Introduction to the usage analysis

Hitherto only general user activity like the number of registered users and proposed ideas has been analysed. The analysis conducted as part of TID+ project goes much further, finding a number of possible performance indicators, harmonizing the content of TOM, assigning tags (keywords) to ideas and performing a review of answers the proposals have received from ministries as well as using Google Analytics to find the sources of visitors. The analysis thus traces trends in usage, including the number of proposals, their authorship and their subject matter, and identifies the factors explaining variations and patterns in usage.

TOM overall activity: number of new ideas vs website visits

This is the most readily available and most often used data concerning TOM performance from its launch in 2001 to the end of 2006. The number of new ideas, yearly:

The number of ideas, monthly, with first year out-scaled:

We see a sharp decline in the year after TOM was launched because initially TOM received a lot of media coverage and the ideas for legislation proposed were mirrored on the largest Estonian portal delfi.ee - yet after 2004 the number of ideas started to rise again. This can be probably be attributed to overall activisation of citizen initiatives and internet usage.

At the same time number of visitors to TOM has remained notably low - typically 150-200 visits per day, peaking at about 300 when an idea author posts a link to online forum or mentions on weblog. Even in Estonian context this is comparable to slightly popular weblog or homepage of a small company. On the public statistics of Estonian online publications available at http://tnsmetrix.emor.ee that would compare to websites of Baltic Business News newsagency and some programmes of Estonian public radio (r4.ee is russian-language channel, klassikaraadio.ee broadcasts classical music).

TOM user activity

In six years of existence, TOM has attracted 6000 users whilst over a 1000 legislative ideas have been proposed. General activity data for proposals that had passed the process at the moment the study sample was selected (March, 2007) are presented in the following table. In addition 20 ideas were still in various phases of completion and thus were not included in the analysis: 2 were still under discussion, 5 were yet to be voted on and 13 were still awaiting a government response. 2/3 of the ideas have recieved the support of majority of votes, although there have been ideas that have passed with a single vote cast (average votes per idea has dropped from 20 in 2001 to 4 in 2006).

Surprisingly, the number of abandoned proposals, which attracted no votes, is very low (2%). This seems to be due to the presence of regular users and/or frequent visitors, who have stumbled upon proposals and voted on them before the cut-off point (three days following ten days’ potential discussion and a further three days for amendment). However, author participation in voting and discussing one’s own proposals is extremely poor: only 40% of ideas have had author commenting and 34% actually vote for own idea.

This can only be interpreted as a sign that the initial TOM tool was not designed with the promotion of citizen debate in mind: there is no simple method for notifying authors regarding comments nor is there an integrated multi-step process for linking together commenting, voting and signing. Thus author participation had little influence on the voting result - in this sense TOM has not created an author-led process of citizen interaction.

Interestingly the TOM website displays two different versions of stats that don't agree with eachother and are wildly off the mark on what might be considered key performance indicator: number of ideas that have received and answer from government as "Vastatud ideed" and "Ideedele saabunud vastuseid" (slight difference in total number of ideas is due to the screenshot being taken at different time):

So the main indication from the service is, that only 13% ideas get an answer - while in reality it is 89% which we consider surprisingly high:

How TOM-generated ideas fared: government responses

It is commonly assumed that most government responses to TOM-generated ideas are negative. To verify this claim, we have examined every single government response and categorized them according to the nature of the answer: those that explain how the problem can be addressed using existing legislation; those informing TOM users that the solution to the problem is already in the pipeline as an amendment to current statutes; those expressing a possible implementation of the idea (see Possibly implemented ideas); those generally supportive; those that were negative; and, as a separate category, TOM-generated ideas that received positive answers and were implemented at least in part. This categorization is of course partially subjective since lengthy answers could often contain criticism and praise, thus we categorized answers as positive if at least some element of the proposal was deemed worthy.

TOM-generated ideas by subject matter

One of the drawbacks of the current version of TOM is the inability to categorized proposals by subject matter. The absence of subject categorization creates several problems: it impedes citizens’ ability to find or track topics that are of interest; leads to duplication of proposals; hampers the process of learning from already posted government responses. For the purposes of the analysis of subject matter, therefore, all TOM-generated ideas were tagged with a set of keywords and the database thus created was posted to the social bookmarking site del.icio.us that contains a striking visual method for highlighting common topics, as can be seen in the graphic below. The categorization used can be seen at http://del.icio.us/tom.al.icio.us

The following visualization is generated from assigned keywords using the extisp.icio.us service at http://kevan.org/extispicious.cgi?name=tom.al.icio.us

The following table is a dual-language glossary of the most common subject matters for TOM-generated ideas:

• liiklus - traffic (parking, speed limit, Driving Under the Influence, insurance, driving licence)
• maksud - taxes (proportional vs progressive income tax, VAT on books purchased online)
• noored - youth (alcohol-tobacco-drugs issues, education, crime)
• pere - family (various forms of state family benefits)
• eestiasi - related to Estonia (from citizenship to having a national anthem online)
• põhiseadus - constitutional matters
• keskkonnakaitse - environmental protection
• riigikogu - parliament (mostly concerning the emoluments of MPs)
• eriik, eteenused - e-government, e-services
• valimised - elections
• alko - alcohol policy

Finding most active users

As there is no incentive to sign up, compared with simply visiting, except for authoring, commenting or voting it is particularly interesting to note that only 45% of users have performed any action after signup (it would be fruitful to compare this percentage with that from other systems that do not require a login to read content).

While the above table lists the average activities per user, the table below dhows the average activities for percentiles of users who have been active in their respective precentile category:

While we could interpret the above data as a sign, that a sizeable proportion of active users has performed more than one action we get completely different result when we look at the proportion of activities by top active users. Following table sums actions by top 10% of users with number and percentage of active users producing 50% of actions on bottom, together with number of actions performed by the user at the 50% position. Please note that ComIdeas is not number of comments like counted for total activity, but number of ideas commented by user so it should be better comparable to ideas and votes.

It is worth noting, that in fact 10% of ideas are generated by a single user (followed by another user with 3% of ideas; the top nine users have each contributed at least 1% of all ideas in system, whilst the top 10 users have generated 25% of ideas).

Distribution like this is can be expected, as explains Clay Shirky in Power Laws, Weblogs, and Inequality - in fact, if the top user is left out, the number of ideas generated follows very nicely a power law distribution of userIdeas(r) = 50 * r ^ 0.71

Although expected this distribution might not be the optimal solution for increasing citizen participation: the system is dominated by a small number of mega-users (<100 in TOMs case) who place a heavy burden on resources, most importantly the officials compelled to respond to the proposed legislative ideas but also administrators and other users.

TOM traffic sources

Google Analytics, a free service provided by the IT corporation Google, generates statistics about website visits, including such elements as where internet traffic came from, length and frequency of visits. According to Google Analytics, actual TOM usage is very low - slightly more than a 100 visits on a typical day, but interestingly usage can double on certain occasions:

All the peaks illustrated above are the result of an idea being discussed outside the TOM tool. For instance, the peak of 9 January 2007 highlighted above was generated by the auto24.ee discussion board, the website of a major daily newspaper epl.ee and two weblogs as demonstrated in the next graphic. This is a crucial finding, which suggests that public interest in e-participation is greatly dependent on how the tool for citizen participation is publicised among internet users, especially the weblog community.

At the time of the 9 January 2007 peak, the most active TOM-generated proposal was idea number 2050:

Proposal number 2050 proposes a solution to an urban parking problem. This particular problem arises from the fact that in Estonia there is no legislation permitting bad parking to be classed as a traffic or parking violation because of constitutional due process applicable for such sanctions. Certain European countries circumvent this constitutional obstacle by having the regime of fines imposed for such traffic violations classed as local taxes raised on parking mistakes. However, no such legislation has been introduced in Estonia, which prompted TOM-users to ask for such a measure. This particular proposal was then discussed on popular forums and in comments to a newspaper article as well as two blogs written by the author of the TOM idea. The following is a snapshot from one of the TOM-user's blogs illustrating nicely the problems associated with the absence of such restrictions on parking violations. It was precisely by publicising the policy issue across various websites that internet traffic directed towards TOM hit a peak.

Incoming search engine keywords

To complete our analysis of traffic source for TOM users and visitors, it was necessary to identify the internet search engine keywords that brought people in contact with the TOM tool. As can be seen from the following top-10 list of keywords generating TOM traffic, the hit parade unsurprisingly consists of expressions related to the site's name but there are also two real names of TOM idea authors/voters (blurred here for privacy reasons):

When examining search statistics beyond the top-10 traffic-generating keywords, it becomes obvious that a very notable amount of inbound traffic is generated by searches for the names of people who happen to have participated in TOM - it is impossible to say whether the searches were conducted because the name searched was known to have authored a proposal on TOM, although prima facie this possibility seems highly unlikely. Out of 5435 search phrases 1955 (35%) are names, out of 8783 search instances 3404 (39%) are names. TOM ranks pretty highly in Google searches (often 1st or 2nd page) so it is not unusual for a search for a person's real name to bring up the idea they have proposed or voted on prominent position. Again, without real usernames, here is a list (difference in number of visits when comparing to above illustration due to consolidating searhces related to same person):

Overview

This section has reviewed the quantitative data relating to TOM general user activity since its launch in June 2001 until March 2007, when the current analsysis was conducted. Whilst the total number of users and legislative proposals was easy to establish, this analysis broke new ground by using a variety of data-analysis techniques to trace usage over time and explain fluctuations therein. In addition, every single TOM-generated idea was tagged according to subject matter to show what type of proposals TOM users generated. Moreover, we were able to trace usage statistics of the individual authors of TOM ideas to show the distribution of ideas generation amongst the TOM community. Finally, using innovative internet tools for tracing internet traffic, the report was able to determine the sites and search engine keywords that brought outside internet users to the TOM site.