Since the news broke that the current budget negotiations in the U.S. Congress placed many of the open government data initiatives squarely on the chopping block there has been much consternation within the data community. Much has been written in defense of these sites, which include data.gov, USASpending.gov, paymentaccurancy.gov, and others. The Sunlight Foundation has even started online petition to rally support. For my part, I believe this is yet another ridiculous attempt to frame the budget debate in terms of minor expenditures rather than focusing on the endemic problems in entitlement and military spending. But I digress.

Clearly these sites are of massive value to researchers, as they provide—in some cases—extremely granular information about government activity. If these sites were shutdown it would certainly affect the work of many scholars, journalist, and active citizens. Likewise, however, in that event the government is not going to suddenly stop collecting data. As the other side of this debate has pointed out, those interested in gathering this data can always make formal requests to receive it, and most agencies are bound by statute to provide it. For researchers working on relatively protracted timelines this will not be catastrophic loss, and journalists were breaking stories using government data long before these sites existed.

The question then is: who really suffers?

One argument I have seen and heard is that there are now many companies and startups using this data to provide tools and services that are a direct byproduct of this level of government transparency. Much of the budget debate has centered on returning the U.S. to prosperity. Politicians often pay lip service to the value of job creation, so perhaps a negative consequence of this proposal would be the loss of these newly created jobs. Anecdotally I have observed the rise of these firms, as I have seen presentations and participated in hack-a-thons that focused exclusively on government data. But is there actual evidence of such a trend?

One way to test this is to count the number of firms in the 'government' and 'data' space that have been founded over the last several years. Since I am primarily interested in technology companies the best source for this information is CrunchBase. This is an open database on all things related to technology firms, and provides a very convenient API for querying. One drawback of the API, as I was able to understand it, is that you cannot query it using Booleans. In my case, I was interested in companies that matched the terms 'government' and 'data,' but had to actually perform both searches separately and then take the intersection.

As such, the companies I focused on lie at the center of the above Venn diagram. That is, their description in the CrunchBase database include both the words 'government' and 'data.' I am perfectly aware of the limitations of this approach for the analysis. There are likely companies in the data set that are not representative of the trend I am attempting to analyze. Furthermore, the CrunchBase database is full of holes, and many companies that met the search criteria did not include founding date information and thus were ignored. Bearing all that in mind, however, the results remain quite interesting.

The above graph shows the frequencies of companies in the dataset founded each year between 1950 and 2010. The blue bars are the raw frequencies, and the smooth red line is a kernel density estimate. Clearly, starting in the late 1990's and through the mid-2000's there was a huge rise in the number of companies working in this space. Since then, however, there has been a decrease.

This result is in stark contrast to my assumptions coming into this analysis. Given the anecdotal evidence I mentioned, I assumed there would have been a steady rise over the past several years, rather than a decline. Perhaps someone who is more knowledgeable of the CrunchBase data can provide some insight as to why? Or, even better, someone in the government data space can provide alternative evidence.

As a final thought, regardless of whether the numbers are increasing or decreasing, this simple exercise shows one important thing: there are many companies already working with government data. It is very difficult to know whether shutting down open government sites would stymie the growth of firms in this space. What is clear is there are already many companies, those under the large curve from 1990-2010, that could be negatively affected by this decision. For the U.S. Congress the important question is: does the ends justify the means?

Code used for analysis