Data Unbound

Earlier in the week, Erik Wilde, Eric Kansa, and I published our technical report Proposed Guideline Clarifications for American Recovery and Reinvestment Act of 2009, a set of technical guidelines for how we think recovery.gov should publish data about how stimulus money is being spent and a prototype of what people can do with the data if data were published accordingly.  Here's the abstract of our report:

The Initial Implementing Guidance for the American Recovery and Reinvestment Act of 2009 provides guidance for a feed-based information dissemination architecture. In this report, we suggest some improvements and refinements of the initial guidelines, in the hope of paving the path for a more transparent and useful feed-based architecture. This report is meant as a preliminary guide to how the current guidelines could be made more specific and provide better guidance for providers and consumers of Recovery Act spending information. It is by no means intended as a complete or final set of recommendations.

The technical heart of the work would be the XSD schemas for communications, formula block grant allocation, and weekly report feeds. But the most fun part is to looking at how some fake data would appear, displayed in a mashup of Google Maps and Simile Timeline. We made up some data because at the time of analysis, there wasn't much in the way of real government data to use. We hope that situation will change soon.

What next for me on this front?  Revisiting the questions I posed in my March 7 post (Some questions about the implementation guidelines for the recovery feeds) to see whether I can now answer them.

The Congressional Biographical Directory contains entries for every congressperson from 1774 to the present.  Each congressional representative is associated with an identifier (a bioguide ID).  For example, the bioguide ID for Edward (Ted) Kennedy is K000105.  With this ID, you can determine the URL for the coresponding biographical directory — e.g., Kennedy's is

http://bioguide.congress.gov/scripts/biodisplay.pl?index=K000105

I would like to make use of the bioguide ID in interacting with Freebase with respect to congresspeople.

http://www.freebase.com/view/en/ted_kennedy

hit explore:

http://www.freebase.com/tools/explore/en/ted_kennedy to see

Outbound key(s):

What's the MQL query to read all the keys for the topic?

{
  "id" : "/en/ted_kennedy",
  "key" : [
    {}
  ]
}

we get among the various keys

{
  "namespace" : "/user/jamie/sunlight/bioguide_id",
  "type" : "/type/key",
  "value" : "K000105"
}

Keys are new to me — so I need to do a bit of learning right now.   Now, let's note the following

• query to read all senators in PolDB listing a bioguide_id whether it's there or not.
• query to read all senators in PolDB w/o a bioguide_id

Let's now figure out how to write the bioguide ID for one of the senators without the bioguide ID:  Jeanne Shaheen facts – Freebase. Her bioguide_id is S001181. Here's a MQL write query that writes the bioguide_id to Freebase:

{
  "id" : "/en/jeanne_shaheen",
  "key" : {
    "connect" : "insert",
    "namespace" : "/user/jamie/sunlight/bioguide_id",
    "type" : "/type/key",
    "value" : "S001181"
  }
}

Things to figure out:  how to create keys in the first place in the freebase UI and in MQL.  I think regular users can create keys but I'm not aware of how to do so in the Freebase UI.  I didn't even see a way to insert the bioguide_id using the Freebase UI.

A project that Erik Wilde and Eric Kansa (colleagues at the School of Information at Berkeley) and I have started tackling is tracking the flow of money from the Stimulus Package (aka the  American Recovery and Reinvestment Act of 2009).  The Obama Administration has set up recovery.gov to "feature information on how the Act is working, tools to help you hold the government accountable, and up-to-date data on the expenditure of funds."   Recovery.gov refers to some implementation details:

To meet these objectives, the President is directing Federal agencies to take critical steps in preparation for the Act’s implementation.  See here, for the White House’s February 9, 2009 initial implementation memorandum and February 18 detailed guidance memorandum.

The detailed memorandum (p. 56) issues the following requirement involving the use of web feeds :

For each of the near term reporting requirements (major communications, formula block grant allocations, weekly reports) agencies are required to provide a feed (preferred: Atom 1.0, acceptable: RSS) of the information so that content can be delivered via subscription.

Erik has been leading our efforts in making sense of the implementation memoranda — you can track his findings in a series of blog posts (listed in reverse chronological order, of course):

• Finding Stimulus Agencies
• Stimulus Feed #2
• Finding Stimulus Sites
• Stimulus Feed #1
• Stimulus Feed Details
• Atom Feed Autodiscovery
• Finding Stimulus Feeds
• Stimulus Feeds Done Right

As we've read through the memoranda, we've been confused by a variety of matters, ones which we can hope others can help us with:

1. On p. 56,  you find "Note that the body of the email should include the appropriate completed template as an attachment and should include the name, title, and contact information for the submitter. Templates for these files can be found at https://max.omb.gov/community/x/doC2Dw"  The URL requires a password to access.  Assuming that the template is not some state secret, can someone make those templates available to the public?
2. Does anyone know of a way to get a list of all "federal block grant programs" relevant to the Stimulus Package?
3. How to get a complete list of  CFDA Program Number?  One way is to go to http://www.cfda.gov/ and download the latest 2000+ page pdf catalog and scrape the list.  Is there an easier way?
4. How to get a list of all Treasury Account Symbols (TAS)?  I think I found a list at http://www.fms.treas.gov/fastbook/, specifically the FAST book (in Word format).  I looked for one of the TAS mentioned in a  stimulus feed from the DOJ 15-0402-OJP and couldn't find that TAS specifically.  I'm not surprised since this might be a new TAS — but should we expect to start seeing the new TAS in the FAST book?
5. Anyone help us decipher Treasury Appropriation Fund Symbols (TAFS). TAFS seem to be a primary mechanism for agencies to distinguish recovery vs non-recovery spending in its reporting. On p. 6, we read  "Agencies must establish unique Treasury Appropriation Fund Symbols (TAFSs) in their financial systems for all Recovery Act funding, unless a waiver is granted by the Director of OMB by February 25th."   Can we get help to get that list of TAFSs?  On p. 32, we have "OMB will post the list of TAFSs on the Budget Execution and Recovery Funding page of the Budget Community; the URL is https://max.omb.gov/community/x/-4BeDw "  Is there any reason that list should be available only to government employees?
6. Can someone show how to map the detailed list of spending items in the Stimulus package (such as the analysis at ProPublica) to Treasury Account Symbols?

Any help with any of these questions would be greatly appreciated!

I'm co-leading a meetup around government transparency in San Francisco next Wednesday, March 11, 2009:

Democracy Web

President Barack Obama has promised an era of unprecedented transparency for the US government. In anticipation of vast flows of data from the federal government in the weeks and months to come, we are organizing a SF Bay Area local interest group around tracking and interpreting this data. An immediate catalyst and focus for this first meeting is the Sunlight Foundation's Apps for America Competition (with entries due March 31, 2009). However, we welcome folks interested in the larger topic of government transparency, whether they plan to take part in the immediate competition.

Location: Metaweb Technologies 631 Howard St, 4th floor San Francisco, CA 94105
Time: 6:30pm

All are welcome  but please RSVP on  meetup.com.