[gist id=5625043]

in the text of this post, I can embed https://gist.github.com/rdhyee/5625043 into the post to get:

Open data — data that is free for use, reuse, and redistribution — is an intellectual treasure-trove that has given rise to many unexpected and often fruitful applications. In this course, students will 1) learn how to access, visualize, clean, interpret, and share data, especially open data, using Python, Python-based libraries, and supplementary computational frameworks and 2) understand the theoretical underpinnings of open data and their connections to implementations in the physical and life sciences, government, social sciences, and journalism.

"""
compare eccoext.ini files

My goal is to edit the new file so that any overlapping values take on the current value

"""
current_file_path = "/private/tmp/14868/C/Program Files/ECCO/eccoext.ini"
new_file_path = "/private/tmp/14868/C/utils/eccoext.ini"
updated_file = "/private/tmp/14868/C/utils/updated_eccoext.ini"

# extract the key value pairs in both files to compare  the two

# http://docs.python.org/library/configparser.html
import ConfigParser

def extract_values(fname):
    # generate a parsed configuration object, set of (section, options)
    config = ConfigParser.SafeConfigParser()
    options_set = set()

    config.read(fname)
    sections = config.sections()
    for section in sections:
        options = config.options(section)
        for option in options:
            #value = config.get(section,option)
            options_set.add((section,option))

    return (config, options_set)

# process current file and new file

(current_config, current_options) = extract_values(current_file_path)
(new_config, new_options) = extract_values(new_file_path)

# what are the overlapping options
overlapping_options = current_options & new_options

# figure out which of the overlapping options are the values different

for (section,option) in overlapping_options:
    current_value = current_config.get(section,option)
    new_value = new_config.get(section,option)
    if current_value != new_value:
        print section, option, current_value, new_value
        new_config.set(section,option,current_value)

# write the updated config file

with open(updated_file, 'wb') as configfile:
    new_config.write(configfile)

The Maryland Institute for Technology in the Humanities will host a two-day workshop on developing APIs (Application Programming Interfaces) for the digital humanities. The workshop will gather 40-50 digital humanities scholars and developers, who along with industry leaders will demonstrate their APIs during this “working weekend.” We will discuss ways that existing and future APIs could be leveraged for digital humanities projects.

As someone who has been fascinated by APIs for years, I hope to learn a lot from my fellow digital humanities about what they care about. One of my tasks is to give an introductory talk about APIs.  What do I want to cover?  I'm still working out the exact structure, but the following topics come to mind:

• What are APIs.  The relationship between web APIs (the focus of our workshop, I believe) and other APIs
• How to learn more about APIs
• APIs of specific interest to the digital humanities, with specific references to Freebase, Google geo-APIs, and OpenLibrary (organizations represented by fellow presenters)
• Why does REST matter. (I'll only anticipate what fellow speaker Peter Keane will be bring up in his talk about REST)
• How to consume APIs; What are mashups
• How to deploy APIs
• Open questions I think about

Stay tuned. Over the next two weeks, I'll work through these topics for myself (writing on this blog). I'll take this time as an opportunity to revisit what I wrote  in Pro Web 2.0 Mashups:  Remixing Data and Web Services and what I taught in my Mixing and Remixing Information course I taught at UC Berkeley over five years.

A lot has happen since I started dataunbound.com, the weblog.  I left my staff position at UC Berkeley so that I could devote myself more fully to the task of teaching others about the world of web APIs and mashups.   I wrote my book on the subject Pro Web 2.0 Mashups: Remixing Data and Web Services, which has been very well-received, I'm pleased to say.  Right now, I'm teaching my course Mixing and Remixing Information for the fifth time at the School of Information at UC Berkeley.  This year, I'm focusing the course on the rapidly expanding area of open government and the web.

And now, I (in partnership with my wife, Laura Shefler) have taken the next step of formally starting Data Unbound LLC:

Data Unbound LLC is a training and consulting company that helps organizations access and share data effectively. The value of your data, when it is scattered throughout multiple databases and applications, grows if you can make it all work together. This value increases further when you leverage your information resources with the vast world of data on the Web. Our specialty is helping you to use APIs (application programming interfaces) to integrate data across your organization and beyond.

We're open for business, ready  to work with clients  to solve their data problems.  Our training will enable their organizations to integrate data, both their own and that of others through APIs and data standards.  I encourage you to read more of what we have written on dataunbound.com, in which we detail our approach and our offerings.  In the next months, I'll be describing how general principles behind data integration and web APIs can solve your  problems in your specific context. And if you know anyone who make use of Data Unbound, by all means, put them in touch with us.

1. the worksheet entitled "92_AARP_TAFS_DD_Detail" in May 8, 2009 weekly report from USAID
2. a pdf published by ProPublica on April 1, 2009.

We specifically asked for an up-to-date Excel spreadsheet with the same columns as the worksheet "92_AARP_TAFS_DD_Detail" — but with an explanation of what each of the columns meant.  We  also encouraged the OMB to make this data available on an ongoing basis as an XML document published on the OMB website and kept up to date, with an explanation of each field.

Last week, we got what we asked for:  an Excel spreadsheet ( see Internet Archive metadata), which I've also uploaded as a Google spreadsheet.  Note the description of the spreadsheet to be found in the first sheet:

In a letter dated August 24 to OMB's Freedom of Information Officer, you requested that OMB provide you with an up-to-date Excel spreadsheet with the same columns as a worksheet you emailed on October 16. The Berk_FOIA_Data tab in this Excel file provides up-to-date information using the same columns in the file you sent. The information is up-to-date as of October 19, 2009, and shows a list of each Treasury Appropriation Fund Symbol (TAFS) associated with the Recovery Act (RA). Below is a description of each column in the Berk_FOIA_Data tab.

I've not had an opportunity to complete my analysis of the FOIA spreadsheet  and to correlate the data to the recipient reporting.   You'll note that there are 342 TAFS in the spreadsheet.  To derive a list of Treasury Account Symbols (TAS as opposed to TAFS), we concatenate the  Treasury Agency Code with the Treasury Bureau Code (separated by a '-') and bundle all  the corresponding TAFS.   See the resulting list, with a total of 313 TAS.  You'll note that a spreadsheet that lists the TAS as of Sept 13, 2009 has 309 symbols, while the HTML list on federalreporting.gov currently lists 327 TAS (along with 32 place-holder symbols).   The differences in those lists is something to nail down next.  At any rate, even something like the list of Treasury Accounts associated with the Recovery Act is more fluid than what I would have expected at this point.

One thing that has puzzled me is why there are so many TAFS with $0.00 for the treasury warrant.  You find an explanation in the FOIA spreadsheet:

Treasury Warrant is the sum that Treasury warranted to the TAFS. You can think of a warrant as being the initial deposit in a new checking account. For many of the TAFSs on the list, you can track the amounts appropriated in the law to the amount of the Treasury warrant. In some cases, however, you cannot track back to actual amounts because the funding in the law is formula based. In many cases, a TAFS has a zero in the Treasury Warrant column. The primary reason for this is that these TAFSs receive RA funds via a transfer from other TAFSs.

Hmmm.  We're going to have to understand the relevant formulas.

Acknowledgement:  A big thanks to Brian Carver for providing us valuable advice on how to formulate, draft and send a FOIA request and helping us to interpret what's happening during a FOIA process.

Here are some highlights from our report, which I quote (with a bit of editing):

• Design priorities for recovery.gov need to shift from focusing on deploying an attractive Web site toward designing ARRA web services to support reuse of data in third-party applications.
• These services should allow any party  to receive the complete set of ARRA reporting data in a timely and easily usable manner, so that in principle, the full functionality of Recovery.gov could be replicated by a third party.
• Our proposed architecture is based on the principles of Representational State Transfer (REST) and always attempting to use the simplest and most widely known and supported technology for any given task.
• We recommend the feed-based dissemination of ARRA reporting data using the most widely used technologies on the Internet today: HTTP for service access, Atom for the service interface, and XML for the data provided by the service. This approach allowing access from sophisticated server-based applications or from resource-constrained devices such as mobile phones.
• The manner which data flows from FederalReporting.gov to Recovery.gov is of critical importance. Ideally, Recovery.gov should use Web services offered by FederalReporting.gov.
• We strongly recommend that Recovery reporting systems adopt the Atom syndication format for feeds.  Feeds represent a major positive development in making government data more open to citizen review and reuse and provide a unique ability to do so by merging utility for humans as well as machines.
• While not formally standardized, feed autodiscovery is well supported by current browsers and could be implemented reliably with a well-defined set of implementation guidelines for Web pages offered by Recovery.gov.
• We strongly recommend making feed paging and archiving mandatory, so that the feeds are not just a temporary way of communicating that information has become available. Instead, the feed pages should be available as persistent and permanent access points, so that accessing information via feeds can be done robustly and reliably.
• ARRA data dissemination services should be more resource-oriented than service-oriented.  XML representations should contain links (in the form of URIs) to related data resources, thereby representing the relationships between the different concepts which are relevant for reporting.
• The Recovery reporting schema uses many different coding systems and identifiers. Publication of resources related to some of these identifiers will be of great value.  (We list key identifiers in the report.)
• There are many possible analyses that people may wish to perform on Recovery data,  making it difficult  to accommodate them all. Therefore, querying services should be oriented toward making machine-readable representations of data available, so that third party developers can easily populate their own analysis engines and run their own specialized algorithms on that data.

Erik Wilde has also commented on our report. We welcome and look forward to your feedback.

Finally, we are grateful to the Sunlight Foundation for a grant that helped to support this effort.

The Recovery, Accountability, & Transparency Board wishes to have an open discussion with all interested developers about how data should be made available via Recovery.gov.

As you are all aware, a new version of Recovery.gov will be released soon. From a data standpoint, the initial release of the new site will replicate existing functionality. However, the Board aims to set a new standard of transparency with this site and would therefore like to make the data available in the most convenient and straightforward way (or ways) possible so you can use and analyze official, up-to-date Recovery Act data. We need your input to achieve this goal.

Please let us know how the site could best meet your needs in terms of  machine-readable data format(s) and standards, APIs, guidance, training, etc. [emphasis mine]

As I waited for Rusty to respond to my question of how best to provide feedback, Luigi Montanez went ahead with posting a series of excellent pointers.  I second Luigi's advice, also commend  the recent OMB Watch Recovery Act Transparency Status Report)  and have similar general web development advice to offer, which I had written up as "Making Your Web Site Mashable" (pdf)  (Chapter 12 of my book Pro Web 2.0 Mashups).

In terms of work specifically related to the Recovery Act. my Berkeley colleagues Erik Wilde, Eric Kansa, and I published a report "Proposed Guideline Clarifications for American Recovery and Reinvestment Act of 2009" in which we proposed and prototyped  the use of Atom feeds to disseminate Recovery spending data.  We are currently at work on updated recommendations based on the latest Recovery Act OMB Guidance.

One of my most important things that has made the Recovery spending less-than-transparent is how difficult it has been to locate basic accounting data.  For example, after looking for months, I have yet to locate a reliable list of Recovery TAFS, basically a list of all the pots of money (as tallied by Treasury) and the maximum amount of money we expect to see in each pot (the dollars appropriated).  Now, Recovery.gov does list the amounts obligated and spent by agency, but how much money has been appropriated?  That basic data should be clearly documented at Recovery.gov, so that we can track the flow of money reliably from the originating legislation to Treasury out to the agencies  and then to contractors and grantees  or the states.  (I will note that ProPublica's Stimulus Tracker does break down the totals by agency but doesn't publish the list of individual accounts.)

At any rate, there is more to say — but I'll wait until Rusty responds to what is here.