ROI on Apps for Democracy?

Hopefully, you have heard about Apps for Democracy (AfD), an innovation competition created by Vivek Kundra of the Office of the CTO of Washington, DC and Peter Corbett of iStrategy Labs.

Innovation competitions, properly designed, can bring a lot of value. I find it interesting to observe the different ways that innovation competitions are justified. In many cases, the ra-ra-ra behind collective intelligence seems to be a sufficient bandwagon. In the case of AfD, I’m glad to see that the parties involved put some thought into boiling it down to a metric. Often you have to invest (spend) money in order to achieve savings. That’s why the ROI (return on investment) metric is useful. So what is (or what was) the ROI of Apps for Democracy?

ROI

According to an iStrategy Labs blog post, the ROI was estimated at 4,000%:

* OCTO has estimated the value of these submissions at $2,000,000+ including external contracting costs and internal procurement time
* The cost to OCTO was $50,000 including prizes, marketing, management etc. representing an estimated 4,000% return on investment

According to OhMyGov.com:

“If you do the math on the return of the investment, it’s 5,100 percent. That return is based on the fact that it would have taken a year to procure each individual application themselves,” Corbett said.

A ROI of between 40X and 51X — this sounds too good to be true. I don’t buy it.

Digging In

While I appreciate the importance of boiling down a message — and I appreciate that top decision makers do put a big importance on an ROI estimate — such a figure is only the start of the conversation, not the end of it.

Calculating ROI, after all, is not simply about doing the math of dividing numerator by denominator. It is about deciding what the numerator and denominator should be. The essence is deciding what to include and what not to include.

So what are the right things to include in the ROI calculation for Apps for Democracy? We’ll need to be specific. (Perhaps in a later post I can attempt to generalize about ROI for innovation competitions in general.)

Goals

You can’t sensibly talk about ROI unless you clearly define the scope and purpose of a program.

The short-term and long-terms goals are summarized in a press release from November 13, 2008:

“While the immediate goal of the Applications for Democracy contest is to develop innovative software to present District data, its long-term goals are broader,” said District CTO Vivek Kundra. “By making government data easy for everyone to access and use, the District hopes to foster citizen participation in government, drive private-sector technology innovation and growth, and build a new model for government-private sector collaboration that can help all governments address the technology challenges of today and tomorrow.”
…
Entries were judged by an appointed jury based on criteria including usefulness to citizens, usefulness to government and originality

Let’s take these goals as given for the purposes of argument and ask some follow-up questions:

Questions

1. The cost side of the calculation is relatively easy to measure. The contest gave out $20,000 in prize money, meaning that iStrategy Labs was paid $30,000.

2. How was the benefit estimate done exactly?

3. If one uses the rough process of taking the estimate benefit given above (B = $2,000,000) and dividing by the number of applications (N=47), then you get a benefit per app of approximately $42,500. This estimate seems implausible to me.

4. Was there any accounting of the quality and functionality of each app? Did all the apps work?

5. Was there any accounting of duplication of effort? If two apps did roughly the same thing, do they both get counted?

6. To what degree, if any, does DC “own” the projects generated from the competition? By ownership, I mean code ownership, domain name ownership, operational ownership, and oversight ownership. (Note: I am *not* saying that ownership is the key factor, just saying that it needs to be factored in.)

7. Related to the above, were all of the projects required to open source their code bases?

8. How was “internal procurement time” accounted for?

9. Are we comparing apples to apples? Using a normal procurement process would result in a different kind of ownership in the projects.

10. How long will the projects be around? If one vanishes tomorrow, what happens to the ROI calculation?

11. Did the ROI calculation make any accounting for alignment with their mission? If an app did not align with the mission, was its value discounted?

12. If the mission was to raise awareness of the DC Data Catalog, it would be good to gather statistics about their usage. Speaking of oversight, does DC have some mechanism for observing the apps and seeing how they do, traffic wise? Does it have an idea of how much traffic it drives to the Data Catalog?

I’d appreciate if we could dig into these questions together. Please use the comments to share your thoughts.

Finding New Ground

Based on the questions above, I find it implausible that the benefit of AfD is as high as $2,000,000.

It is easy to do a quick back of the envelope calculation and arrive at a really high ROI. It is easy to point to the unbridled potential of crowd-sourcing or the potential of social networks. But these approaches are just wild guesses — relying on them will likely bring about a backlash.

So far, I have heard too few grounded discussions about the tough questions about innovation competitions. In the spirit of accountability, we need to have a discourse about what matters and how we can design our competitions to emphasize those objectives.

I’m a supporter and an evangelist for innovation competitions, provided that they are consciously designed. But I have seen too many technologies get oversold. I do believe that there are compelling reasons to go with an innovation competition, but I think we might want to have a more nuanced way to articulate and value them.

Furthermore, I don’t want government agencies to simply jump on the innovation competition bandwagon out of mania.

There are real, tectonic shifts happening with the social Web, but an organization needs to embrace it on their own terms and for their own purposes. I want organizations, especially government and non-profits, to actively design innovation competitions so that they meet their needs. One size does not fit all.

Sweeping (but over-generalized and perhaps premature) claims such as “States, counties and municipalities please take note. You must enable and encourage your IT departments with a similar strategy” are bold and exiting. I believe in the potential of the democratization of data. But such boosterism must be counterbalanced by a hard look at accountability, transparency, benefits, and costs.

A Beginning

I am happy to see Apps for Democracy (AfD). It is a starting point for DC, not a proven standard for all the world to blindly copy. We owe it to ourselves to ask the tough questions so that we can iterate and improve it.