Solving the Surveyor’s Dilemma

Dr Robin Gaster

Dr. Robin Gaster
rgaster@incumetrics.com
240-462-4409

Note: This article was originally published by the Information Technology and Innovation Foundation on May 4 2020

KEY TAKEAWAYS

  • It is well-established that innovation-support programs like SBIR/STTR offer significant benefits to companies and a return on the government’s investment in the form of new commercial revenues. But conventional evaluations are imprecise.
  • Surveys face one particularly challenging problem: They are essentially snapshots, capturing outcomes at the time they are taken. They don’t capture outcomes afterward. But new data offers a new opportunity to address this issue.
  • The new data suggest that the positive impact of the SBIR/STTR programs is significantly greater than has been reported—and that the return on investment was in fact 22:1 compared with the previous estimate of 15:1.
  • These results can be applied to other evaluations of innovation programs where surveys are used to estimate outcomes.

Introduction

In the United States and elsewhere, government programs have emerged to fill gaps left in the innovation ecology by market failures. In particular, small, innovative businesses may find that their projects do not fit the profile or the timelines required for venture capitalist funding. Their markets may be too small or too difficult to conquer, or the expected time to payback and success may be too long.

Support from programs like the Small Business Innovation Research (SBIR) and Small Business Technology Transfer (STTR) programs plays an important role in filling these financing gaps.1 Offering more than $3 billion annually in nondilutive finding for small businesses, they have funded some notable successes, including Qualcomm, iRobot, and Illumina. Thousands of less-known companies have benefited as well.

Understandably, Congress has tried to make sure that funding is being well spent. A long series of National Academies reports generally established that the programs are well-run and that they offer significant benefits to companies, as well as a return on the government’s investment in the form of new commercial revenues. Those revenues then turn into new jobs and further company growth—sometimes very substantial and long-lasting growth.

Measuring ROI With Surveys Creates a “Snapshot” Problem

Because companies do not report back to agency funders on a consistent basis and over a long enough period of time, the preferred methodology of assessing outcomes is through wide-scale surveys. This methodology was pioneered by the Government Accountability Office (GAO), adopted and enhanced by the National Academies, and most widely deployed by TechNet in a series of surveys for the Department of Defense and its agencies.

Surveys are undoubtedly the best mechanism for identifying program outcomes in detail, although they can also be supplemented by other approaches—for example, using administrative data from the Census Bureau. But surveys have a number of flaws, which are described in the National Academies studies.2

Surveys measuring outcomes for innovation programs like SBIR/STTR face one particularly challenging problem. Surveys are essentially a snapshot. They capture outcomes at the time of the survey. What they do not capture is outcomes that occur after the survey ends. For some kinds of programs, that’s not a problem. But it’s a major challenge for evaluating innovation programs, where the time to market is often long, and the lag even after that can be substantial before a product cycle peaks. In fact, it can be decades before a product is completely outmoded. As a result, innovation outcome surveys substantially under-report total commercialization, but to an unknown degree.

New Data Offers a Better View

Major studies have acknowledged this problem, but they have nonetheless persisted in using surveys to assess outcomes in the absence of better data. Thus the National Academies, for example, clearly identified this problem—before going on to use those results as the best available data.3

However, a set of survey data collected by TechNet for DOD in 2017–2018 offers a new opportunity to address this problem. The size and extent of the new data makes it possible to generate statistically sound estimates for future project outcomes, even though those outcomes have not yet arrived. As a result, we can generate at least a preliminary estimate of the extent to which surveys under-represent commercial outcomes for SBIR/STTR, and by extension for similar innovation programs as well. That research is presented more fully in a recent paper and is summarized here.4

Innovation outcome surveys substantially under-report total commercialization, but to an unknown degree.

The TechNet surveys used substantially enhanced resources from DOD and its agencies to build a data set that encompassed almost all Phase II SBIR/STTR projects funded by the U.S. Navy and Air Force between 2000 and 2013. TechNet acquired about 6,700 survey responses against a universe of 7,216 funded projects—a response rate of 93 percent.

This large sample made it possible to develop a picture of the revenue life cycle for projects. Of course, even such a large sample must still be treated with some caution—the results below may need adjustment as additional data becomes available. Nonetheless, developing a preliminary estimate for median project outcomes over the product life cycle is an important step forward, because it allows us to develop estimates for the future revenues to be generated by products that had not completed their life cycles at the date the survey was administered.

The missing revenues emerge from two distinct sources. First, there are projects that have not exhausted their life cycle at the time of the survey. Indeed, some may have only just reached the market, and are still in the growth phase typical for a recent market entrant. Others may be older but still generating substantial revenue growth. Second, there are projects that have not reached the market at all, but that will do so at some point after the date of the survey.

To estimate missing sales for projects that were funded more recently and have already reached the market, we developed an estimate for median sales for each elapsed year after the conclusion of the SBIR/STTR award. Unsurprisingly, we found that median returns from older projects were indeed much larger than those from more recent projects. In fact, commercial revenues stemming from Air Force and Navy SBIR/STTR contracts completed 16 years before the date of the survey at the median generated more than five times as much revenue as products from contracts ending three years before. The predicted median sales per project with sales, derived from our regression model, are shown in figure 1.

Figure 1: Predicted sales by elapsed year, estimated from regression model

 

Using this analysis as a baseline, we could then estimate the percentage of total revenues not captured by the survey, by elapsed year. For projects surveyed three years after the conclusion of their SBIR/STTR awards, 72.8 percent of revenues were not captured by the survey (see figure 2).5

Figure 2: Percentage of total estimated sales not captured by survey (estimated from regression model)

Applying these estimates to the data collected by TechNet, we can estimate how much of the total return from the program was not captured by the survey data. Overall, across 2000–2013, we estimated about $13.1 billion in company revenues was not captured, amounting to 44 percent of estimated total revenues over the product life cycle.6

Turning now to the second source of missing data, we used a similar approach to estimate the share of projects that had not reached the market, but would eventually do so. For these projects, the returns captured by the survey were of course zero.

Again, we used the 14-year time period covered by the TechNet surveys to estimate the likelihood that projects would at some point reach the market. Using the oldest projects in the data set as a baselines, we estimated that approximately 35 percent of projects will eventually generate some sales (see figure 3). We used standard statistical techniques to generate an estimate for predicted the number of projects that will eventually achieve sales greater than $1,000 by elapsed year. This estimate is roughly in line with those produced for DOD as a whole by the National Academies.

Figure 3: Percent of projects reporting sales

Again, fewer recent projects have reached the market, and hence have any outcomes captured by survey data. Only 17 percent of projects in elapsed year three had reached the market, compared with 35 percent from year 16.

To estimate the additional revenues from these projects, we performed a two-step calculation: First, we estimated the number of additional projects that would eventually reach the market, for each elapsed year, using statistical tools to smooth the pathway to the market over time (see predicted sales in figure 3 above). Then we applied those additional projects to the estimated total revenues to be generated from that elapsed year. The first analytic step indicated that an additional 14 percent of projects that were not reporting sales at the time of the survey would eventually reach the market. The second showed that these additional projects can be expected to generate an additional $1.4 billion in sales over their product life cycles.

Standard methodologies as used in the original TechLink reports captured $17 billion out of a total of $31.5 billion in estimated total project revenues. $14.5 billion—or 46 percent—of the total value was not captured by the surveys.

Together, the missing sales from projects already in the market, and sales from projects that will at some point reach the market provide a preliminary estimate of under-reporting from innovation surveys of commercial outcomes. Standard methodologies as used in the original TechLink reports captured $17 billion out of a total of $31.5 billion in estimated total project revenues. $14.5 billion—or 46 percent—of the total value was not captured by the surveys.

What the New Data Implies

This has obvious implications. To take only one example, the TechNet report used IMPLAN economic modeling software to develop a return on investment estimate for the Federal government’s SBIR/STTR investments. Plugging revised commercial revenues into the IMPLAN model, the mythology described above showed that the already-positive impact of these programs was significantly greater than had been reported, and that the return on investment (ROI)—defined as the ration of funding provided to eventual commercial returns to companies—was in fact 22:1 compared with the previous estimate of 15:1. The estimated employment effect was 50 percent greater as well. In other words, as Congress and the administration consider these small business innovation programs, they should understand that the actual impacts are even larger than conventional estimates indicate