Charles Wessner and Robin Gaster

May 2008

One consistent criticism of the SBIR program is that some firms “game” the system by winning numerous awards but then fail to commercialize. These frequent award winners (often referred to as “SBIR mills,”) are criticized for using funds that could be better spent on firms more closely focused on a primary objective of the program, namely commercialization of their research.

This critique has gained some traction; some Congressional staffers believe that limits should be placed on the number of awards a single firm can win. NSF firms are limited to four applications annually.

We believe both the critique and the resulting policy adjustments are profoundly misplaced: the problem of the “mills” is indeed a myth, for several interlocking reasons:

first and foremost, the dimension of the problem is overstated. Given the size of the program, only a small number of firms, indeed very few, win any significant number of awards, and collectively these multiple award winners garner only a modest share of program funding;
the purported consequences, i.e., less commercialization, is also overstated: overall, firms that win multiple awards generate more commercial returns than other firms in the program, not less;
commercial returns are only one objective of the program. Others are equally important and there is strong evidence that these firms help agencies meet these objectives;
some of the most criticized firms are no longer eligible for the program; and finally,
the cure will be much worse than the disease

1 How many mills are there? And how many awards do they get?

Agencies do not systematically publish their own lists of multiple award winners. And the tech-Net database is not guaranteed accurate, especially as searches by company name may understate extent of award concentration. Still, Tech-Net data provide at least an approximation of the overall situation at all agencies:

Table 1. Multiple award winners, by number of awards, 1992-2005[2]

		% of total
Top 1	151	1.0
Top 5	454	3.1
Top 10	850	5.7
Top 20	1,276	8.6
Top 50	2,012	13.6
Remainder (6336)	12,812	86.4
Total	14,824	100.0

Looking across all five major SBIR awarding agencies,[3] we find that according to the SBA, there were between 1992 and 2005 14,824 Phase II awards made, to a total of 6,386 firms. Two firms received more than 100 awards. Overall, 28 firms received at least 28 Phase II awards during this period – an average of at least 2 per year. Only 15 firms received at least 3 awards per year. [check].

Collectively, the top 20 award winners received 5.7% of all Phase II awards during this period; the top 50 received 13.6%. In contrast, the remaining 86% of awards were spread across 6,336 companies.

Clearly, some companies do win more awards than others. The most prolific won about 11 awards per year – a sizeable amount, generating revenues on the order of $3.5M annually.

To put this amount into perspective, DoD maintains a database of research, development,. Testing and evaluation (RDT&E) contracts.[4] In 2004, EDS was the top contract winner, with an astounding 10,065 contracts valued at well over $1.5 billion. The top 10 averaged more than 6,000 contracts each, and more than $7 billion in annual revenue.

2 What commercial results do they generate?

One critique leveled against SBIR “mills” is that these are essentially contract research operations, providing sufficient services to attract more SBIR awards, but never making any real effort to commercialize their work independently of the SBIR program.

This argument is misleading, on two grounds. First, contract research is itself potentially valuable to agencies in meeting their specific research needs – a specified Congressional objective for the SBIR program. Second, the data show that the mills do in fact in aggregate commercialize.

Over time, some firms have built substantial and versatile research capabilities, and have become reliable research vendors to government agencies. When called on, these firms can provide specific research in response to closely articulated questions whose answers are needed by the government, especially in the procurement agencies. Some of these answers are extremely valuable, closing off potentially expensive lines of futile research, for example, or providing informed, reliable answers in a timely and cost-effective manner. The legitimate role of these firms in the SBIR program is discussed in section 3 below.

So far as commercialization itself is concerned, the plain fact is this: looked at in aggregate, firms with the largest numbers of awards generate more commercial results, not less, than firms with few awards.

The following table is taken from the DoD CCR database, which closely tracks commercial outcomes related to SBIR awards at DoD and other agencies. It is the most accurate available gauge of outcomes at DoD, and provides some insight into activities at other agencies as well. However, as the “mills” argument is applied most often to firms preponderantly operating within DoD, it is especially relevant here:

Table 2.

Data taken from the DoD Company Commercialization (CCR) Database
# of Phase II SBIR per Firm*	# of Firms	# of Projects in CCR database	# of Projects with Award Years prior to 2004	Average Commercialization of projects with Award Years prior to 2004	Average Sales of projects with Award Years prior to 2004	Average Funding of projects with Award Years prior to 2004
>125 projects	5	941	823	$2,067,719	$1,384,571	$683,148
>75 and <110 (no firms had between 111 and 124 projects)	5	485	411	$1,117,325	$526,623	$590,703
>50 and <75	17	1067	945	$4,103,125	$3,586,611	$516,514
>25 and <50	77	2692	2330	$1,710,140	$1,048,787	$661,354
>15 and <25	101	1858	1535	$1,375,061	$863,310	$511,750
>0 and <15	2715	8101	6243	$1,300,886	$751,418	$549,468
* Awards are all Phase II from any agency, not just DoD awards

Source: DoD Company Commercialization Reports database

These results are telling. They indicate that on average Phase II awards to firms with the most awards in fact generate more sales/revenues per project than do awards made to any other group.

There are a number of reasons why this might be so – such firms are, typically, larger than firms with a handful of awards. They have often been in business longer, and have developed a better marketing network, especially within the somewhat arcane environment of DoD acquisitions.

But the fact remains –according to the survey data, companies with the most awards, criticized for being service organizations not commercializers, in fact commercialize more than companies with fewer awards.

3 Agency mission and knowledge effects: meeting other SBIR objectives

The “mills” argument is also flawed because it takes a single goal of the program as the only goal. This approach has been explicitly rejected by Congress in setting program goals.

Criticism of the “mills” implies that the value of SBIR awards can be judged solely in terms of commercial results. But this is only one objective among the four detailed by Congress in the legislation. And at DoD in particular – which accounts for half of the overall program – a successful SBIR is not one that generates commercial results in the sense of sales on the commercial market, but one that contributes to technology that is eventually built into DoD weapons systems.

The Academy reviewed the relationship between SBIR and agency mission. It focused on three elements:

Selection process. At all the major SBIR agencies, these procedures are designed to make sure that projects are selected in part because they will help to meet the needs of the agencies. We found no projects that were in obvious ways misaligned with agency mission. This is not to say that there are none; it does suggest that such misalignment is not widespread.
Repeat custom. The fact that these firms consistently win awards suggests that they continue to provide effective services to their clients and customers in the agencies.
Specific projects. All of the agencies maintain databases of “success stories” – projects funded by SBIR that were in the view of the agency noteworthy and successful. Numerous projects from multiple award firms are found in these databases.

In all of the Academy’s analysis – and indeed in that provided by those critical of SBIR “mills” – there is no indication that multiple award winners provide less value to the agencies than other firms. If anything, the evidence – both in terms of repeat business (i.e. customer satisfaction) and of successful commercialization – suggests that the agencies are generally satisfied with the performance of these firms.

Beyond agency mission, the Academy also looked at knowledge effects, and found that there is no evidence that multiple award winners generate fewer patents, publications, and other indicators of technical knowledge than do firms with fewer awards, on a per award basis.

4 Ongoing eligibility

It is worth noting that some of the most prolific award winners are no longer eligible for future SBIR awards. Foster-Miller, for example, the most prolific of all, was acquired by a foreign corporation, making the firm ineligible for future awards.

Firms become ineligible because they are acquired, because they become too large to meet SBA size guidelines for participation, because their ownership structure changes in some specific ways, and they become predominantly institution-owned.

Firms that are as successful as the most prolific SBIR award winners may be especially attractive acquisition targets, as the profile of the program continues to rise, and potential purchasers give more credence to the validation effect that accompanies SBIR awards.

5 Why the cure is worse than the disease

Finally, we should consider possible effects of solutions to the “problem” of the “mills.”

Most proposed solutions are simple – they suggest that firms be allowed to apply (or in some circumstances, win) no more than a set number of awards in a given fiscal year once it has reached a certain threshold of total awards won.

This simple solution will not work for firms closely focused on winning SBIR awards. Firms can easily reconstitute themselves – for example by reincorporating under a different name. They can start subsidiaries. They can subcontract. All of these tools are likely to help determined firms evade quantitative limits.

But such limits are also a bad idea. They raise a stream of questions and concerns:

Who decides which projects to accept? If it is the firm, their priorities may not be those of the agencies, and critically important research may not be undertaken.
How many is “enough”? Should the bar be set at 3 Phase II awards, four, five, ten? On what basis should this be determined?
Should Phase I awards be limited, Phase II, or both. There are some Phase I “mills,” which seem to specialize in winning Phase I awards. But Phase I awards can provide valuable results even if they are technical failures: as Edison said at one point, “I now know 2,000 ways not to make a light bulb.”
One year or cumulative? Even the most prolific multiple award winners rarely receive more than a handful of Phase II awards in a single year. So should limits be imposed on a single year or multiple years? Over time, and the SBIR program has been making awards for 25 years, the cumulative number of awards to any firms with ongoing interest in the program will inevitably rise – should the number of “permissible” awards increase as well?
Coordination. Firms apply separately to different agencies, which hold competitions at different times. How should this be coordinated? Or should the limits, if any, be on an agency by agency basis? If the former, who decided which agencies should get the applications?
Exceptions. What if an agency identifies a firm as having unique capabilities needed for a high priority project – but with no remaining allowable SBIR awards?

There are only a few of the practical questions. Failing to answer them effectively could mean that a limit-based rule could have highly negative consequences for the agencies as well as the firms affected.

6 Conclusions

Critiques of the SBIR” mills” are largely based on a misunderstanding of the purposes of the SBIR program, and on the absence of important data. Together, as the National Academy SBIR studies clearly demonstrate, there is no good case for imposing further limitations on access to the SBIR programs for small businesses based on the number of awards they have won in the past or the number of current applications pending. The only meaningful criteria are the needs of the agency, the quality of the proposal, and the past performance of the applicant.

Further, imposing such limits carries with it a significant degree of risk for the program. These limits would likely prove hard to design and harder yet to administer. That is why the Academy has recommended that the only existing limit –imposed at NSF primarily for administrative convenience – be discontinued. And it is why we conclude that such limits are not in the national interest.

____________________

[1] This article is based on the research conducted by the National Academy in the course of developing a six-volume report on the SBIR program at DoD, NIH, NSF, DoE, and NASA. See [ref.]. All opinions expressed here are those of the authors, not of the National Academy.

[2] This time period was selected for the recent National Academy study of the SBIR program at five agencies, and we maintain it here to allow for comparability with that study.

[3] These five agencies – DoD, NIH, NASA, NSF, and DOE account for more than 97% of all wards.

[4] The DD350 database