Company Outcomes Research and Evaluation for SBIR (CORES) is an effort to build a sustainable capability to identify and measure firm-level outcomes for SBIR awardees. Hitherto, evaluation and assessment has focused exclusively on project-level outcomes. CORES will address the impact of SBIR on firms. To do so, the CORES team will develop new capabilities that:
- Build a data bridge between the SBIR/STTR awards database at SBA and the Business Dynamics and other data sets at the US Census Department;
- Identify an evolving set of SBIR/STTR awardees along with critical identifying information for each firm – notably the Employer Identification Number (EIN);
- Develop a matched set of firms to act as a control group;
- Design a database structure for maintaining data on both the awardee and the control group firms; and
- Ensure that the resulting database is fully and accurately populated, and is updated regularly across the life of the project.
The CORES data platform will be constructed in collaboration with SBA, Census, and NSF.
The CORES vision is that the data platform will eventually cover every firm in the SBIR universe. It will be regularly updated as additional firms enter that universe and as Census refreshes firm records. SBIR firms can then be grouped by year, geography, NAICS code, number and scale of awards, and ownership characteristics for extended analysis. Census will also create a control group of firms, based on criteria developed in conjunction with the CORES team and SBA. Possible identifiers for control group firms include NAICS codes, revenues, patent applications, and SBIR Phase I awards only (no Phase II).
Once the data platform and initial data acquisition are complete, the project can be extended in various ways described below, and will offer sponsoring agencies access to a high quality and low-cost resource of undertaking further studies and assessments.
Additional tools will enhance the new data platform’s capabilities. Economic impact analysis will require the application of IMPLAN software and methodologies to the data developed under CORES, following the methodology established in the 2014 TechLink study of Air Force SBIR impacts and TechLink’s subsequent 2016 study of impacts at Navy.
Dr. Gaster is the project director, working on this project in partnership with Dr. Andrew Reamer of George Washington University.
Congress, SBIR-awarding agencies, and other stakeholders seek cost-effective way to monitor and evaluate SBIR/STTR programs. Previous impact evaluation efforts, such as those conducted by the National Academies and Montana State University (MSU), have used surveys to track individual project outcomes. Though valuable, surveys are expensive and rely entirely on self-reported data of limited quality. At a time when Federal programs are under increasing pressure to show they provide value for money, an alternative approach is needed.
Using administrative data to track firm outcomes. CORES looks at the outcome histories of SBIR-funded companies, not individual projects. It uses administrative data collected by the Census Bureau, the Internal Revenue Service (IRS), the Bureau of Labor Statistics (BLS), and the U.S. Patent and Trademark Office (USPTO) to develop detailed outcomes data for SBIR recipient firms. Data elements include revenues, employment, firm growth and age, ownership, patents, and SBIR award characteristics.
These outcomes data can be further disaggregated by applying variables developed at SBA based on SBIR/STTR award characteristics. These include variables such as the date of first and most recent SBIR/STTR award, number of awards, awards by agency, participation in formal agency initiatives (e.g. various Phase II+ initiatives, commercialization training). In principle, any differentiation within SBA awards data can generate outcomes for analysis through CORES. CORES will also provide opportunities for comparison group analysis as Census will develop and maintain a shadow database of nonSBIR firms that have characteristics similar to participating firms.
Estimating economic impacts from administrative data. Administrative data can be used as inputs into IMPLAN’s detailed input-output model of the US economy to estimate the economic impacts and ROI for SBIR programs. This approach was used in both the Navy and Air Force impact studies and has become a standard methodology used for impact analysis. Impacts can be estimated for individual SBIR agencies and components and for selected geographies (the nation, states, and counties).
The data platform
The CORES data platform will be constructed through the collaboration of SBA and Census. SBA will provide a list of SBIR firms and their EINs. Census will match these firms to their records in its Longitudinal Business Database (LBD), a compendium of firm- and establishment-specific records drawn from multiple federal sources and going back to the 1970s.
SBA and Census then will collaboratively design the analyses to be undertaken. Census will conduct the specified analyses and deliver the tabulated/aggregated results to SBA, with complete protection of firm confidentiality.
The CORES data platform will cover every firm in the SBIR universe. It will be regularly updated as additional firms enter that universe and as Census refreshes firm records.
Census will create a control group of firms, based on criteria developed in conjunction with SBA. Possible identifiers for control group firms include NAICS codes, revenues, patent applications, and SBIR Phase I awards only (no Phase II).
Census variables can be disaggregated by year, geography, NAICS code, and ownership characteristics. At present, the following variables are expected to be included:
- Company formation, age, and operating status
- Legal organization
- Ownership characteristics (such as acquisitions, owner demographics)
- Patent citations
- Federal non-SBIR grants and contracts
These variables will be combined with SBIR variables, including:
- Year of SBIR award
- Year of most recent award
- Awarding agency
- Award size
- Number of awards
- Participation in specified agency initiatives (e.g. Phase II+, iCorps, other training programs, DoE white papers)
- Relationship to CCR outcomes
Through the Census Bureau’s LBD, federal data sets utilized will include the Census Bureau’s Economic Census, Survey of Business Owners, and Longitudinal Employer Household Dynamics (LEHD); USPTO patents; the Federal Procurement Data System (FPDS); and the System of Awards Management (SAM).
Analyses commissioned by agencies and SBA
Using the CORES platform, SBIR-awarding agencies and other stakeholders can commission analyses that can be completed relatively quickly and inexpensively, with no additional data collection required. This is the key advantage of the CORES approach.
In Phase 2, CORES will prepare an analysis of the SBIR program at one participating agency. In subsequent phases, additional analyses will be tailored to the specific needs of the commissioning agency. We anticipate that each study can be completed and delivered in about five months at a cost of $80,000.
In addition, multi-agency analyses can be commissioned by SBA with funding from participating agencies. Such studies would have the same timeline and cost, but would address program-wide issues—such as preparing estimates of program-wide ROI.
The work-flow for commissioned studies is described in Figure 1. It shows how study requests are approved by SBA, which then provides the relevant input data on which Census builds the specialized data set being requested. Those data are tabulated by Census and returned to SBA for further analysis. SBA provides the completed study to the commissioning agency. This parsimonious approach allows Census and the commissioning agencies one point of contact, uses economies of scale to reduce costs, and retains flexibility for agency-defined studies.
Dr. Robin Gaster