Report Focuses on Evaluating R&D

A new report from the Advanced Technology Program (ATP), A Toolkit for Evaluating Public R&D Investment, provides useful information to anyone interested in evaluating publicly-sponsored research and development (R&D) programs. While the report focuses on more than 40 evaluations that have been performed for ATP, it offers one of the most comprehensive and understandable overviews of evaluation methods and applying those approaches.

In one section, the authors provide a general framework for evaluation, discussing evaluation fundamentals, including the objectives and steps of evaluation, and choosing various methods of evaluation. The advantages and disadvantages of a variety of approaches, including surveys, case studies, bibliometrics, historical tracing and expert judgment, are described.

The bulk of the report is focused on how the evaluation methods have been applied to ATP, drawing on various studies commissioned by ATP between 1990-2000. Technology-based economic development practitioners will be able to use this section to consider how ATP's experience may be applied to their programs.

The final section centers around the results of ATP in five major themes: firm or industry effects, collaboration effects, spillover effects, interfaces and comparisons with other programs, and measures of overall performance. Prospective case studies show the benefits of the program far exceed the costs, the authors note. The National Research Council concludes that ATP is effectively meeting its legislative goals. According to the case studies, approximately 16 percent of completed ATP-funded projects showed strong progress toward creating and disseminating knowledge and commercializing projects and processes, and 26 percent showed substantial progress.

The report concludes with recommendations for future directions in evaluation.

Established in 1988, ATP is part of the National Institute of Standards and Technology and uses project selection and funding processes to bring together private, nonprofit and public sector organizations to advance the development and commercialization of new technologies. The full report is available at: http://www.atp.nist.gov/eao/gcr03-857/contents.htm