Measuring Impact: NSF STEM Efforts at 25

As most practitioners know, measuring progress for tech-based economic development efforts can be difficult given the long lead time necessary for most research investments to yield results. Consequently, many programs rely on interim measures to evaluate a policy or program's impact. Still others use input measures such as amount of money distributed through a particular program, the number of grants made, the number of companies involved, or the amount of money leveraged by the state's investment. Unfortunately, none of those measures provide much insight on a program's true impact.

A new assessment of the National Science Foundation's efforts to attract underrepresented population groups into science, technology and math (STEM) fields provides an example of the difficulty programs have assessing interim progress.

Broader inclusion in S&T is an important issue for future U.S. competitiveness given the varied growth rates among population groups and flattening immigration rates. A more diverse and scientifically more literate workforce should lead to significant increases in innovation and opportunity for the entire nation. To address this goal, NSF has increased its financial investments targeted to underrepresented groups by 87.5 percent over the past 25 years.

The increased spending, on its own, sounds like an impressive step and a useful input measure of the agency's commitment to the issue of broader participation. Broadening Participation in America's Science and Engineering Workforce, the new report by the Committee on Equal Opportunities in Science and Engineering (CEOSE), however, reveals the 87.5 increase is only "slightly higher than the overall budget growth during that time period." Efforts to broaden participation continue to capture less than 5 percent of NSF's total budget, similar to 1994 levels.

Now, with the passage of time, CEOSE is able to rely less on input measures to determine whether or not NSF's investment and efforts have paid off. The study analyzes the first 25 years of NSF policies and programs related to broadening participation in STEM fields.

The good news is that persons from underrepresented groups are now submitting a substantially greater number of proposals to NSF and make up a larger percentage of the professional staff at NSF, the report states. More importantly, perhaps, proposal success rates among women, underrepresented groups and people with disabilities are comparable with the foundation-wide average of 31 percent.

However, CEOSE concludes there is still a long way to go before these individuals have full access to STEM education and opportunities, as progress remains uneven across underrepresented groups, science and engineering fields, and career paths.

The report commends NSF for being among the first federal agencies in 1980 to work toward broadening participation of underrepresented groups. This change, however, is proving to be slow and difficult, the report states. Anecdotal evidence suggests that to achieve successful institutional transformation, factors affecting persistence and attachment of students and professionals demand attention. Such factors are said to require focused research and include curriculum, teaching approaches, mentoring, career opportunities, role models, decision-making processes, reward structure, resource allocation, and ways of collaborating.

In addition, it is necessary to overcome the low societal expectations and common biases about the roles and capabilities of women, minorities, and persons with disabilities, the report adds. With a focus on increasing retention rates and broadening participation, CEOSE developed recommendations for NSF within four priority areas:

• Increase social-science research on the factors that influence progress and provide barriers for underrepresented individuals in STEM at all levels;
• Conduct systematic and objective evaluation of its broadening participation policies;
• Continue to use policy "levers" to focus attention of researchers and institutions on the need to broaden STEM participation; and,
• Engage more Native Americans in STEM fields by enhancing research capacity and opportunities at Tribal Colleges.

The report summarizes the results of a quantitative analysis of the trends in participation in STEM over the past 10 years and includes COESE's 2004 Biennial Report to Congress. Recommendations for priorities and directions in the 2005-06 biennium also are provided. The full report is available at: http://www.nsf.gov/od/oia/activities/ceose/