When benchmarks breed bad behavior

February 06, 2020
By: Mark Skinner

The old adage, “we become/are what we measure” can push behavior of individuals and organizations into unanticipated negative space when the selected key performance indicators take on too much importance — sometimes amazingly negative space that becomes common practice and potentially damaging for the entire industry.  Counting life science startups created through the licensing work of technology transfer offices (TTOs) at the nation’s research universities appears to be the latest example of KPI (key performance indicator) pursuit potentially going bad, based on a recent Nature Biotechnology article.

“The biotech living and the walking dead,” written by three Utah-based academicians, Paul C Godfrey, Gove N. Allen and David Benson, describes their research, which found over one-fourth of the 498 life-science university-licensed start-ups created since 1969 by the 50 top patenting universities in the U.S. existed merely on paper. During a forensic research project lasting eight years, the trio was unable to find any evidence that 25.1 percent of the startup firms studied ever advanced beyond the original technology transfer documents. The article includes an outline of their search, using the National Establishment Time-Series database, state corporate dissolution databases, press release archives, LinkedIn, Pitchbook and Internet searches to track each firm’s activity or lack of activity.

These 125 startups were startups in name only, then?  It seems so, but that isn’t the entire story. They also represent individual tally marks of success in the performance measurements that university TTOs, university administrators, the tech-transfer trade associations, the broader economic development community, and state and federal policymakers have used to assess success and innovation-based entrepreneurial capacity for the institutions and regions for decades. 

The logic for using startups from university licenses as a KPI for university startups seems sound. High-growth startups are associated with greater job growth and entrepreneurial culture within a community. Innovation-based startups, in fields such as the life sciences and biotech, tend to pay higher wages. Achieving more measurable results from the billions of dollars the federal government spends on R&D each year is certainly desirable.

How metrics are used both internally and externally affects the KPI’s long-term value. If widely accepted KPIs lead to larger salaries, prestige in the field, prominent standing for the university among its peers and higher placement in national indices, benchmark studies and rankings, then bad behavior may result. And if that behavior goes unnoticed or seems accepted, more TTOs may adopt the practice.

The practice was not limited to a single higher education institution going rogue, the authors note.  As the authors explain, “We saw a significant rise in the number of these [nonstarter, ‘walking dead’] firms as driven by university administrators attempting to game the system, to claim, but not create economic development.”

The article points to another significant challenge to assessing the economic development impact of life science TTO practice: “Just under 90 percent of the [498] university-license life science startups in our sample never operated as public companies and did not report revenues, expenses, direct labor costs or direct taxes paid; hence a full economic impact analysis exceeds our grasp.”

How then to deal with the issue of non-starting startups from tech transfer? The authors suggest, “University TTOs should no longer only report the number of licensing agreements consummated or the number of startups created, in a single year, but a broader set of data over longer time horizons.” They offer: “a focus on economic impact data, such as number of employees, total wages, other expenditures, VC raised and any liquidity event.”

Competition within the academic community doesn’t end with TTO metrics however. While the authors point out in the article that the “zombie” startups problem was uncovered in the portfolios of several of the 50 top-patenting universities, they mentioned only one by name: a rival institution in their own state.

The Nature Biotechnology article is available here.

metrics, tech transfer