Report prepared for the AUTM Board of Trustees
July 2007

GOAL OF TASK FORCE
In July 2006, the AUTM Board of Trustees authorized a Board-level task force to assess what “new metrics” AUTM, or others, should collect and report in order to provide more useful and comprehensive information about academic technology transfer and its impacts on society. The Task Force defined, and the Board approved, a phased project beginning with a process to gain insights from relevant audiences regarding necessary and effective metrics, and identified appropriate partners for an initial study. AUTM’s primary goal was to suggest audiences and information needs regarding academic innovation that either AUTM or other organizations should collect and report. Specifically, AUTM was interested in learning more about how people use metrics now, what metrics they would like to have which they do not have now and what other parties are involved in the environment or ought to be. AUTM asked its audiences questions such as:

1. What information do people need, why do they need it and on what schedule (annual, decade, etc) do they need it?
2. What would they do with the information?
3. What information do they use now, and how do they get it?
4. What is the most critical piece of information they could have, that they lack now?
5. What formats help them the most – stories, hard numbers, comparisons, benchmarks?
6. How does technology transfer information roll up into other measures associated with the institutions our professionals serve, specifically regarding institutional-created innovation?

As a means of soliciting broader open-ended input, the audiences were also asked, “What other information/training/other needs do they perceive AUTM could help in providing?”

FINDINGS TO DATE

1. AUTM’s has a continuing role to play in metrics.
   1. The Annual Licensing Survey, because of its history and stability, is a very useful tool for multiple audiences, and the data it provides is seen as valuable.
   2. The policy community, research institution administration and funding agencies strongly believe that AUTM should not attempt to measure or report information about all aspects of research institution innovation. AUTM members can report about one component of the knowledge transfer and innovation activity which a research institution engages in. We note that the Task Force believes this view is consistent with AUTM’s role as a professional society supporting academic technology transfer.
   3. AUTM found that there are no U.S. surveys of university or other research and innovation activity equivalent to AUTM’s Annual Licensing Survey. There are other surveys which address some components of university or academic innovation, such as the National Science Foundation Academic Research and Development survey.
2. AUTM must define metrics, or be defined.
   1. AUTM members, the policy and government community, and research administration do not believe it is appropriate that they are often ranked based on financial performance of their office, given maximizing financial return is not usually the highest goal of a research institution, nor its innovation management functions.
   2. Absent appropriate metrics, financial performance is often used the default proof of success. It is therefore essential that AUTM defines appropriate metrics to prevent this from happening.
   3. The political pressure on the public universities both inside and outside the U.S. to connect funding received from public sources to impacts and outcomes is escalating. Technology transfer is viewed as a key component connecting them and existing AUTM metrics, in the absence of other measures, are taken to be the total output of research performing institutions.
3. AUTM must provide context for its data.
   1. AUTM does not provide context for its data, but other organizations do use the AUTM data embedded within their own context. Information such as overall patents, research expenditures, and the scope of technology transfer within the knowledge transfer environment, may better help readers understand the impact of technology transfer alongside other knowledge transfer impacts. AUTM should therefore more proactively seek appropriate context for its data
   2. Policy makers and funders are interested in more detailed studies of individual innovations – more case studies in the lifecycle of innovation. This information would provide them greater proof of funding effectiveness, and allow opportunity for greater understanding of how policies and funds can accelerate the effectiveness of academic innovation.
   3. The Better World Project does provide a counterbalance to the financial numbers and a better context for the range of activities and efforts undertaken by our members. We see this from the contributions the Better World Project has made to changing perceptions among outside groups, opening discussions with policy makers, senior research administrators and sister organizations and positioning AUTM to conduct the New Metrics Activities. This underscores the importance of considering metrics broadly and balancing quantitative and qualitative approaches to our activities. It also presents an opportunity for creation of the longitudinal data being sought.
   4. AUTM members report a core number of items about their own activity: Office mission statement; Number of licenses executed; Patents filed; Invention disclosures; Revenue from licensing; and Number and name of new start-ups. Many members also include information on sponsored research; Patent reimbursement; and data about other kinds of agreements the specific office is responsible for.
4. AUTM members need operational information.
   1. AUTM members generally find the information which AUTM collects through its Annual Licensing Survey useful for their regular practice, which was the original goal of the survey when it began in the 1990s.
   2. More information about office “operations” would be helpful to members. Information such as the scope of an office, the budget and goals of an office, none of which are currently systematically collected, would be helpful to members and academic institution administration. The AUTM Salary Survey, currently executed once every two years, does collect and report limited data on office personnel and salary. Members find this survey very useful for managing recruiting and budgets.
   3. AUTM members do not want operational information for ranking so much as for benchmarking. The mode of presentation and availability of such data is an important consideration and needs to avoid the issues arising with use of the existing AUTM Annual Licensing Survey.
5. More parties should be involved in innovation measurement.
   1. Many parties involved in academic innovation – in some place in the innovation lifecycle – desire to be more involved in defining, collecting and reporting metrics regarding its utility and effectiveness. Several associations are conducting their own metrics activities, exploring what role they should play in measuring and reporting innovation activities.
   2. U.S. State and local economic development agencies (EDAs) often use AUTM data to benchmark and conduct other analysis as they see appropriate. However, there are different levels of usage of the survey data, and differential levels of understandings of AUTM data. Many EDAs are not well connected to their local technology transfer function.

PARTNERS
AUTM is focusing its initial efforts in the new metrics project in the United States, and is pleased to work with UNICO (United Kingdom) and the Alliance for the Commercialization of Canadian Technology (ACCT) in Canada in this phase of the New Metrics project (the MoUs between the Associations are shown in an Attachment). This builds upon strong existing relationships between the three associations and upon UNICO’s and ACCT’s pre-existing activity and interest in metrics definition. ACCT and UNICO are conducting similar analyses as AUTM is in their respective jurisdictions and expect to have initial conclusions later this year.

AUTM specifically worked with Diane Hoffman and Kate Phillips to lead certain tasks under AUTM’s portion of this project, in order to more efficiently achieve certain tasks.

ACTIVITIES ENABLED THROUGH THE NEW METRICS PROJECT TO DATE

1. Seek advice. AUTM formed an Advisory Board of individuals and organizations key to U.S. competitiveness, research institution innovation, academic analysis of innovation and federal relations in order to provide AUTM context regarding others’ U.S. metrics activities. The Advisory Board is currently composed of:
   • Don Siegel, University of California, Riverside
   • Lesa Mitchell, Kauffman Foundation
   • Bob Hardy, Council on Government Relations
   • Ed Linsenmeyer, Federal Lab Consortium
   • Toby Smith, American Association of Universities
   • Maryann Feldman, University of Georgia
   • Jenny Bond, Council on Competitiveness
   Other key contacts also attend the Advisory Board meetings:
   • Kevin Cullen, University of Glasgow (UNICO representative)
   • Carolyn Lee, CONNECT (University of California, San Diego)
   • Angus Livingstone, University of British Columbia (ACCT representative)
   • Alan Rapaport, National Science Foundation
   • Kate Phillips, The Regulatory Strategy Group
     
     
   The Advisory Board met in conjunction with the 2007 AUTM Annual General Meeting in San Francisco in March 2007. The Advisory Board is expected to meet in fall 2007. The Advisory Board primarily advises AUTM on other parties involved in innovation and metrics, and provides AUTM advice on metrics.
2. Seek input from research institution leadership. AUTM or its designees attended several meetings with Senior Research Officers of universities to learn their perspectives on university innovation, identify areas for future data collection, and discuss how metrics (other than financial) could be used and communicated to demonstrate the value of technology transfer.
3. Seek input from political and other groupings. AUTM, in partnership with New Economy Strategies, held a three-hour roundtable discussion in Washington D.C. in April 2007, bringing together representatives from a variety of federal agencies, national organizations and private sector groupings. The roundtable initiated a broad discussion on roles in the innovation ecosystem and the information needs and perspectives of various segments of that system.
4. Learn member’s perspectives. AUTM conducted a mini-survey of AUTM members to better understand their preferences for surveys and data about their own activity.
5. Examine how AUTM members report about their own activity. AUTM examined members’ annual reports to discern what AUTM members report to their audiences as most important to them, and demonstration of their effectiveness.
6. Identify possibilities for new metrics. AUTM constructed several schema and lists of “possible” metrics which AUTM or others may possibly consider using to illuminate other aspects of research institution innovation activity.
7. Establish relationships with key stakeholders. AUTM met, and will continue to meet, with policy and national federal funding organizations to determine their current involvement, interest and future need for data regarding research institution metrics.
8. Learn how AUTM data is used. AUTM conducted a review of how other organization use AUTM data to better understand what variables are of interest to the external community.
9. Link scholars and practitioners. AUTM is working with the Technology Transfer Society, and conducting part of a workshop at the Academy of Management to establish closer connection with scholars and academics studying academic technology transfer, to enable scholarship which more completely illuminates the knowledge transfer activities of research institutions.
10. Motivate others in the ecosystem. Through its existing and new relationships, AUTM is talking with other parties involved in U.S. innovation to encourage activity in measurement and reporting, with such measurement activities designed to coordinate as much as possible to enable a deeper understanding of research institution’s innovation.
11. Analyze AUTM’s activity. AUTM examined its own Annual Licensing Survey identify the definitions AUTM has used for its variables, and document how these definitions changed over time.

SPECIFIC RECOMMENDED ACTIONS
AUTM is a comprehensive , worldwide network of professionals involved in technology transfer in research organizations. AUTM’s core strength is its network of members, and its members ought to be integrally involved in defining the schedule, variables and survey activities authorized under this project.

1. Present the outline of a plan for New Metrics to AUTM members in September, and a plan incorporating as much members’ feedback as possible to Senior Research Officers in October.
2. Refer the plan outline, feedback and recommendations to the Metrics & Survey committee, and allow them to suggest the schedule, variables and report structure for changes to the Annual Licensing Survey.
   1. If appropriate, hire consultant(s) to assist in defining and crafting survey questions, variables and plans.
   2. Conduct a series of “pilot” surveys, and begin planning for those surveys in Fall 2007, using a schedule which works for AUTM members, to test such questions and variables. The pilot surveys may cover issues such as: types of technologies disclosed at research organizations; nature of markets that research technologies enter; process and length of innovation creation and transfer from research organization to consumer.
   3. Provide a better basis for identifying “peer” organizations.
      1. Identify the “goals” of specific research institutions and their technology transfer functions
      2. Report core operational information of research institutions’ technology transfer office, such as budget.
3. Consider how AUTM reports its data and make the reports more useful to audiences to place AUTM’s data in comparison with other innovation actors. Reference other data sets relevant to the academic technology transfer lifecycle, such as U.S. patent information, U.S. research and development statistics, venture capital information, etc. Providing such information will allow readers to better understand the relationship academic technology transfer has to the overall economy.
4. Define variables, elements and/or factors that AUTM believes will be relevant to parties studying and evaluating the research environment innovation ecosystem. Identify a strategy for seeking other parties’ actions to enable such data collection and availability.
5. Create “matchmaking” tools which allow researchers and practitioners to partner together to conduct research.
6. Investigate creating funding or other resources for academic – practitioner research partnerships, and a forum for such research to be highlighted. Consider allocating or raising specific funds for this purpose, perhaps through the AUTM Foundation. Especially if AUTM does not receive National Science Foundation funds, AUTM will require funding to continue this important, but expensive, relationship management activity.
7. Post this report, absent appendices and “recommendations” section, to AUTM website, and share with interested parties.
8. Establish partnerships with other interested research technology transfer organizations and/or countries, and encourage adoption of a similar approach; coordinate with results and survey changes as appropriate.