Purpose

Measuring research excellence (Outcome 1, Programme 1.3)

Description

Excellence in Research for Australia (ERA) assesses research quality by research discipline at Australian higher education institutions. Evaluated against international benchmarks, ERA identifies excellence across the full spectrum of research activities giving a rating between five, well above world standard, and one being well below world standard. ERA aims to improve Australia’s research capacity and inform government policy.

In 2015–16, the ARC commenced developing a new engagement and impact assessment framework that will run in conjunction with ERA and assess the engagement of university researchers with end-users, and show how universities are translating their research into economic, social, environmental and other impacts.

Together the frameworks:

• provide a unique, evidence-based resource to inform Australian government research policy and the strategic direction of higher education institutions
• encourage researchers to produce high-quality and impactful research with real world benefits.

Highlights of delivery

The deliverables for ERA, as identified in the Portfolio Budget Statements (PBS) 2015–16 are the ERA 2015 evaluation, ERA 2015 National Report and strategic policy advice on research matters relating to the measurement of research quality (PBS, page 165).

In 2015–16, the ARC:

• completed the ERA 2015 evaluation process and published Statement of Australian University Research 2015–16, Volume 1 ERA National Report
• completed extensive outreach visits to universities to discuss their ERA 2015 outcomes and obtain their feedback on the process
• undertook a range of evaluation-related policy development activities (see Section 2.4 for further information).

Results

Intended result: Outcomes of benefit to Australia

Description

Improvements in research quality lead to a better return on investment in research, and improve the social rate of return of research (Benefits Realisation Review of Excellence in Research for Australia, September 2013, ACIL Allen Consulting). The ARC’s aim is to ensure these outcomes can be realised for Australia through the conduct of a best practice evaluation of the quality of research at Australian higher education institutions.

Performance

Measure 1	Evidence that ERA improves the research performance of eligible Australian higher education institutions
Source	CP 2015–19 page 27; PBS 2015–16 page 165
Target	The impact of ERA is demonstrated through analysis of Australian higher education research performance
Results	The outcomes of ERA 2015 showed that the quality of research conducted by eligible Australian higher education institutions had improved since the previous evaluation.	Target met

Analysis

The results of ERA 2015 provided evidence that ERA is improving the research performance of eligible Australian higher education institutions.

To date, the ARC has conducted three ERA evaluations–in 2010, 2012, and 2015. The ERA 2015 National Report shows that overall the quality of research conducted by eligible Australian higher education institutions has improved, reflecting the important role that ERA plays in focussing universities on research quality.

The unit of evaluation in ERA is the discipline, defined by the Australian and New Zealand Standard Research Classification, at the broad field and specific discipline levels. Of the Units of Evaluation (UoEs) assessed in both ERA 2015 and the previous round, ERA 2012, 56 percent of these maintained the same rating, while 29 per cent improved their rating by one. The ERA results show an extraordinary breadth of outstanding research performance in Australia.

There was also steady growth in Australia’s university research activity between the ERA 2012 and ERA 2015 rounds, including increases in:

• the total number of assessed UoEs, up six per cent to 2460
• the number of research outputs submitted, up five per cent to 432,747
• the number of research staff submitted, up 11 per cent to 67,579
• total number of patents, up 20 per cent to 936.

ERA also helps identify areas of particular research strength. A wide variety of disciplines were represented in these national strengths in the 2015 evaluation—from basic sciences, engineering, to the humanities—including pure and applied mathematics; astronomical and space sciences; macromolecular and material chemistry; soil sciences; plant biology; civil engineering; clinical sciences; nursing; cultural studies; historical studies; philosophy and law. A graphic of the ERA 2015 outcomes released at the same time as the ERA 2015 National Report is provided in Figure 2.10.

Measure 2	Evidence that ERA improves strategic planning at eligible Australian higher education institutions
Source	CP 2015–19 page 27; PBS 2015–16 page 165
Target	Majority of higher education institutions report that ERA data and results are used to assist their strategic planning
Results	A survey of the 41 ERA participating universities revealed that the majority of institutions referenced ERA in their strategic planning documents.	Target met

Analysis

A survey of university websites provided evidence that ERA is informing the strategic planning at eligible Australian higher education institutions.

The survey of the websites of the 41 ERA participating universities revealed that the majority (approximately 85 per cent) of institutions referenced ERA in their strategic planning documents and in their annual report of performance, while more than 90 per cent made references to ERA and their performance in the 2015 ERA round on their website.

One of the aims of ERA is to improve Australia’s research capacity and it is evident that Australian universities are using ERA data to support their strategic planning activities. ERA results are routinely used in universities’ internal reporting and planning documents (such as annual reports and strategic plans) and the promotional material of universities frequently emphasise ERA outcomes as an indicator of current performance and/or to set targets in relation to future performance. These documents compare performance relative to both international and national benchmarks. They demonstrate that Australian universities are using ERA data to:

• build on existing and emerging research strengths
• address weaknesses and/or gaps in their research capacity
• identify opportunities for cross disciplinary research and collaboration
• benchmark performance against national strengths and weaknesses
• develop capacity in areas of strategic priority to their institutional missions, including aligning capacity with undergraduate teaching profile, research training load, and the concerns of local communities and other key stakeholders.

Measure 3	Evidence that ERA positively informs Australian government policy
Source	CP 2015–19 page 27; PBS 2015–16 page 165
Target	At least two case studies per year
Results	Three case-studies are provided at the end of this section. The case studies relate to the Australian Government’s Science and Research Priorities, the Australian Government’s National Strategy for International Education and the Australian Academy of Science’s Decadal Plan for Chemistry.	Target met

Analysis

The three rounds of ERA have established a data collection that spans over a decade of the research activity and performance of Australia’s universities. The dataset is a unique and valuable resource for a range of stakeholders. The ARC regularly responds to requests for information about the nature, extent and location of research strengths in particular disciplines from individual researchers, postgraduates, government agencies and entrepreneurs.

ERA has a range of broader impacts from assessing the quality of Australian university research in all academic disciplines. It is also designed to have real effects, including to:

• set national and institutional performance targets by benchmarking against the rest of the country and the world
• give industry, students and academics real data to inform their decisions about postgraduate options, research collaboration and job opportunities
• build incentives that drive research performance through resource allocation mechanisms
• align research training with areas of both institutional and national strength
• improve our understanding of our innovation inputs through improved data collection, monitoring and review.

Figure 2.10: ERA 2015 outcomes

Broad research disciplines: 01 (Mathematical Sciences), 02 (Physical Sciences), 03 (Chemical Sciences), 05 (Environmental Sciences), 07 (Agricultural and Veterinary Sciences), 09 (Engineering), 11 (Medical and Health Sciences), 17 Psychology and Cognitive Sciences), 18 (Law and Legal Studies), 21 (History and Archaeology)

Intended result: Deliver high-quality evaluation

Description

Conduct of a high-quality (efficient and effective) evaluation process helps ensure that stakeholders can have confidence in the results of the process. The ARC aims to ensure that the ERA process incorporates all elements of best practice evaluations.

Performance

Measure 4	Evidence that the ERA framework remains relevant and best practice
Source	CP 2015–19 page 27; PBS 2015–16 page 165
Target	Feedback received demonstrates confidence in ERA framework; ERA compares well to equivalent research evaluation exercises internationally
Results	Feedback received through post-ERA 2015 reviews indicated a high level of sector satisfaction with the methodology. ERA is a uniquely comprehensive collection with data submissions covering all eligible researchers and their research outputs. Other evaluations undertaken around the world are selective exercises.	Target met
Measure 5	Evidence that the ERA National Reports provide assurance of the excellence of research conducted in Australia
Source	CP 2015–19 page 27; PBS 2015–16 page 165
Target	Stakeholders are satisfied with the assurance provided by ERA National Reports
Results	Feedback received through post-ERA 2015 reviews indicated a high level of sector satisfaction with the outcomes.	Target met

Analysis

Feedback received through post-ERA 2015 reviews indicated a high level of sector satisfaction with the ERA methodology and outcomes.

To prepare for ERA 2015, the ARC built on the methods used in ERA 2010 and ERA 2012. A key principle underpinning all three evaluations was the consistent application of the framework to allow comparison of the results from one round with the previous rounds. To that end the ARC:

• maintained the length of the reference periods applied to research outputs and other indicators
• used the same key data and evaluation methodologies.

At the same time, possible improvements were identified following sector consultation and further internal analysis. Changes implemented in the conduct of ERA 2015 included:

• requesting gender data (not for evaluation)
• open access repository (not for evaluation)
• requiring that peer review research outputs must be in an ERA repository
• tightening eligibility requirements for staff engaged at less than 0.4 full-time equivalent (publication association required)
• a request to nominate outputs for peer review.

The ARC visited all participants in ERA 2015 to provide information on the outcomes of the evaluation. The visits provided an opportunity to engage with universities to discuss the results as well as to seek feedback on the process.

Analysis of the impact of changes on performance

There were no changes in the ARC’s purposes, activities, organisational capability or environment that impacted on its performance in delivering Purpose 2: Measuring research excellence.

As noted on page 3,a key development during the year was the announcement that the ARC would be responsible for the development and conduct of an engagement and impact assessment to be run concurrently with ERA 2018. The addition of this new responsibility had no impact on 2015–16 outcomes of the ARC activities in relation to ERA.

The ARC will receive $9.4 million over four years from 2015–16 to establish the new system to assess the engagement of university researchers with end users, and to measure the commercial, economic, social and other impacts of research. As a result of this change, two new deliverables were identified for the ARC in the Additional Budget Estimates documentation—‘consultation and development of impact and engagement exercise’, and ‘strategic policy advice on broad research measures relating to measures of research quality and impact’. During 2015–16, the ARC:

• helped to establish the Engagement and Impact Steering Committee, Technical Working Group and Performance and Incentives Working Group to assist with the development of the engagement and impact assessment framework
• consulted with industry, other end-users of research, and Australian universities regarding the development of the engagement and impact assessment framework
• developed key principles for the engagement and impact assessment framework.

ERA data contributed to the development of Australia’s Science and Research Priorities. These national priorities are designed to guide investment in areas of immediate and critical importance to Australia and its place in the world.

In 2015 panels of experts were appointed to identify Practical Research Challenges within each of the Science and Research Priorities. As part of this effort to build a comprehensive picture of Australia’s research capability and activities, and to track these into the future, the ARC provided detailed analyses of Australian publication activity from the ERA dataset. The aims of the analyses were to provide a comprehensive map of capability and potential capacity across the nation taking scholarly publication output as a proxy measure, as well as to identify the research on which Australia focuses and in which it excels.

Case study 1: Outcomes of benefit

Informed by ERA: Australian Government’s Science and Research Priorities

For each Practical Research Challenge, the mapping analyses depicted where Australia is placed relative to other countries, trends in research output, which institutions within Australia produce high volume and high quality research, and Australia’s main international collaborative partners. The analyses provided the expert panels with robust evidence to inform their deliberations and form a baseline for tracking the nation’s progress in future.

The nine Australian Government Science and Research Priorities announced in May 2015 were Food, Soil and water, Transport, Cybersecurity, Energy, Resources, Advanced Manufacturing, Environmental change and Health. The Practical Research Challenges identified for Food, for example, were:

1. Knowledge of global and domestic demand, supply chains and the identification of country specific preferences for food Australia can produce.

2. Knowledge of the social, economic and other barriers to achieving access to healthy Australian foods.

3. Enhanced food production through:

• novel technologies, such as sensors, robotics, real-time data systems and traceability, all integrated into the full production chain
• better management and use of waste and water; increased food quality, safety, stability and shelf life
• protection of food sources through enhanced biosecurity
• genetic composition of food sources appropriate for present and emerging Australian conditions.

ERA data were used to inform the development of the draft National Strategy for International Education 2025. As noted in the policy description, ERA is designed to have real effects including the alignment of research training with areas of both institutional and national strength.

The final strategy, released by the Department of Education and Training on 30 April 2016, is a 10-year plan to develop Australia’s role as a global leader in education, training and research. It is aimed at strengthening Australia’s internationally recognised education system, increasing global partnerships and driving collaboration with local communities and international partners.

Case study 2: Outcomes of benefit

Informed by ERA: Australian Government’s National Strategy for International Education

ERA results were used in the Decadal Plan for Chemistry (2016–25) as part of the description of the current state of chemistry. The ten-year plan, developed by the Australian Academy of Science’s National Committee for Chemistry, envisages improved links between chemistry researchers and industry, as well as better quality school and tertiary chemistry education in Australia. It also outlines strategic goals for chemistry, including improving teacher expertise, strengthening links between industry and the chemistry research community, and creating a positive image for chemistry.