Chapter 6 Programme 1.3 Excellence in Research for Australia

Summary of performance

2014–15 performance is summarised in Table 6.1.

Table 6.1: Excellence in Research for Australia, performance summary

This summary reflects the framework identified in the 2014–15 Portfolio Budget Statements which incorporates a number of changes from that used in 2013–14. In particular:

• the deliverables and Key Performance Indicators were revised to reflect current activities
• three performance measures were introduced (Evidence of stakeholder confidence in the ERA framework as indicated by use of ERA data and results to inform policy advice across government and the strategic research agendas of higher education institutions; Achievement of milestones for delivery of ERA 2015; Sector contribution to development of ERA 2015).

Deliverables

2014–15 deliverables are summarised in Table 6.2.

Table 6.2: Excellence in Research for Australia, deliverables

Development of the ERA 2015 evaluation

Through ERA, the ARC delivers regular evaluations of the excellence of research undertaken in eligible Australian higher education institutions (mainly universities). Each evaluation is conducted using a combination of indicators and expert review by committees comprised of experienced, internationally recognised experts. The 2015 ERA evaluation is currently underway with outcomes to be reported at the conclusion of the process in late 2015. The key milestones achieved during the process are described on page 84.

Strategic policy advice

During 2014–15 the ARC used ERA data to analyse research activity and outputs to provide ongoing policy advice to government, including the Department of Education and Training, the Department of Industry and Science, the Chief Scientist and other government stakeholders.

The results from ERA 2010 and ERA 2012 provide detailed information about the research landscape in the Australian university sector. Data on research outputs (including books, journal articles, conference papers and other non-traditional research outputs) submitted for ERA cover a six year reference period.

Objective—TO IMPROVE RESEARCH OUTCOMES BY IDENTIFYING UNIVERSITIES AND DISCIPLINES THAT ARE INTERNATIONALLY COMPETITIVE

Description

ERA is an evaluation framework that gives government, industry, business and the wider community assurance of the excellence of research conducted in eligible Australian higher education institutions.

ERA collects a wide variety of indicators for evaluating research quality, research application and esteem measures. Outcomes of each ERA round are published in a national report. In addition to the national report, the ARC uses data collected through ERA to analyse research activity relating to specific areas of research.

The evaluations are conducted by ERA Research Evaluation Committees. The committees comprise distinguished and internationally-recognised researchers with expertise in research evaluation. Their judgements are informed by a range of indicators as well as more traditional measures of quality, such as citation analysis and peer review.

Committees are responsible for determining the rating given for each discipline in Australia's higher education institutions. Committees use the ERA information to determine ratings for a unit of evaluation (UoE). A UoE is a discipline area within a university.

The ERA rating scale is from 1 to 5, where:

• a rating of 1 is for research that is well below world standard
• a rating of 2 is for research that is below world standard
• a rating of 3 is for research that is at world standard
• a rating of 4 is for research that is above world standard
• a rating of 5 is for research that is well above world standard.

ERA methodology requires universities to submit comprehensive information about their research activities, including: information about staff; research outputs; awards; grants; patents; and commercial income. Using the information provided, the RECs make judgements about the quality of Australian research relative to world standards. Reports from previous ERA rounds have provided assurance about the quality of Australian university research against world benchmarks.

In 2014–15 the ARC commenced the 2015 evaluation after extensive preparations conducted during 2013–14.

Performance

ERA 2015 uses the evaluation methodology originally developed in 2008–09 by the ARC through extensive consultation with the sector and the advice of international experts in research evaluation. The methodology was specifically designed to identify excellence across the full spectrum of research performance. It uses discipline-specific performance indicators, which reflect discipline-specific research practices, and indicators which reflect a range of research activities, including basic and applied research.

In July 2014 the ARC released the guidance material for ERA 2015, which included refinements and a small number of improvements, informed by sector consultation, to ensure the methodology continues to identify excellence across the full spectrum of research performance. Changes to the methodology used for ERA 2015 included:

• reintroducing a conference list (not ranked) to ensure consistent reporting of conference data and identification of field of research codes for conferences
• introducing a book publisher list to ensure consistent reporting of book publisher data
• including a new research output type, 'research report for an external body', to capture research activity published in this format
• requiring that peer review research outputs must be in an ERA repository to ensure reviewer access to required information for evaluation
• tightening the requirements for evidence of the eligibility of staff with FTE of less than 0.4.

ERA methodology was specifically designed to allow for comparisons of research in Australia, nationally and internationally, for all discipline areas. It uses national and international benchmarks, and a range of moderation points are built into the expert and peer review processes to further allow for national and international benchmarking across disciplines.

The ARC ensures that results from the ERA 2010 and ERA 2012 evaluations (and the ERA 2015 results, when they are delivered at the end of 2015) are comparable, and performance can be tracked over time.

Key Performance Indicator 1: Establishment of a high-quality evaluation framework

Evidence of stakeholder confidence

In 2014–15 ERA data and results were used to inform policy advice across government, as well as the strategic research agendas of higher education institutions. Further details are provided in Chapter 3 .

During the year ongoing stakeholder confidence in the ERA framework was also observed during ARC interactions with the sector. The ARC conducted a range of outreach activities with university researchers, research office staff and university managers to assist them to prepare for ERA 2015. During these interactions these key stakeholders provided positive feedback on their confidence in the ERA framework.

Key Performance Indicator 2: Successful implementation of the 2015 ERA evaluation

All 2014–15 milestones for the delivery of ERA 2015 were met, including:

• release of the ERA 2015 Submission Guidelines and associated technical guidance material to enable universities to prepare their data for submission for the ERA 2015 evaluation
• conducting outreach activities with university researchers, research office staff and university managers to assist them to prepare for ERA 2015
• seeking nominations from the research sector for membership of the Research Evaluation Committees and for peer review in ERA 2015
• appointing eight committee chairs and 142 committee members, as well as over 1300 peer reviewers (see also page 86)
• submission of ERA 2015 data from all 41 eligible higher education institutions (checked and collated by the ARC for evaluation)
• committee members and peer reviewers commencing their evaluations according to the timetable to ensure ERA 2015 results are available in December 2015.

The release of ERA 2015 guidance material, in July 2014, was the culmination of extensive consultation with the sector regarding the development of ERA 2015. The ARC continued to ensure that all submission documentation for an ERA round is available six months prior to the opening of an ERA round. In 2014–15 the ARC continued to provide opportunities for the sector to contribute to ERA 2015 processes through a range of outreach activities with university researchers, research office staff and university managers in preparation for ERA 2015.

ERA Research Evaluation Committees

A key delivery milestone completed during 2014–15 was the establishment of ERA Research Evaluation Committees (RECs).

ERA evaluates the excellence of research undertaken in Australia's higher education institutions using a combination of indicators and expert review by RECs comprising experienced, internationally-recognised experts. For ERA 2015 there are eight RECs established according to broad discipline groupings.

In July 2014 the ARC invited eligible institutions and Learned Academies to nominate potential members from their organisations and overseas. The selection criteria for committee members included: research excellence; broad discipline expertise; professional and academic standing; and interpersonal and team skills and experience. Other considerations in the selection process included the spread of expertise, institution affiliation and geographical regions, workload, university cohort and gender.

The ARC undertook a rigorous process in the selection of REC Chairs and members to ensure the best quality outcomes for the evaluations and the integrity of the ERA process. A total of 152 distinguished researchers were appointed for ERA 2015 across the eight RECs from a pool of over 700 national and international researchers nominated by the sector.

As distinguished research leaders with broad discipline expertise REC Chairs had an important role in the selection of the members for their committees. The REC Chairs for ERA 2015 are:

• Physical, Chemical and Earth Sciences: Professor Stephen Buckman, The Australian National University
• Humanities and Creative Arts: Emeritus Professor Graeme Turner, The University of Queensland
• Engineering and Environmental Sciences: Professor Hugh Possingham, The University of Queensland
• Education and Human Society: Professor Brenda Cherednichenko, Deakin University
• Economics and Commerce: Professor Deborah Cobb-Clark, The University of Melbourne
• Mathematical, Information and Computing Sciences: Professor Leon Sterling, Swinburne University of Technology
• Biological and Biotechnological Sciences: Professor Merlin Crossley, The University of New South Wales
• Medical and Health Sciences: Professor Terry Nolan, The University of Melbourne

During 2014–15 REC Chairs and members were provided with detailed information about the full suite of ERA indicators and underwent thorough training conducted by the ARC.

Reducing red tape—Single Higher Education Research Data Collection

A key priority identified in the ARC Strategic Plan for 2014–15 to 2016–17 was to implement the government's commitment to the reduction of red tape by: investigating, in consultation with the Department of Education and Training, options for a single higher education research data collection.

Following the Review of the Reporting Requirements for Universities (December 2013) the Government agreed that the ARC and the Department of Education and Training would work together on the development of a single higher education research data collection—with implementation planned for 2016–17.

In 2014–15 the ARC and the Department of Education and Training jointly consulted with universities regarding the sector's views on combining the ERA data collection and the Higher Education Research Data Collection (HERDC).

In contrast to the periodic collection of data for ERA evaluations, HERDC collects research income and publications data from eligible higher education providers annually. HERDC data, along with higher degree by research student data, is used to inform the allocation of the Department of Education and Training's Research Block Grants. Whereas, ERA outcomes do not feed into the competitive grants processes, but do moderate the allocation of the funding provided to universities through the Sustainable Research Excellence research block grant scheme.

A joint ARC-Department of Education and Training discussion paper Combining Excellence in Research for Australia (ERA) and the Higher Education Research Data Collection (HERDC) was released in December 2014. The paper outlined the aims of a single higher education research data collection and invited feedback on three possible options for reporting publications and income data to the ARC.

In total, 38 universities and 4 non-university organisations provided submissions to the ARC. The submissions identified a range of options and issues in relation to combining the data collections. The ARC and the Department of Education and Training are continuing to work with the sector, to address the issues and harmonise ERA and HERDC policy, eligibility and data requirements.