ARC response to the Evaluation of the implementation of the continuous Linkage Projects processes
Purpose of this document
The purpose of this document is to provide the ARC’s response to the recommendations arising from the evaluation of the ARC’s implementation of the continuous Linkage Projects processes.
The terms of reference of the evaluation are as outlined below.
Terms of reference
- Assess the effectiveness of the continuous Linkage Projects process in supporting research collaboration, including:
- its influence on the participation of partner organisations, including industry and other research end-users
- its facilitation of support for research that responds to time critical innovation opportunities
- Assess the efficiency and effectiveness of the continuous Linkage Projects process and its implementation, from the perspectives of the ARC, higher education institutions and partner organisations, including:
- application and assessment processes
- fast tracking processes
- announcement and commencement of funding.
ARC response to recommendations of the evaluation
Recommendation 1: Application deadlines
Recommendation
To address the decline in total partner organisation participation, the value of partner organisation contributions, total applications, funded projects, and overall collaboration in the Linkage Projects scheme since the introduction of the continuous process, the ARC should introduce three or four application deadlines and assessment rounds each year. The introduction of deadlines would respond directly to feedback from key Linkage Projects stakeholders.
A series of regular application deadlines would help the ARC to address a range of issues associated with the implementation of the continuous process by:
- providing greater incentives for researchers and partner organisations to prioritise and complete Linkage Projects applications
- supporting improved planning and workload management for universities, researchers and partner organisations
- providing incentives for more even submission of applications throughout the year
- reducing the administrative burden on SAC members and the ARC associated with the management of assessment processes and SAC meetings throughout the year
- reinforcing the capacity of SACs to benchmark and assess relative application quality through fewer, larger batches of applications.
There are positive changes and benefits arising from the introduction of the continuous process that the ARC should seek to protect as part of the implementation of any application deadlines and assessment rounds. These include:
- an increased average number of partner organisations participating per application and funded project, indications of more frequent involvement by partner organisations who are repeat participants, and indications of increasing value of partner organisation contributions per project
- stakeholders reporting increased and improved collaboration within applications and funded projects, and facilitation of more timely collaboration and greater responsiveness to innovation opportunities
- efficiencies in assessment and SAC processes, and satisfaction among successful stakeholders with ARC implementation of the process.
The introduction of three or four assessment rounds per year should be accompanied by an undertaking to provide applicants with an outcome within six months from each application deadline. This will represent a shift from the current undertaking to provide outcomes within six months from the submission of applications (as intended by NISA and the Watt Review). Given that stakeholders expressed a desire for faster assessment processes, the ARC should consider the number and dates of deadlines carefully to minimise the potential time taken for applicants and their partners to receive advice of their outcomes.
ARC response
The ARC accepts this recommendation.
Since the finalisation of the evaluation, the Linkage Projects 2018 round has been finalised. The low proportion of Australian industry engagement has continued in the most recent round (see chart below).
Figure 1: Partner organisations by type (funded projects) (%)
Note: in the chart, the letters beside the year represent the type of round
B= Biannual; A=Annual and C=continuous.
The ARC notes the strong stakeholder feedback from end-users and universities that while there are positive aspects to the introduction of the continuous application cycle, publicly announced deadlines throughout the year would be beneficial in supporting planning and development of grant applications.
The ARC also notes the finding of the evaluation that the introduction of the continuous process does not appear to have “incentivised increased participation by business and industry partner organisations”.
Given the findings of the evaluation, which appear to be supported by the most recent data, the ARC is of the view that it should maintain the continuous submission process, but introduce three published deadlines each year for batching applications. According to feedback gathered through the evaluation such a move would support increased engagement by end-users.
Three selection rounds per year for the Linkage Projects scheme will address a range of issues identified in the evaluation. It is also consistent with (i) Recommendation 6a of The Review of Research Policy and Funding Arrangements Report released in November 2015 (the Watt Review) and the National Innovation and Science Agenda, and (ii) the original aim of improving opportunities for partner organisations (including industry) to participate in the scheme.
The ARC will provide funding recommendations to the Minister within six months of the relevant batching application deadline.
Recommendation 2: Information on assessment process
Recommendation
To address stakeholders’ concerns about a lack of transparency, difficulty in tracking applications through the continuous process, and the quality of feedback on unsuccessful applications, the ARC should provide additional information on the ARC website and in RMS regarding assessment processes and announcements.
The introduction of three or four assessment rounds per year would support this recommendation by enabling the ARC to manage and communicate the following information:
- expected timeframes for the different stages of the assessment process for each assessment round—published on the ARC website
- a selection report for each completed assessment round—published on the ARC website
- provision of feedback on the scoring bands in which unsuccessful applications fell (within the relevant assessment round)—available to applicants through RMS.
The provision of this additional information will support improved planning, workload management and prioritisation for universities, researchers and partner organisations.
ARC response
The ARC accepts this recommendation.
By publishing three batching application deadlines per year, this will allow for greater transparency of assessment processes and timeframes for applicants and their partners, as well as for ARC assessors.
Complementary to batching into three assessment rounds, the ARC will publish additional information on the ARC website about assessment processes and expected timeframes, publish a selection report following the announcement of each batch of applications, and develop functionality to provide additional feedback to unsuccessful applicants via the Research Management System.
Recommendation 3: Funding commencement
Recommendation
To better align grant announcements and the commencement of ARC funding for successful projects, the ARC should modify ARC financial systems to allow for more frequent releases of initial funding.
It may be possible to coordinate these releases with the expected timeframes for assessment and announcement of grants. This would help to address stakeholder concerns about potential delays in commencing projects.
ARC response
The ARC notes this recommendation.
While capability currently exists within the ARC’s Research Management System to establish funding commencement dates for any month of the year, a detailed analysis of the impact of this proposed change will need to be conducted. In particular, the ARC will consider when all agreement deliverables are likely to have been met, rather than just the announcement date.
Any response to this recommendation will also be affected by proposed changes to application deadlines (Recommendation 1). The ARC will give further consideration to this recommendation after the additional analysis has been conducted and a decision regarding Recommendation 1 has been made.
Recommendation 4: Guidance to SAC members
Recommendation
To address the decline in business and industry organisation participation as a proportion of all partner organisations under the continuous process, the ARC should revisit the Watt Review recommendation to provide guidance to Linkage Projects SACs to ensure high quality applications that involve business partner organisations are given greater priority.
Increasing business and industry organisations’ collaboration (and investment) in research was a specific focus of government under the National Innovation and Science Agenda (NISA), and an intended outcome of the continuous process. The evaluation notes that business and industry organisations have provided the largest proportion of partner contributions under the continuous process.
ARC response
The ARC accepts this recommendation.
The ARC is responsible for funding the highest quality projects on the basis of a competitive peer review process against established assessment criteria. This is the key principle underpinning the ARC’s administration of the National Competitive Grants Program (NCGP) funding schemes.
Within this context, the ARC will provide advice to Linkage Projects SACs that at the funding margins, where appropriate, preference should be given to funding high quality applications that involve business partner organisations. The ARC will continue to monitor the outcomes of the Linkage Projects scheme with regard to the participation of different types of partner organisations.
Following the release of the report Boosting the Commercial Returns from Research in 2015, the ARC made a number of changes to the Linkage Program funding rules to strengthen the industry focus of the Program, including revising (i) an objective of the Linkage Projects scheme to emphasise opportunities to obtain commercial benefits from research; and (ii) the selection criteria to strengthen the focus on collaborative links with industry and commercial outcomes.
Recommendation 5: Data and monitoring
Recommendation
The ARC should consider using metrics provided in this report as benchmarks to monitor future performance of the continuous process and inform improvements.
Additional insights may also be gained from revisiting areas where sufficient data and information were not available to identify clear trends or draw firm conclusions at the time of this report’s preparation. The completion of the 2018 continuous round will offer numerous opportunities to build upon this report’s analysis. Among others, areas where further research and analysis may be useful include:
- the average value of partner organisation contributions per project
- partner organisation involvement by discipline
- total funded projects, success rates and return rates
- analysis of SAC member participation in meetings
- partner organisations entering new collaborations.
ARC response
The ARC accepts this recommendation.
The ARC will develop a set of metrics, based on those provided in the evaluation report, to monitor the performance of the Linkage Projects scheme application and assessment process.
The ARC notes that, given that the changes to the Linkage Projects processes have only been in place since 2016, clear trends could not be ascertained from the scheme data available at this stage. The ARC will release updated data incorporating the 2018 selection round.
Summary of actions
Table 1: Summary of actions identified in response to each recommendation
|
Recommendation |
Action |
|
1 |
|
|
2 |
|
|
3 |
|
|
4 |
|
|
5 |
|