Transcript: Detailed Assessor Video
Video Link
Introduction
This video provides instructions and advice for Detailed Assessors about the assessment process of applications submitted for grant opportunities under the Australian Research Council’s National Competitive Grants Program (or NCGP).
The Assessment Process
The following flowchart shows the ARC’s assessment process. Peer review plays a critical role in the assessment of ARC applications. It is undertaken by two groups of experts known as General and Detailed Assessors. Experts from each group assess applications against the relevant grant opportunity assessment criteria and contribute to the process of scoring and ranking research applications.
The objective of the assessment process is to ensure that the highest quality research applications are recommended by the Selection Advisory Committee (or SAC) to the ARC Chief Executive Officer for funding.
The ARC CEO considers the SACs recommendations and, in accordance with the ARC Act, makes recommendations to the responsible Minister who ultimately decides which applications will be allocated funding under the NCGP.
The Role of an ARC Detailed Assessor
At the core of the ARC NCGP is a community of Australian and international researchers who peer review all research applications submitted to the ARC.
As a Detailed Assessor, you will review research applications within your field. You will make a valuable contribution to the research community by contributing your discipline specific expertise and knowledge to the ARC’s robust peer review process. In addition, you will keep up to date with current research and increase your own knowledge and ability to write a strong application.
Detailed assessors provide a written assessment and scores against specific assessment criteria for each application to which they are assigned.
These assessments are central to the considerations of the General Assessors in determining which applications to recommend for funding.
Detailed Assessors typically receive between five and ten applications a year for review, but this may vary depending on your specific expertise and the types of applications the ARC receives. Applications are matched to the expertise listed within your profile in the ARC’s Research Management System (or RMS).
The Research Management System (RMS)
RMS is a web-based computer system for the preparation and submission of research applications, assessments and rejoinders to the ARC. All ARC Assessors use RMS to submit their assessments.
To assist General and Detailed Assessors to navigate the assessment functionality in RMS, there is a RMS Handbook for Assessors available on the ARC website.
As a Detailed Assessor, you must ensure that your RMS profile is kept up to date.
Your expertise text should highlight your discipline-specific and/or interdisciplinary knowledge, including areas of speciality and techniques.
Please select your most relevant Field of Research (FoR) Codes.
An accurate and up to date Employment History helps minimise the chances that you will be assigned an application with which you have an organisational conflict of interest.
Confidentiality and Conflict of Interest (COI)
The ARC Conflict of Interest and Confidentiality Policy is designed to ensure that all Conflicts of Interest (or COIs) are managed in a rigorous and transparent way. It aims to prevent individuals from influencing decisions unfairly and to maintain public confidence in the integrity and fairness of the peer review process.
Any individual who is reviewing material for the ARC must agree to the ARC’s COI and confidentiality statement.
As a Detailed Assessor, you must declare any COIs related to your peer review task as soon practicable in RMS and reject assignments if a COI exists. This will assist in the timely re-assignment of applications.
Failing to declare a COI may result in a Detailed Assessor being investigated for breaching the ARC’s Research Integrity Policy.
For any COI queries, please contact the ARC using the peer review mailbox.
What to Disclose?
Detailed Assessors must disclose any close personal relationships with named participants on a research application as well as any professional relationships with that named participant including:
- Conjointly holding funding within the past two years with that named participant
- Having a current application or negotiating an application for funding with that named participant
- Being a collaborator or co-author with that named participant on a research output within the past four years
- Being a co-editor of a book, journal, compendium, or conference proceedings with that named participant within the past two years
- Being a postgraduate student or supervisor of that named participant within the past five years
- Benefitting materially from the awarding of funding to the application involving that named participant and
- Other situations as outlined in the COI Policy on the ARC website.
Detailed Assessments
You may be assigned a number of applications within your field of research or across a broader disciplinary area based on your RMS profile expertise text and FoR codes.
Detailed Assessors are asked to review the application to identify merits or otherwise of the proposed research and complete in-depth assessments of applications in RMS. This includes providing scores and comments against grant opportunity assessment criterion separately.
Detailed Assessors may receive applications to assess at any stage of the assessment process due to late COIs being declared by other assessors.
Please remember that Detailed Assessor comments are made available to Applicants anonymously once an application is open for Rejoinder. The purpose of the Rejoinder process is to allow applicants to respond to comments made by Detailed Assessors. Detailed Assessor comments will also be accessible to the SAC.
Please note that ARC-funded researchers are obligated by their Grant Agreement to provide assessments. Assessing is a great way to support Australia’s research sector and read interesting grant applications.
What Makes a Quality Assessment?
A quality assessment:
- is a thoughtful, meaningful and balanced assessment that provides fair and objective information about the key merits or otherwise of the application with respect to the assessment criteria
- includes relevant comments and criticisms that are justified and aligned to the scores
- enables applicants to undertake an informative and reasonable rejoinder
- enables the SAC to make informed decisions about funding recommendations
- does not include any inappropriate elements as outlined in the Assessor Handbook.
How to Ensure High Quality Detailed Assessments
Detailed Assessors are asked to provide high quality, constructive assessments with the following elements:
- Objective, constructive and professional comments.
- Detailed comments on the merits or otherwise of the application with respect to the selection criteria.
- Sufficient information to allow applicants to provide a Rejoinder.
- Comments that align closely with scores
- Comments that are fair, meaningful and balanced.
- Comments free from exaggeration and understatement.
- Timely submission via RMS by the ARC deadline.
- Observation of the ARC Conflict of Interest and Confidentiality Policy.
Please refer to the ARC Peer Review webpage for examples of good Detailed Assessments.
Avoiding Inappropriate Assessments
When completing an assessment, do not:
- Restate or rephrase excessive parts of the application
- Include acronyms or scores within the assessment text
- Provide scores which do not align with the assessment text
- Provide comments comparing one application with another
- Provide very brief assessment text without any explanation or justification
- Provide information that identifies researchers named on other applications
- Simply quote the rubric or restate the metrics provided by the applicant
- Use the same generic comments across multiple assessments
- Provide advice about your own identity or standing in the research field
- Comment on the outcome or status of relevant research not mentioned in the application
- Comment about the potential ineligibility of an application
- Provide comments that can be perceived to be discriminatory, defamatory or distastefully irrelevant
- Provide comments that bring into question the integrity of an application
- Provide comments regarding whether the application satisfies the National Interest Test.
Under no circumstances should Detailed Assessors contact researchers or institutions about a submitted application or seek additional information from any sources. This includes following any hyperlinks that may have been included in the application.
Treatment of Inappropriate Assessments
Inappropriate Assessments compromise the integrity of the peer review process.
To be fair to all Applicants, the ARC will review assessments that have been identified to contain inappropriate comments.
If the ARC considers an assessment to contain inappropriate comments, the ARC has absolute discretion to decide what action will be taken.
Actions may include requesting a Detailed Assessor to amend their assessment text, removing the assessment from the peer review process or investigating a breach of the ARC’s Research Integrity Policy.
Scoring and Ranking Assessments
The Scoring Matrix table on the screen shows an example of the scores applicable to grant opportunities.
It is important that Detailed Assessors follow the Scoring Matrix applicable to each individual grant opportunity.
Objectives and Assessment Criteria
Assessors must have regard to both the objectives and the assessment criteria as outlined in the relevant Grant Guidelines of each scheme.
Grant Guidelines can be accessed on GrantConnect.
Research Opportunity and Performance Evidence (ROPE)
Research Opportunity and Performance Evidence (or ROPE) provides a framework within which the quality and benefit of achievements is given more weight than the quantity or rate of particular achievements.
ROPE aims to ensure that NCGP assessment processes accurately evaluate a researcher’s career history and personal circumstances relative to their current career stage.
It also considers whether their productivity and contribution is commensurate with the opportunities that have been available to them.
Unconscious Bias
As Assessors you should also be aware of how your unconscious bias could affect the peer review process. The ARC encourages you to recognise your own biases and be aware of them in your assessments.
All Detailed Assessors are encouraged to complete Implicit Bias training offered by Harvard.
The National Interest Test
Detailed Assessors may consider the content of the National Interest Test Statement when assessing the overall application against the assessment criteria. However, it is not your role to assess whether the application satisfies the National Interest Test. Your assessments should not include comments about the National Interest Test.
As a separate process, the ARC CEO will consider the National Interest Test statements of all applications recommended by the SAC.
Interdisciplinary Research
The ARC is committed to supporting interdisciplinary research. Assessors are required to assess all research on a fair and equal basis, including applications and outputs involving interdisciplinary and collaborative research.
Applications outside your area of expertise
If you believe that the ARC has misunderstood your expertise, or has made an error in assigning an application to you, please give early notice to the ARC by rejecting the application in RMS and entering a reason in the comment box.
To assist the ARC to accurately assign applications, please review your expertise text in RMS regularly to ensure it remains accurate and up to date.
Eligibility
If, while assessing an application, you have concerns about eligibility, ethics or other issues associated with an application, you must not include this information in your assessment.
Please send an email highlighting your concerns to the peer review mailbox as soon as possible. Please complete your assessment based on the merits of the application without giving consideration to the potential eligibility issue.
The ARC’s NCGP Eligibility Committee is responsible for reviewing the eligibility of applications and will be advised of your concerns.
Conclusion
Thank you for watching this short video on the assessment process for Detailed Assessors and thank you for your valuable contribution to Australian research through your participation in the ARC’s peer review processes. We look forward to reviewing your assessments.
If you require any assistance with your assessment activities, please email ARC-Peer_Review@arc.gov.au