A quality assessment:
- is a thoughtful, meaningful and balanced assessment that provides fair and objective information about the key merits or otherwise of the application with respect to the assessment criteria
- includes relevant comments and criticisms that are justified and aligned to the scores
- enables applicants to undertake an informative and reasonable rejoinder
- enables the Selection Advisory Committee (SAC) to make informed decisions about funding recommendations
- does not include any inappropriate elements as outlined in the Assessor Handbook.
Quality Assessment Criteria
Expectations of the ARC for high quality detailed assessments to support a robust peer review process include:
- Objective, constructive and professional comments.
- Detailed comments on the merits or otherwise of the application with respect to the selection criteria. One or two sentences is not sufficient – minimum of one paragraph.
- Sufficient information to allow applicants to provide a Rejoinder to comments about the application.
- Comments that align closely with scores—for example, if you have given significant criticisms an ‘A’ rating is unlikely to be appropriate. Further, if a ‘D’ rating is given, then suitable constructive criticisms and comments justifying the score are required. It is important to remember that applicants see only the comments and the SAC will see both comments and scores. Assessors are encouraged to identify specific aspects of applications that warrant the score given, rather than provide broad general comments.
- Comments that are fair, meaningful and balanced, addressing only issues relevant to the application in terms of the selection criteria. Comments should provide a sound, comprehensive account of, and justification for, views about the application, while respecting the care with which applications have been prepared.
- Comments free from platitudes, exaggeration and understatement.
- Timely submission via RMS by the ARC deadline.
- Observation of the ARC Conflict of Interest and Confidentiality Policy and reporting of anything you are concerned about to the ARC.
Remember that while your assessment is anonymous to the applicant it will be seen by a panel of your peers, the General Assessors, often consisting of the ARC College of Experts members and highly esteemed industry representatives.
Inappropriate Assessment Elements
In order to ensure you provide the most helpful information in your assessment it is important that you do not:
- Restate or rephrase excessive parts of the application.
- Include acronyms or scores within the assessment text.
- Provide scores which do not align with the assessment text.
- Provide comments comparing one application with another.
- Provide very brief assessment text such as ‘Good application’ without any explanation or justification.
- Provide information that identifies researchers named on other applications – respect confidentiality and privacy concerns.
- Just quote the rubric or restate the metrics provided by the applicant – instead provide comments detailing or questioning the projects quality and innovative value.
- Use the same generic comments across multiple assessments or text that has been copied from a previous assessment.
- Provide advice about your own: identity or standing in the research field in the application.
- Comment on the outcome or status of relevant research not mentioned in the application.
- Comment about the potential ineligibility of an application as this process is independent of the peer review process. Eligibility concerns should be provided by email to ARC-Peer_Review@arc.gov.au.
- Provide comments that can be perceived to be discriminatory, defamatory or distastefully irrelevant (such as gratuitous criticism of a researcher and/or Eligible Organisation).
- Provide comments that bring into question the integrity of an application (any concerns in relation to potential non-compliance with the Australian Code for the Responsible Conduct of Research or the ARC Research Integrity and Research Misconduct Policy should be raised with the ARC Research Integrity Office by email to researchintegrity@arc.gov.au).
- Provide comments regarding whether the application satisfies the National Interest Test.
Assessment Examples
High Quality Assessments for Strong Applications
Example 1:
Investigator(s) (A)
The CI has an extremely strong track record overall. She has been a competitively-funded research fellow for her entire career with relatively few teaching duties and has made the most of these opportunities. Her research activities have made a significant contribution to her university department’s international standing. The CI has published a large body of work and has made important contributions – particularly to our understanding of the capacity of mixed methods research to provide insights into complex social issues associated with diverse regional labour markets. It is clear that she has a high standing in this field internationally, judging from the citation of her work across a range of social science disciplines and invitations to speak at international conferences that focus on the design and implementation of mixed method research programs in key areas of social and economic policy. I note particularly the regular publication (averaging 5 papers annually over the last ten years) in very highly regarded international journals as well as second-tier journals that focus on the Australian policy environment. She is a corresponding author on well over half of these, which demonstrates that she is a major driver in these pieces of work. My only negative comment relates to the lack of a strong record of public outreach/engagement, however recent contributions to both The Conversation (2017) and ABC Radio National interviews (2016, 2018) suggest a growing track record in this area.
Program of Research Activity (A)
The proposed program of research represents a new direction focused on regional life, well-being and access to economic resources among an ageing population. The program builds on and extends the CI’s established areas of expertise. A key part of the application is to build capability at the Administering and Partner Organisations as an exemplar in the integration of social research methods to address key social and economic challenges. The suggested program of research is the first in Australia to adopt such an integrated and large range of disciplines (anthropology, information and communications technology, geography, psychology, sociology, economics and public policy) to this particular type of program. This approach has the potential to provide a much deeper understanding of the implications of the challenges and opportunities for regional Australia if it can address practical issues of cross-discipline communication and coordination. The CI’s previous intellectual contributions and national and international networks, providing some confidence that these challenges will be addressed successfully. The application also carefully describes and justifies an appropriate budget for the significant administrative resources that will be required to manage successfully a large and geographically dispersed team. Overall, this exciting application has the potential to make major contributions to theory, applied research and policy.
Mentoring/Capacity Building (A)
Capacity building: CI has an extremely impressive track record of obtaining research funding and building research capacity. Particular highlights include leadership of a major Research Centre at the admin org, which attracts significant external funding, alongside two ARC DP grants in the last 6 years. These, together with her other leadership roles, clearly demonstrate an ability to lead and to contribute to large initiatives that strengthen the admin organisation’s and Australia’s ability to carry out the highest quality research in the social sciences. Mentoring: The CI has worked with a number of mid-career researchers and supervised doctoral students who have graduated and later been awarded independent fellowships and research grants. These outcomes appear to have been supported by the CI’s provision of lead authorship opportunities to early career researchers in the formative stages of their careers. I am unclear about the development opportunities in the long term for the named research assistants. Perhaps the CI could expand a little in this area if there is a chance to do so.
Comments ()
Overall, this is a very strong application. The CI has an enviable track record and the application describes an exciting and innovative initiative. My key reservation is her capacity to maintain publication output while mentoring junior colleagues to independence. Further evidence of the CI’s systematic approach to mentoring and links with the career outcomes of former students would be helpful.
This assessment meets criteria 1 to 6 of a high quality assessment. It is:
- objective
- detailed
- appropriate for applicants to address in a rejoinder
- consistent with scores and assessment text aligning
- fair and meaningful
- free of inappropriate elements.
Example 2:
DECRA Candidate (A)
The candidate’s research performance to opportunity is excellent, demonstrated through highly regarded journal publications that are being cited by some very senior and well respected researchers and policy makers. His position as first author on 15 of the 17 journal papers is a testament to the critical role that he has played in driving and executing this research. This is particularly impressive given the breaks in the candidate’s career progression for challenges associated with a medical condition.
Project Quality and Innovation (B)
The candidate demonstrates an excellent grasp of the sociological theories that underpin this project on technology in the workplace. The project is challenging and innovative, especially for an early career researcher who is in the initial stages of developing the type of international networks required for a multi-site, international study of this nature. The participant samples are appropriate but I have reservations as to whether the very broad scope of this project is practical for a three year time-frame and modest budget. While I believe the project, as stated, could deliver excellent results with important implications for theory and policy, I am concerned that the scheduling and anticipated costs of data collection underestimate the requirements of the project as it currently stands. Challenges with data collection are likely to impose constraints on the analysis that can be undertaken, particularly if it compromises the number of sites and reduces the comparative nature of the project.
Feasibility (B)
Some issues regarding feasibility have been noted above with respect to the scope of the project and possible constraints on data collection. I also note a lack of international collaborations among the co-authors on the candidate’s journal publications. Again, there is limited evidence that the candidate has succeeded in building the international and interdisciplinary collaborations that would facilitate project implementation and poses a risk to achieving its outcomes. A greater demonstration of building or working in international collaborations would provide confidence that an ambitious and well designed project can be implemented successfully.
Benefit and Collaboration(C)
The project as described has significant potential benefits for Australia’s research profile in the sociology of technology and important implications for policy. Although there is an impressive academic publication plan, the scope of the proposed data collection and analysis makes it likely that much of the publication output will occur well after the completion of project funding and so will be, in part, contingent on the candidate’s academic role and duties following the DECRA. The supportive letter from the administrative organisation is encouraging in this respect. It is also possible that systematic policy discussions may occur following the conclusion of project funding but there is limited information about such plans within the project description. I have already noted the advantages that could result from greater collaboration to assist with implementing the design of this project and so will just note that the candidate may benefit from greater attention to this aspect of the planned research activities.
This assessment meets criteria 1 to 6 of a high quality assessment. It is:
- objective and constructive
- detailed in its justifications
- appropriate for applicants to address in a rejoinder
- consistent with scores and assessment text aligning
- fair and respectful when posing questions
- free of inappropriate elements.
Example 3:
Investigator(s) (A)
The lead CI is one of Australia’s most prolific political scientists as evident by 21 book manuscripts and more than 100 journal articles, many of which are in leading international journals. In addition, she has the required prior experience and necessary networks to facilitate project implementation across all three sites (one in Australia and two international). The project’s PI is an early career researcher with an emerging record of high quality publications and successful collaborations with more senior researchers in the discipline. The PI’s record of designing survey instruments, and collecting and analysing survey data on sensitive political issues will be an asset to the project.
Project Quality and Innovation (A)
The research project is designed around three research questions, which are well defined and complementary in addressing the project’s overall objective of understanding challenges to the Westminster system of democracy in three countries. Each research question focuses on a different unit of analysis and aligns with detailed plans for data collection and appropriate analytical approaches. The proposed data collection from individual citizens (research question three) is the most challenging but is also innovative and provides the largest potential for making an insightful theoretical contribution. Therefore, I believe the project makes a compelling case for both innovation and promises to contribute to knowledge as well as has important policy implications.
Feasibility (A)
The project is well designed and achievable within the proposed time and budget. It will build on an existing project and so much of the required infrastructure and networks are in place. The CI and PI have appropriate experience and training to conduct both proposed data collection and analysis. The established international links with seven highly relevant international research centres provides further assurance of a collaborative research environment that enhances the feasibility of the project.
Benefit (A)
The findings will provide important insights useful for scholars as well as policy-makers and non-governmental actors working in the area of citizen engagement and thus will have great benefits beyond academia. Outreach activities, including three workshops and an international conference are planned and supported with appropriate budget items and associated duties allocated to project personnel. The outcomes of this project can be expected to raise debate and have policy implications well beyond academia.
This assessment meets criteria 1 to 6 of a high quality assessment. It is:
- objective and professional
- detailed with justifications
- appropriate for applicants to address in a rejoinder
- consistent with scores and assessment text aligning
- fair and respectful when posing questions
- free of inappropriate elements.
High Quality Assessments for weak applications
Example 1:
Investigator(s) (B)
Both Professors (CI1 and CI2) are extremely well established scholars in the field of primary school education, with a clear track record of contributions to pedagogy and classroom practice which has spanned their careers. They are certainly amongst the leading scholars in their field and have both previously held ARC grants in this area. As an ECR, CI3 understandably has a less established record in the area, though he has published several good journal articles and appears to play a very active role in engaging with professionals within the education sector.
Project Quality and Innovation (D)
The project aims to refine the ABC learning program and evaluate its success with a well-specified and sensible selection of primary school environments. The research involves two streams of analysis: (a) refinement and piloting of the ABC learning program, with a consultative program that incorporates feedback from teachers, parents and education departments, and (b) a larger scale program of implementation and evaluation. It is certainly an interesting project, but it is unclear from the research description just how much this particular project will advance previous work which has already been undertake by Smith, Nguyen and Miller. In what particular ways will this project take the program forward and how does it differ from the international and Australian studies cited in the application that are also investigating ABC programs? The proposed extensions to the ABC model appear to be incremental. There is a lack of clarity around the purpose and research outcomes associated with three different sets of focus groups. The analytical approach is also unclear with some sections of the proposal focussing on content analysis and others on emergent theories of teaching. The number of groups and individuals proposed in the research description (six groups and 40 participants) seems out of proportion to the breadth of the study – why so few across multiple sites with different stakeholders? There is also a discrepancy here with the interview transcription services provided for in the budget (over $30,000 for 65 interviews). It would be helpful to have greater clarity on the focus groups’ purpose, design and data analysis.
Feasibility (D)
As noted, several aspects concerning the specific scope and breadth of the research are unclear from reading the description of the research project (for example, timeframe, number of groups and participants), which makes it difficult to evaluate feasibility. That said, it appears to be a very substantial project and I wonder whether the CIs might benefit from additional consultation with others engaged in broadly similar projects.
Benefit (C)
The proposed publication path is sound, and the plan of holding academic and practitioner workshops is also a good idea. Most of this is focused on an Australian audience, and some consideration could be given to international collaboration and engagement. On an organisational note, who would be responsible for organising the planned conference for year three and maintaining the project website? There is no mention of these activities in the roles of the project personnel.
Improvements ()
As noted already, the application would benefit from greater care in defining key project activities (focus groups) with clear methods and analysis and consistency with budget requests. Greater detail on the allocation of responsibilities to manage outreach activities would also provide greater confidence about the likely benefits of the project.
This assessment meets criteria 1 to 6 of a high quality assessment. It is:
- objective and constructive
- detailed in its justifications especially for low scoring criterion e.g. Project Quality and Innovation
- appropriate for applicants to address in a rejoinder
- consistent with scores and assessment text aligning
- fair and respectful while providing constructive comments
- free of inappropriate elements.
Example 2:
Investigator(s) (B)
There are 4 academics and 3 partner investigators on the team. The academics, all from a single university, are experienced researchers - some very senior with past competitive grant projects success. The expertise of the researchers is highly relevant to the project and the partners will play a significant role in providing context for the intended research. Though the senior academics adopt the lead researcher roles, the commitments of all researchers is 0.2 (each). While this may be workable, it is the case that the lead researcher could show a higher commitment in time than the others to indicate a greater responsibility for seeing the project through. This is a minor quibble in what looks to be a strong team with runs on the board in terms of publications and research project experience.
Project Quality and Innovation (C)
The projects' interest in the examination of the relationship of School-based learning outcomes and community sports participation is interesting and potentially important. There is a statement that this approach is based on earlier research projects undertaken by the CIs. My question then is not so much about the methodology itself but how this project can be framed as an innovative approach. A few of the participants have the "word" innovation in their titles, but there is little specific evidence that the methodology is innovative. The project design suggests that there will be substantial data generated by the project. Much of this data will cross the quantitative/ qualitative divide and while this is not unusual in projects such as these, the stated methods appear too general in the way they are both rationalised and outlined. There is a lack of specificity in how the data, once collected, will be analysed and then translated into outcomes with specific impacts.
Feasibility (C)
The case for feasibility is made in a section in the application under that title. It states how the research will be connected to patterns of the school year with the expectation that these will afford opportunities for researchers to align themselves to periods when students are undertaking community sports activities. There is a sense that this strategy has worked in the past and is being deployed here on that basis. Beyond this, the application establishes that the University's various research groupings provide organisational backing should and when it is required. Whether these argue successfully for feasibility is based on whether the idea that past practices and formally constituted maintenance groupings will ensure a successful project.
Benefit (C)
Under the section with the title of benefit, a relatively brief section highlights the benefits the project will return to students, the partner institutions and the University. As the main target of the research, the students should, in principle, be the main beneficiaries but the possible benefits are not well defined. Benefits to the partner organisations are implied rather than stated, although their participation in the project may provide some implicit evidence that benefits, while undefined, are expected. Finally the University is seen to benefit out of the expanded relationships to these institutions in the sporting community that will give the University a better profile as an actor with impact in local community organisations. I am not sure any hard evidence has been provided to demonstrate benefit here, however the likelihood of these benefits accruing is quite strong given the previous track records of the CIs.
Improvements ()
I think the application lacks a degree of specificity when it comes to both the statement of the problem and the methodology, particularly in regards to the innovations. The project appears feasible but the potential benefits are implied rather than clearly stated.
This assessment meets criteria 1 to 6 of a high quality assessment. It is:
- objective and constructive
- detailed in comments relating to the applications merits with respect to the selection criteria
- appropriate for applicants to address in a rejoinder as the assessor has provided points requiring further clarification
- consistent with scores and assessment text aligning
- fair and respectful while providing constructive comments
- free of inappropriate elements.
Example 3:
Investigator(s) (C)
The investigators are an interdisciplinary team with appropriate expertise to study psychological, organisational and industrial factors that might be associated with physical work hazard exposure and resulting illness and disease. CI3 and CI4 bring expertise in organisational psychology and their publications evidence their expertise in relevant quantitative methods. CI2 has a strong publication history of applied occupational safety research particularly on shiftwork and operator fatigue, including studies of commuting among workers. CI3 and CI4 do not report experience conducting research in heavy industry contexts but have proven experience completing occupational health and safety research in the professional service sector. CI1 has a strong track record of publication in the area of management and labour relations and his significant publications include a very high proportion of outputs on which he is first author. However, it is not made clear how his core expertise as a historian will contribute to this project. Three of the CI team have experienced significant health or life events which would have affected their productivity for a period and all the CIs have experienced periods of heavy teaching and/or administrative demand during their academic careers. Nonetheless, each has built a substantial body of research outputs relative to their career duration, usually publishing in high quality journals in their area.
Project Quality and Innovation (C)
The problem is important. The reason existing risk management regimes have become less effective in controlling related diseases in workers needs to be understood to underpin prevention efforts. The conceptual approach is not very novel, contradicting the CIs’ claim of innovation. It uses fundamental, orthodox ideas of modern WHS. It is confusing that Figure 3 does not show the relation between hazards and health being mediated by other factors. The relevance to the study of the distinction between the current risk management approach to WHS regulation and the earlier prescriptive approach is not well-argued. Nor is it clear why behavioural approaches to health and safety are listed as approaches to regulation. The method captures a range of relevant data but will self-reported, historical work and exposure data. There is a legitimate concern as to whether this will be reliable and detailed enough for the intended purpose A clearer statement is needed of how the sources of data, collected on different timescales, will be combined. The design will not include data relevant to workers health outcomes before the current increase in disease and so will preclude finding a change in substance characteristics related to change in disease occurrence.
Feasibility (C)
The project timeline and budget generally seem feasible with some exceptions. For example, the part-time casual Research Assistant (RA) is expected to enter all employee surveys every year (p.12). This is unrealistic. Have sufficient funds been requested for adequate support staffing? The prolonged support of the Staff Advisers from Industry participants is critical to the success of the study. Will there be Ethical issues for them in providing data for the study? Some assurance of clear attention to the need for Human Research Ethics clearance would be helpful in assessing feasibility.
Benefit (B)
Dissemination is not well considered. There is no aim to publish in specialist WHS journals or to brief Unions, Industry or Regulators. If done well, the results of a study such as this have the potential to benefit the Australian community by reducing the social and economic burden of occupational disease among Australian workers. The cost of the study would be small in comparison to the economic and other burdens of a resurgence of chronic debilitating or fatal illness in this industry. Many of the costs of conducting the study are minimised because sources of data that are already collected for other purposes will be used.
Improvements ()
Typos were distracting and did not inspire confidence in the application. Much material presented as conceptual background was not clearly linked to the study. The description of the rationale and methods could have been more streamlined. Who is the Professor referred to on p.31?
This assessment meets criteria 1 to 6 of a high quality assessment. It is:
- professional and constructive
- detailed with specific examples to demonstrate the points made
- appropriate for applicants to address in a rejoinder as the assessor has provided points
- requiring further clarification
- consistent with scores and assessment text aligning
- fair and respectful while providing constructive comments
- free of inappropriate elements.
Poor Assessments regarded as Inappropriate/Unhelpful
Example 1:
Investigator(s) (B)
Excellent background well suited to the project.
Project Quality and Innovation (A)
This is a clear and persuasive application. The literature review is comprehensive, the core questions well defined, and the research design is appropriate and thoughtfully constructed.
Feasibility (A)
Solid and clear
Benefit (A)
Good
This assessment does not meet criteria 1, 2, 3 or 4 of a high quality assessment. It is:
- not constructive
- fails to provide justification for the judgements of “solid and clear” or “good”
- does not provide enough detail for the participants to address in the rejoinder
- disparate with comments such as ‘Excellent’ conferred a score of B and ‘Good’ scored as an A
It is considered inappropriate as it:
- Is too brief
- Does not outline the merits of the application and is therefore unhelpful in informing the applicants and the General assessors of why this application is deserving of funding i.e. What discriminates this good application from the excellent within a pool of high-quality applications?
Example 2:
Investigator(s) (B)
Solid track record
Project Quality and Innovation (B)
This is a clear application and cross-national research is welcome, although it remains unclear how the research will relate to the initial problems which are presented in the opening section of the application. The research questions remain very descriptive rather than drawing upon clear theoretical arguments and the existing research lit. And why not include a broader set of case studies?
Feasibility and Benefit (C)
Fair
Research Environment (C)
Fair
This assessment does not meet criteria 1, 2, 3 or 5 of a high quality assessment. It is:
- not constructive
- fails to provide justification for the judgements of “solid” or “fair”
- does not provide enough detail for the participants to address in the rejoinder
- not addressing issues relevant to the application and accounting for their ‘fair’ assessment
It is considered inappropriate as it:
- Is too brief
- Generic with comments such as ‘fair’ not qualified
- Is unprofessional
Example 3:
Investigator(s) (C)
The team should have the appropriate skills and experience for the project research.
Project Quality and Innovation (E)
The theoretical section is underdeveloped, weak, and confusing, with a lot of repeated assertions but no sense of what existing theoretical frameworks this is being built upon, which theories would disagree with the argument, what core propositions are being tested, and how the cases will contribute towards the answers. The case selection also seems ad hoc and the rationale for selection remains unclear.
Feasibility (E)
Not feasible
Benefit (E)
Unrealistic
Improvements ()
This application needs rejecting, not amending.
This assessment does not meet criteria 1, 2, 3, 5 or 6 of a high quality assessment. It is:
- not constructive, objective or professional
- fails to provide justification for the judgements of “not feasible” or “unrealistic”
- does not provide enough detail for the participants to address in the rejoinder
- not meaningful or helpful and shows no respect for the applicants efforts
It is considered inappropriate as it:
- Is too brief
- Is offensive and dismissive of the applicants’ research
- Is insolent in the ‘Improvements’ recommendation
Example 4:
Investigator(s) (C)
Fair
Project Quality and Innovation (E)
The theory starts with a series of assertions but this reflects one perspective and the applicants should acknowledge extensive theory about rational workforce participation decisions in the context of household roles and obligations. Throwing in a few ad hoc and vague country names as examples is not evidence of anything. The 3rd sentence in the aims section (economic independence requires that...), followed by a series of diverse citations thrown together, is incomprehensible. The idea that economic independence can develop from greater welfare assistance is misguided and this certainly needs unpacking not just asserting. Overall the project description is very, very incoherent and confused. Not anywhere near the quality for fundable research.
Feasibility (E)
Poor and confused
Benefit (E)
No benefit apparent.
Improvements ()
Reject
This assessment does not meet criteria 1, 2, 3, 5 or 6 of a high quality assessment. It is:
- not objective or constructive
- fails to provide justification for the judgements of “fair” or “poor and confused”
- does not provide enough detail for the participants to address in the rejoinder
- not meaningful or helpful and shows no respect for the applicants efforts
It is considered inappropriate as it:
- Is too brief
- Is offensive and dismissive of the applicants’ research
Example 5:
Investigator(s) (E)
I have serious doubts about the lead CI’s ability to deliver this project. I was unfortunate to collaborate with him on a number of publications in recent years, and partnered on an ill-fated DP application last year which fell apart due to his overbearing and tendentious manner. If this is how he behaves towards other researchers it is unlikely he will be able to bring this project, which requires extensive collaboration across institutions, to fruition.
Project Quality and Innovation (C)
Not at the forefront but solid.
Feasibility (C)
Generally OK.
Benefit (D)
Unclear.
This assessment does not meet criteria 1, 2, 3, 5, 6 or 8 of a high quality assessment. It is:
- an obvious conflict of interest (on a professional level) that should have been rejected by the assessor and warrants referral to the ARC Research Integrity Officer
- not constructive, objective or professional
- fails to provide justification for the judgements of “generally OK” or “unclear”
- does not provide enough detail for the participants to address in the rejoinder
- not meaningful or helpful and shows no respect for the applicants’ efforts
It is considered inappropriate as it:
- Is too brief
- Is offensive and insulting of the lead applicant on a personal level
Note: Unprofessional behaviour and the failure to declare a conflict of interest such as that demonstrated in Example 5 above may warrant referral to the ARC Research Integrity Officer.
Further educational resources are located on the Assessor Resources page.