QAF 2.6.2 notes that in rare instances, the Appraisal Committee may determine that the original external review of a new program proposal was inadequate and therefore invite further input from an external expert, either through desk review, or in person or virtual site visit.

The Appraisal Committee might decide to request an additional external review for the following reasons:

• The original reviewers did not have the appropriate expertise/qualifications and/or they were not at arm’s length.
  • QAF 2.2.1 states that external reviewers should normally be associate or full professors (or equivalent) and will have suitable disciplinary expertise, qualifications, and program management experience, including an appreciation of pedagogy and learning outcomes. 
  • Under the 2021 QAF, the Protocol for New Programs relies more heavily on peer-evaluation than it did under the previous QAF; the qualifications of external reviewers are therefore critical to ensuring a rigorous appraisal process.
  • Combined, the external review team must have adequate disciplinary expertise to cover all aspects of the new program (this is a particularly important consideration for interdisciplinary programs) and program management experience. Additionally, the external review team for proposals for graduate programs should include at least one member with experience with graduate programming.
  • The Quality Council’s guidance on choosing an arm’s length reviewer can be found here: Choosing Arm’s Length Reviewers (QAF 2.2.1 and 5.2.1) — Ontario Universities Council on Quality Assurance (oucqa.ca)
• The external reviewers’ report does not reflect a substantive engagement with the proposal.
  • The report should demonstrate that the reviewers considered the proposal with a critical lens and with an eye to how it could be improved, with recommendations that are clear, concise, and actionable.
  • A superficial or entirely positive report could indicate to the Appraisal Committee that the review was not sufficiently critical/engaged.
  • Similarly, the external reviewers’ report needs to provide an adequate degree of engagement with and coverage of the QAF’s evaluation criteria, as detailed in the new program proposal.
• The report does not include recommendations for how the proposal could be improved and/or the recommendations that are included are largely out of scope.
  • A key aspect in the quality assurance process is the academic unit and decanal engagement with the external reviewers’ report. To be able to make clear, concise, and actionable responses to the external reviewers’ recommendations, these need to be identifiable. Ideally, they will be listed in a separate section of the report. In the absence of discernible recommendations and a resulting set of challenging internal responses, a request could be made for the university to seek a revised external reviewers’ report and for the internal responses to be rewritten accordingly.
  • Note that if the report contains recommendations, but they are not clearly delineated, the Committee may not find it necessary to request a new external review. Instead, the university may be asked to pull out the recommendations, list them, and provide internal responses.

Steps the university can take to help prevent a request for an additional review:

• Provide a detailed orientation to the quality assurance process for Ontario’s universities, highlighting the critical role of external peer review in the appraisal process for new programs. You may want to provide this guidance, or a version thereof, directly to external reviewers: Guidance for External Reviewers of New Programs (QAF 2.2.1) — Ontario Universities Council on Quality Assurance (oucqa.ca).
• Provide external reviewers with a clear template for their report, which includes a section for a list of their recommendations. The template should also include all the evaluation criteria listed in QAF section 2.1.2, preferably in the same order. An optional template for use under the 2021 QAF is available on this page of the Quality Council’s website: https://oucqa.ca/resources-publications/templates/
• Communicate clearly (and early) with external reviewers that if their report is not adequate, e.g., if it does not contain a clear list of recommendations or if it is not sufficiently detailed, the university will follow up to ask for a revised report. Some universities include this as part of the external reviewers’ appointment letter or contract and withhold full payment of the honorarium until an acceptable report is received.
• For interdisciplinary programs and other programs covering a wide range of disciplines, carefully consider whether it is possible to cover all disciplines as well as the QAF requirement that external reviewers have program management experience (and experience with graduate programming, where applicable) with only two external reviewers. In some cases, you might consider engaging more than the required number of external reviewers from the beginning of the process.