Guidance for External Reviewers of Existing Programs (QAF 5.2.1)

Independent expert review is foundational to the Quality Assurance process for Ontario’s universities. Thank you for participating in this essential process.

This document provides an overview of Ontario’s quality assurance process, the Protocol for Cyclical Program Reviews, and includes an appendix detailing the distinction between Program Objectives and Program-level Learning Outcomes – a key aspect of the assessment of the quality and continuous improvement of existing programs.

Quality Assurance of Ontario’s Universities

The Quality Council is the provincial body responsible for assuring the quality of all programs leading to degrees and graduate diplomas granted by Ontario’s publicly assisted universities. It operates at arm’s-length from both the provincial government and the universities. The Quality Council does not make decisions regarding the funding of university programs; however, the provincial government’s Ministry of Colleges and Universities (MCU) will not fund a program which has not been quality assured and approved by the Quality Council.

Ontario’s universities have committed to a process to ensure the quality and continuous improvement of their academic undergraduate and graduate programs. The degree of rigour established throughout the Quality Assurance Framework (QAF) and in particular, the Protocol for Cyclical Reviews plays an essential role in ensuring the ongoing improvement of existing academic programs using internationally accepted quality assurance practices.

The Cyclical Program Review of existing programs is the key quality assurance process aimed at assessing the quality of existing academic programs[1], identifying ongoing improvements to programs, and ensuring continuing relevance of the program to students and other stakeholders. The Cyclical review consists of the following elements:

  • The self-study and external review provide internal and external perspectives on the institutional goals, program’s objectives, program-level learning outcomes, and graduate outcomes.
  • Degree Level Expectations, combined with the expert judgment of external disciplinary scholars, provide the benchmarks for assessing a program’s standards and quality.
  • The internal (i.e., program-level and decanal) responses to the externals’ reviewers’ report identifies changes needed to maintain the quality of the academic programs through the Final Assessment Report, which includes an Implementation Plan.
  • The required program changes identified in the Implementation Plan become the basis of a continuous improvement process through monitoring of key performance indicators. Independent expert review is foundational to this process.

The Role of the Quality Council

The Quality Council does not “re-do” the earlier external reviewers’ assessments; rather, it evaluates whether those assessments were comprehensively well done (that the main issues are addressed) and well received. This does not necessarily mean that the conclusions and recommendations are always welcomed; but that each has been reasonably considered and an appropriate plan has been developed to effect program improvement.

To this end, the Quality Council reviews the Final Assessment Reports and Implementation Plans, which provide the institutional synthesis of the external evaluation of the program and its strategies for continuous improvement. If the Quality Council finds an issue or potential area of concern in a university’s Final Assessment Reports and Implementation Plans, it may decide to take further action accordingly.

Therefore, when universities conduct Cyclical Program Reviews, they must demonstrate that the expert independent peer review adequately addressed all the main issues and was conducted at arm’s length.

Requirements of the External Reviewers’ Report

The elements that the external reviewers must address are specified in the Quality Assurance Framework (QAF), Section 5.2.1 and in the university’s Institutional Quality Assurance Process (IQAP). Minimally, the reviewers’ Report must:

  1. Address the substance of the self-study (see Section 5.1.3), with particular focus on responding to the evaluation criteria detailed therein;
  2. Identify and commend the program’s notably strong and creative attributes;
  3. Describe the program’s respective strengths, areas for improvement, and opportunities for enhancement;
  4. Provide evidence of any significant innovation or creativity in the content and/or delivery of the program relative to other such programs;
  5. Make at least three recommendations for specific steps to be taken that will lead to the continuous improvement of the program, distinguishing between those the program can itself take and those that require external action; and
  6. Identify the distinctive attributes of each discrete program documented in the self-study in those cases where a university chooses to simultaneously review more than one program / program level (for example, graduate and undergraduate), program modes, and/or programs offered at different locations.

While the external reviewers’ report may include commentary on issues such as faculty complement and/or space requirements when related to the quality of the program under review, recommendations on these or any other elements that are within the purview of the university’s internal budgetary decision-making processes must be tied directly to issues of program quality or sustainability.

Final Assessment Reports and Implementation Plans

Universities are required to prepare a Final Assessment Report which provides the institutional synthesis of the external evaluation of the program and strategies for continuous improvement. The Final Assessment Report includes all the recommendations of the external reviewers and the associated separate internal responses and assessments from the unit and from the Dean(s). The Implementation Plan prioritizes those recommendations that have been selected for implementation and sets out a clear action plan for implementation.

Units and Deans/Divisional Heads are best able to make concrete, considered responses when the external reviewers’ Recommendations are clear, concise, and actionable.

It is important to note that the QAF requires that an Executive Summary of the Final Assessment Report and the Implementation plan be published on the institution’s website. This ensures that students and other stakeholders can access information about program’s quality and its commitment to continuous improvement.

Footnotes

[1] In order to reduce confusion in cases where a single self-study refers to multiple degree options, streams and/or levels, institutions must clearly define the scope of the program to be reviewed in the Cyclical Program Review process and should convey this information to the external reviewers accordingly. This ensures that Recommendations are directed at the correct program and responsibility for implementing changes is assigned appropriately.