Guidance for External Reviewers of New Programs (QAF 2.2.1)

Independent expert review is foundational to the Quality Assurance process for Ontario’s universities. Thank you for participating in this essential process. Your Report will be the primary focus of the Ontario Universities Council on Quality Assurance (the Quality Council) and its Appraisal Committee as it considers the quality of the New Program.

This document provides an overview of Ontario’s quality assurance process, the Protocol for New Program Approvals, and includes an appendix detailing the distinction between Program Objectives and Program-level Learning Outcomes – key criteria in the appraisal of New Programs.

Quality Assurance of Ontario’s Universities

The Quality Council is the provincial body responsible for assuring the quality of all programs leading to degrees and graduate diplomas granted by Ontario’s publicly assisted universities. The Quality Council operates at arms-length from both the provincial government and the universities. While universities have vested in the Quality Council the final authority for decisions concerning approval of new programs, universities must apply separately to the provincial government’s Ministry of Colleges and Universities (MCU) for funding. The MCU will not approve funding for a program which has not been quality assured and approved by the Quality Council.

Ontario’s universities have committed to a process to ensure the quality and continuous improvement of their academic undergraduate and graduate programs, from inception. The degree of rigour established throughout the Quality Assurance Framework (QAF), and in particular, in the Protocol for New Program Approvals plays an essential role in ensuring that new programs are developed using internationally accepted quality assurance practices and that the quality of that new program is sustained.

The primary responsibility for the design and quality assurance of new programs lies internally, with universities and their governing bodies. When preparing a New Program Proposal, universities are responsible for the development of program objectives (see Guidance) and curriculum design, the creation and clear articulation of program-level learning outcomes (see Definition and Guidance), their monitoring and the design of their assessment, and generally for the assembly of human, instructional and physical resources needed to achieve those program-level learning outcomes.

The role of expert independent peer review

There are three levels of assessment for quality assurance: primary, secondary, and tertiary. Primary assessment occurs at the unit level where the program itself engages in the development of new programs.

Secondary assessment involves independent expert review conducted at arm’s length. This includes recommendations from you as the external reviewer that are clear, concise and actionable.

The Quality Council and its Appraisal Committee engage in tertiary assessment. They do not “re-do” the earlier assessments; rather, they evaluate whether those assessments were comprehensively well done (that the critical criteria required by the Framework have been addressed) and independently and appropriately assessed (that the appraisers are arm’s-length, have an appreciation of pedagogy and learning outcomes, and are appropriately knowledgeable in the proposed program’s area of discipline).

For New Program Proposals, these evaluations are made by the Appraisal Committee, which will, in the first instance, focus its review of a new program proposal on the following elements of the submission:

  1. Overall sufficiency of the External Review Report(s);
  2. Recommendations and suggestions made by the external reviewers, including on the sufficiency and quality of the planned human, physical and financial resources;
  3. Adequacy of the internal responses by the unit and Dean(s) to the recommendations, or otherwise for single department Faculty; and
  4. Adequacy of the proposed methods for Assessment of Teaching and Learning given the proposed program’s structure, objectives, program-level learning outcomes and assessment methods. (See Evaluation Criteria 2.1.2.4 a) and b))

Once the Committee has completed its review of the submission to its satisfaction, it makes one of the following recommendations to the Quality Council:

  1. Approved to commence[1];
  2. Approved to commence, with report; [2]
  3. Deferred for up to one year during which time the university may address identified issues and report back;
  4. Not approved; or
  5. Such other action as the Quality Council considers reasonable and appropriate in the circumstances.

Therefore, when universities submit New Program Proposals to the Quality Council’s Appraisal Committee, they must demonstrate that the expert independent peer review addressed all the main issues and was conducted at arm’s length.

Evaluation Criteria

The elements that the external reviewer must address are specified in the Quality Assurance Framework, Section 2.2.2 and in the university’s Institutional Quality Assurance Process (IQAP). Minimally, the reviewers’ Report must:

  1. Address the substance of the New Program Proposal;
  2. Respond to the evaluation criteria set out in Framework Section 2.1.2 (see the Sample Template for the External Review Report for a detailed list of minimally required criteria);
  3. Comment on the adequacy of existing physical, human[3] and financial resources; and
  4. Acknowledge any clearly innovative aspects of the proposed program together with recommendations on any essential or otherwise desirable modifications to it.

It is important to note that, while the external reviewers’ report may include commentary on issues such as faculty complement and/or space requirements when related to the quality of the new program, recommendations on these or any other elements that are within the purview of the university’s internal budgetary decision-making processes must be tied directly to issues of program quality or sustainability

An important outcome of the Protocol for New Program Approvals is a demonstrated commitment to ongoing and continuous improvement of the approved program, particularly in the areas of program-level learning outcomes and the assessment of the student achievement of these learning outcomes. External reviewers should pay particular attention to this aspect of the New Program Proposal. Please see the Guidance on Assessment of Teaching and Learning for detailed information about the assessment and monitoring of student achievement of program-level learning outcomes. Appendix 1 of this document provides further detail on program objectives and program-level learning outcomes.

Internal Response

The QAF requires that programs and Deans/Divisional Heads provide separate responses to the external reviewers’ recommendations (QAF 2.3). This internal response is an important part of the tertiary assessment. The Quality Assurance Framework (Part 1) notes that recommendations from external reviewers must be “reasonably considered and an appropriate plan has been developed to effect program improvement. What is praised is continued and strengthened; what is in need of improvement is in fact improved.”

When evaluating new program submissions, the Appraisal Committee typically expects distinct responses to each of the external reviewers’ recommendations. Units and Deans/Divisional Heads are best able to make concrete, considered responses when the external reviewers’ recommendations are clear, concise, and actionable.

Appendix 1: Program Objectives and Program-level Learning Outcomes

Footnotes

[1] The Quality Council may provide a note regarding an issue(s) to be considered at the time of the program’s launch, or for its first cyclical program review, or for audit.[2] The with report condition implies no lack of quality in the program at this point, importantly, does not hold up the implementation of the new program, and is not subject to public reference on the Quality Council’s website. The requirement for a report is typically the result of a provision or facility not currently in place but considered essential for a successful program and planned for later implementation.[3] Based, in part, on the external reviewers’ assessment of the faculty members’ education, background, competence and expertise as evidenced in their CVs.