In accordance with the Royal Military College of Canada (RMC) Institutional Quality Assurance Process (IQAP), this Final Assessment Report (FAR) provides a synthesis of the external evaluation and internal response and assessments of the graduate program in Defence Studies offered by the Faculty of Social Science and Humanities. This report identifies the significant strengths of the program, together with opportunities for improvement and enhancement, and sets out and prioritizes the recommendations that have been selected for implementation.
This report includes an Implementation Plan that identifies who will be responsible for approving the recommendations set out in the Final Assessment Report; who will be responsible for providing any resources entailed by those recommendations; any changes in organization, policy or governance that will be necessary to meet the recommendations and who will be responsible for acting on those recommendations; and timelines for acting on and monitoring the implementation of those recommendations.
Overview of Program Review Process:
The Program Self-Study Report was completed in 2016. For the program under review - the Master in Defence Studies (MDS) - it contained the degree level expectations for these programs, an analytical assessment of the programs, course outlines, program-related data, some survey data from the Office of Quality Assurance and appendices with sample rubrics and CVs of faculty members.
Two arm’s-length external reviewers (Dr. B.L. Job, University of British Columbia, and Dr. J.W. Yaeger, National Defence University) were selected from a list of possible reviewers and approved by the Deans of Arts and Graduate Studies. An internal reviewer, Dr. M. Hefnawi of RMC’s Electrical and Computer Engineering Department, was also selected for participation on the External Review Committee (ERC). They reviewed the self-study documentation and conducted a site visit to the Canadian Forces College (CFC) on 30 May to 1 June 2017. During the site visit, the ERC met with the CFC Commandant, BGen Kevin Cotton; Vice-Principal, Academic, Dr. Phil Bates; the Vice-Principal, Research and Dean of Graduate Studies, Dr. Pat Heffernan; the Deputy CFC Commandant, Col Nishika Jardine; the CFC Corporate Secretariat, LCdr Joel Kam; the Head of Defence Studies, Dr. Eric Ouellet; the Chair of MDS, Dr. Christopher Spearin; the CFC Heads of Curriculum Departments; the Chief Librarian, Ms Cathy Murphy; and held a roundtable with DS professors. The ERC also had the opportunity to meet with several students currently enrolled in the program. The ERC subsequently produced a report based on the Self-Study and site visit.
The reviewers submitted their report in September, 2017. In their report, they found that the MDS program was consistent with the standards, educational goals and learning objectives/outcomes of the graduate degree offered and consistent with the RMC and CFC missions. While it found the requirements well laid out in course outlines and rubrics, it found, however, that the Graduate Degree Level Expectations (GDLEs) were not developed and mapped to specific Learning Outcomes contained within the various rubrics. The ERC identified strengths/exceptional achievements, as well as areas that presented opportunity to enhance the program.
Significant Strengths and Areas of Concern of the Program:
The ERC identified a number of strengths of the MDS:
- Consistent to mission, tailored to stakeholders;
- Fair admission process consistent with norms for degree sought with opportunity for those denied entry to reapply;
- Curriculum met or exceeded the current state of relevant disciplines in virtually all cases;
- High quality Directed Research Project (DRP) outcomes either leading to publication or informing practice; and
- Good quality core faculty active in research.
The ERC identified a number of areas of concern for the MDS:
- Graduate Degree Level Expectations (GDLEs) were not developed and mapped to specific Learning Outcomes contained within the various rubrics;
- Information management in Quality Assurance (QA), particularly for student instructor assessment and program data collection and dissemination, is inconsistent;
- A requirement to upgrade the CFC learning environment including IT based tools such as the Learning Management Systems (LMS);
- Involvement of non-PhD members in teaching and supervising projects at the graduate level creates challenges for graduate level instruction;
- Over reliance on contract/visiting faculty could jeopardize the effectiveness of the programme;
- Inadequate involvement of permanent faculty and over reliance on contract/visiting faculty for DL variant of program is inappropriate for graduate level education (comments recognized existing decision and date to cease DL variant in advance of visit); and
- Staffing of the library is inadequate to ensure appropriate access to the excellent resources held.
The Program Chair, after consultation with faculty and staff in the program, submitted a response to the Reviewers’ Report in March, 2018. The Dean of Graduate Studies, in conjunction with the Dean of Social Science and Humanities, and in consultation with the Vice-Principal, Academic, and the Deputy Commandant of CFC prepared this Final Assessment Report in January, 2019. Specific recommendations are discussed, and follow-up actions and timelines provided.
Summary of the Reviewers’ Recommendations with Dean’s Responses
The ERC identified a number of areas of concern or issues that require attention. This report will address those recommendations included as “Summary Recommendations”. Minor recommendations identified in the report have been left for consideration by the department and not included in this report. These issues are discussed in the order that they appear in the ERC Report:
- Workload. The ERC concluded that adding additional full-time academic faculty should be a priority and necessary to bring faculty-taught components of the JCSP and particularly the MDS into line with expectations of a professional, masters-level degree program, to be taught by qualified faculty. The JCSP (MDS?) is largely reliant on assigned military staffing, whose training and capacity as instructors is variable. Also, the reliance on sessional instructors should be reduced.
The JCSP is a purely Professional Military Education (PME) product and only subject to review inasmuch as it impacts on the MDS. Directing Staff (DS) are an essential and critical component to the professional nature of the degree and their contribution cannot and should not be assigned to civilian faculty. However, it is important that DS be selected, at least in part, based on their suitability for instruction at the graduate level to ensure quality standards are maintained.
Part of workload stress has been implementation of MDS DL variant without the investment of resources. The decision to cease the DL variant should free up indeterminate faculty for greater contribution to the residential variant of MDS. In addition, CFC has reallocated resources to allow RMC to hire an additional two UTs in the Department of Defence Studies. These changes should more than address the workload concerns for UTs.
- The ERC is in full agreement with the CFC and RMC decisions to suspend the MDS option. Monitoring and quality control in this program, as currently delivered almost entirely by sessional instructor hires, is agreed to be insufficient.
As per prior decision, enrolment in the DL variant of MDS was intended to be ceased in the summer of 2017 and ultimately stopped in the summer of 2018.
- Systematic data on the MDS program is apparently not collected. If collected, it does not appear to be readily available and was not available to the ERC. This includes basic information on the numbers and composition of the MDS student cohort, but extends beyond to data on student evaluation of courses, grade distributions, etc. The RC strongly recommends attention to these issues.
Data on student populations for the MDS are held by the Registrar at RMC and is consistent with all other programmes. It is unclear why this data wasn’t available to the ERC. It is noted that the ERC did not meet with anyone from the Registrar’s staff. Program Representatives as noted by the Departmental reply, are not integral in the data collection process, nor are they provided for programs integral to a single department where admin support comes from that department.
The student evaluations used for MDS courses should be reviewed for relevance for the professional academic nature of the program. Processes for their development and administration will be reviewed in conjunction with CFC to investigate if one instrument is suitable for both PME (JCSP) and program (MDS) requirements.
- Maintenance of the excellence to the CFC IRC should remain a priority. Staffing levels should be addressed.
CFC needs to ensure adequate resources to maintain student access to IRC. RMC to discuss and monitor impacts on QA concerns.
- Updating facilities to state-of-art instructional technological levels needs to be done, if the program is to serve the requirements for CAF personnel trained to adapt and take advantage of innovative approaches and information sources.
CFC learning environment meets all essential needs for graduate level education. RMC encourages CFC to continue to invest in their learning environment including new pedagogical methods and tools. RMC and CFC should investigate optimization of their independent LMSs to consider integration or, at least, harmonization.
- An identified program priority is attention to gender, in curricular terms and in terms of student experience in the program. (Indeed, the positive integration of women into the CAF is a highlighted priority in the recently announced National Defence Review.) The RC received little information on how the JCSP and MDS address these issues. That program administrators are apparently unaware of the number of women in the program was noted. No complaints were raised with the RC by students regarding the program per se. Female students did comment negatively on the CAF’s attitude and “culture” concerning understanding of, and response to, gender related issues. Given the position of the JCSP within the CAF structure and the leadership role expected of its, and the MDS’ graduates, consideration of directly addressing gender issues in the CAF, either through courses or other required participatory workshops and events, should be given priority.
RMC supports all efforts towards addressing gender-related issues. The curriculum changes which will have been introduced through the GBA+ review will address the stated concern.
- The standardized surveys administered by RMCC/IQAP is not appropriate for MDS students and faculty. It should be reviewed.
See response to Recommendation 3 above.
- The resource requirements to make CFC IRC collections and library staff available to those taking distance learning courses.
Cessation of MDS DL greatly reduces the DL demand for IRC resources. However, post-JCSP MDS students remain the program’s and thus DDS’s responsibility. As such, they should be afforded continued access to the CFC IRC collections until completion of their degree.
Given the large number of recommendations above that are of a coordination nature between RMC and CFC, there is benefit to formalizing the expectations of each party in a Service Level Arrangement (SLA).
Implementation Plan
| Recommendation | Proposed Follow-up | Responsibility for Leading Follow-up | Timeline for Addressing Recommendation |
|---|---|---|---|
| 1. Workload. |
|
DGS DSSH, Cmdt CFC, DAcad CFC | In place to inform postings for APS 2020. |
| 2. MDS DL variant. | Cease enrollment to MDS DL variant. | DGS | Complete Summer 2018. |
| 3. Data and Evaluations. |
|
VPA, Registrar, DCmdt CFC | Complete for AY 2019/20. |
| 4. IRC Resources. | Engage CFC to ensure IRC resources are adequate for appropriate student access. | VPA, DGS, Chair MDS, DCmdt CFC. | Ongoing. New staffing by AY 18/19. |
| 5. Gender curriculum concerns. | Complete GBA+ analysis of JCSP/MDS curriculum and delivery. | DGS, DDS Dept Head, Chair MDS. | Completed. Will required ongoing monitoring and adjustment. |
| 6. IRC Resource DL Availability. | Engage CFC to ensure IRC resources are available to all MDS students, residential and DL, until degree completion. | VPA, DGS, Chair MDS, DCmdt CFC | In place ASAP and nlt AY 2019/20. |
| 7. Graduate Degree Level Expectations mapping exercise. | Fully map learning outcomes against GDLEs and specific evaluations. Establish a data collection plan to measure success and review on a periodic basis. | DGS, Chair MDS | In place for AY 2019/20. DGS to review. |
| 8. RMC/CFC coordination. | Collaboratively develop an SLA between RMC and CFC to clarify responsibilities of each institution with regards to MDS delivery. | VPA, DGS, DSSH and DComd and DAcad, CFC. | In place for AY 2019/20. |
Conclusion:
The ERC Report provides positive feedback on the outcomes of the Master of Defence Studies program. It confirms that RMC’s Department of Defence Studies is delivering an academically rigorous program to its students at CFC Toronto and that its standards meet those of similar programs in Ontario, nationally and internationally. However, the ERC also identified some areas requiring improvement, and some risks to the future health of the program. RMC is already taking steps to address the issues raised and will continue to collaborate with CFC to do so.
The Department of Defence Studies is RMC’s contribution to the collaborative development and delivery of the MDS with CFC. Therefore, RMC issues are DDS issues. It is essential that DDS adopt holistic ownership of the challenges towards their students and programme delivery from admissions to degree completion.
The Dean of Graduate Studies, in consultation with Dean of Social Science and Humanities and the Head of the Department of Defence Studies, is responsible for monitoring the Implementation Plan and will engage DCmdt and DAcad CFC as required.