2018 IQAP Review of Bachelor of Aeronautical Engineering Programme
Final Assessment Report and Implementation Plan
In accordance with the Royal Military College (RMC) Institutional Quality Assurance Process (IQAP), this final assessment report provides a synthesis of the external evaluation and internal response and assessments of the Bachelor of Aeronautical Engineering programme offered by the Department of Mechanical and Aerospace Engineering.
This report identifies the significant strengths of the programme, together with opportunities for programme improvement and enhancement, and sets out and prioritizes the recommendations that have been selected for implementation.
This report includes an Implementation Plan that identifies who will be responsible for approving the recommendations set out in the Final Assessment Report; who will be responsible for providing any resources entailed by those recommendations; any changes in organization, policy or governance that will be necessary to meet the recommendations and who will be responsible for acting on those recommendations; and, timelines for acting on and monitoring the implementation of those recommendations.
Overview of Programme Review Process:
The IQAP programme review process was conducted in concert with the Canadian Engineering Accreditation Board’s. The Programme Self-Study report, finalized in September 2018, consisted of the CEAB-mandated self-study components plus an IQAP supplement which addressed elements of the IQAP process not covered by the CEAB’s requirements. In combination this included the CEAB graduate attributes and IQAP degree-level expectations for these programmes, an analytical assessment of the programmes, course outlines, programme-related data, survey data from the Office of Quality Assurance and appendices with sample examinations and CVs of faculty members.
The CEAB Programme Visitor, selected by CEAB and approved by the Dean of Engineering (Dr Andy Simoneu, PEng) acted in the role of external reviewer. He was aided in this by the other members of the CEAB visit team who looked at issues of interest across all of RMC’s undergraduate Engineering programmes and whose observations formed part of the CEAB visit report on the programme. The internal reviewer (Dr Fiona Kelly) was selected by the Dean from a list of possible reviewers. The internal reviewer and CEAB team reviewed the self-study documentation and conducted a site visit to RMC on 4–6 November 2018. In the remainder of this document, the CEAB visit team and the internal reviewer are collectively referred to as the External Review Committee (ERC).
At the College level, interviews were conducted by the CEAB Visit Chair, Vice Chairs, and General Visitors. This included meetings with the Commandant, Principal, Vice-Principal (Academic), Vice-Principal (Research) and Dean of Graduate Studies, Dean of Engineering, Dean of Science, Interim Dean of Social Sciences and Humanities, Associate Dean of Engineering (Accreditation and Programme), Chief Librarian, Comptroller, Director of Cadets, Director of the Success Centre, Registrar, Unit General Safety Officer, and the Heads of the Departments of English, Mathematics and Computer Science, and Physics and Space Science.
At the programme level, interviews were conducted jointly by the CEAB programme visitor and the IQAP internal reviewer. This included meetings with the Dean of Engineering, Associate Dean of Engineering (Accreditation and Programme), Department Head, members of the faculty, the Technical Officer and members of the technical staff, the administrative and clerical staff, teaching assistants, and a selection of students from all years of the programme. The visit also included tours of all undergraduate laboratory facilities and representative classroom spaces.
The CEAB sent RMC their Report of the Visiting Team on 24 December 2018 and their Decision Letter on 27 June 2019. These were provided to the IQAP internal reviewer as input to the ERC report, which was finalized by the internal reviewer. The originally-submitted ERC report appears to have been lost in the cyber-attack that RMC suffered in July of 2020. Dr Kelly was able to locate a copy in her personal archives, which was provided to the Dean of Engineering on 27 January 2022.
The self-study documentation, CEAB report, CEAB decision letter, and ERC report together provide a thorough analysis of the current situation in the programme. In the following sections, observations are taken from both the CEAB and ERC reports. Recommendations are taken from the ERC report, which is specific to IQAP, and the CEAB decision letter which includes recommendations relevant to both CEAB and IQAP processes.
The Dean of Engineering and the Head of the Department of Mechanical and Aerospace Engineering, after consultation with faculty and staff of the programme, produced this Final Assessment Report in May of 2022. The Faculty and the Department look upon this as an important opportunity to better understand the programme’s context and identify opportunities for improvement.
Significant Strengths and Weaknesses of the Programme:
The CEAB report and ERC identified a number of strengths of the Bachelor of Computer Engineering programme:
- The labs are well supported by technicians, professors and the department. Teaching assistants do a preliminary outline of the lab, highlighting theory. Technicians assist and watch over students, professors often participate. In particular, the machining, material processing and welding laboratories are exceptional and of tremendous value.
- The Aeronautical program has more hands-on elements than programs at other universities.
- Professor – student engagement is excellent. Some professors make themselves available to students outside of the classroom and are often available after hours to assist students. The students were very happy with the professors, TA’s and technicians.
- Military faculty represent a unique resource for the program. They are able to draw on some hands-on, “real world” experience that is extremely beneficial to these future engineers.
The CEAB report and ERC also identified several areas of concern that influence the status and quality of the programme:
- Course outlines and syllabi should be uniform. There is no uniform approach or template for course outlines and syllabi. Some are very detailed while others say very little. A formal template for course syllabi and outlines would be effective in communicating course content, expectations, and outcomes and their relationship to indicators and CEAB Graduate Attributes.
- New course assignment occurs frequently for term hires and Military Faculty which could impact quality of course content and delivery since there is little consistency. More advanced timing on course assignment and changes could help.
Summary of the CEAB Findings and ERC Recommendations with the Programme’s and Dean’s Responses
The ERC report does not include any recommendations beyond those from CEAB.
The following findings and recommendations are taken from the CEAB decision letter. In some cases recommendations have been reworded for clarity. Suggestions from the CEAB Visit Report are not included since CEAB identifies them as just that – suggestions from an individual visitor – and does not require a response from the institution.
Responses from the Programme and the Dean are provided in italics below each finding or recommendation with a corresponding implementation plan in the next section.
CEAB Findings that apply to all Programmes
The following CEAB Findings, reported in the CEAB decision letter, are common to all Engineering programmes.
CEAB Common Finding 1. Learning outcomes and graduate attribute indicators used interchangeably. There appeared to be some confusion between the differences between the two.
In the tracking system that RMC used for Graduate Attributes (GA) up to the 2018 CEAB visit, each GA was mapped onto multiple “indicators” which were learning outcomes in specific courses, or occasionally appearing in multiple related courses. Comments provided in our 2012 CEAB Accreditation, when CEAB’s GA programme was still in its initial phase, suggested that this met CEAB’s intent. Since then, CEAB has provided further direction that the tracking system should have three levels: the twelve CEAB GAs, each mapped onto multiple “indicators”, and each of those mapped to multiple “learning outcomes” which are specific to courses. Indicators and learning outcomes are determined by the institution and programme. RMC’s response to this finding is described in CEAB Common Finding 3, below.
CEAB Common Finding 2. Insufficient indicators were used. In some cases, there was only a single measurement for an indicator or there was reliance on a single course and/or the indicator. For some graduate attributes not all indicators were used. Often, Introduced and Applied were identified, but no Developed.
This is related to CEAB Common 1. Since each indicator was specific to a single course, or at most a small number of courses, there were very few measurements associated with each indicator. CEAB further expect that each GA be Introduced at some point in the program, then Developed (deepened or enhanced) and then finally Applied. While RMC’s programmes do generally follow this pattern, the structure of RMC’s GA mapping did not make this clear. See the response in CEAB Common 3.
CEAB Common Finding 3. Stakeholder engagement is limited mostly to internal representation. A broader set of external stakeholders including alumni (still in military and ex-military) should be engaged. Also, stakeholder roles in the improvement process is inadequately demonstrated.
CEAB mandates a Continual Improvement (CI) process, to which the GAs are one input. Other inputs are expected to come from both internal and external stakeholders. While the RMC Engineering programmes do receive significant input from external stakeholders, this has been neither systematic nor well documented.
Programme response. The Mechanical and Aerospace Engineering programme has been fully involved in the development of the new GA and CI processes and is committed to implementing them in conjunction with the rest of the Faculty.
Dean’s response. In response to CEAB Common Findings 1, 2 and 3, as well as certain programme-specific findings, the Dean of Engineering convened a committee with representatives from all programmes, chaired by the Associate Dean (Accreditation and Programme). The committee’s mandate was to recommend changes to Engineering programmes’ GA and CI practices and processes, with the aims of addressing CEAB’s concerns, improving the processes’ value to RMC as an institution, and keeping the administrative burden of the processes as lightweight as possible. The committee’s work was delayed by the advent of the pandemic, then further by the cyber-attack that struck RMC’s computer networks in July of 2020. However, the committee’s recommendations are now being provided as a Faculty-wide directive on GA and CI. The intent is that the new GA and CI processes will be in place for the fall semester of 2022.
CEAB Findings that apply to the Aeronautical Engineering Programme
The following CEAB Findings, reported in the CEAB decision letter, are specific to the Aeronautical Engineering Programme.
CEAB Aeronautical Finding 1. There was an insufficient breadth of data collected from year to year to be able to show progression of indicator performance through program and demonstrate remedial action.
Programme response. This is related to CEAB Common Finding 2 and is addressed by the plan described under CEAB Common Finding 3.
Dean’s response. I concur with the Programme response.
Implementation Plan
| Recommendation | Proposed follow-up and resource implications | Responsibility for leading follow-up | Timeline for addressing recommendation |
|---|---|---|---|
| CEAB Common Finding 1. Learning outcomes and graduate attribute indicators used interchangeably. There appeared to be some confusion between the differences between the two. | See CEAB Common Finding 3 | ||
| CEAB Common Finding 2. Insufficient indicators were used. In some cases, there was only a single measurement for an indicator or there was reliance on a single course and/or the indicator. For some graduate attributes not all indicators were used. Often, Introduced and Applied were identified, but no Developed. | See CEAB Common Finding 3 | ||
|
CEAB Common Finding 3. Stakeholder engagement is limited mostly to internal representation. A broader set of external stakeholders including alumni (still in military and ex-military) should be engaged. Also, stakeholder roles in the improvement process is inadequately demonstrated. |
In response to CEAB Common Findings 1, 2 and 3, as well as certain programme-specific findings, the Dean of Engineering convened a committee with representatives from all programmes. The committee’s recommendations are being provided as a Faculty-wide directive on GA and CI. Implementation of the changes will impose significant additional work on all Engineering Departments as well as the Dean’s office, both during the transition and on an ongoing basis. It may be appropriate to re-create the position of Associate Dean (Accreditation and Programme) on an ongoing basis to provide oversight and support. |
Dean of Engineering |
Directive on GA and CI to be released June 2022. New processes to be implemented starting in September 2022. Review of process outcomes and revision of processes in June 2023. |
| CEAB Aeronautical Finding 1. There was an insufficient breadth of data collected from year to year to be able to show progression of indicator performance through program and demonstrate remedial action. | See CEAB Common Finding 3. |
Conclusion
The External Review Committee, consisting of the CEAB visit team and the internal reviewer, provided a thorough and fair review of the Bachelor of Engineering in Aeronautical Engineering programme. Activities are underway to address the ERC report’s recommendations. CEAB-specific recommendations are also being addressed in the context of ongoing CEAB accreditation activities.