Periodic Appraisal PhD in Population Health The Student Experience Cyclical Review of Graduate Programs The University of Ottawa Brief for the Periodic Appraisal of the PhD in Population Health (student experience Submitted to the Graduate Program Committee Volume I: Program January 2013
Periodic Appraisal PhD in Population Health The Student Experience 1. Program overview Table of Contents 1.1. Brief listing of program(s and brief history 1.2. Method used for the preparation of the cyclical review 1.3. Objectives of program 1.4. Fields 1.5. Review concerns 1.6. Special matters and innovative features 1.7. Bilingualism and language requirements 1.8. Student with disabilities 1.9. Essential requirements 1.10. International collaboration and exchanges 1.11. Professional skills 1.12. Mentoring 1.13. Communications 1.14. Academic services that contribute to academic quality of programs 1.15. Summary and analysis 2. Faculty members 2.1 List of faculty members 2.2 Research funding of core faculty 2.3 Supervision experience 2.4 Current teaching loads 2.5 Commitment of faculty members from other graduate programs and/or from other institutions 2.6 Assessment of teaching and learning 2.7 Quality of graduate supervision/mentoring 3. Physical Resources and financial support 3.1 Library 3.2 Lab/computer 3.3 Space 3.4 Financial support 4. Program 4.1 Intellectual development and the educational experience of the student 4.2 Program regulations 4.3 Progress report 4.4 Language rights 4.5 Mode of delivery 4.6 Part-time studies
Periodic Appraisal PhD in Population Health The Student Experience 4.7 Graduate courses 4.8 Collateral and supporting departments 4.9 Governance structure 5. Outcomes and quality indicators 5.1 Quality enhancement learning outcomes 5.2 Enrolment and graduations 5.3 Initial employment 5.4 Publications 5.5 Projected enrolments 6. Degree level expectations and expected learning outcome and Definitions 7. Distance delivery 8. Appendices: Appendix A PhD Student Survey Appendix B Sondage des étudiants au doctorat Appendix C Quantitative Data Combined English and French Results Appendix D Email to students
Periodic Appraisal PhD in Population Health The Student Experience 1. PROGRAM OVERVIEW 1.1. Brief listing of program(s and brief history 1.2. Method used for the preparation of the cyclical review. Briefly describe the full process In June 2012 a working group (Student Cyclical Review Working Group; SCRWG of three students (Tanya Hewitt, James Crispo, and Flora Nassrallah from the Population Health Graduate Students Association was assembled to inform the PhD Population Health Programme and the Graduate Diploma in Population Health Risk Assessment and Management Programme cyclical reviews. Survey Design & Delivery Survey Monkey was chosen to prepare and disseminate the survey. This platform guaranteed respondent anonymity, which was identified as an essential attribute of the survey. Survey questions were posed using three distinct formats - multiple choice, Likert scale, and open text box. The first question addressed current student status, and a response was mandatory. All subsequent questions were optional. The survey was available in both official languages (English & French. An electronic link to the survey was first delivered to the relevant PHGSA PhD Programme contacts (75 current & former students via email by the SCRWG on September 5 th, 2012. Note that for the purposes of the student experience, all alumni (whose responses cannot be identified, by design will be referred to as students. Their alumni experience was not interrogated only their previous experience as a student. The SCRWG decided to keep the survey active for data collection for nine weeks, until November 9 th, 2012. In order to increase reach and survey responses, Roseline Savage, Program Coordinator, sent out four email reminders (September 26 th, October 10 th, October 24 th, and November 7 th, 2012 on behalf of the SCRWG to her relevant PhD Programme contacts (101 current & former students. Additionally, in effort to further increase surveys responses, the SCRWG gave both diploma and PhD respondents the opportunity to participate in a draw for two $50 Chapters gift cards and two $25 Starbucks cards. Data Analysis Following data collection, the SCRWG manually examined each survey and discarded those that were deemed incomplete. The SCRWG leveraged both qualitative and quantitative methods to analyze and interpret survey responses. Simply put, emergent thematic coding was used to identify recurring themes expressed within written responses, while percent responses were computed for each multiple choice and Likert scale question. Since we are unsure of the total reach of this survey, a rough approximation of the response rate was computed to be 66 respondents / 101 PhD contacts = 65%. All the tabulated data are found in Appendix C.
Periodic Appraisal PhD in Population Health The Student Experience 1.3. Objectives of program 60%# Objec&vesoftheprogrammethroughtheadmissionprocess 50%# 40%# 30%# 20%# 10%# 0%# Strongly## Disagree# Neither#agree#nor# # Agree# Strongly#agree# The#programme#requirements#meet#the#needs#in#order#to#be#successful#in#the# programme.# The#objecEves#of#the#programme#(e.g.#intellectual,#technical#skills,#etc.#were#clear#at# admission.# Some expressed that this programme was a compromise for another programme (most notably epidemiology.
Periodic Appraisal PhD in Population Health The Student Experience 1.4. Fields Survey responses show that 59% of students found information on the advertised programme fields to be clear when applying to the programme. Further, 48% (32 students of students stated that their application to the programme was influenced by one or more fields. 60%# 50%# 40%# 30%# 20%# 10%# Forthe48%ofstudentswhoseapplica&onwasinfluencedbyoneor morefields 0%# Health#determinants# Global#and#local#health# inequiees# Health#intervenEons# and#policies# Health#risk#and# resiliency# Of the students who commented on fields, some said that the fields had no influence on their application. Many noted that the fields peaked their interest in the programme (but not any specific one, while others specifically identified global and local health inequities, risk and resiliency, and health determinants as influencial. Alumni of the graduate certificate noted the expected continuation of their previous studies in the PhD progamme.
Periodic Appraisal PhD in Population Health The Student Experience 45%# 40%# 35%# 30%# 25%# 20%# 15%# 10%# 5%# Thiscoursepromotedthefouradver&sedfields(healthdeterminants, globalandlocalhealthinequi&es,healthinterven&onsandpolicies, healthriskandresiliencyequally 0%# Strongly## Disagree# Neither#agree#nor# # Agree# Strongly#agree# POP8910#SCIENTIFIC#PARADIGMS#IN#POPULATION#HEALTH# POP8920#INVESTIGATIVE#METHODS#IN#POPULATION#HEALTH# POP8930#POPULATION#HEALTH#INTERVENTIONS# 1.5. Review concerns if any Nothing to contribute
Periodic Appraisal PhD in Population Health The Student Experience 1.6. Special matters and innovative features 60%# Thefollowingaspectswereinfluen&alinmydecisiontoapplytothe Popula&onHealthdoctoralprogramme 50%# 40%# 30%# 20%# 10%# 0%# Strongly## Disagree# Neither#agree#nor# # Agree# Strongly#agree# Interdisciplinary#studies# Unique#in#Canada# Students mentioned the uniqueness of the Population Health programme as a prime reason for their application, as well as a bilingual programme, and the advertised interdisciplinary nature of the studies.
Periodic Appraisal PhD in Population Health The Student Experience 1.7. Bilingualism and language requirements Thiscoursesupportedbilingualdiscussion 40%# 35%# 30%# 25%# 20%# 15%# 10%# 5%# 0%# Strongly## Disagree# Neither#agree#nor# # Agree# Strongly#agree# POP8910#SCIENTIFIC#PARADIGMS#IN#POPULATION#HEALTH# POP8920#INVESTIGATIVE#METHODS#IN#POPULATION#HEALTH# POP8930#POPULATION#HEALTH#INTERVENTIONS# Some noted that while a bilingual programme served to attract their application, it simultaneously detracted others from applying. The language requirement seems to have been complied with for some students at admission (they needed to prove their competency, but not for all students. Once in the programme, many students remarked that their experience was not bilingual, nor were their classmates most notably a lack of a passive understanding of French. Others believed that the programme did not support the unilingual students enough once admitted. A few students commented that their experience reflected the advertised bilingualism.
Periodic Appraisal PhD in Population Health The Student Experience 1.8. Student with disabilities 70%# 60%# 50%# 40%# 30%# 20%# 10%# Theprogrammetakesdisabili&esandotherextenua&ng circumstancesintoconsidera&onthroughouttheadmissionprocess 0%# Strongly## Disagree# Neither#agree#nor# # Agree# Strongly#agree# Out of the 66 possible respondents, 37 individuals chose to answer this question.
Periodic Appraisal PhD in Population Health The Student Experience 1.9. Essential requirements 60%# Therewassufficientexplana&onofthefollowingitemsinthe applica&onprocess 50%# 40%# 30%# 20%# 10%# 0%# Strongly## Disagree# Neither#agree#nor# # Agree# Strongly#agree# Minimum#grade#requirements# Previous#degree#requirements##(i.e.conferred#Master's# degree# Language#proficiency# Required#knowledge#(Epi#&#Biostats# Proposed#research# Students commented on the dubious relevance of the week-long epidemiology and biostatistics courses, given that their content was not revisited, and that the knowledge assessment form did not mention an appreciation of social justice, which would have been helpful. Conferred master s degrees were not enforced as an essential admission requirement, nor was bilingualism. A description of the proposed research in the admissions package (the expectations for which could be clarified was seemingly ignored, according to student s comments. Transparency in the admissions process was stressed in comments, yet seemingly absent.
Periodic Appraisal PhD in Population Health The Student Experience 1.10. International collaboration and exchanges Some noted the possibilities for international collaboration through this programme, specifically through supervisors, although distance learning and overseas practicums could be enhanced. 1.11. Professional skills Students mentioned feeling prepared for the real world, and having a strong emphasis on equity and social justice throughout the programme. 1.12. Mentoring Students noted that their supervisors may or may not have been familiar with the programme requirements, creating difficulties for some. Some students mentioned great strengths of specific course professors (but weaknesses were also noted. However, the most significant mentoring in the programme seems to come from interacting with other students in the programme, especially in preparation for the comprehensive exam and in the face of significant challenges. Furthermore, the programme does not facilitate this interaction beyond the courses, particularly between cohorts, and students need to search out connections on their own (despite being in the same programme.
Periodic Appraisal PhD in Population Health The Student Experience 1.13. Communications Howdidyouhear/findoutabouttheprogrammeinPopula&onHealth? 50%# 45%# 40%# 35%# 30%# 25%# 20%# 15%# 10%# 5%# 0%# University#of#O`awa# Recommended#by#a# Recommended#by#a# Website# supervisor#/#mentor# student#/#colleague# I#don't#remember# Other# Students expressed that the programme is well described on the website, although others noted more clarification on admission requirements and the degree process would be helpful. Students reported knowing of the programme through word-of-mouth, a recommendation, following completion of the graduate certificate in risk assessment and management in population health, the stature of a specific researcher, and being in Ottawa.
Periodic Appraisal PhD in Population Health The Student Experience 60%# Theexpecta&ons(orlackthereofofthefollowinglearningac&vi&eswere clearatadmission 50%# 40%# 30%# 20%# 10%# 0%# Strongly## Disagree# Neither#agree#nor# # Agree# Strongly#agree# Required#coursework# ElecEve#coursework# Comprehensive#exam# Thesis#proposal# Thesis#defence# Seminars# Work/study#opEons# Students reported that there are two main ways in which information gets communicated via the University of Ottawa website or via front line administration. Students had difficulty knowing the most up-do-date version of any given policy regarding the programme (e.g. thesis proposal different versions are in e-mails, and there seems to be no reliable central repository for these essential documents. Some students did not find the University of Ottawa website helpful, nor front line administrators (constant referral to someone else to answer questions. An exception to this is the Program Coordinator for the PhD Population Health programme (and other programmes many students commented on how helpful the Program Coordinator was in their specific case, available and willing to help students navigate the programmes and is the main source of communication from the programme to students. They noted that there is little two-way communication; students feel that their concerns are rarely heard let alone acted upon. 1.14. Academic services that contribute to academic quality of programme Nothing to contribute
Periodic Appraisal PhD in Population Health The Student Experience 1.15. Summary and analysis 45%# 40%# 35%# 30%# 25%# 20%# 15%# 10%# 5%# Pleaseanswerthesefinaloverallques&onsonyourexperienceasa graduatestudentinthisprogramme 0%# Strongly## Disagree# Neither#agree#or# # Agree# Strongly#Agree# If#I#were#to#start#my#graduate/#professional#career#again,#I#would#select#this#same#university# If#I#were#to#start#my#graduate/#professional#career#again,#I#would#select#the#same#programme# I#would#recommend#this#university#to#someone#considering#my#programme# The students overwhelmingly noted that the programme s interdisciplinary nature both attracted them to the programme, and was the most beneficial aspect of their studies. Some noted that the cohorts were very interdisciplinary (different backgrounds, life experiences, cultures, and perhaps this is key to the admissions process. The core course assignments were interdisciplinary and guest speakers were a highlight for many, as they helped reinforce the transdisciplinary approach and helped make the learning applicable to many fields. Some students noted some core courses are not so oriented, and that many theses are not interdisciplinary. Other strengths included: the Program Coordinator, the core course discussions and readings, the quality of the teaching professors, the practical exercises in the courses, and the experience of the comprehensive exam. Flexibility in electives was also mentioned as an appreciated aspect of the programme. Many noted that they thoroughly enjoyed their experience and highly recommend the Population Health programme to others.
Periodic Appraisal PhD in Population Health The Student Experience In the vein of continuous improvement, students were asked for specific suggestions on potential changes to improve the quality of the programme. Specific Suggestions: Make information more available on: Admission rates of past years The social justice aspect of the programme The expressed disconnection between the graduate certificate and the PhD degree Admission selection criteria Admissions process Feedback / information from previous graduates (website course sequencing and progression of degree Comprehensive exam process Comprehensive exam subjects Change admissions process to: Be interview based Select a cohort that will work well together Enable potential students to meet others in the programme Test for bilingualism (if this is what is desired Have the knowledge assessment form be more relevant (social justice, human rights, social sciences instead of epi and biostats Establish and/or enforce admissions requirements on: Master s level thesis completion Having one (not two supervisors Online applications Bilingual students Describing proposed research (and do not accept students whose research is not population health Establish and/or change pre-courses/pre-programme experiences to cover: Social sciences Upgrade course options Professors interested in students describing their research Moral philosophy A week long summer institute introduction The programme should: Establish strong visionary leadership Ensure adequate administration help Ensure up to date documents are on the programme website Provide students with an admission scholarship
Periodic Appraisal PhD in Population Health The Student Experience Revisit the fields, and delete/align accordingly (notably risk/resiliency Reduce the number of required credits Reduce red tape regarding elective courses Not recommend elective courses that conflict with the core courses Offer more international distance learning Offer TA ships Offer guidance on supervisor selection / resolving supervisor-student conflict Offer an environment for the 5 faculty interdisciplinary exchange (especially Law Ensure more interaction between the Institute of Population Health (IPH researchers and the PhD students Establish semi-annual research update meetings for all students and IPH researchers, or other opportunities for cohorts to collaborate Establish a requirement to present an IPH seminar Establish a requirement to attend IPH seminars Require regular thesis committee meetings Allow students space, photocopying priviledges Give the substance of the capstone meeting (comprehensive preparation when sending the invitation. Give automated reminders of deadlines to students in their last year Offer thesis defense coaches Shorten the time between thesis submission and defense Core courses professors should: Be comfortable with interdisciplinary teaching Have access to the accepted students applications Cover the programme fields equally Cover policy Cover environmental determinants of health Ensure discussion and assignments that challenge the status quo Ensure different points of view are presented Cover quantitative methods and qualitative methods in two courses Align with the programme s endorsement of mixed methods Cover data analysis methods Not dwell on one topic for too long Cover evaluation of complex interventions Ensure a manageable reading list Prepare students for the comprehensive exam Ensure bilingual course delivery The comprehensive exam should: Follow the written guidelines Involve professors who taught the courses Give feedback Be available more frequently then annually
Periodic Appraisal PhD in Population Health The Student Experience Evaluate students through a portfolio of student activities Query on research design Be a discussion or quiz about the literature Not cover subjects that could negatively impact students Not cause undue stress to students (is comprehensive exam necessary? Overall, students found the programme very beneficial although this view was not universal. From this point forward, the student experience is in only selected sections of the report. 2. Faculty members 2.1 List of faculty members 2.2 Research funding of core faculty 2.3 Supervision experience 2.4 Current teaching loads 2.5 Commitment of faculty members from other graduate programs and/or from other institutions 2.6 Assessment of teaching and learning Many students commented on the high quality of discussions in their classes, where students were encouraged to share their diverse opinions, and the expert and passionate instructors enriching their learning. Students who did not share the above commented on the lack of feedback, seemingly closed minded instructors, and lack of interdisciplinary teaching skills. 2.7 Quality of graduate supervision/mentoring Many students noted that they were drawn to the programme based on the reputation of their supervisor, and that their collaborative efforts are due to their supervisor s connections, not due to the programme. Others noted that their supervisors are outside population health, and consequently knew very little of the programme and its rules. Students felt unsupported when issues arose with supervisors. 3. Physical Resources and financial support 3.1 Library 3.2 Lab/computer 3.3 Space 3.4 Financial support Students mentioned unclear guidance concerning funding opportunities, and that the notification / information sessions for funding opportunities were scheduled too close to the funding deadlines.
Periodic Appraisal PhD in Population Health The Student Experience 4. Program 4.1 Intellectual development and the educational experience of the student Students commented on the lack of integration with the Institute of Population Health, and the absence of guidance in supervisor selection or support when barriers are encountered. Students commented on the existence of population health seminars, but not being required (or even encouraged to attend, which may support student interactions once courses and comprehensives are finished. 4.2 Program regulations 4.3 Progress report 4.4 Language rights Commentary offered in section 1.7 4.5 Mode of delivery 4.6 Part-time studies 4.7 Graduate courses
Periodic Appraisal PhD in Population Health The Student Experience Students noted how courses helped expand their vision of health, encouraging some students to change the direction of their intended research. Flexibility in elective choice was noted and appreciated. In POP8910, students commented on the strength of the paradigm focus (or lack thereof; in POP8920, the course being a review of methods (not necessarily mixed methods and the utility of writing the thesis proposal and receiving feedback; in POP8930, the amount of reading, the quality (or lack thereof of feedback, and the well-rounded approach. Comments concerning the strength (or weakness of professors and/or guest speakers, the inconsistent support of bilingual delivery, and the discussions (most very beneficial, some very constrained were made for all courses.
Periodic Appraisal PhD in Population Health The Student Experience Many students noted that the core courses did not prepare them for the comprehensive exam the study sessions students formed within their cohort and independent work was the best study preparation. Students overall noted that the comprehensive exam as a worthwhile experience, albeit unrelated to their further studies and interests. Students commented on the utility of preparing their thesis proposals in a course. Some students mentioned that their comprehensive exam jury professors were not experts in population health, making their experiences very challenging. Most students (52% stated that their research direction significantly changed from what they had in mind during the application process. While the majority of students cited typical reasons for changing their initial project (changing interest, supervisor issues, etc., a few noted that they had to change their preferred research to align with population health.
Periodic Appraisal PhD in Population Health The Student Experience 4.8 Collateral and supporting departments 4.9 Governance structure A few students noted the difficulties of being supported by 5 faculties, and the apparent lack of vision for interdisciplinary studies. The accolades for the Program Coordinator were highlighted, although difficulty in finding information was also noted. The lack of programme integration with the IPH was noted as a weakness by many students. 5. Outcomes and quality indicators 5.1 Quality enhancement learning outcomes 5.2 Enrolment and graduations 5.3 Initial employment 5.4 Publications 5.5 Projected enrolments 6. Degree level expectations and expected learning outcome and Definitions 7. Distance delivery 8. Appendices
Appendix A PhD Student Survey
1. INTRODUCTION As a student in the Doctoral programme in Population Health, your participation in the current cyclical review is crucial to shed light on the student experience. The intended focus of this review is the student without your participation, the process will not be as beneficial to yourselves or future students. PLEASE NOTE: As the Health Services and Policy Research (Graduate Diploma programme is a joint programme, and the Population Health Graduate Students' Association (PHGSA did not have the expertise to set questions for such a programme, this diploma is not evaluated here. All answers are COMPLETELY ANONYMOUS we thus ask you to please answer the questions honestly, and only once per person. Taken from the Institutional Quality Assurance Process document (p. 34, "In essence, the self-evaluation report is much more than a description of the aspects under evaluation. It must be the product of a thorough examination into the programme s strengths and weaknesses; where applicable, it must suggest how the programme can be improved and what its future directions are." This survey should take you 15-20 mins, depending on how much commentary you volunteer. This survey has 7 sections (and 7 corresponding pages : Introduction General information Admission process Required courses Comprehensive exam Thesis Overall impressions In order to preserve anonymity, survey respondents must complete the survey in one session. We greatly appreciate your feedback and your time taken to complete this survey. Responses to questions (aside from #1 are not mandatory. You may wish to participate in the draw for one of four prizes (one of two $25 STARBUCKS gift cards or one of two $50 CHAPTERS gift cards. Following completion of this survey and in order to ensure its anonymity, you will be asked to copy and paste a url link into your browser. This will redirect you to another website where you can enter your contact information for the draw.
2. GENERAL INFORMATION * 1. What is your current status in the programme? Current student Former student graduate Former student non-graduate Prefer not to say
3. ADMISSION PROCESS 2. How did you hear/find out about the programme in Population Health? University of Ottawa Website Recommended by a supervisor / mentor Recommended by a student / colleague I don't remember Other (please specify - 10 line maximum 3. The Population Health doctoral programme advertises four fields (health determinants, global and local health inequities, health interventions and policies, and health risk and resiliency. Was this information clear to you when applying to the programme? Yes No Please offer any helpful commentary - 10 line maximum
4. Was your application influenced by these four fields? Yes No Please offer any helpful commentary - 10 line maximum 5. If yes, which one(s? Health determinants Global and local health inequities Health interventions and policies Health risk and resiliency Please offer any helpful commentary - 10 line maximum
6. The following aspects were influential in my decision to apply to the Population Health doctoral programme. Strongly Disagree Neither agree nor Agree Strongly agree N/A Interdisciplinary studies Unique in Canada Please offer any helpful commentary - 10 line maximum 7. The expectations (or lack thereof of the following learning activities were clear at admission. Strongly Disagree Neither agree nor Agree Strongly agree N/A Required coursework Elective coursework Comprehensive exam Thesis proposal Thesis defence Seminars Work/study options Please offer any helpful commentary - 10 line maximum
8. There was sufficient explanation of the following items in the application process. Minimum grade requirements Previous degree requirements (i.e.conferred Master's degree Strongly Disagree Neither agree nor Agree Strongly agree N/A Language proficiency Required knowledge (Epi & Biostats Proposed research Please offer any helpful commentary - 10 line maximum
9. Please rate the following statements regarding the admissions process. The programme requirements meet the needs in order to be successful in the programme. The objectives of the programme (e.g. intellectual, technical skills, etc. were clear at admission. The admission requirements to the programme are well defined on the website. The programme takes disabilities and other extenuating circumstances into consideration throughout the admission process. I described myself and my research intentions in the application. I have an appropriate academic background for this programme. The selection criteria are transparent. I was satisfied with the application process. Strongly Disagree Neither agree nor Agree Strongly agree N/A Please offer any helpful commentary - 10 line maximum 10. Please offer any experiences and/or opinions you believe could help strengthen the admission process - virtually unlimited text box.
4. REQUIRED COURSES Please rate the following aspects of each required course. NOTE: All written information for course evaluations is given to the course instructor(s only. The information sought here is to enable the entire programme to benefit from the student's experience. 11. POP8910 SCIENTIFIC PARADIGMS IN POPULATION HEALTH Strongly Disagree Neither agree nor Agree Strongly agree N/A This course encouraged a variety of viewpoints This course had interesting guest speakers This course supported the objectives of the programme My opinions were welcomed in this course This course supported bilingual discussion Written feedback was constructive This course was well taught This course provided me with depth and breadth of knowledge This course was useful to me in my academic pursuits This course promoted the four advertised fields (health determinants, global and local health inequities, health interventions and policies, health risk and resiliency equally This course was useful for the comprehensive exam This course should remain in the programme Please offer any other helpful commentary with respect to this course - 10 line maximum
12. POP8920 INVESTIGATIVE METHODS IN POPULATION HEALTH Strongly Disagree Neither agree nor Agree Strongly agree N/A This course encouraged a variety of viewpoints This course had interesting guest speakers This course supported the objectives of the programme My opinions were welcomed in this course This course supported bilingual discussion Written feedback was constructive This course was well taught This course provided me with depth and breadth of knowledge This course was useful to me in my academic pursuits This course promoted the four advertised fields (health determinants, global and local health inequities, health interventions and policies, health risk and resiliency equally This course was useful for the comprehensive exam This course should remain in the programme Please offer any other helpful commentary with respect to this course - 10 line maximum