As the Fall semester begins, the College of Pharmacy and Health Sciences, its program leaders, faculty, and administrators aim to provide students with the information and resources needed to succeed. On August 31st, 2015, the incoming Fall 2015 didactic cohort of Physician Assistant (PA) students joined Dr. S. William Zito and the Office of Assessment for its annual Assessment Workshop led by Dr. Marc Gillespie.
Prior to the workshop, this cohort of students was invited to participate in a survey regarding their experience and knowledge of assessment. The Pre-Assessment Workshop Survey received a 46% response rate and based on a summary of responses, it was evident students were seeking more information on assessment. Following the workshop, the Post-Assessment Survey was administered and received a 58% response rate.
Some highlights of the survey include that 94% of students are now aware of the assessment process in the college in comparison to 37% before our workshop.
84% of students are now willing to participate in assessment as compared to only 66% of respondents beforehand.
Now, 96% of students either strongly agree or agree that they are involved in improving the culture of assessment in comparison to 28% before the workshop.
A majority of respondents now know that the College of Pharmacy and Health Sciences has an Assessment Committee and an Office of Assessment. Previously, less than 30% were aware of their roles. 89% are now in agreement that the results of assessment initiatives are used effectively.
Results of the Post-Assessment Workshop Survey demonstrate students’ interest and willingness to participate in the College’s assessment process, as well as, the effectiveness of the Assessment Workshop. The Assessment Team encourages all students to share their feedback by electing to participate in any survey invitations they may receive. We look forward to next year’s PA Assessment Workshop and hope to hear from all students throughout the year.
Often, accrediting agencies require that programs complete course evaluations when classes end. As an alternative to the traditional course evaluations completed on paper, the College of Pharmacy & Health Sciences has already implemented the use of online evaluations across a number of its health sciences programs. As the spring 2014 semester comes to a close, we have completed the transition from paper-based evaluations to online evaluations for our Clinical Laboratory Sciences (CLS) program which had yet to changeover.
After meeting with the CLS program director, it was decided to implement online course evaluations to CLS students, so as to allow for more efficient data collection and better organized analytics. To ensure higher rates of student participation, a well-defined plan was discussed with the program director. Class announcements were made by instructors to ensure that students are aware of this opportunity to provide feedback, which can, in turn, impact their current learning experience. E-mails were also sent to encourage participation prior to the closing of each course’s evaluation.
Based on the amount of course evaluations needed to be completed, embracing a system of online course evaluations has its benefits. Online evaluations can be cost-effective, while also allowing the Office of Assessment to focus on managing important analytics from a digital perspective. Furthermore, continuing to introduce forward-thinking ideas of assessment will also contribute to the ongoing efforts of improving the “culture of assessment” in the college.
A heat map is a visual representation of data, utilizing colors to represent certain values. The assessment team decided a few years ago that a heat map would be a good way to take the complex set of PCOA data provided by NABP, and provide a quick visual overview. We found this colorful visualization to be a useful way to compare our own PCOA results against other schools in the nation. You can read more about “information aesthetics” (defined by Lev Manovich) here: Infosthetics.com.
The idea here, is that by giving a visual representation of data, we can engage our stakeholders, while helping them understand precisely what this data means. As assessment professionals, we often find ourselves mired in the same types of data reports, absorbed often in personality-less information. Our hope is, through the idea of ‘showing’ data, rather than just providing it, we can give our faculty and students important information in a more engaging way.
The College administered the PCOA (Pharmacy Curriculum Outcomes Assessment) examination to P1-P4 pharmacy students in January for the third consecutive year. As stated on NABP’s web site, “The PCOA is a comprehensive tool for colleges and schools of pharmacy to use as they assess student performance in the curricula.” Every year, after this examination is administered at SJU, the assessment team generates a heat map to visualize how our students have fared against the national reference group. This heat map takes advantage of the conditional formatting/color scales in Excel, and allows us to see the overall visual difference between % correct scores for SJU and the national reference. Ideally, this heat map should give a visualization of our curriculum. More specifically, the heat map provides the difference between the college and the national number for the mean percent correct scores for each topic and subtopic within the exam. For example, our P1 students achieved a ‘2’ in microbiology this year, meaning that their microbiology mean percent correct score was 2 points above the national number. The conditional formatting tool in excel takes whatever selection you choose, and conditionally formats it based on the existing numbers. For our purposes, we conditionally format each separate section, so that we could look at the ‘heat’ of individual areas. Each blocked group has been conditionally formatted separately, which explains why one zero may be yellow, while another zero is green.
The PCOA heat map is one way for us to look at our PCOA scores, and gives us something to compare against nationally. As ACPE considers whether or not to include the PCOA in the new standards, it is important for us to be prepared, and utilize the PCOA as a validated assessment tool to compare our curriculum to those across the country. I have attached a Sample School Heatmap, drawn from data provided by NABP on their published ‘Sample Score Report’. This should give a good idea of what we are looking at on our own heat map.
The College of Pharmacy and Health Sciences’ Committee on Assessment and Outcomes utilizes a built-in, cyclical assessment process throughout the year. This process allows for each program’s achievement of goals and outcomes to be analysed at some point in the academic year by the committee. Each month when the committee meets, a different program presents their ‘assessment matrix’. This matrix is a functional table, that organizes and describes the program’s goals, outcomes, measures, targets, findings and associated action plans. We have found this process to be useful, and it has turned into an organizational fundamental for our institution. We view this systematic approach to the measurement of goals and outcomes as a type of internal program review. The process itself forces us to continuously look at ourselves, and think about ways we can improve how and what we assess within our programs.
By taking a big project (analysing each programs’ goals and outcomes) and breaking it up into smaller bits, one month at a time, we have given the committee time to look at other things. This allows us to get away from having a rushed large scale review of all programs in a month or two at the end of the academic year. This ongoing programmatic review process has simply become another part of what we do, on a rolling basis, and enhanced the ‘continuous’ part of our continuous quality improvement efforts.
As a follow-up to the prior post, here’s another interesting article on Outcomes Assessment. Acceptance, as we know, is key. This article speaks to that, as well as the specific types of assessment that are needed to begin to see progress.
We recently ran our second iteration of the PCOA exam, and managed to administer the exam very well electronically. Students were asked to bring their laptops with them to the exam with a web-lock browser pre-installed. IT technicians were on hand to handle any issues that could would inevitably arise. While we had several hiccups, such as individual student machine problems, as well as a low University wireless signal, we were able to have all students complete the exam in the allotted time without any major issues.
Next year, we know we’d like to continue to work with our IT department, as they were instrumental in monitoring the network and troubleshooting student machines during the process. We also need to stress that students bring their University-issued laptops with them, as personal laptops had more issues than University-provided machines. IT has also assured us that we can request to have an additional 10 IT laptops on hand for students that are having computer issues. This will speed the entire process up.
It will be interesting to see how the results look as compared to last year’ paper and pencil version of the exam. Going forward, we’d like to continue with the electronic method, keeping in mind the issues we had, and the things we’ve discussed to avoid them in the future.
We’ve completed some analysis on the PCOA exam (Pharmacy Curriculum Outcomes Assessment) that was taken by students in professional years 3-6 in January, 2012. The results proved that, although we have a small sample size, the PCOA exam reflects our curriculum pretty well and will serve as a good tool for us going forward. This is important, as we are always striving to find tools to better the assessment capabilities of our curriculum. Obviously, we need more data, and will really be able to look at correlations more deeply once we have another year or two of information. The exam will be given to another cohort of PharmD students in January, 2013. Hopefully, the data continues to show that the PCOA exam is a good barometer of our curriculum, allowing us to make inferences based on our findings.