Mobile Survey Lab

The College recently purchased several Nexus 7 tablets for the Office of Assessment to use for survey and assessment purposes in a newly created Mobile Survey Lab. We rolled out the tablets this week for two separate surveys: a satisfaction survey for the ‘Finals Grab & Go Breakfast’ and a survey on the utilization of professional development funds of graduating seniors, during cap and gown distribution. The tablets are being set up to loop to the start of a new survey once a student completes one, allowing students to take the survey on a rolling basis. During the breakfast, students stopped at our Mobile Survey Lab table as they exited, to complete the satisfaction questionnaire:

During the funding survey administered at cap and gown distribution, students passed the tablets around as they waited to be called up. This instantaneous survey method allowed for real-time, instant results, and is already proving to be a useful tool for increasing our survey response rates greatly. We were able to achieve an 83% response rate on the funding survey, as 276/332 graduating seniors completed the survey on tablets in person. We are looking forward to using these tablets in new and innovative ways for surveys and other assessment related tasks. This is just another way that assessment is allowing us to push into new areas and affect positive change. Look for us at the ‘Finals Grab & Go Breakfast’ for the rest of finals week, as we continue to collect data, and give our students an opportunity to provide valuable feedback.

Health Sciences Assessment: Course Evaluations

Often, accrediting agencies require that programs complete course evaluations when classes end. As an alternative to the traditional course evaluations completed on paper, the College of Pharmacy & Health Sciences has already implemented the use of online evaluations across a number of its health sciences programs. As the spring 2014 semester comes to a close, we have completed the transition from paper-based evaluations to online evaluations for our Clinical Laboratory Sciences (CLS) program which had yet to changeover. 

After meeting with the CLS program director, it was decided to implement online course evaluations to CLS students, so as to allow for more efficient data collection and better organized analytics. To ensure higher rates of student participation, a well-defined plan was discussed with the program director. Class announcements were made by instructors to ensure that students are aware of this opportunity to provide feedback, which can, in turn, impact their current learning experience. E-mails were also sent to encourage participation prior to the closing of each course’s evaluation.

Based on the amount of course evaluations needed to be completed, embracing a system of online course evaluations has its benefits. Online evaluations can be cost-effective, while also allowing the Office of Assessment to focus on managing important analytics from a digital perspective. Furthermore, continuing to introduce forward-thinking ideas of assessment will also contribute to the ongoing efforts of improving the “culture of assessment” in the college. 

2014 Exit Interviews

The Office of Assessment recently conducted PharmD exit interviews on April 25 and 28 with the graduating Pharmacy class.  This process requires students to meet in small groups with a faculty member and discuss their experiences in the program.  Students are asked to provide three time slots that are most convenient and are scheduled on a first come, first served basis. A professor provided the Assessment Office with two sections of a scheduled class that the students were asked to attend for the interviews. The students discussed five questions that are related to the three areas of the program (Laboratory, Experiential and Didactic).  The comments and recommendations that were provided are currently being compiled. A report that includes these recommendations will be presented to the Committee on Assessment and Outcomes during the Fall semester.  The participation rate was very high this year with 212 students taking part in this process.   A response rate of 99% is the highest that the Office of Assessment has had in several years.  As part of the exit interview process students also complete the AACP survey, which is used as another assessment tool to gauge student experiences. Exit interviews continue to be an important means to include students in the assessment process. 

2014 PCOA Heat Map

A heat map is a visual representation of data, utilizing colors to represent certain values. The assessment team decided a few years ago that a heat map would be a good way to take the complex set of PCOA data provided by NABP, and provide a quick visual overview. We found this colorful visualization to be a useful way to compare our own PCOA results against other schools in the nation. You can read more about “information aesthetics” (defined by Lev Manovich) here: Infosthetics.com.

The idea here, is that by giving a visual representation of data, we can engage our stakeholders, while helping them understand precisely what this data means. As assessment professionals, we often find ourselves mired in the same types of data reports, absorbed often in personality-less information. Our hope is, through the idea of ‘showing’ data, rather than just providing it, we can give our faculty and students important information in a more engaging way.

The College administered the PCOA (Pharmacy Curriculum Outcomes Assessment) examination to P1-P4 pharmacy students in January for the third consecutive year. As stated on NABP’s web site, “The PCOA is a comprehensive tool for colleges and schools of pharmacy to use as they assess student performance in the curricula.” Every year, after this examination is administered at SJU, the assessment team generates a heat map to visualize how our students have fared against the national reference group. This heat map takes advantage of the conditional formatting/color scales in Excel, and allows us to see the overall visual difference between % correct scores for SJU and the national reference. Ideally, this heat map should give a visualization of our curriculum. More specifically, the heat map provides the difference between the college and the national number for the mean percent correct scores for each topic and subtopic within the exam. For example, our P1 students achieved a ‘2’ in microbiology this year, meaning that their microbiology mean percent correct score was 2 points above the national number. The conditional formatting tool in excel takes whatever selection you choose, and conditionally formats it based on the existing numbers. For our purposes, we conditionally format each separate section, so that we could look at the ‘heat’ of individual areas. Each blocked group has been conditionally formatted separately, which explains why one zero may be yellow, while another zero is green.

The PCOA heat map is one way for us to look at our PCOA scores, and gives us something to compare against nationally. As ACPE considers whether or not to include the PCOA in the new standards, it is important for us to be prepared, and utilize the PCOA as a validated assessment tool to compare our curriculum to those across the country. I have attached a Sample School Heatmap, drawn from data provided by NABP on their published ‘Sample Score Report’. This should give a good idea of what we are looking at on our own heat map.