Category Archives: Computational Tools

Computational tools that get us to the business of assessment

The Federal College Scorecard Site Is Up!

Screen Shot 2015-09-14 at 9.47.50 AMThough it has been sometime in the making the Federal College Scorecard site is up and running.

The site uses a combination of Federal data sources including loan and tax return data. The upside of this is the documentation that goes with the site here, contains a wealth of information on how the analysis was done, what data sources were used, and instructions how to tap into (or download) the data using a pretty straightforward API.

There is much discussion concerning the pros and cons of using the data this way. Each position is worth a read. You can read a sample of the concerns from the New York Times, Wall Street Journal, and the Chronicle of Higher Education. The issues raised include fears that this represents another step in the view that Higher Ed Institutions can be viewed as and shopped for. While there is lots of data present here, there is also the underlying worry that the data is incomplete, or provides a skewed view of the institutions.

As with any new data source it is worth a look, especially with the open nature and full descriptions of the data used.

Site – College Scorecard

SJU Information – St. John’s University

Screen Shot 2015-09-14 at 9.48.25 AMThere are also a number of nice US Department of Education Blog posts available on their HomeRoom Blog. For College Scorecard specific posts, have a look here.

This type of data is definitely the direction that the new knowledge based society is taking us in. It provides a snapshot of how students fair after school and it deeply lists the data sources that were used. Incomplete or not, there is only going to be more data like this released and used.

1 Comment

Filed under Computational Tools

Developing an Examsoft User Manual

“This seems like a lot of work.”

“What do I have to do?”

“I’m not comfortable with computer-based testing.”

These are all valid concerns raised by new Examsoft users and the types of concerns which need to be considered when rolling out a new initiative. As computer-based testing was recently introduced to the College of Pharmacy & Health Sciences, the Office of Assessment has been named its Key Examsoft Administrators. A great deal of information was compiled during our recent pilot phase, which ultimately led to the creation of an instructional user manual tailored to the needs of our specific programs.

Information was collected by tracking the issues experienced during exams’ administrations, disseminating student surveys, & collaborating with faculty.  Observing new users as they were introduced to Examsoft during training sessions and taking note of their concerns alerted us of elements which are crucial to helping new users understand Examsoft. All this information accumulated into a wealth of knowledge we now held and the next logical step was to share it with fellow users.

The manual provides a complete look at delivering computer-based assessments. It is intended to be the first resource for new users to utilize when building an exam. It promotes a ‘help yourself’ approach while still having other lines of support in place. The Key Administrators are available to any new or current users and the resources already offered by Examsoft are still actively promoted. In designing the manual, we opted to provide certain details, so users can better visualize how the software’s many functions are connected, as well as, insight into an exam taker’s experience.

As specific plans for computer-based testing in courses of upcoming semesters are finalized, roles & responsibilities within each course will be established.  The manual will be distributed to faculty users whose courses are ultimately authorized to use the software. We will continue to monitor all issues experienced by students and users, so we may continue evolving our computer-based testing practices to best track learning outcomes and student success.

Leave a comment

Filed under Computational Tools, E-learning, Education, Methods

Computer-Based Testing in the College of Pharmacy & Health Sciences

The implementation of computer-based testing is on the rise in the education sector as institutions continue to integrate technology with pedagogical approaches when delivering the curriculum. The demand for computer-based testing aligns with the institution’s continued efforts to seek greater understandings of learning outcomes achievements and student performances. The College of Pharmacy and Health Sciences implemented a trial period to test the functionality of Examsoft, a program which “combines assessment creation, administration, scoring and analysis into a single, platform that provides teachers with direct evidence of student learning and faculty with greater insight for metric-based decision-making.”

There are many advantages to implementing computer-based testing. Students receive prompt feedback on their exam performance and are given access to electronic exam reviews. Students receive detailed reports, which show their strengths and weakness following each exam, allowing for self-directed learning. Faculty are able to create, deliver, and score assessments much more efficiently. The reports generated by Examsoft provide analyses on exam taker achievement of learning goals providing powerful data-driven resources to assist in student evaluation, accreditation and program assessment.

The article Computer-Assisted Assessment: Impact on Higher Education Institutions (Bull, 1999), outlines the many factors surrounding computer-based testing. A hurdle to overcome will include the cultural shift for some stakeholders. Students are accustomed to taking pen and paper exams, so there may be some apprehension during this transition. In the attempt to counteract any hesitation, we’ve provided students with opportunities to sit through mock exams and offer informative resources regarding the new platform. Faculty organize the design, implementation, and maintenance of computer-based testing protocols assisted by the Assessment Office staff with each step. These organizational concerns are alleviated by properly training all users, collaborating during these initial stages of development, and establishing a clear picture of all responsibilities and services following the trial period.

As the trial period comes to a close, the organizational structure of this platform will need to be finalized. The experiences of all stakeholders will be taken into consideration, to get a thorough understanding of the trial’s successes and any difficulties we need to address. Nevertheless, the advantages of implementing computer-based testing outweigh the perceived difficulties through proper collaboration and significant thought to ensure a strong protocol is put into practice.

1 Comment

Filed under Computational Tools, Methods

Dashboards

The Office of Assessment recently attended the 2014 AACP Annual Meeting in Grapevine, Texas. There were many interesting sessions related to assessment and its place in pharmacy education. One of the sessions focused on the role of dashboards in organizing data and making it visually appealing. Tableau Desktop was one of the dashboard interfaces discussed at the meeting. This software seems to be an effective means of understanding data and identifying trends especially with large data sets. The raw data from surveys or research projects must be input into Excel before analysis can take place. The Tableau Desktop dashboard displays several views of the data at once. Data can be filtered to highlight specific information depending on the task at hand. Fields can be combined into groups and subsets of data can be separated for a variety of purposes. Statistical analyses like trend analyses, regressions, and correlations can be computed. Dashboards are a good way to help stakeholders become involved in data analysis while presenting the data in a clear format. The Office of Assessment is beginning to discuss the pros and cons of purchasing or creating a dashboard for the Assessment Office. More research needs to be done in order to determine what software would be most appropriate for our needs.

Leave a comment

Filed under Computational Tools, General Observations, Methods

Mobile Survey Lab

The College recently purchased several Nexus 7 tablets for the Office of Assessment to use for survey and assessment purposes in a newly created Mobile Survey Lab. We rolled out the tablets this week for two separate surveys: a satisfaction survey for the ‘Finals Grab & Go Breakfast’ and a survey on the utilization of professional development funds of graduating seniors, during cap and gown distribution. The tablets are being set up to loop to the start of a new survey once a student completes one, allowing students to take the survey on a rolling basis. During the breakfast, students stopped at our Mobile Survey Lab table as they exited, to complete the satisfaction questionnaire:

Image

During the funding survey administered at cap and gown distribution, students passed the tablets around as they waited to be called up. This instantaneous survey method allowed for real-time, instant results, and is already proving to be a useful tool for increasing our survey response rates greatly. We were able to achieve an 83% response rate on the funding survey, as 276/332 graduating seniors completed the survey on tablets in person. We are looking forward to using these tablets in new and innovative ways for surveys and other assessment related tasks. This is just another way that assessment is allowing us to push into new areas and affect positive change. Look for us at the ‘Finals Grab & Go Breakfast’ for the rest of finals week, as we continue to collect data, and give our students an opportunity to provide valuable feedback.

Leave a comment

Filed under Computational Tools, Methods

Assessment Perspectives from the 2014 NMC Horizon Report

ImageThe New Media Consortium (NMC), a group of leading universities, colleges, museums, and research centers, has recently published the “The NMC Horizon Report > 2014 Higher Education Edition”, a collaborative effort between the NMC and the EDUCAUSE Learning Initiative (ELI). This report is a readable blend of higher education trends and predictions of the impact that those trends will have on higher education institutions.

The really nice part of this package, unsurprisingly since we are covering it here, is the tip of the hat to assessments role in mining new troves of data that new modes of education are producing. We have seen this trend noted elsewhere as well as on this blog here, where we covered MIT’s approach to using usage data for the assessment of MOOCs.

In a chapter entitled “Rise of Data-Driven Learning and Assessment”, the report touches on the sensitivity of personal data, the possible use of this data to drive adaptive response, and the value of this data for assessing learning.

You can get get a pdf copy of the report here, or go to the NMC site, here to learn more.

Leave a comment

Filed under Computational Tools, Planning

Google Drive for Project Management

Google drive is great, but it is not always obvious how best to use it for project management. With just a few steps though, you can be up and running with minimal effort.

1) Use the drive like an external drive.
One of the greatest features that Google didn’t have when it first started was a directory structure. Users like a directory (folder) structure and shortly after starting drive Google implemented directories.
The best part of these directories is the way that sharing was implemented. Just like any other file you might have on the drive sharing settings can be implemented on directories.
All of this means, that for each project you can set up a directory, by default you will have read/write access (more in this later), add all of the other users that you would like to share with, and boom, you have central shared space for all project members. Any file you drop into the directory will be shared with all.

2) Pay attention to sharing preferences.
As mentioned above, setting preferences on a directory can be very helpful. Once set, anytime you place a file in that directory, it inherits the sharing preferences of the directory. All your files for a project can be shared with the group, if optimal, you the owner can maintain read/write access, while the group only has read access. This way sets of files can be shared from a single project directory. This includes files and subdirectories that you have included for organization.

3) Let your users use their own credentials.
Implicit in the directions above is your use of individual accounts. The advantage here is that group members log into and out of the shared space using their own credentials that they are responsible for remembering. You must ask what credentials they would like to use, but once you have set this up once, you are done. If you have a user or users that don’t have or don’t want to share their credentials you can create a read only account for them to use.

With just those three you should be off to a good start.

1 Comment

Filed under Computational Tools