College of Health and Human Services
School of Physical Therapy
1997-1998 Assessment Report
Prepared By: Averell S. Overby, Dr.P.H., P.T., Director
on behalf of the Faculty of the School of Physical Therapy
Dennis Cade, Ph.D., P.T.
Gary Chleboun, Ph.D., P.T.
Stephanie Carter, M.S., P.T.
Doug Kohn, Ph.D., P.T.
Chris Petrosino, M.Ed., P.T.
Rosalind Hickenbottom, Ph.D., P.T.
School of Physical Therapy
Assessment Report 1997-1998
Introduction
The School of Physical Therapy has now completed implementation of the entry level Master’s Degree program in Physical Therapy (MPT). We graduated our first class of 35 students on June 12, 1998. We are expecting our on-site visit by the national accreditation team from the Commission on Accreditation in Physical Therapy Education (CAPTE) April 11-14, 1999 and have begun preparation for that event. Although we will have to thoroughly evaluate all aspects of our program for that visit, the preliminary steps which we have completed thus far will be reported here. In addition, this past year, we participated in the Seven Year Review for the University Curriculum Council. The report of the committee which performed our review is attached as Appendix A.
In the following paragraphs, we will attempt to answer the questions posed in the Guidelines for 1998 Assessment Reports.
Assessment Report
For the school’s goals and objectives for student learning, please see Appendix B. These were reviewed again this year by the faculty and were not changed or modified as we agreed that they continue to represent our instructional goals. Furthermore, because we had not graduated a class when these were reviewed, no data were available which would lead us to change them. Therefore, the last modification was done in 1996.
Student learning is assessed in many ways throughout the curriculum, at graduation, and after students have graduated to determine if our curriculum is meeting the students’ needs to practice in the current health care environment.
During the curriculum:
Students are asked in several courses to keep a journal or portfolio (Appendix C) of what they have learned, not what they have been taught. Instructors use these journals, in some cases, to modify content and process throughout the course if it appears that the student is not learning the material. In other cases, the students turns the journal in at the end of the course so that the instructor can assess what students have actually learned. Since our courses are sequential, this allows us to assess and modify the plan for the subsequent courses so that all pertinent and necessary information is covered.
Another way learning is assessed is through problem solving (Appendix C). Students work in groups to determine, from a case history and partial evaluation, what the primary needs of the patient are and which treatment plan should be implemented. The case is then discussed with the entire class which allows for clarification of items which were not clear from lecture or readings.
Of course, students have midterm and final exams which also assess their ability to problem solve and apply their knowledge to case presentations. Many tests include essay questions so that instructors can assess the student’s thought processes. Students are also given practical exams during which they must take information which is provided to them to diagnose the patient’s movement deficits and plan a treatment program to correct those deficits. They must also carry out that treatment plan so that the instructor can evaluate the student’s psychomotor abilities. Students must pass practical exams with an 80%; otherwise they must re-take it so that we are certain that they possess the necessary skills.
In order to ensure that the content of a course remains stable so that we know where constructs are being taught in the curriculum, a course evaluation sheet was devised (Appendix D) and a course notebook has been started for each course containing the course evaluation sheets as well as all handouts and reading assignments for the course.
Students also go on clinical rotations during which they are evaluated by clinical instructors on their professionalism, as well as their clinical skills and critical thinking. They use an instrument called the Blue MACS (Mastery of Clinical Skills). Careful documentation by the student, clinical instructor, and the academic coordinator of clinical education (ACCE) ensures student acquisition of knowledge and skills required to practice safely. An example of a completed skills list is included in Appendix E. Skills which the student must complete during each of the clinical affiliations or practica are included in the Appendix. We have evaluated use of this instrument and have decide to change to an instrument which was perfected by the educational division of the American Physical Therapy Association. Since the new instrument has been designed so that significant outcomes data can be derived from it easily, and this will enable us to benchmark with other physical therapy schools, we have decided to use it starting with the first year class.
We try to ensure that clinical instructors are able to assess student learning and the clinical sites are top quality through a variety of means. One way is one-on-one counseling with the ACCE, one of our faculty. Another is through an evaluation which the student performs at the end of a clinical rotation. This evaluation (Appendix F) is reviewed by the student with the clinical instructor and is kept on file at the school so that other students may see how the facility was rated. If the evaluation is poor, the ACCE schedules a time with the facility and the coordinator of clinical education in that facility to try to remediate the problem. We also ask that the facility and the clinical instructor complete a self assessment (Appendix G) annually so that they are aware of their performance. This year, we have been asked to participate in a study in which the clinical facilities evaluate our ACCE (Appendix H) and we believe that this will help us to improve our performance also.
This past year, we have also utilized focus groups of clinicians in the Columbus and southeastern Ohio area and the Cincinnati area to help us to evaluate what we are doing in terms of sequencing and the curriculum (Appendix I). We plan to do additional sessions next year in the Cleveland area. Major themes from these discussions were analyzed and reported to the faculty. We also used this opportunity to educate them on changes which we had made in the curriculum to keep them up to date. We regularly send them information on the curriculum so that they know what the students have had in classes so that they can expect them to utilize those skills and so they know what needs to be taught to the student if they have not yet covered it in class. We feel that with all of these checks and balances, the goals for instruction in the clinical sciences, communication, and critical inquiry are being met.
Professionalism is also assessed in clinical rotations; however, we also assess it in the curriculum through use of the generic abilities with our advisees and in some courses. We implemented this tool last year and an example of how it is being used with advisees is attached (Appendix J). This tool has also been used when a student exhibited problematic behavior at a clinical site to identify where the weaknesses were and led to a successful completion of the assignment. We also assess this goal through student participation in activities related to the profession, such as attendance at educational and professional seminars. This year, 23 students attended the national annual student conclave in Phoenix, Arizona where they mingled with students from all of the schools in the United States and learned more about their professional association, as well as attended educational seminars. We also had approximately 25 students attend a continuing education series on balance scheduled over a weekend. Of course, they regularly raise money for the Foundation for Physical Therapy Research and for various scholarships in the school.
Communication abilities are assessed in their written and oral presentations and in their clinical experiences. This past year, the third year students presented their research during an afternoon in which an oral or poster presentation was done. First and second year students, as well as faculty, had the opportunity to discuss their research findings with them during this time. In addition, 6 groups of students presented their posters at the Ohio Physical Therapy Conference in Columbus in April and one group has submitted an abstract for a national conference. These activities successfully meet our goal for communication.
At Graduation:
This year, we implemented an exit survey in which we asked students if the curriculum had met their needs to engage in their final clinical affiliations and to begin professional practice (Appendix K). We have not yet had the opportunity to completely evaluate these since the students turned them in at graduation; however, preliminary results are in and will be discussed later in the report.
After Graduation:
We will implement this year (6 months after graduation) an employer survey for our recent graduates (Appendix L). This will be an annual event and should provide us with important information related to how well our graduates are prepared to function in today’s health care system. As can be noted, the questions asked of employers are designed to determine if the goals of our program are met through graduate performance. Therefore, there is great congruence between the goals of the school and the employer survey. Concurrently, we will send the graduates a survey to determine how well we prepared them for clinical practice. The committee working on these surveys has not yet developed the 6 month survey because the results of the open ended survey we gave the students at graduation may shape the content of the subsequent survey.
As the last piece of the evaluation of the curriculum, we will look at the results of the national licensing examination given through the Professional Examination Services. Because errors had been reported in the past, we requested that students give us permission to obtain their individual scores. All but two students agreed, so this information will be accurate and will also help us to evaluate the curriculum.
Admissions Process Review
Since the outcome, or product, of the program depends rather heavily on the nature of the students admitted, our assessment this year included an evaluation of the admissions process (Appendix M) which was done by the faculty before admissions were completed and after admissions were completed. Prior to the current class admissions, the interview procedure was examined. We decided that more training was needed for faculty and clinicians who perform the interviews, since there was great variation in the scores on the same student. (We usually have approximately 8 students in each group with a faculty facilitator and two clinicians from the community.) In addition, we noted that the descriptors were not consistent with the Generic Abilities which we were trying to develop in the students. We felt that we should try to select those students who had the potential to develop those qualities which we wish our graduates to possess. Therefore, we changed the descriptors to be consistent with the Generic Abilities and to more precisely define the scores from 1-5. We held 3 separate training sessions for the interviewers. A training film was developed in order to assist the interviewers in identifying different behaviors. The result was that we noted that the interviewers used more of the entire scale from 1-5 rather than 3 and above and the range of the scores increased from 31 in 1995 to 35 in 1998. The admitted mean was also higher and there was more consistency in scores, although reliability statistics were not as yet computed. We also decided, after reviewing the essay requirement and after the current admissions procedure was completed, that we wanted prospective students to write on a topic on-site since therapists are required to document cogently without having a great deal of time to think about it. This will allow us to compare the writing sample they submit with their application to that done extemporaneously. Another decision on the admissions process was to include life experiences as one of our criteria and not to include GRE since these scores have not been predictors of success with physical therapy students. Finally, we made a decision on how to score the various components of the admissions process.
In summary, we have implemented many of the assessment procedures which we indicated that we would and examples of what we have changed are listed below.
Evidence which indicates we are meeting our objectives can be found in the fact that 100% of our students completed their clinical affiliations and were found to have mastered the clinical skills we required. In addition, 75% of the students reported that they currently have found employment and many were offered employment by their affiliation sites. The others simply have not looked as yet. Seventy-five percent of the students are scheduled to take the national licensing examination in June or July; the rest have not applied as yet. As mentioned, 6 groups of students presented their research findings at a state conference of physical therapists and one group has submitted an abstract for a national conference. The only evidence thus far which indicates that we may not be meeting all of our objectives is that some students did not complete a few of their master’s list of skills during their clinical affiliations and this must be investigated to determine why these skills were not completed. This will most probably require additional education for students and clinical instructors by the ACCE to alleviate this problem.
Finally, the results of the exit survey indicated that students were prepared for their clinical experiences, although some did indicate that they felt weak in diagnosis and prognosis of physical therapy problems. Because they relate to the goals of our program, the responses to the first question will be given here.
Well Prepared Program Lacking/Not Prepared
Patient/Client Management
Evaluation 29 6
Diagnosis 18 16
Prognosis 13 22
Intervention 20 15
Consultant 28 1
Educator 30 2
Critical Inquiry 30 2
Supervision/Health Care Delivery 25 3
* There was a 100% return rate, for a total of 35 students. Where the numbers add to fewer than 35, students did not indicate a response or did not understand the question.
These were addressed at a faculty meeting held June 22-24 of this year and objectives were revised and content increased in these areas. This will now be monitored in the coming year by the faculty and we would expect that we would have less than 25% indicating weakness in these areas next year.
If one looks at the goals and objectives which we submitted last year, the following activities have been implemented:
We have taken steps to ensure that we receive correct information on the national licensing examination for our graduates by having the students release their scores to us directly. We have initiated the exit survey for graduates and have the preliminary results at this time. We have formulated the employer survey based on our goals of the program and this will be mailed in December 1998. We have taken steps to ensure that the clinical experiences for students are developing the qualities which we desire in students. In addition, we have done additional re-sequencing of the courses in the curriculum as a result of student and clinical input. Most recent changes (Appendix N gives examples) include changing the part-time clinical experiences to one hour courses and removing them from the didactic courses; deleting one part time clinical affiliation; moving Clinical Orthopedics II from the Fall quarter to the Spring quarter since most of the students are going on orthopedic affiliations and in Practica I and II and the students needed these skills prior to the practica. We also moved PT 661, Motor Control (Pediatrics) to the Fall quarter of the last year so that students could take full advantage of their pediatric rotations which usually occur in the last quarter of the third year. We moved PT 663, Sensorimotor Physical Therapy to the Winter quarter of the second year to allow students to get a rehabilitation clinical experience earlier in the curriculum, since there are so few of these types of affiliations for students. We have allowed students more freedom in choosing their clinical affiliations, based on their interests, and we have also allowed more flexibility concerning when they may be taken. We also changed the clinical rotations so that most of the full time affiliations or practica are occurring in the final year. This was done because clinicians, who are their instructors in the clinic, wanted them to be at a higher level of patient care when they came to the clinic. We have also considered adding a course on Neurology to the first year since students often complain that they do not remember the information from Neuroscience (a prerequisite) once they are in the curriculum and courses based on this content (Neurorehabilitation) do not appear in the first year. A special topics course will be implemented this Fall as an experimental course. We have also documented the course content of introductory courses so that we do not inadvertently re-teach information which has already been covered. We have recently reviewed and evaluated the objectives from all courses in the curriculum and faculty are re-writing these for the next year.
Goal $5: The student will share information through effective oral, written, and non-verbal communication with patients, peers, supervisors, caregivers, and the public.
The students have presented their research in poster or platform presentations to faculty and students in the other classes. Six groups of students presented at a state conference and one has written an abstract for a national conference.
We are using the Generic Abilities Assessment in an advising capacity with students and some faculty have incorporated these into course objectives. We need further work with using it with students with problems to guide their remediation. Twenty-three students attended the National Student Conclave sponsored by the American Physical Therapy Association. Twenty-five students attended a continuing education course sponsored by the School of Physical Therapy on balance. The students raised money for scholarships, research in physical therapy, and participated in KidsFest in the Convocation Center. Students presented in-services on their clinical affiliations which included research topics as well as new clinical practices.
In academic program/curriculum: At a recent national meeting, an approach to using a document generated by the American Physical Therapy Association (Guide to Professional Practice) was presented. Because of its connection with the accreditation standards of the external review organization, the faculty will carefully evaluate how we will implement this document in the curriculum. Other changes would surely be premature as we have just graduated the first class on this new curriculum.
In your department’s assessment process? At this particular time, it is difficult to recommend changes in the assessment process since we have not fully implemented what we have designed. We would certainly recommend that the assessment process continue to expand to include a review of the school’s mission statement, philosophy, continued review of the admissions process, etc. Next year, we will also add a postcard to the newsletter sent to alumni which they can fill out relative to their continued development; that is, an advanced degree, certification as a specialist, etc. Since many of the faculty area the same ones who taught these alumni, we can try to get an idea if we are instilling the idea of lifelong learning into our graduates.
In conclusion, this has been a busy year with the graduation of our first entry level Master’s degree class and with assessment activities as a result of their graduation. Now we must continue to monitor the curriculum to make sure that we are meting current practice needs.