n Teacher-made tests. This is the primary assessment tool in most IT classes. Both objective and subjective test items are used.

n Evaluation of written reports. Faculty and peer evaluation of written reports by students include professional society reports, production reports, technical reports, informative reports, book reports, and annotated bibliographical note card reports. In addition, students have applied their knowledge and produced more non-traditional forms of written reports, such as the web site produced by the group of students who participated in a manufacturing study-abroad trip this year to Argentina.

n Evaluation of oral reports. Oral reports given by students in class are evaluated on individual and group (team) bases. These reports are evaluated using check sheets listing important topics to be covered and presentation skills.

n Design / Documentation. Students produce packages that include process and operations analysis that guide the planning and procedures for part production, and design manuals that are used to conceptualize and specify components in circuitry.

n External Reports. In one case, parents of our students were surveyed on the performance of our students during our Parent’s Weekend Project. Other examples include feedback from Co-op supervisors and professional society reviews of student projects and works.

n Project work. Project work in this context more specifically relates the actual production of manufactured products such as bench vises, pen sets, roll-top disk files, etc..

n Interviews. The Industrial Advisory Board interviews a selection of students regarding their progression through the IT program. Formal interviews are also conducted by industrial personnel when selecting students for co-op positions, and by Dean Emeritus T. Richard Robe when selecting students to participate in the College’s Leadership Development program.

n Competency testing. Observation of students using manufacturing machinery is accomplished as part of several courses. Students are required to learn safe and proper operation of typical machinery used in industry, and are expected to manufacture assigned products as an introduction to the use of the machinery. Power (timed) practical examinations are given to students on an individual basis to determine their competency in setting-up circuitry and programming machine tool operation. Furthermore, some courses incorporate a competency-based model that requires students to prove competency in machine operation and process knowledge during laboratory activities (e.g. check-outs for skills and knowledge). Most IT courses require the use of computer software to accomplish assignments. Student learning is assessed by their competency in using this software to produce required assignments.

n Pre-test, post-test analysis. During the orientation phase of some courses surveys are given as an overview of the course and to measure pre-existing knowledge. The same survey is then given at the end of the quarter to measure the increase in student knowledge. (See Appendix B)

n Standardized tests. Tests using national norms are administered to students. The primary instrument used is the Society of Manufacturing Engineers’ Certified Manufacturing Technologist Exam. Students take this exam on a voluntary basis.

n Capstone course performance. In our capstone course, Product Manufacture (IT 462), the managerial skills of planning, organizing, leading, and controlling are recorded on a standard form relating to the execution of their final team production project. In addition, this course is used in a variety of assessment activities including machinery usage, production analysis reporting, trouble shooting of machinery and process malfunction in a production environment, and in group dynamics studies. IT-400 – Senior Seminar is another capstone course in our program. This course is not dedicated to manufacturing practices but is more geared toward career development.

n Graduate surveys. The department conducts surveys of recent graduates on an annual basis. Alumni are regularly surveyed to determine the quality of the education they received, to determine their level of satisfaction with the IT program, and to look for ways to improve the program. Survey results from former students and their employers provide information about their positions upon entering the companies.

n Course and class attendance records. Classroom and laboratory attendance is closely monitored by instructors in nearly all cases, using roll call, sign-in/sign-out cards, and group (team) meeting records. Attendance in class is closely monitored and has a direct impact not only on student learning (facilitation of participation) but also on student employability after graduation.

n Longitudinal studies. Data from surveys of alumni are compared from one year to the next to evaluate trend changes in graduate’s status. In addition, the department administers an annual follow-up survey of baccalaureate graduates five years after graduation. The perspective of these graduates often is more valuable than those of recent graduates. In addition, this affords an opportunity to examine career development and has allowed us to develop a compendium of business cards from graduates back to 1970. This "notebook" is often used by senior level students to establish contact with other OUIT graduates in the workplace, and is often the focus of intensive examination by alumni when they return to campus long after graduation.

n Institutional research studies. Data from the Ohio University Office of Institutional Research are used as other summative assessment of the status of departmental graduates and their satisfaction with their baccalaureate program.

n Assessment activities in the past year have been modified to formalize the review of co-op supervisor responses (See Appendix C-1).

n Every IT graduate back to 1980 (rather than our typical one year – five year cycle) was contacted to solicit information about current employment status and to update the business card file.

n Some assessment activities have been altered in our determination of the classroom/laboratory performance of our students. For example, the assessment methods have changed in Engineering Drawing II. In the past, standardized assignments were based on lecture topics. Within the last year the class has changed to include students’ analyses of an assembly to determine the best methods to draw the detail parts. Due to this change, the assessment is more open-ended with a variety of possibilities being acceptable. It is more time consuming to evaluate drawings, but also more realistic.

n A concerted effort was made to analyze the assessment of oral reports given by individuals and groups in classes. A team of faculty members devised categories of observation, applied Likert-type scales with descriptors, reviewed this development with faculty members in the School of Interpersonal Communication, redesigned the instrument, and field-tested its use (See Appendix D). The result is a more standardized format for assessment of students’ individual and group oral reports throughout our program.

n The Society of Manufacturing Engineers drastically modified their summative report format to institutions during the past year on the Certified Manufacturing Technologist Exam. This made it more difficult to assess categorical changes in the group performance of our students when comparing current with past results.

Some examples of how data are analyzed follows:

n Data are being analyzed on a continuing basis to evaluate the salary level and type of employment being achieved by IT graduates. This information is summarized in tabular form, business cards are kept for easy retrieval and review, and averages of salary and tables of job classification are prepared and made available to faculty and students alike.

n The results from teacher-made tests are analyzed using various methods. Students are provided with summary descriptive statistics (means, ranges, and standard deviations) as well as their personal test results for objective items. Class discussion is held clarifying any test item of concern and individual meetings are held with students when requested. In some cases, test item analysis provided by computer services is used to review test item reliability. Subjective testing also tends to be content driven, with feedback provided by the instructor based on omissions and incorrect concepts.

n The analysis of written reports appears to be the most subjective and inconsistent method of assessment used in our department. For the most part, the class instructor analyses written reports and provides feedback to the student based on standards provided in the class syllabus or during class discussion. Collectively, the subjective evaluation of written reports tends to be highly based on content. However, some instructors analyze coherence, structure, mechanics, style, and other characteristics of "good" writing. In one class, an elected peer representative also reviews all written reports. After blind reviews by both the representative and the faculty member, grades are negotiated for each written report.

n The analysis of oral reports also tends to be subjective. However as mentioned in the previous section, departmental faculty have taken the initiative during the past year to remove some of the inconsistency and bias from this analysis. In addition to the incorporated use of a standard form for oral report evaluation, multiple assessors are also typically used. During the past year we have also purchased our own video equipment to use during grade negotiation by multiple assessors, and as an aid for students to practice their presentations and review results with the instructor.

n Data from design / documentation activities are analyzed by instructors and by the students themselves. Operations and process analysis sheets are devised by students and reviewed by the instructor prior to students using tools and machinery to produce products. The success or failure of these procedures provide immediate feedback to the student. Design manuals tend to produce analytical results. Instructors analyze both the answers to those mathematical problems and provide feedback to students on the procedures followed to gain those results. In some cases, computer aided instructional packages that contain embedded expert systems shells allow students to attain immediate feedback on their results.

n Data from external reports appear in many forms. For example, the co-op supervisors report contains descriptions in six categories using forced choice rankings and three summative evaluations (two with polar responses and one with a Likert scale). (See Appendix C-1 ). The assessment committee applied numerical ordering from 1 to 5 for each forced choice rankings and determined means and standard deviations for each category for all 35 co-op students for 1997-98. Similary descriptive statistics were calculated for the remaining polar ratings (only numbers 1 and 2 were used) and the Likert scales (numbers 1 through 5 used). In all cases lower numbers represented superior performance. (See Appendix C-2). Other external reports such as the mailed evaluation survey for the Parent’s Weekend Project allowed parents to choose from a list of activities those that they deemed successful and also those that they recommended to be repeated in future Parents’ Weekend Projects. These results were summarized by the number of "hits" for each activity. In turn, the student committee determined which activities were worthy of further development. They served as guidance for planning next year’s project.

n Analysis of the data provided by project work is the responsibility of the instructor. The instructor will analyze the results by comparing the resulting products to established standards (non-precision and precision linear measurement, surface finish, structural flaws, verticality, etc.). Eventually, these results are interpreted as points on a grading scale provided to the student during class orientation. In some cases descriptive summative statistics are provided on the collective performance of the class.

n Interviews provided by industrial personnel during the co-op screening activity are summarized and provided to the assistant dean for career services. In cases where she believes that intervention is necessary she will provide professional counseling to our students. In extreme cases she will report situations to the department chairperson for further planned actions. Interviews provided by the Leadership Development Program result in two of our students being selected to participate in the program annually. This selection is reported to the department chair, but no other collective or individual evaluation is provided. Similarly, the results of interviews arranged by Ohio University’s Career Planning and Placement Office yield only initial employment results and no evaluation of the actual interviews are provided.

n Competency testing using practical examinations are analyzed by the instructor although the nature of these exams allows students to immediately observe whether their work is correct or not. Quite simply, the circuits or controls operate to provide the desired output or they don’t. Sometimes the level of competency is analyzed by requiring students to set-up multiple operations (either in a random selection model or through planned increasing complexity).

n During the period of this report only one course used pre-test – post-test analysis. Although the validity of this type of analysis is often questioned, in this case the instrument used was perceptual. It simply asked students to identify basic computer skill that they thought they possessed entering the course and contrasted those responses using the same instrument after completing the course. Item analysis is performed by the instructor to determine if instructional topics are appropriate for the class.

n In the capstone Product Manufacture (IT 462) class the instructor recorded notes on students’ managerial skills. They are reviewed with the students so they may improve their actions. This information is reviewed by the department chair and shared with the faculty.

n Data from the SME CMfgT test are reviewed for each test to discern the level of knowledge of the students taking the test compared to previous IT students, and among themselves. The data are summarized using descriptive statistics (mean, standard deviation) and the data are discussed among faculty at staff meetings both in the context of the most recent test results, and compared to past test results. Further, the results provided by the SME to the IT department provide a breakout identifying, by broad subject category, the performance of the test subjects. These data are discussed and a rationale for curricular or teaching method change is developed based on the results.

n 1. Gainful employment goal. Marty North, Assistant Dean for Career Development, reports that no intervention was deemed necessary for any of our students interviewing for co-op positions during the period of this review. The average "overall performance" rating for our students (N=35) participating in the co-op program was 1.7429 on a five point scale. Although this is a formative measurement of the "in-process" capabilities of our students, it does indicate that supervisors in the industrial workplace deem our students to be prepared for entry level positions with their companies. In addition, 18 of the 36 graduates from the department during this period responded to the departmental follow-up survey of recent graduates. All of the respondents from this year’s survey reported attaining gainful employment. Similarly, all of the eligible respondents (N=20) to the most recent (1995) Institutional Research Career and Further Education Study reported attaining gainful employment within eight months of graduation. However, when expressed as a percentage, those currently holding full time employment at the time of the survey was 95% (n=19) and 5% (n=1) reported that they were currently unemployed at that time. We can’t make alumni respond to these surveys but with the data we have gathered it is fairly safe to say that we have reached this goal during the past year (of course, non-respondents may be those whom did not attain gainful employment).

n 2. Knowledge and technical competence goal. The only formative indicator of accomplishment of this goal is overall articulation of the program by students. During the period of this review the total number of major enrollments in technical courses in industrial technology was 903. Of this total only 14 received failing grades. This represents a course articulation rate of 98%. There is no requirement for a grade of "C" or better in any industrial technology course. Therefore, this arbitrary (although institutional) standard indicates that students are demonstrating knowledge and practical competence in the discipline of industrial technology. One summative indicator of accomplishment of this goal is that 84% of the respondents to the Ohio University Career and Further Education Study reported being either "extremely satisfied" or "very satisfied" with major courses (only 17% reported being "somewhat satisfied" and 0% reported being "not at all satisfied"). Another indicator of accomplishment is the fact that 100% (39 of 39) of the students enrolled in our capstone course during the period of this review successfully completed this requirement. Finally, the summative results of the SME Certified Manufacturing Technologist exam reveal that 57% of our students taking the exam (N=21) during the period of this review passed. More important to this goal is that the average percentage on the eleven categories directly related to technical content on this exam was 76%. However, using these results as an indicator it is apparent that we need to examine our performance in drafting/engineering drawing, manufacturing processes/machine tools, process planning, QC/QA, CAM/CIM, and CNC/NC in the areas that are under our direct control as course offerings.

n 3. Manufacturing and Computer Hardware and Software Goal. The use of manufacturing hardware is prominent in the materials process sequence of coursework within our program. Sixteen separate course offerings having extensive orientation to manufacturing hardware were provided during the period of this review. This cumulative offering represents 278 student enrollment, of which 276 were successful completions. In the area of computer hardware and software usage eight separate course offerings were made during the period of this review, representing 179 student enrollments with 173 successful completions. Finally as a positive indication of attainment of this goal, in the three offerings of IT-103 – Computer Usage in Industrial Technology, using pre-test / post-test analysis, pre-testing revealed that of a possible 3312 iterations of software usage familiarity only 1452 responses were positive. During the post-test analysis 3300 responses were positive (must have been absentees that didn’t catch-up). It is notable that we currently do not know, cumulatively, the proportion of evaluated coursework that is dedicated to manufacturing and computer hardware and software. However, it is reasonable to assume from course titles and descriptions that, a major proportion of the classes’ requirements were based on this parameter in the courses described in this section. Therefore, it is reasonable to assume that this goal is currently being achieved.

n Other goals. Evidence of accomplishment of all other goals is speculative and primarily based on anecdotal reports from faculty. We do believe that we are making conscious efforts to attain them, however, we have not, at this time, isolated the data from various sources to reach a decision of whether we are accomplishing these goals or not . (Remember this is the first year that we have had these student outcome goals). The one exception relates to the goal of positive professional attitudes. Co-op supervisors evaluate our students highly in this area as revealed by the following cumulative averages on a scale of 1-high to 5-low in their reports: Relations with Others – 1.69, Attitude – 1.97, Judgement – 2.14, Dependability – 1.62, Quality – 1.56, Attendance – 1.34, and Punctuality – 1.34.

n We recommend that the University grade reporting form provide information on a student’s major instead of his/her college. This would facilitate analysis of the articulation of our majors in the courses under our direct control. Currently, class roles contain majors that evidently are generated by the students themselves (we find it hard to believe that some of these majors actually exist). This information has limited use when determining student outcomes. Grade reports list the college of each enrollee. This may be of benefit to the University but serves little purpose to the department or the assessment committee. It is our understanding that major number codes exist for each recognized degree granting program. Perhaps these codes could be used or an automatic generation of major title could be generated from them as linked to the student’s social security number and DARS or SIS. This would eliminate the need to associate the grade reports with our departmental list of majors when examining grade reports for assessment purposes.

n We recommend that required departmental courses be grouped into broad categories such as materials and processes, design/documentation, power and energy, and information technology. These groupings should be applied to the student check-sheets and should be used in internal data bases and in DARS. This would help current and prospective students to put courses in the program into context and would eventually (with the development of relational data bases) allow us to more easily access groupings of classes that align with student learning outcome goals.

n We recommend that all teacher-made tests be formatted for topical technical content matter (e.g. injection molding questions should be clearly separated from rotational molding questions, word processing questions should be clearly separated from presentation software questions, etc.). Furthermore, faculty members should clearly categorize those questions on each teacher-made test as being technical knowledge; manufacturing and computer hardware and software usage; oral , written, and graphical communications; or analytical and empirical problem solving. Collective and individual records should be kept not only on overall performance and item analysis, but also on the topical and categorical results of each test. Overall performance statistical summaries for each class should be provided to the assessment committee to synthesize departmental student learning outcomes in the categories related to the student leaning goals.

n We recommend that a "writing across the curriculum" department committee be established to develop a coordinated intervention mechanism to improve student writing skills and to establish departmental standards for all categories of student writing and to develop standard evaluation guides to be used by all departmental faculty. It is suggested that these guides incorporate areas such as technical content, creative use of computer software, evidence of information gathering, examples of analytical and empirical problem solving, incorporation of graphical communications, industrial ecology or interpersonal relations, and evidence of international considerations in text or source citations.

n We recommend that at least one faculty meeting for next year be dedicated to allow those faculty members who worked on the performance standards and grading guides for oral reports to present their findings to all faculty. This should be the beginning of discussion that will establish standards for oral presentations for student use and a departmental evaluation guide to be used in all classes.

n We recommend that faculty using 1.design/documentation assessment, 2. project work assessment, and/or 3. competency testing assessment in classes report the results of these activities in summary fashion for each class in terms of average percentage of possible points awarded within each classes’ grading scale. These results should be provided to the assessment committee on a quarterly basis.

n We recommend that a committee of faculty and graduates be established to develop an exit exam to be used in IT-400 – Senior Seminar or to examine the development of the proposed National Association of Industrial Technology Certified Technologist Exam to determine if it would be appropriate for a departmental exit exam.

n We recommend that a concerted effort be made to establish standards for student performance in IT-462 – Product Manufacture and recommend that areas such as knowledge and technical competence; use of manufacturing and computer hardware and software; evidence of managerial skills in the laboratory work environment; oral, written, and graphical communications; technical problem solving; and evidence of ethical standards be considered in individual and team evaluations used in this class.

n We recommend an overhauling of the current survey of recent graduates used by the department to replace the items related to "favorite courses" with questions that would reveal the respondents’ perception of the program’s contribution to their gainful employment, knowledge and technical competence, use of manufacturing and computer hardware and software, development of positive professional attitudes, and awareness of the international aspects of industrial technology.

n We recommend that a concerted effort be made to contact every new alumni eight months after graduation to determine their employment status and information pertaining to the previous recommendation. The mailed survey should be considered as the primary method, however, follow-up for non-respondents should be planned with phone or personal interviews. The same (or a slightly modified) survey should be distributed to each graduate five years after graduation to determine if their careers have progressed as expected and to see if time has changed their perception of the program’s contribution to those categories previously mentioned.

n We recommend that our industrial advisory board be consulted to develop strategies for improving the number of respondents to our departmental survey of alumni employers.

n We recommend that a longitudinal study be undertaken to determine if our students initial salaries have at least kept-up with increases in the consumer price index for the period from the first report of average starting salaries from Ohio University’s Career Planning and Placement data.

n We recommend that categorical test questions should be added to teacher-made test and that pre-test / post-test or longitudinal studies be performed to assess our students’ awareness of the international aspects of industrial technology.

n We recommend that a formalized procedure be implemented to have faculty record contacts with industrial personnel that result in employment of our graduates.

n We recommend that a funding mechanism be devised to allow assistance to outside speakers coming in to give presentations, to offset costs associated with travel to manufacturing facilities, and to assist faculty in making contacts with industrial manufacturing managers to develop relationships that would foster further shadowing of these professionals by students. Faculty using outside speakers in class should develop an instrument to be used by students to express any attitudinal or behavioral changes that may be evident in students as a result of these activities. For example, the minute assessment technique of having students write a brief directed essay on the experience would seem to be appropriate. In turn, the instructor should compile these reports and provide a summary of the outcome to the department chair an/or the assessment committee for inclusion in the annual assessment activity.

n We recommend that end-of-class reporting by faculty members be expanded beyond the current grade report. Eventually, all sorts of individual and class collective summaries of performance could be included. As a starting point, we recommend that a relational data base system be established on our local area network that would allow faculty members to record individual grades and class attendance records expressed as a percentage of potential sessions attended by each student.

n We recommend that the student learning outcome goals established for this assessment reporting cycle be reviewed and performance standards be established for each goal.

n We recommend that the departmental assessment committee membership be expanded to include two student members. If quality is "satisfying the customer" then this would appear to be an appropriate action. It would assist the students in developing positive professional attitudes, help clarify actions from the students’ prospective, and validate our student learning outcome assessment.