Assessment of Electronic Course Evaluation Technology and its Applicability to the University of Alaska Fairbanks - PDF

Please download to get full document.

View again

of 18
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Information Report
Category:

School Work

Published:

Views: 7 | Pages: 18

Extension: PDF | Download: 0

Share
Related documents
Description
Assessment of Electronic Course Evaluation Technology and its Applicability to the University of Alaska Fairbanks University of Alaska Fairbanks P.O. Box Fairbanks, Alaska, Authors: Franz
Transcript
Assessment of Electronic Course Evaluation Technology and its Applicability to the University of Alaska Fairbanks University of Alaska Fairbanks P.O. Box Fairbanks, Alaska, Authors: Franz J. Meyer (CNSM), Eric Madsen (SoEd), Chris Beks (OIT), Andrea Ferrante (CNSM), Kelly Houlton (CRCD/Dev Ed), Brenda Konar (SFOS), Michael Koskey (DANRD), Sally Skrip (Provost s Office), Nathan Zierfuss (OIT) April 30, 2013 Executive Summary In October 2012, the Faculty Development, Assessment, and Improvement (FDAI) committee together with Dr. Eric Madsen, School of Education, were entrusted by the UAF Faculty Senate to study the current state-of-the-art of electronic course evaluation technology and its applicability to UAF. Early in the study it was recognized that course evaluation technology is an integral part of a university s overall evaluation process. Hence, to recommend appropriate course evaluation technology we need to evaluate all other components of an established evaluation process, including (1) the purpose of course evaluation at UAF, (2) the indicators that we want to use to determine success, (3) and the benchmarks we want to use to evaluate performance. With this report, we analyze course evaluation technology as a part of UAF s overall evaluation process and provide guidelines for a step-by-step approach to optimizing UAF s course evaluation philosophy. The main findings and recommendations are summarized in the following: 1. We recommend to formulate a clear understanding of the main purpose(s) of course evaluation at UAF before deciding upon changes in course evaluation technology (see Section 2). 2. If a change in the course evaluation procedure is planned, we recommend to not change technology and question sets at the same time, but instead follow a step-by-step approach. 3. Electronic course evaluation systems have a number of benefits and drawbacks relative to traditional paper-and-pencil technology that need to be carefully analyzed and compared before selecting the most appropriate evaluation technology for UAF (see Section 3.1). 4. While student response rates are an important factor in evaluating the success of a course evaluation system, it is only one of many performance parameters (see Section 3.2). 5. Electronic course evaluation can produce satisfactory student response rates if students are incentivized, if the course evaluation system is easy to use, if faculty and administration actively promote the importance of course evaluation, and if regular reminders of active or upcoming survey periods are provided to faculty and students (see Section 3.3). 6. Nowadays, a large number of highly capable electronic course evaluation systems are available whose capabilities are ever improving (Section 4.3). 7. From our system survey, we conclude that available technology varies widely in aspects including (1) hosted vs. host-yourself solutions, (2) online-only vs. hybrid (paper plus online), (3) University-focused vs. generic survey-focused, and (4) flexible question set vs. fixed survey format. Also the amount of applied data analysis varies widely (see Section 4.3). 8. Three systems were identified that are excellent in their flexibility and functionality and are also well matched with UAF s needs (Section 4.3). 9. We recommend starting a discussion on the development of a culture of course evaluation on campus to improve course evaluation quality independent of evaluation technology. To further analyze the capabilities of a down-selected group of three electronic course evaluation systems, UAF will continue to examine their suitability in fall We will coordinate our activities with UAF faculty and administration. Details of the evaluation activities in the fall will be announced. ii Table of Contents 1 Background and Motivation of This Study Defining the Intended Purpose of Course Evaluation at UAF A List of Potential Goals of Course Evaluation Potential Analysis Methods of Course Evaluation Results Current Course Evaluation Practices at UAF Discussion of Course Evaluation Technology Pros and Cons of Electronic Course Evaluation (ECE) Concepts Parameters for Assessing the Success of a Course Evaluation Technique A Discussion on Response Rates in Electronic Evaluation Systems Background on Student Response Rates Student Response Rates in Electronic Course Evaluation Methods for Improving Response Rates in Electronic Evaluation A Survey of Electronic Course Evaluation (ECE) Systems System Evaluation Procedure Main Evaluation Criteria Main Findings of the ECE System Survey Recommendations and Future Steps References iii 1 Background and Motivation of This Study In October 2012, the Faculty Senate was approached by Provost Susan Henrichs to evaluate a potential implementation of an electronic course evaluation system at UAF. A discussion on electronic means for evaluating courses offered by UAF is both timely and relevant for several reasons. These reasons include, among others, (1) the need to provide an accessible and equivalent course evaluation method for students in traditional and online class environments, (2) the potential for faster processing times of evaluation results, (3) potential improvements of anonymity and safety of evaluation results, (4) the high costs associated with the current paper-based evaluation system, and (5) an increase of flexibility of course evaluation format, questions, and timing. As a consequence, the Faculty Development, Assessment, and Improvement (FDAI) committee together with Dr. Eric Madsen, School of Education, were entrusted with studying the following two main questions associated with electronic course evaluations: 1. Is it advisable for UAF to move to an electronic course evaluation model? 2. If so, what would be the necessary steps towards adopting an electronic course evaluation system? In addition to these main goals, the activities performed were also aimed at reviving the discussion about electronic course evaluations (ECEs) by surveying their current state of the art and by gauging the sentiment towards ECEs on campus. This report summarizes the research that was conducted towards addressing this topic and provides our main initial findings. The report is grouped into three main sections that reflect our view of the chronology of discussions that need to be led when applying changes to UAF s established course evaluation process. It is important to note that the goal of this report is not to make decisions for Steps 1 3, but to provide suggestions and parameters that will support a future decision making process. We suggest to approach course evaluation discussions by following these three main steps: 1. Define the intended purpose of course evaluation: Relevant parameters that inform this discussion are addressed in Section 2 of this report, which summarizes potential uses of course evaluation and assesses current practices at UAF. Defining the intended purpose(s) of course evaluation is essential to be able to select appropriate course evaluation technology. 2. Select evaluation technology in light of the identified purpose and considering UAF s needs: This topic is addressed in Section 3, which analyzes the pros and cons of paper and electronic course evaluation approaches. We also provide information that help in weighing these pros and cons against each other in support of finding an optimal technology for UAF. 3. If electronic course evaluation is favored, select the most appropriate electronic course evaluation system: This topic is addressed in Section 4, where we summarize the results of an extensive study of existing electronic course evaluation systems. Evaluation criteria are listed together with the main findings of the conducted survey. A detailed description is provided for a subset of systems that fit best with UAF s requirements. A summary of our main recommendations and findings concludes this report. Page 1 of 15 2 Defining the Intended Purpose of Course Evaluation at UAF Nowadays, a wide range of course evaluation systems (both paper and electronic) are available from a number of vendors throughout North America. While these systems have common traits, their evaluation philosophies and the applied evaluation technologies vary widely. Evaluation technologies range from pure paper-based systems to hybrid systems that combine paper with web-based services, to online-only technologies. Evaluation philosophies differ depending, for instance, on whether the system is focused on supporting faculty development or on assessing faculty performance. In addition to the diversity of course evaluation solutions, it also has to be recognized that UAF is an institution with very specialized and highly diverse needs requiring a course evaluation concept that is able to address this diversity. Hence, selecting or designing an optimal course evaluation concept for UAF is a non-trivial task that requires the careful consideration of a large number of parameters. In the following, we list and discuss some of these parameters in an attempt to facilitate a discussion on potential modifications to UAF s current course evaluation concept. From our research we have reached the conclusion that it is a necessary requirement to develop consensus on the intended purpose of course evaluation at UAF before modifications of evaluation technology should be attempted. To facilitate a discussion on the main goal of course evaluation, we provide (1) a list of the most relevant course evaluation goals (Section 2.1) as well as (2) a summary of analysis methods that can be applied to generated course evaluation data (Section 2.2). Findings from Sections 2.1 and 2.2 are compared to the current practices at UAF (Section 2.3). 2.1 A List of Potential Goals of Course Evaluation Course evaluation can serve a wide range of purposes. To extract the most relevant of all potential uses, more than 1,500 faculty (90% of all study participants) and members of academic administration (10% of all study participants) were asked to rank purposes of student course evaluation by importance and popularity (Rathke and Harmon, 2011). The following list shows the main results of this study by highlighting the six most popular purposes of course evaluation in descending order: Table 1: List of potential goals of course evaluation in order of importance and popularity and as perceived by both faculty and academic administration (modified from (Rathke and Harmon, 2011)). Relevance Ranking by Faculty 1. Instructor feedback for teaching improvement 2. Evaluation of class quality as a means for course improvement 3. Collection of information in support of program/institutional accreditation 4. Evaluation of faculty performance in the context of promotion and tenure procedures 5. Assessing student learning outcomes 6. Collection of information for program review Relevance Ranking by Academic Administration 1. Instructor feedback for teaching improvement 2. Evaluation of class quality as a means for course improvement 3. Collection of information in support of program/institutional accreditation 4. Assessing student learning outcomes 5. Evaluation of faculty performance in the context of promotion and tenure procedures 6. Collection of information for program review Page 2 of 15 Remarkably, both faculty and administration ranked teaching improvement and course improvement as the most relevant and popular goals of course evaluation. In comparison, evaluation of faculty performance appears low on the priority list. Table 1 can serve as a basis for a discussion on the main goals of course evaluation at UAF. 2.2 Potential Analysis Methods of Course Evaluation Results In addition to the main goal of a course evaluation system, it is equally relevant to discuss how the produced course evaluation data should be analyzed, as not all evaluation systems support all potential evaluation methods equally well. Ways of analyzing course evaluations may include: Comparison of performance measures over time Benchmark comparison against peer groups Comparisons among units Comparisons among faculty Benchmark comparisons against national norms Benchmark comparisons against institutions using the same evaluation system 2.3 Current Course Evaluation Practices at UAF Currently, UAF is using a paper-based evaluation process for large parts of the student population and adds an online evaluation option for students that participate in online and distance classes. Both the paper and online system are administered by IASystems, an instructional assessment system that has been developed over many years at the University of Washington. Two types of forms are provided to students to gather feedback: (1) a questionnaire that asks to rank properties of the class and the instructor on a six point scale ( Excellent to Very poor ), and (2) a comment sheet that allows students to provide open-ended feedback. The questions on both forms are largely focused on assessing faculty performance and comparably little feedback is provided for teaching and class improvement. This focus on assessing faculty performance is further evidenced by references to course evaluation in various UAF guidelines, regulations, and policy documents. Information on course evaluation can be found in (1) The Board of Regents Policy (P Evaluation of Faculty), (2) the UAF Blue Book (Regulations for the Appointment and Evaluation of Faculty, Chapter III Periodic Evaluation of Faculty), (3) as well as the Collective Bargaining Agreements with UA faculty unions (UNAC Collective Bargaining Agreement, Chapter Merit Bonus Factors; UAFT Collective Bargaining Agreement, Chapter 7.1 Merit Pay). The verbiage used in these documents is listed in the following: Board of Regents Policy, Part IV Human Resources, Chapter Faculty Section P Evaluation of Faculty, states in Subsection D (only relevant sections are shown) (University of Alaska Fairbanks Board of Regents, 2006): * * * D. In conducting evaluations pursuant to this chapter, faculty and administrative evaluators may consider, but shall not be limited to, the criteria set out in of this subsection as appropriate to the faculty member's professional obligation. In addition, units may elaborate in writing on these or other criteria that take into account the distinctive nature of the discipline or special university assignment. Criteria may include: * * * Page 3 of 15 2. effectiveness in teaching, demonstrated by such things as: evaluation by peers; reaction of students as determined by surveys and classroom and laboratory observations; development of improved teaching materials and processes; development of new courses; advising of students; assessments of student achievement; and participation in necessary and routine duties that support classroom performance; University of Alaska Fairbanks Regulations for Faculty Appointment and Evaluation (Blue Book), Chapter III Periodic Evaluation of Faculty (University of Alaska Fairbanks, 2006): B. Criteria for Instruction * * * 2. Components of Evaluation: Effectiveness in teaching will be evaluated through information on formal and informal teaching, course and curriculum material, recruiting and advising, training/guiding graduate students, etc., provided by: a. systematic student ratings, i.e. student opinion of instruction summary forms, * * * United Academics (UNAC) Collective Bargaining Agreement, Section Merit Bonus Factors (United Academics, 2013): Recommendations and determinations of merit pay adjustments by the dean/director and provost for exemplary performance shall consider pertinent factors regarding faculty effort, such as the following: * * * quality of student evaluations University of Alaska Federation of Teachers (UAFT) Collective Bargaining Agreement, Section 7.1 Merit Pay (University of Alaska Federation of Teachers (UAFT), 2013): * * * Merit Pay Factors: Recommendations and determinations of merit pay adjustments by the dean/director and provost for exemplary performance shall consider pertinent factors regarding faculty effort, such as the following: * * * quality of student evaluations All of the above information highlights faculty evaluation in the context of promotion and tenure as the main application of course evaluation at UAF. Interestingly, this stands in contrast to the survey results referenced in Section 2.1.1, where instructor feedback for teaching improvement was cited as the most important and popular use of course evaluation. Recommendation: Before discussing changes in course evaluation technology, we recommend to lead a discussion on the desired purpose of course evaluation at UAF and on whether or not a change of course evaluation focus is required. 3 Discussion of Course Evaluation Technology After a good understanding of the intended goals of course evaluation is developed, an appropriate evaluation technology can be selected. To support this step, this section provides some background on the main benefits and drawbacks of various course evaluation technologies. Page 4 of 15 For the purpose of this report, course evaluation technology is divided into two groups: (1) traditional paper-based evaluations and (2) electronic evaluation systems. Due to recent technological advances, more and more institutions are moving to administer course evaluations online, forgoing the traditional paper-and-pencil methods (Adams and Umbach, 2012). In the following, the pros and cons of electronic course evaluation technology are analyzed. 3.1 Pros and Cons of Electronic Course Evaluation (ECE) Concepts A growing body of literature has studied the benefits and drawbacks of changing from paper-based to electronic course evaluation systems (Adams and Umbach, 2012; Anderson et al., 2005; Lieberman et al., 2001; McCracken and Kelly, 2011). Table 2 provides a list of the main benefits and drawbacks of ECE systems that was created based on a review of pertinent literature and based on our own experience from conducting an extensive survey of existing ECE technology (See Section 4). Table 2: List of the main benefits and drawbacks of electronic course evaluation systems relative to traditional paper-and-pencil options. Pros of Electronic Course Evaluation Concepts Flexibility of evaluation design Easy creation of course-specific questions Flexibility of access Flexibility of evaluation period (both length of period and timing) Improved anonymity (especially for written comments) Cost reduction likely Equal course evaluation of classroom-type and online classes Faster production of evaluation results Wide range of statistical analyses possible if data is digital Potentially easier production of teaching portfolios Cons of Electronic Course Evaluation Systems Potentially lower response rates The student population that is evaluating the class may change The course evaluation process becomes less visible as the visual cues (paper forms, envelopes, pencils, ) go away Not all students may have access to the internet on a regular basis Students have to be proactive to visit the web location and complete the form Electronic systems may increase the number of irrelevant and inappropriate comments While the benefits of electronic course evaluations outnumber their drawbacks in Table 2, some of the risks associated with electronic course evaluations can have severe consequences. One of the most discussed drawbacks of electronic means of evaluating courses is a potential drop in response rates that was observed by various universities (Adams and Umbach, 2012). To provide additional background on the relative importance of the benefits and drawbacks listed in Table 2, Section 3.2 provides an assessment of the main parameters that determine the success of a course evaluation system. 3.2 Parameters for Assessing the Success of a Course Evaluation Technique Depending on the identified purpose of course evaluation at UAF, a number of parameters can be identified that indicate whether
Recommended
View more...
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks
SAVE OUR EARTH

We need your sign to support Project to invent "SMART AND CONTROLLABLE REFLECTIVE BALLOONS" to cover the Sun and Save Our Earth.

More details...

Sign Now!

We are very appreciated for your Prompt Action!

x