| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
TEACHING WITH TECHNOLOGY
Department of Physiology and Neuroscience, St. George's University, Grenada, West Indies
Address for reprint requests and other correspondence: J. Kibble, Memorial Univ. of Newfoundland, Div. of Biomedical Sciences, Faculty of Medicine, The Health Sciences Center, St. John's, NL, Canada A1B 3V6 (e-mail: jonathan.kibble{at}med.mun.ca)
Abstract
Online quizzes were introduced into a large Medical Physiology class to provide students with formative assessment before midterm and final summative examinations. Use of unsupervised online quizzes was chosen to provide a flexible supplementary learning tool for students without overwhelming a small faculty. Several quiz models were applied, which varied in the availability of course credit points for participation and performance. The aims of the study were to investigate if participation in formative assessment was associated with improved course outcomes, if offering incentives for completing quizzes affected student participation, and if quiz performance was predictive of summative examination outcomes. Results showed that students who elected to use online quizzes performed better in summative examinations. Offering course credit of between 0.5% and 2% per quiz increased student participation. However, evidence was found for widespread inappropriate use of unsupervised online quizzes when incentives for participation were applied. Predictive validity of online quizzes could be demonstrated when comparing the first of several quiz attempts with subsequent summative examination scores.
Key words: web based
FORMATIVE ASSESSMENT is intended to foster learning, helping students to develop under conditions that are nonthreatening. Providing feedback is an important element of formative assessment, and this should ideally include guidance about how to improve performance (17). Formative assessment should be available early in the learning process for feedback to be most effective (2). Well-designed formative assessment tools should be informative for students in answering the question "How am I doing?" (14). Summative assessment requires students to demonstrate the sum of their learning in terms of knowledge, skills, and attitudes. It is generally a formal and mandatory assessment given at the end of a period of instruction. From a student perspective, summative assessment answers the question "How did I do?" (14).
The concept of formative assessment is firmly established in higher education, generally, and also in medical education, specifically. For example, the United States Licensing Committee on Medical Education (LCME), standard ED-26, states that "The directors of all courses and clerkships must design and implement a system of formative and summative assessment evaluation of student achievement in each course and clerkship." Improved learning outcomes with formative assessment have been demonstrated, especially when accompanied by individual feedback and remedial action (16). Other studies (3, 19) have shown that the use of formative assessment before summative assessment increases grade points. On the other hand, Peat and Franklin (12) found no relationship between the outcome of formative quizzes and the subsequent performance on summative assessments for inexperienced biology students, although a positive effect of formative assessment was established later in their program (13). This emphasizes the importance of auditing use and effectiveness of formative assessment tools.
The Medical Physiology course at St. George's University serves a heterogeneous student cohort in large classes of 
350 students. In recent years, we have introduced formative assessment in the form of unsupervised online quizzes to meet the general trend of increasing student expectations for support. Online quizzes were adopted as a tool that could provide flexible student support without overwhelming a small faculty. Several different presentations of formative online quizzes were used over a period of five semesters, which varied in the amount of course credit available and in the criteria used to award credit. A retrospective analysis of the data was undertaken to address several hypotheses: 1) use of formative assessment is associated with improved outcomes on summative assessment, 2) providing incentives for unsupervised online quizzes increases student participation, and 3) unsupervised online quizzes have predictive validity for summative examinations. In addition to addressing these hypotheses, the study revealed unexpected student behaviors in response to providing credit for participation in quizzes, which may be meaningful to others intending to adopt this type of formative assessment.
METHODS
Background
St. George's University offers a 4-yr medical program in which basic medical sciences are taught during the first 2 yr. Medical Physiology is taught twice a year to cohorts of 350 students. It is a five-credit hour course, delivered over 18 consecutive weeks, consisting of 75 h of lecture and 12 h of formal small-group teaching. Curricular content is based on the American Physiology Society Medical Physiology Objectives Project. The course runs during the second academic semester, concurrently with Neuroscience, Genetics, Immunology, and Medical Parasitology courses. The Medical Physiology course follows the Anatomy and Biochemistry course and precedes courses that include Pathology, Pharmacology, and Pathophysiology.
Learning Management System
A campus-wide learning management system (ANGEL) was introduced at St. George's University in late 2004, and faculty members were encouraged to develop online tools to support learning by the Curriculum Committee. Our use of online quizzes stemmed from this initiative together with the consistent request by students to provide more formative assessment in their course evaluation. All quizzes were administered through ANGEL, which students accessed using their personal username and password. The grade book feature within ANGEL was used to record quiz use.
Online Quiz Presentation
Medical Physiology students were assessed using the aggregate score from a summative examination after 9 wk and another after 18 wk. Each summative examination consisted of 80 multiple-choice questions. Two online quizzes were offered during the learning period before each summative examination. The two quizzes offered in each block were presented sequentially, separated by 2 wk. The second quiz in a pair included new course material to provide ongoing feedback as the teaching progressed. Each online quiz consisted of 20–30 multiple-choice questions designed to match the style and difficulty of summative examinations. Students were made aware of quizzes in the course syllabus and by milestones posted in their personal calendar within ANGEL. Different quiz presentations were applied to five student cohorts between 2004 and 2006, in which there were variations in the amount of course credit available, in the ways of earning credit, and in the length of time quizzes were open to students. These variations reflected ideas of faculty members aiming to encourage student participation and to help students pace their studies. Each model described below was applied once throughout a single semester (Table 1). A given cohort of students was held to the same online quiz policy throughout their semester. Feedback pertaining to quiz items and suggestions about remedial action were available through face-to-face meetings with faculty members on student request.
|
Model 2.
In model 2, total course credit available for each online quiz was set at 0.5%. Credit was awarded for taking a quiz, irrespective of the score obtained. Students could take a given quiz an unlimited number of times leading up to the summative examination. The database was closed immediately before the summative examination.
Model 3.
In model 3, total course credit available for each online quiz was set at 1.0%. Credit was awarded on an all-or-none basis to students scoring 30% or more on a given quiz. A 1-wk window was provided to complete a given quiz for credit, and unlimited attempts were allowed during that week. The database was closed at the end of the quiz week, and accumulated data were downloaded for later analysis. The quiz was then again made available for unlimited review until the summative examination.
Model 4.
In model 4, total course credit available for each online quiz was set at 1.0%. Credit was allotted as a function of the actual quiz score attained from the better of two attempts. After the first attempt, students were provided with a list of questions answered incorrectly along with their original responses. Students were encouraged to review the questions before completing the quiz a second time. Students had a 1-wk window to complete a given quiz for credit, after which time the database was closed and accumulated data were downloaded. Each quiz was then made available for unlimited review until the summative examination.
Model 5.
In model 5, the same design as for model 4 was used except that the total course credit available for each quiz was set at 2%.
Data Analysis
Data analysis was conducted retrospectively for all semesters at the end of the Fall 2006 semester. Approval was granted by the St. George's University Institutional Ethics Review Board. A sampling approach was used to manage data analysis because of the large volume of data produced. Within each model, one pair of quizzes and their associated summative examination were selected at random for analysis. Comparison of quizzes within any given pair showed consistent patterns of student responses. There were no significant differences between mean quiz scores within any given pair of quizzes. For this reason, pooled data from quiz pairs were used for presentation.
On first inspection, quiz performance appeared different for students who took an online quiz only once compared with students who accessed a quiz several times. These data are presented separately for models 2–5, in which quizzes could be taken more than once. Student performance on each quiz or examination is reported as a percentage, and data are expressed as means ± SE. Statistical comparisons between mean online quiz scores and summative examinations within each semester were made using ANOVA followed by post hoc t-tests with Bonferroni correction and 5% significance level. Use of alphanumeric student codes allowed anonymous correlation between individual online quiz scores and summative examination scores.
Student and Faculty Member Surveys
The student survey shown in Table 2 was presented to student cohorts in models 4 and 5 at the end of their course. The survey was available online, and the student response was both voluntary and anonymous. The purpose of the student survey was to gauge how useful and acceptable students perceived online quizzes to be. The faculty member survey shown in Table 3 was e-mailed to 120 full-time faculty members involved in teaching in the medical curriculum to gain insight into their views on the purpose and implementation of online quizzes across the wider curriculum. Four-point Likert Scales were used to omit a neutral response and encourage a clear response to questions (7).
|
|
Student Participation Rates
Table 1 shows the percentage of students who participated in online quizzes for each model. Higher student participation was observed with increasing credit value of quizzes.
Models
Model 1.
The mean online quiz score in model 1 did not differ significantly from the mean summative scores. However, individual quiz scores showed a significant positive correlation with corresponding summative examination scores, and students who elected not to use the online quizzes performed significantly worse on the associated summative examination (75.7 ± 0.6%, n = 167 students) than those who used at least one (80.2 ± 0.5%, n = 179 students). Of the students taking online quizzes, 160 students used both quizzes and 19 students only one of the pair. There were no differences in the summative examination scores between students taking both online quizzes (80.3 ± 0.5%, n = 160) and those taking only one quiz (79.4 ± 0.7, n = 19). Data from these two groups were pooled for presentation and are shown in Fig. 1.
|
Model 2.
The mean quiz score for students using only one attempt (82.9 ± 1.1%, n = 356 quiz scores) was significantly greater than the first effort of students who subsequently repeated a quiz (72.2 ± 1.6%, n = 284 quiz scores; Fig. 2). There were no significant differences in summative examination scores for students who used a quiz only once (80.2 ± 0.5%, n = 356 scores) compared with students who repeated an online quiz (81.6 ± 0.6%, n = 284 scores). Individual scores from the first of several quiz attempts showed a significant positive correlation with corresponding summative examination scores. The mean summative examination score of the 49 students who did not post any online quiz scores was significantly lower (76.3 ± 0.5%, n = 49 students) than the average for students taking at least one quiz (80.8 ± 0.5%, n = 333 students).
|
|
|
|
Faculty member survey.
Toward the end of this study, a number of other disciplines within the basic medical sciences curriculum were starting to use online quizzes. Forty-four full-time faculty members responded. More than 90% of respondents agreed that online quizzes would be useful in helping students keep up with studies. Ninety percent of respondents agreed that online quizzes help students learn and that they provide useful feedback about student mastery of learning objectives. More than 90% also recognized the likelihood that awarding course credit would increase student participation. The faculty member response was divided as to whether unsupervised online quizzes could adequately replace supervised offline quizzes. Two-thirds of faculty respondents felt that online quizzes should be compulsory for students.
DISCUSSION
Summary of Main Findings and Limitations of the Data
The main outcomes of this study were as follows: 1) students who used formative assessment generally performed better on summative examinations, 2) providing incentives for online quizzes increased student participation, and 3) predictive validity of online quizzes was demonstrated. An unexpected but important finding was evidence for inappropriate use of online quizzes when incentives were applied. I use the term "inappropriate use" to refer to the significant number students who scored 100% on quizzes in a single attempt but who did not subsequently perform well on summative examinations. Their strategy in using quizzes appeared to defeat the formative purpose for which quizzes were intended. This pattern of behavior was not defined as cheating because no rules were defined in terms of strategies that students were allowed to use when taking unsupervised online quizzes. Data were analyzed anonymously, presenting limitations in terms of studying effects of gender or other cohort characteristics. Faculty member and student survey response rates were lower than desired, at 37% and 55%, respectively, presenting limitations in terms of true representation of the populations sampled.
Aim 1: use of formative quizzes is associated with better outcomes.
The first study hypothesis considered whether use of formative online quizzes was associated with better outcomes on summative assessment. Data from quizzes in model 1 showed a significant difference between students who participated in quizzes and those who did not. These data were striking because the class distributed itself evenly between participators and nonparticipators. The same effect was repeated for models 2 and 3, where 49 of 382 students and 29 of 358 students, respectively, elected not to participate. No differences were noted in models 4 and 5 with regard to the effect of participation on summative examination outcomes. However, models 4 and 5 had 97% and 98% participation, respectively, so the remaining group of students who elected not to participate was very small for comparison.
There are several examples in educational research literature showing that students who make use of all learning activities have a higher likelihood of successful outcomes (4, 6). In the present study, it should be noted that "nonparticipation" was defined as not recording responses when quiz windows were open. Students were able to access quiz materials again and may have used them at some later time. Lack of compliance may therefore be a more important issue. Understanding why some students behave in this way is important because this group performed worse on summative examinations. Others have noted a relationship between lack of compliance and poor examination outcomes in medical students (18), but this seems too simplistic a view. In our Medical Physiology course, all assessments are computer-graded, multiple-choice questions designed to reflect the United States Medical Licensing Examination Step-1. It is been reported that students with a learning style that shows a strong preference for abstract conceptualization perform better on this kind of assessment (8). Our uniform offering of such assessment may discourage use among students with different learning styles. Similarly, students who adopt a surface learning approach are less conscious of monitoring and managing studying and generally have lower examination performance in medical school (9). Peat and Franklin (12) studied the effectiveness of formative assessment in a group of first-year biology students during their first semester and found no demonstrable benefit. However, a second study, performed with a different cohort, reported that use of self-assessment modules had a significant positive impact on performance for students by the end of their second semester (13). This suggests that benefits of self-assessment become apparent as students learn how to work more independently. In the present study, it seems likely that the nonparticipating groups reflect a population who have not yet optimized their approaches to studying.
Aim 2: student participation is increased by offering incentives.
The second hypothesis addressed in the present study examined whether student participation increases when reward in the form of course credit is available. The study was able to partially address this question because a range of quiz models were presented with different credit values, ranging from 0% to 2% of final course grade per quiz. Quizzes in model 1, for which no credit was available, had a disappointing participation of only 52%, although it should be acknowledged that student motivation to participate at the requested time may be reduced by allowing access to quizzes again later. Student participation rates rose dramatically to 87% in model 2 and to 98% in model 5. These data confirm student survey responses in which >90% of respondents indicated that they completed quizzes to earn available credit. The trend of increasing participation may have been reinforced by criteria that made it progressively more difficult to earn full credit for quizzes. In model 2 simply taking a quiz was sufficient, whereas in models 4 and 5 students needed to score 100% on a quiz within two attempts to earn all available credit. It may be that students viewed earning these course points as an insurance policy in case they were close to a grade borderline.
Offering course credits for online quizzes has the apparent advantage of increasing participation. Given the observation that nonparticipation is associated with lower summative examination scores, increasing participation with a reward might be expected to produce generalized improvements in student outcomes. However, widespread inappropriate use of online quizzes was an unexpected side effect of offering rewards. This was most apparent in models 4 and 5, where students were expected to take a quiz once for formative assessment and then use the feedback given to review weak areas and correct their mistakes in a second attempt. Contrary to faculty expectations, large numbers of students took quizzes only once and were able to score close to 100%. These students did not sustain a high level of performance on summative examinations and even had worse performance than their classmates in the cases of models 3 and 5. For this group of students, any formative value of the quizzes was likely to be lost, defeating the faculty's purpose in providing them. It was of particular concern that students scoring significantly lower in summative examinations appeared more likely to be in the group using quizzes inappropriately (models 3 and 5). Their lower summative scores may reflect a missed learning opportunity from not using quizzes as intended by the faculty, or, perhaps more likely, it reflected students who perceived the need to insure points were scored above other considerations.
Outcome data in the present study suggest that strategies used by students to insure quiz points are scored, but which circumvent learning, are unwise. Are such strategies unethical? A great deal has been written recently, both in the popular press (1) and educational literature (5, 10, 15), about the rising severity of unethical approaches by university students to completing their homework and examinations. Student perceptions of what constitutes "cheating" probably vary significantly from those of faculty members. In a study of engineering undergraduates across 11 institutions (11), it was reported that 41% of students regarded working in groups on web-based quizzes as cheating, 28% considered it unethical but not cheating, and 30% thought it was neither unethical nor cheating. Similarly, only 27% of students regarded asking another student about questions on an exam not yet taken as cheating, 45% felt it was unethical but not cheating, and 27% regarded it as neither unethical nor cheating. In the present study, students were encouraged to take quizzes without assistance and then to review their results using course materials or a discussion with peers and faculty members. The survey given to cohorts in models 4 and 5, where there was evidence of inappropriate quiz use from outcomes data, indicated that use of lecture notes and discussion with peers were widely used to assist in taking quizzes. My intention in utilizing quizzes was to provide students with an indicator of their current mastery and then use such tools to further enhance their performance. I did not give students explicit rules about how the unsupervised quizzes were to be used and what was acceptable or otherwise. I do not, therefore, regard the patterns of quiz use described as unethical or cheating. However, in retrospect, it does seem important to agree with students what is acceptable and to document this before assigning course credit to any form of unsupervised homework.
Aim 3: online quizzes show predictive validity for summative examinations.
Outcome data from a first quiz attempt for students who made a repeat effort was used to address this hypothesis. It was assumed that these students were likely to have used the quizzes as intended and that their first attempt represented the current state of their knowledge. In models 1–4, a consistent pattern was observed in which formative assessment scores were weakly but significantly correlated with summative examination scores. This suggested that the quizzes themselves were constructed from items that tested the same knowledge and understanding as summative examinations.
In summary, students who chose to use formative online quizzes generally had better outcomes on summative examinations. In the absence of course credit, the quiz participation rate was low. Offering course credit and introducing performance criteria to earn credit increased student participation. However, with rising quiz value and performance requirements to earn credit, the incidence of inappropriate quiz use increased. Formative online quizzes had predictive validity for subsequent summative examinations, indicating that quizzes have the potential to be useful learning aids to help students perform better on course assessments. Future work will focus on how to encourage student participation, on understanding reasons for nonparticipation, and on the effect of agreeing ethical standards for quiz use with students.
Received for publication April 19, 2007. Accepted for publication May 9, 2007.
REFERENCES
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
| Visit Other APS Journals Online |
