| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
HOW WE TEACH
1 Educational Research Unit, School of Biomedical Sciences, The University of Queensland, Brisbane, Queensland, Australia
2 School of Education, The University of Queensland, Brisbane, Queensland, Australia
Address for reprint requests and other correspondence: R. W. Moni, Educational Research Unit, School of Biomedical Sciences, The Univ. of Queensland, Brisbane 4072, Queensland, Australia (e-mail: r.moni{at}uq.edu.au)
 |
Abstract |
|---|
 TOP Abstract IntroductionMETHODSRESULTSDISCUSSIONREFERENCES |
|---|
Key words: assessment criteria; concept mapping; dentistry; group work; observations
|
Introduction |
|---|
|
TOPAbstractIntroductionMETHODSRESULTSDISCUSSIONREFERENCES |
|---|
We have previously reported the first phase of an action learning project aimed at improving the quality of physiology assessment in second-year dentistry students at The University of Queensland (14). Given that the appropriate design of rubrics (18), group work (2), and use of concept maps (17) can be effective approaches to learning and assessment, we undertook a study to develop an assessment rubric for group-based concept maps in physiology. We reported how an assessment rubric could be constructed to grade the application of physiology concepts around a patient scenario. The rubric described set academic standards around three criteria. These were 1) content (cardiovascular, respiratory, and renal physiology); 2) logic and understanding (of concepts); and 3) presentation (of the concept map). The intention was to make faculty expectations more explicit and clear. However, this first phase of study reported by Moni et al. (14) focused largely on faculty perceptions and their actions aimed at designing and constructing the rubric. The present study documented how the opinions of second-year dentistry students and faculty members can be integrated to improve the quality of assessment.
In the second phase of the study reported here, we shifted focus to how students used the rubric in completing the assessment task. This task was an open assessment task over 1 wk in which student-to-student and student-to-faculty interactions were encouraged (5). We undertook a small exploratory case study to better understand how students completed their concept maps.
Concept maps are graphical representations of the conceptual, relational, and hierarchical nature of knowledge (1). The process of constructing concept maps entails critical and analytic thinking (12). Learners develop and demonstrate their knowledge through the progressive differentiation of ideas organized as the propositional structures that link concepts. There is evidence that concept mapping stimulates meaningful learning of physiological concepts in undergraduate problem-based learning (PBL) settings (e.g., Ref. 16). In addition, concept mapping can be motivating, although attitudes toward using concept maps reflect noncognitive factors, notably perceived workload (19) and approaches to learning (12). Clearly, these perceptions can be influenced by contextual factors, e.g., whether mapping was assessed and, if so, how it was assessed.
However, there is limited evidence in the research literature about how student project groups work on collaborative tasks in university settings (7). Thus, we recognized that while the product of student's work (i.e., the completed concept maps) could be graded using the rubric, the process of conceptualizing, planning, drafting, and editing the maps also needed to be investigated. This would provide insight as to how students conceptualized and operationalized the rubric. We therefore investigated how some students worked in their groups to complete concept maps.
We now report how students in the 2005 dentistry cohort perceived the concept mapping task, how they perceived the assessment rubric, and how two groups of students worked as teams to complete their concept maps in physiology.
|
METHODS |
|---|
|
TOPAbstractIntroductionMETHODSRESULTSDISCUSSIONREFERENCES |
|---|
Key Research Questions
Three research questions guided the data collection.
Question one.
What are the opinions of students about the concept mapping task in physiology?
Question two.
What are the opinions of students about the assessment rubric for the concept mapping task?
Question three.
How do groups use the criteria to complete the concept mapping task?
The Course, Students, and Concept Mapping Task
Details of the course (Foundation Biological Sciences for Dentistry, DENT2012), the academic background of second-year dentistry students at The University of Queensland, and the concept mapping task are provided in the previous study by Moni et al. (14).
In 2005, 61 students with a grade point average (GPA) of at least 5.9 (from a maximum of 7) were enrolled in the course.
The concept mapping task was the summative assessment task for the cardiovascular, respiratory, and renal (CRU) unit of work delivered in lecture format over 4 wk. The key concepts included control of blood pressure, hypertension, atherosclerosis, thrombosis, heart failure, and, as a minor component, the action of appropriate drugs. In course modules prior to this unit, students learned how to construct simple concept maps as part of two formative PBL sessions on "pain" and "handling a patient on methadone treatment."
A 50-min workshop was run by the CRU module tutors to review concept mapping, previously introduced in the PBL sessions, and to discuss the marking scheme. Students reviewed the basics of how to construct a concept map using an example based on content unrelated to the assessment task and were then taught how to use the assessment rubric. They were subsequently presented with a written, clinical scenario of an unfit, middle-aged male patient with a preexisting gastric ulcer presenting with hemorrhagic complications following a tooth extraction. Broad guidelines on the inclusion of major physiological pathways and drug treatments were given.
Self-selected groups experience increased group satisfaction and better marks regardless of their previous grades (6, 13). Thus, students selected their groups of four and then organized themselves as they wished during the following week to collect relevant resources for their concept maps. At the end of that week, all groups attended a 3-h workshop, during which they completed their hand-drawn concept maps on size A3 large-format paper and submitted them for grading. Two tutors facilitated student learning by responding to questions and, where deemed necessary, provided guidance and feedback on the "open book" assessment task. For this complex assessment task, all tutors (including the authors) consulted extensively before, during, and after the assessment task to ensure that all students received similar levels of support throughout. Tutors later graded all the concept maps.
Grading of Concept Maps
Before the final workshop, the two tutors met for 2 h to 1) design a model answer and 2) decide how the alphabetic grades from the rubric would be converted to numbers. This was necessary because the university requires number grades for the calculation of exit grades and reporting. It was decided that an "A" on the rubric would be 10 of 10 marks, an "A–" would be 9.5, a "B+" would be 9.0, a "B" would be 8.5, etc. Students were informed of these decisions at the start of the final workshop. On submission, each tutor independently graded all group concept maps and pencilled in justifications for their grading. These preliminary grades were subsequently moderated to generate the final grades [moderation is the formal process by which faculty members negotiate consistency and agreement when grading using a rubric (14)]. Final grades and written feedback were then provided for each group.
Student Surveys
Surveys was designed to capture the opinions of all second-year dentistry students (n = 61) regarding 1) the concept mapping assessment task (5 items) and 2) the assessment rubric (6 items). These were both administered immediately after the concept maps had been submitted. For both surveys, respondents were asked to complete Likert rating scales from 1 to 5, where 1 = very strongly disagree, 2 = strongly disagree, 3 = neither agree nor disagree, 4 = strongly agree, and 5 = very strongly agree.
Before completing the surveys, students were informed about the purposes of the study and advised that their participation was voluntary and anonymous and that their grades could not be affected by participating or not participating in the research project. The only institutional requirement was that consent was obtained from volunteers because the study was considered part of the ongoing enhancement of teaching and learning activities. All students volunteered to record their alphabetically coded group (groups A–N) on their survey form. Using the codes identifying each group, it was possible to calculate the average Likert scale value for each survey question for each group. These could then be correlated with the final course grades of each group.
The medians and interquartile ranges of opinions from the survey responses were calculated using GraphPad Prism (version 4). A nonparametric Wilcoxon signed-rank test was used to estimate whether the medians of each item of the surveys were different from the midpoint (hypothetical median = 3) of the 5-point scale. Significance was represented by two-tailed P values. This statistical test is appropriate for sets of ordinal data with skewed distributions (10).
The internal consistency of all items on each survey was measured using Cronbach's 
-coefficient using SPSS version 10.0 (Macintosh). An -value of 0.8 or higher indicates that the survey items reliably measure related themes or constructs.
Digital Videorecordings of Students Completing Group Concept Maps
Two groups volunteered to have their group work videotaped and analyzed during a session that replaced their 3-h workshop. Group membership was determined solely by the students. Group A consisted of four males, and group B consisted of four females. These volunteers had GPAs from 6.0 to 6.5. Each group worked in separate, adjacent videorecording studios set up with tables and chairs to resemble typical teaching rooms.
Both groups were initially given a brief orientation by one of the authors [R. W. Moni (RWM)] who was a course tutor. This tutor then moved between both groups (
23–24% of the time) but was not present with either of the two groups for 75% of the workshop time. Digital recordings were made for each group using two ceiling-mounted cameras and directional microphones for the whole duration of work, i.e., 4 h. It should be noted that although the session was scheduled for 3 h, the students continued to work after the formal time had elapsed. We decided to continue so that the complete process of mapping was recorded. All 15 groups worked beyond the 3-h block.
Analysis of Videorecording Sessions
Four hours of audiovisual tape were recorded for each of the two observation groups. Each tape was then analyzed in three stages (3). These were opening, coding, and analyzing.
Opening.
The complete videorecordings were separated into time segments based on group membership, i.e., the number of people (students and tutor) in the room at any time. This is appropriate because the effectiveness of learning in groups depends on group membership and the quality of interactions (15).
Coding.
This entailed identifying, for each time segment, how much time each group spent on the three criteria of content, logic and understanding, and/or presentation. The times for each segment duration were recorded for each criterion. This constituted the "time on task" related to each criterion. The times during which the students were not focused on work related to the concept map constituted "time off task." For group A (male), in 37 of 44 segments (84%) all of the group were present. In the remaining seven segments, only one group member was missing, and the rest of the group continued to work in their absence. For group B (female), all group members were present and working for 30 of 38 segments (79%). In the other eight segments where one member was missing, the remaining students continued to work. On this basis, the authors decided that coding by group was appropriate.
The coding of the segments was based on language use and associated behaviors. For example, students sometimes explicitly referred to content (facts, concepts, or procedures) or their conversations were about how content was linked (propositional statements about causality or noncausal relationships and questions seeking to clarify or statements demonstrating their understanding) or their talk and actions focused on issues of presentation (the location of concept boxes, ensuring arrows did not cross, and use of color). Table 1 shows samples of each of these taken from partial transcripts.
|
Analyzing.
For each time segment, the group membership, time on task for each criterion, and cumulative time on task for each criterion were tabulated. The cumulative time on task for each criterion was then expressed as a percentage of the total time on task. Broad trends were then identified.
|
RESULTS |
|---|
|
TOPAbstractIntroductionMETHODSRESULTSDISCUSSIONREFERENCES |
|---|
Student Opinions Regarding the Concept Mapping Assessment Task
Fifty-six students (92%) responded to the survey. In general, the most common response was "neither agreeing not disagreeing" to the five specific issues around the task: the link between the task and learning, liking the task, prior use of mapping in problem-based learning sessions, and perceived usefulness of the task in learning (Fig. 1). However, the large interquartile ranges indicated a range of opinions about the mapping task. Overall, students (including the observed groups A and B) perceived that working in a group of four did assist their learning.
|
3.5) or did not favor (scores 
2.5) the concept mapping task. For four groups, these values ranged from 2.8 to 3.1. Because these approximated the scale midpoint of 3, representing neither agreement nor disagreement, there was no strong group opinion about the survey items. For the remaining 11 groups, there was a strong correlation (r2 = 0.75) between favoring the concept mapping task and their grade on the task (Fig. 2). However, the correlation is best considered as being descriptive rather than predictive. Cronbach's -coefficient for this survey was 0.9799, indicating that a single underlying survey construct was being measured. On this basis, therefore, averaging of these Likert scores was considered valid. The two observed groups (groups A and B) clearly favored the mapping task.
|
-coefficient was 0.9824, again indicating very high reliability of survey items in measuring opinions around the rubric.
|
|
The following were noted from observing both groups A and B. First, students worked on task for 93% of the 4-h observation period. Second, they showed evidence of considerable preparation before the observed workshop, having prepared preliminary concept maps and taken notes from texts and reference books on physiology. Both groups claimed that they had already spent many hours of study for this assessment task before the final workshop; notably, group A reported they had spent 12 h in preparation. This was three times longer than the preparation time reported by group B. This is evidenced from the transcript excerpts below (student names are pseudonyms):
Group A.
Segment 43 at 3:12:00. Four students plus RWM present.
Time: 3:12:01
RWM: Have you people been working on this all week?...
Simon: About 14 hours on average.
Kim: We did 4 on Tuesday, we did 6 yesterday.
Ben: And then today is 4.
Simon: I did 2 on the weekend.
Kim: Yeah and then on the weekend I did 2, then 1 on the day before, so what's that? Probably 12 on average each.
Group B.
Segment 37 at 3:18:25. Four students plus RWM present.
Time: 3:20:20
RWM: How many hours have you have spent [on this task]?...
Alice: Each of us individually about 4 hours each.
RWM: So 4 hours, all day Saturday.
Judith: And probably yeah, about half of Saturday.
Alice: Yeah and I spent about 2 hours.
Emma: Yeah and I spent about 6 hours on my own.
Alice: I would have probably spent about 8 because I did all of the rough drafts and pulled them all together.
Third, students spent a greater percentage of their time on presentation than either logic and understanding or content (Table 3). However, group A spent almost twice as much time on logic and understanding than group B. Fourth, the overall time spent on content was relatively short for each group. Fifth, reference to the assessment rubrics were made in 20% of the time segments. Finally, students were frequently challenged by attempting to find the "right" verb to connect two concepts, e.g., on one occasion, there were 16 words proposed before the final choice was made.
|
|
|
DISCUSSION |
|---|
|
TOPAbstractIntroductionMETHODSRESULTSDISCUSSIONREFERENCES |
|---|
What are the Opinions of Students About the Concept Mapping Task in Physiology?
The teaching faculty members involved in this module considered this task to be potentially valuable to students, especially in fostering high-order reasoning skills and team work required for professional dental practice. The task was clearly academically demanding, as evident from the time allocated by some groups prior to the workshop and their struggle with language in finding the "right" words in their maps. The average grade (7.20 of 10) was high. This was lower than that of the previous year's cohort, which had the same two tutors. This may reflect the lower GPAs of students in the 2005 cohort.
Motivation appeared to have a strong influence on academic achievement. However, those students who did favor the task achieved higher grades for their maps. The correlation between favoring the assessment task and final grades was clearly evident (Fig. 2). Most students reported that working in a group helped them learn.
An unexpected finding was that the concept mapping task was not generally more favored by the cohort. In another case study, concept maps were shown to assist nursing students to link theory to practical scenarios encountered in the workplace (7). Similarly, in the present study, the clinical scenario was contextualized to dental practice and designed to motivate students because of its clear relevance. However, while the students in this project found the scenario appealing, they did not enjoy the demands of the process in completing the maps.
It may be that the traditional pen-and-paper block exams were generally preferred over the mapping task. Although concept maps were constructed as part of the PBL workshops throughout other modules of the course, these were relatively simple and brief compared with the summative mapping task. It is likely that these students found the latter stressful. It required much effort, higher-order thinking, and cooperative interactions from students. Dentistry students at our university and others (21) are generally considered to be competitive, high academic achievers. Such students can be prone to competitiveness, excessive industry, and anxiety around the pressures associated with learning and assessment, including time pressure (9). However, it is important to note that the students felt that working in a group helped them to learn. It may be that this perceived benefit outweighed the stress related to completing their maps.
One possible solution from faculty members would be to set a series of mapping tasks (e.g., in the PBL sessions of other units) that more closely reflect the complexity of those set for summative assessment. This could be achieved using the existing timeslots for the PBL workshops. Most students reported that working in a group of four helped their learning (Fig. 1). Although this finding was not further explored, it is likely that students perceived the contributions of others as helping them achieve high grades.
What are the Opinions of Students About the Assessment Rubric for the Concept Mapping Task?
One of challenges to effective teaching is how to ensure that students' perceptions of assessment align with those of faculty members (4). The 2005 students were the first to use the new assessment rubric. The rubric was strongly favored by most students (Fig. 3). The language of the rubric was easy to understand, and the criteria and achievement standards were considered as being appropriate and fair. This was expected, given that much of the language was that used by students from the 2004 cohort. It was encouraging that most students liked the rubric and reported that it helped them to complete the task. Again, because assessment rubrics do not stand alone as learning tools (18), it was not surprising that the students needed guidance beyond the rubric to complete the concept mapping task.
When Completing the Concept Mapping Task, how do Groups Work in Ways That Reflect the Criteria of the Concept Mapping Task?
The two groups (group A: 4 male students and group B: 4 female students) volunteered to be videorecorded as they completed their concept maps. The authors did not analyze the group work from a gender perspective because findings could not be generalized from this small sample (2 of 15 groups). Rather, a case study approach was used to describe in some detail how groups completed their maps framed around the assessment criteria. Broadening the research of group interactions using a larger sample (in the future) may provide more generalizable findings.
The two observed groups were similar, but not typical of the dentistry cohort, in that they favored the concept mapping task (Table 3) and achieved high grades (Fig. 2). How they worked as a group revealed substantial preparation, persistence, and industry in completing their group concept maps.
The analyses of videorecordings clearly demonstrated the cognitive and social complexity of working in a group to complete a complex concept map in physiology. The ebb and flow of group dynamics is graphically captured in Fig. 4. The two observed groups spent more time in the final workshop on presentation of their maps. This probably reflects 1) that they had allocated time during the week to content as well as logic and understanding because they had preliminary maps when they arrived at the workshop and 2) the nature of the task as a form of graphical representation (7).
Different processes of completing concept maps were observed in the two groups. Group A distributed their work across the criteria, whereas the work of group B had more clear stages. In the latter, content as well as logic and understanding were the focus of earlier work during the final workshop, and presentation was the focus of later work as they prepared their maps. Both groups achieved highly; therefore, it was not possible to identify if one process was more likely to lead to high academic success.
The findings from this study clearly underline that, while concept maps provide rich opportunities for learning, faculty members should not ignore the academic challenges required and the amount of time needed, especially for high achieving and competitive groups of students to complete the task. While it is not feasible to assess the complex processes of constructing concept maps in large groups because of the time entailed, this study revealed the importance of allowing students ample time to undertake these processes and allocating sufficient weight to the assessment task.
Future directions might broaden the scope of the study to include larger numbers of groups to explore how diverse groups of students approach complex group concept mapping tasks. Specifically, the relationships among gender, ability, and academic achievement could be investigated. In addition, more information is needed about how different groups of students researched and drafted maps prior to constructing their final complex group map. This could be achieved by sampling students' work in the week prior to the final workshop.
|
Acknowledgments |
|---|
Received for publication April 17, 2007. Accepted for publication January 1, 2008.
|
REFERENCES |
|---|
|
TOPAbstractIntroductionMETHODSRESULTSDISCUSSIONREFERENCES |
|---|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
| Visit Other APS Journals Online |
