Open-ended or closed-ended case study design schemes offer different educational advantages. Anatomy and physiology faculty members who participated in a conference workshop were given an identical case about blood doping and asked to build either an open-ended study or a closed-ended study. The workshop participants created a rich array of case questions. Participant-written learning objectives and case questions were compared, and the questions were examined to determine whether they satisfied criteria for open or closed endedness. Many of the participant-written learning objectives were not well matched with the case questions, and participants had differing success writing suitable case questions. Workshop participants were more successful in creating closed-ended questions than open-ended ones. Eighty-eight percent of the questions produced by participants assigned to write closed-ended questions were considered closed ended, whereas only 43% of the questions produced by participants assigned to write open-ended questions were deemed open ended. Our findings indicate that, despite the fact that instructors of anatomy and physiology recognize the value of open-ended questions, they have greater difficulty in creating them. We conclude that faculty should pay careful attention to learning outcomes as they craft open-ended case questions if they wish to ensure that students are prompted to use and improve their higher-order thinking skills.
- case-based learning
- learning objectives
- convergent questions
- divergent questions
cases are stories with an educational message (7) and have garnered widespread interest among instructors in the biomedical sciences (11). How these stories unfold depends on the ingenuity of the case designer. The best case studies are written to invite students to explore a multitude of intellectual pathways with learning objectives serving as trail markers for the educational journey ahead (2). In the same way that a master storyteller often fashions a tale to have alternate endings, the case author is at liberty to develop case studies in differing directions-narrowing or widening, closing or opening the scope of the study-as pedagogical needs arise.
Previously, we have described some of the essential features that distinguish open-ended from closed-ended case study design elements (2). We noted that open-ended case designs permit multiple solutions [i.e., are divergent (3, 13)] since the knowledge needed to solve the case study may be in flux, the necessary information may not be known or available, or certainty cannot be obtained. Alternately, closed-ended designs usually have a single solution [i.e., are convergent (3, 13)] since the knowledge required is well defined and the information needed is readily accessible. We concluded that, depending on the goal for student learning, either type of case approach can yield outcomes that are beneficial for student understanding. Underlying these contentions was the assumption that, starting from the same scenario, case studies can be profitably unfolded along either open- or closed-ended avenues.
While our suggestions generate a helpful framework for choosing between open- and closed-ended design schemes, we realized that our thesis would gain greater definition if we demonstrated the construction of actual case studies illustrating the salient features of both types of architectures. Even though we could have produced such case studies on our own, we hypothesized that by helping other faculty members understand the features of open- and closed-ended case studies, we could enable them to generate useful exemplars. So, we asked a group of instructors of anatomy and physiology to fashion open- or closed-ended components of a single case study as they participated in a workshop that highlighted the features and benefits of both open and closed case design. By doing so, we saw the potential to obtain a much richer tapestry of case elements than we could have invented ourselves, and we recognized that we might also be able to uncover some of the adequacies and inadequacies of faculty members as they approach the design process. Thus, we saw the value in carefully examining and reporting the efforts of workshop participants to develop open- and closed-ended features of a single case.
Here, we describe on our efforts to 1) solicit from workshop participants their thoughts on the advantages of closed- and open-ended designs and 2) determine how participants would build a case study following either an open-ended approach or a closed-ended approach to case design when starting from an identical case scenario. We chose a short story about blood doping by an Olympic athlete that offered multiple avenues of inquiry, presented a dilemma to be addressed, and was replete with the social, behavioral, and ethical implications of physiology that Penny Hansen has described as a component of the discipline's recondite curriculum (6). Given this scenario, participants were asked to begin developing a case study by creating the design elements, i.e., learning objectives and corresponding case questions, which would guide and stimulate student learning. By examining the products of the participant's work, we intended to examine and analyze the design schemes they used as they created open- or closed-ended approaches to case studies.
Two workshops, entitled “Open-Ended and Closed-Ended Approaches to the Design of Case Studies,” were offered as part of the program of the 2002 national meeting of the Human Anatomy and Physiology Society. The workshop participants consisted of ∼50 anatomy and physiology educators who were interested in the use of case studies. Even though we did not gather background information on the participants, we believe that they represented a typical cross section of Human Anatomy and Physiology Society attendees. Most taught at the level of undergraduate anatomy and physiology and appeared to have a breadth and depth of teaching experience.
The authors provided an introduction to the use of case studies as well as explained the attributes of open- and closed-ended design structures by analogy to the features of ill-structured and well-structured problems (10). The dilemma used to illustrate the difference between ill-structured and well-structured problems involved an everyday example of transporting a child to school. The outcome was stated in terms of an objective: “The child must arrive at school at an appointed time.” The question “What is the best means for getting the child to school?” was given as an example of an ill-structured problem since the optimal solution depends on judgments made about the different means and accessibility of transportation, the availability of drivers, the familiarity of the routes, the occurrence of traffic delays, scheduling conflicts, etc. There could be multiple acceptable solutions to this question. On the other hand, the question “What is the shortest route to school?” was provided as an example of a well-structured problem since it suggests a single correct solution. Participants were instructed to use these analogies to help them construct open- and closed-ended design features of the assigned case study.
Following this introduction, participants were asked to orally report their understandings of the pedagogical advantages and disadvantages of each design (time constraints within the session prevented discussion of disadvantages associated with closed-ended design). Participants were then divided into groups of two or three members and were provided with a design template worksheet that contained the case scenario and space where they were instructed to write specific learning objectives and corresponding case questions (see Fig. 1, italics). Each group was randomly assigned the task of creating an open-ended or a closed-ended case study design, which required them to develop matched objectives and questions. More specific instructions for creating the learning objectives and questions were written on an overhead projection of the design template and are indicated in Fig. 1, bold. In particular, since planning of the workshop was influenced by the “understanding by design” methodology of Wiggins and McTighe (15), the phrase “specific understanding” was explained to involve a summary of the particular understanding desired and the kinds of student work needed to achieve and show such an understanding. After this assignment was completed, some of the groups were asked to report the results of their work during the wrap up of the workshop. As these reports were discussed, some of the participant-generated learning objectives and corresponding questions for both the open-ended and closed-ended approaches were written on a board for all to see. The workshop was concluded with a review and reinforcement of the advantages and disadvantages of open- and closed-ended case designs.
The design templates containing the group-generated learning objectives and questions were collected at the end of each session after participants were given time to review their assigned template. In reviewing these templates, the authors found that the majority of the group questions were not well paired with learning objectives. As a result, the group questions were categorized by general subject area and grouped accordingly. The authors sought to evaluate how accurately participants were able to create questions that corresponded to an open-ended design or a closed-ended design. Using the criteria spelled out in our previous article (2), each author independently rated the faculty member-written questions as open, closed, or ambiguous. An ambiguous question was defined as one that had the possibility of being either open or closed because of uncertain or indefinite wording. The degree of interrater agreement was measured by Cohen's unweighted κ-statistic and accompanying SE (12), and the P value was derived from the resulting z-score (9). The P value tested whether the estimated κ was not the result of chance agreement.
Advantages of open-ended and closed-ended case designs.
The advantages of open- and closed-ended case designs offered by the participants during the two workshops are shown in Table 1.
Case questions written by workshop participants.
The authors received 72 learning objectives and 91 case questions created by the participants during the group work of the two workshops. A review of the learning objectives indicated that all were relevant to the case of blood doping. However, many of the submitted questions were not well matched or matched at all to the corresponding learning objectives provided by the groups. For example, one of the groups assigned to write closed-ended questions gave “Understand the function of EPO [erythropoietin],” “Understand link between RBC [red blood cell] and O2 carrying capacity,” and “Understand homeostatic control of RBC” as learning objectives and offered “What is the advantage of taking EPO to competition?,” “What is the mechanism by which dehydration would alter hemoglobin?,” and “Was the decision to disqualify him justified—Explain” as case questions. A comparison between these two sets showed that both learning objectives and questions were relevant to the case, but there were little or no conceptual links between them. Because this sort of disjunction was commonplace, questions were not categorized by participant-submitted learning objectives. Instead, the authors identified subject areas relevant to the content described in the questions and used the major subject areas to classify each of the questions obtained.
After the questions had been classified by subject area and sorted accordingly, the authors independently determined whether each question was open, closed, or ambiguous. When the individual ratings were compared, it was found that of the total of 91 questions, the authors agreed on the classification of 81 questions (89%). Of the 10 questions where the authors disagreed on classification, 6 questions were categorized as “ambiguous” by one or the other of the authors, indicating that ambiguity was an important factor in interauthor disagreement. In only 4 of 91 questions did an author classify it as open when the other author classified it as closed. The level of agreement was similar whether the participants had written the questions as open or closed ended. The authors agreed on the classification of 30 of 35 total questions (86%) that participants had been assigned to write as open ended (κ = 0.759, P < 0.0001) and 51 of 56 total questions (91%) that participants had been assigned to write as closed ended (κ = 0.663, P < .0001). The high level of agreement about the classification of questions as either open or closed ended indicates a high degree of reliability of the interauthor rating of the questions.
From the total of 91 participant-written questions, the authors selected up to 3 questions to represent each subject area. Selected questions were characteristic of the level and depth of the questions generated by each of the groups. For some subject areas, fewer than three questions were written by the partcipants. In those instances, all submitted questions were selected for Table 2 shows the major subject areas and a representative selection of the corresponding participant-written questions. These questions were selected by the authors to represent the larger, complete sets of open- and closed-ended questions. The questions shown in Table 2 had approximately the same frequency of classification (open, closed, or ambiguous) as those found in the larger sets of questions received from groups. The ratings given each question by the two authors are indicated in Table 2.
Table 3 shows the percentages among a subset of participant-written questions judged as open, closed, or ambiguous by the authors where both authors agreed on the classification of the question into the same category (n = 81). The percentages are separated by open- or closed-ended design according to the original tasks given to the participants on the design template. It is evident that the agreement between the assignment and the unanimous judgment of the authors was greater for the closed-ended questions (88%) than for the open-ended questions (43%).
Workshop participants identify the advantages of open- and closed-ended case designs.
A review of the advantages offered by workshop participants for using an open-ended case design showed participants believe that open-ended questions promote thinking skills, encourage integrative learning, and motivate student learning (Table 1). These assertions are consistent with the claims that we and others have made for the open-ended approach to case design (2, 8). In both workshop sessions, participants were quick to identify enhancement of higher-order or critical thinking as one of the distinct benefits of an open-ended case architecture. Herreid (8) believes that the best case designs provoke this sort of thinking in students when they force students to grapple with ill-structured problems, evaluate incomplete data, and weigh alternate hypotheses to solve the dilemmas. Similarly, we have suggested that open-ended designs promote critical thinking by challenging students to operate at the synthesis and evaluation levels of Bloom's taxonomy of educational objectives (2).
Participant comments on the advantages of closed-ended designs suggest themes of control of learning, focus on course content, support of student learning, and assessment of student understanding (Table 1). These rewards of the closed-ended design closely parallel our previous suggestions (2). In particular, participants felt that cases of closed-ended design can be crafted to promote certain beneficial ways of thinking. One workshop participant described this approach as being “apprenticeship-like.” We have depicted this advantage as the ability to “guide the trajectory of student learning” where the student follows the specific and intentional pathway of thinking laid out by the author of the case (2).
How successful were workshop participants in creating a diversity of open- and closed-ended design elements?
The extensive range of subject areas and case questions shown in Table 2 indicates that workshop participants were able to create a rich variety of approaches toward building an understanding of hematology around a case of blood doping. Consistent with our original hypothesis, participants were successful in generating exemplars of both open- and closed-ended design elements. Open-ended elements ranged from the consideration of the normal concentrations of hemoglobin and contributions of the factors that influence hemoglobin levels to the effects of elevated hemoglobin on athletic performance and the ethics of blood doping. Closed-ended elements ranged from the mechanisms of EPO action to the factors that control hemoglobin levels to the physiological advantages of blood doping and the mechanism of oxygen transport. Since the ability to create such a fruitful range of approaches to case design is a hallmark of a talented designer, these results demonstrate the collective expertise of workshop participants as skillful case developers.
How successful were participants in creating learning objectives that matched design criteria?
A review of the workshop activities and case design templates prepared by the groups gives several indications that the participants undertook the task of creating learning objectives less seriously than they did creating case questions. Informal observations made of the participants during the workshop indicated that some groups appeared to spend little time or effort in establishing the learning objectives. Instead, they moved rapidly to the task of constructing the case questions. Second, we noticed that most of learning objectives submitted were written as brief topic statements or content outlines (e.g., regulation of RBC concentration) rather than as performance outcomes (e.g., students can describe the steps in the regulation of RBC concentration). When fashioned in this way, the objectives do not indicate what students should do to demonstrate their learning and do not fully define the goals of student learning in terms of specific understandings achieved (15). This was a common finding despite the fact that a performance-based objective was used as an example earlier in the workshop during the discussion of the analogy to ill-structured and well-structured problems. Finally, and most significantly, the widespread disparity found between the stated objectives and the expected answers to the case questions suggests that many of the participants failed to deliberately link the learning objectives with the questions. This occurred despite oral instruction during the workshop to make the match. There may have been a number of factors that led to the design flaws we observed in participant-written learning objectives, including our inability to fully delineate the nature and purpose of the objectives and to unequivocally emphasize that they should be connected to the case questions. For whatever reason, the failure to create suitable learning objectives may have contributed to the further difficulties that participants had when creating case questions that matched the design criteria, particularly with regard to open-ended questions.
Is this sort of ineffectual effort to develop student learning objectives that we observed in the workshop participants atypical of professional educators and simply an artifact of the workshop setting? Not according to Wiggins and McTighe (15), who, in making an argument for “backwards design,” point out that many instructors tend to focus on learning activities and overlook learning objectives and assessment. Anecdotal evidence from our own classroom practice and from our experience leading faculty workshops suggests that it is common for faculty members to concentrate on learning activities to the neglect of student learning outcomes. The reasons why faculty members may overlook the creation of suitable learning objectives, or fail to appropriately link learning objectives with corresponding questions or prompts that engage student learning, is a fruitful subject for further research.
These findings highlight a persistent weakness in the way that faculty members tend to approach the design of course learning activities or other aspects of curricular planning. It is easy to lose sight of the learning goals in the midst of excitement over creating or adopting novel teaching strategies such as case study analysis. Wiggins and McTighe (15) remark, “…too many teachers focus on the teaching and not the learning…” As a result, inappropriate learning objectives are set and/or the achievement of the desired objectives cannot be authentically assessed by the questions and prompts that are used to measure student learning. This conclusion reinforces the importance of one of the dictums of backwards design: student learning objectives should be carefully defined before the planning of learning activities if we wish to create instructional units that effectively help students to learn. Furthermore, giving priority to learning outcomes has been suggested to be particularly important for making effective use of higher-order types of questioning (1).
How successful were participants in creating case questions that matched design criteria?
When the bank of participant-written case questions was scrutinized carefully, we found a rich tapestry of useful tools for prompting student learning (see Table 2). We encouraged the participants to work together to create an assortment of open- or closed-ended questions, and we did not constrain their designs to match the specific understandings expected for a particular level of physiology instruction. Indeed, we hoped that participants would generate a broad spectrum of questions that would be helpful for engaging student learning in courses ranging from nonmajors to human biology to medical physiology. Their success with this task is evident in the variety of questions obtained. Moreover, this diverse pool of available questions forms a starting point from which faculty members can choose the particular questions that most effectively help students achieve course-specific learning objectives. As we have pointed out previously (2), it is important to give careful effort to match the level of challenge and the degree of open endedness of case questions with the goals for student learning that are appropriate for a particular course or stage of student intellectual development.
The battery of well-written questions extends across the spectrum of Bloom's taxonomy of educational objectives. Several items are notably provocative of higher-level thinking, for example, “Is there a physiological argument to be made that increased DPO [darbepoietin] had no net effect on athletic performance? Explain” or “How else might (an) athlete achieve increased hemoglobin concentration (besides receiving DPO)?” These types of questions would be useful to physiology faculty members who teach about the regulation of the oxygen-carrying capacity of the blood by EPO and, by implication, its influence on athletic performance. Additionally, the workshop participants provided questions that promoted different degrees of student understanding, from foundational knowledge about specific processes to analyses of the interrelations between multiple body systems. The latter sort of questions would be especially useful in fostering an integrative understanding of the subject matter.
Closer examination of the results shown in Table 2 shows that some of the open-ended questions require at least an “analysis” level of thinking to provide a satisfactory answer. Leading words such as “Would…” and “Could…” and phrases such as “How likely…,” “Does…make sense?,” and “…physiologically justifiable?” prompt students to arrive at an answer through careful appraisal and reasoning and to draw conclusions based on inference and different levels of certitude. Many of these prompts assume that students will take a comprehensive and integrative approach to analyzing the case study. Some of the questions even encourage students to arrive at diverse, original or novel solutions, one of the hallmarks of the open-ended or divergent question (3).
When responding to these sorts of questions, students must analyze the various factors that influence blood hemoglobin to arrive at a credible synthesis of what could be the true cause(s) for the elevated hemoglobin levels in the skier. Students are asked to interpret the clinical findings associated with the case to determine whether there might be other believable explanations for the traces of DPO found. Students need to weigh the physiological benefits expected from blood doping to judge whether banning DPO is physiologically justifiable. Students must apply the ethical principles surrounding the use of performance-enhancing drugs in sports competition, in light of the potential athletic advantages obtained, to judge whether the Olympic committee made a fair decision. In doing so, students are engaged in analytical, synthetic, and evaluative tasks consistent with the goals and advantages of open-ended case design (2). The extent to which the participant-written questions elicit these sorts of student learning activities evinces the degree to which the workshop participants were successful in crafting the open-ended design elements consistent with their assigned task.
As expected, the majority of the closed-ended questions (Table 2) were more straightforward and concrete than the open-ended counterparts and showed features and characteristic language that assume a discrete answer consistent with convergence (3). A review of Table 2 indicates that these questions tend to address the “knowledge” and “comprehension” aspects of cognition. Students are expected to describe the hormone control mechanism mediated by EPO and explain how RBC number or concentration would be altered. Students are called upon to outline important aspects of the pharmacokinetics of DPO (and presumably EPO) in the body. Students are required to identify the normal range of blood hemoglobin, describe the causes for increased hemoglobin, and determine whether a given blood hemoglobin level is high, low, or within the normal range. Furthermore, students are asked to describe the physiological advantages of high levels of hemoglobin. As they address each of these tasks, students would be expected to arrive at a single, satisfactory answer for each of these queries, an outcome that demonstrates the distinctive targeting characteristic of closed-ended case design (2). Thus, we document that participants were capable of generating a broad range of closed-ended questions appropriately derived from a single case scenario.
Table 3 shows that participants were much more successful in writing closed-ended questions versus open-ended questions. The correspondence between the assigned design task and the author's judgment of the product was 88% for closed-ended questions and only 43% for open-ended questions. What might account for the higher “failure” rate that the workshop participants experienced in generating open-ended questions, i.e., why were participants less effective in writing questions that met the criteria for open endedness?
First of all, it is important to note that the failure rate for writing open-ended questions was not uniform over the different subject areas that participants chose to address. When participants wrote open-ended questions about the ethical implications or outcomes of the case, there was a higher degree of match between the assigned task and how the questions were rated. This finding indicates that when participants tackled the ethical implications of blood doping, the open-ended aspects of this topic became readily apparent and more easily translated into appropriate case questions. This accomplishment also shows that, without extraordinary effort, faculty members can successfully incorporate this hidden or neglected aspect of the physiology curriculum (6) into the design of case studies.
On the other hand, when participants attempted to write open-ended questions about the scientific aspects of the case, there was a much lower degree of conformity between the finished product and the assigned task. Here, a much higher frequency of questions written were judged as closed ended. What might have made it particularly difficult for workshop participants to successfully write open-ended questions about the scientific content of the case? We offer the following suggestions.
First, it is more difficult to create well-conceived and constructed open-ended content questions than it is to produce the corresponding closed-ended questions. One of the major reasons for this “hardness” is the difference in the topography of the intellectual domain. Open-ended questions tend to venture into irregular, ill-defined, or even chaotic areas of a subject or discipline. Close-ended questions tend to focus the smoother, more apparent, and predictable regions of the intellectual terrain. It can be difficult for faculty members, particularly if they are not experts in the field, to make full sense of the open-ended features of a case study. If the scientific underpinnings of a case are ill structured (10), it is hard to gain perspective or get a sense of direction and so consistently produce well-crafted open-ended questions.
Furthermore, writing a question that evenhandedly directs students along multiple lines of inquiry and invites multiple solutions is challenging. It is difficult to construct questions that avoid biasing student thinking in one direction or another when many possibilities must be impartially entertained. If instructors try to anticipate student answers as a means for facilitating the creation of suitable questions, it can be even more challenging to imagine all the different sorts of acceptable student responses to an open-ended question. Moreover, the determination of open endedness itself rests in part on subjective judgments of what constitute acceptable or unacceptable answers to a particular question or assignment. A problem that appears to have multiple acceptable solutions to one evaluator may seem to have only a single acceptable solution to another. This can lend an air of ambiguity to the determination of what constitutes an open-ended question.
Given these difficulties, it is not surprising that participants had less success in crafting open-ended content questions during the workshop. The inherent difficulties in this process may have even dissuaded some of the groups from completing the assigned task. In fact, we received requests from some participants to switch from open ended to closed ended because they told us that the process of writing an open-ended question was “too hard.”
Second, open endedness in science education is a foreign and unfamiliar approach to teaching for those of us who supervise content-rich courses such as physiology. In this setting, we become accustomed to writing questions that anticipate a single correct answer, a practice consistent with the priority that we place on student knowledge and comprehension of the subject. As such, we are familiar with, and perhaps proficient at, writing closed-ended questions. Crafting open-ended questions is against the grain of our ordinary practice because open endedness is not typically an important aspect of our learning outcomes or a regular feature of our forms of assessment. It simply does not come naturally.
When challenged with the unfamiliar, “out-of-the box” task of writing open-ended questions in the workshop, some participants may have opted to revert to the more routine habit of writing closed-ended questions. This conclusion is consistent with the interpretations that others have made of the efforts to improve questioning practice by primary and secondary school teachers. Educational researchers have hypothesized that, when prompted to emphasize higher-order questioning, many teachers fail to accomplish this goal because of the daunting barriers posed by the significant alterations in their teaching behaviors that would be required (5, 16). As science educator William McComas has put it, “Study after study reveals that although educators know that higher-order divergent questions hold significantly more power to engage the learner and ensure transfer of knowledge, we consistently retreat to using lower-order, convergent style questions when teaching and testing students” (13).
Our findings indicate that, despite the fact that instructors of anatomy and physiology recognize the value of open-ended science questions, they have considerable difficulty in successfully creating them. Inexperience may contribute to the low rate of success in producing open-ended questions. Many science faculty members who teach content-rich courses simply aren't used to writing questions that promote creative thinking or complex problem solving. The high failure rate we observed in the workshop occurred despite the fact that participants were quick to acknowledge the advantages of open-ended science questions for promoting critical thinking, engagement, and integration (see Table 1). Faculty members may recognize the virtues of open-ended case design, but our results imply that many could not successfully generate open-ended case questions when given the opportunity. Consistent with the recommendations from the literature on effective questioning (1, 4, 14, 16), we suggest that when faculty members decide to create open-ended case questions, they should pay careful attention to the expected outcomes of the case study, the context in which the questions are asked, and the kinds of responses that students will likely provide. These factors must be at the forefront of the instructor's mind if he or she wishes to ensure that, by responding to the question, students are prompted to use and improve their higher-order thinking skills.
In summary, our analysis of the products of a case design workshop indicates that participants were able to produce a wide array of both open- and closed-ended case questions from a single case about blood doping. However, participants were not as proficient in generating open-ended questions, even though they recognized the value of the open-ended case architecture. This observation suggests that a more deliberate effort needs to be made when writing open-ended case questions. Our findings also make visible the thesis that, starting with a suitable story, case studies can be well crafted along both open- and closed-ended lines (2). It remains the responsibility of conscientious case designers to take full advantage of these two avenues for case development as they unfold each story.
We thank Arne Tärnvik (Umeå University) for permission to use the case study on blood doping and Paul Vermette (Niagara University) for valuable discussions.
- © 2008 American Physiological Society