This study examined the effectiveness of a series of Web-based, multimedia tutorials on methods of human body composition analysis. Tutorials were developed around four body composition topics: hydrodensitometry (underwater weighing), dual-energy X-ray absorptiometry, bioelectrical impedance analysis, and total body electrical conductivity. Thirty-two students enrolled in the course were randomly assigned to learn the material through either the Web-based tutorials only (“Computer”), a traditional lecture format (“Lecture”), or lectures supplemented with Web-based tutorials (“Both”). All students were administered a validated pretest before randomization and an identical posttest at the completion of the course. The reliability of the test was 0.84. The mean score changes from pretest to posttest were not significantly different among the groups (65.4 ± 17.31, 78.82 ± 21.50, and 76 ± 21.22 for the Computer, Both, and Lecture groups, respectively). Additionally, a Likert-type assessment found equally positive attitudes toward all three formats. The results indicate that Web-based tutorials are as effective as the traditional lecture format for teaching these topics.
- instructional technology
- World Wide Web
- interactive tutorials
Laboratory experiential learning is an essential component of undergraduate education in all of the life sciences. However, often it is necessary to limit the scope of these laboratory experiences to those that can accommodate the larger class sizes frequently associated with many undergraduate courses. Thus it may be difficult or impossible to provide undergraduate students with hands-on experience in a research facility where the equipment is already in high demand by research faculty and graduate students and the space is not conducive to handling more than a few people at once. The University of Vermont’s General Clinical Research Center, for example, is equipped with a state-of-the-art physiological testing facility for measuring energy intake and expenditure and human body composition. Many academic units at the University attempt to schedule time at this facility for student laboratories and demonstrations. However, due to the very high research use of this facility, no more than a few students can be scheduled for teaching purposes each year.
The rapid advances being made in instructional technologies may provide a reasonable, cost-effective alternative for some of these experiences. The capabilities of the World Wide Web for delivering interactive, multimedia content are allowing educators to develop “virtual laboratory experiences” that can be delivered to students worldwide and on demand. Considering how quickly these on-line learning materials are being made available, it is essential that we conduct studies to determine what works and what doesn’t. A number of published studies in this area are showing promising results. For example, Carew et al. (2) reported significantly improved student performance when traditional lectures were supplemented with on-line tutorials and quizzes in a college-level nutrition course. Janda (4) examined the use of multimedia instruction in an introductory college-level political science course. Lectures were supplemented with traditional discussion groups (Control group), statistical and tutorial software (Single Media group), and hypermedia and videodisc presentations (Multimedia group). Attitudes and cognitive learning were assessed in all three groups. No significant differences were found in cognitive learning among the groups. However, very positive attitudes were reported from participants in the multimedia group. More recently, Lum and Gross (5) developed and tested a Web-based tutorial for teaching spirometry over the Internet. Pre- and posttests were administered to 65 medical students, interns, and senior hospital staff. The spirometry tutorial significantly improved knowledge base and spirometry interpretation in these individuals.
Thus, although research in this area is expanding, more studies are clearly still needed that examine both the effectiveness of these learning paradigms and the related student attitudes (7). Web-based learning is already a significant component of many distance education programs at many universities. The rapidly expanding availability of broadband Internet access and user-friendly multimedia educational software development tools will contribute to a further expansion of this type of learning, allowing for the delivery of engaging, media-rich content. In the present paper, we report the results of a study we conducted to compare the effectiveness of a series of Web-based, interactive, multimedia tutorials for teaching methods of human body composition analysis vs. teaching the same material in a traditional lecture format.
Four interactive, multimedia Web-based tutorials were developed, focusing on four methods of human body composition analysis (http://nutrition.uvm.edu/bodycomp/). The tutorials were developed using Macromedia Flash (Macromedia, San Francisco, CA) for interactive animation, text, and audio. This software produces high quality interactive animations of relatively small file sizes that are quickly downloaded and viewable over the Internet. Other software applications utilized in the development of the tutorials were Cold Fusion Studio (Macromedia), for Web-database connectivity to aid in the collection of data from users, and RealVideo (RealNetworks, Seattle, WA), for delivering streaming videos. The tutorials were designed to help students meet specific learning objectives related to four methods of human body composition analysis: underwater weighing, bioelectrical impedance analysis, total body electrical conductivity, and dual-energy X-ray absorptiometry (DEXA) (Fig. 1). Each method’s tutorial consisted of a series of lessons, with each lesson building upon the material covered in the earlier lessons. Thus, for example, the hydrodensitometry tutorial consisted of a series of seven lessons. The lessons topics are 1) overview of Archimedes’ principle, 2) review of static lung volume measurements, 3) correcting for lung volume, 4) derivation of the Siri equation, 5) complete underwater weighing simulation, 6) interactive sample calculations, and 7) a streaming video of an actual underwater weighing procedure being performed on a volunteer. Students are encouraged to go through each of the lessons in the order presented; however, students can easily move back and forth within a lesson or between lessons. In addition, some lessons provide for the students to enter sample data and perform sample calculations.
To ascertain the effects of our on-line tutorials on both cognitive gain and student attitudes, we developed an evaluation instrument to be administered before the students begin the tutorials and again after completing all of the tutorials. Specific cognitive behavioral objectives were considered during test development, and the objectives were placed in a table of specifications (1). Questions were devised to assess student learning at the highest level applicable to each of the objectives in the table. This was completed in such a manner as to allow ∼25% of the objectives tested to be from each of the four methodologies of body composition analysis being taught.
A sample behavioral objective and corresponding pre- and posttest question would be as follows.
Behavioral Objective: The learner will differentiate between density and area density as it pertains to bone mineral density.
Associated Pre- and Posttest Question: In a two-dimensional DEXA scan, how is bone mineral density expressed?
Mass of bone per volume
Quantity of calcium per cubic centimeter
Mass per unit area
None of the above
Developing the test from a table of specifications was the first measure taken to help ensure high content validity of the assessment instrument. Validity of the instrument was further enhanced by addressing the same behavioral objectives in the test as in the tutorials, thereby ensuring that the material taught matched the objectives tested.
Test reliability was ensured by developing questions in three familiar forms: matching with at least seven items per column (with the columns labeled), completion with all questions independent of each other and with only a single possible one-word or one-term correct answer, and multiple choice (with four viable options and all negative questions grouped). A negative question was one that was phrased so as to ask students to indicate the item from among the choices that does NOT belong. Clear, concise directions and point values were given for each section of the test. Questions were placed in perceived order from easiest to most difficult and grouped by subject matter.
The test was then reviewed by a panel of nine experts consisting of educators from various higher education institutions as well as experts familiar with methods of human body composition analysis. Feedback from the panel of experts was used to make minor modifications to the test instrument and to help ensure an instrument that had high usability and validity, was free of bias, and could be predicted to have high reliability.
To test the tutorials in an actual university teaching environment, a new, one-credit course, entitled Methods of Human Body Composition Analysis, was offered. The class was available to anyone having taken an introductory nutrition course as a prerequisite. A total of 34 students enrolled in the class. During the first class, all in attendance were provided with written information describing the nature of the class and were given objectives, a syllabus, and a tentative class schedule. They were informed that the class was part of a research project to test different teaching methods and that they would be randomly assigned to one of the three different “learning method” groups. Anyone not willing to remain in the class, regardless of their ultimate group assignment, was asked to drop the class at that point. An informed consent form, a five-question attitude assessment, and a demographic information form were given to, completed by, and collected from each of the students in attendance. All thirty-four students were then “pretested” with our evaluation instrument. Two of the original 34 students withdrew from the class, leaving 32 who completed the entire study.
The students were then randomly assigned to one of three learning groups; the “lecture only” group (Lecture), the “tutorials only” group (Computer), or the “tutorial-supplemented” group (Both). Students in the Lecture group were requested to attend all lectures but were not permitted access to the Web tutorials. Students in the Computer group had access to the four computer tutorials on the Web, but were not permitted to attend the lectures. Students in the Both group were requested to attend all lectures as well as to use the Web tutorials. By use of a tentative estimate of standard deviation of ±15, a 95% confidence interval, and having 11 students per group, the power to detect a difference of one letter grade (10% or 20 points on a 200-point test) was estimated to be 87%.
One week after the first class, an orientation class was given for all students in the groups that would be using the Web tutorials (Computer and Both). At this time, the students were given their individual passwords to access the tutorials, and they were instructed on how to access the tutorials from either their home or dormitory computers or from computers in campus laboratories. The Web tutorials themselves were also briefly demonstrated. Students in the Computer group were informed that they were not allowed access to lectures or lecture materials. Similarly, students in the Lecture group were instructed not to attempt to view the computer materials. However, other than password-protecting the Web site and instructing the students on the rules, we made no other attempt to enforce these requirements.
Two weeks after the first class, lectures commenced for students in the two groups with access to the lectures (Lecture and Both). All lectures were ∼50 min in duration. Two lectures were devoted to the topic of underwater weighing, two lectures to DEXA, and one lecture each was devoted to bioelectrical impedance analysis and total body electrical conductivity. Lecture notes and transparencies were developed using the same behavioral objectives as those addressed in the online Web tutorials. Students in these groups were informed that the posttest would test them on the four methodologies equally and to study accordingly - even though the number of lectures had not been divided equally among the topics. None of the students was informed that the posttest was to be identical to the pretest, and the pretests were not returned to the students.
To obtain an estimate of the instructional time spent on the different groups, attendance was taken in lecture for the Lecture and Both groups. In the groups with access to the Web-based tutorials (Computer and Both), Web-tracking statistical software (Webtrends, Portland, OR) was used to obtain data on which students were accessing the tutorials, the number of times they logged on to the tutorials, and the amount of time each student spent while logged on to the tutorials.
One week after the final topic lecture (week 9), the 32 students were posttested with a 66-question, paper and pencil, cognitive assessment instrument (identical to the pretest). Immediately before the posttest was administered, a Likert-type attitude assessment instrument was completed by all 32 students, to assess and compare attitudes toward the class among the three different treatment groups. The responses to the attitude assessment were anonymous, but the treatment group of the respondent was distinguishable. Posttest attitudes were compared using one-way analysis of variance. The posttest was then given with no time limit (as had the pretest). On the basis of the posttest results, difficulty indexes were calculated for all 66 questions to ensure that none of the multiple choice difficulty indexes exceeded 75% and that no difficulty index for the matching and completion questions exceeded 80% (1). If a four-choice, multiple-choice question was answered incorrectly by 75% of the learners, it could be deduced that the answers were the result of guessing rather than learning. On the basis of this difficulty index analysis, no questions were determined to be too difficult. To identify questions as discriminating, nondiscriminating, or reversals, subjects were sorted by posttest scores, and the highest and lowest 25% (8 students each) were identified. The number of correct answers for each question answered by students in each of these two quartiles was totaled and compared. One question was found to have more correct answers in the lowest quartile than in the highest quartile and was deemed a reversal and eliminated. The original test was based on a maximum score of 200 points. However, after elimination of the one reversal and one other question for which the material was covered differently enough in the groups attending lecture compared with the Web tutorials to change the question from analysis to knowledge level, according to Bloom’s taxonomy (1), the final total possible score for both the pretest and posttest was adjusted to 194. The change in score from pretest to posttest was compared among the three groups.
The validity and reliability of any assessment instrument and the resultant data are paramount to meaningful measurement. The cognitive assessment test and results for this study were thoroughly examined and found to be highly valid and reliable, respectively. The split-half method of determining reliability involves correlating the number of correct responses to odd-numbered questions for each student with the number of correctly answered even-numbered responses. Correlation between the two halves of the test resulted in an r value of 0.72. This r value represents the reliability of only one-half of the test. The Spearman-Brown formula was then applied to this correlation coefficient to yield total test reliability. The reliability of the test in this case, as determined by the split-half method, was 0.84. Typically, exams developed by instructors have reliability coefficients between 0.60 and 0.85, and national aptitude tests, such as the SAT, should have reliabilities exceeding 0.90 (3). Thus the reliability of this test was deemed adequate for this study.
Eleven students completed the pretest, the posttest, and the attitude assessment in the Lecture group and the Both group. Ten students in the Computer group completed all three assessment instruments. Students participating in this study were of diverse demographics. Class standing, sex, age, self-reported grade point average, pretest attitudes toward computers, and computer access are presented in Table 1. The sex distribution in the class may seem abnormal, but this is typical for this department at this educational institution.
Because the pretest and posttest were identical tests, changes in scores were the predetermined primary outcome measurement. Cognitive test results are presented in Fig. 2. Mean score changes from pretest to posttest were 76.8 ± 21.22 (SD) for the Lecture group, 78.82 ± 21.50 for the Both group, and 65.4 ± 17.31 for the Computer group. The scores represent the total point improvement, based on a maximum score of 194. Assumptions of normality and homogeneity of variance were found, through the use of normality plots and Levene’s test for homogeneity of variance, not to be violated. Analysis of covariance (adjusting for pretest score) indicated no significant difference in the mean score changes among the three groups. Variability in score changes was slightly higher than estimated, since the final pooled standard deviation was ∼20, rather than the original estimate of 15. For this reason, the study design was determined to have 80% power to detect a difference of 12% in pre- and posttest scores.
Results of the posttest attitude assessment are presented in Table 2. On a five point scale, with five as a very positive attitude, mean attitudes for the three groups were 3.54 ± 0.66 (SD) for the lecture group, 3.70 ± 0.82 for the computer group, and 3.77 ± 0.59 for the both group. Because positive attitude is the value we were measuring, the scores from the two negative questions (questions 7 and 8) were adjusted. For example, with question 8, “There is not enough value in the course to justify offering it again,” a response of 5 (strongly agree) would be a very negative attitude and would be converted to a 1. Analysis of the adjusted results from the Likert-type assessment indicated no significant difference in student attitudes toward all three learning formats. That is, learning via the Web-based tutorials was found to be equally as enjoyable as the other two methods. The one question showing a significantly different score among the groups was question 8 (Table 2). The group receiving only lectures saw significantly less value in the course than did the Computer group or the group receiving lectures supplemented with the Web-based tutorials.
The methods of human body composition analysis covered in these tutorials require an understanding of both basic and advanced physiological, chemical, and physical science concepts. Ideally, students learning these methods would be best served by hands-on experiential learning in a laboratory setting. However, the high cost and limited availability of body composition analysis equipment, particularly hydrodensitometry, DEXA, and total body electrical conductivity equipment, limit this type of experiential learning to a relatively small number of students. Thus, for these topics, undergraduate students are typically limited to traditional lecture-based formats. To address this need for an alternative to “in-person” experiential labs on methods of human body composition analysis, we developed a series of Web-based, multimedia tutorials. Development of these tutorials required ∼80–100 person days per topic. Thus the four lesson topics required a total investment of 320–400 person days. This number includes the combined contributions of a principle investigator, one graduate assistant, one graphic artist, and one videographer. Although this may seem prohibitively high, we believe that it represents a cost-effective means of delivering educational materials when one considers the fact that the Internet allows for the completed course materials to be shared by teaching and research faculty around the world.
The results of this study indicate that the Web-based tutorials can be as effective as lecture-based formats in teaching these concepts. Pretest scores, posttest scores, and changes in scores for the three treatment groups were not significantly different (Fig. 1).
Student attitudes may not be directly related to cognitive achievement, but they are an integral component of learning and may be an indicator of perceived ease of learning. Although attitudes toward the course were not significantly different among the three groups, it should be noted (Table 2) that the most positive mean attitude score was seen in the Both group, and the least positive mean attitude score was seen in the Lecture group (3.77 and 3.54, respectively). Furthermore, the Lecture group saw the least value in the course (Table 2, question 8), suggesting that the multimedia computer tutorials added a measure of relevance to the topic not present in the lectures. Perhaps this was due to the visual nature of the interactive animations, combined with the videos of actual testing procedures.
The mean amounts of total time spent using the tutorials by students in the Both group and the Computer group were 180 and 299 min, respectively. It should be noted that the variance in time spent using the tutorials by students in the Both group was quite large (SD ±165) and the range was 0–450 min. This high variance and range of times may reflect the fact that the material covered in the lectures was essentially identical in content and sequence of subject matter as that presented in the on-line tutorials. This was an intentional part of the experimental design to ensure that the delivery methods, rather than the course content, were being tested and that all students had an equal chance when taking the posttest. However, it may have inadvertently caused some students to view the tutorials as redundant to the lectures and therefore not necessary, whereas those who considered the tutorials as helpful supplements to lecture notes may have been the more frequent users who accessed the tutorials for up to 450 min. In hindsight, perhaps a better experimental design would have been to define a Both group in which the computer is used to present the theory, concepts, and factual knowledge related to the lesson materials, whereas the lecture is devoted to a higher-level discussion of the material, including answering questions about the lessons and discussion of the research and clinical applications of the body composition methods.
One might expect a high correlation between time spent on the tutorials and final test scores, especially in the group with the tutorials as their sole source of instruction. However, the correlations were not high in either of the groups with access to the tutorials (r = 0.42 for the Computer group and r = 0.26 for the Both group). However, great care should be taken in interpreting these results in light of the limitations of the Web-tracking software used. For example, there was no way of confirming who was using the tutorials, since it was possible (although discouraged) for students to share passwords. In addition, incidences of students failing to properly log off from the tutorials before leaving a computer were occasionally observed by the investigators. This would cause the Web-tracking software to overestimate the time the student was using the tutorials. It also allowed the next student to access the tutorials without entering a password, and his/her time on the tutorials could, therefore, be underestimated. Another limitation of the tracking software was an inability to assess the quality of the time spent on the tutorials. For example, the tutorial Web page could have been minimized while the logged-on student was using an unrelated application, watching television, talking with friends, or engaged in some other distracting activity.
In our attempt to control the study carefully and ensure that all students, regardless of group assignment, had equal opportunity to perform well on the posttest, we may have inadvertently limited our ability to evaluate the effectiveness of the tutorials as a course supplement in the Both group. That is, the material covered in the lectures and the tutorials was identical. Students in the Both group may simply have viewed the lectures as an “alternative” to using the tutorials, or vice versa. The real learning advantage to the use of instructional technologies should be to free time for the instructor to devote to the higher levels of cognitive learning such as analysis, synthesis, and evaluation (6). Thus, to better evaluate the effectiveness of the tutorials as a course supplement, it would be necessary not to repeat the tutorial material during lectures but rather to use the lecture time to pursue these higher levels of learning.
In summary, the utilization of instructional technologies, such as Web-based multimedia tutorials, in life sciences higher education offers tremendous potential in both distance education applications, and as a supplement to traditional lectures. The development and delivery of Web-based courses and course-supporting materials will be a driving force for change in higher education in the immediate future. However, it will not require that every faculty member be involved in all stages of the creative process. Rather, each academic field will rely on a relatively small number of its experts to help create the course materials in a particular field. In this way, institutions of higher education will be able to benefit from the cost efficiency associated with the sharing of course materials over the Internet. In the present study, we have demonstrated that interactive, multimedia, Web-based tutorials are as effective as traditional lectures for learning the scientific bases, physiological concepts, and practical applications of four methods of human body composition analysis. In making these tutorials freely available on the Web to all interested life science educators, we hope to encourage similar exchange and sharing of on-line educational resources.
This study was funded in part by a grant from the USDA Higher Education Challenge Grants Program, Grant no. 97–38411–4255.
Address for reprint requests and other correspondence: S. J. Pintauro, Dept. of Nutrition and Food Sciences, 318 Terrill Hall, Univ. of Vermont, Burlington, VT 05405 (E-mail:).
- © 2002 American Physiological Society