"If there is one lesson I would most like to pound home, it is that one cannot evaluate the use of technology separately from the instructional uses made of it. Smith and Dillon (1999) call it the media/method confound, and it is perhaps as clear a term for the interrelationship between technology and instructional design as I have found. In other words, it is not the technology that has an effect, it is the way it is used."

Tomorrow's Professor Msg.#509 Quality in Distance Education Focus on On-Line Learning

 
Folks:

The posting below looks at some key questions that need to be addressed in evaluating on-line, web-based teaching and learning. It is the executive summary from , Quality in Distance Education, Focus on On-Line Learning, by Katrina A. Meyer. ASHE-ERIC Higher Education Report: Volume 29, Number 4. Adrianna J. Kezar, Series Editor Prepared and published by Jossey-Bass, A Wiley Imprint (www.josseybass.com). Copyright 2002 Wiley Periodicals, Inc. A Wiley Company: All rights reserved. 989 Market Street, San Francisco, CA 94103-1741 . Reprinted with permission.

Regards,

Rick Reis
reis@stanford.edu

UP NEXT: Rethinking Critical Thinking: Values and Attitudes

Tomorrow's Teaching and Learning

----------------------------------------- 2112 words -----------------------------------

QUALITY IN DISTANCE EDUCATION - FOCUS ON ON-LINE LEARNING

Executive Summary

Writing about Web-based learning are fraught with misunderstandings, misperceptions, and mistakes. Undertaking this project has been an eye-opener for me, and reading this report will likely be a surprise to the reader as well. The main questions below express most of these misperceptions - and hard-earned insights.

Does the Research Reviewed Apply Only to Distance Education?

This monograph focuses on the research concerning on-line learning, and although it makes some mention of other modes of distance education, it is oriented toward exploring the specific intricacies of learning over the Web. Distance education is a broader term, comprising several different delivery modes. On-line or Web-based distance education is use of the Web to deliver education whether at a distance or next door, although the Web may also be used to enhance on-campus courses in particular ways. Most of the research reviewed may actually be quite interesting and useful to faculty incorporating the Web in their regular classes. And because the line between Web-based learning at a distance and in an on-campus course is murky at best, research on campus-based uses of the Web is also included in the review.

Focusing on on-line education should not be construed as an attempt to ignore some of the fine work of early distance education theorists and researchers. In many cases, this work holds up quite well and can be constructively applied to on-line learning. Many good reviews of this earlier research exist (Gibson, 1990; Moore and Thompson, 1997), but they are beyond the purview of the current monograph, which focuses on on-line learning.

But there Isn't Much Research on Using the Web, Is There?

I had been operating under two misconceptions, which have been duly erased. I thought there would not be much research on on-line learning: I was not only wrong but gravely wrong. I found numerous papers (not included in this review, as they do not describe a research study) where faculty discussed their personal experiences with on-line learning and generated thoughtful and useful insights into how best to use the medium. Literally hundreds of studies had been peer reviewed and published in on-line journals or posted on conference Web sites. Many faculty in Canada, Australia, England, and the United States have been busy conducting their own research into on-line learning, using some very innovative approaches or more classic experimental designs.

I also thought I would need to rely on the research literature in other domains (e.g., cognitive science) but had to do so less than expected. I was also wrong in thinking that the research might still be focused on such outcomes as student achievement without looking at the intervening variables (in the student, the environment, even the instructional design) that might explain the outcome. I saw some of these types of studies, although we need more.

But Aren't These Studies Poorly Done?

The perception is that most studies done on distance education or the use of technology are poorly designed and prone to incomplete analyses. That certainly is true of the simple comparison study, where student outcomes (such as course grades) for an on-line course are compared with a traditional course. It is the source of the "no significant differences" phenomenon, where possible intervening forces are ignored and the researcher and instructor are the same person, further muddying the results. No one would argue that this design is flawed and the results questionable. And I would be happy if I never saw another such study published anywhere.

Some very good studies are out there, however - some of them quantitative and others qualitative and still others a thoughtful or theoretical analysis of what is going on in an on-line course. Some of these studies are quite creative and use interesting approaches to analyze the on-line course or the student learning resulting from using the Web in a course, e.g., applying critical analyses to student understanding implied in his or her contributions to threaded discussions. Many of these studies would pass the harshest peer review criteria, and others are less complicated but no less worth reading. Even though we may value multivariate, controlled-environment research, it is sometimes the serious and balanced personal voice of the writer sharing his or her insights that may influence other faculty members to try the new medium or improve their use of it as an instructional tool.

Will We Ever Have a Definitive Answer on the Quality of On-Line Learning?

It is unlikely we will ever unravel all the factors that impact on-line learning. It is complex and its elements (the technology and the students) keep changing. Because we have no achieved a definitive answer on quality for more traditional classroom situations, perhaps it is unwise to expect such clarity for on-line learning. But more understanding is always better than less, so the search for clarity will (and should) continue.

It is unlikely we will ever unravel all the factors that impact on-line learning. It is complex and its elements (the technology and the students) keep changing. Because we have not achieved a definitive answer on quality for more traditional classroom situations, perhaps it is unwise to expect such clarity for on-line learning. But more understanding is always better than less, so the search for clarity will (and should) continue.

I remain a supporter of on-line learning and think it holds great promise for students who wish or need to learn this way. It forces faculty and institutions to question assumptions and renew their attention to student learning. No student of technology can pretend that any technology is always and unfailingly positive, however, and I urge others to undertake some of the longitudinal and careful studies of this learning to see whether the Web may have some secondary or tertiary impacts - the unintended consequences technology is famous for - that we cannot see at the present. Most technologies are likely to have such impacts, so it is wise to keep a careful eye on what they might be. In the meantime, however, having doubts about technology is no reason for keeping it away from students, for they may be better able to identify what the problems may be and excuse themselves when they feel it is right to do so.

What Might Be the Most Important Lesson to Take from This Review?

Probably because so much of the earlier research on distance education was the simple comparison study, results were attributed to the use of technology without any attention to the instructional design of the course or the instructional uses of a most flexible tool, the Web. If there is one lesson I would most like to pound home, it is that one cannot evaluate the use of technology separately from the instructional uses made of it. Smith and Dillon (1999) call it the media/method confound, and it is perhaps as clear a term for the interrelationship between technology and instructional design as I have found. In other words, it is not the technology that has an effect, it is the way it is used.

If I could have a second most important lesson, it would be based on the work of Reeves and Nass (1996), who studied the relationship between media and reality. They found that people treat media as though it were real life, which can be attributed to the need and expectation for human relationships, even if you are relating only to a computer. Basic human psychology is the key to unlocking our relationship with the Web, and perhaps it explains why instructional design is so important and why we can create community over a bunch of wires.

And if I could be indulged for a third most important lesson, it would be to urge caution when reading so many opinion pieces on Web-based distance education. As may soon become clear, the researcher's biases can went their way into research design and the interpretation of the results, leaving the reader wondering about the true worth of the study. Be a critical reader; although there is nothing wrong with advocacy, beware of advocacy disguised as research.

So How Would You Define Quality?

Although Chapter Five gives several examples of measure for quality Web-based learning, I would recommend focusing predominantly on student learning and augment it with those variables that contribute to learning. Quality learning is largely the result of ample interaction with the faculty, other students, and content. Because the Web enables interaction, it provides an opportunity for faculty to construct collaborative projects for students. A quality course or program would allow for multiple paths to learning, capitalizing on students' different learning styles or intelligences. Quality would also be the result of opportunities for students to construct meaning from experiences, to reflect on meaning, and to test and retest those understandings in new situations. These definitions are not too dissimilar from those of a quality education, so the only difference is that on-line learning uses the Web to make these opportunities available to students in class or at a distance.

And if you do not have the time and resources, focus exclusively on student learning. Student learning is the ultimate reason why higher education exists, and so we need to know how to define, assess, and improve student learning in multiple ways.

Where Is Research Needed?

As long as you do not undertake a comparison study, there is plenty to do and many answers to seek. If I could promote one of the areas that is most needed, it is finding an answer (or answers) to the question of which technology works with which student and which learning objective in which discipline and why. Let me add a plea for answers to the question about what mix of media (including, of course, face-to-face instruction) works best for which purpose. And because we always need to keep a close eye on the legitimate worries of the technology critics, we must ask whether any evidence exists that the Web is having any deleterious effects or unintended consequences on students or their learning. Undoubtedly, the careful reader will find many more areas for research in this document, and I hope they do so and do so quickly.

Are There Some Unforeseen Benefits to Doing Research on the Web?

The Web also drastically changed how I conducted this review. I am grateful that so many of the conferences focused on using the Web for teaching in higher education have made it a practice to make individual papers available on-line, and I am grateful as well that so many peer-reviewed journals put their contents on-line. This availability made searching for studies somewhat easier, and it means that you will also find these papers at the Web sites indicated in the reference list. Although several fine journal still publish in print, finding good peer-reviewed papers on-line is both quicker and easier for those of us interested in learning from others' experiences and for the researcher interested in sharing his or her results.

What About the Connection Between Quality and Cost?

If I have one regret, it is that I could not spend too much time on the quality/cost nexus: the interconnections between raising quality and lowering cost (or improving efficiency). In light of the large number of good studies on quality alone, I chose to focus on those elements that most contribute to quality learning. Van Dusen (2000) provides a good discussion of the interrelationship of cost and quality, and a growing number of studies look at cost and quality, most particularly Twigg's Center for Academic Transformation and the studies on asynchronous learning networks funded by the Sloan Foundation. Obviously, more needs to be done on this topic, but these sources are excellent places to begin a review of the research on whether one can improve quality and efficiency at the same time.

Is This Report the Definitive Answer?

No. Given how rapidly faculty are producing research on Web-based education and how our use of the Web changes from year to year, this document may be out of date in a few years, when another review of the research should be undertaken. Or in other words, as researchers continue their study of on-line learning, the results of research included in this review may be superseded by new and better understandings of what works best and why. That is a humbling truth, but at least it ensures that we must stay current or lose our place in the rapid advance of the field. Although using the Web in education is sometimes fraught with problems - downed servers, inadequate ISPs, bug-prone software, and even viruses that are communicated by e-mail - any errors or inadequacies found in this document have an all-too-human source.

----------------------------------------------------------------------------------------------------
TOMORROW'S PROFESSOR MAILING LIST
Is sponsored by the STANFORD CENTER FOR TEACHING AND LEARNING
----------------------------------------------------------------------------------------------------