Transforming Learning with Mixed and Virtual Reality Technologies

Jamie Reaves Kirkley, Sonny E. Kirkley

Вlended learning has been discussed primarily in the light of using currently available learning approaches and technologies, particularly with a focus on how online learning can be integrated with face-to-face learning. However, as new technologies emerge and training becomes increasingly just-in-time and embedded within specific situations and equipment that people use, instructional designers will be challenged to blend learning in ways that expand beyond our current understandings. This chapter addresses how an emerging class of tech­nologies, mixed and virtual reality, can be blended with current technologies and approaches to create highly innovative and authentic learning opportunities.

Expanding Boundaries of Learning

As stated by Oliver, Herrington, and Reeves (Chapter Thirty-Six, this volume), constructivist learning theory has challenged instructional designers to create learning environments that are more authentic, complex, and geared to support performance-based learning. The hope is that by capitalizing on the elements of increased authenticity, complexity, and real-world performance, we can support learners in developing greater domain expertise, problem solving, and transfer of learning. To do this, we must provide increased immersion into the community of practice, as well as an enculturation into its way of seeing, interpreting, and

The Handbook of Blended Learning

acting. New technologies can potentially provide access to and make visible how experts view, interpret, and act. However, new technologies alone cannot meet the need; instructional designers must design innovative learning environments using appropriate learning methodologies that can support learners with complex problem solving and development of greater expertise.

To meet the need for creating learning environments that provide greater com­plexity and authenticity, instructional methodologies such as problem-based learning (PBL) (Barrows, 1992; Kirkley et al, 2003; Woods, 1992) and case-based reason­ing (CBR) (Riesbeck & Schank, 1989) have been increasingly used as design frame­works in a wide variety of fields, including business, engineering, military, and medicine. Both focus on facilitating processes of student inquiry and real-world cases or problems as an impetus for learning, thus providing greater complexity in the learn­ing environment. Learning occurs through the complex interactions among learners' existing knowledge, the social context, and the problem to be solved (Duffy & Cunningham, 1996). By centering the learning situation in real-world problems, we have the opportunity to acculturate the learner into the processes, practices, and language of a specific domain (Reiser, 2002).

In order to blend learning effectively, we need to better understand how to use learning methodologies such as PBL and CBR, strategies such as discussion and role playing, and various technologies such as face-to-face and online learn­ing in order to make learning effective. However, as new technologies emerge, in­structional designers must expand their notion of blended learning and constantly assess and reassess how to use methodologies, strategies, and technologies in order to create highly innovative learning environments. Thus, we can assume blended learning will become exponentially more complex as we are challenged to deter­mine how to best design and develop a learning environment using specific methodologies, learning strategies, and technologies. This applies to all parties with an interest in the learning environment: instructional designers who must create the instructional content and environment and keep it up-to-date over the life cycle of use; trainers and instructors who must facilitate the learning; learners who must learn how to learn with these new capabilities; and administrators who must decide what technologies to support and how to pay for their implementa­tion within budget constraints.

A secondary challenge lies with determining how to move a learner seamlessly through the overall process in a way that effectively supports learning. This challenge relates to creating appropriate processes and scaffolding that can support learners within learning environments that are cognitively and technologically complex (Hedberg, 2002). This requires special focus on designing effective scaffolding sup­ports that are embedded within these learning environments. Kirkley et al. (2003) de­fine embedded scaffolds as systematically designed elements of support that are integrated

Expanding the Boundaries of Blended Learning

directly within the learning environment. These are based on a performance support metaphor where the goal is to provide just-in-time, adaptive scaffolding using resources, learning tools and software, pedagogy, content, or the environment.

In order to address these challenges, our team has been researching and designing training environments for military, corporate, and educational contexts that use mixed and virtual reality technologies. In this chapter, we present two examples of blended learning environments where different training goals, contexts, strategies, and technologies have been used within specific design frame­works to create innovative training environments. The goal is not only to provide examples of emerging technologies available for blended learning environments but to provide design models of how to combine and use different technologies with specific methodologies and design frameworks.

Applications of Mixed and Virtual Reality to Support Training and Performance

Mixed and virtual reality technologies hold much promise for providing authen­tic and complex learning environments through realistic simulations, visualization of data, and new forms of collaboration and community building. Due to space limitations, we will not report on the effectiveness of virtual reality (such as video games) for training and learning, which has been reported elsewhere. Instead, our focus is primarily on the use of mixed reality technologies, which are likely new to most people. Taken together, mixed and virtual reality technologies provide unique opportunities to support a range of learning goals.

Mixed reality provides the ability to enhance reality. These technologies often involve merging real and virtual worlds in which virtual objects are superimposed on real ones or real objects are used as part of a primarily virtual reality world. As illustrated in Figure 38.1, there is a continuum of types of mixed and virtual reality. On the far left is the real world. On the far right, we find virtual environments in which learners are completely immersed in a virtual world using one or more senses (for example, sight or sound). Mixed reality blends both real and virtual worlds. Augmented vision provides information relevant to a learner's context (for example, location, task, skill level). In this example, a technician sees a job aid while working on the vehicle. Augmented reality overlays virtual information onto the real world so that people perceive that information as part of the world. In this example, a novice ship navigator sees a virtual highway on the water. Augmented virtuality takes a mostly virtual world that is enhanced with some real objects. In this example, virtual building models are intermixed with models of real buildings to analyze architectural designs.

536 The Handbook of Blended Learning

FIGURE 38.1. REALITY-VIRTUALITY TRAINING CONTINUUM.

Reality -Virtuality Training Continuum

Mixed reality systems come in a variety of forms, from stationary systems where the person is relatively immobile but is able to visualize data in useful ways (for example, looking through mounted video binoculars in a museum to see the skin on a dinosaur skeleton) to mobile systems in which wearable computers are used and the display is worn on the head (as when an automotive repair techni­cian looks at a real car engine and sees relevant parts labeled to perform a procedure). For a general overview of the field, two surveys of augmented reality provide an excellent overview (Azuma, 1997; Azuma et al., 2001).

The field of mixed reality began to show the promise of viable applications in the 1990s. Specifically, augmented reality was used for applications to support maintenance and repair, as well as medical appHcations. The Boeing Company used augmented reality to support maintenance and assembly for aircraft wiring har­nesses (Gaudell & Mizell, 1992). Feiner and his colleagues at Columbia University developed KARMA (knowledge augmented reality for maintenance assistance), a test-bed system for automating the design of systems for maintenance and repair tasks. The Columbia group also developed a system termed " architectural anatomy" that enabled a user to " see through" walls and view wiring and other types of infrastructure (Feiner, 1992; Feiner, Webster, Krueger, Maclntyre, & Keller, 1995). Within medicine, augmented reality has been used to support surgeons by over­laying medical information, such as ultrasound images, directly onto the body to guide the doctor in performing a biopsy (Bajura, Fuchs, & Ohbuchi, 1992).

More recently, in a move toward highly mobile environments, Feiner and colleagues (Feiner, Maclntyre, Hollerer, & Webster, 1997; Hollerer, Feiner, Terauchi, Rashid, & Hallaway, 1999) have described prototype wearable sys­tems to be used for travel, history, recreation, and touring. Their tour guide

Expanding the Boundaries of Blended Learning

application provides information about a university campus (names of build­ings, Web information about academic departments) through head-worn displays as well as palm-sized computers.

Building on the work at Columbia University, the Naval Research Lab has developed the Battlefield Augmented Reality System (BARS). Figure 38.2 illus­trates how augmented reality simulations can be used for training. In this exam­ple, we see virtual minefields on the ground, a virtual rock, and information on the screen (for example, location). The BARS system provides situational aware­ness to soldiers by overlaying important information such as routes on the ground, outlining buildings, and identifying the location of enemy soldiers (Gabbard et al, 2002). In related work, the U.S. Army has been investigating how to use augmented reality simulations for training. The Mobile Augmented Reality Contextual Embedded Training and EPSS (MARCETE) project enables three-dimensional computer-generated characters and events to be displayed in the real world. Basically, the soldiers can participate in video-game-like experiences using the real world as the playing board (Kirkley Borland, et al., in press).

FIGURE 38.2. MARCETE CONCEPTUAL TRAINING EXAMPLE.

The Handbook of Blended Learning

The MagicBook project at the University of Washington Human Interface Technology (HIT) Lab (Billinghurst, Kato, & Poupyrev, 2001) seeks to blend mixed and virtual reality. It consists of a video see-through augmented reality system (shown in Figure 38.3) that is used to view a book or other document with embedded symbols. When the system recognizes the symbol, it displays a three-dimensional model such as a building in that location. The person using the system can choose to zoom into a scene and is at that point in a virtual reality scene (for example, walking around inside the building).

The blending of these two technologies offers many possibilities for train­ing and learning environments. However, there are technical challenges that still exist with implementing and using these technologies.

Challenges of Using Mixed Reality for Training

Although mixed and virtual reality technologies offer much promise for creat­ing innovative learning environments, there are still many technical challenges with regard to obtaining accurate and precise tracking systems (for example,

Expanding the Boundaries of Blended Learning

determining exactly where someone is looking), high-quality visual display out­puts, and computer processing power. While these technical issues will be ad­dressed with new technological advancements, it is even more critical that we develop models of training using these technologies. While the technologies them­selves offer much promise for innovating learning and performance, the design of the training methods, processes, and tools is critical to ensuring that these promises of developing expertise, problem-solving skills, and transfer are realized.

Much of the literature on mixed reality training focuses either on providing examples of training environments or addressing technological or hardware issues (Nakajima & Itho, 2003; Schwald & de Laval, 2003). There have also been some studies focused on comparing the use of augmented reality for training versus more traditional techniques (Boud, Haniff, Baber, & Steiner, 1999; Ong & Nee, 2004). In general, research supports that on procedural and performance-based tasks, overall performance improves and error is reduced when using augmented reality as compared to ofher modes of interacting with content. For instance, Tang, Owen, Biocca, and Мои (2004) found that on procedural tasks, error was reduced by 82 percent when using augmented reality when comparing paper instructions, instructions on a computer LCD, instructions on a head-worn display (augmented vision), and augmented reality. In addition, the subjects' perception of task load, that is, how hard something is to do, was reduced with augmented reality. One caveat is that only minimal research has been conducted in this area, so we need to understand what is working better, in what conditions, and for what kinds of learning.

Yet as these new technologies expand the possibilities of training approaches, the field of instructional design and development must provide methodologies, processes, and models that meet the needs of these new training environments. Cur­rently, existing instructional methodologies do not adequately address how to design and deliver blended learning using mixed and virtual reality or how to move seam­lessly among these different modalities in the instructional environment. This requires envisioning design models that are flexible, adaptive, and based on innovative methods as well as technologies.

As new technologies continue to drive performance and training needs, they will also continue to push training methodologies to use more innovative training development and implementation techniques. Iterative cycles of development as well as early input from experts, trainers, and trainees will increase the quality and effectiveness of training designs and development. Reigeluth (1996) has stated that new models of instructional design will rely more heavily on input from user-designers. This requires not only developing new methods of training but also using innovative development processes such as rapid prototyping (Tripp & Bichelmeyer, 1990) and participatory design (Schuler & Namioka, 1993) to meet

The Handbook of Blended Learning

the needs of supporting learners in achieving complex performance goals. In the following section, we describe two blending learning environments that have been designed using mixed and virtual reality technologies.

Designing Blended Learning Environments Using Mixed and Virtual Reality

Mixed and virtual reality can be used in a variety of ways within a learning context. In this section, we describe a few of the ways in which these tools can support standard learning activities in the context of two case studies.

Using Augmented Vision to Provide Training and Performance Support Assembly and Maintenance

While one of the common criticisms of training is that it is often divorced from the situations and ways in which it is used, mixed reality offers the opportunity to provide information, resources, and performance support in the place and context in which it is needed: on the job. On the flip side, the workplace can now be brought into the training environment by using augmented reality-based simulations of the workplace as a context for training.

Using current technologies (Figure 38.4), an automotive service technician or assembly worker could engage in learning activities (see Table 38.1) in which they learn how to perform procedural tasks.

Table 38.1 provides examples of various instructional activities where mixed and virtual reality technologies can be used.

Problem-Based Embedded Training

Within the military, there is an push toward the development of authentic and situational training due to new training readiness goals set by the U.S. Department of Defense (Bonk & Wisher, 2000; Harris, 2002), along with emerging techno­logical advances. Military training is moving to be more soldier-centric, just-in-time, and embedded within the equipment that soldiers use. Embedded training uses built-in or add-on training and support components (hardware and software), such as computer-based tutorials and simulations, in effect, the equipment that soldiers use during operational and training modes (Morrison & Orlansky, 1997). The U.S. Army's Future Force Warrior (FFW) (Figure 38.5) and Future Combat Systems (FCS) will transform the way soldiers fight and are trained (U.S. Army Natick Soldier Center, 2003). Within the first increment, a family of manned

The Handbook of Blended Learning

FIGURE 38.5. PROTOTYPE FUTURE FORCE WARRIOR SOLDIER

UNIFORMS WITH THE CAPABILITY TO DELIVER MIXED

AND VIRTUAL REALITY.

systems will provide tomorrow's soldiers with high battlefield mobility and unprecedented organic reconnaissance and surveillance capabilities. These systems require radically innovative training solutions that can prepare soldiers to meet new military training readiness goals. Both FFW and FCS systems will require the development of embedded training and performance support systems during the systems acquisition process, and mixed and virtual reality technologies will play a vital role in these approaches. In fact, without these training technologies, FFW and FCS cannot be deployed and used effectively due to the increased com­plexity of decision making required in using these systems effectively.

Expanding the Boundaries of Blended Learning

Unfortunately, blended learning approaches that take full advantage of mixed and virtual reality technologies have not been developed for these pro­grams. Much of the embedded training as currently described provides simple access to reference materials or the ability to pull up a simulation and practice. However, these approaches seem to lack the benefit that a blended learning ap­proach could bring, including greater authenticity and the development of more robust knowledge. As part of an effort to help people think about the dif­ferent modes that could be employed in an embedded blended learning capa­bility for FFW, we have developed a matrix of technologies and modes of training support (Table 38.2).

For the past two years, Information in Place, with support from the U.S. Army Research Institute, has been developing and prototyping an instructional method­ology and training support package for embedded training using mixed and virtual reality technologies (Kirkley, Kirkley et al, in press). As part of this initiative, PBL was adapted to develop a training methodology, problem-based embedded train­ing (РВЕТ). The РВЕТ instructional model was designed to include four main components of the learning process:

• The mission is the problem and stimulus for learning.

• The action plan phase is a set of instructional events in which soldiers practice specific skills, rehearse aspects of mission execution, explore resource materi­als related to the mission, discuss solutions with colleagues and commanders, and perform other activities as needed.

• The implementation phase is when the soldiers participate in an authentic simulation where skills can be integrated and tested. This can be in a virtual reality environment or a field exercise.

• The after-action review phase is when soldiers, trainers, and commanders evaluate performance.

In Figure 38.6, the process is generally linear but may be iterative depending on training goals and trainee needs.

Using the РВЕТ methodology as a guide, an instructional designer will develop a comprehensive training support package. Within this package, there is a training module with a range of learning activities, scaffolding elements to support learners at various stages of the process, and guidelines for trainers and instructors.

Using current and next-generation technologies, the FFW soldier would engage in learning activities (see Table 38.3) in which they learn how to perform multiple war fighting tasks (for example, learning to use small unmanned robotic systems to perform reconnaissance). Table 38.3 contains an example of how mixed reality technologies could be used for РВЕТ types of training events.

The Handbook of Blended Learning

Next Steps and Technological Evolution

The case studies are just two examples of how technologies can be used to sup­port instructional and assessment events. As we move into the future with the advancement of both technologies as well as the design of learning environments, we will need to continue research and development efforts in order to identify ap­propriate uses and best practices.

Mixed and virtual realities will be an integral part of future learning envi­ronments across industries, in educational settings, and in the military. In order for these systems to be effective at improving learning and strengthening perfor­mance, the stakeholders in the learning process must be educated about the pos­sibilities and trade-offs for effectively implementing them. As part of this decade-long effort, several research and development activities must continue to advance.

Mixed and virtual reality learning methodologies must be developed and tested to determine the most effective ways to blend these technologies into the overall learning environment. While some research has been conducted on performance improvement (for example, speed and error reduction), little validated research has been focused on what aspects are most beneficial for learning or on how to generalize findings into methodologies that can be ap­plied in other settings. For example, using these technologies to increase do­main knowledge, problem solving, and transfer is particularly important if we are to realize important gains potentially offered by these new types of learn­ing environments.

Authoring tools, instructional design guidelines, and instructor guidelines must be developed that will ensure effective learning occurs using these technologies in a blended learning environment. We are developing what we believe to be the first system like this for the U.S. Army Research Institute for the Behavioral and Social Sciences (Kirkley, Borland, et al., in press).

Mixed and virtual reality technologies must continue to mature in a way that is beneficial for learning. Much of the research in the field is focused on how to make the technology work or be usable. Learning scientists must begin to push the technologists for features that matter to learning. The industry as a whole must begin to explore the return on investment of these technologies.

Implementing and integrating these three research tracks effectively will not be an easy task and will not happen overnight. Our challenge is to look toward fu­ture possibilities as an inspirational guide, while we implement this new kind of blended learning environment in the best way we can, given current knowledge and technologies.

Expanding the Boundaries of Blended Learning

____________ /

References

Azuma, R. T. (1997). A survey of augmented reality. Presence: Teleoperators and Virtual

Environments, 6(4), 355-385. Azuma, R., Baillot, Y., Behringer, R., Feiner, S., Julier, S., & Maclntyre, B. (2001). Recent

advances in augmented reality. IEEE Computer Graphics and Applications, 27(6), 34—47. Bajura, M., Fuchs, H., & Ohbuchi, R. (1992). Merging virtual objects with the real world:

Seeing ultrasound imagery within the patient. Computer Graphics, 26(2), 203-210. Barrows, H. S. (1992). The tutorial process. Springfield: Southern Illinois School of Medicine. Billinghurst, M., Kato, H., & Poupyrev, I. (2001). The MagicBook: Moving seamlessly

between reality and virtuality. Computer Graphics and Applications, 27(3), 2-4. Bonk, C. J., & Wisher, R. A. (2000). Applying collaborative and e-learning tools to military distance

learning: A research framework (Tech. Rep. No. 1107). Alexandria, VA: U.S. Army Research

Institute for the Behavioral and Social Sciences. Available at

https://php.indiana.edu/~cjbonk/Dist.Learn%20(Wisher). Boud, A. C, Haniff, D. J., Baber, C, & Steiner, S.J. (1999). Virtual reality and augmented

reality as a training tool for assembly tasks. In Proceedings of International Conference on

Information Visualization 1999 (pp. 32-36). Los Alamitos, CA: IEEE Computer Society. Caudell, Т., & Mizell, D. (1992). Augmented reality: An application of heads-up display

technology to manual manufacturing processes. In Proceedings of Hawaii International

Conference on System Sciences (Vol. 2, pp. 659-669). Los Angeles: IEEE Computer Society. Duffy, T. M., & Cunningham, D.J. (1996). Constructivism: Implications for the design and

delivery of instruction. In D. Jonassen (Ed.), Handbook of research for educational

communications and technology (pp. 170—198). New York: Macmillan. Feiner, S. (1992, May). Annotating the real world with knowledge based graphics on a see-through head

mounted display. Paper presented at the Graphics Interface '92, Palo Alto, CA. Feiner, S., Maclntyre, В., Hollerer, T, & Webster, T. (1997). A touring machine: Prototyping

3D mobile augmented reality systems for exploring the urban environment. Personal

Technology, 1 (4), 208-217. Feiner, S., Webster, A., Krueger, T, Maclntyre, В., & Keller, E. (1995). Architectural

anatomy. Presence, 4(3), 318-325. Gabbard, J. L., Swan, J. E., Hix, D., Lanzagorta, M., Livingston, M., Brown, D., & Julier, S.

(2002, January). Usability engineering: Domain analysis activities for augmented reality systems.

Paper presented at the SPIE 2002, Electronic Imaging Conference, San Jose, CA. Harris, P. (2002). How the U.S. military is reinventing learning. Learning Circuits. Available at

https://www.learningcircuits.org/2002/nov2002/harris.html. Hedberg, J. (2002). Ensuring high quality thinking and scaffolding learning in an online

world. In Proceedings of ACSILITE 2002. Retrieved October 10, 2003, from

https://216.239.37.104/search? q=cache: vp6znmjhlssj: www.unitec.ac.nz/ascilite/

proceedings/papers/166.pdf. Hollerer, T, Feiner, S., Terauchi, Т., Rashid, G., & Hallaway, D. (1999). Exploring MARS:

Developing indoor and outdoor user interfaces to a mobile augmented reality system.

Computers and Graphics, 23(6), 779-785. Kirkley, J. R, Kirkley, S. E., Lindsay, N., Myers, Т. М., & Barclay, M. (2003). Embedded

training for designing mixed reality based military training systems. In Proceedings of the

Interservice I Industry Training, Simulation and Education Conference (I/ITSEC) 2003, Orlando, FL.

CHAPTER THIRTY-NINE