Blending learning for business impact

IBM's Case for Learning Success

Nancy J. Lewis, Peter Z. Orton

rom its inception, IBM Management Development (MD) has taught leader­ship and management in nearly every business and economic cycle. IBM MD's knowledge base derives from its experts in business and its own experience in managing one of the most dynamic companies in the world. Its programs are designed and taught by successful managers with firsthand, first-rate knowl­edge of their field, and an ability to impart their insights in meaningful and com­pelling ways. This chapter describes a blended learning model for training at IBM, as well as a strategy used to evaluate the effectiveness of this model.

To promote effective training and development, IBM MD uses designs that link active learning with the business environment. One such design, which became fully developed and deployed throughout IBM, is its industry-acclaimed blended four-tier learning model (Figure 5.1). IBM MD took the lead within IBM to develop a model of leadership development that incorporates four distinct instructional approaches (tiers) to provide an array of technology-enhanced learning to support the standard classroom intervention. IBM managers now use this four-tier learning approach to master skills and behaviors that develop them as outstanding managers and leaders.

Tier 1 provides information and just-in-time online performance support. These online resources primarily address an ongoing, immediate management concern. The manager with an existing problem accesses the relevant topic using an index or the key­word search engine and brings the material directly to the desktop for online reading, printing to hard copy, or mailing to an e-mail account. Best thinking on over fifty

The Handbook of Blended Learning FIGURE 5.1. IBM FOUR-TIER LEARNING MODEL

leadership and people management topics of concern to managers are available, in­cluding ManageMentor provided by Harvard Business School Publishing. Tools— printable worksheets and checklists—are also available for specific action issues. Links to important external Web sites are highlighted. Because IBM teams are global, man­agers need to have access to policies and practices in different countries. Tier 1 offer­ings allow managers quick and easy access to all global management material.

Tier 2 is interactive online learning. Managers enhance their knowledge and personal development beyond the awareness level by engaging in immersive sim­ulations of the issues presented in tier 1. The online Coaching Simulator comprises eight different scenarios, with over five thousand screens of actions, decision points, and branching results. Twenty-six other simulations cover other management top­ics, such as business conduct guidelines, multicultural issues, work-life issues, retention, and personal business commitments. Going Global, IBM MD's Web site on multicultural business, features over three hundred interactive culture clashes.

Online collaboration, tier 3, brings e-learners together through technology. Through IBM products such as TeamRoom, CustomerRoom, and Lotus

Blending Learning for Business Impact

LearningSpace, managers team with other managers in virtual groupware spaces. Here they learn collaboration skills and create and build real-life learning networks to enhance the company's intellectual capital. Collaborative spaces using same-place, different-time communication enable a global learning environment, eliminating the problems of time zones and travel. This part of the learning process introduces the give-and-take of human dynamics and uses the benefits of technology to transcend time and space. Management Development supports virtual teams with materials and consulting to maximize business results and learning at the same time.

For developing people skills, face-to-face human interaction is arguably the most powerful of learning interventions. In tier 4, classroom activities provide im­mediate responses, are flexible to human needs, and can adapt to different learn­ers' styles. For leadership development, nothing quite duplicates face-to-face learning. In addition, a classroom of peer learners can provide motivation, in­spiration, and a community environment, which further stimulates interest and involvement. Management Development continues to offer interactive classroom experiences. The in-class experiences require the learner to master the material contained in tiers 1, 2, and 3 so that the time spent in classroom Learning Labs in tier 4 can target deeper and richer skills development.

Basic Blue for Managers: IBM's New Manager Program

Basic Blue for Managers equips all new IBM managers with the fundamental knowledge and skills they need as leaders and people managers (Figure 5.2). Aligned with IBM's strategic focus on e-business, the program combines e-learning with in-class activities. Basic Blue's founding principle is that learning is an ex­tended process, not a one-time event. It immerses managers in a collaborative Lotus Notes LearningSpace with online self-study, online simulators, individual competency assessments, an online tutor, second-line management coaching, and classroom experiences. The ten-month process focuses on developing skilled man­agers and leaders with the competencies required to lead high-performance teams.

Phase I: Twenty-Six Weeks of Self-Paced, Online Learning

New first-line managers engage in forty-eight hours of self-paced instructional ac­tivities within a Lotus LearningSpace, available twenty-four hours a day on the company intranet via each manager's desktop or laptop computer. Each LearningSpace cohort numbers twenty-four new managers, who engage with each other in a virtual, asynchronous workroom. However, each manager

The Handbook of Blended Learning FIGURE 5.2. BASIC BLUE: NEW FIRST-LINE MANAGER PROGRAM.

progresses individually at his or her own pace, at an average of two hours per week, working through five online units of modular content: in all, eighteen mandatory and elective managerial topics, custom-developed for IBM's culture and business environment, with additional content from the online Harvard Man-ageMentor (Harvard Business School Publishing). Phase I has these features:

• Each mandatory topic requires the manager to display knowledge mas­tery by achieving a minimum score on an online test, which can be repeated until mastery is attained.

• Fourteen interactive online simulation modules immerse the user in typi­cal, real-life business scenarios—situations in Human Resource Policies and Pro­grams issues. The manager makes decisions by consulting the company intranet's HR policy database and learns from his or her mistakes how to search effec­tively within this huge and critical online information resource.

• A LearningSpace tutor (an experienced company manager or trainer skilled in facilitating in a collaborative online workspace) guides managers via on­line and telephone support.

Blending Learning for Business Impact

• Each new manager's second-line manager works closely with him or her to support four in-field activities (meeting management, goal setting, and retention, for example) with direct reports.

Phase II: The Learning Lab

The in-class five-day Learning Lab is held at company learning centers world­wide. Because information transfer has occurred in phase I, the Learning Lab focuses on experiential, higher-order learning. For example, the coaching model based on Sir John Whitmore's (1996) Coaching for Performance was introduced in the phase I online activities. Managers practiced applying this new knowledge in the eight interactive scenarios within the online Coaching Simulator, which con­tained five thousand screens of possible actions, decision points, and branching results. In the Learning Lab, students move directly into action learning, as all managers bring a real-life situation from their jobs and are coached by a student colleague to define and address the issue. Participants are thus able to jump into higher-order application because both the basic information transfer and skills practice first occurred through e-learning. Other leadership and management topics are addressed similarly.

The focus of the Learning Lab is for the manager to gain self-knowledge as an individual and understand the role as a team or group leader and as a mem­ber of the organization. Validated 360-degree instruments, case studies, and ex­periential exercises are used to address each perspective. These include the Leadership Competencies Survey, Hermann Brain Dominance Instrument, the Hay/McBer Climate and Managerial Style Assessment, and a validated feed­back tool created by IBM's research division. Case studies include cases customized from Harvard Business Review and several developed by in-house subject matter experts. Experiential exercises were created from work done at the U.S. Military Academy at West Point and the Fuqua School of Business at Duke University. Teaming is important at IBM, and students are assigned to learning teams. This creates small learning communities as teams remain together through all three phases. This recursive approach creates an environment where students learn from each other as much as they learn from experts. Within the five-day Learning Lab, less than one hour is used for lecture.

Phase III: Online Learning

Phase III consists of twenty-five weeks of online learning, similar to phase I. How­ever, the content is more complex and focused on application of skills and knowl­edge. On conclusion of the Learning Lab, each manager creates an individual

66 The Handbook of Blended Learning

development plan and an organizational action plan. During phase III, these plans are reviewed with the first-line's manager. When satisfied that the student has demonstrated competency in the workplace, the second-line manager signs off. This final step in the learning process ensures that learnings are applied in the workplace. In addition to such application activities, students also complete more e-learning modules. Unlike phase I, where most modules are mandatory, the student chooses topics of personal relevance in phase III. This design en­courages managers to take an active role in the planning and execution of their own development. Phase III completes the new manager process.

Evaluation Strategy

Using the Kirkpatrick (1979, 1998) model on training impact evaluation, we eval­uated the overall effectiveness along two dimensions, leadership and e-learning, that correspond to our primary objectives. Outcomes were also evaluated along a continuum comprising a chain of impact leading from direct training effects to higher-order business outcomes. When investigating business outcomes, we dis­tinguished between cost avoidance and business result enhancements. Because this program represents a full year of investment in the development in each manager, we considered it important to measure success throughout the different phases of the program, not just at the end, and also eight months after the intervention. Furthermore, because our ultimate interest is in documenting long-term organi­zational changes, results from the previous year were also used as comparisons. We strive for two goals with our new evaluation framework: (1) to spread train­ing effectiveness information throughout the internal staff in order to drive con­tinuous improvement and (2) to measure with sufficient research rigor to illuminate true training effects.

Level I: Reaction

Two separate and independent studies of representative cohorts of students (JV = 520 managers) were done in August 1999 and August 2000 using confi­dential questionnaires and in-depth telephone interviews by a Harvard Business School professor to assess student satisfaction with content and delivery modality. Also, at the end of every Learning Lab, a company-administered student per­ception survey assesses content and delivery of every participant. Results are an­alyzed formally each quarter, and modules are changed or supplemented to respond to student needs. In phase III, student feedback is posted in Learning-Space, and Learning Lab facilitators receive individual feedback.

Blending Learning for Business Impact

Level II: Learning

In phase I, fifteen mastery tests (220 items on basic leadership and people-management principles and theory, legal and policy understanding, and other areas) are taken; all students must achieve 90 percent passing grades in order to move to phase II. Mastery is also demonstrated in phase II through collaborative role play­ing, feedback, and case studies. In phase III, mastery tests measure advanced con­tent areas that are dependent on knowledge mastered in phases I and II. Individual assessment is done using 360-degree feedback from managers, peers, and direct reports of students on competencies, managerial styles, and climate.

Level III: Transfer

An alumni assessment is conducted eight to nine months after completion (the time lag between course completion and measurement is intended to capture ingrained behavior change versus immediate posttraining effect) to measure two behavior change dimensions: (1) actual observed behavior changes and (2) changes in fac­tors that social science research has indicated are strong predictors of behavior. These behavioral precursors include gains in self-efficacy and reduction of per­ceived barriers. For new managers, perceptions of barriers to effective leadership can be powerful in influencing behavior. Hence, a large part of the training is spent on building skill in overcoming leadership barriers, as well as building intention to increase positive change activity. The assessment is one of two online, anony­mous surveys administered to all graduates (we believe this follow-up intervention reinforces the importance of desired behavior transfer). The 40 percent response rate from the first wave of alumni (637 students who completed course work in 1999) was satisfying, and the data were examined for response pattern match to other assessments to confirm the representativeness of the respondent group.

Level IV: Business Impact

Business impact attributable to training was measured in the same survey used for level III measurement. These include the extent to which students have become better leaders, the extent to which their teams have been positively affected, and types of impact on business results (such as impacts on people, teamwork, and morale; productivity and effectiveness; and customer and financial indicators). Large-scale leadership effectiveness was measured as well.

The Employee Research group conducts a global opinion survey each year in which many items critical to the corporation are measured. Recently a detailed analysis investigated the connection between leadership and the key corporate

The Handbook of Blended Learning

measurement of customer satisfaction. This piece of research was examined for high-level leadership trends and linkage to important business outcomes. These results are widely distributed and strategically used across the corporation, so our quest to iso­late training effects is confounded. For evaluation purposes, the research is important nonetheless because we believe that using multiple measures representing a variety of depth and specificity of impact can help to triangulate global training effects.

Level V: Return on Investment

Most easily measured was the cost efficiency achieved using an e-learning approach over classroom-only delivery. Cost of development returns and learner efficiencies have been quantified and tracked since the inception of the program. More diffi­cult is the measurement of the ultimate success of training: the extent to which it has a noticeable impact on business success. The bottom-line impact of leadership train­ing on business operations is notoriously hard to measure, but we endeavored using a chains of impact approach, following the trail of training effects down two paths: (1) cost avoidance and savings and (2) results enhancement. For example, tangible cost savings from cost of discrimination and harassment lawsuit avoidance are estimated by comparing internal legal action rates with other top-tier corporation rates. In ad­dition to savings, we measured return on investment (ROI) by results enhancements, such as manager estimation of business impact due to leadership improvements.

The Results

The Basic Blue for Managers intervention achieved positive effects at each Kirkpatrick level.

Level I: Reaction

All three satisfaction instruments yielded high participant satisfaction with both content and modes of delivery. The Harvard Business School professor's findings {JV = 520) indicated:

The company made significant strides in selecting learning modalities that are most appropriate to the learning situation, and implementing those modalities in an effective fashion.... The student interview results revealed unequivocal enthusiasm for [the company's] implementation of both the online and classroom components of the program... [and that the company] has appeared to recognize that when implemented appropriately, learning modalities can be synergistic, rather than competing.

Blending Learning for Business Impact

The internally conducted student survey showed that certain modules, such as coaching and climate, consistently receive the most positive scores. Summary ratings (a five-point scale, with 1 the highest score) for overall satisfaction had a mean of 1.16; " lessons learned were useful" had a mean of 1.06; the " overall experience was valuable" had 100 percent yes; and " recommend program to others" was 100 percent yes.

Level II: Learning

The program's second goal was that attainment of level II (knowledge gain) would be greater than the previous classroom-only new-manager intervention. On the fif­teen knowledge mastery tests, slightly over 96 percent of the sixty-six hundred par­ticipants to date have achieved mastery in all fifteen subject areas, and these students attained an average of 92 percent mastery on the 220 online-delivered knowledge items. Moreover, five times as much content is covered in the new year-long process than in the previous five-day new managers' classroom program. Use of the e-learn-ing architecture is regularly monitored. Based on a student population of three thou­sand students per year, nearly 500, 000 intranet Web page requests per year signifies an average of approximately 150 page requests per year per student. Since the pro­gram's inception, there have been 2.3 million page requests.

Level III: Transfer

Based on the alumni survey, significant behavior change occurred as a direct re­sult of training, with the largest behavior changes in the content areas most heav­ily emphasized in training: coaching, competencies, styles, and climate. Samples of the most powerful findings on a five-point scale of degree of change, with 1 representing the greatest degree of change, are straight talk (mean = 1.82), coach­ing as a competency (1.9), teamwork (1.92), active listening (1.94), using the in­tranet to increase leadership knowledge (1.96), team leadership (1.97), and coaching as a managerial style (1.97). These results match facilitator observations of degree of behavior change throughout the course. Furthermore, they demon­strate that the overall goals for themes of behavior change are being met.

Self-efficacy items also showed outstanding results: the graduates believe they can make a difference and they are still enthusiastic eight to nine months after completion of the course; the impact has staying power. The most powerful re­sults are " confidence in ability to be an effective leader" (1.75), representing the biggest change found in the survey; belief that " positive changes in team are within my control" (1.87); " increased knowledge of leadership capabilities and needs" (1.93); and belief that I can make a positive impact on climate" (1.95). The greatest

The Handbook of Blended Learning

barrier reduction was found for lack of understanding of how to resolve " people" issues (2.31) and difficulty in leading remote employees (2.62).

Level IV: Business Impact

Self-reports on observable changes in leadership behavior and impact on the busi­ness were also uniformly positive. Regarding " overall effect of the training on their leadership, " 8 percent of managers reported " extraordinary improvement"; 50 per­cent reported " large amount of improvement"; and 41 percent reported " some im­provement." Most frequently selected types of impact on subordinate teams that resulted directly from leadership improvement were development of the group as a team (71 percent), increased focus on strategy and goals (68 percent), morale im­provement and empowerment of staff (both 65 percent), stronger relationships among teams (53 percent), and increased productivity (50 percent).

To further establish the relationship between behavior change and business impact, eleven indexes were created from the key behavior topics and then cor­related with the measures of business impact. The strongest correlation (or " pre­diction" of impact) was found between behaviors regarding changing organizational climate and impact (correlation =.41, indicating a high degree of relationship, statistically significant at the^ <.0001 level), and leadership com­petency behavior and impact (correlation =.35, p <.0001). A factor analysis on eleven index variables yielded a three-factor solution, consisting of Impact on Business Measures, Impact on Strategic Outcomes, and Impact on Relation­ships. Impact on Business Measures correlates with many of the behavior change indexes, most notably self-efficacy (correlation = .ЪЪ, р — 0.0001) and manager­ial styles (correlation =.33, p = 0.0003).

We conclude that alumni perceive leadership improvements that relate to improvements in their business as a result of training. This finding is supported by results from the internal research study, which showed proof of linkage between leadership and customer satisfaction that were particularly compelling. A company-specific linkage model was created using structural equation modeling to demonstrate that leadership quality influences teamwork and, ultimately, cus­tomer satisfaction.

Level V: Return on Investment

Total financial cost including room and board, travel and infrastructure cost, not typically included in standard education industry accounting templates, is calculated by estimating per student cost for 128 hours of learning. Given the $8, 708 cost per student for program completion, our estimated delivery ROI is 17 to 1. (The ROI is based on the total cost of creating and deploying a module

Blending Learning for Business Impact 71

and the tangible cost benefits based on the use over the past eighteen months.) Reuse of the e-learning methodology, using content object templates or simula­tion templates, is another source of savings.

In the survey, we asked graduates to assess the first-year annual impact, in dollars, that the leadership change due to training has had on their departments. The average direct impact value that managers placed on department improve­ment was $415, 000. This leads to an ROI of 47 to 1.

Correlates of impact on revenues were based on responses from graduates (JV = 121), and impact on financial indicators was predicted by key behavior change indexes. Improvement in the areas in Table 5.1 predicted impact on financial indicators.

Together these analyses led to the conclusion that students perceive real and lasting leadership improvements directly linked to the training, which drive observable and financial value for their business.

Impact of This Learning Practice on the Organization

In making the content available to all managers worldwide over the company Internet, we have contributed to establishing a greater consistency of language, knowledge, and company culture across the globe than previously when different

The Handbook of Blended Learning

geographies developed and deployed their own separate programs for new managers.

The early success of this practice has sparked interest and enthusiasm across other parts of the company to use a similar blended approach for other company professions. Not only do all business units plan their development based on the blended four-tiered learning approach, but because the templates are reusable, they are being adapted for content beyond management development. The cus­tomization of Lotus LearningSpace to students' needs—for example, to provide a progress view for each student and an aggregate progress tracking map for administrators—and the successful deployment for such a huge number of stu­dents has helped inform and improve subsequent LearningSpace endeavors within the company.

There are other selected impacts of the new process as well:

• Adaptation of common nomenclature and conceptual models. Prior to the practice, different organizational functions and geographies used different terms and con­cepts to describe the work of managers. For example, seven dissimilar change mod­els were used within the company. That has now been replaced with one commonly agreed-on model. Common approaches and ideas have subsequently fostered more cross-functional understanding and teamwork.

• Online workplace behavior evaluations. Online 360-degree survey instruments with input from direct reports and peers on managerial styles, leadership com­petencies, and organizational climate are now used over the intranet to measure the behaviors exhibited by participants.

• Participants now develop an organizational action plan aimed at measurable improve­ments in the business. Completion of the program is dependent on the new manager's sign-off that real workplace behaviors have changed, as evidenced by completion of plan objectives. This new approach has been well received.

• Rip and read requests. Participants can use the print version of online learn­ing modules to retain the content in hard copy for later use. Nearly 50 percent of 896, 000 annual site hits are print requests. This suggests that managers are using the material while traveling and in places where online access is unavailable.

All of these impacts are believed to be long term, as this e-learning approach has been increasingly accepted throughout our company.

Lessons Learned '

IBM learning developers gained insight into effective e-learning design from third-party evaluations of Basic Blue for Managers. Most notably IBM learned:

Blending Learning for Business Impact

• Learning preferences are poor predictors of e-learning acceptance. Harvard Busi­ness School professor Youngme Moon's studies (1999, 2000) reveal it is difficult for preintervention students to express accurately their preference for a particular learning modality (e-learning) because their range of classroom experiences far exceeds that of online learning. E-learning should thus be viewed as an innova­tion with attributes unclear to its users. With this approach, we used the findings of nearly fifty years of " diffusion of innovations" research (Rogers, 1995) to inform our design and deployment.

• The relative advantage of e-learning must be salient and promoted. The degree to which the innovation is perceived as better than existing alternatives is the primary driver of its use. With our students, online learning is anytime, anyplace access, and it has the advantage of being able to focus learners on a specific skill or in­formation module desired instead of having to sit through an entire class program covering a broader set of skills or wade through a larger body of information.

• The compatibility of e-learning with existing tools, navigation, and usability is impor­tant to students. Any learning design features inducing students to regard the inno­vation as familiar increases satisfaction and speeds its adoption. For instance, if online applications are consistent with familiar interfaces and navigation, such as their e-mail, learners feel more comfortable adopting the new learning technology (Moon, 2000).

• The simplicity of an e-learning application, as perceived by its potential adopters, will speed its rate of adoption. Conversely, perceived complexity of installing " plug-ins, " commonly required for various online learning programs, is one example of how complexity slows adoption of online learning. Thus, we designed and built everything with a simple " point-and-click" mentality.

• Trialability, or the degree to which e-learning can be experimented with on a limited basis, helps dispel uncertainty and drives its adoption. Deploying e-learning features with " no risk to try" helped speed adoption. All simulators, online cases, and Web pages are neither tracked nor password required, so users could try them—and make mistakes—without feeling they were being watched. Trialability appears to be especially critical for earlier adopters, who have no precedent to follow when they adopt, unlike later adopters, who may be surrounded by peers whom they can observe and experience via " vicarious trialability."

• " Observability, " or the degree to which the innovation's results are visible to others, speeds the innovation's adoption. Some effects of e-learning are more immediate and easier to see, and these help drive future usage. For example, we learned that building and im­plementing observable and practical management skills content that can be used im­mediately by the student helps whet appetites for more e-learning. So we made these available first. Such immediate-skill gains are more quickly observed than other lead­ership skills, which typically have long-term accrued effects. These immediately per­ceived benefits help promote continued adoption of the e-learning approach.

The Handbook of Blended Learning

Moon (1999, 2000) assessed IBM managers' responses to the Basic Blue ex­periences and found that users universally extolled both learning modalities— classroom and online—without reservation and with equal enthusiasm. All respondents reported in interviews that they preferred learning the informational material (phase I, cognitive-based development) online from their own home or office rather than in a classroom setting. Representative comments from interviewees include:

" Because the information was the type of stuff I could learn on my own, there was really no reason for it to be communicated in a classroom. I think I would have been resentful if it had been dumped on me in a classroom. We're no dummies... we can learn this kind of stuff on our own."

" The key thing was the hybrid model. Rather than adopting a totally online training program or a totally in-classroom training program, they decided to take a best-of-both-worlds approach, and it really worked." " Neither phase could have worked without the other. Phase I set up phase II really nicely, and phase II would have been impossible to pull off if we hadn't done the prep work in phase I."

" It's too much information to be taught in a classroom format. You need to be able to sift through this stuff from the comfort of your own home, at your own pace."

" There's no question that the ability to work at home or in my office made some material easier.... This was a huge advantage."

It is clear that technologies and frameworks for learning have delivered much more value to IBM as an organization than just cost savings. These innovations and productivity that have come about as a result of the blended learning pro­grams have enabled IBM to be more responsive to changing business demands, develop deeper relationships within and across the extended enterprise, and de­liver financial results. We have no doubt that in the future, learning will become a competitive differentiator separating those organizations that are merely surviving from those that are leaders in the knowledge economy.

References

Kirkpatrick, D. L, (1979). Techniques for evaluating training programs. Training and

Development Journal, 33(6), 78-92. Kirkpatrick, D. L (1998). Evaluating training programs: The four levels (2nd ed.). San Francisco:

Berrett-Koehler.

CHAPTER SIX