Learning measurement, it looks like, is not an easy term to define. I couldn’t find it on Wikipedia and Google didn’t help much either. Wikipedia kept prompting me to go for learning management instead. It was perhaps my first experience where Wikipedia or Google didn’t open the right door for me. However, to my relief, I did find some pieces of information on Bersin.com about learning measurement.

Before I begin citing some ways to use LMS for Learning Measurement, let me state that Learning Measurement is a very challenging exercise and not one that can be executed easily or quickly with some calculator. The best way (but probably the most difficult one) to measure learning is to measure for the right business indicators like sales figures, productivity measures, defect ratios etc. My post is only to find simple ways where LMS could contribute towards Learning measurement in some way. However the eventual learning measurement depends on a variety of factors including, but not limited to, company policies, overall environment to perform, training process, organization will to enable channels that can lead to learning measurement, etc.

Referring back to the cartoon, the first thing most LMS users (organizations) need to be conscious that MIS reports from an LMS does not inherently imply measurement of learning. The ‘what’ in the reports is only meaningful if the ‘how’ it reached there is thought out carefully and implemented appropriately.

Here are the five ways to use an LMS for ensuring better and effective learning measurement:

1. A good start is half the work well done.  As I mentioned in one of my earlier posts it would be essential to set ‘being able to effectively measure learning’ as one of the end goals of implementing LMS (or using LMS). This will ensure that the inputs received from various stakeholders are aligned with this objective making it a critical part. In most cases this is ignored and the focus remains on ‘automating’ the learning / training function. Believe me – you will get what you will seek.
2. Look at the LMS as an overall strategy rather than just a tool. Learning measurement is already present as a process in the training function. E.g. Assessments after training sessions, On-the-job surveys, Supervisor feedback on team member performance, etc. However what gets missed out is an attempt to get that process completely mapped in to the LMS. Most LMS would offer all these modules – online assessments, surveys, etc. It is up to the organization to sit with the vendor and map out the process.At this stage, the idea is to do things manually (e.g. remember to create a survey after 3 months of training and evaluate the change in performance). This could become a problem as it may not be done reliably and consistently for all training programs and gradually it degenerates into something ad-hoc.
3. Explain the learning evaluation process clearly to the vendor. More the vendor knows, better the chances that they’ll come up with a good and innovative solution to build your process into the LMS implementation.
4. Define automated, scheduled process for conducting evaluation surveys at regular interval after the training is over. Ensure it is automated so it can be activated for all training programs. Exceptions can then be managed individually. This will trigger notifications to the user after the time period to take the survey again and could help in analyzing the change in behavior/performance after applying the training back to the job.
5. Use the assessment module of the LMS to create relevant and timely assessments which can be run either along with or independently of the training. Online assessments, if designed properly can be a powerful tool to measure learning.

All these are simple measures and if accounted for in the beginning will definitely result in a better process to generate meaningful reports. These reports will surely contribute positively, on a regular basis, to the learning measurement process which is in place.

Jim Kirkpatrick (is SMR USA’s vice president of global training and consulting, and presents workshops for and provides consulting to Fortune 500 companies around the world) and Wendy Kirkpatrick (is a director of Kirkpatrick Partners, LLC and a certified instructional designer)  have written a paper which honors Don, and takes a fresh look at Kirkpatrick’s Four Levels. You can download the paper here.  (registration required)

The whitepaper delves into the many issues associated with the use and implementation of the model to evaluate training. The authors espouse Five Foundational Principles

1. The end is the beginning – An incisive comment: ‘For decades, practitioners have attempted to apply the four levels after a program has been developed and delivered. It is difficult, if not impossible, to create significant training value that way.’ They go on to add it’s important to distinguish a plan to build effective training & the evaluation methodology from the actual collection of data.
2. Return on Expectations (ROE) is the ultimate indicator of value – Despite what they might say, training professional tend to jump into the task of designing and developing learning without a proper needs assessment. Learning professionals do nothing to track the expectations of the business stake holders. Without this critical assessment, it’s not possible to map expectations to observable and measurable success outcomes.
3. Business partnership is necessary to bring about positive ROE – The authors point out that they do not believe that training events in and of themselves can deliver positive bottom-line outcomes. They go on to add that research suggests that 70% of learning happens on the job in many different ways. In this context, they emphasize the role of the supervisor or manager after the training event. Reinforcement and coaching by such individuals directly correlates to improved performance and positive outcomes.
4. Value must be created before it can be demonstrated – Research has identified statistical correlations between the four levels. In short, it suggests that providing excellent training does not lead to significant transfer of learning to behavior and subsequent results without deliberate and consistent reinforcement. They mention that learning designers are putting most our time on designing, developing and delivering training and getting only about a quarter of the result. This is mainly because we spend virtually no time on follow-up reinforcement activities to promote behavioral change that leads to results.
5. A compelling chain of evidence demonstrates your bottom-line value – Authors mention L1 and L2 as consumptive metrics, L3 and L4 are classed as impact metrics. While working at L1 and L2, we are talking only about costs – because we only talk about number of programs, attendees (typical L1 and L2 data). If we had the data for L3 and L4 metrics, the data would point to the value the training delivers. They go on to mention that a chain of evidence connects the four levels and will actually show the contribution the training has made to the business.

Other than listing and describing the Five Foundational Principles, the authors also introduce a revised new-look Kirkpatrick Model. The authors mention that the model retains the fundamental ideas that Don (& the authors) have been communicating, it’s now said to offer a more complete illustration so both learning professionals and their business partners can create maximum training value. They further give detailed information about the model and various examples. The last example they give of ‘The Window Washers’ makes an excellent case for ‘not just training’.

Anyone interested in evaluating learning programs based on Kirkpatrick’s Four Level model should read this paper; if offers great insights, I’d recommend it.