A new model of evaluation’ by Bob Little, Training Journal, May 2012

The author positions himself as a writer on corporate learning and the article is well written.  However, there is an element of sycophancy in the way he reports the latest work in the area of learning evaluation by John O’Connor.  There is no doubt that the ideas behind O’Connor’s ‘Results Assessment (RA) Model’ have some merit but to position it as breakthrough thinking is an insult to those who have battled with the thorny issue of evaluation for many years.  

The RA model is based on the idea that any evaluation of learning effectiveness must be based on results which are linked to business goals.  It claims to go beyond Kirkpatrick but there seems little to distinguish his thinking from Kirkpatrick level 4.  There are a number of references to research carried out by O’Connor but no convincing evidence and no examples of its application in practice.

There is great emphasis on the importance of stakeholder engagement and understanding business goals, which is valid but hardly new to those of us who are involved in targeted, tailored learning solutions.

The frameworks that are offered make sense but are quite similar to what we see in many top companies as a matter of course when they launch major programmes, for instance;

-        -    Think and act with customers in mind

-         -   Focus on performance outcomes

-         -   Measure results that add value

Or how about these four RA processes?

-          - Alignment

-          - Planning

-         -  Data Collection/Analysis

-         -  Reporting results

My reluctant conclusion is that this is not ‘beyond Kirkpatrick’ as the author claims; it is just another vain attempt to find the Holy Grail of learning evaluation.  It could be argued that this is harmless but I believe there is potential damage from the suggestion that this kind of evaluation (and here I quote Dr O’Connor) is ‘easy to do’.  It is not easy because it is rarely possible to isolate the impact of learning from the many other factors driving business results.  How can you know what those who receive learning would have done without the benefit of their increased knowledge and skills?  This fundamental challenge is not even acknowledged in the article.

This is not to suggest that we should not try to measure outcomes where possible and cost effective and the sort of questions posed by O’Connor’s framework may be helpful to those who have never tried to get beyond Kirkpatrick’s first two levels.  But to suggest that this RA model is new or makes things easy is misleading.  And the author makes things worse by a failure to question, challenge or acknowledge the model’s limitations.  

Read the article
http://www.trainingjournal.com/feature/2012-05-01-a-new-model-of-evaluation/