‘Proving your worth’ by Jenny French, Training Journal, June 2010

I could have chosen at least three other articles on evaluation of training in the last month, which makes me wonder if this is a bandwagon that too many consultants are jumping on, knowing that it is a sure route to getting an article published. We all know that evaluation is important but is there anything new to say?

I chose this article because of one fundamental difference from the others; it is written by a senior learning professional in a well-regarded company and not by yet another self-serving consultant. Jenny French is Group Head of Leadership Development at BT and her article describes how she introduced a new approach to evaluation as part of a plan to communicate the value of their Leadership Development programmes. This creates a good impression at the start of the article; unlike many others BT have thought through the reasons why they want to evaluate training rather than seeing it as an end in itself.

One interesting comment early in the article is that BT would ideally like to turn the clock back and start the whole design process with ROI in mind. This raises the critical issue that has hardly been addressed in other articles, whether evaluation should be pre or post the training activity. My view is that there is no doubt in a perfect world; evaluation should be both pre and post and the actual outcome should be compared to plan. The parallel with capital expenditure evaluation is powerful; how many companies would wait until the money has been spent before evaluating it? With capital proposals, the problem is reversed – companies often do great evaluations before spending but never bother to compare with actual outcomes.

So is there anything new about BT’s approach? Their first important step has been to develop common measures for all their leadership development programmes and show these as a ‘dashboard’ which presents measures in digestible form. In addition to standard measures from course evaluation forms, they have tried to introduce the impact on key value drivers like revenue, cost, quality, customer/employee satisfaction, all the measures that you might see in a balanced scorecard.

The dashboard is analysed during monthly meetings and then there is a further stage of ‘digging deeper’ through in-depth interviews which are designed to further assess business impact. Two points occurred to me here; firstly the cost of doing this must be substantial unless you are being highly selective; secondly, how objective are the responses? We have had some experience of post-course follow-up workshops and interviews at MTP and have reason to be cautious.

The author reports that sometimes the interviewers have to dig deep to uncover monetary impact that was not immediately obvious; this enables stories to be revealed and ROI to be assessed. This may well work in some cases but I can recall when running follow-ups for MTP that you can find yourself pushing people into claiming benefits; they may eventually comply because they want to be helpful or because it seems the right thing to do. Certainly the examples quoted by BT show how difficult and arbitrary such assessments can be – for example’ ‘I became more proactive and was able to bring in 5% of revenue earlier’. This raises the hoary old question – how do you know what would have happened if you hadn’t attended the course?

This is not to devalue what BT has done; it sounds like an excellent effort and the idea of making a film to summarise and highlight evaluated benefits is an original one. But I would like to have seen more understanding of the costs and other resource implications of the assessment process and acceptance of the fact that, while it may be viable for a high profile and high investment leadership programme, the use of such evaluation methods may not be practical in a wider context. And that any ROI calculation on the basis of the processes described has to have a health warning about assumptions and limitations.

To read this article go to:
http://www.trainingjournal.com/tj/2919.html