Level 4 – Business Impact

Level 4 is the final level of the Kirkpatrick model of training evaluation. It seeks to measure changes in business performance that have come about through learners applying their new learning to the workplace.

For organisation leaders in particular, this may be regarded as where the bottom-line value of training lies. However, gathering evaluation data at this level is a complex task which involves measuring the impact of training on business performance measures as reported by learners’ managers and other key stakeholders.

The specific performance measures used will depend on the individual organisation and, crucially, on the agreed objectives and expectations of outcomes from the training programme.

Examples of measures include changes to:

  • productivity/output rates
  • sales volumes
  • employee turnover rates
  • customer satisfaction and retention rates
  • number of customer complaints
  • wastage rates
  • non-compliance
  • rate of accidents per year
  • number of sick-absence days per month
  • number of cancelled training days/sessions
  • recruitment costs.

In many organisations some or all of these will be in place alongside other performance measures within existing management systems and reporting.

Note: Financial data collected in response to level 4 questions can also be used as a basis for calculating the Return on Training Investment.

Taking account of influencing factors

It is often commented that isolating the impact of training on business performance is difficult. The main reason for this is that there are likely to be a number of other factors which will have an influence on how the organisation performs in any one area at any given time. For example, the organisation may have introduced new working practices or performance incentives, or there may be new competitors, legislative or environmental factors that influence performance.

Within TrainingCheck we have taken a pragmatic approach to this issue by creating example questions and evaluations which ask key stakeholders to identify any major influencing factors on performance changes and to estimate the relative influence of the training in relation to these. The proportion of influence ascribed to the training programme can then be taken into account when considering the evaluation results and be applied to any quantitative/financial data.

For example, if gains in business performance have resulted in a financial benefit of £5,000, and it is estimated that the training programme is responsible for 50% of the change in performance (the other 50% being due to other influencing factors), then the total financial benefit attributable to the training can be calculated as £2,500 (ie £5000 x 50%). It is this figure that can then be used, for example, as part of a Return on Training Investment calculation (see the guidance on Calculating Return on Training Investment).

While this is clearly not an exact science, relying on the estimates of those who are in the best position to make them will result in data that is likely to be taken seriously by relevant decision makers.

Is it always advisable to evaluate at level 4?

It may not always be advisable to evaluate the impact of training at this level. The decision whether to do so will ultimately depend on factors such as the length and type of training, and, most importantly, on what the training was designed to achieve. For example, for some compliance training it may be prudent to simply measure subsequent compliance rates, while in other cases (eg short, low-cost programmes), it may be possible get a great deal of the information you need by focusing on different kinds of satisfaction and learning measures.

As mentioned elsewhere, there may also be practical issues with attributing results at this level to training. For example, if an organisation-wide quality improvement initiative has recently been introduced of which the training programme was just one small part, then it makes no sense to try to evaluate the impact of the training programme alone. Instead the impact of entire quality improvement initiative should be evaluated. The important thing is that for an evaluation of a training programme at this level to be meaningful the training programme must be largely separable from other significant influencing factors.

Engaging managers

It is vital from the outset to engage managers in identifying training needs, in establishing expectations regarding training outcomes, in setting training and evaluation objectives and in agreeing what will happen with the evaluation results. Among other things, these actions will help to:

  • motivate managers to support learners to transfer their learning to the workplace (and thereby potentially improve business performance)
  • ensure managers and other stakeholders are engaged in the evaluation process (this is especially critical when collecting job and business impact data)
  • ensure that the evaluation outcomes are meaningful to the organisation at a business/strategic level.

Seeing the big picture

As with the other Kirkpatrick levels, level 4 should not be seen in isolation. Feedback from other levels can provide important data that may either support any conclusions drawn about the linkage between training and business performance, or, if necessary, help to diagnose potential causes where there has been little or none of the expected impact.

Choosing your evaluation questions

When choosing your specific evaluation questions at this level you will need to consider the following:

  • Which performance indicators (PIs) and/or other business impact measures are relevant to the training programme? If the training programme development has been linked to specific business needs, then you should be able to identify the relevant performance areas relatively easily. However, as the programme may have a wider impact than intended you may also want to try to capture unexpected impacts by including a variety of other PIs/business impact measures (see the example evaluations and questions provided within TrainingCheck for some ideas).
  • Is data on these PIs/business impact measures currently collected for the group, team, department or organisation? If not, then pre-learning data will either need to be collected before the programme begins, or, if this is not possible, then key stakeholders should where possible establish an estimate of pre-learning performance. (Note: When using actual pre-learning data, this should ideally have been collected over a long enough time period to be able to determine whether any changes are likely to be for seasonal and/or cyclical reasons)
  • When, and over what timescale, will changes to the PIs/business impact measures be measured? Bear in mind that it in most cases it takes some time (usually at least 3 months) before the impact of training on workplace performance becomes evident.
  • What other factors might influence changes to each PI/measure? For example, have new working practices or performance incentives been introduced, are there any new competitors, or are there any legislation or environmental factors that might influence performance? You will need to bear in mind that respondents may be asked to estimate the percentage of influence of these factors on changes to performance.
  • Will you want to be able to calculate the Return on Training Investment (ROI) from the programme? If so, respondents will need to assign financial values to changes in performance where possible.
  • What other quantifiable and/or 'intangible' (non-quantifiable) benefits might there be as a result of the training programme and how might these be measured or captured?

What data collection methods can be used?

Data collection at level 4 can be carried out through evaluations, interviews, and focus groups involving managers and other stakeholders, and/or through desk research.

If budget and resources allow it, you might consider conducting two or more evaluations at this level. These should ideally be at regular intervals (eg 3 and 6 months) after the training programme has been completed. Doing so will help to evidence that any impact was actually due to the training programme rather than any chance (eg seasonal) variations, and will also help you to develop a clearer picture of the impact of the programme on performance over time.

Evaluating Business Impact using TrainingCheck

Creating your evaluation

When creating your evaluation you will be able to:

  • copy an existing evaluation (eg you can choose to copy the example evaluations provided)
  • create a new evaluation from scratch
  • choose from the questions within the ‘Participant Reaction’ sections of the Question Library
  • copy individual questions from existing evaluations
  • create your own questions. (See also the guidance on creating effective questions)

The evaluations you create should generally be short (between 5 and 15 questions). Longer evaluations tend to have much lower response/completion rates. Therefore question choice is very important.

Once you have created an evaluation it is advisable to pilot test it before deploying it with the target group.

Deploying your evaluation

Evaluation respondents at this level should include anyone who has access to relevant performance data. This can include learners’ managers, line-managers, supervisors, team leaders, Human Resources, Finance, Production departments etc.

At this level evaluations should usually be deployed a suitable period of time (usually about 3 months) after the completion of the training programme to allow time for learning to be transferred to the workplace .

The deployment options (via the ‘Collect Responses’ page) are to:

  • send your evaluation to contacts in your Address Book
  • place a link to your evaluation in an email using your usual email program (eg. Outlook)
  • place a link to your evaluation on a web page .
  • launch the evaluation immediately so that you can manually add data directly into it ('Add Data Manually' button) - useful, for example, if you have collected evaluation data through paper based evaluations, interviews, or focus groups

Please note: Evaluations can be printed so that they can be completed manually. Responses from printed evaluations can be uploaded to TrainingCheck via the 'Add Data Manually' function.

You can deploy the evaluation as many times and using as many of the different methods as you wish.

It will be important to consider the timing to ensure a good response rate. For example, does your evaluation coincide with other surveys, or is it a particularly busy time for respondents?

Data analysis and reporting

Once you have collected the data from respondents or manually entered data, you can view the responses by clicking on the ‘Analyse icon. You will be able to filter the responses according to criteria you choose, and download responses as CSV (Excel) or XML files. You will also be able to create custom reports (via the My Reports and ROTI Calculator pages) and share these with key stakeholders.

You may want to discuss the results of the evaluation with learners, the trainer(s), and learners’ managers. This can be an effective way of encouraging, and identifying potential barriers to, the transfer of learning to the workplace.

As with all other levels of evaluation, it is vital that the outcomes of the evaluation at this level are acted on. Not doing so will, at a minimum, undermine the credibility of the evaluation process.

Back

Article Quick Jump :