Level 3 – Job Impact

Level 3 of the Kirkpatrick training evaluation model involves evaluating the extent to which the training participants have applied their new knowledge and skills back to their work and what effect this has had on their work performance.

It goes without saying that evaluating at this level is important if it was an aim of the training programme to improve workplace performance in some way. More broadly speaking, it can help (along with level 4 evaluations) in establishing the ‘business value’ that training has added to an organisation.

What are the key questions?

The key questions that evaluators seek to find answers to at this level include:

  • To what extent were knowledge and skills acquired through training programme used in the workplace?
  • Were there noticeable and measurable changes in the activity and performance of the training participants when back in their workplace?
  • Was the change in performance and new level of knowledge or skills sustained?
  • Were there any particular barriers to or promoters of the application of learning to the workplace?
  • What influence have factors such as the workplace environment, the ‘learning culture’, the support of managers and supervisors, target setting and the availability of on-the-job support had on the application of learning?

When combined with evaluations at other levels (particularly levels 2 and 4), level 3 evaluations can help to evidence the link between learning taking place and changes to individual and business performance. Similarly, where these changes have not materialised, evidence from evaluations at this level can play an important role in diagnosing factors which may be limiting the effectiveness of training programmes.

Choosing your evaluation questions

When choosing your specific evaluation questions at this level you will need to consider the following:

  • Which particular job behaviours and/or competencies are expected to change as a result of the training programme?
  • Is data on these job behaviours and/or competencies currently collected (eg through performance management appraisals)? If not, then where possible the initial data should be collected before the programme begins. If this is not possible, then key stakeholders should establish an estimate of pre-programme performance.
  • When, and over what timescale, will changes to the job behaviours and/or competencies be measured? You will need to bear in mind that in most cases it takes at least 3 months before the impact of learning on workplace performance becomes evident.
  • What factors other than the training programme might influence changes to each job behaviour and/or competency? For example, have managers and supervisors supported learners in applying their new skills, have there been changes to organisational structures, have performance incentive schemes been introduced, or have there been other external environmental influences?

Where possible each of the above questions should be discussed and, where relevant, agreed with key business stakeholders in advance of the training development. The conclusions can then be integrated within both training and evaluation objectives and plans. Stakeholders should also agree how success will be judged and how the evaluation results will be used.

These actions will help to ensure that the evaluation is aligned to the organisation’s strategic objectives, is feasible to carry out and can also help in gaining support and engagement from training participants and other stakeholders during the evaluation process. They will also help to motivate managers to support learners to transfer their learning to the workplace, which should in turn have a positive impact on the training outcomes.

Calculating 'Competency Gains’

As part of your evaluation at this level you may wish to calculate 'competency gains'. This shows how effective the training programme has been in improving a specified competency. It can be calculated using numerical ratings on changes to competencies using the following formula:

(Most Recent Post-learning Competency Rating minus Pre-learning Competency Rating 1 / Maximum Rating minus Pre-learning Competency Rating 1) X 100

For example, using a 10-point scale, if the pre-learning rating was 5, and the post-learning rating is 8, then you get:

((8-5) / (10-5)) X 100 = (3 / 5) X 100 = 60%

This shows that the training programme was 60% effective in improving the competency.

Note on Job competencies

The following are commonly used groupings of job competencies:

The technical skills and competencies needed to carry out the job. These are, for the most part, unique to each business unit in the company. For example, there are technical competencies needed by production line employees (eg measuring, grading, assembling and reporting) and different competencies needed by ICT technicians (eg installing, maintaining and upgrading hardware) etc. There may be some overlap of technical competencies across business units.

The personal management skills and competencies needed to manage both self and others, eg planning, organising, implementing, monitoring and reviewing.

The emotional and behavioural competencies needed to carry out a job eg self-confidence, motivation, conscientiousness, teamwork and collaboration

It is important to identify (eg through performance management processes) which competencies need development before delivering the training programme and any subsequent evaluation.

Note: It is not advisable to create new job competencies for the purposes of an evaluation without guidance from an expert.

What data collection methods can be used?

Evaluation at this level is typically carried out using questionnaires (evaluations) aimed at training participants and/or their managers, and it usually begins at least 3 months after the training programme has been completed. This allows time for the learning to be applied to and impact on the workplace. You may also consider using interviews and/or focus groups to collect evaluation data.

Evaluations focused on training participants and managers can be carried out independently of each other or they can be combined, via e.g. 360-degree / multi-rater evaluations, to generate a more comprehensive picture of the impact of a training programme on job performance.

If budget and resources allow it, you may wish to consider conducting two or more evaluations at level 3. These should ideally be at regular intervals (eg 3 and 6 months) after the training programme has been completed. Doing so will help to evidence that any impact was actually due to the training programme rather than any chance variations, and will also help you to develop a clearer picture of the impact of the programme on performance over time.

Learner View and Manager View

To help make evaluation at level 3 easier within TrainingCheck, two separate but closely related topic areas have been created within the Question Library:

  • Job Impact (Learner View) - aimed at capturing data from learners on perceived changes to their own performance and influencing factors.
  • Job Impact (Manager View) - aimed at capturing data from learners’ managers on observed changes to job performance and influencing factors.

You can find out more about using TrainingCheck to evaluate these areas by clicking on the links below: