Collecting Data

What you choose to evaluate and the amount and type of data you collect will inevitably depend on the reasons you want to conduct the evaluation. For example, if you want to identify future learning needs, measure how far learning is being applied to the workplace, or find out if a programme produces a Return on Investment (ROI), each of these will require the collection of different sets of evaluation data .

Your data collection methods will also depend on the resources available. For example, you may want to evaluate at all four of the Kirkpatrick levels, but this can be a lengthy and costly process. Where resources are limited you may wish to limit comprehensive evaluation to training programmes that require the greatest investment and/or are the most strategically important for the organisation.

Your decisions on data collection will also depend on the level of skills and knowledge in, and commitment to training evaluation within the organisation, and on whether the methods employed are likely to produce relevant and useable results.

Whichever data collection methods you choose, it is important to recognise that training evaluation is generally more about collecting evidence rather than proof. The main reason for this is that many of the changes measured are complex in nature and can be influenced by a number of factors other than the training programme. For example, changes in organisational performance may be as equally influenced by the introduction of a new organisational structure or process as by the learning from a programme.

It is likely that in order to convince others in your organisation about the impact of the training programme, and to get decision-makers to do something about the results, you will need to collect a range of both ‘hard’ quantitative evidence and ’soft’ qualitative supporting evidence. For example, to evaluate changes to organisational performance you may want to collect both data on key performance metrics and supporting statements from learners and their managers concerning their views and observations on how the training programme has influenced work performance.

Overall your data collection activities will need to be something of a balancing act, both brief and focused enough for respondents to want to contribute, but also broad enough to gather the evidence you need to make a convincing case for the impact of the programme.

Keeping to the following ‘Golden Rules’ will help to ensure that the data collection process is successful:

• Gain management support for and engagement with both the training and evaluation processes early on.
• Ensure that respondents and other stakeholders clearly understand their roles and responsibilities in the evaluation process.
• Think about metrics throughout the life cycle of the training programme - focus on metrics that are meaningful to the business and are already being captured.
• Where practical, use consistent methods to compare the outcomes of similar programmes.
• Ensure that the evaluation process does not disrupt the situation or objective being evaluated.
• Encourage and support respondents to be as open and honest in their training feedback as possible.
• Ensure that confidentiality is maintained where appropriate, eg by not revealing learners’ names within reports or by using external people to facilitate focus groups etc
• Ensure that there is a clear understanding of how the results will be used.
• Be flexible in your approach to data collection. If something is not working, do not be afraid to change it.

The following sections provide further guidance on data collection methods.

• What data collection methods should I use?
• When should data be collected?
• How can I create effective evaluation questions?
• When should I use sampling techniques?
• When should control groups be used?

Back