The E-learning Curve at Edublogs

The Retrospective Pre-test method is an interesting if controversial data collection design. I will examine its advantages and negative aspects here, as well as an argument for including this strategy in in assessing organisational learning initiatives. The retrospective pre-test method (see Table 1 is an extension of the one-shot case study design where data are gathered from participants following the learning intervention (for example, a series of non-formal Information Sessions).

Table 1 Retrospective Pre-test Design

However, participants (N) report on their knowledge, understanding or skills (Observation O1) after the intervention (X), and then also reflect and answer (O2) what they believe their understanding or skill was before the intervention. Rockwell & Kohn (1989) applied this method to testing the effectiveness of achieving program outcomes when interventions such as training programs are implemented and concluded that “using a post-then-pre design [retrospective pre-test] to identify self-reported behavioural changes can provide substantial evidence for program impact”. The retrospective pre-test enables researchers to reduce the response shift bias - defined by Klatt & Taylor-Powell in their 2005 paper Synthesis of Literature relative to Retrospective Pretest Design as

the change in the participant’s metric for answering questions from the pre-test to the post-test due to a new understanding of a concept being taught

- because the participants are able to give pre-test responses which are based on a post-intervention frame of reference.

Using the retrospective pre-test, response shift bias can be reduced increasing the likelihood that the observable results are due to intended intervention effects (Pratt, McGuigan & Katzev, 2001). This proposition is complicated and not without its critics – Theodore Lamb (2005), an advocate of the retrospective pre-test method, describes it as “an imperfect but useful tool.” Robson (2002, p.139-141) expresses concerns that used purely as a quasi-experimental method, this strategy could lead to an internal validity threat through regression to the mean.

References:

Klatt, J. and Taylor-Powell, E. (2005) Synthesis of Literature relative to RetrospectivePretest Design. Presentation to the 2005 Joint CES/AEA Conference, Toronto [Internet] Available from: http://www.citra.org/Assets/documents/evaluation%20design.pdf [Accessed 5th August 2007]

Lamb, T. (2005) The Retrospective Pretest: An Imperfect but Useful Tool. Evaluation Exchange. [Internet] 11(2). Available from: http://www.gse.harvard.edu/hfrp/eval/issue30/spotlight.html [Accessed 21st March 2007]

Pratt, C.C., McGuigan, W.M., & Katzev, A.R. (2000). Measuring program outcomes: Using retrospective pretest methodology. American Journal of Evaluation, 21(3). 341-349.

Robson, C. (2002) Real World Research. 2nd ed. Oxford: Blackwell Publishing.

Rockwell, S. K. & Kohn, H. (1989) Post-Then-Pre Evaluation. Journal of Extension. [Internet] 27(2). Available from: http://www.joe.org/joe/1989summer/a5.html [Accessed 21st March 2007]

–