1. Determine the current state and needs of the learner
   2. Define the end goal of instruction
   3. Develop a learning intervention to assist in the acquisition of new skills, knowledge or expertise.

Once these stages have been undertaken, the learning materials need to implemented and improved, or evaluated, and that’s what we’re going to investigate today.

IMPLEMENTATION

The implementation phase of the instructional design process is when the instruction as designed is actually delivered. Materials are collected, the environment is arranged, and the course is prepared for delivery.

For some courses, (particularly in e-learning environments) a pre-test is made available.  This can determine whether the learner can already perform some of the objectives (or larger units of instruction). If that is the case, certain modules or part of the curriculum can be skipped. A pre-test can also determine whether a learner is undertakes an individualized learning path – a basic form of adaptive learning design.

When designing a new course or learning program, the design and materials should be tested during a pilot course. The pilot enables the instructional designer and trainers to have an opportunity to review and revise the course before it is fully implemented. The instructor who actually delivers the training must be knowledgeable and skilled in the competency-based training approach used by the instructional designer when designing the course.

IMPROVEMENT PHASE

A common useful way and comprehensive way of looking at improvement is in terms of the Kirkpatrick model:

Donald L Kirkpatrick first published his ideas on evaluating learning in 1959 in a series of articles in the US Training and Development Journal. The articles were subsequently included in his book Evaluating Training Programs (originally published in 1975; I have the 2006 edition).

In this text he outlined and further developed his theories on evaluating culminating in the Four-Level Model, arguably the most widely used and popular approach for the evaluation of training and learning. Kirkpatrick’s Four-level model is now considered an industry standard across the HR and training communities (see Table 1).

Table 1 Kirkpatrick’s Four-level Model

According to the model, evaluation should always begin with Level One, and as time and budget allows, should move sequentially through Levels Two, Three, and Four. Information from each prior level serves as a base for the next level’s evaluation. Each successive level represents a more precise measure of the effectiveness of the training program, however, each level also requires a more rigorous and time-consuming analysis.

Kirkpatrick’s model aside, the course materials, objectives, delivery, test items, audience profile – all of the instructional components in fact – need to be evaluated. Assessing these elements regularly is especially important for repeating courses or asynchronous courseware.  As an example, if substantial majority (70% or 80% of the learners) fail a criterion test item, it would be reasonable to look again at the design of the related piece of instruction.

_________

References:

Kirkpatrick, D. & Kirkpatrick, P. (2006) Evaluating Training Programs. 3rd ed. San Francisco, CA: Berrett-Koehler Publishers, Inc.

–

The methodology I’ll be describing is a generalised account of the type work I carry out when developing and implementing non-formal learning initiatives in organisations – I invite you to argue, discuss, praise etc. as appropriate.

Typically, I consult for small- to mid-sized enterprises in Ireland. The type of learning interventions being discussed here are what I call ‘Information Sessions.’ Information Sessions are learning events designed to disseminate a broad range of knowledge and information about an organisation and its products. The title ‘Information Session’ is carefully chosen: these events are not formal training (see this blog entry), nor are they informal learning; they are informational by definition and design, though they are developed according to a strong instructional design process. The Info Session events are hosted as synchronous, SME-led presentations. The content is repurposed post-event for asynchronous distribution on-demand over the Web, as well as being distributed to learners on DVD-ROM. Information Sessions are distributed in the following formats:

Table 1 Information Sessions format

In the first phase of data collection, quantitative data are collected on every attendee participating in both the live in situ and the synchronous and asynchronous distribution of the events. An E-learning Readiness Survey in the second phase of research is distributed to internal experts who fall under the aegis of what William Horton (2006, p.107) describes as “those who should know” – managers, learning and knowledge professionals, and key stakeholders in the organisation. A representative sample (proportionate to staff numbers) of employees participate in Phase III of the research design. Informed consent is obtained from each contributor by asking them to complete a sign-off statement form in the questionnaire (we’ll get back to the questionnaire) and asking their assent to use their responses before follow-up interviews are conducted.

Note: As the research design follows a case study approach and is based on a learning initiative in a single organisation, I am cognisant of the fact that the potential exists for “individuals’ interests to be affected” (Robson, 2002, p.501) by participating in the research, particularly if their opinion deviates from the corporate policy.

References:

Horton, W. (2006) So how is elearning different? IN: Kirkpatrick, P. & Kirkpatrick, J. Evaluating Training Programs. 3rd ed. San Francisco, CA: Berrett-Koehler Publishers, Inc.

Robson, C. (2002) Real World Research. 2nd ed. Oxford: Blackwell Publishing.

–

However, as Mr. Clark points out, rejecting Kirkpatrick doesn’t provide a answer to how else we should evaluate the effectiveness of learning (particularly non-formal learning) so in lieu of another approach I will carry on discussing how I think Kirkpatrick can be used to assess the effectiveness of NFL initiatives.

Pay attention, here comes the science bit…

I’ve already discussed why I think Kirkpatrick’s guidelines for evaluating learning is incompatible with the assessment of non-formal learning (particularly in the context of NFL), and today I’ll discuss how you can begin to counter these deficiencies. Like Steven Kerr, CLO at Goldman Sachs, Kirkpatrick Level 3 is the most interesting to me; in the context of NFL, Level 2 cannot be used as an instrument for evaluation: with no summative assessment at the end of an NFL session, you cannot evaluate knowledge transfer by traditional testing methods.

To understand why NFL cannot be assessed at Level 2, look at what Michael Eraut says in Non-formal learning, implicit learning and tacit knowledge (2000). Eraut makes a clear distinction between his interpretation of the term ‘non-formal learning’ and what others including Scribner (1988), Conner (2002) and Cross (2003) would call ‘informal learning’ – what Eraut terms ‘incidental learning’ or ‘the acquisition of knowledge independently of conscious attempts to learn and the absence of explicit knowledge about what was learned’ (Reber, 1993, quoted by Eraut 2000, p.12)

This is, he argues, because most workplace learning takes place outside formal learning contexts, and informal learning carries with it connotations of

so many other features of a situation, such as dress, discourse, behaviour, diminution of social differences – that its colloquial application as a descriptor of learning contexts may have little to do with learning per se.

Not only does the term carry unwanted and confusing implications, but it is too vague to be of any real utility. For Eraut, an analysis of learning must focus on activities and the outcomes that that contribute to significant changes in capability or understanding. In a sense, Eraut does not define non-formal learning; rather, he defines the characteristics of formal learning (p.12) as:

1. A prescribed learning framework
2. An organised learning event or package
3. The presence of a designated teacher or trainer
4. The award of a qualification or credit
5. The external specification of outcomes.

The implication of this categorisation is that any learning that does not exhibit all of these characteristics should be classed as either informal or non-formal learning. Some reviewers (Colley, Hodkinson & Malcolm, 2002) make the point that Eraut does not make clear what the status is of learning in situations that meet some, but not all, of his ‘formal’ criteria. My interpretation of his characterisation is that the very nature of a formal activity – “following or according with established form, custom, or rule” (Merriam-Webster Online, 2007) validates Eraut’s description.

I am particularly interested in exploring the relationship between Level 1 (Reaction) and Level 3 (Behaviour); as Kirkpatrick & Kirkpatrick (2006, p.22) assert “learning has taken place …when one or more of the following occurs: Attitudes (sic) are changed. Knowledge is increased. Skill is improved. One or more of these changes must take place if a change in behaviour is to occur.” In this context, I believe a mixed method of quantitative data-gathering and a more naturalistic or qualitative information collection approaches can be employed to capture data that will assist in evaluating the success of a non-formal learning initiative.

And it is this approach that I will be discussing tomorrow.

References:

Colley, H. Hodkinson, P. Malcolm, J. (2002) non-formal learning: mapping the conceptual terrain. a consultation report [Internet] Available from: http://www.infed.org/archives/e-texts/colley_informal_learning.htm [Accessed 28th January 2008]

Conner, M. (2002) Informal Learning: more than “What we do between classes”. [Internet] Available from: http://www.productivity-solutions.net/documents/
informallearning-mconner-071202.pdf [Accessed 19th October, 2006]

Cross, J. (2004) An informal history of eLearning. On the Horizon [Internet] 12(3). pp.103-110. Available from: http://www.emeraldinsight.com/Insight/viewPDF.jsp?
Filename=html/Output/Published/EmeraldFullTextArticle/
Pdf/2740120301.pdf [Accessed 20th February, 2007]

Kirkpatrick, J. & Kirkpatrick, P. (2006) Evaluating Training Programs. 3rd ed. San Francisco, CA: Berrett-Koehler Publishers, Inc.

Scribner, S. (1988) Mental and manual work: An activity theory orientation. In E. Tobah, R. J. Falmagne, M. B. Parlee, L. M. Martin & A. S. Kapelman (Eds.), Mind and Social Practice: Selected Writings of Sylvia Scribner. (pp 367-374).Cambridge: CUP

Whitney, K. (2004) Steve Kerr: Managing the Business of Learning [Internet] Available from: http://www.clomedia.com/content/
templates/clo_cloprofile.asp?articleid=582&zoneid=4 [Accessed 28th February 2008]

–

In this text he outlined and further developed his theories on evaluating culminating in the Four-Level Model, arguably the most widely used and popular approach for the evaluation of training and learning. Kirkpatrick’s Four-level model is now considered an industry standard across the HR and training communities (see Table 1.1).

Table 1.1 Kirkpatrick’s Four-level Model

According to the model, evaluation should always begin with Level One, and as time and budget allows, should move sequentially through Levels Two, Three, and Four. Information from each prior level serves as a base for the next level’s evaluation. Each successive level represents a more precise measure of the effectiveness of the training program, however, each level also requires a more rigorous and time-consuming analysis. As we shall see, the characteristics of NFL mean that not all of the four levels can be applied to the evaluating of this approach to learning.

In Chapter 5 of Evaluating Training Programs (2006) Kirkpatrick and Kirkpatrick stress the significance of measuring learning, because “no change in behaviour can be expected unless one or more of the learning objectives have been accomplished” (p.42). They set out “helpful” guidelines for the measurement of learning:

Table 1.2 Guidelines for evaluating learning

This approach is incompatible with the assessment of non-formal learning (particularly in the context of e-learning) in a number of ways:

1. The singular nature of each non-formal learning event means that no control group can exist to measure a difference between it and the “experimental group” (p.43).
2. Similarly; a significant number of the learners that access NFL do so asynchronously, via a number of different learning channels and over a broad time span in a relatively ad hoc manner. This “just enough, just in time” aspect of non-formal learning is one of its strengths, but it was unrealistic to attempt to measure learning using an experimental method in such an environment.
3. The lack of summative assessment in non-formal learning precludes both pencil-and-paper and performance testing to measure learning.
4. The distributed nature of access to these NFL interventions, over both time and location makes getting a 100 per cent learner response rate practically impossible.

I’ll be discussing approaches to overcoming these challenges in my next posting.

References:

Kirkpatrick, J. & Kirkpatrick, P. (2006) Evaluating Training Programs. 3rd ed. San Francisco, CA: Berrett-Koehler Publishers, Inc.

–

This assists in providing a context, and helps define the NFL initiative’s value to the learners and to the organisation. An added benefit of the case study approach is that it enables the learning practitioner to employ a number of different data collection methodologies: participant observation, archive data collection, surveys, and interviews to answer the questions outlined in Table 1. The main purpose of using a number of data collection methodologies was to capture a range of quantitative and qualitative data, and to establish a means of triangulating that data to enhance validity and “trustworthiness” (Russ-Eft & Preskill, 2001, p.153).

Table 1 Questions the case study approach attempts to answer

Quantitative Data

Qualitative Data

Who attends live “in situ” the learning event? (dept., role, skill level, location)

Description and chronology of a the learning event

Who views synchronous live stream of the learning event? (dept., role, skill level, location)

Transfer of new knowledge to the job (Kirkpatrick – Level 3)

Who views asynchronous on-demand version of the learning event? (dept., role, skill level, location)

Does the pedagogical approach used in the learning event fulfil participants’ needs?

Who requests asynchronous DVD version of the learning event? (dept., role, skill level, location)

Do non-formal learning event enhance individuals’ and organisation performance? (Kirkpatrick – Level 3)

Participants satisfaction with the learning event? (Kirkpatrick – Level 1)

What is the Business Impact of the the learning event or series of events over time (Kirkpatrick – Level 4)

What do attendees know that they didn’t know before? (Kirkpatrick – Level 2)

References:

Russ-Eft, D. & Preskill H. (2001) Evaluation in Organizations: A Systematic Approach to Enhancing Learning, Performance and Change. New York, NY. Perseus Books.

–

Figure 1 the Learning Continuum

and that learning is situated in a larger socio-cultural context – a cognitive web – that provides the knowledge-centric framework within which learners operate (see Figure 1.2).

Figure 1.2 Knowledge-centric View of Knowledge (after Marc J. Rosenberg)

Similarly, NFL shares characteristics with both informal and formal learning, but is differentiated enough from both of these approaches to merit it’s own category (see Table 1)

Table 1 Definition of learning types

Given the non-formal nature of the learning (and by extension a paucity of the type of assessment metrics available when evaluating the efficacy of formal training programmes) I am going to explore the challenge of assessing non-formal learning in the context of Kirkpatrick’s 4-level model, particularly between Level 1 (Reaction) and Level 3 (Behaviour); as Kirkpatrick & Kirkpatrick (2006, p.22) assert “learning has taken place …when one or more of the following occurs: Attitudes (sic) are changed. Knowledge is increased. Skill is improved. One or more of these changes must take place if a change in behaviour is to occur.”

More tomorrow…

References:

Kirkpatrick, J. & Kirkpatrick, P. (2006) Evaluating Training Programs. 3rd ed. San Francisco, CA: Berrett-Koehler Publishers, Inc.

Rogers, A. (2004) Looking again at non-formal and informal education – towards a new paradigm [Internet] Available from: http://www.infed.org/biblio/non_formal_paradigm.htm [Accessed 30th January 2008]

Rosenberg, M. J. (2001) e-Learning: Strategies for Delivering Knowledge in the Digital Age London: McGraw-Hill.

See Also:

Holford, J. Patulny, R. & Sturgis, P. (2006) Indicators of Non-formal & Informal Educational Contributions to Active Citizenship [Internet] Available from: http://crell.jrc.ec.europa.eu/active_citizenship.htm [Accessed 24th February 2008]

–

As organisations are at root, cognitive enterprises, and the sum of the knowledge of the organisation is expressed collectively through the behaviour, skills and attitudes of its employees, the organisation is itself a cognitive entity. One of the ways that such an entity continually constructs itself is through the learning processes its members engage in.

Situated, experiential learning, delivered:

1. synchronously in a social setting where workers can engage in a dialogue with More Knowledgeable Others, subject matter experts and their peers,

   or,

2. asynchronously accessed when the individual learner has a requirement to acquire information and knowledge relevant to a task at hand

in relevant, context-specific formats, whether in a classroom, live online, on-demand over the web, or via an easily accessible format such as a DVD-ROM to suit a range of learning styles and environments, creates the conditions for “meaningful learning” (Jonassen, 2001) to occur.

Providing workers with a knowledge-centric learning and performance architecture, and structured, goal-oriented content, gives them the motivation or intention to learn, enables them to process “raw” information into actionable knowledge on an ongoing basis, and regularly inhabit a scaffolded environment which encourages the learner’s development. Digitally-based technologies are the optimal mediator for this learning process, which is called ‘non-formal’ in recognition of its flexibility and accessibility for learners, and also because of its pedagogical structure and outcomes-focus.

Over the next week or so, I’m going to look at ways to quantify the learning effect of non-formal learning its target audience beginning with an overview of that stalwart of evaluation, Donald Kirkpatrick.

References:

Argyris, C. and Schön, D. (1974). Theory in practice: Increasing professional effectiveness, San Francisco: Jossey-Bass.

Jonassen, D. H. (1996) Computers in the Classroom: Mindtools for Critical Thinking. Upper Saddle River, NJ. Prentice Hall Inc.

–