Formative Evaluation

You should choose one of the project below to carry through a full formative evaluation.

Your group are members of an instructional design team. Your team has followed the Morrison, Ross, and Kemp model of instructional design to produce an elearning product. The instructional design documentation (also known as design documentation) used to produce the elearning product and the elearning product are listed below.

1. Rapid Prototyping elearning module | Loren Breland

a.and the appeal of the product.

b. Audiences

c. Issues

d. Resources

e. Evidence

f. Data-gathering techniques Your team will need to plan and create the instruments necessary to conduct your formative evaluation. Your team will need instruments that align with the different stages (i.e., SME review, small group trial) of your formative evaluation. All of your formative evaluation may be completed at a distance. It can be completed asynchronously or synchronously. You choose what works best for you.

1. Expert/SME Review (at least 1 SME) You will need an SME Notes form for your SME Review. At the minimum, the SME should review your elearning module. You need your SME to complete the SME Notes form you created and submit it back to you for you to analyze and report.
2. Small group trial (8-10 learners) For the small group trial, you will need to collect data for (1) the reactions of your learners and (2) the improvement in your learners learning. The first one is very often completed with an attitude survey, and the second is collected with an assessment of learning. Your team has previously produced the objective assessment items located in your documentation. Since you may not be able to collect the practice items data from your small group testers as it is embedded in your elearning module, you may wish to create a quick answer sheet for them to use or you may wish to just copy the items onto a one-page quiz (e.g., Google Forms, , SurveyMonkey) that they can submit back to you. You will need the assessment data for preinstruction and postinstruction, and you will need the reactions of your learners (in an attitude survey) for postinstruction.

g. Analysis For the SME review, you can summarize the text comments. For the small group trial, you should plan analyze (1) the reactions of your learners with mean scores and (2) the improvement of your learners’ learning by objective with percentages correct (i.e, frequencies, So what percentage of learners got each item correct?) for preinstruction and postinstruction.

h. Reporting See below.

2.You will use the following headings/sections for reporting your formative evaluation as part of your documentation format described in the text:

1. Exective summary Three to five paragraphs. This section comes first, but you will write this last. Write in past tense.
2. Purposes

a. Purposes statement The purpose of your team’s formative evaluation is to improve the instruction and improve the elearning product.

b.Evaluation objectives These are the objectives for the formative evaluation not your performance objectives. Formative evaluation focuses on improving the instruction and improving the product. Your team’s formative evaluation objectives should be focused on the effectiveness (i.e., what gains in learning occurred from pretest to posttest), the efficiency (i.e., the learner time required to complete the instruction), and the appeal of the product (i.e., did the learners find the instruction appealing).

c. Description of the target course/unit See your design documentation for complete details.

3.Methodology

a. Overview of formative evaluation process Describe the formative evaluation using a Kirkpatrick Level 2 Evaluation. Use references as needed. Describe the two stages your team has conducted.

b. Participants See your design documentation for complete details of who your participants should be.

c. Instruments Describe each of your instruments (i.e., SME notes form, objective assessment, attitude survey) from each stage of your formative evaluation and how they were administered. When you describe the instrument, be sure to describe its structure (e.g., sections, number of items, scale) and how the data were collected.

1. SME notes form You will need to create this instrument and collect data from an SME.
2. Objective assessment Your team has already created this instrument. See your design documentation. You will want to collect your objective assessment data for both preinstruction and postinstruction. Include a copy of the instrument in an appendix. Do not just link to an online form. Since one of your purposes has to do with the efficiency and the learner time required to complete the instruction, be sure to collect that data. Since you may not be able to collect the practice items data from your small group testers as it is embedded in the elearning module, you may wish to create a quick answer sheet for them to use or you may wish to copy the items onto a one-page quiz (e.g., Google Forms, , SurveyMonkey) that they can submit back to you. You will need these data for both preinstruction and postinstruction.
3. Attitude survey You will need to create this instrument and collect data from your participants. You will only need postinstruction data for this instrument.
4. Results

a. Describe how you analyzed each of your instruments (i.e., SME notes form, objective assessment, attitude survey) from each stage of your formative evaluation. Did you summarize notes? Did you calculate mean averages of survey items? Did you calculate frequency percentages for test items associated with each objective in a pretest-posttest design?

b. Present the findings of each instrument from each stage of your formative evaluation. For the SME review, you can summarize the text comments. For the small group trial, you should plan to analyze (1) the reactions of your learners with mean scores and (2) the improvement of your learners’ learning by objective with percentages correct (i.e, frequencies) for both preinstruction and postinstruction. Don’t forget to report the average time required to complete the instruction.

c. Results should include tables and figures as needed. Please do not present unedited graphs or tables from an online form (e.g., pie charts from Google Forms). All tables and figures should be cited in your text (e.g., “see Table 1” or “see Figure 1”). All tables and figures should be described or summarized, that is, tell us what the table or figure depicts.

5.Conclusions and recommendations

a. Address your formative evaluation objectives individually (i.e., effectiveness, efficiency, appeal). What do your results say for each of these? Make statements about your results that address each objective. Consider all the sources of your results. Was your elearning product effective? Efficient? Appealing? to your learners.

b. Next, be specific about recommendations for changes that need to be made to your team’s elearning product. From the results your team reported, it should be obvious in your recommendations the changes that need to be made. For example, if your SME and both small group testers reported grammar and spelling issues in your content, a recommendation to revise the module for grammar and spelling errors on screens/pages #3, 5, 12, and 15 should be listed. Be specific here not general. If one of your small group testers noted that the color of your elearning module’s navigation text was difficult to read, then you will need to determine what type of recommendation you should make to address a change in the product.

c. Where possible, combine the results (e.g., SME plus small group testers) to make a recommendation for improvement.

3.Your document should include references formatted for APA.

4.Your report should be a word-processed document that is free from grammatical errors and formatted in a professional manner, including headers, footers, and page numbers. APA for references and citations should be the norm, but your report should be formatted like a professional document not a research paper. This should be your best document design. Consider using one of Microsoft Word’s templates with styles in order present a professional design. Color, text variations, headings, subheadings, and different type sizes are all appropriate. You may at your discretion, choose to use a Google Docs or Office365 file instead of Microsoft Word. In either case, the document should be professionally formatted.

Add a title page with the following: (a) all team members’ names, (b) email addresses, (c) course number and section and (d) the date.

Resources

Use the following 3 resources to help you with your team’s evaluation plan:

• from Reeves, T.C. & Hedberg, J.G. (2003). Interactive learning systems evaluation. Educational Technology Publications. [ PDF ]
• from Reeves, T.C. & Hedberg, J.G. (2003). Interactive learning systems evaluation. Educational Technology Publications. [ PDF ]

Morrison, G.R., Ross, S.M., Kalman, H.K., & Kemp, J.E. (2013 or 2019). Designing effective instruction(7th or 8th ed.). Wiley.