Page 22 - 372.w

Moving from

Observation to Evaluation

With an evaluation plan in place, the evaluation process begins with observing the exercises.
Data collectors and evaluators will need to know what to look for, both general and specific.
They must be aware of the exercise goals and objectives, the overall scenario, as well as what
activities and actions will take place during the exercise. It is helpful if data collectors and
evaluators have some subject matter expertise for the activities they will observe.They should
also create written records, using a checklist or free-notes, to assist with data collection to ulti-
mately derive the most value from the evaluation (Appendix F).

Evaluation is a common process in the public health sector, so the eight steps recommended
by HSEEP, should be familiar.

Plan and organize the evaluation in
1 advance.All data collectors/evaluators and 5 Conduct the exercise debriefing/hot wash
(See Appendix F, page 67). Depending on
controllers need a complete briefing on the size and complexity of the event, this
exercise and evaluation materials. process may consist of a single debriefing
or a series of debriefings with various
2 Observe the exercise and collect data subgroups of exercise participants.
Moving from Observation to Evaluation
using the materials provided (i.e., observer
and data collector logs and workbooks, Identify opportunities for improvement
including the data collector/evaluator 6 and lessons learned. Update the plan
handbook). accordingly.

3 Analyze data and assess performance at 7 Finalize the AAR, which should include
the task, departmental, discipline-of-
an assessment of strengths and weaknesses.
function, and mission levels. Reconstruct The AAR will in turn guide the develop-
the exercise events from logs, workbooks, ment of the Improvement Plan (IP).
hot wash, and other debriefings. Identify
the root causes of differences using critical Develop the IP. This converts the lessons
thinking to determine why things 8 learned from the exercise into measurable
happened as they did. steps that will result in improved response
capabilities.Then track the implementation
4 Draft the After Action Report (AAR) of the IP (See Appendix F, page 69).
(See Appendix F, page 53 for details).

20