In addition to determining the level of outcomes at which they are going to evaluate their program, educators also need to determine the design by which they will assess if there is desired improvement of knowledge, skills, performance or patient care has occurred.

The most straightforward method for conducting a program evaluation is to measure the impact of the program on the desired outcome measures after the program has been delivered. The limitation with this type of measure is that it does not provide any information regarding a learner’s level of knowledge or performance before the course. As such, this approach does not allow the educator to determine whether a learner’s current level of performance or knowledge is due to the program itself, due to baseline levels before the course, or other factors that occurred in parallel to the course.

One way to determine whether improvements occurred through the program is to use a pre–post design, where the outcome of interest is measured before the program, and then once again after the completion of the program. While this design allows educators to determine whether knowledge or skills improved from the baseline, it does not eliminate the possibility that the improvements were due to other factors that occurred in parallel to the program or to increase in performance merely due to repeated testing.

The next level of program evaluation design is to have an experimental design, with pre course and post course assessments, and a comparison group that is identical to the learners in the program, but does not receive the program. This design is considered superior for program evaluation, as it allows educators to separate out the influence of baseline ability and parallel events from those of the program itself. However, the limitation of this type of design is that it is resource and time intensive, and may create ethical concerns if the learners in the control group are withheld a particular learning opportunity. This last concern can be overcome with a crossover design, in which the students in the control group are offered the learning intervention after the post-course measures have been obtained.

In designing their outcomes based program evaluation, educators are often challenged by the difficulties in obtaining outcome measures at the level of “behavior” or “does,” as well as with creating rigorous experimental designs. One question often asked is how strong is the need to prove that a program delivers the desired outcomes. In the case of a program that is already supported and established, in which other educators have shown that it works, it is generally acceptable to focus on learner feedback, pre- and post-course knowledge and skills assessments, mostly as a form of continuous monitoring of the quality of a program. If a program is novel, and potentially contested and challenged, educators will need to focus on behavioral or patient outcomes data, and will need to move towards a true experimental design with pre and post course measures and a comparison group. If one objective is to publish the results of the program evaluation in a peer-reviewed journal, the standards for publication have been consistently increasing over the years. A design with a control group or comparison training method, with validated measures of performance, is generally required for publication in peer-reviewed journals targeted to health professions educators.