Measures of Effectiveness in Disaster Management

Measures of effectiveness (MOEs) define quantitative and qualitative management tools that provide means for measuring the effectiveness, performance, and outcome of an operation, strategy, or project. They allude to a set of parameters that ultimately determine the degree to which an operation has been a success or failure relative to the operation’s objectives. MOEs define needs and objectives and the seeking and selection of solutions to address those objectives. Measure implies a metric or standard; effectiveness suggests how well something is done or whether the intended objectives were met. Disaster management operations rely on indicators as the fundamental units measured against an agreed set of criteria for critical operations performance and impact in relationship to operational goals. Indicators, whether quantitative or qualitative, usually have limits or a range of values. These are integrated into the evaluative design process at the outset.

MOEs should be differentiated from more granular metrics such as measures of performance (MOPs), which are bundled sets of rates, averages, counts, or percentages—parameters that determine a system’s or operation’s level of performance against a benchmark. The MOEs assess the impact of those performance levels and are necessarily more complex, often cross-referencing MOPs for a given operational objective. The term monitoring is the ongoing, systematic collection, analysis, and use of data during the course of a project. Evaluation is the periodic review of program activity, outcome, and impact, with an emphasis on lessons learned.

Systems interact with their surrounding environments. Disaster response systems and the variety of organizations that compose and support them, operate in dynamic climates that evolve in both deterministic and stochastic patterns. These patterns are represented by numerous ever-changing variables. As systems provide solutions to problems (e.g., disaster response systems respond to public crises), organizations rely on operational procedures as solutions to those problems. MOEs measure how well a system’s objectives are being met in applying solutions to problems; in a disaster, the efforts to mitigate against further loss of life and to restore order similarly require a measure of response effectiveness and impact.

Historical perspective

The term MOE was first applied by Morse and Kimball to operational research on weapons systems and strategies used in World War II. In the 1990s systems engineering and industry adapted the term to the performance of products and to the ability of solutions to manage problems. It encouraged building consensus among stakeholders—those affected by products and solutions—on selecting the MOE metric and determining the nature of the MOE and whether a performance, product, or solution was functionally successful and met their needs.

Systems engineers adapted MOEs to understand how organizations work, perform, and deliver, specifically on measures of solution to problems experienced in a variety of industries. The concept has subsequently been modified to address strategic problems posed to militaries and public agencies. In the 1990s, as humanitarian and disaster relief agencies strove to professionalize their operations through evidence-based research and donor accountability initiatives, the idea of MOEs took hold likewise across the multiple sectors involved in crisis response (food, health, water, sanitation, shelter, etc.). For instance, the health sector embraced critical consensus-driven mortality, morbidity, and nutritional indicators that gauged the mission’s success or failure on the affected population. With humanitarian assistance projects, MOEs have measured access, influence, sectoral impacts, and local capacity building.

Current practice

MOEs should represent the needs and concerns of the stakeholders defining the operational problem; in the setting of disaster response operations, these would include beneficiaries and local populations, multilateral organizations, nongovernmental organizations (NGOs), and officials of the affected country. Engaging these critical stakeholders engenders consensus agreement and clarity and facilitates communication between them in real time to see if goals are being met and where operations should be improved. When disaster organizations agree to share information and work with stakeholders for a common goal, MOEs become a priority in management, evaluation, and monitoring for all concerned, especially the beneficiaries of relief and development. Only with transparency can stakeholders feel confident and organizations be held accountable. These measures can promote unification of medical assistance, allow for comparison of responses, and bring accountability to postdisaster acute-phase medical care.

The key to understanding MOEs and the process by which they are generated in disaster response lies in (1) the recognition of which indicators are considered essential measures of sector performance, (2) the assessment of the applicability and reliability of these essential indicators in whether they meet usefulness criteria, and (3) the ability of these essential indicators to provide the coordinating language of the timeline or critical pathway specific to the disaster event ( Fig. 62-1 ).