The first step in establishing a defect management program is to understand the organization's objectives. These objectives should point to the defect measures that are most important to the organization. While the measurement data will actually be collected and reported in a bottom-up manner, the actual requirements for measurement data should be defined in a top-down manner as follows:
Identify Senior Management Sponsors: By definition, the Critical Metrics are those metrics which senior management deems critical to the successful operation of the organization. Therefore the Critical Metrics should be chosen by senior management. Ideally senior management from user areas as well as IT management would be involved.
Educate Senior Management: Before they are asked to define the Critical Metrics, senior management should be given an overview of the proposed defect management process as well as relevant quality and measurement concepts.
Critical Metrics Joint Definition Session: The Critical Metrics should be chosen by the senior managers as a group decision. This is best accomplished in a group meeting so that each senior manager understands that even if his or her favorite metric was not chosen for the Critical Metrics Set, the Critical Metrics reflect a group consensus of the metrics important to the organization. Normally the Critical Metrics Set consists of 5-7 metrics of which 2-3 are defect related.
This step is designed not only to understand what metrics are most important, but also to get senior management involvement and commitment, and set the stage for ongoing interest in the Defect Management Program.
Because the Critical Metrics Set is highly situational, we can only provide examples of defect related Critical Metrics. Such examples are as follows:
Failure index (failure cost/IT budget)
Defect removal efficiency
Mean time to failure of critical systems
Defect arrival rate for critical development projects