Using Metrics to Support Investment in Software Testing

In the current economic environment, one would think that we would be using all means to secure investment in software testing. However at a recent industry forum it became apparent that the level of maturity in using metrics is somewhat limited.

Of the participants, many admitted to using defect metrics, but rarely used other metrics such as time and budget allocation, effort, coverage, and test outputs. This is somewhat disappointing to have a poor measure of the work that we undertake. It leaves us vulnerable to cutbacks, as other managers who may be more verbally gifted may take away our budget. Metrics provide a useful way of supporting our benefit.

The meeting went on to review the kinds of metrics that testers use. The following list provides some suggested metrics:

• Defects
Type
Cost
Age
Status
proportion found in stage

• Coverage
automated vs manual
planned vs executed
requirements
code

• Effort, time, cost, schedule

• Test outputs
test designed
test executed

• Test inputs
size
complexity
developer hours
project budget

• Risk
Some comments were made that metrics can be broadly grouped into two categories:
• Efficiency - can we do more testing using less effort
• Effectiveness - can we acheive a better outcome from our testing effort
While the discussion focused on supporting the business case for software testing in tighter economic times, it is important to note different uses of metrics. Metrics are used by test managers for the following:
• Managing Progress - producing estimates, how complete is testing, how much more to go, ...
• Quality Decisions - is the product good, bad or indifferent, are we ready for release, ...
• Process Improvement - what are the areas that future improvement should target, how do we know our process has improved, ...
• Business Value - what benefit has testing produced for the business in terms of reduced costs, increase revenue, reduced risk, ...

The choice of what metric to use can be daunting. It is not a good idea to collect everything, as you get overwhelmed in what data to use to make a management recommendation.

There was a lot of discussion that metrics assessing productivity can be dangerous. My personal view is that we should you use metrics to influence productivity, but that the following points need to be kept in mind:

• Productivity or performance measures do change behaviour. People will start aligning behaviour towards meeting the metric target. A poorly chosen metric or target could create behaviour that you never intended.
• One metric wont tell the correct story. Measuring productivity in terms of tests per hour completed, may mean that people run poor tests that are just quick to run. You may need several metrics to get a better perspective, for instance collecting information on defect yields, defect removal ratios from earlier phases and so on to get better picture.
• Given that metrics will change behaviour, you may change your metrics from time-to-time to place emphasis on improving or changing other parts of your process or performance.
• Metrics should be used by the manager to ask more questions, not a hard and fast rule to make a decision. A metric may lead you to make deeper enquiries with individual testers or developers.
• Managers need to build trust with the team in how they use metrics. If the team don't trust how you will use the metrics they will likely subvert the metrics process.

When we were discussing using metrics to justify the business case for testing it is very easy to get caught up in the technical metric, that is important to the test manager. However, when discussing with other business stakeholders, you need to talk their language. You may need to explain the metric in terms of what it means in terms of cost saving or increased revenue for it to have its impact. Don't explain it in terms of how many more tests cases are executed per hour, instead explain it is $'s per hour saved. Other business may need it explain in other terms, such as risk reduction or satisfying compliance.


* Thanks to Kelvin Ross for the original version of this blog