Software Testing Metrics

- Based on Open Tech Talk on 29/12/2011.

Agenda
• What and Why
• Metrics classification
• Metrics life cycle
• Some crucial Software Testing metrics
• Do’s & Don’ts

Can You Answer these Questions about Your Software Product?
• How large is the product?
• What is the test coverage?
• What is the automation reliability?
• When can you hit ZBB?
• How many bugs were found before it was released?
• How many bugs did the customers find in UAT?
• How fast are fixes to customer reported problems made?
• What is the quality of fixes delivered?

What is Metric?
• Metric is a quantitative measure of degree to which a system, component or process possesses a given attribute.
Why to measure software?
Computing metrics –
• Improves project planning.
• Helps us to understand if we have achieved the desired quality
• Helps in improving the process followed.
• Helps in analyzing the risk associated.
• Analyzing metrics in every phase of testing improves defect removal efficiency .

Metrics classification
• Product Metrics
Product metric – A metric used to measure that characteristic of any product of the software development process
Assesses the state of the project
Track potential risks
Uncover problem areas
Adjust workflow or tasks
Evaluate teams ability to control quality

• Process Metrics
Process metric – A metric used to measure characteristics of the methods, techniques, and tools employed in developing, implementing, and maintaining a software system
Insights of process paradigm, software engineering tasks, work product, or milestones
Lead to long term process improvement

Few Product Metrics
• Number and type of defects found during requirements, design, code, and test reviews
• Number of pages of documentation delivered
• Number of source lines of code delivered
• Total number of bugs found as a result of system testing
• No. of test cases passed, failed and blocked
• Defects by Action Taken/Resolution
• Defects by Type
• Automation coverage
• Code coverage

Few Process Metrics
• Average find-fix cycle time
• Number of person-hours per review
• Average number of defects found per review
• Average amount of rework time
• Average number of bugs per tester per day
• The ratio of Severity 1 bugs to Severity 4 bugs
• No. of test cases planned Vs. Ready for execution
• Total time spent on test design Vs. Estimated time
• No. of test cases executed Vs. Test cases planned
• Defect effectiveness, Defect Quality
• Avg. time taken for automation suit execution

Metrics Life cycle

Crucial Metrics used in Software Testing

• Requirement Volatility
Number of requirements agreed v/s number of requirements changed.
(Number of Requirements Added + Deleted + Modified) *100 / Number of Original Requirements

Example: VSS 1.3 release had total 67 requirements initially, later they added another 7 new requirements and removed 3 from initial requirements and modified 11 requirements

So, requirement Volatility is
(7 + 3 + 11) * 100/67 = 31.34%
Means almost 1/3 of the requirement changed after initial identification

• Test coverage on Functionality
Total number of requirement v/s number of requirements covered through test scripts.
(No of requirements covered / total number of requirements) * 100


Example: Total number of requirements estimated are 46, total number of requirements tested 39, blocked 7…define what is the coverage ?

So test coverage on Functionality is
39 / 46 *100 = 84.78%

• Code Coverage
No. block or functions of code covered through testing v/s Total No. of blocks or functions of code developed

We can use tools like Magellan to instrument the code and get the code coverage as part of test pass or automation suit execution.

This is a good indication of stability of the product and it is also useful to identify the gaps in testing done.

• Automation Reliability Index

Reliability Index tells about the reliability or stability of the Automation scripts designed.
(No. of scripts passed/ [No. of scripts passed + failed with code issue]) * 100
Automation Pass % is the % of scripts passed Vs. total no. of scripts executed.
Failed cases could be because of product issues or environmental issues or automation code issues.

• Test Case defect density
Total number of errors found in test scripts v/s developed and executed.
(Defective Test Scripts /Total Test Scripts) * 100

Example: Total test script developed 1360, total test script executed 1280, total test script passed 1065, total test script failed 215
So, test case defect density is
215 X 100
---------------------------- = 16.8%
1280

This 16.8% value can also be called as Test case efficiency , which is depends upon total number of test cases which uncovered defects

• Defect Density
Total number of valid defects v/s size of the product
(no. of defects/ KLOC or Estimation in Functional points) * 100

Example: Total defects logged 38, and the product has 200 KLOC of code
So, defect density is
38 X 100
---------------------------- = 19%
200


• Bug Convergence
Bug convergence is the point at which the number of bugs fixed exceeds the number of bugs reported.

• Defect Find Rate
This is a most useful derived metric both for measuring the cost of testing and for assessing the stability of the system.
It can give a good indication of the stability of the system being tested.

• Review Efficiency
The Review Efficiency is a metric that offers insight on the review quality and testing

Review efficiency=100*Total number of defects found by reviews/Total number of project defects

Example: A project found total 269 defects in different reviews, which were fixed and test team got 476 defects which were reported and valid

So, Review efficiency is [269/(269+476)] X 100 = 36.1%

This is also called as Static Testing efficiency

• Defect Slippage Ratio
Number of defects slipped (reported from production) v/s number of defects reported during execution.
Number of Defects Slipped / (Number of Defects Raised - Number of Defects Withdrawn)

Example: Customer filed defects or defects found in UAT are 21, total defect found while testing are 267, total number of invalid defects are 17

So, Slippage Ratio is
[21/(267-17) ] X 100 = 8.4%


• Defect Removal Effectiveness
Number of defects found in development phase v/s number of defects reported during execution.

DRE= Defects removed during development phase x100%
Defects latent in the product

Defects latent in the product = Defects removed during development
phase+ defects found later by user

Example: Customer filed defects or defects found in UAT are 21, total defect found while testing are 267, total number of invalid defects are 17

So, Defect Removal Effectiveness is
[(267-17)/(267-17+21) ] X 100 = 92.25%