Luke… Beware of the Dark Side
As with anything in life, metrics analysis can have a light and a dark side. I inevitably think of Star Wars and the constant struggle between the two sides of the force—the dark and light. Depending on your perspective and experience, analyzing your teams’ data can provide tremendous insight and competitive advantage. However, if you do not show wisdom, care, and understanding, this analysis can quickly be misused and become highly disruptive.
I was recently teaching a course on test case construction in which we were exploring a section on test metrics to measure testing progress and product quality. One of the slides contained a set of example metrics. Here’s one that caught the attention of the class:
Metric - Measure test team efficiency by calculating the number of testing hours per defect detected.
The class immediately went to the "Dark Side" and reacted viscerally to the example. Discussions churned around different perspectives. For example:
- From a development perspective, this measure did nothing to evaluate the quality of the development team deliverables—the code. It simply tries to reward fewer bugs, but didn’t speak to the nature of them, including priority, severity, pervasiveness, or complexity. Nor did it speak to bugs that ultimately reached the customer.
- From a testing perspective, the metric revealed nothing about the quality of the testing team’s efforts towards detailed planning, broader strategy, types of testing being applied, and project schedule support. It also does not consider the quality of the bugs found nor the maturity level of the product as it is released.
- From a management perspective, it is not clear what the measurement was driving towards or what behavior the metric was trying to encourage. This metric might be valid. However, more likely it would drive negative behavior within both teams as they try to positively support the metric.
This is clearly an example of a poor metric. One that illustrates why many software engineers react poorly to requests for more information on their working habits and project progress. It also alludes to the fact that most good metrics require deeper probing, drive additional questions and rarely lead to a simplistic response.
Instead of looking at software development data with this approach, our analytics should lead you in another direction. No, you are not "Big Brother" or Darth Vader or their victim. Instead, you are an experienced software developer or leader who understands that detailed data can provide tremendous insight and lead towards carefully crafted adjustments in individual and team performance—which can truly be the difference in project success or failure!
Previous Post:
The App Integration Challenge
Next Post:
6th Sense Analytics — Big Brother Watching or Data-Driven Coaching?
