Responses to Making Metrics Work Questions, Part 1
Last week, we hosted a very well attended webcast and panel discussion called "Making Software Development Metrics Work: A Framework for a 2nd Generation Metrics Model." The event featured PJ Connolly from SD Times; Bob Galen, author of Software Endgames; Todd Olson, co-founder & CTO of 6th Sense Analytics; and Zach Neis, VP of product development at Rally Software. If you weren't able to attend, you can watch a replay here. In total, we had nearly 100 questions, many of which we covered during the discussion. Others will be answered by e-mail, and through a series of blog posts. Here is the first set:
Q: How does 6th Sense collect the base data? Do you need any tool like a time reporting system to do that?
Actually, that is part of the unique value of the 6th Sense solution--the data is automatically gathered based on the execution of the software development process. We embed a very lightweight sensor in the developers' toolset and capture data about how, where and to what extent time is being spent by activity (e.g., coding vs. editing vs. debugging), project or team.
Q: What do you think of the statement: "People do what you measure"?
You're absolutely right and that is one of the reasons many 1G metrics programs failed. The Hawthorne Effect suggests that behavior aligns to what is being measured, which means that counting lines of code will invariably lead to LOTS of code. One of the key critics of this sort of metrics dysfunction is Dr. Robert Austin from the Harvard Business School. He sits on the 6th Sense Analytics Technical Advisory Board, in part, because he believes our units of measure (Active Time and Flow Time) are not subject to this sort of dysfunction. I would suggest reading our white paper "The 5 Essentials of Effective Software Development Metrics", which discusses the issue of metrics dysfunction. For a more rigorously defended argument, read Dr. Austin's book, Measuring and Managing Performance in Organizations.
Q: 1. How important is time reporting data in 2G metrics?
The short answer is that time matters in 2G metrics programs, because time equals investment and every investment has a direct and an opportunity cost. So it is important to know where time is being spent across your teams, activities and project portfolio. We've seen virtually unanimous support for this notion. Where the world becomes divided is on how much time capture is enough? Some organizations want a complete accounting of how time is being spent--whether or not that time is directly related to the creation of software and whether or not there is a cost to collecting that data. Organizations can achieve this by using 6th Sense to automate the collection of the time actively spent in the toolset and augmenting it with time outside of the toolset (e.g., meetings, whiteboard, thinking, personal). But for the majority of our customers, they're satisfied with a meaningful proxy for how time is being spent and they prefer complete automation over complete time accounting. Just as you must swing a hammer to build a house, you need to design, code, edit, debug and test to build software. The good news is that these core activities are executed within a desktop toolset, which makes the capture of data automatable and thus highly accurate.
Q: how do you minimize the "big brother is watching" syndrome amongst your technology teams?
This is a great question. We certainly do get the "Big Brother" reaction from some developers early in conversations, but we typically find that they like the 6th Sense solution. Here's why:
- It automates a low-value activity -- developers would much rather spend their time building software than manually reporting on their time. Most developers come to see this as best-use of automation.
- It is accurate -- the data the 6th Sense solution collects is unquestionably accurate because it is a direct result of the execution of the software development process. It accurately reflects where investments are being made.
- It's not typically about individual data -- the most value is experienced in looking at team-level aggregations, not data about an individual developer. In fact, organizations can disable individual level data.
- It's empowering -- 6th Sense doesn't segregate access to the data, so any stakeholder can look at aggregated views and help to identify opportunities and risks within a project. This makes them more valuable and more differentiated. It also helps developers to *prove* the need for additional resources when they (and teams) are being stretched beyond sustainable utilization.
Q: I understand the concept of active time and flow time but I don't believe that this is enough detail to measure and identify performance across teams. Shouldn't we also be looking at deliverables? And if so, what are the deliverables?
Absolutely. Active Time provides a powerful proxy for effort and investment and flow time provides a proxy for focus. On its own, this information helps managers to understand important things like resource alignment, capacity utilization and process adherence. But it becomes even more powerful when this data is linked to fine-grained units of work such as tasks, and outcomes-oriented data such checking in code or closing a defect. Since productivity equals effort divided by output, this gives you the complete picture. The 6th Sense solution provides the effort data. When you couple this with output data, you have a very complete picture and a very reliable measure for productivity.
This post is continued here.
Previous Post:
Integration Developer News interview with Todd Olson, Co-Founder, CTO of 6th Sense Analytics
Next Post:
Responses to Making Metrics Work Questions, Part 2
