Safety in Numbers

How often have you asked how delivery of an initiative, programme or portfolio is going and you’ve been handed a dashboard with metrics showing positive trends but doubted whether they are actually telling you anything meaningful?

Having worked in the world of delivery for over 15 years, both as a practitioner and consultant, I’m going to discuss how I’ve seen data used to make people look good but, more importantly, I’m going to share how to use data in the right way to enable a data-driven culture.

So what is a data-driven culture and why do you need one?

Well, it promptly helps organisations identify gaps and opportunities in their growth strategy. The most compelling benefit of becoming a data-driven culture is the ability to increase agility and make fast decisions based on accurate data insights, not just customer insights but also internal operations. How can you remove waste from your system to increase delivery flow, reduce cost of delivery and improve return on investment, as you’re able to realise value sooner..

How do we make data meaningful (by which I mean more than just a number)?

By converting the data into a metric that is a quantitative measurement of data in relation to what you are actually measuring. What you measure should inform your situational awareness.

Let’s explore the different type of metrics:

Vanity Metrics: These are metrics that make you look good by showing positive trends, but when you scratch the surface don’t really help you understand the performance of what you’re measuring in a way that informs future strategies, i.e. they’re not actionable.

Examples of such metrics I’ve come across:

• Number of page views (the context… what about conversion?)
• Number of app downloads (the context… did this lead to ongoing use of the app and conversion?)
• Number of story points dev done, (the context… did they release?)
• Number of commits (the context… is this efficient engineering, striving for technical excellence?)

Edward Deming put it best.

“Just because you can measure it doesn’t mean you should.”
(Spoiler alert! This won’t be the last Edward Deming quote in this blog.)

Political Metrics: These are metrics that show your stakeholders what you want them to see. You might use a political metric to support an alternative agenda, be a little opaque about progress or, more often from my experience, use this type of metric when there is a lack of understanding of what should be measured.

Examples of such metrics I’ve come across:

• Number of user stories releases, without lead time
• Number of production defects resolved, without severity or resolution times
• Redefined lead time and/ or cycle times (excluding release in production)
• Velocity of teams, which has no context outside of the team
• Number of features released, but without the effect on business KPI feedback

“If you can’t describe what you are doing as a process, you don’t know what you’re doing.” Edward Deming

There is also a type of metric, which helps encourage behaviour change called an Incentive Metric. But be careful, they  can become a double-edged sword. There’s a great Dilbert post here (https://dilbert.com/strip/1995-11-13) which sums up perfectly what can happen if you don’t invest in designing the correct metrics.

I’ve found incentive metrics really useful when trying to encourage members of the team to cross-skill or when reinforcing positive behaviours. I’ve used techniques like badges, ranking or the number of lunch & learns conducted.

Systemic metrics: My go-to type of metric, systemic metrics can represent a single team or a portfolio of teams. These metrics highlight opportunities for systemic improvements, such as reducing inefficiencies by increasing throughput or improving the way you prioritise to maximise benefit realisation, etc.

When designing a good systemic metric, ask yourself the following questions:

• Do meaningful data provide insights through trend analysis?
• Do metrics illustrate cause and effect?
• Do metrics enable continual learning and improvement?
• Are metrics designed against the end-to-end value stream?

Recommended Metrics

Metrics are typically unique to both the organisation and system you want to measure, however the core areas I would recommend measuring as a bare minimum are:

Productivity:  Are you improving your ability to deliver products over time?

Value Delivery: Are you prioritising the right feature to deliver first?

Quality:  Are you meeting your quality standards?

Sustainability: Are you working in a way you can continue for the long run?

When designing appropriate metrics, my goals is to: 

• Provide the appropriate context with which to make data-driven decisions
• Streamline the work required by teams, stakeholders or leadership
• Reduce decision latency and lag between cause & effect
• Enable continual learning and improvement

The success in using systemic metrics is underpinned by one key behaviour: radical transparency of the data. This is fundamental, enabling you to honestly assess your progress, to inspect and adapt your product and process.

Systemic metrics enable you to learn how to improve efficiency, learn how to release value faster, learn how to prioritise better, learn how to empower and engage your people, and learn how to better serve your end user. It all leads to increased benefit realisation.

I’ll leave you with one last Edward Deming quote: “Learning is not compulsory… neither is survival.”