Anatomy of a Metric: Build Time

Updated: September 20, 2024

Original post: January 17, 2024

Is the Build Time metric the ultimate inner-loop productivity indicator?

LinkedIn recently shared its approach to measuring developer productivity and happiness and the three company-wide engineering metrics it tracks for all but the developer platform teams.

The organization reached a consensus that one of those key metrics is the Developer Build Time metric “because it happens so frequently, you can potentially save a ton of engineering time and make engineers much more efficient by improving build time.”We were reminded of our own experiences advocating for the Build Time metric as a key productivity indicator at other Silicon Valley companies. Spoiler alert: It wasn’t easy.

But we learned a lot, and we’re sharing our learnings here.

A two-fold challenge for developer productivity leaders

Imagine a mid-sized tech company in the Bay Area, where 1,000 engineers build and maintain a popular SaaS product.

The Developer Productivity team comprises 30 engineers and is responsible for all the developer tools and services, including developer environments, build systems, source code and code review processes, CI/CD, and testing environments. It’s also responsible for measuring, reporting, and improving developer productiviy.

Despite this team’s efforts, developer surveys repeatedly highlighted a significant pain point: prolonged build times during what some term “inner loop” activities — those solitary, focused periods of coding and problem-solving.

These complaints also reached the ears of executive leadership, who were always concerned with the organization’s productivity. The anecdotal grumbles prompted the leaders to ask the Developer Productivity team for solutions to the problems and evidence of improvement over time.

For Developer Productivity leaders, the challenge is always twofold:

1. Identifying metrics that genuinely reflect productivity improvements.
2. Justifying investments in the tools and environments that facilitate these gains.

The team aimed to identify a clear, singular metric that would effectively showcase their success in reducing build times and the positive impact on the business.

Was the Build Time metric the one?

The Hypothesis for the Build Time metric: Faster builds improve productivity

The Developer Productivity team laid out its hypothesis that improving build execution time is a worthy investment:

1. Build execution time constitutes most of the developer wait time in inner loop activities of coding and testing.
2. Shorter build execution times contribute to faster task completion times.
3. Shorter task completion times lead to higher throughput (engineers can complete more PRs during the same period).
4. Completing more PRs will have a positive impact on business results (as the team completes more product work faster).

Thus, the Developer Productivity team would begin investing in build optimization and observe their impact on build execution time over time.

The Implementation for the Build Time metric: Multiple iterations

The implementation of this hypothesis went through multiple iterations. Here’s how it went:

Step 1: Measure build execution time

There are many ways to crunch and present a metric like the  metric. The Developer Productivity team chose to implement it as the sum of total build times over time

• What they measured: Sum of total build time over time (Total Build Time).
• What they expected: Total Build Time would decrease.
• What actually happened: Total Build Time was unstable, unpredictable, and hard to understand. The team suspected it was being influenced by spikes in usage. And, as individual build times decreased in the real world, teams were able to run more builds, making Total Build Time a poor proxy for productivity.
• What they learned: As is, the learnings were unclear and the Build Time metric couldn’t be presented to leadership.

Step 2: Measure build execution time in a controlled environment

To isolate the Total Build Time metric from the various spikes, the team opted to measure it in a controlled environment.

• What they measured: Sampled build time over time in a controlled environment (Build Time).
• What they expected: Build Time would decrease.
• What actually happened: Build Time stabilized and indeed decreased thanks to the optimizations introduced by the Developer Productivity team. The metric was stable and useful for the team. However, it was still unusable for leadership.
• What they learned: Leadership struggled to understand the value of the metric and how it translated to business impact.

Step 3: Measure build time as a percentage of a PR’s cycle time

The team sought to find a better signal to monitor. They introduced a more precise metric that could show that the build bottleneck was decreasing and engineering productivity was increasing.

• What they measured: The ratio of build time to the PR's complete cycle time (from code checkout to PR merge). If this Build Time Ratio metric decreased over time, the team could show it demonstrably relieved a significant inner loop bottleneck.
• What they expected: Build Time Ratio would decrease over time as optimizations were introduced (see note).
• What actually happened: Build Time Ratio decreased over time.
• What they learned: This metric was better, but it was still difficult for leadership to associate directly with business impact.

Two things were found to make this metric more impactful:

1. Converting the time savings from improved Build Time Ratio into dollars.
2. Correlating the decrease in Build Time Ratio with an increase in completed tasks in a given period. This would explicitly show that the time savings were being converted into increased productivity.

Note: The team assumed that the number of times the average engineer builds their code on an average PR is relatively stable.

Step 4: Create a dashboard that includes economic benefit and throughput

The team concluded that the Build Time metric needed to be presented in context:

1. Show build time is decreasing relative to the other steps in the developer’s inner loop workflow (Build Time Ratio).
2. Translate the time savings generated by optimized build times into an economic benefit. Multiply the time savings by the number of engineers and by the engineer’s loaded hourly rate.
3. Demonstrate that the time savings impact the ultimate goal of delivering more business value faster by showing that engineers are now completing PRs faster.

Note: The team assumed that the engineers are working on the right things as determined by the product and engineering leaders who prioritize their work.

Key learnings for the Build Time metric

In this article we followed the evolution of one single metric — the Build Time metric — to act as a signal or proxy of developer productivity. As you can see, it wasn’t a slam dunk on the first try.

We learned a lot from this one instance about what it takes to identify the right metric, calculate it, and present it in the right context.

Leaders want to know the engineers are working on the right things and having an impact, but struggle to define how they want that represented.

• Reaching a consensus about “good metrics” is hard. Leaders often don’t know what they want or what will work for them until they see it, probe it, and consider the data. It will take trial and error to figure it out.
• Try to anticipate the “so what?” that leaders will ask. This metric improved — so what??? If you anticipate the question, you can construct metrics that are more self-explanatory, contextualized, and tied to business impact.
• Leadership changes and you may find yourself going through this process again and again with new leaders.

Any metric you put on a productivity report is going to get tremendous scrutiny and some resistance.

• Be prepared to defend your chosen metric and explain why you’re measuring it. In this example, the Developer Productivity team was aiming to prove that their investments in build optimization were bearing fruit on engineering productivity and translated to business impact at large.
• Every metric will be questioned, and you’ll need access to other types of data to confirm, defend, and dispel objections.

There is no silver bullet.

• Engineering is a complex and sprawling function. You have to be prepared to measure all aspects of engineering if nothing else then to ensure you are balancing all the different elements of performance and efficiency without creating unwanted consequences.
• Context is king, and rarely can the sum of all your considerations and tradeoffs be captured in a single metric. You will need to have more than a single metric at your disposal.
• Data engineering is time-consuming and specialized. It helps to have a dedicated data expert to create different versions of metrics and analyze them. Most of the Developer Productivity team has their hands full with the optimization work itself.
• Industry benchmarks can help your organization know what good looks like, how you compare, and what to prioritize.

Faros AI is a specialized data platform for software engineering that supports data-driven developer productivity and developer experience initiatives. Learn more here.