SmartBear’s Software Engineering Scales to Support Rapid Growth by Measuring Outcomes with Faros AI

Results

• Unified Visibility: Data from disparate teams and portfolios coalesced into a single, accessible platform.
• Enhanced Predictability: The centralized data analytics helped keep product deliveries on track, enabling on-time major product releases and increasing customer satisfaction.
• Optimized Resource Allocation: Portfolio investment and outcome monitoring leads to data-driven resource allocation and informed decision-making.
• Streamlined Efficiency: Lead time and effort of reporting and data analysis decreased from days to minutes.

Background

SmartBear is a leading provider of software development visibility tools, with the goal of helping software teams make each release better than the last. Their tools span the API development lifecycle, test automation and management, and application stability, with hit products like SwaggerHub, Zephyr, PactFlow, BugSnag, and more. They’re used by 16 million developers, testers, and operations engineers at 32,000+ organizations – including world-renowned innovators like Adobe, JetBlue, FedEx, and Microsoft.

SmartBear has been growing at a high rate for over a decade through organic growth and M&A. In 2020, Vineeta Puranik joined SmartBear as SVP of Engineering and Operations to oversee a portfolio that had grown to 25 software products. “As a developer myself, I love how I can relate to SmartBear products. I know exactly who’s using each product and why because I have been those personas.”

Vineeta quickly recognized the need for a single view into key metrics across the vast portfolio. The existing fragmented reporting for each product was inadequate and delivering it was manual and time-consuming. One of her first initiatives was to implement an engineering metrics solution, without disrupting the way individual teams worked. Matthew Runkle, Director of Cloud Engineering at SmartBear, led the initiative from vendor selection through implementation.

“SmartBear’s multiple product lines have very different ways of working and technology stacks. Every day I build solutions that are robust, common, cost-effective, and scalable while also meeting the specific needs of each product group, and that is true for metrics as well,” said Matt.

Centralizing Software Engineering Intelligence on Faros AI

SmartBear’s scale was a trigger for seeking out a metrics solution. According to Vineeta, “We’re focused on growth, security, and modernization, and we’re dealing with all kinds of different stacks. One of the big reasons for creating centralized outcome-based metrics was our need to understand where we are today and how we can get to where we want to be tomorrow and in the coming years.”

SmartBear selected Faros AI for its ability to integrate with its diverse stack and be customized to its taxonomy. Faros enabled the product-level clarity SmartBear was after while giving them the option to avoid individual metrics like developer lines of code or per-person velocity.

Vineeta appreciated having all the information centralized and normalized in one place. “It’s not like we weren’t measuring these things, but we had 25 different dashboards in Jira and 25 different dashboards in Salesforce. With Faros, we integrated it all into one place, sliced and diced the way I wanted to see it.”

Optimizing Portfolio Investments

SmartBear monitors its portfolio’s performance through two lenses: BCG’s growth rate matrix and SmartBear product family. Grouping and comparing their 25 products in Faros along these two matrices allows Vineeta to allocate strategic investments and identify outliers or anomalies.

“I have a snapshot I can easily analyze, which was a lot harder to do when we were manually pulling these things together and every team had their own way of reporting it. I use the metrics to make sure we are using our resources effectively.”

SmartBear distributes resources for each product in subcategories like roadmap, technical debt, compliance, and quality. “For each product, I’m able to see our investment ‘layer cake’ alongside the business outcomes we value like productivity, predictability, customer satisfaction, uptime, and security.”

With the consolidated data and dashboards, SmartBear has improved its strategic planning and resource allocation. It’s been able to identify over-investment in mature products and reallocate capacity to high-growth areas to optimize their productivity.

“The visibility helps raise red flags and inform us if something is off in the way we are staffing the teams. We adjust our resources very quickly and efficiently based on those signals.”

Data-Driven Decisions Lead to Better Predictability

SmartBear uses Faros to measure DORA metrics that cut across tools, which help ensure products are releasing consistently and frequently and identify areas that may be bottlenecked.

Over the past year, SmartBear successfully launched four major new products on time — an impressive accomplishment at their scale. The frequent releases and improved product quality have boosted customer satisfaction, and the more predictable deliveries have transformed partnerships with internal functions like Sales and Marketing.

Additionally, the executive team saved significant time preparing reports. Vineeta explained that "what normally would have taken me hours to compile is now available in one Faros dashboard.” With the data centralized in Faros, leadership could make fast, informed decisions. Ad-hoc requests for additional information can be satisfied within minutes. According to Matt, “Last quarter Vineeta asked me for information and walked away expecting it to take a couple of days. I was back in five minutes with all the data.”

Transparency Through Company-Wide Access

Faros dashboards are accessible to everyone at SmartBear, which provides transparency into how products are performing. Engineers can self-serve insights to improve their work. Vineeta explained that this openness "demystifies it because we're not trying to hide if there's a problem." They collectively solve issues using the same data. “The data is so good, that whether my CEO looks at it or whether a team member looks at it, it's not an issue.”

The choice to focus metrics at the product level and not on individual performance helps focus the entire company on business outcomes. It’s a future-proof approach as well, as Vineeta explained. “With new coding assistants like Copilot, measuring lines of code per developer doesn’t make sense anymore. We need metrics that can survive the test of time.”

Customization Enabled by Strong Partnership

Both Vineeta and Matt credited collaboration with Faros as key to their progress. According to Matt, "The partnership with Faros has been fantastic. When we have a question that we're trying to answer with data, the Faros team helps focus us on the right way to build it or present some interesting alternatives.”

“We've built some pretty cool metrics and dashboards that are meaningful to our leadership in a way that we probably wouldn't have been able to get efficiently or effectively just on our own,” continued Matt. Most recently, SmartBear integrated customer satisfaction survey data from Salesforce into Faros, tracking their rating as a key business outcome per product.

Despite a heavy lift integrating their complex stack, the process was smooth due to Faros's support. Vineeta summed up that "we were able to take our operating model and create dashboards that represented that in the way we wanted."

The Future Looks Bright with AI-Infused Metrics

With visibility in place, SmartBear is looking ahead to leveraging predictive capabilities and metrics that uncover problems and opportunities. Vineeta shared that they "...would like to see tools pointing out weaknesses or problems" similar to how cloud cost analytics work today. Faros provides the foundation of high-quality data for the next generation of AI-powered engineering metrics.

Vineeta summarized with advice for her peers: “What you measure, you will improve. So measure the right things and focus on business outcomes.”

Start your journey with Faros.ai