‍

Strengthening engineering culture at Riskified, a leader in ecommerce fraud and risk intelligence

Riskified is an enterprise-grade fraud and risk intelligence ecommerce solution that efficiently combats fraud and curbs policy abuse. It boosts merchant revenue by leveraging big data and machine learning to approve transactions that merchants might otherwise decline.

At a time of accelerated growth, the engineering organization doubled in size. The challenge facing its leadership was maintaining the commitment to delivering business value with speed and quality as they scaled. To that end, they sought a solution that would help strengthen the culture of Lean, Agile, and DevOps across the growing team.

Riskified chose Faros to provide the visibility that empowers its highly autonomous teams to optimize their efficiency and effectiveness.

A commitment to business value is a commitment to relentless improvement

Shai Peretz, SVP of Engineering at Riskified, joined the company in 2021. A Lean, Agile, and DevOps enthusiast and mentor, Shai was impressed with Riskified’s existing engineering culture.

“At Riskified, we have a strong DevOps culture, characterized by a mindset of end-to-end ownership. Our teams are responsible for their uptime, quality, performance, and even cost,” describes Shai.

But from his years of experience, Shai also knew that maintaining the culture is hard when you’re growing and hiring extremely fast. “That’s when you need tools to help supplement gut feelings with more scientific, objective data,” says Shai. “Faros adds a new layer of visibility that helps the organization measure itself and continuously improve. It reflects our tenet of relentless improvement.”

Shai strongly believes that decisions should be made by subject matter experts and delegated to the teams. He jokes that if a decision is escalated to him, he conducts a retrospective to understand what went wrong. He never wants to become a bottleneck, and — to that end — he invests in tools like Faros to help decentralize decision-making.

The goal: To improve processes and reduce risk

Tal Levinger is Riskified’s Agile Program Manager, who reports to Shai. Her role is to implement a Lean-Agile culture, help teams thrive, and implement tools and processes that improve their workflows and outcomes. It was natural for her to lead the selection and implementation of a visibility solution.

“Our goal was to first determine the baseline of the organization, get the numbers. Then we would assess the good and the bad, set some action items based on the metrics, and allow each team to track them and improve them independently,” says Tal. “For example, if we didn’t deliver well, we wanted to understand why — was it due to resource constraints, poor definition, or something else?”

Tal evaluated many tools, but Riskified chose Faros for three reasons.

“First, Faros was the only tool that met Riskified’s stringent security and compliance requirements. It was the only solution that could accommodate our data ingestion preferences and allow us to run the connectors ourselves to avoid providing system credentials to a third party,” says Tal.

“Second, Faros was the only software engineering intelligence platform that had the customizability and flexibility to meet us where we are and produce the metrics based on how we work today. All our tools are highly customized and we needed a vendor with the expertise to extract and standardize the data for reporting. For example, the Faros AI team provided a custom connector to ingest events from our bespoke deployment pipeline, so we could link PRs and commits to deployments for a complete picture.

“Third, we wanted to be able to easily build our own metrics and dashboards, beyond what comes out of the box with most vendors,” continues Tal. “Faros gives us the flexibility to create new metrics and views as needed.”

Tal also appreciated that Faros AI let Riskified select the granularity of the metrics. “We use Faros to learn and improve team by team. I like that we’re able to configure our dashboards to look at what the teams are doing, not the individuals.”

Becoming data-driven in retros and planning

Elad Kochavi is an engineering team leader at Riskified. He was one of the earliest domain leaders to run his sprint retrospectives off of Faros dashboards. Elad’s goal is for the team to learn and improve together.

In their retrospectives, Elad's team examines their metrics to ensure they’re focusing on the right priorities, becoming better at estimation, and keeping WIP under control. “Running a retro with concrete data naturally opens the door to more creative and critical thinking,” says Elad.

They also measure KTLO (keeping the lights on) work vs. value-adding feature work, something that was not possible before Faros.

“The biggest benefit we see is that we no longer rely on gut feelings to set our action items,” says Elad. “We now have a combined picture from all the tools we use and can do much more sophisticated analysis in place of the naive and simplified views in Jira. Our transition to data-driven retros has energized and motivated the team; they love seeing the impact of their efforts in the charts.”

Like Tal, Elad loves the flexibility he has to build his own metrics. “The beauty of Faros is that we can keep adding more and more metrics if something comes out of our retros that we want to analyze or improve. It’s pretty simple and intuitive for us to pull together the data from our different sources, build new charts, and add them to our retro dashboard.”

Improving quality of delivery to protect profitability

Reliability and quality are critical to Riskified, as some of the company’s offerings assume the financial responsibility for ecommerce fraud. A bug in production can have significant financial repercussions.

Thus, Tal focuses on how to improve Riskified processes to reduce risk, minimize dependencies, and streamline handoffs. Domain leaders like Elad focus on improving deployment quality, to minimize the number of times a rollback or hotfix is required.

For that reason, Riskified leverages a wide set of metrics on Faros, covering DORA Metrics, engineering productivity, agile health, software quality metrics like test coverage, and lagging indicators like production incidents.

“The ability to connect workflows from all our different data sources from development through production is so significant in understanding our performance. Jira only tells you how a task’s status has changed over time," says Shai.

"The metrics in Faros have more complexity, depth, and dimensions because they draw directly from the code itself—which is always more reliable and preferred. For example, with Faros AI I can understand how many times we deployed to test environments or how many times a task was reopened due to unclear requirements.”

Transitioning to data-driven tech QBRs

Shai introduced Faros at Riskified for the teams to have a mirror for themselves, to drive bottom-up improvement. However, the data is proving valuable to the ELT as well. Today, Faros AI dashboards are used in Tech QBRs.

“In our QBRs, metrics in Faros help map engineering’s work to business value. The excellence with which our engineering teams deliver and operate can be tied directly to the lagging results, like helping the business acquire, retain, upsell or increase customer satisfaction,” says Shai.

Elad also feels more empowered in these discussions with leadership. “Having the insights from Faros has increased my confidence when working with leadership on priorities, capacity, and resources,” he says.

Shai is pleased with how the data-driven mindset is strengthening the engineering function. “When an organization becomes data-driven, they’re more dialed into why they are doing these things and developing these specific features. That helps them learn the language of the business, foster stronger cross-functional collaboration, and become better leaders.”

Riskified’s advice for implementing engineering metrics

Having gone through the implementation and adoption of a software engineering intelligence platform, Shai shares his advice with others embarking on a similar journey.

First, generating visibility is very dependent on the tools you use and how you use them. This is where a lot of human expertise is required, to decide how to best reflect your business processes and workflows in your metrics. “No tool can magically do this on your behalf or create data where none exists, so be prepared to devote some time and thought to these questions,” counsels Shai.

Second, get creative when it comes to motivating teams to improve. For example, the Riskified office has monitors in the kitchenettes and around the office that display company info and announcements. Tal had the idea to display some engineering metrics from Faros  on them, to show how the tech org was meeting business goals. “I’ve gone into the kitchen a few times and found people discussing the stats. It motivates them to dive deeper and think about improvements we can make, which is fantastic,” says Shai.

Finally, the entrance of AI-driven development tools like GitHub Copilot demands better visibility into tech productivity and cost. “While tools like GitHub Copilot have the potential to increase productivity, evaluating their impact scientifically will help build the business case for the investment.”