Midway through 2024, we surveyed 85 technology leaders about the questions they wish their engineering data could answer. Three themes emerged: AI, developer productivity, and business outcomes.

Here are the top 10 questions engineering leaders want answered.

1) Which AI-driven development tool is best for our company?

Challenge: The driving force behind this question is the rapid proliferation of AI-powered development tools in the tech industry, creating both opportunities and confusion. As there are many competing products, such as GitHub Copilot and Amazon Q Developer, the challenge lies in selecting the most suitable tool that aligns with your company’s specific needs, technical stack, and long-term goals.

Opportunity: Using the right AI-driven development tool has the potential to significantly enhance productivity, code quality, and innovation within an organization. Some organizations opt for an initial comparison process to determine which tool will best fit their specific development workflows, security requirements, and long-term goals while balancing costs and potential productivity gains.

If you’re looking to compare AI pair programming tools scientifically, here’s how to conduct a bakeoff.

2) How do we track AI usage, expand adoption, and monitor downstream impacts?

Challenge: The dynamic here stems from the desire to ensure a smooth integration of new AI tools into existing workflows to maximize adoption and deliver business value. The challenge lies in overcoming potential resistance, fostering efficient usage, and quantifying the impact of the tool on key performance metrics. Understanding low adoption rates and identifying unused licenses are also key to organizational learning.

Opportunity: Many organizations are looking for a single, unified view across the SDLC to monitor and support adoption, from pilot through rollout and broad implementation. Their goal is to accurately measure the tool’s impact on team performance, provide adequate enablement and training for the team, make sure the the usage justifies the investment, and have a clear view of downstream impacts to mitigate unintended consequences. One software company set up an internal champion program to drive adoption.

3) How do we achieve the highest ROI from AI copilots?

Challenge: Hefty investment towards AI technology creates pressure to demonstrate concrete returns. While traditional ROI frameworks focus predominantly on time savings and cost reduction, this narrow approach fails to capture the full spectrum of AI’s impact on developer experience and productivity. Software engineering leaders then face the complex task of articulating AI's broader benefits and translating them into quantifiable business outcomes.

Opportunity: To justify the investment in AI code assistants, Gartner® recommends reframing the ROI conversation from cost reduction to value creation. Doing so allows engineering leaders to adopt a new and improved ROI framework that builds a holistic value story to analyze the full return to the organization. Download the complimentary Gartner® research.

4) Which high-impact areas should we address to unlock major productivity gains?

Challenge: Within rapidly evolving technological environments, engineering leaders face pressure to improve development speed and quality while managing limited resources and technical debt. Additionally, they must balance immediate productivity gains against long-term scalability and sustainability of their engineering processes and systems. This creates the challenge of identifying and prioritizing the most impactful areas for investment in tools, platforms, and practices.

Opportunity: Platform Engineering is a growing practice, where centralized teams build and maintain common capabilities on shared, self-service platforms that sit between developers and underlying services or platforms, while addressing critical requirements like security, compliance, and governance. These platforms help users navigate complex architecture and reduce their cognitive load, thus improving developer experience, delivery speed, process agility and business value.

5) How do we optimize our processes and reduce bottlenecks?

Challenge: With increasing complexity in software development workflows and globally distributed teams, the challenge lies in identifying inefficiencies across multiple time zones, departments, and stages of development. Companies struggle with communication gaps, handoff delays, and synchronization issues, which ultimately lead to bottlenecks, hindered productivity, and late deliveries.

Opportunity: Insights generated from the intersection of multiple data sources — SDLC, HR Systems, FinOps, Customer Data, and more — can help identify the root cause of bottlenecks, confirming existing hypotheses or bringing a new issue to light. For example, joining GitHub PR data with employee location data from Workday helped one software company confirm the root cause for long lead times: cross-geo dependencies. A re-org that kept ownership of features and repos within regional hubs helped increase velocity by 16% within two months.

6) Which strategies will increase confidence in the quality of our outputs?

Challenge: Engineering leaders grapple with this question due to challenges balancing speed with reliability, maintaining quality across distributed teams, and adapting to new technologies and methodologies. Not only do quality issues erode customer confidence, but they also can have significant financial repercussions.

Opportunity: Balancing velocity and quality, or innovation with keeping the lights on (KTLO), requires visibility into how teams spend their time between planned and unplanned work. Many organizations create a combined picture from all their tools to support a much more sophisticated analysis of their delivery. Software engineering intelligence platforms like Faros AI combine visibility into DORA Metrics (velocity and quality), engineering productivity, agile health, testing and quality metrics, and lagging indicators like production incidents to create this understanding.

7) Are we focusing on the right priorities?

Challenge: Engineering leaders frequently struggle to provide comprehensive insights into their department's operations due to limited visibility and inadequate tools. This lack of information hampers their ability to answer crucial questions about project alignment, resource allocation, and business impact. Consequently, teams face persistent challenges in delivery timelines, quality assurance, and reliability, which are difficult to identify and address proactively.

Opportunity: Engineering leaders can enhance alignment by implementing self-serve dashboards for stakeholders to track current and upcoming initiatives. This, coupled with company-wide understanding of the ‘product development system’ (i.e. the inputs, outputs, and feedback loops that inform decision-making and prioritization), ensures focus on top priorities and enables timely recalibration. Such transparency in decision-making and prioritization processes helps keep engineering efforts aligned with larger business priorities.

8) Are our investments and resources aligned to our business goals?

Challenge: As organizations grow, maintaining visibility across distributed engineering resources gets harder. Teams often struggle to prioritize effectively, feeling pressure from all sides — Sales, Product, and Customer Support. Additionally, a lack of clarity about the company's overarching strategy and engineering's role in it can lead to misalignment between team objectives and organizational goals. This disconnect hampers the ability to assess current states, set clear targets, and align efforts with the company's strategic direction.

Opportunity: Some companies tackle these issues by implementing unified dashboards to consolidate data across their portfolios. For instance, SmartBear leverages Faros AI to create a comprehensive view of their investments alongside key business outcomes. This integrated approach enables rapid identification of resource misalignment, allowing for swift and efficient adjustments based on real-time signals. This visibility enhances decision-making, ensuring resources are optimally aligned with strategic priorities and performance metrics.

9) Can we use our data to predict and mitigate issues before they occur?

Challenge: Without predictive capabilities, engineering leaders are constantly in reactive mode, addressing problems only after they've escalated and caused significant disruption. This lack of foresight erodes trust with stakeholders as milestones are missed, and increases stress on engineering teams who must repeatedly engage in crisis management.

Opportunity: To better leverage engineering data for issue mitigation, leaders can implement real-time monitoring systems that track key performance indicators across projects and teams. With data integrated across multiple sources, they’ll gain a holistic view of their development ecosystem. From there, they can identify patterns and anomalies that precede issues, enabling them to take preventive action and make data-driven decisions to optimize processes and resource allocation before the issue escalates.

10) How do we get the most value out of our engineering department?

Challenge: Recognizing software as a key competitive advantage, businesses may allocate 35–50% of revenue to software engineering. But as the organization grows, visibility into engineering activities and their impact often diminishes. This leaves engineering executives struggling to answer C-Suite and board questions about project statuses, alignment with key initiatives, resource allocation, and the impact of emerging technologies on productivity.

Opportunity: Tech organizations can become much more data-driven throughout the core pillars of how they work — from monthly productivity reviews to weekly sprint deliveries, from quarterly outcome tracking to annual budgeting, and even within bi-annual talent reviews. Integrating the tools the organization already has, tying the information to business data sources like financials, HR, customer, and product data, can provide the visibility and line of sight into better decision-making. Centralizing the data also makes it possible to apply AI to optimizing these workflows, ensuring teams are aligned with corporate strategy and effective.

With great power comes great responsibility

In summary, with great power comes great responsibility. Engineering leaders are facing many challenges in today's rapidly evolving technological landscape, but that’s because the company is more dependent than ever on the the competitive edge tech innovation provides.

Ultimately, the ability to harness data effectively will be a key differentiator in maximizing the impact and value of engineering departments, driving innovation, and gaining a competitive edge in the market as AI transforms every single aspect of software engineering as we know it.

Ready to close your visibility gaps? Request a demo of Faros AI today.

Back to blog posts