The AI Transformation Imperative, Part 2: Building AI-Native Products in 2025

The product leadership landscape is in the midst of a transformation at a speed faster than we’ve seen before. Generative AI is changing things faster than the cloud and mobile advances that came earlier. What does this mean for product leaders?

AI remains a powerful differentiator—for now. What we're witnessing is a race, with every product leader scrambling to figure out how to harness AI's potential before their competitors do.

I wrote previously about how product leaders are using AI in their day-to-day work. Now I’m going to write about how product leaders are adapting their craft when it comes to building AI-native products.

The stories that follow come from the trenches—from product managers and leaders who are figuring out in real-time how to build meaningful AI products.

The Strategic Decision: When and Where to Build with AI

As product leaders, we have to start with a clear understanding of the customer need combined with clarity on how AI can address it better than other solutions.

Kristy Hissa, General Manager at Kairos, an AI startup focused on improving meeting outcomes, shares, "The biggest mistake I see is integrating AI features without a strategy. AI isn't a shortcut to identifying business problems, validating their impact, and then solving them." Hissa offers a pragmatic framework for evaluation: "Cost + Time + Talent." Before diving into AI development, product leaders must honestly assess whether they have the financial resources, timeline, and team capabilities to execute competitively.

Matt Luppino's journey at Formation Bio illustrates the importance of using product skills to figure out the right product opportunity to go after. Rather than chasing the crowded field of AI-powered drug discovery, Formation Bio focused specifically on a category of drug development bottlenecks - running clinical trials. This strategic focus has yielded remarkable results: Formation Bio can now run clinical trials "in about 30% of the time with 50% of the costs of a standard trial." They’ve identified a specific, high-value problem where AI could make a meaningful difference.

Defining and Measuring AI Success

One of the most perplexing challenges in AI product development is determining what ‘good’ looks like. AI outputs exist on a spectrum of quality that can be frustratingly subjective. The non-deterministic nature of AI means that measuring 'good' becomes more complex—you can't simply test if a feature works as designed when the output varies each time you run it.

A technique that is quickly rising as a top skill for AI product managers is crafting evals, or evaluations. As Aman Khan says in Lenny’s Newsletter,

“Evaluating AI systems is less like traditional software testing and more like giving someone a driving test:

• Awareness: Can it correctly interpret signals and react appropriately to changing conditions?

• Decision-making: Does it reliably make the correct choices, even in unpredictable situations?

• Safety: Can it consistently follow directions and arrive safely at the intended destination, without going off the rails?”

Another skill that remains critical for product managers in AI as it is in other domains is product discovery and design research. Is what you think good looks like also what your customers think good looks like?

Mijeong Kim, Product Manager at Kairos, confronted this challenge head-on: "One big challenge is that it's hard to measure the quality of AI-generated answers. As a product manager, I need to define what 'good' looks like and find a way to measure the value AI is creating."

To answer this, Kim conducted blind surveys comparing AI-generated meeting agendas to human-generated ones. The result? Users rated the AI agendas as more outcome-focused and helpful. This kind of discovery research on the AI output is a valuable way of keeping humans in the loop.

Kim's advice crystallizes an essential principle for AI product management: "Start by knowing what you're actually selling to the user. Sometimes it's a measurable outcome, like saving time or reducing costs. Other times, it's a psychological value, like feeling more confident or prepared. Either way, define the user's desired outcome first. Then measure whether your AI feature is helping them achieve it, without getting too caught up in the AI part itself."

Formation Bio's approach offers another lens on measuring AI success. Luppino's team tracks concrete business outcomes: time and cost reductions in clinical trials.

In the end, measuring product success for AI products has a lot in common with measuring product success for any kind of product. Determine how to measure if you are delivering on your core value proposition, regardless of how much you use AI to get there.

Building AI Products Outside of Big Tech

The conventional wisdom suggests that competing with Big Tech in AI is a fool's errand. How can smaller companies possibly match the resources and reach of Google or Microsoft?

Kristy Hissa offers an optimistic perspective: "FAANG is so busy with world domination, that if you do one thing really, really well and you treat your customers right, you have a good chance of being successful."

Hissa's experience working closely with FAANG companies reveals another advantage: "We work very closely with one of the FAANG companies and there are use cases where they have told their customers 'No', while making an introduction to Kairos as a vertical, specialized vendor." The giants can't be everywhere at once, creating opportunities for specialized solutions that serve specific market needs better than generic platforms. Thus, we’re seeing a rise of Micro SaaS products that may be too small to be funded as VC-backed startups and will compete based on how close of a relationship the product develops with its user base.

Managing the Human Side of AI Products

Kristy Hissa points to another human-centered challenge: managing user expectations based on their familiarity with AI. "The broader community of users who have started adopting AI is generally familiar with what to expect, but you might need to invest more in customer relations for setting the right expectations for the late majority that is coming to the AI party."

As AI becomes more mainstream, product leaders must navigate varying levels of user sophistication. Early adopters might push for cutting-edge features, while mainstream users need hand-holding and clear value propositions.

The human side also extends to team dynamics and organizational readiness. Building AI products requires new skills, new workflows, and often new mindsets. Product managers often find themselves at the forefront of learning how to use AI in our work as well as learning about what AI can do for our products.

The Product Leader's AI Playbook

The experiences of these product leaders coalesce into practical lessons for anyone embarking on the AI product journey:

Start with the problem, not the technology. Kristy Hissa's warning about strategy-less AI implementation serves as a north star. The most successful AI products emerge from deep understanding of customer problems combined with nuanced understanding of how to build on new technological capabilities.

Define success in user terms. Mijeong Kim's approach to measuring AI quality through user outcomes rather than technical metrics provides a template for meaningful evaluation. Your AI is only as good as the real problems it solves for users.

Embrace the ecosystem. Modern cloud infrastructure and AI services level the playing field considerably. Learn what’s available and what it can do for your products. You don't need to build everything from scratch—focus your resources on what makes your solution unique.

Keep humans in the loop. Whether it's user research, evaluation of results, communication, or team skills, the human elements of AI product development deserve as much attention as the technical ones.

As we navigate this AI transformation, Kristy Hissa's enduring principle resonates: "Truly understanding your ideal customer and creating business value by solving their most impactful problems... doesn't change." The tools may be revolutionary, but the fundamentals of great product management remain constant.

For product leaders standing at the crossroads of AI adoption, the message is clear: start small, think strategically, and never lose sight of the customer problems you're solving. The AI transformation isn't just about keeping up with technology—it's about thoughtfully applying that technology to create meaningful value in the world.