Today we’re announcing that Fin can now be used as a Service Agent on top of HubSpot and Freshworks, meaning you can use the world’s best Agent without migrating off your helpdesk.

Hubspot and Freshdesk customers can now:

• Get Fin live, integrated, and working seamlessly in less than an hour

• Delivering a 76% average resolution rate

• Across all customer channels (voice, email, chat, social and more)

• Resolving complex queries that require reading and writing to 3rd party systems

• With everything fully configurable to follow the unique policies of every individual business.

Our very visible commitment to being open

Today’s announcement is a deepening of our commitment to build an open platform that is customer first.

We’ve long known that businesses want flexibility in how they configure their customer-facing tech stack. Since the very beginning, we have built Fin as an open platform, with APIs, MCPs, CLI, and opening up access to Apex, our proprietary trained model that delivers best in class performance.

To make things easy for our customers, we have extensive public documentation of our product on our website, in our help center, and in our developer docs. We are the only Agent company in our space to do this, others hide most details behind sign-in screens, which we don’t believe is the right thing to do.

Open Agent platforms will win

We now believe our category has reached a stage where customers demand open platforms, that those who open up are more likely to win, and those who remain closed and protectionist will accelerate their demise.

We are operating in a fast changing world, and customers do not want to be locked into a single vendor or closed ecosystem. They want the ability to experiment, to swap things in and out, and move everything with ease, technically and commercially.

In an open world, the best product will win

In a world where businesses can easily swap vendors, the best product will win. We are happy to compete on that front, confident that Fin delivers the best customer experience and the highest performance.

If you are a Hubspot or Freshdesk customer, you can now have Fin integrated and live within an hour, without needing any help from us.

We’re here if you want us, but as part of our commitment to building an open platform, we’ve designed everything to be self-servable, and you can get started right here or watch a quick demo of how everything works below:

fin.ai/hubspot

fin.ai/freshdesk

For about 15 years, most websites have had the same little circle in the bottom-right corner. Sometimes blue, sometimes pulsing, mostly polite enough to stay out of the way. Click it and a panel slides up with a friendly greeting and an estimated wait time. On the other side, was your support team. We know a lot about this widget because we invented it, and the industry followed – down to the design.

But on most websites today, what’s on the other side of the circle isn’t a person, it’s an Agent, always on and ready to respond. Despite the backend evolving significantly, the look remains the same.

This is fine for support where customer behavior is established, but as Agents move into other parts of the customer lifecycle, they’ll need a different interface to encourage engagement.

Different roles deserve different front doors, and this became especially clear when we designed Fin for Sales.

AI-first, not AI-added

Almost every design assumption baked into the chat widget is about the human support team behind it.

The launcher icon is small and discreet to discourage overuse, as the reps have limited capacity. The Messenger’s UI manages expectations around response times, as teams are frequently overloaded or only available during set hours.

But when building for an Agent, especially one outside of support, none of these assumptions hold.

This means we have to approach designing differently. Instead of just taking what was and building on top of it, we look at things with fresh eyes and ask how the form factor must change when the experience is first and foremost agentic. The difference is going AI-first, not just AI-added.

Designing for agentic experiences beyond support

When we set out to design Fin for Sales, here’s what we considered:

Entry points set expectations. For many, a chat widget signals “click here when something’s broken,” thanks to more than a decade of that being its only use case. When a visitor is on a pricing page evaluating options, they may not think a widget is the best place to ask questions. The same goes for ecommerce.

Show, don’t tell. Text was always a placeholder. When you ask about pricing, the natural answer isn’t a paragraph describing a comparison – it’s a chart. When you ask where something lives, the answer is to walk you there. Cards, tables, voice, video, inline navigation – the modality should match the question. A pure-text thread will quickly become tiring.

Visibility comes with responsibility. A widget hidden in the corner has the luxury of being ignored. A composer that sits openly on a page does not. While we want to encourage interaction with the Agent, we don’t want it to be intrusive. Color, motion, copy, weight, and placement are all important. It needs to suit whatever’s around it – on every site, in every brand.

What we built

Holding to those properties led us somewhere different than another iteration on the widget. We called it the Spotlight Messenger – a floating bar at the bottom of the page, centered, and dismissable in one click. It’s the entry point used by Fin Customer Agent on fin.ai and customers like Attio and Fellow.

The bar shows Smart Suggestions: AI-generated questions written from the customer’s own site and knowledge base. It shows three questions at a time, rotating across sessions. A visitor can click one to start a conversation without typing. That’s a big change from a support widget’s usual ritual of: spot the icon, click, wait for the panel, scan the greeting, find the input, and type the first sentence.

We would have never designed this look in the past. Putting a live composer on every page of a customer’s website, making it visible by default, and surfacing questions a visitor can fire off instantly would drown a human team in five minutes. It only makes sense when the thing on the other side is instant, infinite, and never has a bad day. Spotlight is downstream of Fin, not the other way around.

The same principle that reshaped the entry point reshaped the answers. The right answer to a question often isn’t a sentence, so Fin decides how to respond based on what’s being asked, not just what to say. Ask how two plans compare and Fin lays them out as a table, with features, limits, and prices aligned so the differences are obvious at a glance. That extends to anything where structure beats prose – tiers, specs, availability.

Spotlight’s visual identity went through several iterations. We explored about a dozen aesthetic directions – glassmorphism, terminal-style monospace, custom typography systems, branded gradient washes – and rejected all of them. Distinctive AI styles tend to age fast; they look like the year they were designed. We wanted Spotlight to live on a customer’s page, not on top of it.

The treatment we landed on is deliberately quiet – a soft, translucent backdrop tint behind the composer that picks up the accent colour of the page it’s sitting on. The goal is to ensure it harmonizes with its surroundings, present but never competing. Fin’s own personality shows up most significantly in motion, small cues signal it’s thinking or streaming.

We also tested how it should interact with the surrounding webpage as users navigate. Customers asked us to close the messenger when a visitor scrolled the page, but it felt fiddly in testing – a small accidental scroll dismissed the whole conversation. Instead, we opted to close the chat window when the user clicks outside of it.

The impact

On fin.ai we ran an eight-week experiment on the new entry point against the old widget, and it produced a sharp lift in new conversations and engagement, enough to roll out to 100% of the site. Individual customers have seen larger swings – one early beta reported a 9x jump in engagement on Spotlight versus the widget Messenger they used to have.

Where this goes

We think websites are about to go through the most interesting visual change since the move to mobile. For a long time, a website was a brochure: read it, then go elsewhere to talk to a person. Then it became a brochure with a chat button: read it, and if you really need to, summon a person. The next thing is closer to a conversation that happens to have a brochure underneath.

The change is already impacting the language we use. Internally, we’ve started saying surfaces, entry points, and modalities.

Surfaces are where the Agent lives: a floating bar on a marketing page, a side panel inside a product, an embedded block on a product detail page.

Entry points are how it gets triggered: proactive, contextual, persistent, ambient.

Modalities are how it answers: text, table, voice, video, navigation.

The interesting design work for the next few years happens in the combinations – which surface, which trigger, which modality, for which job.

Spotlight is one answer for one of those combinations, the front door for a sales Agent on a relevant page. We’ll keep iterating on it. But overall, we believe the chat widget as a universal default is finished. The bulk of our efforts in the coming years will focus on defining the landscape that succeeds it.

Brian Boyle is a Principal Engineer and Daria Voronstova is a Principal Designer at Fin. They co-led the design and development of Spotlight Messenger, the entry point used by Fin Customer Agent on intercom.com, fin.ai, and customers like Attio and Fellow.

AI is breaking down the rigid boundaries between roles. PMs are taking on design and designers are running analysis. But what does that mean for specialists? I don’t think the answer is that specialists go away. I think part of their job changes from doing the work for everyone else to making sure everyone else, and every Agent, can do the work well.

Non-engineers will keep shipping code. Non-analysts will keep running analysis. We are not going backwards. Still, that work can’t be improvised against raw systems. To be most effective, this new way of working will require the people with deep knowledge to encode that expertise into tools, definitions, guardrails, and evals the whole company can use.

That’s the direction we’re taking with Ground Truth, the AI-first semantic layer we’re building at Fin. Ground Truth gives Agents approved metric definitions, SQL templates, and business context to query the warehouse and help teams throughout the business produce accurate analysis.

The problem we ran into

We rolled out Claude Code across the company a few months ago. To say it ripped through the place like wildfire is an understatement. Of the 1,400 people here, around 700 now query our warehouse on a typical day using Claude, up from 220 six months ago.

That unlocked a lot of speed. It also exposed a problem.

When we audited the analysis output, we found consistent errors from the Agent: wrong tables, wrong metric definitions, wrong inferences, and wrong answers. The worrying part was that every error we found had only been caught because a human happened to notice. We had no systematic approach for catching the errors no one was watching for.

During testing, a skill that had been scoped to a single customer was picked up on an org-wide query and returned an 80% automation rate as the number. The real number was 32%.

When we ran an internal eval across a sample of key metrics, resolution rate, one of our core business metrics, was accurate about 65-70% of the time. The Agent was often close, but close isn’t the bar if the output is going into comp, QBRs, or customer-facing analysis.

What we’re building

We’re building Ground Truth to solve this problem: an AI-first semantic layer for our data warehouse.

The core idea is simple. Instead of letting the Agent guess how to calculate resolution rate or ARR every time it is asked, we codify each definition once in a central place, written and reviewed by the domain experts who own that metric.

When someone asks a data question, the Agent searches the semantic layer first. It pulls the right definition, the right table, the right filters, the right SQL pattern, and the known traps before it touches the warehouse.

The Agent stops improvising from raw schema information and starts from the same definition an analyst would use.

How the definitions are structured

The layer is organised by business domain: Fin, helpdesk, finance, GTM, and so on.

Each domain contains two kinds of files.

The first is a _context.md file. Think of it as the five-minute briefing you’d get if you grabbed an analyst who knows that area of the warehouse and asked them what you need to know before you start querying. It maps the domain: which metrics exist, what their synonyms are, how they relate to each other, which entity filters to apply, and how to interpret common patterns.

The second kind is a metric file - one per metric. Each includes the description, formula, SQL templates at each grain, variants, and the gotchas that tend to trip people up.

For resolution rate, that includes things like:

• Never average daily rates. Always sum the numerator and denominator separately, then divide.

• Use the approved resolved conversation definition, not a status field that happens to look similar.

• Apply the right customer, paying, and channel filters before aggregating.

These files are the source of truth for Agent-facing metric definitions. They are version-controlled, owned by named domain experts, and reviewed like code.

How the Agent uses it at runtime

At runtime, the Agent does three things.

First, it searches the semantic layer by metric name, synonym, and related concept.

Second, if it finds a match, it injects the full metric definition into context: the correct table, filters, SQL template, grain, variants, and known failure modes.

Third, it generates and runs SQL from that approved definition rather than reconstructing the metric from scratch.

If there is no match, the Agent can still fall back to exploring the warehouse schema on its own. But that answer is treated with lower confidence.

More importantly, the miss is logged.

How the semantic layer keeps itself current

When the Agent cannot find a definition, the query and surrounding context are logged. A separate Agent then picks it up with the goal of creating a new metric definition or improving an existing one.

That Agent does three things.

First, it searches source tables that could plausibly answer the question. It ranks candidates by schema priority, with curated marts ahead of raw sources, and by how often each table appears in past queries on the same topic.

Second, it drafts the semantic metric file: description, formula, SQL template, variants, and gotchas inferred from related metrics and previous failure patterns.

Third, it routes the change to the right human reviewer. That might be the owner of the underlying table, the most active querier in that area, or the domain analyst.

The human then reviews, edits if required, and approves. The result is a PR into the semantic layer.

The point is that every question the system cannot answer well becomes a chance to improve the system. Accuracy compounds with usage.

Early results

We’re still early, but the results have been strong

• Accuracy on core metrics like resolution rate has moved from ~70% to 100%. How we ultimately evaluate the system is a topic in itself, which I’ll cover in a follow-up post on the eval system we built as part of Ground Truth.

• The median number of SQL queries the agent writes to land an answer dropped from six to two, because it isn’t exploring the warehouse from scratch.

• Time to answer is down about 90% for the same reason.

And, just as importantly, the system now has memory. When it fails, we can see where it failed, turn that failure into a reviewed definition, and make the next answer better.

The role shift

As more people across a company run their own analysis, analysts become part of the underlying infrastructure. The repeat questions, like “what’s our resolution rate this quarter” or “how has it trended,” get handled by the system.

Analysts spend their time on higher-leverage work: improving the system, adding guardrails to keep Agent-led analysis accurate and unbiased, and pushing into more complex forms of analysis.

I see specialists across the company are undergoing similar changes in responsibility.

Designers will not just design every experience. They’ll build and maintain systems that help others design better. Engineers will build and maintain platforms that make it safe for others to ship.

Analysts will not answer every data question. They will define what good analysis looks like, encode it into the tools, and measure whether the Agents are living up to it.

When competing in business, you have two choices. Try to win the game your competitors have chosen to play, or play a different game. In our category, which we call customer agents, our well-financed competitors have made generous use of fabricated demos, particularly when it comes to their voice technology. Voice is just extremely hard. And while we all know that the future of customer experiences will be agent-driven voice, we’re not there yet today. How many times have you called a business and spoken with a modern AI agent? The technology is not generally in the wild because it’s just not been ready. That changes today. And because we finally reached a level we’re proud of, we’re going to play a different game. No smoke or mirrors are required. We’re sharing a live demo of our tech for the world to see and it’s very, very good.

For those still reading let me give you some context on this new product and explain what makes it so special.

Yesterday we announced Apex Flash, our newest and fastest model yet, and one we built for the unique demands of low latency channels like voice. Today we’re announcing Fin Voice 2, a major upgrade to Fin Voice with over 20 new features, and our first product built on Apex Flash.

Here are three things you need to know:

First, thanks to Apex Flash, Fin Voice 2 is now the fastest, most natural Agent for phone, with higher resolution rates and customer satisfaction scores than ever before. Apex Flash is trained on millions of customer experience interactions, fine tuned for customer service, and can be configured to understand all your knowledge and follow all your policies. This means Apex Flash delivers higher resolution rates whilst being significantly faster, delivering the best of both worlds.

Fin Voice 2 is the fastest and most natural voice agent because it was built in a superior way. Most voice AI products are slow because they convert speech to text, send it to a general model, get a text answer, and then convert it back to speech. Fin Voice 2 was designed to work differently, separating the real time layer that handles speech processing, and the layer that generates answers. It is purpose built for the demands of customer service on voice.

Second, Fin Voice 2 can handle all your complex queries, for example taking actions in external systems, verifying callers’ identities, processing refunds, booking appointments, and so much more. Phone is a high stakes channel, and Fin can very naturally deal with customers in many different emotional states, adapting when their emotional state changes. Fin will clarify when needed, and confirm key details before taking action. Most of the time, Fin can resolve the query in full, and when it can’t, it seamlessly hands off to the human team, maintaining full customer context and history. Fin Voice 2 has multiple improvements to call quality, and can call proactively to follow up with people who have unresolved queries.

Third, Fin Voice 2 gives you total control, with the best tools in the industry to configure and manage how Fin behaves. We’ve shipped even more rich, detailed insights into areas like call behavior and quality, the most common topics of calls, and one-click recommendations to improve. As with everything in Fin, you can fully self-serve and then manage it all with ease, without requiring professional services. Other companies will only let you set up their voice agent under their supervision, meaning you need to contact them to change anything. With Fin, we give you everything you need.

If you’ve not yet tried the demo, go check it out. Or just wait a bit and you’ll likely end up speaking with it at your favorite brand soon.

AI and coding Agents make it much simpler, cheaper, and faster to build targeted internal solutions to go-to-market problems. Nowhere is that clearer than in sales. There, companies can gain an edge over competitors still relying on off-the-shelf software that is often bloated, expensive, and built for the last generation of SaaS.

When Nicolas Sharp, co-founder of Attio, spoke at our Fin for Sales event recently, he put words to something I’d already been seeing: a growing need for more technical roles across GTM, not just in sales, but across marketing and customer success too. He framed it through Jevons paradox. When steam engines became more efficient, coal use didn’t fall, it surged. His view is that as AI systems become more capable, demand for the people who can build, operate, and improve them will grow, not shrink.

That resonated with me because it’s close to what we’re building at Fin. We’ve set up a new engineering team dedicated to improving GTM effectiveness and efficiency. Its mission is to bring a commercial edge to the organization by applying the latest technology and techniques. It’s still early, but we’re already seeing results against our commercial targets, with adoption of these tools well beyond what we expected. As the team grows, some interesting technical and organizational dynamics are becoming clearer, and we’ve been leaning into them eagerly.

The power of the pair

We’ve had real success pairing an AI-forward generalist engineer with a domain expert who deeply understands the problem space and the organization. That has pushed me to think less in terms of “what can this team achieve” and more in terms of “what can one engineer achieve,” and increasingly, the answer is a lot more than before.

Taking this approach, we built an answer engine optimization (AEO) engine that dramatically increased Fin’s visibility in AI chatbot responses and grew bot-sourced revenue pipeline. We had similar results building an outbound engine, and now we’re applying the same model to optimize other sales workflows.

In conversations with peer companies, I’m seeing a broader pattern emerge. Engineers are working much more closely with frontline contributors and managers, and are embedded directly in the teams and workflows they’re trying to improve. Rather than building generic products, they’re creating bespoke solutions from the bottom up. This feels like one of the most important developments inside GTM right now.

Building bespoke

It’s easy to hear “build your own tools” and conclude that SaaS is dead. I don’t think that’s true, though I do think SaaS is being dramatically disrupted.

Foundational platforms like your CRM, payments infrastructure, and customer communication layer are systems of record. They’re deep, broad, and embedded in the ecosystem in ways that make them genuinely hard and unwise to rebuild. You should keep buying Salesforce. You should keep buying Stripe. You should keep buying Fin. These products encode years of hard-won capability that no small internal team should be trying to replicate.

What’s different now is the layer above them: the orchestration layer. It’s the customization that connects these systems, fills the gaps between them, and makes the job easier for the rep, the marketer, or the CSM. That used to require a vendor relationship, a six-month implementation, and a dedicated admin. Now it requires a very strong data and systems foundation, a GTM engineer, and a couple of weeks focused on a clear problem to solve.

We built a cockpit to run outbound across Outreach, our data warehouse, and our own internal infrastructure. It made it possible to turn new ideas into pipeline in a way the team could actually run. Without it, the process was extremely manual and error prone. It has already generated $1.2 million in sales pipeline, and we’re tracking toward $5 million by year-end, without any migration or the need to learn a new tool. We also built another internal tool in roughly three weeks that replaced a vendor product we had been evaluating. We found the gaps, shipped something better, and tuned it to our exact needs.

We’re seeing the same logic at other companies too. Will Jones, a GTM operations associate at Attio, is in a role that barely existed a few years ago. He implemented Fin for Sales largely himself. He turned help center content, testimonials, social proof, and past call transcripts into tailored context docs, then built and tested a working Sales Agent in a matter of weeks. Now he reviews conversations, patches content gaps, and works on data pipelines that weren’t there six months ago.

What stuck with me most was how he went on to describe his job. It was less about doing the work directly, and more about directing Agents to do it well. That feels like the clearest articulation of the GTM engineer role I’ve heard so far. We’re no longer trying to find the perfect point solution in the market. We’re building the connective tissue between our foundational platforms ourselves, and for that class of problem, the advantage of building internally has been clear.

We’re seeing the same idea reshape adjacent technical functions too, though that’s a broader story in its own right.

What I’m nervous about

I’d be dishonest if I presented this as a clean story. It isn’t.

One question I keep coming back to is whether the cockpits our teams are building are durable, or whether they’re sophisticated throwaways. I genuinely don’t know whether the orchestration layer we’re investing in now becomes a lasting foundation or scaffolding around something that hasn’t been invented yet.

I’m also thinking about the staffing implications. What happens when the tools get good enough that everyone in GTM is self-serving more of their own analysis, workflows, and automation? That may be a success story for the company, but it fundamentally reshapes what my team needs to look like, how big it is, and what roles I hire for. It will probably get smaller, and the roles will become less engineering-heavy.

Build for the flex, not the forecast

My biggest concern isn’t any single one of these questions. It’s that, for all the ambiguity, excitement, and energy around what’s changing, the core principles of building and delivering software still apply.

The creative trial and error required for this transformation means you have to throw away half of what you try, maybe more. At the same time, there’s a sales team hungry for pipeline this quarter and for tools that work right now. I don’t know what the right balance looks like between exploring the next evolution of GTM software and delivering on near-term objectives that fuel the business.

What Nicolas said resonated with me here. He said Attio doesn’t try to perfectly predict what the future will look like in 18 months and then build an organization around that vision. Instead, they try to build one that’s so agile it won’t matter if their vision is 50% wrong. Build for the flex, not the forecast.

I think that’s right. I also think it’s hard to put into practice when near-term commercial pressure doesn’t go away.

Where this leaves us

GTM engineering isn’t a hypothesis anymore. The pattern is clear: embed a builder next to a domain expert, give them a fast feedback loop, and get out of the way.

Exactly where this leads is still unclear. What we’re building now may prove durable, or it may simply be a bridge to whatever comes next. Demand for GTM engineers may grow as Nicolas predicts, or the tools may get good enough that the role evolves into something else entirely.

What I do know is that the old model of buying software, plugging it in, and trying to make it all hang together is no longer the only option for the orchestration and customization that makes GTM function. The foundational platforms still matter. But the layer that connects them, shapes them around the workflow, and makes them coherent for the people doing the job is increasingly where real advantage gets built. That’s the work of GTM engineers, and it’s why this role is becoming real.

It was very much a live demo that let us walk through some of the key components of our system.

If you missed it, you can watch the recording here.

We loosely committed to answering all questions in some way, so here it is. Feel free to continue the conversation in the comments, or reach out if you have any questions about Fin, Intercom, our AI models, or anything we discussed on the webinar.

Thanks to my colleagues, Brian and Kesha, and to everyone who showed up and asked great questions. Special thanks to our guest, Claire Vo, for joining us and having Brian on her “How I AI” podcast recently.

We hope to do more events like this very soon!

Connect with us here:

X: Darragh | Brian | Kesha | Claire

LinkedIn: Darragh | Brian | Kesha | Claire

Org structure, team, and staffing

What size team works only on the agentic harnesses and how do you handle roles and responsibility? (Asked by AS, similar by EL)

How do you staff development of Fin’s “agentic coding system” (skills, Shrek, agent observability, etc)? How much is dedicated platform team(s) vs. grass roots contributions from product engineers. (Asked by NS)

Currently, a very senior core team of three IC engineers (Principal/Senior Principal), two Senior+ engineers, a Staff TPM, and a VP eng. We’d had a couple of other engineers tour in and out based on opportunity/capacity. They’re pretty autonomous on deciding what to work on. I meet with them weekly to stay close and provide input. This team has existed for about six months now. We wish we’d started sooner, but it always feels very hard to pull some of your best off other critical projects. The payback has been immense though. In addition to the team, it’s explicitly everybody’s job to contribute to the system, help encode context or skills unique to them or their team, and act on feedback loops to drive improvements in the system.

Have you restructured your org to adapt to these new processes? (Asked by YG)

Essentially no, not dramatically. We’ve formed our team-2x, and there are many local experiments with different/leaner team structures, but no dramatic re-orgs. We do, however, intend to push harder in this direction, as often making changes (for example, shrinking teams) forces valuable lessons around what works better, what breaks, how to adapt, etc. – things you can be insulated from in old world structures.

How will roles in engineering change? (Asked by KG)

At a high level, we see many adjacent roles blurring together: your best engineers already have great product judgment, and will take on more of the PM role, creating space for the PM on higher impact work, and reducing coordination overhead internally. Directionally, what is a team today should be a single person on some near-ish timeframe. In that world, engineers need to be well equipped to strongly own areas of the product/system, set great direction, and execute with high levels of independence.

Productivity metrics and impact measurement

How do you measure the actual impact beyond throughput? Does 3x increase in PR correlate to business outcomes and product value that’s relevant for your organization? (Asked by AK)

The real prize is our ability to solve customer problems, and bring our vision to life dramatically faster than before. This shows up for example in our rate of significant product launches, or in our defect backlogs burning down. You should expect revenue to be a lagging indicator; we see revenue growth to continue accelerating.

More directly to your point, it was and still is true that the number of PRs doesn't directly translate into business impact. However, it is a required pre-condition to be able to ship at least something: required, but not sufficient. At this stage, we are heavily investing in ensuring all our operations are streamlined to make sure if we want to ship something, it is agentic-first, fast, safe, with minimal human intervention.

Do your engineers know how you're measuring productivity? How do you prevent them from gaming the system or account for gamification? (Asked by AK)

Claire said this best: gamification is a culture problem, not a measurement problem. That’s the broad strokes answer. We’ve a high trust, high transparency culture with people who care about their work. Having said that, we are all human and the pressure to hit/increase this metric can be a source of stress. When we introduced tools like Buzz, which enable the offloading of work to cloud agents, initial versions didn’t attribute the work to the person who initiated it, and there was reluctance to use this system out of fear the impact wouldn’t be attributed to the person. We fixed the attribution to prevent resistance to using this power tool. Ultimately, anything egregious would become obvious quickly, energy spent trying to game is far better spent trying to win together.

What business metrics have you tracked and noticed changes in as a result of AI-assisted development? I noticed DORA-like metrics earlier – I am interested more in business metrics/OKRs etc. (Asked by SC)

Your metrics showcase eng metrics, what about customer delight or revenue? Were you able to show causal changes? (Asked by AL)

Best ones we have are product release throughput and time from idea to release, tier 0 marketing releases, or our recent Intercom 2 launch. Like I said earlier, revenue growth is accelerating for us. Another example is that we are able to say yes much more frequently, which translates into deals closing that would have been blocked, or accounts churning because we can’t support their evolving needs.

What was the target metric you started with? PR count? (Asked by AL)

Yes, merged PRs, but specifically total merged PRs divided by total number of people in the entire R&D org. This is about measuring the throughput of the entire system, not just the per engineer output. Total throughput has increased further as we’ve been growing our team too.

Have you also started measuring the change in PR size? Or ways to keep PR sizes low/discrete? (Ask by DK)

A: Yes, some details here. TL;DR; size has increased a little alongside volume of PRs. Intuitively, agents great at writing code and tests tend to get more tests as a result. Our AI approval system provides natural incentive to keep PRs small and focused, as they’re more likely to be AI approved, which enables faster flow through the system. Small focused PRs are also inherently safer, so that’s win, win. It’s early days, but we expect this to apply downward pressure on PR sizes.

Did the limit of 150 lines cause the increase in PRs per dev? (Asked by RM)

As mentioned above, there is a healthy incentive towards smaller PRs that are more likely to be AI approved and flow through more quickly. Smaller independent changes tend to be safer too.

PR workflow, review fatigue, and auto-approval

In Darragh’s article on X, you mentioned about 20% of your PRs are created and reviewed using AI. How do you maintain cognitive control, and what happens when things go wrong on these fully agent-driven PRs? (Asked by MK)

For human-initiated PRs, authors shouldn’t be blindly shipping and hoping AI allows it through. Engineers still need to have an opinion and take responsibility for the shape of the solution. In addition, our system encodes opinions about how we should solve problems, patterns to follow, etc. Our review system helps enforce this. When things truly go wrong and, for example, we have an incident, we do a deep postmortem and figure out the right corrective measures.

Curious to know how you deal/dealt with PR review fatigue. (Asked by NC)

AI review and approval is a big part of this, as well as humans having a way to encode the perspective they’d typically bring to a review into repeatable instructions/guidance for the review and approval systems.

How do you gate “Eligible for Auto Approval”? (Asked by RG)

Multiple inputs to this: PR size is one heuristic, other signals that indicate the level of risk/blast radius etc. More details here.

Do you have a classification / risk rating for PRs so you can auto-approve lower risk PRs? Do you have different agents running for this? What do your scheduled processes look like? (Asked by GF)

Yes, sub-agents with discrete tasks/responsibilities, all triggered on PR submission. Auto approval focuses most on low risk changes.

If the PR is not auto-approved, what are the next steps that take place in an agentic flow? (Asked by NF)

Today, it passes to the typical human approval flows, with some automation to notify the right people/teams in Slack.

Is everybody allowed to edit the reviewer prompts? (Asked by MB)

Yes, but this requires peer approval. In practice, owners/experts in particular parts of the system own/take responsibility for the review/approval requirements for their area of expertise.

Is it only one reviewer, or have you split it into a few different agents in parallel? (Asked by MB)

Many specialized review agents in parallel, and again, owners/experts in particular parts of the system can guide additional ways that their code is reviewed.

Skills: Creation, distribution, evals, governance

How do you distribute your skills? (Asked by IK)

A hierarchy of Claude Code plugins. Initially, we did that via a private marketplace, relying on Claude Code to auto update. We found the auto update wasn’t as reliable as we’d hoped, so we forced this using our IT MDM systems to ensure everyone has the latest version of master ~ at all times.

What skills and MCPs have you got? (Asked by GF)

LOTS. The intention is that the agent has access to all the tools a human engineer would, and that they work out of the box. Quite a bit of care and attention have gone in to achieve this. Of course, auth is required and some hooks that help make sure you resolve auth issues, rather than the agent deciding to not use the ideal tools available. We organize skills in hierarchy: our base plugin holds skills core to every person’s job, then separate plugins, many disabled by default for things like different languages/frameworks, and team/role/product area specific skills. We have hundreds of skills in total, but we also have processes for gardening and removing skills that don’t get traction, and promoting/improving skills that do.

Are the skills you’ve placed in the plugins actually reusable across multiple projects? How do you avoid them becoming too generic but still usable across several projects? (Asked by EL)

For starters, we’ve largely a monolithic architecture, so lots of general reuse. We don’t really have this problem, but do also leverage, for example, team-specific plugins where the goal isn’t general purpose. If we need to, we can promote skills that end up being generally useful from team to shared plugins.

I love the idea of constructing many small skills designed for individual parts of processes. Did you guys develop a methodology for how skills should slot in with each other? And how do you determine the architecture for how these skills interact as a whole? (Asked by JG)

Basically, not really. Emergent for now, we instrument all skills automatically so that we can spot trends in usage and effectiveness, and investigate how skills interact through session analysis.

How do you decide on narrow skills vs skills with many references, for example, a “query-snowflake” skill with many references for how to query certain table schemas where the skill.md progressively discloses which references to load, vs. individual skill workflows like “gong-call-insights” that tells Claude how to use Snowflake to query? We made a marketplace but are finding it hard to find the balance. (Asked by RC)

Agree there is a bit of exploration to find the balance. In this specific example, we’ve gotten a lot of general purpose utility by giving Claude select star and Snowflake access and that being enough to serve open ended analysis. However, it runs the risk of being good enough to be dangerous by creating a bunch of impressive but incorrect output. So alongside this, there are layers of guidance or skills that put guardrails or manners on how it will do certain things, make sure it understands some of the nuance of how data is captured, how to accurately report on ARR, for example, or more specific, like your example, task specific skills like analysing Gong calls.

How do you direct the agent to preload skills? Or does that just happen? (Asked by RC)

We use Claude Code and it puts a system reminder about every available skill into the system prompt. Unfortunately, that’s unreliable, so we use deterministic Claude Code hooks (regular scripts) that inject the context into the session based on certain conditions. For example, we have create-pr skill and we really want Claude to use it every time it creates a PR. To ensure that, we monitor Claude attempts to create a PR using gh tool, intercept it with a hook, and, if create-pr skill wasn’t loaded in the session yet, inject a prompt for Claude to load that skill.

Do you have evals that you run against the skills? How have you curated examples and set that up if so? (Asked by MK)

We have a set of evals in the same repository along with the skills. Basically like automated tests. As skills are used and new edge-cases come up, we habitually update evals with more test cases. For our PR review agent, Shrek, we do batch evaluations – we have a human-curated set of historical PRs with confirmed issues. Any time we change the agent or the prompts, we re-run evals to confirm its performance (alignment with ground truth) hasn’t degraded. Building and curating a dataset is by far the hardest and most time consuming part. As we are not experts in every repository we have to engage with experts in corresponding areas and create incentives for them to invest their time. Spending less time on reviewing PRs manually is a good incentive for them to contribute.

What’s your skill governance workflow? How are they authored, validated, reviewed? The same as code? (Asked by SS)

Yes, same as code. Then rules for how/when skills get promoted higher up the hierarchy. For example, 50 weekly active for a skill to be eligible to promote to a base plugin.

How do you avoid context explosion with so many skills? (Asked by MR)

Splitting into the hierarchy, and most plugins disabled by default, for example, team specific ones.

Agents and Shrek: Infrastructure

Can you elaborate a bit more on the infrastructure behind Shrek? Where is it running and what are the guardrails instrumented on the runtime platform? (Asked by LdAS)

Shrek is running on AWS infrastructure in isolated, short-lived containers, triggered by GitHub webhooks. Each container has read access to the codebase and the code changes diff, as well model providers’ API access (it uses both Claude and Codex). Shrek reviews are guardrailed using strict safety criteria per code repository that define what areas of the code are safe to auto-approve (if the review verdict is positive) or require an engineer in the loop.

What is Shrek built on? Is this home-rolled based just on a model, or based on something else? (Asked by IK)

Shrek is a Python app built on top of Claude SDK and the Codex APIs, with custom prompts and curated evaluation criteria for every type of review it needs to do (from basic things like PR descriptions, to company and team-specific code review and safety rules, to general logic bug finding investigations).

Which tools do you use to construct Shrek? (Asked by VD)

As above, in conjunction with a custom framework to run statistical analysis of the impact that individual prompts (and changes to those prompts) have on correctly judging a code change. Engineers use the framework to curate a dataset of PRs of various types with expected review outcomes and the framework runs simulated tests, producing statistical significance results on whether the prompt changes move the needle in terms of recall and precision.

To what extent do you use cloud environments where an engineer can start working immediately? (Asked by NG)

Engineers work on local development environments, but we invest a lot to make those work seamlessly/smoothly, have agents to help resolve issues, have the dev env in CI to automatically catch regressions, etc.

Do you have a unified solution for agent observability? How much of the agent evaluation and observability stack is in-house? Can you recommend a solution that a small company can easily adopt? (Asked by NG)

Yes, some hooks into Claude Code platform (part of what we like about Claude Code here is the extensibility,) and OTEL outputs into honeycomb.

Do you have autonomous agents that propose pull requests or any sort of changes without being explicitly asked (e.g. triggered via cron schedule, Claude routine)? (Asked by NG)

Yes, for example, our Buzz system can be triggered on schedule or by webhook.

How do you store product requirements / intents that are typically not captured in the codebase, so they can be accessed by agents? (Asked by NG)

Our agents can access docs just like humans and sometimes that’s the answer here, but there is also some discipline around capturing important context like this in the codebase documentation, and in git history.

Are you keeping your SDLC flows autonomous by creating tools/apps where engineers just interact to provide agency, or is it done in remote setups with all skills shared across the org? (Asked by AM)

A combination of local developer environments and cloud agents, skills are shared across the org (via plugin system, auto updating.) Shared skills was one of the big unlocks for us, gains compound vs. hide in silos.

Tool and model choice

Do you limit tooling? E.g. only Claude or only GPT models? If so, why or why not? (Asked by AVH)

We used to be open here, and permit every engineer/team to choose their own tools, but then got opinionated and doubled down on a single primary tool (Claude Code), which helped ensure gains compounded across the whole team, and we weren’t wasting effort trying to solve for multi-system compatibility with our skills/guidance, which can pull you to lowest common denominator and/or give you inconsistent impact across tools. So Claude Code, with bias towards Opus, and then Codex in some of our autonomous parts of the system (e.g. approval/review).

Do you only use Claude Code directly or also use non-Anthropic models? Anyone also use other tools like Cursor? Do you have Bedrock-flavour hosted models with guardrails from AWS? (Asked by SS)

We mostly use Claude Code models. We use OPEX GPT models a bit in our autonomous systems. We’re exploring Bedrock-hosted models primarily for availability reasons. Some people use Cursor, or historically did, but most of that has shifted to just vanilla vs code now. Increasingly, people aren’t in their IDEs nearly as much, if at all.

Are you migrating to GPT Codex, or introducing it in parallel to Claude? (Asked by RS)

We’re introducing it in parallel to Claude and concretely right now in our review and approval systems.

Aren’t you worried to be locked into Claude Code? (Asked by NF)

Not really worried about lock in. We love the trajectory Claude Code is on, but we also expect that other tools and systems will converge to support the platform surface area that Claude has exposed.

Are you using Cloud Agent / Dev Agents, and does the 3x number include those PRs? (Asked by KF)

We’ve started ramping up our use of cloud agents, and while it’s a small contribution right now to those PR numbers, I expect it to increase significantly as people, for example, schedule or trigger agents on queues of work.

4.6 – is that for cost? (Asked by IK)

While we don’t have any token budgets or hard restrictions on models we are constantly experimenting with different models. Cost-effectiveness is a hot topic at the moment as our usage continues to grow. Some tasks don’t require the most intelligent model assuming the prompts are in good shape. Moreover, switching to a newer model doesn’t guarantee better results. Prompts are tuned to a specific version and even switching within the same family (Opus 4.6 → Opus 4.7) might result in a degradation. We learnt this lesson while building Fin – when you ruthlessly optimize for quality it takes a lot of prompt-engineering to update the model, and it’s often the case that newer models are worse for specific tasks. With Fin, we see far more of the gains come from the architecture and system around the models, than model version changes themselves.

Security, PII, and compliance

How did you navigate DPIA or workplace monitoring laws for the prompt monitoring (as seen in the “How I AI” podcast)? (Asked by RM)

People can opt-out from session transcripts on a per-session or permanent basis, and we pseudo-anonymize session transcripts. Data collection is restricted to Claude Code, and we are not collecting personal Claude Chat sessions. All session data collected is stored in AWS with least-privilege access controls applied.

Do you conform to ISO27001 or ISO9001? How does that affect the agentic workflows and guardrails? (Asked by RM)

How does allowing an agent to approve a PR affect SOC2 compliance? (Asked by PB)

We are compliant with SOC 2, HIPAA, ISO 27001, ISO 42001, and many more. Specifically with our PR review agent, we engaged with auditors long before we even had the agent in place to ensure we have a path forward agreed. Our very first iteration was an agent based on deterministic rules approving changes to documentation/tests/etc – low risk changes. You can read more in this blog post.

Do you have to deal with PII issues when giving Claude access to certain systems, like Datadog, or have you gone through the obfuscation process everywhere? (Asked by IK)

Yes. We are lucky in a way as we already have mutual agreements with all major labs in place since we are using their LLM models inside Fin and trusting them to handle our most precious data: customer and end-user data. This enables us to feed telemetry (logs/metrics/traces) into the Claude session.

Do you have a policy for developers on what they can grant Claude access to? And do you audit that to prevent unintended access? (Asked by IK)

Yes. We have strict controls over allowed Claude marketplaces, MCPs, plugins, shell tooling, etc. There’s a massive attack vector inherent to this new developing area and we treat it accordingly. At the same time we aim for Claude to be able to do/access anything an engineer would. For example, we expose read-only access to our production infrastructure, data warehouse, etc., honoring a given engineer’s existing roles and permissions.

How are you managing access and provisioning tokens for devs and their agents? (Asked by AC)

All engineers/employees have access to Claude Code & Cowork, so far unlimited tokens.

What kind of anonymization do you do on sessions for the session analysis? (Asked by RC)

We have a best-effort obfuscation in place but of course it can’t fully anonymize the session – stripping all the data out if it would render it useless for quality analysis. We are transparent about sessions being stored (with opt-out available) and we treat sessions data in the same way we treat our customers data – there could be some PII in sessions afterall (e.g. logs). For example, we expire session data after 14 days to ensure we are compliant with GDPR data deletion requirements.

Observability, telemetry, and evals

What does your telemetry setup look like? How much visibility do you have into which skills and plugins are being used? How effective they are? (Asked by CT)

We started with the out-of-the-box OTEL monitoring that is built into Claude Code. We use Honeycomb for collecting telemetry, and starting instrumenting other flows as needed. We track telemetry events for every Claude Code Hook event, and we have significant instrumentation for SessionStart that gives us insights into any Claude Code or Plugin misconfiguration.

Have you experienced model drift, and how do you manage that? What about updates to system prompts? Do you execute a similar level of eval to any changes as you do with model changes? (Asked by GF)

We have a range of agents/prompts/skills with different levels of eval discipline depending on the use-case criticality. So far we haven’t experienced model drift in our internal dev tooling. For high-stakes agents like Shrek, virtually any change goes through evals first: changes to the agent code, system prompts, repo-specific code review skills, safety criteria, etc. We treat any change as a hypothesis that needs validation.

SDLC, codebases, and refactoring

Tell us about your SDLC? Are eng moving up front to problem discovery/framing/speccing to lessen the coordination need with PM? (Asked by AL)

Yes! We see that general trend. PM/Design incredibly focused on higher ambiguity and closer to customer and GTM.

Where does refactoring fit in? “Make the change easy, then make the easy change.” (Asked by GC)

Part of the system is encoding preferred patterns/architecture, so low level refactoring can be achieved that way. Higher level system refactoring guided by an engineer (typically in response to running into some limitation that we need to evolve our design around).

Multi-part: (1) How are you making your codebases AI-readable, especially at your scale of millions of lines across Ruby, TS etc.— have you seen the limitations of grep yet? (2) Have you found particular codebases have been harder to bridge to being agentic-first development? (3) How have you been measuring whether a codebase change actually improves agent accuracy? (Asked by OT)

(1) Honestly, not doing much here, some in directory overview documentation, some top level repo documentation to help explain how things are laid out. (2) Yes, a good example of this has been seeing agents/models far more effective writing high quality react code vs ember code. (3) Using a system of evals.

Do you have a monorepo / multiple applications which all share structure so that skills translate across projects? (Asked by RM)

We have a small number of very large repos - e.g. our rails app, a front end application repo, a python repo where much of our machine learning lives etc. We also have a general philosophy of making boring technology decisions, so there is a lot of consistency across all our code bases. So much of the skills that are focused on those primary repos become useful beyond, e.g. we have other rails apps, react apps.

Cost, token spend, and ROI

Were you able to get to an ROI calculation per agent (tokens + cloud costs vs. value)? (Asked by AC)

Yes, kind of. So far, we look at fully loaded cost per PR, and as token spend goes up, and PR throughput follows, fully loaded cost drops dramatically.

How do you manage spend of your tokens across the org? Did you consider leveraging open source tooling? (Asked by AK)

Right now we monitor, but don’t limit. As spend has ramped up we spot systematic improvements that yield cost savings. For example, we realized playwright was defaulting to screenshots that were needlessly high resolution, incurring unnecessary token usage, optimising this saved a projected $300k annual costs without any loss in performance. We aren’t using any specific open source tool here.

How much of developer AI spend ends up being spent on AI skill development? (Asked by RG)

We would have to check more deeply, but by ~PR count it’s ~5%.

API usage billing here – what about the bundled team costs or the enterprise plans? Huge discounts there. Not material? (Asked by IK)

At our scale, API based billing was the only path available to us, some discounts at scale, but not massive. Costs definitely require oversight in the medium term.

Roles, juniors, design, and cognitive load

How do you see the effectiveness of different seniorities when using AI tools? How are juniors growing now that there’s a push to outsource all coding to Claude? (Asked by AK)

The general trend is that some of our highest judgement most experienced people are seeing the highest leverage, however some junior people bring wonderful levels of curiosity, and hunger to learn, and near unbounded levels of energy, and much less hard wiring to the old way and are really thriving. I think we’ll bias towards AI-pilled people with experience who can likely own discrete areas of the product/system, but I’m rooting for the juniors too.

How has this transition affected design? Has design Xx’d output also? (Asked by DB)

Yes, designers are shipping, designers are contributing to the system, mindset of a designer, and generalisable to everyone, is for the things I do, that are often repetitious - is there a way I can encode that into the system so that it can be automated, and similar thinking for places where historically work would block on their input. Trend is they work on higher ambiguity work, and are perhaps pulled in async on other things. Easier/faster to build real prototypes/explore multiple directions in parallel. Now it is expected that designers can/do fix papercuts or small bugs, rather than cut issues, or badger their engineer peers to do it.

Have you found any issues with cognitive debt as Addy Osmani defined it: comprehension debt is the growing gap between how much code exists in your system and how much of it any human being genuinely understands. When we wrote code by hand we would find we couldn’t remember writing it 3 months later; now with AI writing the code we really don’t form a lot of memory about the code and over time have a widening gap of our understanding of the codebase. (Asked by RC)

We think about it conceptually, but haven’t seen major evidence of it being a problem for us. One counter balance to it being hard to keep up with all the change, is that it’s easier than ever to absorb unfamiliar context just in time, and our systems also encourage richer context being captured in our code base and in our git history that adds richness to the context that’s available at our fingertips.

Are you struggling with way more context switching? (Multiple projects/things to think deeply about, etc.) If not, how? We are. 😂 (Asked by L M-C)

Yes! We are all managers now, multi-tasking is often the default, it can be exhausting, we need to still be deliberate at taking focus time for deeper thought. AI does buy more time for us, frees us up from so much of the toil - our approach is that directionally that our agents should be able to do all aspects of the job a senior engineer does, and most obvious places to start are taking the least interesting work off our plates so we can focus our precious brain cycles on the hardest/highest leverage ones.

Culture, training, mindset, and cautionary tales

You mentioned about “pulling up the floor” so that all your staff embrace AI into their BAU. What cultural, training and awareness practices have you implemented (that are beginning to show returns) that have staff both engage and utilise AI – especially staff outside of the classic technical dev roles? (Asked by BMcA)

Couple of things. Early on we wanted to make sure everyone was having a real ‘aha’ moment with respect to how powerful AI tools, for us specifically Claude Code, can be. We had an internal competitive hackathon where people aimed to build a very simplified version of Fin, and compete against some secret LLM judges for the highest performance. Everyone, no matter what their role, had to participate. It got us all tackling problems harder than most of us would have believed we could tackle in a very compressed time window, and exposed us all to how powerful the tools are. At a lower level, yes, plenty of training, plenty of show and tell, people who’ve done interesting stuff demo’ing or documenting it, to inspire others, things like no-IDE weeks, to bootstrap people into an agent first mindset.

What about your situation before you engaged in this effort — systems, processes, people – was crucial to enabling it? (Asked by CS)

Culture that was focused on shipping, and systems/processes optimized around that (e.g. CI/CD).

How hard was it to let code quality slide at the beginning and what was the mindset you needed to have there? I guess that’s the scariest thing for teams that are high performing with high quality bars. Was there any time you weren’t sure you were on the right path? And what was the turning point? (Asked by SA)

Honestly, our take at the time was that much of the measurable drop in quality was acceptable as it was often within reasonable encapsulation boundaries - i.e. a class that’s internally a little messier or more verbose than we might like, or tests that are less precise/targeted than we’d like, but overall working as intended. We also believed in the eventual exit velocity, that as we improved the guidance and context to our agents we could steer them towards higher levels of quality and adherence to our preferred approaches, and that in doing so we could also target and improve legacy areas of lower quality in scalable and autonomous ways.

What’s the biggest mistake / cautionary tale that we should avoid? 🙂 (Asked by DW)

Dragging our heels, not pushing hard enough, being slow to invest in a team to own the holistic system, being slow to pick an opinionated approach, being slow to make it clear how our jobs are/need to change, and the evolving expectations on us all, under forecasting token spend (even though we all thought we were adding generous padding) and now dealing with all the reactive financial planning work.

Tell us about a failure of applying AI in your organization. What did you learn? (Asked by AK)

I think resolving open-ended investigations such as in incident response quickly and accurately is still WIP for us. More often than not, even with a good deal of context about our environment, our agents still confidently identify plausible suggestions that are red herrings. More work to do here! More meta but when we started the effort, our approach was to be maximally permissive, supporting everybody to try any/all tooling. We weren’t opinionated enough early enough, there were gains but mostly in isolation, individuals figuring out personal leverage but rarely compounding across the whole team. It took us too long to centralize on a platform. after choosing claude code, it became immediately obvious that the impact was in improving that platform, and seeing every gain automatically compound across the whole organization.

What are the 1 or 2 things that teams should focus on to get 80% of the value? Fin has a massive R&D team so I am curious about how smaller companies can learn from you. (Asked by MK)

Decide what you want to see, measure it, dedicate appropriate capacity to it, approach it systematically, smaller teams, or teams starting now likely have better off the shelf options for some parts of the system, tools are improving at a rapid rate, so maybe especially for smaller teams they can catch up quickly.

You must have had some scary stuff or near misses happen during the adoption and learning process — tell us! (Asked by SS)

Danny tweeted about this war-story, our biggest WTF so far was when an agent got quite confused, and ended up pushing hundreds of thousands of changes to our Rails monolith, which initially slowed CI down substantially, which escalated to completely breaking our ability to use the repo at all. The good folks at Github ended up having to do surgery on our repo so that we could get back shipping. Agents really will try anything to work around blockers, and sometimes get things terribly wrong.

Another type of nearmiss is in how easily Claude Code will innocently exfiltrate internal info - we realised this early and locked it down - e.g. private gists with problem spec md files - you need to understand and support the kind of workflows your team needs and make sure the path of least resistance isn’t a bad path.

Beyond engineering: Bottlenecks and other teams

Can you also share details how the Sales, CS, and Marketing teams are utilizing AI? Are they using the same infra you built? Do they serve themselves? (Asked by AC)

We built something called “Claude4Data,” built on same CC system, more non-engineers than engineers active Claude Code users. Expanding approach to Claude for sales, general approach is role specific skills and agents that share same data/tool access as human team members, skills that automate common tasks for that team. Also similar to R&D building our the general systems around these teams, e.g. automatic review/feedback on customer interactions, just in time enablement etc. Expect us to share more in the year ahead.

When R&D output 3x’s, the bottleneck tends to move – generally out of engineering and into decision-making, PM throughput, or the narrative layer for the rest of the business. Where did it land for you, and what did the communication design have to become to absorb and benefit from this 3x increase? (Asked by IG)

Claire answered this excellently. Check out what she had to say in the recording.

What’s next?

Why isn’t this a product yet? 😉 (Asked by CS)

It would be a fun one to productize, but we’ve got to focus! I expect there are / will be great products in this space and that e.g. Claude Code is already eating into some of the areas we had to build (e.g. session analysis like Brian talked about). I would add a cautionary note here that the things we are doing are a direct continuation of the culture we’ve been building for over a decade, it’s hard to productize culture, and some of these systems will be very opinionated/over-fitted to our culture. Even with off the shelf tools, you need a very strong internal push, business goals alignment, and willingness to take risks to make progress here.

That’s it! Thanks for reading this far. Thanks for all the questions, and your time/attention on the webinar. If there’s demand we’ll likely do more of these. Always happy to talk 1-1 with our customers who are looking for advice or insight into how to do similar things within their organizations.

When a frontier lab releases a new model, you might assume it’s gotten better at everything. Reasoning, coding, Ancient Greek, customer service, all enhanced in one fell swoop. And as this model moves horizontally, mastering disciplines, it will gobble up all the AI products built atop it.

But we’ve seen evidence that isn’t true.

Before we launched Apex – our own model, trained specifically for customer service – we ran Fin in production on Claude Sonnet 4.0. We tested later releases, Sonnet 4.5, 4.6, and Opus 4.5, and none outperformed Sonnet 4.0 on our RAG customer service task.

Turns out, the latest aren’t always the greatest – at everything.

As Fergal, our Chief AI Officer, puts it: “I think there will be some tasks that you can saturate in terms of intelligence and extra general intelligence doesn’t really move the needle for those tasks.” That means, some AI products, built with specific judgment and expertise can outperform the general models, even if they rely upon them.

Fergal recently joined The Chief AI Officer Show to talk about what three years of building AI products teaches you about models, pricing, and why Agent adoption is slower than it should be.

Unseen effort

Two years ago, many AI businesses heard the same criticism: “you’re just wrapping the model.” Cynics assumed that the journey from ChatGPT to Fin was easy – that what we were building was just a thin layer on top.

Fergal thinks that was “a total misunderstanding… People did not understand how much work you have to do around the model in order to deliver a product that was valuable.” He believes our investment in AI was “deeper and broader than any new entrant,” requiring a significant portion of R&D effort.

Today, people talk about “the harness around the model.” That framing is also flawed.

An AI model isn’t something you build a harness around, but a component, one building block in a larger engineered system. For Fin, the general models showed up at defined points in its architecture to deliver specific constrained functionality. Everything else was underlying infrastructure.

That will change over time. Models will take on more. But as they do, the pressure to specialize – rather than rely on horizontal general intelligence – increases alongside it. “You want to make sure that you’re continuing to add a big thick layer of value between the model and the set of business problems that you’re addressing.”

Why the frontier model isn’t always right

When trying to explain a model’s general intelligence vs. ability on specific tasks, and why AI products can survive outside of frontier labs, Fergal provides a thought experiment.

You’re hiring a retail assistant in a clothing store and you have two choices: a Nobel Prize winning physicist with a really high IQ or a retail assistant with 20 years of expertise.

Initially, you might want to go for the physicist – they’re smarter. But in reality, you only need so much general intelligence for the task. You also need emotional intelligence, sales experience, and personality. That’s true in the real world and Fergal believes it’ll be true for models.

This is one of the reasons he and his team began building our own specialized model, Apex.

The other was the significant improvement in open-weight model performance. Once they began to approach frontier models, it opened a new strategic option: take a strong open-weight model, apply a significant reinforcement learning program, and teach it the exact tradeoffs your domain requires. Initially, Fergal just expected Apex to be cheaper to run, but during training, he soon realized “it’s going to be better by the time we’re done.”

The long-term play for application-layer companies is vertical integration: build or fine-tune models optimized for your specific domain, rather than relying on horizontal models carrying capability you don’t need.

The gamble of outcome-based pricing

When we launched Fin with outcome-based pricing at $0.99 per resolution, we were pricing at a loss. The cost to serve a single resolution (running GPT-4 with dedicated capacity) was around $1.50 at the time.

The decision to price that way came down to two bets Fergal owned: that inference costs would fall, and that resolution rates would rise. Both felt like genuine risks but he “was very confident in them.”

However, the pricing model wasn’t primarily a unit economics decision. The more important reason was alignment. Outcome-based pricing makes every part of the organization – sales, customer success, R&D – want exactly what the customer wants: a resolution that actually works. “That’s not to be underestimated.”

Three years later, outcome-based pricing has become the expectation for high-value AI products. The market caught up.

Adoption is slower than the technology deserves

Fin now automates 83% of our own customer support volume and we’re not an outlier – we have customers with even higher automation rates. The technology works. And yet customer service adoption across the industry has been slow.

According to Fergal: “To my mind we should have seen much more radical adoption of Agents in customer service than we actually have.”

The reason is structural. Many businesses view customer service as a cost centre and are therefore slow to adopt new technology. Unlike coding Agents, there’s no product-led growth dynamic pulling people in – no engineer spinning up a trial, no bottom-up pressure from people who’ve already seen it work.

The skepticism is understandable, even if it’s expensive. When we tell people that 83% of our support volume is fully automated, they often don’t believe it. “In the back of their head they’d be like, it’s probably 30%.” We’ve had to learn to show the distribution of automation rates across our customer base to make the numbers credible – even internally.

AI is underhyped, Fergal argues as people are pattern-matching to previous hype cycles and find it hard to take genuine capability seriously. The diffusion curve is playing out but slower than the technology warrants.

To hear the full conversation and Fergal’s thoughts on evaluating models, the company’s initial reaction to ChatGPT, and if society is prepared for the impacts of AI, tune in here.

In B2B AI, there are two users:

1. The customer configuring the system.

2. The end-user asking a question and getting a resolution.

It’s tempting to collapse these groups together when designing, but the first user, the customer-builder, needs different evidence to understand and trust the system they are helping to create.

Most AI systems are black boxes by default: inputs go in, answers come out, and the system in between is hard to inspect. A lot of trust work focuses on making the answer more transparent – with citations, visible sources, input context, status signals, and clear admittances of uncertainty. That matters. But it mostly serves the end-user.

Customer-builders need visibility one layer deeper: “Why did this happen, and what can I do about it?”

Consequently, you must build two glass boxes.

When there’s only one visible thing, that’s what gets blamed

In AI products, the final end-user answer is what both the customer-builder and end-user see and judge. But a bad answer can come from many places: missing content or knowledge, incorrect config, e.g., guidance or guardrails, or the model itself. If those layers are invisible, every failure collapses into the same explanation: “the AI is broken.”

Here’s a story from the trenches back when I was on the Fin Content team. A prospective customer-builder was trying out Fin for the first time. As part of the setup, they entered their website URL into the content page and saw hundreds of pages come back as “enabled.”

What the content page looked like before. No visibility, simply “enabled” / “not enabled”. That didn’t communicate the nuance of website imports, which are the most popular type for new customers trying out Fin for the first time.

To them, that meant the system was ready, so they used the Preview panel to test Fin. They asked a product question they already knew the answer to and Fin got it wrong. From the outside, it looked like a hallucination, since the correct answer was on their site inside an FAQ accordion.

The actual cause was upstream: the page had been detected, but the content inside the FAQ accordion’s collapsed section hadn’t been successfully read, so the answer never made it into Fin’s knowledge base and it responded with incomplete information.

Two trust failures in the same moment, for two different parties. Had this reached an end-user, they would have seen a wrong answer with no signal it was wrong. The customer-builder almost concluded “Fin is not good” because the import surface said “synced” and gave them nowhere else to look.

Fin doesn’t usually fail in isolation, so the fix is rarely at the model layer. Most issues are caused during set-up – wrong content, configuration, or guidance. So the product team must build a glass box around these processes.

In this instance, that means allowing the customer-builder to see what has been read, what was excluded, what failed, and why. Otherwise, “Fin is broken” is the only conclusion the customer can draw, because Fin is the only part of the system they can see.

What goes under each glass box

When we rebuilt the content import experience to expose failure modes and enable troubleshooting, we built a glass box for that part of the experience.

Glass boxes build trust. But “show everything” isn’t a strategy. Revealing too much creates noise and false confidence. Too little leaves people guessing. So the real work is judgement: what goes in the black box, what goes under glass box one for the customer-builder, what goes under glass box two for the end-user.

The principle I keep coming back to is one I borrowed from data science. Early in my DS career I used to put two decimal places on high-level trend reports for product marketing because it felt more rigorous. Actually… for product marketing, what mattered was the trend. The decimals were noise dressed as precision. The lesson learned: detail to the extent it’s actionable for the audience. Anything beyond that is distraction, regardless of how true it is.

The same principle applies to glass boxes, with one twist: actionability isn’t the same for everyone. We must ask: what can each audience do with what they’re seeing?

• Glass box one: the customer-builder. Their action is “iterate the configuration,” so the surface shows what’s accessible to Fin and what’s configurable; what content was available and retrieved; what guidance shaped the response; what task is being executed. Anything they can change, they need to see. Anything they can’t act on stays out.

• Glass box two: the end-user. Their action is “decide whether their query is resolved,” so the surface shows what helps them make that decision: the answer and sources that they can verify, should they want to. Anything beyond it is two-decimal-places of noise.

A small example: one team thought Fin had become inconsistent because PDF source links appeared in some answers and not others. Initially, the model was deemed flaky, but the real cause was a guidance rule that needed iteration. The team could see and fix it because the Preview tool showed the one configuration choice that was producing this behaviour.

The harder version of this judgment is the one that goes the other way.

Sometimes the right call is to leave detail out because it confuses more than it helps. The website import surface is an example we lived through. Underneath it, website imports can fail for a long list of low-level technical reasons: different kinds of network issues, blocking, parsing problems, etc. We could have surfaced every one but chose not to, as most customers can’t act on the raw error type even when they see it. We collapsed the long tail into a small number of high-level categories, each tied to a fix the customer could actually make. The glass box got smaller.

Remember: visibility isn’t a virtue. It’s a tool, and like any tool, it can make things worse when it’s the wrong size for the job.

Once you accept ‘detail to the extent it’s actionable,’ how and where you present it matters just as much

A glass box is only useful if it sits next to the thing it’s about. Configuration on one page and logs on another means the customer has to reconstruct context every time they look. Where possible, don’t make them travel or hold things in working memory to find the evidence for a decision they’re making right now.

Example 1: the Content page after. Beyond “enabled”, the page tells you what failed, what’s live, and once the fixes are addressed, the user can take an action to re-sync.

Example 2: the Preview panel. It sits inside the same pages where Fin is configured. It works as a glass box in two ways: it lets customers test a new configuration in the place they just set it, and it surfaces every event Fin was configured around so the customer can trace the root cause of an answer.

Beyond Fin

If you’re developing AI for builders – people who configure it for their own end-users – you are building two glass boxes at once. One you control end-to-end, and one your customer assembles in your interface.

The second one is the part most teams underbuild. It’s also the part that decides whether your customer can trust your product, and confidently put their name on it.

Sometimes you see a corporate announcement that’s so obvious and so late, it’s almost an admission of failure. And I’m open to feedback on how I should have pulled the trigger on this change much sooner. But the tech industry is so littered with awkward and arguably unsuccessful name changes that I thought it prudent to procrastinate on this one.

But finally, today, we’re changing the name of our company to Fin. The name Intercom will still live on as the name of our customer service software platform that increasingly the best brands are choosing as their primary help desk; we just launched a complete rebuild, Intercom 2, and we’re dramatically increasing our investment in it. But the company behind it will be named after our leading customer agent platform, Fin. All of our 1,400 employees are now employed by Fin. You’re reading a post by the CEO of Fin.

In this moment of great change, for our industry, and for all economic players, the winners of yesteryear have largely been left holding on to their former positions that brought them so much glory. Yet, as I’ve written before, the only path to success in the future is through destroying your past. We’ve done this with our culture, our pricing model, our product lineup, and our investment priorities. But the company name stayed the same.

However, when we started the service agent category three years ago, we promoted our new technology with the name Fin, with Intercom very much in the background, and usually not even mentioned at all. The simple reason is that humans are very good at mapping meaning onto the world and then updating that map rarely, with a very strong bias for the things that are new and shiny. We couldn’t possibly wake up every morning questioning what a chair was and how it works. But when a new type of drone or artificially intelligent software product reaches our attention, we do our best to understand it and add it to the repertoire of things we understand about the world.

This is true in all aspects of life, including with companies and their brand names and the positions they hold in the market. I believe this so thoroughly that I actually think that the relative success of the newcomers in our category, despite the fact that we have provably superior technology, is a result of the fact that they have no baggage. They don’t need to convince anyone of their new position in the market, because they never had an old one.

So, today we say, “baggage be gone.” Fin is clearly our future, and the future of this new customer agent category. It’s driving our renaissance and is about to be the largest part of our business. We thank sincerely everyone who supported Intercom in these past 15 years. And again, the product and its name will very much live on. But our company is now finally, fully reborn and its name is Fin.

AI will always fill in the gaps. Whatever you don’t specify, it decides – toward the average of everything it’s trained on. A generic purple gradient or repetitive sentence structure (hello ‘it’s not X, it’s Y’) or corners that always seem slightly-too-rounded.

The interface encourages short, simple input. Most AI tools have a prompt bar only about three lines tall. And while that level of detail might work for quick fixes, it will leave much to be desired when you’re building something more meaningful.

As Michelle says: “Every blank you leave is a decision you’ve handed over.”

When it comes to getting worthwhile performance out of your Agents or AI systems you need diligence upfront. For us, that looks like providing Claude with 300 detailed skills, hooks that enforce how it works inside our codebase, and read-only access to production data.

The goal is simple: the less Claude has to guess, the better the output.

This week, we recap stories from across our team on how they collaborate with AI. Including: the workflows and org structures that deliver best results, the experiments they’re learning from, and how they’re feeling about the changing nature of work.

When you don’t have the words

Over the past few months, Róisín has been building Nudge – an app that helps busy friends stay in touch.

The frontend came easily. She’s a designer; wireframes are her bread and butter and Claude turned these into working prototypes. With her experience, she could identify an issue: “This button is too far from that input” or “The error message doesn’t explain what went wrong.” And then she and Claude would debate the best solution and action it.

The backend is a different story.

Claude had to take the lead.

“The terminology alone was like Klingon to me.”

Róisín started every session by reminding Claude to do everything in the browser rather than the terminal and that she was a complete beginner. It walked her through procedures step-by-step, explaining as it went, and worked hard to detangle screenshots of console errors. Over time, she’s learned what error codes mean and how to debug simple things on her own.

But Claude can only do so much.

Nudge’s pending invite status kept throwing 400s and after debugging it three times, Claude suggested cutting it. The feature wasn’t essential for beta testing and it was delaying the valuable insights she’d gain from shipping to users. It was the right call.

Learning what to cut, when to stop iterating, and what’s good enough is something Róisín is learning, and in her own words: “You learn it by shipping.”

Read more about Róisín and Nudge here.

Three lines isn’t a brief

Most coding Agents have a mode called “dangerously accept all permissions.” Michelle argues the bigger risk is the posture most people bring to AI: dangerously accept all opinions.

AI presents recommendations confidently, making you feel like it knows best. It doesn’t – it knows the average of its training data. In most cases, that’s not enough.

Her piece walks you through three practical tools for building the muscle of better direction:

• Plan mode: Separates thinking from doing. Map out intent before anything gets built, then push beyond the build plan to the product plan: what should this feel like, what would make someone use it a second time?

• Skills: Small bundles of instructions that give AI better judgment in specific areas. Worth bookmarking: Superpowers for brainstorming, Impeccable’s Shape skill for structured discovery interviews, Get Shit Done for spec-driven planning.

• Living docs: Brief.md, design.md, architecture.md saved in your project folder. The AI references them every session. Your prompts get lighter because it already knows where you’re coming from.

According to Michelle: “The building is the easy part. The hard part is knowing what to build and being detailed and opinionated enough about it so what gets built is actually yours.”

Get all of Michelle’s tips here.

The treadmill that punishes mastery

With the rise of AI tools, designers are now being asked to run two scorecards simultaneously – technical fluency and craft – and it’s a tough balancing act. Worse still, according to John, the demands on each are ramping up.

As the AI space gets more competitive, new tools and models come to market almost everyday, making it hard to achieve true mastery. The external pressure doesn’t help. The AI corner of design Twitter and every internal Slack channel runs on a constant drip of “look what I shipped this weekend.” Any designer who pauses to think feels like they’re falling behind.

“At the same time, you’re still expected to be a world-class designer. Now that anyone can ship, the bar is only going up – for craft, for taste, for the thousand small decisions that make products feel great.”

John fears the designers attempting to excel in both disciplines risk running themselves into the ground. But he also thinks a reprieve could be on the horizon: “There’s probably a correction of some sort coming. The pace isn’t sustainable.”

In the end, “the designers who come out of this strongest won’t be the ones who ran hardest on either axis. They’ll be the ones who stayed clear-headed enough, and rested enough, to do the judgement work that compounds.”

Read his full piece here.

Exit the comfortable middle

Karen leads a 40-person Research and Data Science (RAD) organization at Intercom. Her conclusion from the last few months of working with AI isn’t what you might expect.

“AI commoditises execution. It does not commoditise judgment. And for research and data science teams, that’s the difference between being relevant and being replaced.”

The work most at risk to AI automation, she argues, isn’t the obvious tactical work. But the comfortable middle – the incremental analysis that feels strategic, a dashboard update that is visible but never changes a decision, or simple research exercises. All of this can now be handled by stakeholders with the help of AI.

So how can researchers and data scientists maintain their relevancy?

According to Karen: they can move in three directions:

• Down: Encode business context as infrastructure to make self-serve trustworthy.

• Up: Move from delivering insights to building decision systems.

• Back: Lean into the human edge, the slow, formative work that doesn’t exist in any dataset yet.

Read her full analysis here.

Write it all down

We’ve written a lot about our 2x initiative but no one has spoken about it more than Brian.

Last week, he sat down with Akash Bajwa to talk through what building AI-first looks like inside a 15-year-old Rails monolith.

Getting Claude to perform reliably within such a code base meant building a harness around it – with linters, standardised patterns, and explicit written guidance about the right way to do things. Brian actually thinks the process “mirrors human onboarding”: you show Claude what good looks like, which parts of the code represent the old way, and how to get things done. The difference is you have to write it all down.

How is the harness working?

Brian highlights two metrics: quality of code and cost per PR.

Over the past few weeks, code quality at Intercom is going up, after an initial period of expected decline. Brian credits the auto-approval process for this as it encourages engineers to produce good code on the first pass. PRs that are small, single-purpose, use feature flags, and stay out of dangerous code paths get approved without a human reviewer. Kesha and Niamh recently wrote about this process.

When it comes to costs, Brian encourages companies to avoid becoming fixated on tokenspend. What matters more is the fully loaded cost of a PR including salaries, management overhead, review time, and rework. When you measure that, the economics of AI-assisted engineering look very different. “We have an awful and ever-growing bill,” he says, “and yet the cost per change is dropping. That’s exactly what you want.”

The interview also covers hiring for openness over experience, why committing to Claude Code is less of a risk than staying agnostic, and how org structure has changed.

Read the full interview here. Or Brian appeared on the On Rails podcast and the HangarDX podcast to talk about our use of Claude Code, so tune in if you want to go deeper.

Our final thought

When anyone can ship, the real difference maker is your point of view and your ability to communicate it to your AI. Getting great results requires great documentation, down to every detail.

The idea that software is developing at a pace never seen before is now more than trite. But implications of it are still being reckoned with. One interesting result is the breakneck pace at which categories are evolving. It used to be the case that software categories would mature for a decade before there were opportunities to specialize them for specific verticals, but that timeline just won’t make sense anymore. This Fin announcement is an interesting example of this.

Today we’re announcing Fin for Ecommerce, by far the world’s best shopping assistant agent. The best real life, in-store experiences almost always involve interactions with knowledgeable, helpful experts who close the gap between your problem and their solution. But online you’re left alone, navigating menus, dropdowns, and filters that represent inventory, trying to find products by trial and error, and unsure what’s even available. This is a big reason why 70% of carts are abandoned. Customers are unsure what to buy and have no help.

Fin for Ecommerce fixes this. The world’s first expert shopping assistant, working across the full customer journey, that can move seamlessly from sales to service, delivering perfect customer experiences and driving conversion and revenue.

It works to understand customer intent and has deep knowledge of real time inventory and price changes. It handles abstract questions and guides the customer to the right product. It manages difficult product and service questions, and moves customers towards conversion, adding products to the cart. After purchase, it knows everything about the order, can chat about order status and problems, and can take action to resolve customer change requests. Best of all, you can connect to your store and have it working with customers literally in minutes.

This all runs on the Fin architecture that serves over 12k customers and many millions of conversations per week, and on Apex, the world’s first and best customer agent model. It’s driving epic early results already, with significant lift in average order value (AOV) and higher cart conversions.

What surprised us most about Fin for Ecommerce is how quickly it delivers high-quality support with minimal, non-technical setup

Fin for Ecommerce is already driving meaningful revenue, with 10% of conversations converting to orders averaging 20% above our store AOV. It’s doing the work of a sales and support team combined.

The handoff between support and sales is so smooth I can’t tell the difference without checking the filters. Fin talks policy, sells products, and references our mattress break-in period all in one conversation.

This represents a further demonstration of our philosophically unique approach to this category. Our competitors are still mired in their consultingware approach, where they must hack together, customer by customer, brittle bespoke solutions. While we’ll always customize our offerings for our customers who need it, you can go from zero to one (with one being the best performing in the market) faster with Fin than anything else out there. If you’d like to see the latest and greatest example of this, I encourage all ecommerce companies to sign up for Fin for Ecommerce today.

Today, people use Agents to do tasks – Claude Code can quickly fix that bug that’s been annoying your customers, or OpenClaw can optimize your calendar each day. But increasingly, Agents are also making decisions on our behalf.

If you’re a SaaS business, this means you’re no longer just selling to a CTO or founder, but the Agents they’re using. Ensuring your product is discoverable, configurable, and trustworthy to an Agent is the next product surface worth designing for and prioritizing.

Agents build what they can’t buy

At Intercom, one of our engineering philosophies is to run less software. Don’t build what someone else has already built better.

Agents should work the same way. But they often can’t, because SaaS wasn’t built for them. And as a result, they end up preferring to build instead of buy.

To sign up for most software products, you open a browser, fill in a form, verify an email, and navigate a configuration UI. Sometimes you even have to get on a call with the sales team. That’s fine for humans, but Agents can’t do any of it (yet). So when an Agent needs feature flags, it probably won’t sign up for LaunchDarkly, it’ll build its own feature flag system from scratch.

Agents love to solve problems on their own and they always try their hardest; they’re not always going to ask you to help them navigate a sign-up flow.

The implications of this for providers are best articulated by Aaron Levie, CEO of Box:

“The platforms that are easier for agents to adopt, and solve the agent (and user’s) problems the best, will get ahead far faster than those that don’t. Agents won’t be going to your webinar or seeing your ad; they’re just going to use the best tool for the job, and you’ll want it to be yours.”

We’re already seeing the first wave of providers prioritizing agentic accessibility. When combined with strong brand recognition, they become an Agent’s go-to solution. For example, when setting up payments, Claude almost always picks Stripe. If you need website hosting, it heads to Vercel.

Right now, the financial impacts of Agents’ software preferences are limited. After all, developers are the largest users of coding Agents and they tend to have their own preferences for software and can steer an Agent elsewhere, especially if they’re working in an existing tech stack.

But over time, as Agents become more capable and non-technical users adopt them to set up and run their businesses, an Agent’s choice of software quickly becomes the market’s default choice. Providers hoping to get a leg up in this environment must make SaaS directly consumable by Agents and not assume that an OpenClaw-style browser and email access will provide a quick fix.

What Agent-first means

Since the launch of ChatGPT, there’s been considerable chatter about Answer Engine Optimization (AEO). It’s the LLM counterpart to Search Engine Optimization (SEO). In brief, you optimize your website and content so it will be understood and cited by an LLM. By doing so, you bring organic traffic and potential customers to you. If the LLMs can’t find you, neither can consumers, and it will only get worse as LLMs take on more search volume.

In a similar way, a Command Line Interface (CLI) is the Agent’s equivalent of easy-to-navigate and appealing UI. It allows Agents to understand and use your product. The better your CLI, the more likely it is that an Agent will choose your product when building. Some providers opt to use a Model Context Protocol (MCP), which functions like a wrapper around an API, but CLIs can take on tasks with multiple steps and help Agents get more done.

Creating a CLI is step one of preparing for the Agent-first future, but step two is having a product that can function out-of-the-box. Not “easy to sign up for” – actually deployable by a non-human without any engineering help.

That’s a much larger product issue, not just an interface one. You can’t wrap a CLI around a product that needs hand-tuning. Agents will hit a wall and look for something else or deliver a disappointing experience to users. This grants a considerable advantage to SaaS providers that are already self-configurable and places pressure upon those that are not.

What we built

I’d been watching the rise of agentic software preferences for a while – talking about it on Slack, posting about it, generally making noise. Jordan eventually told me to stop complaining and actually build something. So we did.

The Fin CLI is built for Agents and can be launched by a single prompt. It signs up to Intercom and Fin, configures the Messenger, pulls in your help content, and sets it live. It also returns contextual feedback to the Agent at each step – what to check, what to set up next, and where to go if something isn’t working.

The vision is this: someone asks Claude to build them a product, then says, “I want to talk to my customers and handle support automatically.” Claude says “yes,” and gets Fin up and running all on its own.

AI Agents orchestrating AI Agents. That’s where this goes.

Check out our CLI at fin.ai/cli

Shipping fast is our priority

We’ve long held a belief that might sound counterintuitive: speed is not the enemy of safety. It’s a prerequisite for it. Accumulating code creates risk. Shipping small batches minimizes it. The faster you ship, the smaller each change is, and the easier it is to catch problems, and roll back when something goes wrong as the context is still fresh in your head.

Today, over 93% of our pull requests (PRs) across our two main codebases are Agent-driven. And over 19% are auto-approved with no human reviewer in the loop.

This post is about that second number, and why we think it makes us safer. Most people hear “AI is approving our pull requests” and think that’s reckless. We think the data tells a different story.

The 2x effort

Last year, our CTO Darragh Curran set an explicit goal: double the productivity of our entire R&D organization within 12 months. Because the faster we can build and ship, the faster our customers get the capabilities they need.

Nine months later, we did it. The results have been significant across the board, but the number that matters most for this post: downtime from breaking code changes dropped 35%, even as our deployments doubled. Shipping faster made us safer. As we modernize how we build and ship software, we systematically surface bottlenecks and tackle them. One of the biggest we’ve found? PR review.

The PR review problem

Humans simply don’t have the time or mental capacity to properly review the volume of AI-generated code we’re now producing.

When an AI Agent can produce a working implementation in minutes, waiting hours or days for a human to review it is an impedance mismatch. The production line is moving faster than the quality gate can keep up.

When that happens, one of two things follows: either the queue backs up and velocity drops, or, more dangerously, humans start rubber-stamping. Glancing at a diff, skimming the description, clicking approve.

Some companies are drifting into this failure mode silently. We chose to confront it head-on and built a rigorous solution.

PR review, done properly, is a genuinely complicated process. A good reviewer assesses the quality of the problem statement based on the PR description. They confirm the changeset actually matches the stated intent. They review the code against best practices, look for logical issues, apply their personal product context to validate the changes make sense, and check for performance issues, safety concerns, and more. No single human reviewer realistically has experience to properly cover every one of these dimensions on every PR, especially not under time pressure. And as we’ll see in the data, the baseline we’ve been comparing against, human review, was weaker than most of us assumed.

So we asked ourselves: what if we could do better?

How our PR review Agent works

Our PR review Agent doesn’t treat code review as a single task. It decomposes it into separate sub-jobs, each handled by an independent sub-Agent. One assesses the quality of the problem description. Another checks whether the diff actually aligns with the stated intent. Another reviews for safety concerns. Another checks for logical correctness. Another reviews against best practices and known anti-patterns. And so on.

The result is that every PR is reviewed as if a dozen of our most tenured and knowledgeable engineers were all looking at it simultaneously, each bringing their own specialist lens. In the past, getting that breadth of review on a single PR was impossible. Now it’s the default.

A human reviewer typically focuses on the actual code changes, the diff. Our Agent goes deeper. It traces execution paths, following the implications of a change through the codebase. This is something humans rarely had time to do, even when they wanted to.

A real example:

While testing our new PR review Agent on a set of historical PRs, we found it flagging a one-line text copy change as incorrect. On the surface, it looked completely harmless, just a text update. We assumed it was a mistake, but it wasn’t. Our Agent caught that the new copy contradicted an existing validation mechanism elsewhere in the codebase. No human reviewer would have realistically found this unless they happened to have written that validation code very recently. Our Agent catches this kind of thing consistently, every time, because it’s always tracing execution.

The review isn’t generic either. It’s grounded in Intercom-specific guidance that our engineers have built and continue to refine, encoding the same context, standards, and product knowledge they’d apply if they were reviewing the PR themselves. When the Agent reviews a PR, engineers flag whether the review comments were helpful or not, and that feedback continuously sharpens the guidance. It’s a flywheel: the more our engineers invest in teaching the system how to think about our codebase, the better every subsequent review gets.

Automated approval is also never forced. Any engineer can request a human review on any change, at any time. The system is a tool, not a mandate. At Intercom, shipping code doesn’t end at merge. The engineer who ships a change is expected to watch it go live, monitor its behaviour in production, and be ready to roll back if something isn’t right. AI approval doesn’t change that. The human who ships the code remains accountable for the outcome.

Not just faster; safer

The naive take on AI-approved PRs is that it’s just a rubber-stamp LLM call so that humans don’t have to bother. A convenience feature. That misses what’s actually happening.

Our Agent is strict. It won’t approve large PRs. If a change is too big, too complex, or too broad in scope, it flags it and requires it to be broken down. Creating a direct, positive incentive for engineers to ship smaller, more incremental, well-scoped changes.

This matters enormously for safety. Small changes are easier to review, easier to test, easier to understand, and, critically, easier to roll back when something goes wrong. This is the same principle that has always underpinned our shipping culture, but now the PR review Agent actively enforces it.

It’s tempting to look at a goal like “>50% AI-approved PRs” and worry that we’re optimizing for the metric rather than the outcome. We see it differently. Our goal was to remove a bottleneck that, if left unchecked, risked pushing engineers towards rubber-stamping reviews under time pressure.

On the surface, our PR review Agent is only a solution to “humans don’t have time to review PRs.” But what it really is, at its core, is a safety mechanism. It ensures we continue shipping fast, in small increments, maintaining and increasing our heartbeat, even as the volume of AI-generated code grows.

The data

We didn’t assume AI review would be good enough, we actively ran an experiment.

Our hypothesis was that AI review could match or outperform human review quality, measured by the outcomes that actually matter: were the changes correct? Did they cause problems in production? How quickly were they reviewed and approved?

We started with a controlled pilot of over 100 PRs through the AI approval pipeline. The results: zero reverts of AI-approved PRs, and a 6–16x improvement in time-to-approval at the 75th percentile. Since then, the system has scaled significantly. In the first four weeks of broader rollout, 497 PRs went fully autonomous, with Claude writing the code and our AI approval system reviewing, approving, and shipping to production.

Beyond the approval pipeline itself, we also looked more broadly at how AI-authored code performs in production compared to human-authored code. AI-authored backend code had a revert rate of 0.53%, compared to 5.39% for human-authored. On the frontend, it was 0.22% versus 2.00%.

AI-authored code, reviewed and approved through our automated pipeline, is being reverted at a fraction of the rate of human-authored, human-approved code. We don’t expect that to hold at zero forever, but the data so far tells us that the quality bar our Agent holds is at least as high as the one humans were holding, and in many cases higher.

And perhaps the most important perspective shift: those product changes that did cause outages in the past? They were all reviewed and approved by humans. Human review is not a guarantee of safety. It never was. It’s a useful heuristic, but one with real limitations that we’ve been quietly accepting for decades.

Staying compliant while we innovate

Everything we’ve described in this post, the sub-Agent architecture, the traceability, the labelling, the data, none of it was built just to make the system fast. It was built to make the system auditable. That was a design constraint from day one.

Every AI-approved PR is labelled, logged, and queryable. The review comments, the approval decision, the test results, the merge event: all recorded. The evidence an auditor expects to see is the same whether a human or an AI approved the change. The “who” may change, but the “what” doesn’t.

We engaged our auditors, Schellman, early, before we scaled. We proactively worked with them to confirm that our automated review processes and the evidence they produce meet the requirements of our compliance frameworks, including SOC 2, HIPAA, ISO 27001, ISO 42001, and AIUC-1, among others. We think AI-driven change management can meet and exceed the standards that human-driven processes set, and we want to help prove that.

We see this as a feature of building the system the right way, not an extra hurdle we had to clear. When you build for safety, compliance follows.

What’s next

You can only go so far with PR review as a safety mechanism, no matter how good the reviewer is, human or AI. Only in production do you discover the unknown unknowns. The majority of Intercom’s largest outages weren’t even caused by changes to product code at all. They were infrastructure issues, unanticipated customer usage patterns, or third-party outages. PR review, whether human or AI, was never going to catch those. That’s why, in parallel, we’re also working on an Agent that proactively diagnoses issues in production. We’ll share more on this soon.

Speed has always been at the core of how we build, not in spite of safety, but because of it. And we’re getting even faster with AI. It’s easy to assume that AI-approved PRs would lead to a drop in quality and safety but our data proves otherwise. We’re just getting stronger.

We’re announcing the most significant new strategic development since the start of the Customer Agent category: Fin now has specialized roles. And starting today anyone can sign up for and deploy the Fin Sales Role in minutes.

Fin is now by far the very best sales agent on the market, and it’s been live with some of the most innovative digital brands for months, conversing with thousands of prospects, aiding discovery, building pipeline, booking meetings and starting trials.

We started the Customer Agent category with a focus on service by launching Fin three months after the launch of ChatGPT. Since then Fin has continued to lead the space, today delivering over 2 million resolutions a week for over 8k customers, including Anthropic, DoorDash, Snowflake, Asana, Mercury, Polymarket and many more exceptional brands.

But our vision for Fin has always stretched far beyond service. And for the past 6 months we’ve been building Fin as a single Customer Agent, delivering a seamless experience across all stages of the customer lifecycle.

We do not believe customers or businesses will want multiple agents for different parts of the lifecycle—because the agents won’t have shared memory and goals, and will have to compete with each other, delivering a poor customer experience and sub-optimal business results.

We’ve more roles to come, including another dropping in two weeks. For now please watch our launch video or visit our launch page to learn more: fin.ai/sales

In my last post, I shared a conversation with Matt, Dan, and Jack about what makes engineers effective with Claude Code. I was trying to understand what’s behind the success that some people are having with these tools – is it years of experience? Deep judgment? Strong mental models? And whatever it is, can we encode it and help everyone get there?

That conversation was with people who’ve been here a while – tenured, senior, already thriving. This time I wanted to talk to somebody new. Salman Shah has only been at Intercom a few months and he’s already having an outsized impact – not just on his team, but on AI adoption across the company. He’s shipping at a rate that stands out. While also contributing to shared tooling, running informal coaching sessions with teammates, and building relationships across the org that most people don’t establish in their first year, let alone their first few months.

I wanted to understand what’s going on – what is it about how he works and thinks that’s producing these results? And what that tells us about the AI traits we should be hiring for?

AI + human, not AI vs. human

Salman’s foundational mindset comes from an unexpected place: a chess book. After Garry Kasparov lost to IBM’s Deep Blue in 1997, he created advanced chess, also known as centaur chess – tournaments where both sides played as AI + human teams. The question was no longer: “Can AI beat humans?” but “What does AI + human look like when it’s working well?”

Salman read Kasparov’s Deep Thinking and it changed how he approaches working with AI:

“My mindset for every task is not ‘how do I type something and go away?’ That’s where I feel a lot of people have felt frustration. They try and then realize, ‘oh, it just wasted my time.’ Instead, how can you do AI + human through everything?”

You always have to ask: what does the human add, and what does the AI add? Not “can AI do this for me?” but “what does the AI + human version of this look like?”

Nobody said no

Salman and I talked a bit about his career path – seven years across various technology companies before joining Intercom. At those places, you get used to asking permission. At Intercom, the opposite happened.

He said: “Here I have tried asking for permission and people have been like, ‘you don’t need permission to do that.’ I’ve just done so many things and up until now, no one’s come and told me, ‘don’t do this.’”

In his first month, this culture was tested. He pushed something, broke something, got paged at eight o’clock in the evening for a minor incident.

“I was very worried the next day. But then I was told not to worry about it – just get the learning out of it. Which is to have an additional test that would have caught it.”

That feedback emboldened him and the message was clear: we’d rather you move fast and learn than move slow and play it safe. And each time he tested that boundary without getting burned, the boundary moved further out.

“Fueled by AI, but also fueled by energy, ambition, and nobody saying no.”

Claude Code first – in everything

This is the most actionable theme from the conversation, and probably the biggest mindset shift.

Salman doesn’t just use Claude Code for writing code. He uses it in incidents, planning, QA, and code review. His question for every process is: “Can we do Claude Code first in this?“ – and not just within the existing process. Can we tear the process apart and rebuild it around Claude Code?

He built an agentic QA skill that runs test scenarios overnight. The old process: someone prepares a QA sheet, it takes three days to get through QA and fix what comes up. Now by the time a human QA session starts, most issues have already been caught and resolved.

An incident example: Salman threw everything at Claude Code – Sentry MCP, admin tools MCP, the codebase – and had a PR out within 30 minutes, on a part of the codebase he’d never worked in before.

“I did nothing special. It was related to reporting, which I have no idea about. So we had someone from the Reporting team look at it. But the fact of the matter is – can we have this mindset shift, that Claude Code can just do things better than us if we give it the right context and just go Claude Code first in everything?”

That phrase – “if we give it the right context“ – is doing a lot of work. Salman’s insight is that when Claude Code fails, it’s almost always a context problem, not a capability problem. Give it the right MCPs, codebase access, and framing, and it performs. The skill is in the setup.

He’s done this in three incidents now. Each time, he shipped a PR during the incident that would have taken much longer without the tool.

An incident is high-stakes and high-visibility – not an obvious moment to reach for a relatively new tool. But when Claude Code is already your default for everything else, there’s no switching cost. You just use it.

Build the network

Before he even joined Intercom, Salman reached out to Brian Scanlan, one of his interviewers, to talk through whether he should accept the offer.

“I spoke to him once and I was like: ‘hey, I’m not sure, should I join? Can we just chat?’ And he said: ‘okay.’ And then we met in Dublin my first week. And since then we’ve been talking monthly.”

Salman also met Eugene at a talk before he joined Intercom and reached out to set up a chat. Eugene and Brian recommended other people for Salman to meet. Within a few months he’d built relationships with senior engineers across the org, using them to understand the landscape, find problems, build trust, and contribute to shared tooling.

This is the human side of thriving with AI tools, and I think it’s easy to overlook. No amount of Claude Code proficiency replaces the fact that you need to understand the people, the problems, and the context around you. Salman is explicit about this:

“There is always a human aspect of things. When you’re working with people, I think that probably just helps.”

The principle here is to seek out context actively, don’t wait for it to come to you, and invest in relationships that help you understand the system you’re building in. It’s a skill that will compound.

The local maximum problem

We got into a back-and-forth about something I keep seeing: the local maximum problem. Most people don’t know what they don’t know or the extent of these tools’ capabilities.

I described the pattern as I see it:

“Someone adopts Claude Code and starts shipping AI-written code. They get comfortable and that feels like the ceiling. But the ceiling isn’t a missing feature or a tool they haven’t installed – it’s the boundary of what they’ve imagined the tool can do. Someone sees Claude Code as a coding tool, and then one day realizes it can also do code review, or planning, or incident response, and suddenly the ceiling moves.”

Salman sees the same thing from the ground. He put numbers on it: maybe 50% of the org is at beginner, 30% in the middle, 20% advanced. The adoption curve is left-shifted. And the people at the top are compounding each other’s knowledge while everyone else doesn’t know what they’re missing.

This echoes what came up in my earlier conversation with Matt, Dan, and Jack – the forums and channels we have for discussing AI tend to select for the enthusiasts. The people who most need help aren’t represented in the conversation. When Jack ran his learning sessions in London, he was hoping to reach people at the far left of the curve. Lots of people showed up, which was great – but the hardest problem is still reaching the people who don’t yet know what they don’t know, what they’re missing, or how to engage.

Push the graph up

Salman tracks his own DX graph as a personal benchmark. He pulled it up during our conversation – a steady upward trend since his first day at Intercom.

“Whenever I find anything that is blocking me, I put it into a skill and see if I can push this graph up.”

This is his operating principle: each week, what’s slowing me down? Can I make it faster? Encode the answer into a skill. Compound the gains.

For example: the React inbox refactor. 280 PRs in four weeks driven by an autonomous Claude Code workflow. When Anthropic released agent teams, daily PRs went from four or five to 15. And the flywheel doesn’t stop at the end of the project:

“We’re going to spend a week collecting all that information and putting it back into a skill so that the next time we do review, we don’t have to think about those things because they’re already captured.”

This is knowledge encoding – he doesn’t just solve a problem, he encodes the solution so it persists for the next project and the next person. The skill becomes institutional memory.

At Intercom we’re using DX metrics to help us to understand Claude usage and proficiency. Sometimes when I talk to engineers about these metrics, they cite Goodhart’s law, and worry that the metrics are gameable and therefore meaningless. Salman flipped it on its head. For him, it’s not a performance evaluation metric, but rather a self-reflection tool.

“What’s helpful for me at the end of the week, if I’m doing self-reflection, is to ask: ‘is there a way to improve this?’ It’s easy to count. If you want to find how to get efficient, whatever can show you that makes sense.”

He’s deliberately gaming his own graph – but not in a cynical way. He’s not finding low-value PRs to inflate a number. He’s driving through high-impact work while also compounding low-hanging fruit on the side.

The 80/20 split

Salman has always been a split-focus person. Eighty percent on the main project, 20% on side quests – defects, feature flag cleanup, plugin contributions, whatever catches his eye.

“80/20 was always the principle. It’s just now I’m able to close the loop on the 20, even if the 80 gets harder – because AI is doing a lot of it for me.”

Before AI, the 20% was aspirational. You’d pick up a side thread and it would balloon in scope and you’d quietly drop it. Now the closing cost is so low that side quests actually get finished. Last cycle he cleaned up 30 feature flags. This cycle he’s chipping away at defects – two this week, compounding over time.

He’s not precious about it either. His PR close rate is about 7% – meaning roughly one in fourteen PRs he opens, he closes without merging.

“Sometimes you pick up something, you feel like it’s a lot more effort than what you initially thought. You don’t have the mental bandwidth for it. Just close it.”

Energy management over completionism. Starting something new is so cheap, being willing to abandon the ones that aren’t worth finishing becomes a real advantage.

Right time, right place – and right mindset

Salman is self-aware about the luck component. He arrived at Intercom with a lot of energy and a great mindset – but he also arrived at just the right moment.

“I came when the models were just getting better and better. And I leaned into betting on Claude Code, which is what Intercom also bet on.”

Seven years in technology companies and he’s never had this kind of impact before. At one, the culture dampened down exactly the energy that Intercom rewards. At another, the business wasn’t ready to bet on AI. The timing never aligned.

“I’ve never done this in my career. This is the first time I’m doing something like this in a very big way. I have never shipped as much as I shipped last month.”

But it’s not just luck. The combination is specific: culture (Intercom’s permission to fail) + timing (the agentic inflection) + personality (ambition, extroversion, initiative) + AI (the compounding enabler).

“It’s never just one thing. It’s probably a combination of everything. The culture at Intercom allows you to do that. Plus maybe some part of my personality always wanted to. But I have never done this – whatever this is.”

What’s staying with me

My earlier conversation with Matt, Dan, and Jack was about learning from tenured engineers at Intercom who’ve already shifted their mindset and found great success with Claude Code. Salman shows that’s possible even without long tenure at Intercom – with initiative, a compounding mindset, and a habit of treating every friction as something to encode and eliminate. He doesn’t have twenty years of engineering judgment. He has a relentless instinct to close loops, share what he learns, and ask “can I do this with Claude Code first?”

So what are the AI-native traits we should be hiring for? I think this conversation surfaces some of them:

• Initiative without permission.

• The instinct to tear apart a process and rebuild it, not just optimize within it.

• A network-building drive that generates context and trust before you need either.

• The discipline to encode what you learn into something that persists.

• Reaching for Claude Code first – not just as a coding tool but as the starting point for every problem.

These traits reinforce each other: the network gives you context, context lets you deliver, delivery builds trust for bigger bets, and each bet generates knowledge you encode into the next one. Over time, they compound.

I want to push the adoption curve to the right – to accelerate as many people as possible through their own adoption journey. Conversations like this one help. They surface the ways of working that make the difference, and encourage us to create an environment to foster them.

If there’s one thing to take away from this conversation, it’s the question Salman asks himself for every task: “Is there a Claude Code first way to approach this problem?“ That’s the mindset shift. Start there.

To hear more from Salman on what it’s like to work at Intercom, check out this video:

And if you’re looking to join a team that embraces this mindset, check out our careers page.

Nine months ago, we set a very deliberate goal to 2x the productivity of our R&D org within 12 months. We’d already seen early benefits of AI-assisted engineering, but knew there was so much more coming. My instinct at the time was simple: even if AI progress stalled, it was already good enough for us to hit this goal, but we must deliberately chase it and overcome all obstacles in our way, as opposed to being passive and hoping it happens to us.

This level of public accountability was scary, and it wasn’t always clear to me that we’d pull it off, but…. we did it, with time to spare. In fact, looking back over 16 months we’ve 3x’d and I see no signs of plateauing. I anticipate we will 2x again in a short number of months. I want to give you an unfiltered look into the impact we’ve seen holistically for our product and our org, and how to go about making similar changes in your own company.

This will be a multi-part post. This first part will mostly be about what we’ve seen. In the next, I’ll talk about our messy journey, some of the lessons learned along the way, and what’s coming next.

A culture optimized around moving fast

Our culture has always been optimized around building great product, fast. If you think of your R&D org as a factory, ours was a fairly impressive factory, one others would aspire to replicate. However, technological change, in this case with AI, can very quickly raise the ceiling of what is possible, and you must modernize your factory to not fall behind. We’ve never had a shortage of product vision but have always been constrained by executional capacity. AI unlocks an abundance of capacity, dramatically amplifying our ability to execute against our product vision.

We aspire to be at the forefront of exploiting what is possible with AI, and as a result, more quickly and efficiently build our vision for the future, solving customers problems faster than anyone else. Ideas that would previously take weeks or months can now take hours or even minutes.

We aren’t the only large R&D organization deliberately chasing this kind of outcome, and indeed, I respect and applaud large organizations who have also gone really hard (e.g. this wonderful post shared a few days ago by Ramp). But I’ve also talked to enough people to know that most are struggling to make the right changes or see close to the level of impact that’s possible. I hope by being transparent about our journey we can inspire others to follow a similar path and encourage others to push us to go further too.

For context, while we are going through an exhilarating rebirth phase, we are nonetheless a big, old, beautiful, messy company. Thirty thousand businesses rely on us for how they run their business. We’ve almost ~8.5 million lines of application code (across Ruby, TS, JS, Python, Go, Swift, ObjC, Kotlin, Java). Our PlanetScale datastores alone peak above 2 million QPS. Over the last 24 hours, we ran 2,539 Continuous Integration jobs, each consuming days of compute time, and made 313 distinct automatic deployments to production. We see velocity as a strong driver of stability. There are ~500 people building and operating the product and software that we run, these people are spread across four offices in two continents.

In short, we are not a small startup with the benefit of a blank page, but we know full well that if we don’t adapt aggressively we are heavily threatened by them, and so are you.

There are so many companies similar to us, alive today based on great success in the old world, but dead if they don’t adapt aggressively to the new world. We want to thrive in this new world. And we want to show you it’s possible and inspire you to make the, at times, difficult decisions and changes to achieve it.

Wait, you measured what now?

We measured the number of merged PRs divided by the total number of employees in our R&D org. This particular measure is easy to criticize – are we really going to fall into the trap of just looking at the output of the engineering team in this naive way? This isn’t far off measuring LOC? None of this is done in isolation, our thesis for measuring PR throughput is as follows:

• We believe in fast, safe, and frequent shipping. A new engineer on the team will ship to production on their first day. Shipping is our heartbeat.

• Conceptually, our org is a factory for producing high-quality increments to our product. These leave the production line as merged PRs (pull requests). Our job is to increase the throughput of the factory, while maintaining or improving quality and decreasing cost. Time, cost, scope, pick three!

• Putting higher expectations on merged PR throughput puts pressure on the whole system, exposing bottlenecks which we then systematically tackle.

• We also look at this top-down. What we actually want to optimize is how frequently and how quickly we can go from problem to solve or idea, to solution in our customers’ hands. Both views are useful, but at different zoom levels.

• It’s important to highlight precisely how we define our metric, total number of merged PRs, divided by total number of people in R&D (everyone from IC engineers to VP of product design). We divide by all the people, because 1) that’s all the cost; 2) we expect all people to be able to directly and indirectly impact the metric; and 3) it puts healthy pressure on our org design, by default we should assume all our existing org structure patterns are wrong for the new world.

So many people internally and externally get caught up on this question of how to measure this, afraid to measure the wrong thing, or hold back from driving change because they can’t find the perfect measure. My advice to you is to be at peace with potentially imperfect measures, don’t let that hold you back from the outsized outcomes that lie in store.

So, yes, this metric is not quite perfect, but we’ve driven massive movement in it, hit the goal/milestone we set, AND paid close attention to everything else that changed.

I want to give you an unfiltered and transparent view into a range of benefits we’ve seen, that are well within reach if you push for them with determination.

What has been the holistic impact of our AI acceleration?

1. Product defect backlog shrunk by 54%

Since the start of the year, we’ve reduced a core product defect backlog by 54%, closing all critical and high defects. We are also on track to imminently close all medium defects. Our intent here is to stop wasting time on the drudgery of prioritization and backlog management, the issue ping pong, etc., and instead attempt to operate around an SLO of fixing all reported issues in less than 24 hours (currently at 10.4% and climbing), and maintaining effectively no backlog. Very few product teams I know have ever achieved this high standard, but it’s now well within reach. All this is obviously a well-deserved win for our customers, and just the start of all the ways they’ll start to feel the impact of our internal improvements.

2. >2x product changes, 39% faster from idea to shipped

We’ve started to see two strong trends. We are shipping more product changes, and we are shipping them faster. This is noisy data, not all product changes are equal in size, scope, etc., you can have spiky months, but it’s directionally indicative. Median time from idea to product change is nearly cut in half, and even if we ignore December, we’ve more than doubled our product changes in March vs January. We are able to say yes to more of the things that matter most to our customers, and be more ambitious in what we aim for too.

The faster we can responsibly ship, the faster our customers get leading edge capability, the faster Fin resolves their support conversations, and the faster they can grow their own businesses.

3. Code quality improving, after period of consistent decline

A legitimate worry with the use of coding Agents, is that they won’t write high-quality code and the craft we’ve fought to protect will be undermined by slop cannons. We have a system to rate the structural quality of code contributions using static analysis and various rules/heuristics. It’s clear that prior to agentic coding, this metric would oscillate up and down above the line. As we started to use AI for writing more and more of our code, the overall quality (by this measure) declined. My intuition was that this was inevitable in the short term, but correctable in the medium term, as models and harnesses get better. We are starting to see this and recently had possibly our first ever five-week streak of net positive code quality overall. I’m sure this streak won’t last forever, but my bet is that average quality will continue to trend up.

4. Downtime from breaking code changes down 35%

It’s natural to assume that doubling our deployment throughput means we’re breaking things more often. But that hasn’t happened.

The data shows something even better than “no degradation.” The downtime we can attribute to breaking code changes dropped by 35% over the last 11 months, even as our deployments doubled. Our culture has always been optimized around building great product, fast, but also safely. What’s really happening is that by letting AI handle the mechanical aspects of coding, our engineers have the mental bandwidth to be far more deliberate about architectural quality.

5. AI costs growing exponentially BUT costs per PR cut in half

Our AI spend has been climbing sharply, exactly what we want to see. We’ve not tried to optimize this despite believing there are ample opportunities to do so. For now, our focus is on AI leverage and we believe we are only scratching the surface. This chart is our weekly spend on Claude Code this year. I expect this to continue to grow steadily through the year.

When Agents can do work humans would otherwise do, it’s almost always a huge ROI. One way we look at this is by tracking fully loaded costs per PR. Payroll-related costs massively dominate this, and tend to have strong YoY upward pressure (promotions, pay increases, market adjustments, etc.). However, with ever-increasing PR per head, the overall cost per PR is in a steady decline. December is higher because between time off and other end-of-year stuff, people are far less productive, but the overall trend continues.

6. 93.6% of PRs are Agent-driven

We drove a big behavior change in a short number of weeks. As soon as we made the decision that we’d double down on Claude Code as our primary agentic coding system, we set a goal that within six weeks, >80% of PRs would be driven by Claude Code. We then raised that to 95%, which we are currently hovering just below.

Many of our engineers rarely even use an IDE/editor.

This is all inline and driven by new principles that describe how we should now work. The first and most important is:

“All technical work is becoming agent-first. This is the top priority for R&D.”

We will share our full set of “2x principles” in the days ahead.

7. 19.2% of PRs are now AI-approved. Near-term goal is >50%

Creating PRs isn’t the only part of the “factory” to apply an Agent-first mindset to. With the ramp-up in PRs, unsurprisingly, one of the bottlenecks that gets exposed is PR review and approval. If we want our factory to move faster, we can’t have every increment waiting in a human review queue.

We took an ambitious goal to hit >50% AI PR approvals by the end of April. We’ve started with the lowest risk changes – feature flag cleanups, small bug fixes, focused improvements. Only a few weeks in and we’ve already seen 19.2% of all merged PRs auto-approved with no human reviewer in the loop. The system evaluates roughly 60% of PRs and we expect both coverage and approval rates to climb as we expand the criteria.

This not only reduces the demands on others to spend time on PR review, but reduces the waiting time to get PRs shipped. Auto-approved PRs merge in a median of 14.6 minutes compared to the org median of 75.8 minutes. Eighty-six percent of auto-approved PRs are 20 lines or fewer – exactly the kind of well-scoped changes that shouldn’t bottleneck in review queues.

Perhaps the most striking number: 497 PRs went fully autonomous in the first four weeks Claude wrote all the code, with our AI approval system reviewing, approving the change, and shipping to production. We are also ramping up the percentage of PRs driven entirely by Agents, without humans triggering them.

You’d be right to be anxious about making a change like this, but we are approaching it in a very methodical way, and will share more about our auto-approval approach in a follow-up post.

8. 31% of our org are active contributors to our auto-updating suite of Claude Code plugins

A big turning point in our progress was the decision to go all-in on Claude Code and build a shared platform around it. We spun up a dedicated team (“team-2x”) and created a private plugin marketplace that distributes configurations and skills – specialized capabilities for common tasks – via auto-updating plugins. When someone builds a skill to do a task in the best way possible, everyone on the team immediately benefits. When we improve a skill that everyone relies on, the improvement ships to everyone automatically. In the three months since we started this, our Claude Code plugins repo has become the most actively contributed-to project at Intercom – 153 contributors (31% of R&D) and 267 skills. This isn’t a platform team building for others; modernizing the factory is now part of everyone’s job. Some more details in Brian Scanlan’s viral thread about this.

9. Top 5% of contributors produce 6x the median PR throughput

I still think we are only getting started. We are learning at a rapid pace, but a big area of opportunity is the unevenly distributed nature of progress across our team. There is *huge* latent potential. Some people are really pushing the limit of what is possible, tokenmaxxing, doing really interesting things, while others have only really made incremental changes to how they’re working and don’t see much change in their personal throughput. Ultimately one of the biggest bottlenecks to progress is with humans; how we work together, how we change behavior, etc. We’ve made some great progress and I’m confident we can crack this too, but I want to show a sober view of how unevenly distributed the impact is. In general, the people who are able to spend the most with Claude are having the largest increase in personal throughput.

This scatter plot of our engineering team shows that the vast majority of people who are meaningfully pulling up the average are spending >$1k per month on tokens, but spending >$1k doesn’t guarantee that.

It’s ironic that the sentiment amongst many leaders or CFOs is that this is all becoming too expensive, our evidence suggests that it’s way more expensive in the long run to hold back spend, because of the downward pressure AI leverage plays on the fully loaded cost of a PR, which is dominated by salary costs in almost all organizations.

We need to figure out ways to help everyone move up and to the right. We think about this as a progression from minimal, to elite use of agentic tooling. Our top 5% of engineers operating at an elite level produce >6x the output of the median engineer.

The tiering is computed based on an assessment of an individual’s intensity of AI usage, overall output, depth of AI usage, efficiency of AI usage ($/PR), and prompt quality. All our telemetry on these dimensions helps us spot opportunities to better equip or enable the team to make progress. We will share more on this later.

Changes beyond R&D

The impact has been significant within R&D, but also inspiring beyond R&D. I’ll give a glimpse into that here.

Everyone reflexively uses AI now as part of their job, whether that’s Glean, ChatGPT, Claude, Claude Cowork, or in our case, Claude Code. We’ve 473 people in R&D, our peak has been almost 1,100 active Claude Code users across the company (out of total company size of 1,305). That’s over 600 people who got over the temporary intimidation of learning how to use the command line and are now able to create reports in minutes that would have taken them hours, or that they simply couldn’t have done before. Regularly, I bump into people brimming with excitement, from our finance team, or people team, or recruiting team, or customer service team, or sales team, or legal team, who are on a really fun journey of discovery around what they can get done from an AI Agent with access to the right data and tools. Some of this use is one-off analysis, some of it is replacing complex parts of our operations, with specialized skills being built to report and critique monthly performance. And people push this beyond just reporting, building special purpose tools for sellers or even prototypes of potential product capability.

Cormac wrote about how “every business leader deserves their own data analyst,” which totally undersells the impact of the tooling his team have created. For starters, it’s not every business leader, it’s everyone at your company. The tooling allows you to ask any question you want about the data in Snowflake and any other systems or context you are interested in,has skills for creating interactive reports or mini applications, and then publishing them securely with the right access controls, versioning deployed using Streamlit on Snowflake.

Usage has exploded from initial versions of this in February, and no sign of slow down in April so far, stats here for reports actually published to the internal platform for sharing them:

What next…

I hope I’ve shown you the power in picking a target, no matter how imperfect, and pursuing it with great determination, can lead to massive cultural change and impact for your company. I hope that the transparency I’ve afforded helps motivate and inspire you. I hope it gives you benchmarks to try and smash – or if you’ve already done so, I hope you’ll share your receipts too for further inspiration.

Tune back in soon for my next post, where I’ll share all about the messy journey, how we did this, what we’ve learned along the way, and what’s next.

One last thing, I and other leaders at Intercom have been inundated with customers and peers in the industry wanting to pick our brains and draw inspiration for how we are leveraging AI in our teams. So we are going to do a big group webinar, targeted at people leading organizations like ours, where we’ll go into more depth, and have very open and honest Q&A sessions too. Please save the date, May 19th, 9am PT, 5pm GMT, and register your interest here.

Big thanks to Claire Vo for very kindly reviewing a draft of this post at short notice and providing really useful feedback.

“Product strategy means saying no” became a popular line because it captured something essential: great products are not just collections of useful features. They are coherent systems. In a world full of good ideas, the job was to protect that coherence. To say no to tangents, no to edge cases, no to things that made the product busier but not better.

That is still true. But for years, product teams also lived with hard constraints. Too many good ideas. Too little time. Too few people. In that world, many no’s were strategic: this muddies the product, distracts from the core, or adds more complexity than value. Those no’s still matter.

But many others were different: this could be right for the product, but we do not have the capacity to prioritise it now.

Scarcity did not just kill ideas. It lowered the ceiling on ambition. It trained teams to think in increments. To pitch the safer version. The version most likely to survive planning. We told ourselves we would start small and iterate later. But later often never came. Another urgent problem arrived, we went shallow again, and the pattern repeated.

That tension was always there. The choices were often reasonable, but it never felt great to make a small dent in one important problem, then move on to the next, and rarely have the capacity to go meaningfully deep.

That is what AI changes.

AI changes the economics of software in three ways. Software that was already viable gets cheaper to build and improve. Software that was too niche or too low-value to justify before can now become viable. And the frontier moves outward: ambitious, high-value ideas that once felt out of reach can now be seriously pursued.

That changes opportunity cost too. It used to be the cost of taking capacity away from the obvious priority. Increasingly, it is the cost of not pursuing the more ambitious idea now that it is finally within reach.

So yes, product strategy still means saying no.

But teams that keep operating with the instincts of a scarcer era will miss the shift. The question is no longer just what to cut. It is where to go deeper, move faster, and be more ambitious because the executional constraints have changed.

The best teams will still protect focus. But on the right ideas, they will go harder, deeper, and further than before.

We spent 4 years building a world-class data foundation at Intercom. Trusted models, clean pipelines, proper documentation. And the people who needed it most couldn’t touch it.

When our President wanted to know why pipeline dropped, she filed a ticket with an analyst and waited.

That’s broken.

The access problem

It’s not that the data doesn’t exist. It’s that getting to it requires SQL and a dev environment. Business leaders don’t typically have those things, and they shouldn’t need them.

Meanwhile, generic AI tools can write SQL but they don’t know your data. They can’t tell the difference between your trusted source-of-truth table and the unvalidated one two schemas over. They give your VP a confidently wrong number, and they have no way to spot it.

The experiment that changed our minds

One of our engineers connected Claude Code to Snowflake for two business leaders as an experiment. The BizOps leader produced a private equity partner dashboard in minutes (previously days of analyst time). Our VP of Customer success diagnosed complex conversation volume trends solo.

Their response: “spectacular.”

But the setup needed terminal config, API keys, and multi-repo dependencies. We couldn’t scale that to 50 people, much less >400. So we built something that could.

What we built

We almost built a chatbot with a nice web UI. We didn’t. Because generic AI fails at business intelligence not because it lacks a pretty interface, but because it lacks opinion. Instead we built Claude for Data: an opinionated Agent on top of Claude Code, pre-loaded with our data team’s knowledge. Which tables to trust, how metrics are defined, where the gotchas are. Allowing it to start every session like a seasoned analyst who knows the business.

Four things made it work:

A zero-config installer. SSO auth for Snowflake connections, dependency checks, all handled invisibly. A business user gets their first answer in minutes. This was harder to build than any of the AI, and it mattered more for adoption than anything else we shipped.

A trust hierarchy. The Agent steers toward the best data model for the question. It knows a core, tested table from raw, unvalidated data. It can join across product usage, billing, and CRM to answer cross-functional questions.

A feedback loop. When something goes wrong, we capture the context. Domain experts review and update the guidance. Fixes reach every user automatically. Every bad answer makes the system better.

Shareable reports, and a portal to browse them. This turned out to be the real wow moment. A user can ask a question, get an answer, and then with a single command turn it into a polished, interactive report with charts and tables. That report gets published to the Claude4Data Catalog, a portal where anyone in the company can discover, browse, and learn from what others have built.

The catalog changed the dynamic completely. Before, insights lived and died in Slack threads. Now, someone in Sales can browse what Marketing built last week, fork it, adapt it for their own use case, and publish their version back. Reports show the SQL behind every number, so there’s full transparency on how the data was sourced. People leave comments directly on specific sections, ask follow-up questions, and star the reports they use regularly.

This is what made the tool go viral internally. They could see what their colleagues were producing, click through the catalog, and think “I want to build something like that.”

What happened

Within three weeks, weekly active users went from under 100 to nearly 400, mostly from non-technical teams. We’d planned a “Report of the Month” award to celebrate the best work in the catalog. We switched it to weekly because the output was too good to wait.

One member of a non-technical GTM team told us: “I truly believe this tool is the most transformative one I’ve ever seen in my career. My team is building new things every day, and we are not technical teams.”

A RevOps leader replaced a clunky weekly Tableau workflow with a self-serve dashboard in a single session. He called it “the data equivalent of a Waymo.” A Field Marketing manager built a live weekly report that evolved into a global marketing calendar used across the entire org, no data team ticket required. A Product Manager submitted her first-ever pull request to contribute a reusable analytics skill. Her reaction when it went live: “wowowowow.”

The last mile

Every GTM org we know is sitting on this problem: real investment in data, gated behind a small number of analysts. A decade of building dashboards hasn’t removed that bottleneck.

An opinionated Agent that knows what your data team knows, gets smarter from feedback, and turns conversations into reports people can actually share and build on. That’s the last mile.

Claude for Data is built on Anthropic’s Claude Code.

Today we’re announcing the launch of the Fin API platform. Our best-in-class vertical customer service models that power Fin are now available for you to mix and match and deploy at insane scale to create the perfect customer agent for you. In this post, I’ll explain exactly what we are announcing, why we’re doing so, and how I think we’re going to see a lot more of this in the industry at large.

The news

Fin is a customer agent platform that at present resolves over 2M customer issues a week, growing at a rapid exponential pace. It’s relied on by the best brands, large and small, in every vertical you can imagine. From Doordash and Riot Games, to smaller hot upstarts like Mercury and Polymarket. It runs on a family of models trained by our AI group. Last week, we announced Apex, which is the world’s first specialized customer service LLM. In production tests over the last 6 months, it beat every single frontier model, including those from Anthropic and OpenAI, on resolution rate, latency, hallucination rate, and cost.

We’re now going to allow you to access all of this power and all of our core models directly via API, with contracts starting at $250k per year, and usage rates that are by far the cheapest in the industry for each of the model’s subcategories.

But why?

It’s simply that our customers want it. We hear from people far and wide who want to build their own agents. So starting today we’re providing three ways to do so.

First, for the vast majority of companies, they will want to run their operations on the Fin Agent Platform. We have ~8k companies on it today. This takes care of the needs of 99% of customers and allows them to configure it easily without the exhausting consulting engagements of our startup competitors. It delivers the very best resolution rates in the industry, but straight out of the box.

Second, we have also had an offering for people who want to present Fin in a unique context. For this they can use the Fin Agent API. In this mode, you get all of the magic of the Fin platform, but you don’t have to use our messenger (or our email or voice or other prebuilt channel) and can display the agent in bespoke ways.

But there are also companies out there who want to build hyper-specific and specialized agents for their business. Perhaps they want to build an agent that does service and is also a product agent that lets users interact with their product. In this world, the best and most obvious decision for them is to use Apex and the collection of models we use in the broader system, because they’re trained for exactly that purpose—unlike the generalized models. This is our third and new offering launching today.

We’re also excited to see new startups build Fin-like businesses that cater to hyper-specific verticals too. Fin for dentists? Why not? Fin for car dealerships? Sure. We’re never going to build for these specific verticals, but we’d love someone else to. In fact, if any of our direct startup competitors would like to substantially improve their offering and give us a little cut of the action, we’ll be more than happy to license our models to them too. Decagon, Sierra, and the rest, you know where to reach me. Let’s be friends! :)

Coming soon to an agent company near you; the defensive reason

Unlike many, I’m not 100% ready to write off all of software. But it’s true to say that the software landscape is certainly about to change dramatically before our eyes. In extremely recent times, differentiation came from software functionality that acted as a moat because it was hard to build. But building software is simply less hard now.

We’ve already managed to more than double our measurable productivity on our engineering team. We’ve also created insanely deep new products that previously existed as separate businesses, built by single Intercom engineers, in literally one week.

Going forward, the differentiation that came from features and interface is, at the very least, going to diminish. Serious software companies must not only move from being a features company to an agents company, because the work they used to assist with that was done by humans will now be done by agents, but they must also be building those agents on differentiated AI. We do believe that more and more value will accrue to the model layer, and so, as we did when we started to disrupt our software business with our agent business, we will now begin the processes of disrupting our agent business with our AI business.

Where this all ends is anyone’s guess, but it’s hard to not imagine we’ll see this with many other companies too. For now, we’re excited to be out there first and best with this new platform and business and we can’t wait to see what people build.

When you’re a thriving business, it can be easy to fall asleep at the cash register. You’ve had success for so long, you forget how to adapt and react.

Take Blockbuster.

It kept dismissing the internet, there was always a rational-sounding reason to wait: people won’t want to watch movies on their laptop, patrons love the experience of strolling the aisles, the economics won’t work at scale.

Every single one of those lines crumbled. And by the time Blockbuster took the threat of Netflix seriously, it was too late.

Des has been thinking a lot about Blockbuster lately – because he believes most SaaS companies are living that story right now, just with AI. Last week, he joined the Billions podcast to talk about what it takes to avoid that fate and adapt to the AI age.

Here are the biggest takeaways:

A pivot among chaos

Fifteen days after ChatGPT launched in November 2022, Intercom was already building Fin.

The urgency was prompted by a demonstration from Ciaran. He asked ChatGPT how to install Intercom on a mobile app – the kind of question the support team gets regularly – and it answered it perfectly in under five seconds. “No typical human efficiency software would have got to that outcome and then of course it could answer it in any question in any language and it could do it 24/7.”

Even accounting for hallucinations and guardrails, it was clear this would change customer service irreparably.

What followed was a hard pivot: ripped-up roadmaps, reallocated resources, a new domain, and a new brand direction. Prior to that, Intercom’s topline revenue wasn’t shrinking, but growth had slowed and the company had already begun to restructure.

Adding AI to an already manic sea of changes ended up being a blessing in disguise.

Would Intercom have moved as fast if it had been growing at full speed? Maybe not.

According to Des: “The mistake we would have made if the business was doing great and we saw this opportunity, would be underinvestment… We might have said, ‘Oh, that’s cool. Fergal has a fun little experiment. Let’s let him work on that.’”

The fact that the company was already mid-pivot meant one more radical change was easier to absorb. As Des puts it: never waste a good crisis. If the floorboards are already up, fix everything.

Building AI is not like building SaaS

In SaaS, development is deterministic. You talk to customers, you scope the work, you build it, you ship it, you move on.

AI doesn’t work like that. As Des says: “You don’t know what’s possible. And if something’s possible, you don’t know how reliable you can make it.”

The process Des describes looks nothing like traditional product development.

Before a designer or product manager is involved, you start with the model and test for what new capabilities are now possible. This might involve what Des calls a “torture test”. The AI team constructs hundreds or thousands of scenarios with a known correct answer, and run the model against them. You need to ensure it can complete the task reliably, at or above human quality.

Then you can move into productisation, and determine what infrastructure must be built around it for it to work in the real world.

And once it’s live, you’re not done. You run millions of real scenarios through the system, watch how customers configure it, measure how it performs in the wild, and build reporting and oversight tools so customers can see what’s happening in real time.

You should not chase models

It might be tempting to assume that the path to a better AI product is upgrading to the latest model, but Des pushes back hard on this. When a company announces it has switched to a new model two hours after launch, it means one of two things: they had extremely early access, which is possible but rare, or they have no evaluation criteria.

The AI team’s process for evaluating a new model runs thousands of scenarios and compares three things: what current Fin would say, what the candidate model would say, and what the ideal answer looks like – what Des calls “God’s own support agent.” Only when the new model is demonstrably closer to the ideal does it earn a place in production.

Additionally, most of Fin’s improvements haven’t come from model upgrades. They’ve come from better AI architecture, prompting, and ways to disambiguate. Fin runs across 25 subsystems, many of which now run on models post-trained internally for specific aspects of customer experience. Change one subsystem, and you have to re-evaluate the entire system – because there are trickle-up and trickle-down effects you won’t see otherwise.

That kind of rigour is where the real competitive advantage lives.

You have to know what “good” looks like

If you want to do AI properly, you have to take it seriously. And that means having a clear definition of what good looks like. Without one, Des asks, “What are your team doing? They’re just kind of performatively running around in circles.”

In customer support, good is relatively well-defined. There’s a correct answer to “how do I reset my password?” You can write it down, put it in the torture test, and benchmark off of it.

In other domains, it’s harder, but the discipline is the same. Des uses an example of a LinkedIn post generator. The only true measure of good is whether the post performs. You probably can’t see that directly, so you find a proxy: acceptance rate, how often users actually click post.

You then follow these posts and collect examples that performed well, and maybe some human-written examples. Over time, you build a spectrum from terrible to excellent, and you use it to calibrate your model.

It’s difficult but it’s non-negotiable. Every AI team needs to be able to answer: what does great look like, and are we getting closer to it? Without that, you’re not improving.

What founders should do right now

We’re four years into AI. Des’s advice to founders is to stop thinking in co-pilots and start treating it as a reinvention.

The first step: find someone you trust who is genuinely good at AI, and ask them what percentage of your product still requires a human in the near future. For most SaaS companies, the answer is going to be a lot smaller than expected.

According to Des, you then ask yourself: “If you were starting this product again today and you were good at AI, what would you build and where are humans, if anywhere, essential? And you would build a product that uses AI everywhere it can be made reliable… and boring old SaaS UI wherever you need humans involved. That’s your new product direction.”

Getting there will require painful decisions – roadmap cuts, unhappy customers, engineers resistant to new tools. But the alternative is the Blockbuster path: a string of rational-sounding reasons to delay, right up until it’s too late.

The time is now, as Des says: “You might have one or two years left.”

To hear the full conversation with Des, where he talks managing AI hallucinations, pricing AI products, and Fin’s future as a customer experience Agent, watch on YouTube or listen on Spotify.