Who is liable when AI gets it wrong?

The conversation has moved on. Have our operating models?

Something shifted in the last twelve months. The CX world stopped asking "can AI do this?" and started asking something much harder: "Who's in charge of it?"

The numbers make uncomfortable reading. 95% of customers want to understand why AI makes decisions about them. Only 37% of CX leaders currently offer any reasoning at all.

Some major brands that committed to full AI automation have quietly reversed course. Gartner predicts no Fortune 500 company will fully eliminate human agents by 2028.

The technology gap has closed. The operating gap is wide open. Practitioners want operating model guidance, not technology demos.

Accountability and how to live with Insights being "wrong"

Ian's question isn't really about blame. It's about the absence of agreed rules and process. But before we get to process, there's something more fundamental worth naming: customer feedback data is messy.

Just because a lot of people are talking about something doesn't mean fixing it will solve the problem you think it will—sometimes it's signal, sometimes it's noise.

The best organisations we work with have stopped treating VoC analysis as a machine that produces definitive answers and started treating it as a tool for generating and testing hypotheses:

1. Listen — Look at what your customers are telling you.
2. Hypothesise — Form a hypothesis about what's driving it.
3. Act — Change something.
4. Measure — Quickly check whether it moved the KPIs you care about.
5. Iterate — If it did, do more of it. If it didn't, back to the data.

That shift—from VoC as verdict to VoC as continuous improvement loop—is enormously helpful from an accountability perspective.

The reframe that matters: You're not asking "was the AI right?" You're asking "did our process help us learn faster?" That's a question any organisation can own.

Transparency—a defensible position, not a black box

A head of insight at a major travel business, watching her team try to QA an AI-generated thematic report: "I sat down to check what the AI gave me, and I'm like... no, that's not it."

She couldn't tell how the AI reached its conclusion, so she couldn't interrogate, defend, or fix it. That's a transparency failure as much as an accuracy one.

Our work on definition-led themes is aimed directly at this. Every classification model we build comes with a clear description of the criteria it uses to decide whether a piece of feedback is in or out. That's what gives insight teams a defensible position when leadership, auditors, or regulators ask "how do you know?"

We're sometimes asked whether we use LLMs for classification. The honest answer is: yes… and no. With our new Definition-led Themes, we use LLMs to create and train our classification models.

They're extraordinarily good at understanding nuance in language, generating training data, and helping us build models that genuinely reflect how customers express themselves.

But when it comes to applying those models to your data at scale, we use conventional machine learning. Customer data never touches an LLM. That matters for two reasons: cost—running LLMs across tens of thousands of feedback responses gets expensive very fast—and consistency.

A machine learning model trained on defined criteria makes the same decision with the same data every time. An LLM might not. In a regulatory report or a board presentation, that inconsistency isn't a quirk. It's a liability.

Accuracy

AI is never 100% correct—but neither are humans, and language is unpredictable in ways that make human-only analysis less reliable at scale than people tend to assume.

Reach 85-95% accuracy and you have more than enough to see in real time how customer sentiment is shifting and to identify what's worth acting on. The question isn't perfection. It's whether your tolerance level is documented and defensible.

Fairness, security, and staying awake

We've been looking at bias in AI since 2020—earlier than most—and it remains something we actively monitor rather than assume is solved.

On security: our industry is starting to see a rise in AI-generated survey completions—automated responses that look like genuine feedback but aren't. Distinguishing real human sentiment from AI slop is becoming a live challenge.

And none of this matters if nobody's watching continuously. A theme that captured genuine signal in January can become a noisy catch-all by June. Governance isn't something you set up once.

So, back to Ian's question

Your organisation is liable and this isn't just a governance opinion, it's a legal reality. Under English law, AI has no legal personality. It cannot be held responsible for the harm it causes.

Legal responsibility attaches to the integrator—the organisation that controls the system and benefits from its use. If analysis from an AI tool informs a decision that harms a customer or a tenant, the liability sits with you, not with your vendor.

The AI vendor is responsible for accuracy, transparency, and auditability—and should be held to account for all three. But the decisions made on the basis of that analysis, the actions taken or not taken, those sit with you. As they should.

That's entirely manageable, as long as you've built the process around it. A named owner. A documented QA approach. An honest conversation about accuracy tolerances and trade-offs. A model that's monitored, not just deployed.

When it comes to analysing free text feedback we've passed the point where properly-configured tech is better than humans by any measure: speed, accuracy, cost, a predilection for karaoke on a work night out and, worse still, questionable song choices.

Despite this, many companies remain much more comfortable from a governance perspective recognising that humans are flawed and dealing with the risks than understanding and accepting similar (lower!) risk from machines.

The teams that will benefit quickest are the ones facing and conquering this governance gap.

Until next time, keep learning.

Pete

*Me for one. It's not that I'm avoiding the subject—we're off to Tenerife for some emergency half term sun and drinking. It's been a loooooong, hard winter.