Chapter 9 · Designing for Human Connection: Trust and Customer-Facing AI

The best customer-facing agent is not the most capable one. It is the one that earns and maintains the appropriate level of trust.

---

The Trust Problem Is a Design Problem

When a customer-facing AI agent fails, the failure is usually attributed to the model: it hallucinated, it misunderstood, it gave bad advice. What is less often examined is whether the design of the system created conditions in which that failure was likely, and whether the failure could have been caught or mitigated before it reached the customer.

Trust in customer-facing AI is not an emergent property of capability. A highly capable agent that operates in ways users cannot understand, predict, or override will be trusted less than a less capable agent that behaves transparently and consistently. Trust is a design outcome, and like all design outcomes, it can be planned for or left to chance.

This chapter is primarily about the former.

---

The Trust Calibration Problem

The central challenge of customer-facing AI is not building a trustworthy system — it is building a system whose trustworthiness is accurately perceived by the people using it. Both failure directions are costly.

Under-trust causes users to work around or ignore the agent entirely — checking every output, adding friction rather than removing it. This is the failure mode of agents that are deployed without adequate communication about what they can reliably do.

Over-trust is more dangerous. Users who trust the agent beyond its reliable capability stop applying judgment. Errors that a watchful human would catch propagate downstream — into customer communications, financial decisions, medical information, or legal records.

Calibrated trust — where users rely on the agent in proportion to its actual reliability — is the target. Getting there requires deliberate design choices, not just a capable model. Research on trust in automation establishes that calibration — where a user's trust matches an agent's actual capability — is the design target, and that both overtrust and distrust produce systematic, predictable failures in reliance.³

Key takeaway: The goal is not maximum trust but accurate trust — users relying on the agent precisely where it is reliable and exercising judgment where it is not.

---

Transparency Mechanisms That Actually Work

Transparency is frequently cited as the solution to trust calibration. It is necessary but not sufficient. The challenge is that most transparency mechanisms are ignored in practice — users do not read disclaimers, do not click through to confidence scores, and do not engage with explanations that interrupt their workflow. Empirical evaluation across 20 popular AI-infused products found that the guideline to explain why a system behaved as it did — and to communicate how reliably it performs — ranked among the most violated of all human-AI interaction design guidelines, with users able to identify clear opportunities for transparency that products consistently failed to provide.²

Effective transparency mechanisms share three properties: they are contextual (appearing at the moment of relevance, not buried in settings), proportional (signalling uncertainty when it is high, not on every interaction), and actionable (giving the user something to do with the information).

Mechanism	Effective When	Ineffective When
Confidence indicators	Integrated into the response, proportional to actual uncertainty	Displayed uniformly on all outputs regardless of actual confidence
Source attribution	The source is meaningful to the user and verifiable	Sources are generic or inaccessible
Scope disclosure	Presented at task initiation, not buried in terms of service	Disclosed in documentation nobody reads
Escalation pathways	One-click, low-friction, clearly labelled	Requiring a separate workflow to reach a human
Uncertainty hedging	Natural language signals ("I'm not certain, but...")	Excessive hedging that undermines confidence in reliable outputs

---

Conversational Design Principles

The way an agent communicates shapes the trust relationship as much as what it communicates. Several principles have emerged from deployment experience that are specific to customer-facing agents.

Set scope expectations early. The most effective agents open interactions with a clear implicit or explicit signal about what they can help with. Users who understand the agent's domain are less likely to ask questions it cannot reliably answer, and less likely to over-rely on answers to questions outside its reliable range.

Be explicit about limitations, not apologetic about them. An agent that says "I don't have access to your account history — you'll need to speak with the account team for that" is more trustworthy than one that attempts an answer based on incomplete information, or one that responds with generic apologetic language. Precise limitation acknowledgment signals competence, not weakness.

Match formality to context. Customer-facing agents that communicate with inappropriate formality or informality signal misalignment — the agent does not understand its context. This is a trust signal even when the content is accurate.

Avoid sycophancy. Models trained on human feedback have a tendency to affirm what users say before correcting them, or to avoid delivering unwelcome assessments. In a customer context, this can mean validating an incorrect understanding before providing the right answer — which confuses more than it helps. Design prompts and evaluation criteria that specifically penalise this pattern.

---

Failure and Recovery Design

How an agent handles failure is as important as how it handles success. The trust damage from a poorly handled failure often exceeds the trust damage from the failure itself.

Three principles apply to failure recovery design:

1. Name the failure accurately. "I don't know" is more trustworthy than a hedged guess. "I made an error" is more trustworthy than a vague correction. Specificity in failure acknowledgment signals that the agent has genuine self-awareness of its limitations. Trust in automation also tends to be disproportionately shaped by failures rather than successes, and recovery after a fault is slower than erosion — making the quality of failure handling a more powerful determinant of long-term trust than routine performance.³

2. Offer a path forward. A failure acknowledgment without an alternative path leaves the user stuck. Good failure design always includes a next step: escalation to a human, a suggested reframe of the question, or a pointer to another resource.

3. Do not over-apologise. Excessive apology is as much a trust signal as no apology. It suggests the agent is unreliable by default, which raises the question of why it is deployed at all.

---

The Regulatory Context

Customer-facing AI is the deployment category attracting the most regulatory attention globally. Practitioners need to be aware of three active regulatory frameworks.

EU AI Act (2024) classifies AI systems by risk level. Systems that interact with customers in ways that could affect their rights, health, or financial wellbeing are subject to transparency obligations: users must know they are interacting with an AI, and systems must be designed with human oversight mechanisms. High-risk systems (healthcare, financial advice, legal guidance) face additional conformity assessment requirements. The Act further introduces a layered compliance structure for customer-facing agents built on general-purpose AI models: the foundation model provider carries obligations around technical documentation and training data transparency, while the deployer retains responsibility for ensuring appropriate human oversight and correct risk classification in their specific deployment context.¹

FTC Guidelines on AI (United States) apply existing consumer protection law to AI deployments, focusing on deceptive practices — particularly AI systems that misrepresent their nature or capabilities to consumers.

GDPR and data protection law applies to any customer-facing agent that processes personal data of EU residents. Agents that personalise responses based on user history, store conversation data, or profile users' preferences trigger data protection obligations that must be designed for from the outset, not retrofitted.

The practical implication for design is that regulatory compliance and good trust design are largely the same things: clear disclosure, meaningful human oversight, accurate scope representation, and data minimisation. Meeting the regulatory bar is a consequence of meeting the trust design bar. Bank of America's Erica — deployed in 2018 and handling more than 3 billion client interactions across 20 million active consumer users, while operating under the strict data and disclosure requirements of US financial services regulation — demonstrates this alignment in practice: the same design disciplines that earned sustained user trust at scale are precisely the disciplines that satisfy the regulatory bar.⁴

---

References

1. Future of Life Institute (2024). High-level summary of the AI Act. Available at: artificialintelligenceact.eu/high-level-summary
2. Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P.N., Inkpen, K., Teevan, J., Kikin-Gil, R., & Horvitz, E. (2019). Guidelines for Human-AI Interaction. In Proceedings of CHI 2019, Glasgow, Scotland. ACM. https://doi.org/10.1145/3290605.3300233
3. Lee, J.D. & See, K.A. (2004). Trust in Automation: Designing for Appropriate Reliance. Human Factors, 46(1), 50–80.
4. Bank of America (2025). AI Adoption by BofA's Global Workforce Improves Productivity, Client Service. Bank of America Newsroom. April 8, 2025. https://newsroom.bankofamerica.com/content/newsroom/press-releases/2025/04/ai-adoption-by-bofa-s-global-workforce-improves-productivity--cl.html