Chapter 9 · Designing for Human Connection: Trust and Customer-Facing AI
The best customer-facing agent is not the most capable one. It is the one that earns and maintains the appropriate level of trust.
The Trust Problem Is a Design Problem
When a customer-facing AI agent fails, the failure is usually attributed to the model: it hallucinated, it misunderstood, it gave bad advice. What is less often examined is whether the design of the system created conditions in which that failure was likely, and whether the failure could have been caught or mitigated before it reached the customer.
Trust in customer-facing AI is not an emergent property of capability. A highly capable agent that operates in ways users cannot understand, predict, or override will be trusted less than a less capable agent that behaves transparently and consistently. Trust is a design outcome, and like all design outcomes, it can be planned for or left to chance.
This chapter is primarily about the former.
The Trust Calibration Problem
The central challenge of customer-facing AI is not building a trustworthy system — it is building a system whose trustworthiness is accurately perceived by the people using it. Both failure directions are costly.
Under-trust causes users to work around or ignore the agent entirely — checking every output, adding friction rather than removing it. This is the failure mode of agents that are deployed without adequate communication about what they can reliably do.
Over-trust is more dangerous. Users who trust the agent beyond its reliable capability stop applying judgment. Errors that a watchful human would catch propagate downstream — into customer communications, financial decisions, medical information, or legal records.
Calibrated trust — where users rely on the agent in proportion to its actual reliability — is the target. Getting there requires deliberate design choices, not just a capable model. Research on trust in automation establishes that calibration — where a user's trust matches an agent's actual capability — is the design target, and that both overtrust and distrust produce systematic, predictable failures in reliance.3
Key takeaway: The goal is not maximum trust but accurate trust — users relying on the agent precisely where it is reliable and exercising judgment where it is not.
Transparency Mechanisms That Actually Work
Transparency is frequently cited as the solution to trust calibration. It is necessary but not sufficient. The challenge is that most transparency mechanisms are ignored in practice — users do not read disclaimers, do not click through to confidence scores, and do not engage with explanations that interrupt their workflow. Empirical evaluation across 20 popular AI-infused products found that the guideline to explain why a system behaved as it did — and to communicate how reliably it performs — ranked among the most violated of all human-AI interaction design guidelines, with users able to identify clear opportunities for transparency that products consistently failed to provide.2
Effective transparency mechanisms share three properties: they are contextual (appearing at the moment of relevance, not buried in settings), proportional (signalling uncertainty when it is high, not on every interaction), and actionable (giving the user something to do with the information).
| Mechanism | Effective When | Ineffective When |
|---|---|---|
| Confidence indicators | Integrated into the response, proportional to actual uncertainty | Displayed uniformly on all outputs regardless of actual confidence |
| Source attribution | The source is meaningful to the user and verifiable | Sources are generic or inaccessible |
| Scope disclosure | Presented at task initiation, not buried in terms of service | Disclosed in documentation nobody reads |
| Escalation pathways | One-click, low-friction, clearly labelled | Requiring a separate workflow to reach a human |
| Uncertainty hedging | Natural language signals ("I'm not certain, but...") | Excessive hedging that undermines confidence in reliable outputs |
Conversational Design Principles
The way an agent communicates shapes the trust relationship as much as what it communicates. Several principles have emerged from deployment experience that are specific to customer-facing agents.
Set scope expectations early. The most effective agents open interactions with a clear implicit or explicit signal about what they can help with. Users who understand the agent's domain are less likely to ask questions it cannot reliably answer, and less likely to over-rely on answers to questions outside its reliable range.
Be explicit about limitations, not apologetic about them. An agent that says "I don't have access to your account history — you'll need to speak with the account team for that" is more trustworthy than one that attempts an answer based on incomplete information, or one that responds with generic apologetic language. Precise limitation acknowledgment signals competence, not weakness.
Match formality to context. Customer-facing agents that communicate with inappropriate formality or informality signal misalignment — the agent does not understand its context. This is a trust signal even when the content is accurate.
Avoid sycophancy. Models trained on human feedback have a tendency to affirm what users say before correcting them, or to avoid delivering unwelcome assessments. In a customer context, this can mean validating an incorrect understanding before providing the right answer — which confuses more than it helps. Design prompts and evaluation criteria that specifically penalise this pattern.
Failure and Recovery Design
How an agent handles failure is as important as how it handles success. The trust damage from a poorly handled failure often exceeds the trust damage from the failure itself.
Three principles apply to failure recovery design:
-
Name the failure accurately. "I don't know" is more trustworthy than a hedged guess. "I made an error" is more trustworthy than a vague correction. Specificity in failure acknowledgment signals that the agent has genuine self-awareness of its limitations. Trust in automation also tends to be disproportionately shaped by failures rather than successes, and recovery after a fault is slower than erosion — making the quality of failure handling a more powerful determinant of long-term trust than routine performance.3
-
Offer a path forward. A failure acknowledgment without an alternative path leaves the user stuck. Good failure design always includes a next step: escalation to a human, a suggested reframe of the question, or a pointer to another resource.
-
Do not over-apologise. Excessive apology is as much a trust signal as no apology. It suggests the agent is unreliable by default, which raises the question of why it is deployed at all.
Handoff, Redress, and Human Continuity
The most important design choice in a customer-facing agent is often not what the agent can answer, but how gracefully it exits. A customer who reaches the boundary of the agent's competence should not have to restart the interaction, repeat the problem, or discover that the human support team cannot see what already happened. Poor handoff design converts a contained AI limitation into a broader service failure.
A strong handoff has four properties:
| Property | What It Requires | Why It Matters |
|---|---|---|
| Continuity | The human receives the conversation history, attempted actions, relevant account context, and unresolved question | Prevents the customer from repeating themselves |
| Authority | The human has the permission and information needed to resolve the issue, not merely apologise for the bot | Avoids escalation loops |
| Redress | The customer can challenge, correct, or reverse an AI-mediated decision | Protects trust when the agent affects rights, money, access, or service |
| Feedback capture | The handoff reason is logged as training and evaluation data | Turns failure boundaries into product intelligence |
This matters especially when the agent is not only answering questions but influencing outcomes: refunds, eligibility, account restrictions, appointment scheduling, claim handling, or financial decisions. In those cases, handoff is not a courtesy feature. It is part of the accountability mechanism.
Key takeaway: A customer-facing agent should never be a dead end. The boundary of automation must be the beginning of a better-supported human interaction, not the point where the customer loses context and control.
The Regulatory Context
Customer-facing AI is the deployment category attracting the most regulatory attention globally. Practitioners need to be aware of three active regulatory frameworks.
EU AI Act (2024) classifies AI systems by risk level. Systems that interact with customers in ways that could affect their rights, health, or financial wellbeing are subject to transparency obligations: users must know they are interacting with an AI, and systems must be designed with human oversight mechanisms. High-risk systems (healthcare, financial advice, legal guidance) face additional conformity assessment requirements. The Act further introduces a layered compliance structure for customer-facing agents built on general-purpose AI models: the foundation model provider carries obligations around technical documentation and training data transparency, while the deployer retains responsibility for ensuring appropriate human oversight and correct risk classification in their specific deployment context.1,5
FTC guidance and enforcement on AI (United States) apply existing consumer protection law to AI deployments, focusing on deceptive practices — particularly AI systems that misrepresent their nature or capabilities to consumers. The FTC's 2024 Operation AI Comply enforcement sweep made the principle explicit: there is no separate exemption for AI systems when companies make deceptive claims or use AI-enabled tools to mislead consumers.6
GDPR and data protection law applies to any customer-facing agent that processes personal data of EU residents. Agents that personalise responses based on user history, store conversation data, or profile users' preferences trigger data protection obligations — including lawful basis, transparency, purpose limitation, data minimisation, access rights, and safeguards around automated decision-making — that must be designed for from the outset, not retrofitted.7
The practical implication for design is that regulatory compliance and good trust design are largely the same things: clear disclosure, meaningful human oversight, accurate scope representation, and data minimisation. Meeting the regulatory bar is a consequence of meeting the trust design bar. Bank of America's Erica — deployed in 2018 and handling more than 3 billion client interactions across 20 million active consumer users, while operating under the strict data and disclosure requirements of US financial services regulation — demonstrates this alignment in practice: the same design disciplines that earned sustained user trust at scale are precisely the disciplines that satisfy the regulatory bar.4
Customer-facing agents expose the human side of agentic systems first: trust, expectation-setting, failure recovery, and escalation. The next chapter turns from the customer interface to the systems underneath it. Once an organisation promises an agent can help, the agent has to reach into the messy reality of legacy infrastructure — and that is where many elegant user experiences either become real or collapse.
References
- Future of Life Institute (2024). High-level summary of the AI Act. Available at: artificialintelligenceact.eu/high-level-summary
- Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P.N., Inkpen, K., Teevan, J., Kikin-Gil, R., & Horvitz, E. (2019). Guidelines for Human-AI Interaction. In Proceedings of CHI 2019, Glasgow, Scotland. ACM. https://doi.org/10.1145/3290605.3300233
- Lee, J.D. & See, K.A. (2004). Trust in Automation: Designing for Appropriate Reliance. Human Factors, 46(1), 50–80.
- Bank of America (2025). AI Adoption by BofA's Global Workforce Improves Productivity, Client Service. Bank of America Newsroom. April 8, 2025. https://newsroom.bankofamerica.com/content/newsroom/press-releases/2025/04/ai-adoption-by-bofa-s-global-workforce-improves-productivity--cl.html
- European Parliament and Council (2024). Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). Official Journal of the European Union, L 2024/1689. https://eur-lex.europa.eu/eli/reg/2024/1689/oj
- Federal Trade Commission (2024). FTC Announces Crackdown on Deceptive AI Claims and Schemes. September 25, 2024. https://www.ftc.gov/news-events/news/press-releases/2024/09/ftc-announces-crackdown-deceptive-ai-claims-schemes
- European Parliament and Council (2016). Regulation (EU) 2016/679: General Data Protection Regulation (GDPR). Official Journal of the European Union, L 119, 1-88.
Building agentic AI and wondering why alignment is harder than the technology? Get in touch