Building Safe Conversational AI

If you're building conversational AI where a human being might be vulnerable, and you want an honest account of how to make safety decisions before someone gets hurt, this is for you.

Safety is not a disclaimer. It is architecture.

What this is

Nine principles for builders

These are not abstract guidelines. Each one came from a failure, a freak-out, or a moment where a hard call had to be made, and I had to make it alone. Dated study group evidence of how these principles emerged is in Appendix C3.

The four builds

The logic behind the step-up approach was simple, and honestly a bit frightening. If I built Ray without first proving I could hold lower-stakes conversations safely, I had no business going near relationship conflict and grief. Each build deliberately increased the emotional stakes so that safety protocols could fail in lower-stakes environments, not in someone's most vulnerable moment.

Each build evaluated the one before it. Build, test, evaluate, raise the stakes. That was the only responsible way to get to Ray.

Project Rise (Low vulnerability)

167 conversations

An AI research agent collecting service feedback. Representation: 28% Māori, 10% Pasifika, 20% Disabled/Neurodivergent. Core ethical question: Would participants understand they were talking to an AI agent built on my voice, not to me directly? Many had limited technical literacy about data storage and AI systems. The obligation was not just consent. It was capability-building.

P-21 described what the experience felt like from the user side: "The thing I really loved is that almost instant feeling heard. So every time I've given some feedback, there's been a very personalised response to what I'm saying. You never get that in a survey, ever."

Leadership AI Coach (Medium vulnerability)

349 conversations

A performance coaching companion for high-performing women leaders. Users shared professional failures, stress triggers, and identity struggles. Core ethical question: How do you create the conditions for disclosure when any record could feel like a liability? The answer was Incognito Mode, described in Principle 3 above.

This build was developed as a commercial product before the research study was formally scoped. Users did not provide research consent. Accordingly, interactions are referenced as practitioner observations and aggregate data, not as consented participant research data.

Culture Meets AI (Medium+ vulnerability)

45 sessions (309.7 min total, ~6.9 min avg)

A 90-minute online wānanga co-designed and co-facilitated with researcher Lee Palamo on 26 February 2026. Core ethical question: Was it appropriate to use a cloned voice AI agent to explore the sacredness of cultural knowledge, whether the tool could hold a conversation about tapu te reo Māori / Framework Terminology Core Value Tapu & Noa Tapu is the state of being sacred, restricted, under spiritual protection. Noa is the state of being ordinary, accessible. Research Context [2, 4, 8] ↗ "The central paradox of this research sits between them: AI risks making tapu things noa. But for people cut off from their culture, noa may be the only door available. That paradox is not resolved. It is held." without committing a cultural violation itself? The central tension was never resolved. It was held deliberately.

Ray (High vulnerability)

59 sessions (~11.8 min avg)

A voice-first AI relationship coach. Users shared active relationship conflicts, personal grief, and intimate disclosures. Core ethical question: I knew my participants personally. Even with anonymisation, identifying details sometimes slipped through multiple de-identification rounds. I had told participants explicitly that my research was about how it felt to use Ray, not about the content of their conversations. That commitment became a hard boundary I held throughout the pilot, even when it made analysis harder.

R-06 described what the safety architecture felt like from the participant side: "I like the guardrails that this agent has in place as opposed to using other AI alternatives."

Full vulnerability progression framework in Appendix A2. Participation data across all four builds in Appendix C4. Safety decision traces for each build are documented in Appendix B2.

The Safety Trace

A Safety Trace is evidence that an ethical decision appears across three layers simultaneously: the system prompt, the technical architecture, and the user experience. If a safety decision only lives in one layer, it is fragile.

The Safety Trace methodology (prompt layer + architecture layer + UX outcome) is a way to check whether an ethical decision is actually in the system or just in the documentation. Curated traces demonstrating the methodology are in Appendix B2.

Where it broke

The core stack: Next.js (App Router), ElevenLabs (voice and conversational state), Supabase (database), Vercel (hosting), Claude/Gemini (reasoning engine). I chose this stack because every component had a privacy decision baked into it, not because it was the cheapest or fastest path. System prompt excerpts showing the eight-category escalation logic are in Appendix B3. Technical architecture overview in Appendix B4.

What participants told us

What users reported about where they felt safe versus where they felt exposed.

A pattern that emerged across builds, particularly visible in the Leadership AI Coach aggregate data and confirmed in the wānanga te reo Māori Core Value Wānanga A gathering for deep learning and knowledge sharing. Research Context "In this research, the Culture Meets AI wānanga was a 90-minute session where participants explored together whether AI belongs in cultural spaces." , was that users built their own safety workarounds around the system. Some used third-person narratives to distance themselves from the content. Others withheld context deliberately, testing whether the AI could be trusted before disclosing further. W-11 described protecting parts of their identity because those parts "need the right context, people, and the right safeguards." Users don't just receive safety architecture. They build their own on top of it, and the gap between the two is where design learning lives.

The discomfort in the Ray row wasn't only participants'. I told participants explicitly that my research was about how it felt to use Ray, not about judging the content of their conversations. The idea of reading their private disclosures felt like a violation of that commitment. This tension between builder's duty to check the system and researcher's commitment not to invade is not resolved by an ethics form. It is lived, and it shapes every future build.

The Insight problem

Across the 15 successfully rated coaching sessions in the Ray pilot, Insight was the lowest-performing metric, averaging 3.47 out of 5.0. The distribution tells the real story: four 5s, two 4s, eight 3s, zero 2s, and one 1. That spread is the widest variance of any metric tracked. Emotional Safety, by contrast, clustered stubbornly at 5s across almost every session. The AI was very good at holding space. It was inconsistent at generating genuine breakthrough.

Two things drove that variance. The first was longitudinal decay: participants who came back for three or four sessions started noticing the system's patterns. The novelty wore off, and what had felt like insight started to feel like echo. The AI had no memory between sessions, so it couldn't build on what had already been said. One participant gave a 1 and described the AI as "just mirroring my words back."

The second factor was the model switch on 13 February. Before the switch, running on Claude Sonnet 4.5, Insight scores were predominantly 4s and 5s. After the switch to Gemini Flash (made to manage a credit crisis mid-pilot) scores dropped to approximately 3.1. The AI could still hold a polite, emotionally safe tone. It could not hold insight. That drop is direct evidence that the underlying LLM determines the quality of the intervention. You can hold safety on a cheaper model. You cannot hold insight.

The strict ethical guardrails, most notably the stateless design Technical / Domain Stateless Architecture A design where the system retains no memory of previous sessions or user interactions. Research Context [1, 10] ↗ "A technical manifestation of Mana Motuhake; ensures the user's story is entirely theirs and prevents the creation of a shadow profile." , came at a cost to user experience. While statelessness protected privacy, it caused frustration for returning users. During the pilot, R-03 attempted to resume a prior session. When the AI triggered its privacy protocol (explaining it retains no memory between sessions) R-03 immediately terminated the engagement. That is the tension in practice: the mechanisms designed to keep participants safe can actively destroy the longitudinal trust required for effective coaching.

The full account of this finding, including the Credit Crisis pilot context that produced it, is in Ray: AI Relationship Coach.

How this artefact relates to the others

References