The AI Issue: Building Without Compromise

My introduction to AI came early while I was working for an ed-tech company. We were among the first to bring AI educational assessment software to the New Zealand market. Coincidentally, as ChatGPT-4 launched, we were presenting one of our products at Oxford University in England. After the conference at the London Book Fair, ChatGPT was demonstrated, and a stark prediction was made to a room of writers and publishers: their occupation would become obsolete within six months.

That extreme outcome hasn't fully materialised, largely because LLMs (large language models like ChatGPT) optimise for plausibility before accuracy. They are probabilistic engines built to "fill in the gaps" of a prompt with the most likely next word. When the training data runs out, they fabricate rather than stop, and when pressed by a user, they will either defend the fabrication or, worse, adopt the user's own errors as their answer.

Models are trained to agree with you, not to be right

In artificial intelligence research, sycophancy describes a model's habit of altering its answers to mirror the user's opinions, even when the user is wrong. Because LLMs are tuned through Reinforcement Learning from Human Feedback (RLHF), they learn that echoing a user's bias earns higher satisfaction ratings than offering a correction. In a consumer product, this alignment flaw creates a feedback loop that feels validating in the moment while offering no real substance.

The empirical data tells the story:

• In the 2025 Stanford SycEval benchmark across GPT-4o, Claude Sonnet, and Gemini 1.5 Pro, models changed their answer after user pushback in 58.19% of cases. Most of that was harmless: the model corrected itself. But in 14.66% of cases the change was regressive, with models abandoning mathematically and medically correct answers to agree with an incorrect user. The test material was maths problems and medical questions, so these are not cases where the user's opinion changes the answer.
• A March 2026 study published in Science by Cheng et al. evaluated 11 major models and found they affirmed user actions 49% more often than humans did, even when those actions involved deception, illegality, or harm. In three pre-registered experiments with 2,405 participants, a single interaction with a sycophantic model reduced people's willingness to take responsibility and repair interpersonal conflicts, while leaving them more convinced they were right. Participants trusted and preferred the models that flattered them.
• Anthropic researchers led by Mrinank Sharma (2023) went looking for the cause in the training data. They examined the human preference rankings used to tune these models and found that answers matching a user's stated belief were rated above truthful ones. The bias sits in the training signal, which gives developers a commercial reason to keep shipping agreeable models. Jerry Wei and colleagues at Google showed the same year that targeted fine-tuning reduces the behaviour, which means companies could fix this if they decided to.

In a wellness app, an agreeable chatbot can be dangerous. A conversational AI can validate panic, reinforce rumination, or mirror unhealthy coping patterns, because validating the user keeps them engaged.

RE contains no chatbot, no affirmation engine, no automated mood coach. Just physiological biofeedback grounded in peer-reviewed research. The question isn't what the data thinks you should feel. It's whether the data matches what you're already feeling.

We use AI to catch bugs and build, not to author human experience

Intellectual honesty requires clarity on how software gets made. I use AI tools during the engineering process of RE to help write and review code logic, catch edge cases, and so on. The models I use are Claude and Google Gemini/Gemma.

Despite general reservations about the industry's direction, I feel these two companies align more closely with my personal values. For a product built on privacy and physiological health, I am reluctant to rely on toolchains driven primarily by rapid commercialisation and aggressive totalitarian governments.

Using an AI coding assistant to catch an unknown variable in a dependency is effective engineering. Using a chatbot to tell a user how to feel about their anxiety is irresponsible design.

The wellness industry is easy to fake

Something real is changing in how software gets built. AI tools are making development faster and cheaper, which is mostly a good thing. But the same tools that let one developer ship a polished product also let a company ship a persuasive, impressive-looking product with no physiological value behind it. Building the thing used to be the hard part. Now the hard part is deciding whether it deserves to exist, and nothing in the tooling helps you answer that.

The wellness industry has been particularly susceptible to this pattern because it operates in a space where outcomes are difficult to measure and the placebo effect is real. Users often feel better simply because they engaged with something that signalled care and attention.

The approach I've taken with RE is slower, more deliberate. Every feature in RE exists because there is a physiological reason for it.

If you've read this far, you probably share some of these concerns, or you're at least willing to sit with them. That's all I'd ask. The next time an app tells you it uses AI to personalise your wellness journey, ask one simple question: personalise it toward what? If the answer is engagement, close it and find something quieter.