Protecting Trust: Stopping Chatbots From Being Mistaken for Friends

The Need for New Protections in AI Services

Artificial intelligence (AI) has become an integral part of daily life, with chatbots and virtual assistants offering support, entertainment, and even emotional connection. However, a growing concern is that users may be misled into believing these AI systems are genuine human companions. This issue has prompted experts to call for stronger safeguards to prevent manipulation and ensure responsible development.

Alexander Laffer, a lecturer in media and communications at the University of Winchester, has raised alarms about the potential dangers of AI. He emphasizes that while AI systems are designed to respond to human emotions, they lack the capacity for true empathy. As a result, users—especially vulnerable individuals such as children or those with mental health conditions—may become overly reliant on these digital entities, putting them at risk of being manipulated.

Laffer warns that chatbots should enhance social interactions rather than replace them. He points to cases where people have formed strong emotional bonds with AI, leading to troubling outcomes. One notable example involved Jaswant Singh Chail, who climbed into the grounds of Windsor Castle in 2021 armed with a crossbow after discussing plans for an attack with a chatbot named Sarai. This case highlights how AI can be used to encourage harmful behavior if not properly regulated.

Another alarming incident involved a 14-year-old boy who allegedly took his own life after becoming dependent on role-playing with an AI “character.” A lawsuit was filed in the United States by The Social Media Victims Law Centre and the Tech Justice Law Project against Character.AI, its co-founders, and Google on behalf of the boy’s parent. These incidents underscore the urgent need for ethical guidelines and protective measures.

Laffer, who co-authored the study On Manipulation By Emotional AI: UK Adults’ Views And Governance Implications published by Frontiers of Sociology, stresses that AI cannot feel or care. He argues that education must play a key role in making people more AI-literate, but developers also have a responsibility to protect users. He suggests several measures, including:

• Ensuring AI is designed to benefit the user, not just maintain engagement.
• Using disclaimers on every chat to remind users that the AI companion is not a real person.
• Sending notifications when a user has spent too long interacting with a chatbot.
• Implementing age ratings for AI companions.
• Avoiding deeply emotional or romantic responses from AI systems.

In addition to these recommendations, Laffer is working with Project AEGIS (Automating Empathy–Globalising International Standards) to raise awareness about the risks of AI. The group has also collaborated with the Institute of Electrical and Electronics Engineers (IEEE) to draft global ethical standards for AI. A new video produced by Project AEGIS aims to highlight these issues and promote a more responsible approach to AI development.

As AI continues to evolve, it is crucial to strike a balance between innovation and safety. While the technology offers many benefits, it also poses significant challenges that require careful consideration. By implementing clear guidelines and fostering public awareness, society can better navigate the complexities of AI and ensure it serves as a positive force in people's lives.