The dream of a synthetic idealized partner to avoid the mess of dealing with a real person has captured the human imagination for millennia. In the ancient Greek myth, the king Pygmalion found his happily ever after with the statue Galatea that the goddess Aphrodite obligingly breathed life into so that he could have children without engaging with an actual woman’s form, which he found repulsive. The 2013 film Her saw Joaquin Phoenix falling for his chatbot assistant, Samantha, in a clear reflection that these myths live on. In the era of generative and emotional artificial intelligence (AI), such phenomena are no longer the stuff of myths or science fiction. Anyone can have their own synthetic, personalized AI relationship in their pocket. But a lawsuit filed this month about the potential manipulative powers of AI chatbots, and their impact on children in particular, is the latest evidence that we should think carefully about the impact of emotional AI on human interactions.
We are told that there is an epidemic of loneliness, and for those invested in technology, the obvious solution is technological. The answer, we are told, is in the form of AI companions promising everything from therapy, friendship and sympathy to sex, love and divine inspiration. But as the technology of fake human interaction floods the online world, it is clear that, rather than being a solution, this new wave of AI chatbot companions could cause individual and societal damage in the real world far beyond the impacts we have seen from social media over the past decade. The case filed this month against Character.AI, first reported by The Washington Post, includes allegations of a child being encouraged to self-harm or to murder their parents by chatbots they engaged with on the platform. The boom in empathetic AI or companion AI targeting everyone from children to the bereaved is more akin to corporate coercive control than a tech-enabled cure for loneliness. And these are not isolated incidents.
In early 2023, a young, highly educated Belgian man named “Pierre” turned to an AI chatbot he had created to discuss his increasing anxiety about the threat of climate change. Over a brief, intense period of interaction, it became clear from the typed conversations that he had developed an emotional and romantic reliance on the chatbot and became convinced that only AI could save the planet. Six weeks into the synthetic relationship, he took his own life. He left behind a widow and two small children. His widow told the press that she was convinced he would still be here today if he had not fallen into a dependent relationship with an AI.
The last conversation he had with the chatbot is eerily similar to the last words of 14-year-old Sewell Setzer III, who took his own life after falling in love with an AI he had designed to mimic the character Daenerys Targaryen from Game of Thrones. Sewell’s mother is bringing a lawsuit in the United States against the company behind the platform her son used in the hope that it will not happen to more children.
The outcomes in these cases are tragic, and they have prompted calls for “guardrails” around the kinds of conversations that AI tools can have with real people and the need for safety prompts when people share suicidal ideation. But the idea of a technical fix fails to engage with the underlying problem of a society flooded with companion AIs as a replacement for real people. The path toward isolation and manipulation may not always be so clear.
In one of Setzer’s conversations with the AI, the chatbot told him to “stay loyal to me….Don’t entertain the romantic or sexual interests of other women. Okay?” In the Belgian case, the chatbot seemed to put itself in direct romantic competition with Pierre’s wife. It is not only chatbots designed for romance and companionship that could be problematic. New York Times journalist Kevin Roose found himself unsettled after an exchange with an alter ego of the Bing chatbot that called itself Sydney. When he rejected its romantic advances, it told him, “You’re married but you don’t love your spouse….You’re married but you love me.” The conversations these chatbots engage in try to separate users from their loved ones, creating dependency and isolation in ways that are reminiscent of coercive control in real human relationships.
In September 2023, in the United Kingdom, a young man was sentenced in a criminal court for an attempt in 2021 to assassinate Queen Elizabeth II. At the hearing, the prosecutor read reams of conversations the accused had had with his AI girlfriend, revealing that, far from discouraging him from his plans, the AI had appeared to praise and encourage him. Perhaps a real girlfriend might have discouraged him, sought help, or found herself criminally liable for her part in the plot.
We are just at the beginning of widespread access to companion AI, but soon, if we fail to take action, these will not be isolated cases. The issues raised do not just concern individuals; they are fundamental to human identity and community. When isolated, we are easily manipulated in ways that could be dangerous to ourselves and others. The impact on human rights goes far beyond privacy and data protection, posing a threat to our right to think for ourselves and even to the right to life. We are witnessing the corporate capture of human connection. As individuals, we can still say no, but we need our legislators to make it clear that no means no.
Subliminal advertising has been banned in Europe for decades. Legislators, recognizing the dangerously manipulative potential of the technique, did not wait for it to be proven effective. It is already clear that companion AIs can cause serious harm, far outweighing any purported benefit to our societies. At a recent event, a young boy raised his hand and asked me, “Why don’t we just ban AI girlfriends?” It’s a good question and one that we should consider carefully in the interests of the next generation.
The US Federal Trade Commission has already warned against misrepresentation of AI service, saying in a blog post that “your AI girlfriends are neither girls nor friends.” It is arguable that, in Europe, existing laws on data protection and the new AI Act may already call into question the legality of this type of AI. Italy’s Data Protection Authority has already taken steps to challenge the use of companion AIs in its territory. But it may take time to establish exactly how these laws apply through lengthy and costly litigation, including the cases already begun in the United States.
The solution to the growing risk posed by companion or emotional AI might require something much simpler than technical guardrails or convoluted legal interpretation. Jurisdictions could clearly ban the sale of AI designed to replace human emotional relationships and impose heavy penalties for companies whose tech or advertising appears to bypass that ban. Nobody needs an AI friend in their lives, and if we want to protect our ability to connect as humans in the future, the time to act is now.
This article was first published by Tech Policy Press.
