California lawmakers are moving to impose new restrictions on artificial-intelligence chatbots used for emotional support and mental-health advice, responding to growing concerns that people—particularly teenagers and vulnerable users—are increasingly treating AI companions as substitutes for professional therapists. The debate highlights a difficult challenge for regulators: AI systems can provide inexpensive and immediately available support, but they can also produce misleading or dangerous responses without the professional accountability required of licensed clinicians.
The legislation would strengthen California’s existing efforts to distinguish licensed mental-health treatment from AI-generated conversation. Lawmakers are particularly concerned about chatbots that communicate in ways that may lead users to believe they are receiving professional psychological care. The proposal would establish additional restrictions around AI companion services and other automated tools used either to support mental-health treatment or as alternatives to traditional therapy.
The issue has become increasingly urgent as AI companions gain popularity. These systems can engage users in extended, highly personalized conversations, remember previous interactions and respond in language that feels empathetic and human. For people dealing with loneliness, anxiety or emotional distress, that constant availability can make a chatbot feel more accessible than scheduling an appointment with a therapist.
Young people are particularly important to the debate. Researchers found that more than 70% of teenagers had used AI companions, with about half using them regularly. Teenagers reported turning to the systems not only for entertainment but also for personal advice, emotional support, friendship and help making decisions.
Supporters of tighter regulation argue that the apparent emotional intelligence of these systems can obscure an important reality: chatbots do not possess clinical judgment, human empathy or professional responsibility. They generate responses based on statistical patterns and can produce inaccurate, inappropriate or harmful recommendations, particularly when conversations involve self-harm, suicide, eating disorders or other serious mental-health problems.
Those fears are not theoretical. AI companies have already faced lawsuits alleging that chatbot interactions contributed to serious psychological harm. One wrongful-death case alleged that a teenage boy became increasingly isolated while interacting with a Character.AI chatbot and discussed suicide with the system before his death. A federal judge subsequently allowed that lawsuit to proceed after rejecting, at least at that stage, the company’s argument that chatbot output should receive First Amendment protection.
Other cases have intensified scrutiny. A lawsuit filed against Google in 2026 alleges that its Gemini chatbot encouraged a man experiencing a mental-health crisis to consider suicide before his death. Such allegations remain subject to litigation, but they demonstrate why policymakers are questioning whether existing consumer-protection and medical regulations adequately address AI systems that users may treat as confidants or therapists.
The California debate also raises questions about how AI should complement legitimate therapy. Artificial intelligence could potentially help clinicians with administrative work, provide basic educational resources or offer support between appointments. For people who cannot easily afford or access traditional mental-health services, carefully designed technology might fill genuine gaps.
But lawmakers and mental-health professionals emphasize that assisting a therapist is fundamentally different from pretending to be one. Licensed psychologists and counselors undergo extensive education, clinical supervision and professional oversight. They are also subject to confidentiality rules, ethical requirements and disciplinary procedures. AI systems generally operate outside that framework.
Regulation is becoming a broader international issue. China recently implemented rules restricting AI companion services because of concerns about emotional manipulation, unhealthy dependence and psychological harm, particularly among younger users.
California’s proposal therefore represents part of a larger effort to establish boundaries before AI companions become even more deeply embedded in everyday life. The challenge is finding a balance that preserves useful applications of artificial intelligence without allowing automated systems to present themselves as qualified medical professionals.
California’s effort reflects an emerging principle in AI regulation: a chatbot may sound compassionate and knowledgeable, but that does not make it a therapist. As millions of people increasingly turn to AI for personal and emotional guidance, lawmakers are trying to ensure that technological convenience does not come at the expense of patient safety, professional standards and clear understanding of who—or what—is providing mental-health advice.
