California advances bill to curb AI mental health tools

California lawmakers are advancing legislation that would bar artificial intelligence services from diagnosing or treating mental health conditions unless a licensed clinician oversees the care. The bills target consumer-facing chatbots and apps that rely primarily on automated systems to offer counseling, crisis intervention or clinical advice.

Proponents cite rapid consumer adoption of chatbot-based tools that use large language models and scripted conversational agents to offer emotional support, symptom screening or coping strategies. They point to reports of inaccurate or inappropriate responses, inconsistent handling of crises and gaps in privacy protections for sensitive health information.

The proposal would cover apps and online services that effectively act as diagnostic or therapeutic platforms while relying mainly on automation rather than credentialed providers. Sponsors say the aim is to reduce the risk that people seeking help for depression, anxiety, suicidal thoughts or other serious conditions receive unvetted algorithm-generated guidance. Supporters also raise concerns about how companies collect, store and share mental health data outside the federal Health Insurance Portability and Accountability Act framework.

Technology companies and some digital health startups oppose broad restrictions. They argue limits could reduce access to support for people who face cost barriers, provider shortages or stigma and could slow clinical research and product development intended to improve safety and effectiveness.

Developers have adopted varied approaches: some offer self-guided apps with scripted chat and peer-style conversations, while others integrate licensed therapists into their platforms. The bills would distinguish general wellness or educational content from tools that provide diagnosis or treatment. Legislators, clinicians and industry representatives are negotiating how to define that distinction in statute.

Privacy and data security are central to the debate. Many consumer mental health apps are not covered by HIPAA, leaving users subject to app terms of service and state privacy rules. California’s consumer privacy law provides residents certain rights over personal data; lawmakers drafting the AI restrictions say they will coordinate mental health provisions with state privacy protections to limit unauthorized sharing of sensitive information.

Regulatory jurisdiction is contested. The U.S. Food and Drug Administration plans to focus on higher-risk medical devices, including some software tools, while federal and state agencies weigh oversight for fast-evolving AI products. California’s standards would create state-level requirements that other states could reference.

Companies continue clinical trials and peer-reviewed studies to validate AI-driven mental health interventions. Lawmakers who back the bills say clinical validation should be required for services that deliver treatment, while developers seek regulatory pathways that allow iterative improvement and controlled research.

If the legislation advances, it will need clear definitions and enforcement mechanisms to separate wellness chatbots from systems that provide clinical care. The bills are scheduled to move through committee hearings where testimony is expected from mental health professionals, patient advocates, privacy experts and company representatives. Amendments and carve-outs may be added as stakeholders negotiate ways to balance safety, access and innovation.

Consumer interest in digital mental health tools has grown amid limited access to in-person care and long wait times. Available products range from scripted chat tools that offer coping exercises to telehealth platforms that connect users with licensed therapists.