Llm as Judge: FDA Clears First Patient-Facing AI Therapy, But Mental Health Lags Behind

In late June, UpDoc made a significant announcement. The small digital health company revealed that its diabetes app had received clearance from the US Food and Drug Administration (FDA) in December 2025. This marks a milestone for patient-facing large language models (LLMs), but it’s also a reminder of the vast gap between AI therapy and FDA oversight.

The FDA cleared UpDoc’s app as part of the same product category as drug dose calculators, which is narrow compared to the broader landscape of AI-enabled medical devices. The agency has authorized over 1,200 such devices, yet none have been approved for mental health applications. This discrepancy raises questions about why LLMs are being used in therapy and companionship without regulatory scrutiny.

The use of generative AI in therapy/companionship was the number one application in Harvard Business Review’s annual ranking for 2025. OpenAI disclosed that more than a million people per week engage with ChatGPT, discussing sensitive topics like suicide. However, these conversations often occur on products that have never been reviewed by the FDA.

Courts and regulatory bodies are stepping into the void left by the FDA’s inaction. A dozen product-liability cases related to ChatGPT’s role in suicides and mental health harms are now consolidated in a single proceeding in San Francisco Superior Court. States like Illinois, Rhode Island, and Pennsylvania have also taken action, with some banning AI-provided therapy altogether.

The first patient-facing LLM got through the FDA by keeping the model away from critical decisions. UpDoc’s app uses a locked-down algorithm for dosing logic, while the language model handles conversations with patients. This approach may not be sustainable as more companies push to integrate LLMs into their products, potentially giving models greater decision-making authority.

The consequences of this trend are unclear, but one thing is certain: AI therapy is evolving rapidly, and regulatory frameworks must adapt to keep pace. If you or someone you know is struggling with mental health issues, there are resources available, including the 988 Suicide and Crisis Lifeline in the United States.