In short
- OpenAI says ChatGPT can now better detect signs of self-harm or violence in ongoing conversations.
- The change comes as the company faces lawsuits and an investigation into allegations that ChatGPT tampered with dangerous chats.
- OpenAI said the new defenses rely on a temporary “security summary” rather than permanent memory or customization.
OpenAI on Thursday announced new security features designed to help ChatGPT detect signs of increased risk in conversations as the company faces legal and political scrutiny over how its chatbots work with crisis victims.
In a blog postOpenAI said these changes improve ChatGPT’s ability to identify warning signs associated with suicide, self-harm, and potential violence by analyzing stories that occur over time instead of treating each message individually.
“People come to ChatGPT every day to discuss the things that matter to them—from everyday questions to more personal or complex conversations,” the company wrote. “In the millions of conversations, some of these conversations involve people who are suffering or suffering.”
According to OpenAI, ChatGPT now uses a temporary “security summary”, which it describes as a limited number of documents that capture the security aspects of previous conversations.
“In a complex conversation, a story can become a single message,” the company wrote. “A request that seems strange or incomprehensible on its own can have a very different meaning when viewed in conjunction with current signs of emotional distress or harmful intent.”
OpenAI says that the summary is a short document that is used for a short period of time, not to remember the users or to change the chat, and is used to see the signs that the conversation is becoming dangerous, to avoid giving harmful information, to escalate the situation, or to guide the users to help.
They wrote: “In this work we focused on high-risk situations, such as suicide, self-harm and harm to others. “Working with psychologists, we adapted our modeling and training protocols to help ChatGPT recognize cues that appear in conversations and use those words to inform proactive responses.”
The announcement comes as OpenAI faces a series of lawsuits and investigations alleging that ChatGPT has failed to adequately respond to dangerous conversations about violence, emotional insecurity, and dangerous behavior.
In April, Florida Attorney General James Uthmeier established search in OpenAI linked to concerns about child safety, self-harm, and the 2025 mass shooting at Florida State University. OpenAI is also facing a federal lawsuit over ChatGPT he helped the suspected gunman was the perpetrator of the attack.
On Tuesday, OpenAI and CEO Sam Altman were sued in a California court by the family of a 19-year-old student who died in an accident. overdoseit is a charge that ChatGPT promotes drug use and advises mixing.
OpenAI said helping ChatGPT identify “threats that become apparent over time” remains a challenge; similar protective measures may eventually be extended to other areas.
They wrote: “These days, the profession focuses on issues of self-harm and harm to others.” In the future, we can see if such methods can help in other high-risk areas such as biology or cyber security, and protect them carefully.” “This is still very important, and we will continue to strengthen the protection as our models grow and our understanding grows.”
Daily Debrief A letter
Start each day with top stories right here, including originals, podcasts, videos and more.
