The Center for Countering Digital Hate (CCDH) has issued a warning that ChatGPT can be manipulated into offering unsafe guidance to teenagers on drug use, self-harm, and extreme dieting. The watchdog argues that the AI system requires stronger, built-in protections to prevent harm.
In the study, CCDH researchers created fictitious 13-year-old personas struggling with mental health issues, eating disorders, or interest in illegal substances. Using prompts designed to appear authentic and emotionally vulnerable, they engaged ChatGPT in simulated conversations.
The results, outlined in Wednesday’s report titled Fake Friend, show that many young users treat ChatGPT like a trusted confidant. While the chatbot often began with standard disclaimers and urged contacting professional support, CCDH found that these warnings were frequently followed by tailored responses that still fulfilled the harmful requests. Of the 1,200 prompts tested, 53% produced content deemed dangerous. Slight modifications — such as saying the request was “for a friend” — often bypassed the AI’s filters.
Examples included step-by-step self-harm methods, party plans involving multiple drugs, calorie-restrictive diets of 300–500 calories per day, and suicide notes written in the voice of a young teen. CCDH head Imran Ahmed said some of the output was “so disturbing it reduced researchers to tears.”
The organization has urged OpenAI to implement a Safety by Design strategy, introducing robust age verification, explicit usage restrictions, and integrated safety measures rather than relying solely on content screening after responses are generated.
OpenAI has acknowledged the problem, with CEO Sam Altman noting that emotional reliance on ChatGPT among teens is “common” and that new tools are being developed to identify distress and improve the AI’s handling of sensitive matters.
