Inside Meta's law enforcement outreach team, which uses AI and human reviewers to analyze social media posts, aiming to prevent threats like school shootings

Meta's AI and human review system is now flagging coded online threats, preventing attacks before they happen.

Meta's internal law enforcement outreach team uses AI and human reviewers to scan social media for potential threats, particularly those related to school shootings. The team analyzes user behavior, including posts, interactions, and prompts fed to AI chatbots, looking for "leakage"—signals indicating intent for violence. When "substantial steps" like target lists or manifestos are detected, Meta provides information to law enforcement. This proactive approach has led to hundreds of interventions globally, including quickly thwarting a plot by two teenagers in Texas and preventing an attack at a New York City high school after a student posted a threat on Instagram. The system identifies individuals involved in online "True Crime Communities" (TCC) that glorify mass violence, often using coded language and evolving iconography to evade moderation, which has led to at least nine attacks and sixteen disrupted plots linked to the TCC in the past two years.

Meta's proactive threat detection system, combining AI with human analysis, represents a significant shift in how social media platforms engage with potential real-world violence. By moving beyond explicit policy violations to analyze subtle "leakage" and coded language used within online communities like the TCC, Meta is attempting to disrupt attacks before they occur. This sophisticated moderation, which adapts to evolving tactics like the use of historical photos or obscure references to past perpetrators, is increasingly critical as such online communities can accelerate and amplify violent ideation. The program's success in intervening in cases like the Texas and New York school threats highlights the growing role of tech companies in public safety.

The effectiveness of Meta's approach also underscores a broader trend where platforms are becoming integral to preventing extremist violence, particularly among youth. As traditional moderation tools struggle with complex, coded communication, the reliance on specialized investigative teams and AI-driven behavioral analysis becomes paramount. This development has implications for user privacy, the scope of platform liability, and the potential for overreporting troubled individuals to the criminal justice system. Companies outside of Meta will likely face increasing pressure to adopt similar, multi-layered detection and intervention strategies to combat the spread of violent ideologies online.