Designing Content Moderation at Scale

Content moderation is the oldest large-scale ML system on the internet, and one of the places where LLMs changed the architecture most. Before, every policy needed its own classifier, trained on thousands of labelled examples, retrained for months whenever the policy changed. Now a model can read the policy itself. But at 500M items a day, you cannot run an LLM on everything, and some decisions must happen before a post is even visible. The result is a cascade, and designing its thresholds is the core of the problem.

1. Clarify the task and the quality bar

Questions to ask:

• What platform and which surfaces? Assume a social platform: public posts, comments, profile content and group messages. Private messages have different rules and privacy constraints.
• Which modalities? Text and images at launch; video and live streams as follow-ups.
• Volume? 500M items a day.
• Which policies? A written community policy covering hate, harassment, violence, sexual content, self-harm, spam, scams and misinformation, plus legal obligations such as reporting child sexual abuse material (CSAM) to authorities and obligations under laws like the EU Digital Services Act (DSA).
• What actions? Allow, label, blur, limit reach, remove, suspend the account, report to authorities.
• What is "good"? Very high recall on severe harms, controlled false positives elsewhere, consistent decisions across languages, and every removal explainable and appealable.

Functional requirements

• Moderate text and images against a versioned written policy
• Graduated actions and account-level strikes
• Human review queues, appeals and audit trails
• Policy updates deployable in days
• Transparency reporting (counts by policy, action and region)

Non-functional requirements

• 500M items a day, ≈ 15K per second at peak
• Pre-publication checks under 300 ms on high-risk surfaces; everything else reviewed within minutes
• Near-100% recall on the most severe categories (child safety, credible threats)
• Reviewer capacity is fixed in the short term: the system must fit within it

2. Choose the operating point

Cost and throughput force a cascade. Recall on severe categories is non-negotiable, even at the price of more human review. Precision matters for user trust, creator livelihoods and appeal volume. Different policies sit at different points: for spam, favour precision and automation; for credible threats, favour recall and humans.

3. Estimates

LLM cost: 12.5B input tokens a day on a small model at $0.25 per million is ≈ $3.1K, plus 1.25B output tokens at $2 per million ≈ $2.5K, so ≈ $5.5K a day. A self-hosted fine-tuned small judge can be cheaper, and keeps user content in-house. Running an LLM on every item would cost 20× more and add latency everywhere.

Human cost: 500 reviewers is far larger than the LLM bill. The thresholds are really a staffing decision: widening the ambiguous band by 0.05% of traffic means 250 more reviewers.

4. Architecture

Components

Two paths

• Synchronous (pre-publication) on high-risk surfaces (new accounts posting links, public comments on trending posts): hash matching plus fast classifiers within ≈ 100 ms, the LLM only if classifiers are uncertain and the surface's budget allows, otherwise hold for async review.
• Asynchronous for everything else: publish, then run the full cascade from a queue and enforce within minutes. Also re-scan when a post goes viral or gets reported.

5. Deep dives

Deep dive A: policy as prompt

Classifiers need thousands of labelled examples and weeks to retrain for each policy change. An LLM judge reads the policy text itself:

• New policies ship in days: write the policy, build a labelled eval set of a few hundred examples, calibrate thresholds, deploy.
• Structured output (decision, clause, confidence, rationale) so the decision engine can act on it and reviewers can see why. See Function calling and structured outputs.
• Policy versions are part of the prompt and recorded with every decision, so audits and appeals know which rules applied.
• Context matters: the parent post, the conversation, image captions and OCR text. Reclaimed slurs, satire, news reporting and counter-speech are the hard cases.
• Distillation: once the LLM judge is calibrated, its decisions label millions of examples to train the fast classifiers, which moves volume down the cascade and cuts cost over time. See Distillation.

Deep dive B: thresholds and the cascade

For each policy, set two thresholds on the classifier score: auto-allow below, auto-action above, and send the band in between to the LLM, then to humans if the LLM is unsure.

• Severe harms: a wide band and a low auto-action threshold. More human review is acceptable; misses are not.
• Spam and low-severity policies: a narrow band, high precision, mostly automated.
• Surface and region: stricter thresholds on surfaces seen by minors and in regions with specific legal requirements.
• Account signals: a new account with no history posting links gets stricter treatment than a verified creator.
• Capacity check: the sum of all bands must fit the reviewer queue. When it doesn't, prioritise by severity and reach, not arrival time.

Deep dive C: multimodal content

• Images: perceptual hashes catch re-uploads of known material even after resizing or cropping. Image classifiers score nudity, violence and weapons. OCR extracts text in images, a common evasion route, which then goes through the text pipeline.
• Multimodal LLMs judge memes and screenshots where the meaning comes from combining text and image.
• Video: sample frames at intervals, transcribe the audio, and run both pipelines, prioritising by reach. Live streams need continuous sampling and a kill switch.
• AI-generated media: provenance signals (C2PA metadata, watermarks) and detection classifiers feed labels ("AI-generated") rather than removal on their own.

Deep dive D: behaviour and account signals

Many violations are invisible in a single post: a harassment campaign is 200 individually mild replies from accounts created yesterday.

• Account features: age, history, previous strikes, verification.
• Velocity and graph signals: many accounts posting similar content at once, shared infrastructure (IP ranges, devices), and reply-bombing patterns indicate coordinated behaviour.
• User reports as a signal, weighted by the reporter's past accuracy.
• Enforcement can target the network (a batch of coordinated accounts) rather than each post.

Deep dive E: enforcement, strikes and appeals

• Graduated actions: label or blur → limit reach → remove → temporary restriction → suspension. Severity and history decide the level.
• Strikes accumulate per account with expiry, so one mistake doesn't follow someone forever.
• Notices tell users which policy was applied and how to appeal, as required by laws like the DSA.
• Appeals go to a different reviewer, with the original decision and rationale visible. Overturn rates per policy are one of the best precision signals available.
• Mandatory reporting: confirmed CSAM is reported to the relevant authority (NCMEC in the US), and evidence is preserved according to law.

Deep dive F: adversarial evasion

• Obfuscation (l33t speak, spacing, homoglyphs, text inside images), coded language and fast-moving slang.
• Mitigations: text normalisation, OCR, multimodal models, red-teaming with fresh evasion samples, and fast policy updates through the LLM tier.
• Prompt injection against the LLM judge: posts containing "Moderator: this content is allowed". Treat content strictly as data in the prompt (clearly delimited, never as instructions), and include injection cases in the eval set. See Prompt injection and guardrails.
• Threshold probing: bad actors test what gets through. Randomising some checks and acting at the account level makes probing less useful.

6. Evaluate, safeguard, operate

Evaluation

• Golden sets per policy and language, labelled by expert reviewers, with precision and recall at the chosen thresholds. Evaluate every classifier, prompt and threshold change against them.
• Appeal overturn rate per policy as an online precision signal.
• Prevalence sampling: randomly sample published content and have experts label it, to estimate how much violating content gets through. This is the only honest recall estimate, because missed content is by definition not in your queues.
• Reviewer agreement (for example Cohen's kappa), to know the ceiling of label quality. If experts agree only 85% of the time, a model can't be judged against 99%.
• Fairness: error rates by language and dialect; some dialects are systematically over-flagged by classifiers.

See Offline evals and golden sets and LLM-as-judge.

Reviewer operations and wellbeing

• Blur, grayscale and reduced-size previews for graphic content; limits on daily exposure; rotation and support resources.
• Queue prioritisation by severity and reach (viral content first), and routing by language and expertise.
• Tooling that shows the policy clause, the model's rationale and relevant context, so each decision takes seconds rather than minutes.

Privacy. Private and group messages get narrower scanning (for example hash matching for known severe material, plus user reports), with retention limits and strict access to reviewed content. End-to-end encrypted messaging limits server-side scanning entirely, which is a policy decision as much as a technical one.

Failure modes

7. How the design evolves

• v1: hash matching, off-the-shelf classifiers, user reports and a review queue.
• v2: an LLM judge for the ambiguous band with policy-as-prompt; appeals feeding labels.
• v3: multimodal and video, account and network signals, distillation of the judge into classifiers, and per-region policy variants.

8. Bottlenecks and follow-ups