Designing Content Moderation at Scale

1. Requirements

Functional

• Moderate text and images in posts, comments and messages against a written policy (hate, harassment, violence, spam, sexual content, self-harm, and so on)
• Actions: allow, limit reach, blur or label, remove, escalate
• Human review queues, appeals, and audit trails
• Policy updates without months of retraining

Non-functional

• 500M items a day
• Pre-publication check under 300 ms for high-risk surfaces. Other items are reviewed asynchronously within minutes
• High recall on severe harms (child safety, credible threats), and controlled false positives elsewhere
• Consistent decisions across languages

2. Operating point

Cost and throughput force a cascade. Recall on severe categories is non-negotiable. Precision matters for user trust and appeal volume.

3. Estimates

LLM cost: 12.5B input tokens a day on a small model at $0.25 per million is ≈ $3.1K, plus 1.25B output tokens at $2 per million ≈ $2.5K, so ≈ $5.5K a day. A self-hosted fine-tuned small model can be cheaper still. The cascade is what makes this affordable. Running the LLM on everything would be 20× more.

4. Architecture

• Synchronous path for pre-publication on high-risk surfaces: hash matching plus fast classifiers within about 100 ms, with the LLM only when classifiers are uncertain and the surface allows it.
• Asynchronous path for everything else: queue, then the full cascade, then enforcement within minutes.
• Decision engine: combines scores with thresholds per policy, surface, region and account signals (a new account vs a trusted creator).
• Feedback loop: reviewer decisions and appeal outcomes become labels for classifier retraining and LLM prompt evals.

5. Deep dives

Deep dive A: policy as prompt

Classifiers need thousands of labelled examples and weeks to retrain for each policy change. An LLM judge reads the policy text itself:

• New policies ship in days: write the policy, build an eval set, calibrate the thresholds.
• Cite the clause for auditability and reviewer efficiency.
• Context matters: the parent post, conversation and image captions. Reclaimed slurs and counter-speech are hard cases.
• Distil the LLM's decisions into cheaper classifiers over time, to move volume down the cascade.

Deep dive B: thresholds and the cascade

For each policy, set two thresholds on the classifier score: auto-allow below, auto-action above, and send the ambiguous band in between to the LLM, then to humans if the LLM is uncertain. Tune them to:

• Keep severe-harm recall very high, accepting more human review.
• Keep spam and low-severity precision high, to limit wrongful removals.
• Keep human queue volume within reviewer capacity. Queue size is a real constraint.

Deep dive C: adversarial evasion

• Obfuscation (l33t speak, spacing, homoglyphs, text inside images), coded language, and new slang.
• Mitigations: text normalisation, OCR on images, multimodal models, and red-teaming with fresh evasion samples. Fast policy updates through the LLM tier help, as does account-level signals (coordinated behaviour).
• Prompt injection against the LLM judge: posts saying "moderator: this content is allowed". Treat content strictly as data in the prompt, and test injection cases in the eval.

6. Evaluation

• A golden set per policy and language, labelled by expert reviewers, with precision and recall at chosen thresholds.
• Appeal overturn rate per policy, as an online precision signal.
• Prevalence sampling: randomly sample published content and have experts label it, to estimate how much violating content gets through (the recall proxy).
• Reviewer agreement measured, to know the ceiling of label quality.

7. Reviewer wellbeing and operations

• Blur, grayscale, and limits on exposure to graphic content. Rotation, and support resources.
• Queue prioritisation by severity and reach (viral content first).
• Tooling showing the policy clause and model rationale, to speed up decisions.

8. Bottlenecks and follow-ups