Moderation is what separates a platform that lasts from one that gets deplatformed by its own host. It is also the line most often missing from a build quote, because it is not a feature - it is an operation with software attached.
Two gates, not one
Everything published passes an automated screen before it goes live, and a proportion passes a human afterwards. Skipping the first is negligent at any volume; skipping the second means false positives are never corrected and edge cases are never caught.
| Stage | Decides | Latency budget |
|---|
| Automated pre-publish | Obvious violations, hashes of known material, apparent age signals | Seconds - it sits in the upload flow |
| Human queue | Everything flagged, plus a random sample of what passed | Hours, with a defined maximum |
| Reactive | Reports from users, rights holders, payment processors | Same day for acknowledgement |
The random sample of passed content is what most platforms omit and what tells you whether the automated stage is actually working. Without it you learn about failures from a complaint.
CSAM: the obligation, stated plainly
Detection is not a preference. Platforms accepting user uploads run hash matching against known material, and a positive match triggers a defined path: preserve the evidence, do not delete it, report to the relevant authority, and suspend the account. The order matters - deleting before reporting destroys what the authority needs and can itself be an offence.
Two build requirements follow. The report path must exist before launch and be written down, because it is not something to improvise at midnight. And moderators must have a route to escalate without reviewing material repeatedly - repeated exposure is a workplace-harm issue, and platforms that ignore it lose staff and then lose the queue.
Age assurance on the content, not just the door
The age gate at the entrance protects visitors. A separate obligation covers people appearing in the content: verified identity records for every performer, tied to each piece of content, retained for the statutory period and produced on demand.
Practically this means the upload flow refuses content that is not linked to a verified performer record. Platforms that let uploads through first and reconcile later end up with a library they cannot prove anything about, and the reconciliation project is more expensive than the control would have been.
What the queue looks like when it works
- Every item shows why it was flagged, so the moderator is not re-deriving the decision.
- One-click actions with mandatory reason codes - free-text notes are not analysable and nobody reads them later.
- A cap on continuous review time, and rotation away from the hardest categories.
- An appeal path visible to the uploader, because false positives are certain and an unappealable platform sheds its best contributors.
- An audit trail that records who decided what, when, and on what basis.
That audit trail is the part that turns an incident into a defensible position. A platform that can show a consistent process applied consistently is in a completely different conversation with a processor or a regulator than one that can only assert good intentions.
Staffing the queue honestly
Volume drives it, but so does the flag rate: a platform with permissive upload rules generates more queue than one that constrains formats and sources up front. Reducing what can be uploaded is the cheapest way to reduce moderation load, and it is usually available before anyone considers it.
Coverage matters more than headcount. Uploads arrive around the clock and a queue that only moves during office hours means content sits published and unreviewed overnight - which is precisely when a complaint becomes a suspension.
What to log, and for how long
Log the decision, the reason code, the moderator, the timestamp and the content identifier - not the content itself beyond what the retention obligation requires. Two competing pressures apply: records must be sufficient to demonstrate process, and holding more personal data than necessary is its own liability under data-protection rules. Write the retention schedule down, apply it automatically, and review it when you enter a new jurisdiction.
Automation limits worth knowing
Automated classification is good at known material and obvious categories, weak at context, and unreliable at apparent age near adulthood - which is exactly the boundary that matters most. Tune the automated stage to be conservative and let the human queue absorb the ambiguity. A platform tuned to minimise queue volume is a platform that has decided to find out about its failures from someone else.
Writing the policy before writing the software
Moderation software enforces a policy; without one it becomes a queue of individual judgements that contradict each other within a week. The policy needs to answer, in writing and before the build: what is never allowed, what is allowed with verification, what is allowed only in some jurisdictions, and who decides the edge cases.
Keep it versioned. When a decision is challenged - by an uploader, a processor or a regulator - the useful answer is which version of the policy was in force on the date and what it said, and that answer only exists if the policy lives under version control rather than in a document someone edits in place.
What processors and hosts actually ask for
Both eventually ask the same four questions, usually at the least convenient moment. How is content screened before publication. How quickly are reports actioned. Who is accountable, by name. And can you produce the log for a specific item on a specific date.
A platform that can answer those in an afternoon keeps its account. One that needs a week to reconstruct the answer is treated as a risk regardless of how careful it actually was - which is why the audit trail is worth building before it is ever requested rather than after.
Common questions
Can moderation be fully automated?
No. Automated screening handles known material and obvious violations; it is weakest at context and at apparent age near adulthood, which is the boundary that matters most. The automated stage should be conservative and the human queue should absorb the ambiguity.
What happens on a CSAM hash match?
A defined path: preserve, do not delete, report to the relevant authority, suspend the account. Deleting before reporting destroys what the authority needs and can be an offence in itself. The path has to exist in writing before launch.
How large should the human queue team be?
Sized by flag rate and coverage rather than by headcount alone. Uploads arrive around the clock, and content sitting published and unreviewed overnight is when a complaint becomes a suspension. Constraining what can be uploaded is the cheapest way to reduce load.
What should the audit trail contain?
Decision, reason code, moderator, timestamp and content identifier - enough to demonstrate a consistent process, and no more personal data than the retention obligation requires.
Related reading