What is content moderation?
Content moderation is the review of user-generated or AI-generated content to decide whether it is allowed, using rules, classifiers, and human reviewers. In natural language processing it is a text classification problem. A model labels each message, post, review, or prompt as acceptable or as one of several violation types, such as harassment, spam, or a request the system should refuse. Images and audio have their own moderation models, and large language models increasingly apply detailed written policies rather than fixed categories.
In AI deployments the term covers two directions. Input moderation screens what users send to a model, where guardrails against misuse sit. Output moderation screens what the model produces before a customer sees it.
Content moderation in practice
A regional retailer moderates product reviews and questions on its website. A classifier removes spam and profanity automatically, flags reviews that mention a safety incident for the product team, and routes anything ambiguous to a reviewer. The retailer’s chatbot runs output moderation to catch a reply that exposes another customer’s order details or promises something the company cannot deliver.
What to watch for
- Measure false positives as carefully as false negatives; blocking legitimate complaints damages trust as much as missing abuse.
- Keep a human in the loop for borderline cases and appeals, and publish the policy being applied.
- Audit the classifier across dialects and languages, since moderation models often flag some communities’ speech more than others.
- Log every moderation decision, because the EU AI Act’s transparency obligations may require you to explain them.