AI Securityintermediate11 min · 9 steps
Content Moderation
Classify both inputs and outputs into policy categories and severities, then apply a defined action — allow, warn, redirect, block or hand to a human — consistently and auditably.
requesterrortokensignalresponse
ProblemStep 1 of 9
One filter, one action, three wrong answers
The IT assistant gets a frustrated user venting about the VPN, someone asking how to switch off endpoint protection so a game runs, and a user whose messages show hopelessness. A keyword filter with a single block action treats all three the same.
The venting user gets scolded, the bypass request gets no alternative, and the person in distress gets a cold refusal — the worst outcome of all. Meanwhile nothing checks what the model says back.
Watch outModeration is not one decision. Different categories and severities need different actions.
What changed
- +Three very different messages
- ~One block action for all of them
Lesson 17 of 22 in AI Security