AI & Society
Commercial moderation APIs over-moderate counter-speech and under-moderate coded hate speech, audit finds
In a 2025 paper, researchers audited five widely used commercial content moderation APIs. These are automated services that screen posts for hate speech. The audit ran five million queries based on four datasets. The researchers found that the APIs frequently relied on group identity terms, such as “black,” when deciding what to flag. All five services under-moderated implicit hate speech, meaning they missed some harmful posts that used coded messages. That problem was especially clear for coded hate aimed at LGBTQIA+ people. At the same time, the APIs over-moderated legitimate speech, including counter-speech, reclaimed slurs, and content related to Black, LGBTQIA+, Jewish, and Muslim people.
What did the audit measure?
The authors introduced a framework for auditing black-box natural language processing systems. These are automated language tools whose inner rules are hidden. They used the framework to systematically evaluate five widely used commercial content moderation APIs. The authors checked for two risks: silencing legitimate speech and failing to catch harmful speech.
What did the APIs get wrong?
The services often treated a person’s group identity as a signal of hate speech. They flagged too much counter-speech, meaning speech that pushes back against hate, and too many reclaimed slurs, meaning insult words a group uses about itself. They also missed too much implicit hate speech that used coded language. OpenAI’s and Amazon’s services performed slightly better, but the authors report that every provider still under-moderated implicit hate speech.
What do the authors recommend?
They recommend that API providers give better guidance on how to implement the tools and set thresholds. They also call for more transparency about each API’s limitations.
Frequently asked questions
Did any provider avoid these errors?
No. The paper reports that OpenAI’s and Amazon’s services performed slightly better, but all providers under-moderated implicit hate speech.
What is counter-speech?
Counter-speech is speech that pushes back against hateful messages. The audit found that the APIs sometimes treated that pushback as hate speech.
Which groups were affected?
Over-moderation hit content related to Black, LGBTQIA+, Jewish, and Muslim people. Under-moderation was strongest for implicit hate speech aimed at LGBTQIA+ individuals.
Limits and what's next
The abstract does not state the study’s limitations or a future research plan. The authors recommend that API providers offer better guidance on implementation and threshold setting, and be more transparent about their APIs’ limitations.
Source: David Hartmann et al., “Lost in Moderation: How Commercial Content Moderation APIs Over- and Under-Moderate Group-Targeted Hate Speech and Linguistic Variations,” 2025.