AI & Society
A custom text classifier can outperform off-the-shelf models at detecting many kinds of undesired content
In a 2023 paper, researchers at OpenAI in the United States describe a text classification system for real-world content moderation. The system sorts text into categories of undesired content, including sexual content, hateful content, violence, self-harm, and harassment. The authors report that the approach generalizes across different content category schemes and can be used to create high-quality classifiers that outperform off-the-shelf models. The paper appeared in Proceedings of the AAAI Conference on Artificial Intelligence, a peer-reviewed conference.
What did the researchers build?
The work describes a natural language classification system, a system that reads text and assigns each item to a category. The paper says the system is trained to detect a broad set of undesired content categories. Its approach includes designing content category schemes and labeling instructions, checking data quality, using an active learning pipeline that captures rare events, and applying a variety of methods meant to make the model robust and to avoid overfitting.
How well does it perform?
The abstract states the advantage in general terms. It reports that the approach can create high-quality content classifiers that outperform off-the-shelf models and that it generalizes to a wide range of content category schemes. The abstract does not include numerical performance figures.
FAQ
What content categories does the system detect?
The paper lists a broad set of categories, including sexual content, hateful content, violence, self-harm, and harassment.
Does the paper report how much better the system is?
No. The abstract reports the general finding that the approach can outperform off-the-shelf models, but it does not include a numerical comparison.
Who authored the paper?
The authors are researchers at OpenAI in the United States.
Limits and what's next
The abstract does not state limitations.
Source: Todor Markov et al., “A Holistic Approach to Undesired Content Detection in the Real World,” Proceedings of the AAAI Conference on Artificial Intelligence, 2023.