tech
Mistral Moderation API
Safety plays a key role in making AI useful. At Mistral AI, we believe that system level guardrails are critical to protecting downstream deployments.That's why we are releasing a new content moderation API. It is the same API that powers the moderation service in Le Chat. We are launching it to empower our users to utilize and tailor this tool to their specific applications and safety standards.
TL;DR
- Mistral AI is releasing a new content moderation API.
- The API uses an LLM classifier trained to categorize text inputs into 9 categories.
- It is designed to be multilingual, supporting languages including Arabic, Chinese, English, French, German, Italian, Japanese, Korean, Portuguese, Russian, and Spanish.
- Two endpoints are available: one for raw text and one for conversational content, classifying the last message within a conversational context.
- The classifier addresses model-generated harms such as unqualified advice and PII.
- Mistral AI is sharing performance metrics (AUC PR) internally and aims to contribute safety advancements to the research community.