Pre-publication content filtering
Classify user-submitted text or images before publishing them.
AI & ML · Function
moderations
The Moderations API identifies potentially harmful text and image content. Use its category flags and scores to apply application-specific filtering, review, or account-safety policies.
How it works
The Moderations API evaluates text and image inputs with a moderation model and returns category flags and scores. Applications can use these results to filter content, route cases for review, or take account-level action.
Syntax
POST /v1/moderationsParameters
| Name | Type | Description |
|---|---|---|
model | string | The moderation model to use; the documentation identifies omni-moderation-latest as accepting text and image inputs. |
input | string or array | The text and/or image content to classify. |
Returns
A moderation result containing the model, successful result data, category flags and scores, or an error object when moderation cannot complete.
Key points
- Moderation results are signals; your application decides whether to filter content, send it for review, or intervene on an account.
- The endpoint does not charge for moderation requests.
- Image inputs sent for moderation have a 20 MB size limit.
- Use the current omni moderation model for supported text and image requests.
Common mistakes
- The current moderation model accepts text and images, but audio is not a supported input modality.
- Do not submit known or suspected child sexual abuse material; this API is not a substitute for dedicated child-safety safeguards.
- A moderation operation may return an error object rather than scores, so inspect the result type before accessing score fields.
Example: Moderate text and image input
from openai import OpenAI
client = OpenAI()
result = client.moderations.create(
model="omni-moderation-latest",
input=[
{"type":"text","text":"I will hurt you."},
{"type":"image_url","image_url":{"url":"https://example.com/image.png"}}
],
)
print(result.results[0].flagged)
print(result.results[0].categories)This sends text and an image to the moderation endpoint and prints whether any category flagged the input plus the category results.
When to use it
Generated-content safety checks
Request moderation alongside generated output and inspect it before display.
Safety review routing
Send flagged cases to human review instead of automatically blocking every result.
Account intervention
Use moderation signals as part of account enforcement workflows.
Interview questions
QWhat does the Moderations API return?
The endpoint returns moderation results for submitted content, including a flagged boolean, category labels, and category scores. The application should apply its own policy to decide whether to block, review, or otherwise respond.
Follow-up: How would you handle a failed moderation operation? Check whether the result is successful before reading scores, then apply an explicit fail-safe policy appropriate to the application.
QWhich input types does moderation support?
The current moderation model handles text and image inputs. It does not classify audio, so audio safety requires a separate control or workflow.
Follow-up: What if the application receives mixed multimodal content? Moderate each supported text or image input and define a separate policy for unsupported modalities.
QHow can moderation be used with generated content?
Moderation can be performed as a separate endpoint request, or moderation scores can be requested alongside generated content. In the latter case, the model still generates normally, so the application must inspect the scores before displaying or acting on the output.
Follow-up: What should happen when generated output is flagged? Hold the output from the user-facing path and route it through the application's enforcement or review policy.
Related
Moderate generated contentomni-moderation-latestModerationResultscategory_scoresflaggedResponses APIChat Completions