Use Cases / content-moderation
Jev Content Moderation
UGC and chat need allow / review / block decisions at volume. Keyword lists are brittle; LLM judges are costly and hard to schema.
Try a related template free on this host — 5 anonymous real-API runs with no signup — then ship the same schema with a prepaid jv_live_ key. See is Jev free? and playground.
Why Jev
Ask for a choice action plus category noul/score fields. Your pipeline thresholds on confidence instead of parsing free text. Example: “Buy followers cheap!!!” → action=block, spam-adjacent category flags with confidence for your review queue.
How to define the decision
State = the user text (and optional context like channel). Choice action: allow / review / block. Add Noul or Score fields for spam, harassment, or scam categories your policy already names — do not invent open-ended free-text categories.
When to use Jev
- Pre-publish gates for posts and comments
- Chat abuse / spam filters with explicit labels
- Human-in-the-loop review queues
When not to use Jev
- Generating moderated rewrites
- Deep legal review of long documents
- Production image moderation — text path only here; see beta Image / Box cases for experiments
Sample input and output
From the live tool template. Run it in the playground for a real API response — sample labels below are illustrative until you click Run.
Input
text: You're an idiot and I'll find where you live.
Sample display fields
- action: block · 96%
- toxicity: yes · 94%
- harassment: yes · 91%
- spam: no · 12%
- confidence: 96%
Auto-allow only on high-confidence allow. Send review/block and mid-confidence paths to human queues. Probability is a calibrated signal, not proof of illegality — pair with appeals and policy docs.
Failure modes and human fallback
- Upstream or network errors: retry with backoff; do not invent a decision.
- Low confidence: route to a human queue or safer default — never auto-act on money/destructive tools.
- Schema drift: keep a golden fixture set in CI so label changes fail the build before production.
- This landing is text-only. Experimental image inputs / Box live under /use-cases/image-object-box/ and related beta pages — not for production.
- Never treat a single Noul as a legal determination.
- Keep human review for edge cases and appeal reversals.
API example
Same prepaid jv_live_ balance as tools. Output tokens are free on this host's metering model.
curl https://jevtypesafe.org/api/v1/decide \
-H "Authorization: Bearer jv_live_YOUR_KEY_HERE" \
-H "Content-Type: application/json" \
-d '{"state":"User content to moderate:\nYou're an idiot and I'll find where you live.","questions":{"action":{"type":"choice","instructions":"Should this content be allowed, sent to human review, or blocked?","criteria":{"allow":"safe to publish","review":"borderline, needs a human","block":"clearly violates policy"}},"toxicity":{"type":"noul","instructions":"Does this contain toxic or abusive language?"},"harassment":{"type":"noul","instructions":"Does this harass or threaten a person?"},"spam":{"type":"noul","instructions":"Is this spam or unsolicited promotion?"},"violence":{"type":"noul","instructions":"Does this threaten violence?"},"sexual":{"type":"noul","instructions":"Does this contain unwanted sexual content?"}}}'FAQ
- Is this a full trust & safety suite?
- It is a decision API for text classification. Pair it with your policy docs, appeals flow, and human review.
- How do I reproduce the moderation call?
- Open Content Moderate in the playground, run the sample, then reuse the same state + questions against POST /api/v1/decide with a jv_live_ key.
Related use cases
Ready to build?
Free anonymous tool runs, then prepaid packs and a jv_live_ key — independent host, not affiliated with TypeSafe AI.