Moderation Pipeline
PSI uses a multi-layered AI moderation system to screen user-generated content. The system is designed to be cost-effective (using a cheap model for initial screening) while maintaining high accuracy (using a more capable model for flagged content).
Two-Pass Architecture
Moderation runs in-process in the PSI backend (server/modules/moderation.ts) — there is no separate moderation microservice. The pipeline uses two OpenAI models plus a blocklist gate:
sequenceDiagram
participant User
participant Frontend as PSI Frontend
participant Backend as PSI Backend
participant Light as Light Model<br/>(gpt-5.4-mini)
participant Heavy as Heavy Model<br/>(gpt-5.4)
User->>Frontend: Submit comment
Frontend->>Backend: POST comment
Backend->>Backend: Check blocklist
alt Blocklisted
Backend-->>Frontend: Reject (blocklist)
else Passes blocklist
Backend->>Light: Check comment (fast, cheap)
Light-->>Backend: Result (pass / flag / reject)
end
alt Comment flagged by light model
Backend->>Heavy: Re-check comment (detailed, accurate)
Heavy-->>Backend: Final decision (approve / reject)
end
Backend-->>Frontend: Moderation result
alt Comment approved
Frontend->>User: Comment published
else Comment rejected
Frontend->>User: Comment rejected with explanation
end Models
| Model | Default | Role | Cost |
|---|---|---|---|
| Light model | gpt-5.4-mini | Initial screening of all comments | Low |
| Heavy model | gpt-5.4 | Detailed analysis of flagged comments | Higher |
Both OpenAI models are configurable via environment variables (OPENAI_MODEL_LIGHT, OPENAI_MODEL_HEAVY). The light model runs with thinking effort none; the heavy model uses thinking effort low. Azure OpenAI is also supported as an alternative provider.
Moderation Prompts
AI moderation uses configurable prompts stored in server/prompts/. These prompts instruct the LLM to evaluate comments against community guidelines, checking for:
- Toxicity and hate speech
- Personal attacks
- Misinformation
- Off-topic content
- Spam
Pre-Moderation Review
AI-flagged comments are stored for human review in the Moderation Dashboard (unless disabled via PREMOD_REVIEW_DISABLED). This enables:
- Moderator annotation -- moderators mark whether AI decisions were correct
- False positive identification -- helps improve moderation accuracy over time
- Evaluation datasets -- builds training data for model improvements
- GDPR compliance -- review data is stored for 30 days; user IDs are not stored
Standalone Moderation Service (historical)
Note: An earlier iteration of PSI used a standalone moderation microservice (
moderation-service/, Express 4 + Firebase Firestore/MongoDB + OpenAI/Perspective API). That prototype was never merged intomain; the code lives only on other branches (atproto,it-sec-audit). As of the current codebase, moderation runs entirely in-process in the PSI backend and no standalone service is deployed. This section is retained only for historical context.
The abandoned prototype exposed REST endpoints including GET /health, POST /moderation/check-comment, POST /moderation/check-perspective, POST /moderation/check-jigsaw, and /blocklist/* routes. None of these are part of the current system — modern moderation, blocklist management, and pre-mod review are all implemented inside the PSI server modules and exposed via typed oRPC procedures (server/orpc/routers/blocklist.ts, server/orpc/routers/premod-reviews.ts).
Configuration
| Environment Variable | Description |
|---|---|
OPENAI_KEY | OpenAI API key |
OPENAI_MODEL_LIGHT | Light model name (default: gpt-5.4-mini) |
OPENAI_MODEL_HEAVY | Heavy model name (default: gpt-5.4) |
OPENAI_ENDPOINT | Custom OpenAI endpoint |
OPENAI_PROVIDER | Provider: openai, azure_openai, zdf_openai, or vercel_ai |
AZURE_OPENAI_KEY | Azure OpenAI key (if using Azure) |
PREMOD_REVIEW_DISABLED | Disable pre-moderation human review |
Further Reading
- Configuration reference (psi-product) -- all server environment variables
- Pre-Moderation Review (psi-product) -- detailed review workflow