Skip to content

Moderation Pipeline

PSI uses a multi-layered AI moderation system to screen user-generated content. The system is designed to be cost-effective (using a cheap model for initial screening) while maintaining high accuracy (using a more capable model for flagged content).

Two-Pass Architecture

Moderation runs in-process in the PSI backend (server/modules/moderation.ts) — there is no separate moderation microservice. The pipeline uses two OpenAI models plus a blocklist gate:

sequenceDiagram
    participant User
    participant Frontend as PSI Frontend
    participant Backend as PSI Backend
    participant Light as Light Model<br/>(gpt-5.4-mini)
    participant Heavy as Heavy Model<br/>(gpt-5.4)

    User->>Frontend: Submit comment
    Frontend->>Backend: POST comment

    Backend->>Backend: Check blocklist
    alt Blocklisted
        Backend-->>Frontend: Reject (blocklist)
    else Passes blocklist
        Backend->>Light: Check comment (fast, cheap)
        Light-->>Backend: Result (pass / flag / reject)
    end

    alt Comment flagged by light model
        Backend->>Heavy: Re-check comment (detailed, accurate)
        Heavy-->>Backend: Final decision (approve / reject)
    end

    Backend-->>Frontend: Moderation result

    alt Comment approved
        Frontend->>User: Comment published
    else Comment rejected
        Frontend->>User: Comment rejected with explanation
    end

Models

Model Default Role Cost
Light model gpt-5.4-mini Initial screening of all comments Low
Heavy model gpt-5.4 Detailed analysis of flagged comments Higher

Both OpenAI models are configurable via environment variables (OPENAI_MODEL_LIGHT, OPENAI_MODEL_HEAVY). The light model runs with thinking effort none; the heavy model uses thinking effort low. Azure OpenAI is also supported as an alternative provider.

Moderation Prompts

AI moderation uses configurable prompts stored in server/prompts/. These prompts instruct the LLM to evaluate comments against community guidelines, checking for:

  • Toxicity and hate speech
  • Personal attacks
  • Misinformation
  • Off-topic content
  • Spam

Pre-Moderation Review

AI-flagged comments are stored for human review in the Moderation Dashboard (unless disabled via PREMOD_REVIEW_DISABLED). This enables:

  • Moderator annotation -- moderators mark whether AI decisions were correct
  • False positive identification -- helps improve moderation accuracy over time
  • Evaluation datasets -- builds training data for model improvements
  • GDPR compliance -- review data is stored for 30 days; user IDs are not stored

Standalone Moderation Service (historical)

Note: An earlier iteration of PSI used a standalone moderation microservice (moderation-service/, Express 4 + Firebase Firestore/MongoDB + OpenAI/Perspective API). That prototype was never merged into main; the code lives only on other branches (atproto, it-sec-audit). As of the current codebase, moderation runs entirely in-process in the PSI backend and no standalone service is deployed. This section is retained only for historical context.

The abandoned prototype exposed REST endpoints including GET /health, POST /moderation/check-comment, POST /moderation/check-perspective, POST /moderation/check-jigsaw, and /blocklist/* routes. None of these are part of the current system — modern moderation, blocklist management, and pre-mod review are all implemented inside the PSI server modules and exposed via typed oRPC procedures (server/orpc/routers/blocklist.ts, server/orpc/routers/premod-reviews.ts).

Configuration

Environment Variable Description
OPENAI_KEY OpenAI API key
OPENAI_MODEL_LIGHT Light model name (default: gpt-5.4-mini)
OPENAI_MODEL_HEAVY Heavy model name (default: gpt-5.4)
OPENAI_ENDPOINT Custom OpenAI endpoint
OPENAI_PROVIDER Provider: openai, azure_openai, zdf_openai, or vercel_ai
AZURE_OPENAI_KEY Azure OpenAI key (if using Azure)
PREMOD_REVIEW_DISABLED Disable pre-moderation human review

Further Reading