Skip to content

Moderation Pipeline (Code Level)

C4 Level 4 detail of the PSI moderation system. This page shows the internal structure of the moderation module and its in-process two-pass AI flow.

Module Structure

The moderation system lives entirely in the PSI server. The core logic is in server/modules/moderation.ts; the LLM/blocklist/pre-mod support lives in adjacent modules and the LLM adapter layer.

classDiagram
    class ModerationModule {
        +publicFunctions
        +adminFunctions
        checkCommentWithGpt(store, params)
        checkCommentWithGptHeavy(store, params)
        checkPerspectiveWithGpt(store, params)
        recordJudgement(store, params)
        getJudgementHistory(store, params)
    }

    class BlocklistModule {
        +publicFunctions
        checkBlocklist(store, params)
        list / add / remove terms
    }

    class PremodReviewModule {
        +publicFunctions
        +adminFunctions
        storePremodReviewEntry(store, params)
        getPremodReviewData(store, params)
        annotatePremodData(store, params)
    }

    class LLMAdapter {
        <<interface>>
        +chatCompletion(messages, options) Promise~Response~
    }

    class OpenAIAdapter {
        +chatCompletion(messages, options)
    }

    class AzureOpenAIAdapter {
        +chatCompletion(messages, options)
    }

    class VercelAIAdapter {
        +chatCompletion(messages, options)
    }

    class ZDFOpenAIAdapter {
        +chatCompletion(messages, options)
    }

    ModerationModule --> LLMAdapter : uses for AI checks
    ModerationModule --> BlocklistModule : checks blocklist gate
    ModerationModule --> PremodReviewModule : stores flagged data
    LLMAdapter <|.. OpenAIAdapter
    LLMAdapter <|.. AzureOpenAIAdapter
    LLMAdapter <|.. VercelAIAdapter
    LLMAdapter <|.. ZDFOpenAIAdapter

Two-Pass Moderation Flow (Detailed)

sequenceDiagram
    participant Client as PSI Frontend
    participant Backend as PSI Backend (Hono)
    participant Prompts as server/prompts/
    participant Light as Light Model (gpt-5.4-mini)
    participant Heavy as Heavy Model (gpt-5.4)
    participant Store as ServerStore
    participant Blocklist as Blocklist Module
    participant PremodReview as PremodReview Module
    participant Email as Email Adapter

    Client->>Backend: POST comment

    Backend->>Blocklist: Check terms
    alt Blocklisted
        Blocklist-->>Backend: reject
    end

    Backend->>Prompts: Load moderate.txt (system prompt)
    Prompts-->>Backend: System prompt with community guidelines

    Backend->>Light: chatCompletion(system + user prompt)
    Note over Light: User prompt includes:<br/>comment text, parent context,<br/>community guidelines, response format
    Light-->>Backend: JSON: {decision, explanation, confidence}

    Backend->>Store: Record light model result

    alt decision = "flag" (needs heavy review)
        Backend->>Prompts: Load heavy/moderate.txt
        Backend->>Heavy: chatCompletion(system + user prompt + light result)
        Heavy-->>Backend: JSON: {decision, explanation, confidence}
        Backend->>Store: Record heavy model result
    end

    alt PREMOD_REVIEW enabled AND comment rejected
        Backend->>PremodReview: storePremodReviewEntry(comment, decision)
        Note over PremodReview: Stores comment without user ID<br/>30-day TTL for GDPR compliance
    end

    alt Comment rejected
        Backend->>Email: Send rejection notification to user
    end

    Backend-->>Client: {decision, explanation}

Comment State Machine

stateDiagram-v2
    [*] --> Submitted: User posts comment

    Submitted --> LightModelCheck: Sent to gpt-5.4-mini

    LightModelCheck --> Approved: Light model approves
    LightModelCheck --> Rejected: Light model rejects
    LightModelCheck --> HeavyModelCheck: Light model flags

    HeavyModelCheck --> Approved: Heavy model approves
    HeavyModelCheck --> Rejected: Heavy model rejects
    HeavyModelCheck --> PendingReview: Heavy model uncertain

    PendingReview --> Approved: Moderator approves
    PendingReview --> Rejected: Moderator rejects

    Approved --> Published: Visible to all users
    Rejected --> Hidden: Not visible, user notified

    Published --> [*]
    Hidden --> [*]

Pre-Moderation Review Data Flow

graph TD
    subgraph "Comment Submission"
        A[User submits comment] --> B{AI moderation}
        B -->|Approved| C[Published]
        B -->|Rejected/Flagged| D[Comment blocked]
    end

    subgraph "Pre-Mod Review Pipeline"
        D --> E[Strip PII: remove user ID]
        E --> F[Store with 30-day TTL]
        F --> G[Available in Moderation Dashboard]
    end

    subgraph "Human Review"
        G --> H[Moderator reviews AI decision]
        H --> I{Correct?}
        I -->|Yes| J[Mark as true positive/negative]
        I -->|No| K[Mark as false positive/negative]
    end

    subgraph "Feedback Loop"
        J --> L[AI Evaluation Dataset]
        K --> L
        L --> M[Model accuracy metrics]
        L --> N[psi-ai-eval analysis]
    end

Prompt Architecture

The moderation system uses structured prompts stored as text files:

Prompt File Purpose Used By
moderate.txt Core moderation: evaluate against community guidelines Light model (gpt-5.4-mini)
heavy/moderate.txt Detailed re-evaluation with additional context Heavy model (gpt-5.4)
moderatePerspective.txt Perspective identifier moderation (LLM, not an external API) Alternative flow
comment_quality.txt Score comment quality (constructiveness) Ranking module
comment_bridging.txt Score bridging potential (cross-perspective) Ranking module
name_check.txt Check if display name violates guidelines Profile module
name_looks_real.txt Assess if a name appears realistic Profile module
tag_article.txt Auto-tag articles for categorization Article module
translate.txt Translate user content between languages Translation module
conversationhelper_*.txt (5) AI-assisted conversation guidance ZDF conversation helper

Standalone Moderation Service (historical)

Note: An earlier prototype used a standalone moderation microservice (moderation-service/, Express 4 + Firebase Firestore/MongoDB + OpenAI/Perspective). That prototype was never merged into main — the code lives only on other branches (atproto, it-sec-audit). In the current codebase, all moderation runs in-process in the PSI server (server/modules/moderation.ts). This section is retained only for historical context.

The abandoned prototype exposed REST endpoints including GET /health, POST /moderation/check-comment, POST /moderation/check-perspective, POST /moderation/check-jigsaw, and /blocklist/* routes. None of these are part of the current system — modern moderation, blocklist management, and pre-mod review are implemented inside the PSI server modules and exposed via typed oRPC procedures (server/orpc/routers/blocklist.ts, server/orpc/routers/premod-reviews.ts).

Further Reading