Moderation Pipeline (Code Level)
C4 Level 4 detail of the PSI moderation system. This page shows the internal structure of the moderation module and its in-process two-pass AI flow.
Module Structure
The moderation system lives entirely in the PSI server. The core logic is in server/modules/moderation.ts; the LLM/blocklist/pre-mod support lives in adjacent modules and the LLM adapter layer.
classDiagram
class ModerationModule {
+publicFunctions
+adminFunctions
checkCommentWithGpt(store, params)
checkCommentWithGptHeavy(store, params)
checkPerspectiveWithGpt(store, params)
recordJudgement(store, params)
getJudgementHistory(store, params)
}
class BlocklistModule {
+publicFunctions
checkBlocklist(store, params)
list / add / remove terms
}
class PremodReviewModule {
+publicFunctions
+adminFunctions
storePremodReviewEntry(store, params)
getPremodReviewData(store, params)
annotatePremodData(store, params)
}
class LLMAdapter {
<<interface>>
+chatCompletion(messages, options) Promise~Response~
}
class OpenAIAdapter {
+chatCompletion(messages, options)
}
class AzureOpenAIAdapter {
+chatCompletion(messages, options)
}
class VercelAIAdapter {
+chatCompletion(messages, options)
}
class ZDFOpenAIAdapter {
+chatCompletion(messages, options)
}
ModerationModule --> LLMAdapter : uses for AI checks
ModerationModule --> BlocklistModule : checks blocklist gate
ModerationModule --> PremodReviewModule : stores flagged data
LLMAdapter <|.. OpenAIAdapter
LLMAdapter <|.. AzureOpenAIAdapter
LLMAdapter <|.. VercelAIAdapter
LLMAdapter <|.. ZDFOpenAIAdapter Two-Pass Moderation Flow (Detailed)
sequenceDiagram
participant Client as PSI Frontend
participant Backend as PSI Backend (Hono)
participant Prompts as server/prompts/
participant Light as Light Model (gpt-5.4-mini)
participant Heavy as Heavy Model (gpt-5.4)
participant Store as ServerStore
participant Blocklist as Blocklist Module
participant PremodReview as PremodReview Module
participant Email as Email Adapter
Client->>Backend: POST comment
Backend->>Blocklist: Check terms
alt Blocklisted
Blocklist-->>Backend: reject
end
Backend->>Prompts: Load moderate.txt (system prompt)
Prompts-->>Backend: System prompt with community guidelines
Backend->>Light: chatCompletion(system + user prompt)
Note over Light: User prompt includes:<br/>comment text, parent context,<br/>community guidelines, response format
Light-->>Backend: JSON: {decision, explanation, confidence}
Backend->>Store: Record light model result
alt decision = "flag" (needs heavy review)
Backend->>Prompts: Load heavy/moderate.txt
Backend->>Heavy: chatCompletion(system + user prompt + light result)
Heavy-->>Backend: JSON: {decision, explanation, confidence}
Backend->>Store: Record heavy model result
end
alt PREMOD_REVIEW enabled AND comment rejected
Backend->>PremodReview: storePremodReviewEntry(comment, decision)
Note over PremodReview: Stores comment without user ID<br/>30-day TTL for GDPR compliance
end
alt Comment rejected
Backend->>Email: Send rejection notification to user
end
Backend-->>Client: {decision, explanation} Comment State Machine
stateDiagram-v2
[*] --> Submitted: User posts comment
Submitted --> LightModelCheck: Sent to gpt-5.4-mini
LightModelCheck --> Approved: Light model approves
LightModelCheck --> Rejected: Light model rejects
LightModelCheck --> HeavyModelCheck: Light model flags
HeavyModelCheck --> Approved: Heavy model approves
HeavyModelCheck --> Rejected: Heavy model rejects
HeavyModelCheck --> PendingReview: Heavy model uncertain
PendingReview --> Approved: Moderator approves
PendingReview --> Rejected: Moderator rejects
Approved --> Published: Visible to all users
Rejected --> Hidden: Not visible, user notified
Published --> [*]
Hidden --> [*] Pre-Moderation Review Data Flow
graph TD
subgraph "Comment Submission"
A[User submits comment] --> B{AI moderation}
B -->|Approved| C[Published]
B -->|Rejected/Flagged| D[Comment blocked]
end
subgraph "Pre-Mod Review Pipeline"
D --> E[Strip PII: remove user ID]
E --> F[Store with 30-day TTL]
F --> G[Available in Moderation Dashboard]
end
subgraph "Human Review"
G --> H[Moderator reviews AI decision]
H --> I{Correct?}
I -->|Yes| J[Mark as true positive/negative]
I -->|No| K[Mark as false positive/negative]
end
subgraph "Feedback Loop"
J --> L[AI Evaluation Dataset]
K --> L
L --> M[Model accuracy metrics]
L --> N[psi-ai-eval analysis]
end Prompt Architecture
The moderation system uses structured prompts stored as text files:
| Prompt File | Purpose | Used By |
|---|---|---|
moderate.txt | Core moderation: evaluate against community guidelines | Light model (gpt-5.4-mini) |
heavy/moderate.txt | Detailed re-evaluation with additional context | Heavy model (gpt-5.4) |
moderatePerspective.txt | Perspective identifier moderation (LLM, not an external API) | Alternative flow |
comment_quality.txt | Score comment quality (constructiveness) | Ranking module |
comment_bridging.txt | Score bridging potential (cross-perspective) | Ranking module |
name_check.txt | Check if display name violates guidelines | Profile module |
name_looks_real.txt | Assess if a name appears realistic | Profile module |
tag_article.txt | Auto-tag articles for categorization | Article module |
translate.txt | Translate user content between languages | Translation module |
conversationhelper_*.txt (5) | AI-assisted conversation guidance | ZDF conversation helper |
Standalone Moderation Service (historical)
Note: An earlier prototype used a standalone moderation microservice (
moderation-service/, Express 4 + Firebase Firestore/MongoDB + OpenAI/Perspective). That prototype was never merged intomain— the code lives only on other branches (atproto,it-sec-audit). In the current codebase, all moderation runs in-process in the PSI server (server/modules/moderation.ts). This section is retained only for historical context.
The abandoned prototype exposed REST endpoints including GET /health, POST /moderation/check-comment, POST /moderation/check-perspective, POST /moderation/check-jigsaw, and /blocklist/* routes. None of these are part of the current system — modern moderation, blocklist management, and pre-mod review are implemented inside the PSI server modules and exposed via typed oRPC procedures (server/orpc/routers/blocklist.ts, server/orpc/routers/premod-reviews.ts).
Further Reading
- Moderation Pipeline Overview -- C4 Level 3 view
- Adapter Pattern -- how adapters enable provider switching
- Pre-Moderation Review (psi-product) -- detailed review workflow