Skip to main content

OpenRouter Q&A Service

SECTION 22B — OPENROUTER Q&A SERVICE

22B.1 Overview

The Q&A generation feature (§10.10) is powered by AI models accessed through the OpenRouter aggregator API. OpenRouter exposes a unified OpenAI-compatible REST surface across many model providers, including OpenAI’s Structured Outputs spec. The Notesglider backend:
  • Holds a single platform-wide OpenRouter API key.
  • Routes every Q&A request through this key.
  • Resolves the model to call per a Super-Admin-configurable resolution chain (§22B.2).
  • Enforces spend guardrails per Organisation (§22B.4).
  • Returns Structured Outputs validated against the JSON Schema for the requested question type.

Why Structured Outputs

“Structured Outputs is a feature that ensures the model will always generate responses that adhere to your supplied JSON Schema, so you don’t need to worry about the model omitting a required key, or hallucinating an invalid enum value.”
Benefits we rely on:
  • Reliable type-safety — the response shape matches our schema; no defensive parsing of free-form prose.
  • Explicit refusals — safety-based model refusals are programmatically detectable (handled via the §22B.3 refusal branch, not the validation chain).
  • Simpler prompting — no need for strongly-worded “RESPOND IN JSON ONLY” instructions; the schema is the contract.
Schemas are defined once as Pydantic models on the FastAPI side (used both for outbound JSON Schema generation against the OpenRouter request and for inbound validation). The TypeScript frontend uses Zod mirrors of the same shapes for wizard-side edit validation.
Note on the inspirational reference prompt: legacy markdown-driven prompt patterns exist in industry write-ups (which use #/##/### headings to encode document structure). Our project does NOT use markdown input for the prompt. Notesglider passes news items as a structured JSON payload extracted directly from the canonical Lexical tree (ephemeral ref handle, headline, category, Brief text-flattened). atomic_uid values are stripped server-side before payload assembly and never reach the model — see §22B.3.A “Identity Boundary”.

22B.2 Model Resolution Chain

Only the Super Admin can configure models. No other role has any visibility into model configuration. Resolution order at request time:
  1. per_org_override_model if set, else default_model.
  2. On failure → iterate the resolved fallback list (per-org override if set, else system-wide).

22B.3 Request Lifecycle

  1. Receive request from Q&A wizard with: number of questions N, per-question type spec, selected news-item atomic_uids (resolved server-side from the wizard’s grid selection).
  2. Prompt assembly:
    • Backend builds the request-scoped ref → atomic_uid map (\{ "n1": <uid_1>, "n2": <uid_2>, … \}) held in memory for this single OpenRouter call.
    • Backend constructs the payload with only ref values, headlines, categories, and flattened Brief text (Lexical text-only walk; images and formatting stripped). atomic_uid values are stripped out before the payload leaves the backend boundary.
    • Backend appends the per-type prompt template (§22B.3.A) appropriate to the question type. Mixed-type requests dispatch one call per type group.
  3. JSON Schema attachment: per the OpenAI Structured Outputs spec, the request includes the JSON Schema for the exact response shape. Schemas (one per question type variant) are defined in code as Pydantic models (server side) and JSON Schema (sent to OpenRouter).
  4. Model dispatch: call the resolved primary model.
  5. Validate response:
    • Parse JSON (Structured Outputs guarantees JSON, but parse for safety).
    • Run JSON Schema validation against the canonical shape for the requested type (per §11.1 qa.items[]):
      • subjective.straightforward: \{ statement: string, answer: string \}.
      • objective.direct: \{ statement: string, options: \{ A, B, C, D: string \}, correct_option: "A"|"B"|"C"|"D" \}. All four option keys required.
      • objective.statement_analysis: \{ topic: string, statements: [string, string, string] (exactly 3), options: \{ A, B, C, D: string \}, correct_option: "A"|"B"|"C"|"D" \}.
    • Run business-rule validation:
      • correct_option must reference a key actually present in options.
      • For objective.statement_analysis: each of the 4 options values must be drawn (without repetition) from the allowed set \{ "1", "2", "3", "1+2", "2+3", "1+3", "All of the above", "None" \}.
      • statements array length must be exactly 3.
      • Exactly one correct_option per item — multi-correct or zero-correct payloads are rejected.
  6. Retry chain:
    • Same model → up to 3 retries on validation failure or transport error.
    • If still failing → advance to the next model in the fallback list, restart at retry 0.
    • 4 fallback models × up to 3 retries each + 1 primary × 3 retries = 15 attempts maximum per request.
  7. Return to client: validated Q&A items.
  8. On total failure (all 15 attempts exhausted): return error response. Wizard surfaces “Q&A generation failed after retries. Please try again later, or skip Q&A for this document.”

22B.3.A Per-Type Prompt Templates

Each question type uses its own dedicated prompt. The selected news items are serialised as a structured JSON payload (not markdown) and appended after the prompt. Inspirational interrogative styles are listed inline so the model adopts the right voice; strict rules eliminate prose padding.

Identity Boundary — atomic_uid Never Reaches the Model

This is the non-negotiable rule for identity handling in Q&A generation: Why: atomic_uids are immutable identity primitives that must never be fabricated. Allowing the model to echo an atomic_uid back exposes us to hallucinated or malformed UIDs that would either pollute the audit chain (§11A) or fail the atomic_uid_log lookup. The model is scoped strictly to content generation for the three question types. Identity is always backend-assigned and backend-resolved. The model speaks only in ref handles — short opaque tokens (n1, n2, …) scoped to a single request. After validation, the backend swaps source_refs[]source_atomic_uids[] via the request-scoped lookup map (§22B.3.D), then assigns the QA item’s own qa_id. Common payload structure sent to the model alongside every prompt:
Note: ref values are simple positional handles (n1, n2, n3, …) assigned by the backend at request assembly time. They have no meaning outside this single OpenRouter call and are discarded after post-processing. The backend keeps a private in-memory map \{ "n1": <atomic_uid>, "n2": <atomic_uid>, … \} for the duration of the request.
Template T1 — subjective.straightforward
Template T2 — objective.direct
Template T3 — objective.statement_analysis
When the wizard request mixes types across the N requested questions, the backend dispatches a separate generation call per type group (T1, T2, T3) in a sequence one by one — never a single call mixing schemas — and merges the validated items in wizard order before returning, until then shows a UI level feedback of ‘loading state’ for enhanced UX.

22B.3.B Sample Structured Outputs

These are the exact response shapes the model must return for a single-item generation. Multi-item requests return an array of these. Sample for subjective.straightforward (Template T1):
Sample for objective.direct (Template T2):
Sample for objective.statement_analysis (Template T3):
These samples are also stored in tests/fixtures/qa/ and consumed by the §32.6 Phase 8 snapshot tests.

22B.3.C Validation, Rejection & Regeneration Logic (Model Live)

The model is treated as a live, occasionally-drifting collaborator. Even with Structured Outputs the response can deviate in subtle ways (semantic-rule violations, allowed-set drift on Template T3). The backend handles every deviation deterministically without manual intervention. Validation pipeline (runs for every response, every attempt):
Rejection (terminal, this attempt) — any of the above checks failing marks the attempt failed. The deviating response is discarded entirely — no partial-accept, no field-level patching. The next attempt regenerates from scratch. Regeneration logic when the model is live and accepting requests but its output deviates:
Augmentation rule for attempt 2 (last attempt on a given model): the failure reason is summarised to ≤ 200 chars and prepended to the prompt as a system message. This is the only attempt where prompt is mutated; all other retries replay the exact original prompt. Per-attempt logging (writes a row to openrouter_call_log, §22B.4 telemetry):
Live-model edge cases handled: Why all-or-nothing: partial accept means the saved Q&A set mixes high-quality and degraded items; users cannot tell which is which. Cost of regeneration is bounded by the 15-attempt cap.

22B.3.D Backend Post-Processing & Identity Assignment

After the validation pipeline (§22B.3.C) returns a green response, the backend performs mandatory identity translation before the items are presentable to the wizard or persisted to the database. The model’s output never touches the database unmodified. Step-by-step:
Invariants enforced at this layer (no exceptions): What this gives us:
  • Zero risk of model-fabricated atomic_uids ever entering the audit chain.
  • qa_id ordering is deterministic and survives multiple wizard runs (new acceptances append after the existing max sequence).
  • Aggregation linkage (source_atomic_uids used by Compilation/Magazine Q&A merging in §10/Phase 10) is provably correct — every uid was sourced from the backend’s own resolved map, not from the model’s text.
  • Late-add regen (§13.6) reuses the same pipeline — the only difference is the actor role on step 3 and the resulting qa-late-added = true flag.
Implementation note for the developer: place this post-processing in a single qa/post_process.py module. Any code path that writes to qa_items must go through it. Direct INSERTs from model output are an architectural defect.

22B.3.E Assembly, Merge & Atomicity (Multi-Call → Single qa Node)

A single wizard run can request a mix of types (e.g. 3 × T1 + 2 × T2 + 5 × T3 = 10 items). Per §22B.3.A, the backend dispatches one OpenRouter call per type group — never a mixed-schema call. That produces multiple parallel responses that must be merged into a single, slot-ordered, persistable qa node. This subsection is the full backend walkthrough from “Generate clicked” to “row in qa_items + node in content_json.qa.” 1. Wizard slot model The wizard collects N questions as an ordered list of slots (1-indexed in wizard order):
slot_index is the single ordering authority for the entire pipeline. The model never sees it; it lives only on the backend. 2. Group → Dispatch → Per-Group Ref Maps Group slots by type while preserving each slot’s original slot_index:
For each group, build a request-scoped ref → atomic_uid map and a parallel ref → slot_index map:
Refs are per-groupn1 in the T2 call has no relation to n1 in the T1 call. Maps live only for the duration of their group’s call. 3. Dispatch (sequential, one per group) Per §22B.3.A: a sequence of three calls in fixed order T1 → T2 → T3 (whichever groups are non-empty). The wizard shows a single progress indicator that advances per group. Each call follows the §22B.3 lifecycle independently (own 15-attempt retry chain, own validation pipeline).
4. Re-association (per group) Each group’s response is \{ "items": [...] \} in payload-positional order — the model produces one item per news_item entry it received. Walk the response and re-attach slot_index via the per-group map:
If a returned item’s source_refs resolves to more than one distinct slot_index (model fabrication across slots), the entire response is rejected per §22B.3.C step 5. 5. Merge into slot order + Identity Assignment Collect all re-associated items from all groups, sort by slot_index, then run §22B.3.D post-processing in that order. Because slot_index ascends 1..N, the resulting qa_id sequence numbers (assigned by §22B.3.D step 2) honour wizard order.
Provenance (model_used, generated_by_role, generated_by_id) is set per item from the group that produced it (different type groups may have succeeded on different fallback models — that detail is preserved per item). 6. Atomicity rule — all-or-nothing across groups If any group’s OpenRouter call terminally fails (15-attempt cap exhausted, model refusal, transport collapse), the entire wizard run is aborted. No partial qa_items are saved, no qa node is written. The wizard returns to the type-selection step with an error banner: “Q&A generation failed during the {type} group. Please retry, or skip Q&A for this document.” Rationale: a mixed wizard run is a single user intent (“give me these 10 items”). Persisting 6 successful items and silently dropping 4 produces a misleading saved set. The user always knows their full request either succeeded entirely or failed cleanly. 7. Wizard review → Accept → DB transaction (provenance split) The merged, identity-assigned items are returned to the wizard for Teacher review/edit (§10.10). Teacher can edit any field (statement, answer, option text, correct_option, statements text, topic). Teacher cannot change: qa_id, source_atomic_uids, type. Those are immutable post-assignment. Provenance split — two destinations for each item, never mixed: Top-level qa node fields (set ONCE, on the first acceptance for a document; later acceptances do NOT overwrite): Rationale: top-level fields are render-time metadata (e.g. PDF footer might show “AI-generated by gpt-4o-mini on 2026-03-01”). They must be stable; otherwise late-adds flip the displayed model name and acceptance time, which confuses users. Per-item provenance for full audit (every actor, every model, every acceptance time) lives on the qa_items rows. The transaction:
If late-add (§13.6), the qa_late_added event fires the PDF regen pipeline outside this transaction. 8. Final shape of the assembled qa node — matches §11.1 byte-for-byte After commit, documents.content_json.qa matches the canonical §11.1 schema exactly. One items[] entry per accepted question, in slot_index order, each with a unique sequential qa_id. Top-level fields (generated_by_role, generated_at, model_used) are render-only metadata set ONCE on first acceptance. Per-item provenance does NOT appear here — it lives on qa_items rows. Concrete output for the 5-slot wizard run above (mixed T1+T2+T3), produced by the §22B.3.E step 7 transaction:
Diff vs your §11.1 reference: identical structure. Same top-level fields, same per-item field set per type, same key ordering convention. The only addition is multi-uid source_atomic_uids on item 5 (slot 5 had 2 news_items selected — entirely valid per the wizard model). Single sources of truth — non-overlapping:
  • documents.content_json.qarendering (WeasyPrint PDF, PptxGenJS PPTX, HTML export, translation). Lean, matches §11.1.
  • qa_items rows → relational joins (per-item provenance, aggregation linkage in §10/Phase 10, telemetry, late-add audit trail). Carries every field both content_json has AND the provenance fields content_json deliberately omits.
9. Late-add wizard run (§13.6) The pipeline above runs identically. Two differences:
  • seq starts from the existing MAX(qa_id_seq) for the document (not from 0). New items append: QA-\{doc-id\}-006, QA-\{doc-id\}-007, ….
  • is_late_add = true → sets documents.qa_late_added = true and fires the qa_late_added pipeline event that triggers Stage-3 PDF regen (§13.6).
10. Concurrent wizard acceptances If two actors (e.g. Editor and sub-member both with permissions.qa = true) hit “Accept Q&A” on the same document simultaneously, the SELECT FOR UPDATE in step 7 serialises them. The second-committing transaction observes the first’s writes and re-assigns its qa_ids starting from the updated max sequence. Both sets persist in slot order, just with a sequence-window offset for the second. Neither loses items; neither overwrites the other.

22B.4 Spend Guardrails

Per-organisation monthly token cap (configurable by Super Admin per Org; default applies system-wide): Cap-breach behaviour:
  • Wizard “Generate” button is disabled; tooltip surfaces “Your organisation has reached its monthly Q&A generation limit. Contact your administrator to increase the limit or wait for the next billing cycle.”
  • No partial-completion: a request that would push the running token total over the cap is rejected up front (estimated tokens computed from prompt size + max completion tokens for the requested type).
Telemetry:
  • Every OpenRouter call is logged in a openrouter_call_log table with: timestamp, org_id, document_id, model used, attempt index, prompt_tokens, completion_tokens, total_tokens, success boolean, validation_errors (if any).
  • Super Admin dashboard surfaces aggregate spend per Org per month and a system-wide total.

22B.5 RBAC for Q&A Actions

A new boolean permissions.qa is added to the sub_members.permissions JSON (defaults to false).

22B.6 Translation Interaction

When the source document has translations (is-translated = true), the qa node is included in the translation request to Google Translate (§18). The translated qa node is written into each parallel translated document. Q&A items in a translated document use the same qa_id value as in the source — this enables back-mapping during aggregation.