Assessor_2.0/docs/REGULATORY_RAG.md

# Regulatory-control RAG with Qdrant

Status: Phase 13F one-finding and filtered Result DB batch vertical slice. The server prepares the licensed German ISO 27001/27002 corpus, indexes section-safe ISO 27002 semantic chunks plus ISO 27001 references, retrieves and groups candidates, requests grounded LiteLLM proposals, persists runs and reviewer decisions, and exposes manual proposal generation in Work and Result DB. Sparse retrieval, reranking, and confidence calibration remain out of scope.

## Boundaries

- PostgreSQL remains the only system of record for ASSESSOR projects, assessments, scans, findings, results, reviews, mapping runs, and mapping decisions.
- Qdrant is a derived, rebuildable search index. Losing or deleting its volume must not lose an ASSESSOR scan, finding, result, or review.
- The versioned regulatory corpus JSON is the rebuild source for Qdrant. It must be retained and governed separately from the index.
- React does not connect to Qdrant or the embedding provider. Both are backend-only boundaries.
- Regulatory retrieval produces candidates, not compliance decisions. The constrained mapping stage retains retrieved chunk IDs, scores, corpus/source hashes, model, prompt/schema versions, and append-only reviewer decisions in PostgreSQL.

```text
versioned regulatory corpus JSON
        | validate and chunk
        v
approved multilingual embedding service
        | dense vectors
        v
Qdrant assessor_regulatory_chunks_v2
        ^
scanner finding -> public-signal query builder -> same embedding model -> dense candidates
        -> group top 32 chunks into top 8 controls
        -> optional future hybrid fusion -> optional reranker
        -> constrained mapper -> human decision -> PostgreSQL audit record
```

## Collection contract

The active collection is `assessor_regulatory_chunks_v2`. It has two named vector slots:

- `dense`: the configured embedding dimension with cosine distance.
- `sparse`: reserved now so a later sparse/BM25-style encoder can be added without changing the retrieval abstractions.

Qdrant payloads use schema version 2 and contain:

| Field | Purpose |
|---|---|
| `corpus_id`, `corpus_version` | Versioned rebuild source |
| `document_id` | Stable control-language document identity |
| `framework`, `framework_version` | Regulatory framework scope |
| `control_id` | Native control identifier |
| `chunk_id`, `chunk_index` | Deterministic chunk identity and order |
| `language` | Lower-case BCP 47 language tag |
| `title`, `text`, `text_sha256` | Retrieval content and integrity metadata |
| `embedding_model` | Prevents querying vectors from a different model space |
| `source_uri` | Optional public/source reference |
| `annex_a_control_id`, `control_theme` | Shared `5.1`–`8.34` identity, Annex A alias, and theme |
| `source_standard`, `source_edition`, `national_edition` | ISO/base and DIN source identity |
| `section_kind`, `source_page_start`, `source_page_end` | Section-safe retrieval and page provenance |
| `retrieval_role` | `semantic` for ISO 27002 or `reference` for ISO 27001 |
| `parser_version`, `source_sha256` | Reproducible parser/source provenance |

All fields used by filters have Qdrant payload indexes. An existing collection with a different dense dimension, distance, or missing sparse slot fails closed and must be rebuilt from the corpus.

## ISO corpus preparation and ingestion

The server-only PDF.js CLI accepts explicit ISO 27001 and ISO 27002 paths and writes corpus-v2 JSON below the ignored `.assessor-data/regulatory-corpora/` directory:

```text
npm run rag:prepare-iso -- "ISO27001 2024.PDF" "ISO27002 2024.PDF"
npm run rag:ingest -- .assessor-data/regulatory-corpora/iso-27001-27002-de.json
```

The checked-in [`tests/fixtures/regulatory-corpus.synthetic.json`](../tests/fixtures/regulatory-corpus.synthetic.json) is an unlicensed synthetic EN/DE smoke corpus. It exists to verify rebuilding and retrieval mechanics only; it is not a regulatory source and must never be presented as compliance guidance.

Configure `.env.postgres` from the commented variables in `.env.postgres.example`, then run:

```text
npm run db:up
npm run rag:ingest -- path/to/regulatory-corpus.json
```

The parser accepts encrypted documents only when no password is required and copying is permitted. It rejects scanned/empty, oversized, copy-prohibited, structurally unexpected, incomplete, duplicate, or mismatched catalogs. The build must contain the matching 93 controls with theme counts 37/8/14/34. It retains hashes, editions, control/section metadata, and page provenance without committing licensed text. Ingestion splits only inside a control section, embeds all chunks in bounded batches, and then replaces indexed chunks per `(corpusId, documentId)`. If Qdrant fails after deletion, rerun ingestion from the unchanged corpus JSON.

## Configuration and trust

Qdrant 1.19.1 runs in Docker with a persistent named volume and outbound usage telemetry disabled. Local REST access is loopback-only; deployment access stays on the Compose network. An API key is optional for an unsecured loopback/internal self-host and required for remote hosts. Remote plain HTTP Qdrant URLs are rejected.

The embedding adapter uses an OpenAI-compatible `/v1/embeddings` endpoint. Remote endpoints must use HTTPS. Plain HTTP is accepted only for `localhost`, `127.0.0.1`, or `::1`, enabling a same-machine llama.cpp development server without weakening remote transport. Configuration must explicitly provide the model name and its exact native/requested dimensions. Redirects are rejected, response size and time are bounded, and API keys remain in backend environment variables.

### Recommended local development model

Use BGE-M3 for the first local dense baseline. Its published model contract is multilingual, 1,024-dimensional, and supports up to 8,192 tokens. Those dimensions match the current Qdrant collection. llama.cpp supports dedicated embedding mode, model/default or explicit pooling, bearer API keys, and the OpenAI-compatible `/v1/embeddings` endpoint.

Example command, using a locally obtained compatible BGE-M3 GGUF file:

```text
llama-server.exe -m path/to/bge-m3.gguf --embedding --pooling cls --host 127.0.0.1 --port 8080 --api-key <local-secret> -c 8192
```

Use matching ASSESSOR settings:

```text
ASSESSOR_RAG_EMBEDDING_BASE_URL=http://127.0.0.1:8080/v1
ASSESSOR_RAG_EMBEDDING_MODEL=bge-m3
ASSESSOR_RAG_EMBEDDING_API_KEY=<same-local-secret>
ASSESSOR_RAG_EMBEDDING_DIMENSIONS=1024
ASSESSOR_RAG_EMBEDDING_REQUEST_DIMENSIONS=false
ASSESSOR_RAG_ACTIVE_CORPUS_ID=iso-27001-27002-de
ASSESSOR_RAG_ACTIVE_CORPUS_VERSION=<generated-corpus-version>
```

For a later LiteLLM comparison with `text-embedding-3-large`, use the approved HTTPS gateway and set `ASSESSOR_RAG_EMBEDDING_REQUEST_DIMENSIONS=true` if requesting 1,024-dimensional output. OpenAI supports the `dimensions` parameter for `text-embedding-3` models; llama.cpp currently returns the model's native dimension and should leave this flag false. Changing model identity or dimensions requires rebuilding the derived Qdrant corpus before comparing results.

Embedding is an external processing boundary. Regulatory text is sent during ingestion. The finding retrieval service sends only the privacy-safe check query described below; the result table, paths, tenant identities, reviews, raw evidence, and local provenance IDs remain outside the embedding and completion requests.

## Finding query privacy boundary

`buildRegulatoryFindingQuery` accepts the existing `UnifiedResultProjection` and builds the query from the **shared privacy-safe finding projection** (`src/domain/privacy-safe-finding.ts`, `privacySafeFinding`) — the identical boundary used by the AI overview:

- the rule-level **check text** — title, plus description and recommendation when present — which comes from the deduplicated finding *definition* (canonical key `tool:ruleId`) and describes the check, not the tenant; email addresses are redacted;
- a catalog Maester/PingCastle rule ID, a fixed public scanner name, and recognized public technical topics, all as optional query **enrichment**.

The query never reads the **result table / occurrence data**: affected asset, source locator/path, per-object values, project/tenant identity, review owner, review text, original values, or provenance IDs. Because the query is built from the rule-level check text, **every scanner finding with a check title maps**; only non-scanner records and findings without a check title fail closed as `insufficient-public-context` and do not call the retriever. The local `sourceResultId` is returned only for in-process correlation and is not included in the embedding request.

GPO baseline comparisons (2026-10-02, explicit user request): `policy-evaluation` rows map the same way as scanner findings. The query uses the setting title, guidance text and recommendation after the Result DB disclosure projection, so the configured customer value, its principals and source counts never enter retrieval or the mapper prompt. The public tool label is `Microsoft Group Policy`; there is no rule catalog for GPO settings, so the specificity stays `topic`. The global rule key is `GPOs:<technical setting ID>`: every baseline package comparison of one setting, in every project, shares one mapping and its decisions. In Work the rolled-up setting names one of its current stored comparisons; a historical comparison offers no mapping until it is re-evaluated. Raw GPO observations and environment observations remain excluded context rows. `REGULATORY_RETRIEVAL_VERSION` is unchanged by explicit user decision, because scanner queries are identical and existing scanner mappings stay reusable.

This is a technical minimization boundary. Activation still depends on the approved embedding/LiteLLM destinations, catalog coverage, retention terms, and the precise finding classes allowed to cross that boundary. Residual note: `privacySafeFinding` redacts emails only; it relies on definitions being rule-generic, so a scanner that folds an identifier (host, IP, path) into a *definition* title/description would carry it into the query. Strengthening that redaction is a shared decision affecting both AI features (see implementation plan).

## One-finding mapping workflow

The Work finding reader loads mapping state only after `Map ISO controls` is opened. Mappings are **global**: the finding contributes only its non-personal rule identity and check context, and the mapping is keyed by that identity plus the pipeline versions, so one rule is mapped once and every occurrence in every project reuses it. The API returns HTTP 202 and the UI polls while queued/running. The service retrieves 32 eligible ISO 27002 chunks, groups by maximum dense score, keeps eight controls, attaches up to two semantic passages and the matching ISO 27001 statement, and bounds the LiteLLM payload to 48,000 characters.

The completion schema permits at most five `primary` or `supporting` proposals, advisory low/medium/high confidence, a bounded rationale, grounded evidence IDs, or an explicit `no_relevant_control`/`insufficient_context` outcome. Unknown controls, foreign evidence IDs, duplicates, malformed JSON, and oversized content fail closed. A reviewer can accept or reject any grounded top-eight candidate with an optional comment; each change appends a **global** decision event keyed by rule identity and control, using optimistic concurrency, shared across every project. Acceptance means relevant control, never compliance.

Routes:

- `GET /api/v1/projects/{projectId}/regulatory-mappings/{resultProjectionId}`
- `POST /api/v1/projects/{projectId}/regulatory-mapping-runs`
- `POST /api/v1/projects/{projectId}/regulatory-mapping-decisions`
- `GET /api/v1/projects/{projectId}/regulatory-mapping-batches`
- `POST /api/v1/projects/{projectId}/regulatory-mapping-batches`

On the consolidated migration line (2026-10-02), the single migration 026 `regulatory-mapping` creates the global tables directly, together with the project-scoped `assessor_regulatory_mapping_batches`; the superseded occurrence-bound tables are never created. The remainder of this paragraph records the original history on the former experimental line, where migration 025 originally added occurrence-bound, project-scoped run and decision tables and migration 026 superseded it: it dropped those tables and created the global `assessor_regulatory_mappings` (keyed by `rule_key` + pipeline versions, `UNIQUE(rule_key, corpus_version, retrieval_version, prompt_version, model)`) and a global, rule-keyed `assessor_regulatory_mapping_decisions`; neither carries `project_id` or occurrence identity, and both survive project/finding deletion. Before 026 was applied, the 025 tables held only regenerable local test data (19 runs, 10 decisions); a verified backup of both tables was taken at `backups/regulatory_mapping_025_backup_20260915-155410.sql` and the data was then intentionally discarded per the user's decision. A live development backend applied 026 through its own migration runner (checksum recorded and re-verified against the source file); no reimport is required because mapping runs are regenerable and no scanner data changed.

## Filtered Result DB batch workflow

`Map all ISO controls` sits beside Refresh and Export JSON in Result DB. Both actions derive their scope from the same normalized `UnifiedResultQuery`: the selected assessment and scan-run IDs are the outer timeslot boundary, and every active Result DB filter narrows that scope. This includes deviation, source, severity, outcome, result type, processing state, review state, categories, and supported text/date filters. The default `deviation: true` filter therefore maps deviations only until the user changes or resets it. Offset, limit, client project IDs, and client audience values are discarded. Display pagination, Load more, and the number of mounted rows never change batch membership.

The server enumerates the complete query through stable pagination and fails the batch if the filtered total changes during enumeration. It counts raw GPO observations and environment observations as excluded context rows, scanner findings without usable public check context as unsupported, and deduplicates eligible occurrences by global `ruleKey`. One deterministic representative is selected per rule. Current successful or active mappings are reused; missing mappings and previously failed mappings are atomically claimed and queued. Single-finding and batch work share the same queue with at most two concurrent retrieval/model pipelines, so simultaneous requests cannot create duplicate model calls for a deterministic mapping identity. A completed batch may still report individual rule failures.

The batch table (part of migration 026 on the consolidated line, originally migration 027) holds only project-scoped batch orchestration records: the normalized timeslot/filter snapshot, canonical filter hash, idempotency key, state, aggregate counters, timestamps, and bounded failure details. Global individual mapping rows remain authoritative; there is no batch-item table. Batch records are exported with their project and removed with project deletion. Active mappings and batches are marked failed after an interrupted server process; a later batch reuses successful work and retries only missing or failed rules. The migration is additive and does not change scanner processing, so no processing-manifest bump or reimport is required. After 027 had been applied, a later ordering-only edit incorrectly changed that migration and the checksum guard stopped backend startup. The file has been restored to its applied SHA-256 state and locked by regression; no follow-up schema migration is required.

The filter snapshot is captured when Start is pressed and is never copied into embedding or LiteLLM payloads. Result DB reloads the latest batch for the currently selected normalized scope, so changing filters does not relabel an active run. Batches generate proposals only. Accept/Reject remains the existing explicit human decision per global rule/control in Work; there is no automatic acceptance or batch decision queue.

### Local Qdrant integration check

`tests/server/qdrant-regulatory-index.integration.test.ts` is opt-in and uses deterministic synthetic vectors. With `ASSESSOR_QDRANT_INTEGRATION=1` and the normal Qdrant/model identity environment variables, it verifies the real collection, payload indexes, document replacement, filters, and retrieval without calling an embedding provider. The standard suite skips it so tests do not unexpectedly mutate a developer's derived index.

On 2026-09-15 this check passed against Qdrant 1.19.1 in WSL: the collection was green, Dense used 1,024-dimensional cosine vectors, the Sparse slot existed, all 12 filter payload indexes existed, and one synthetic integration point was retrieved and then removed.

## Dense retrieval evaluation

The evaluation CLI runs the configured dense retriever against a bounded, versioned labeled case file and reports control-level HitRate@K, Recall@K, and mean reciprocal rank. Queries can declare a language different from the expected control language so English-to-German and German-to-English cases remain explicit.

```json
{
  "schemaVersion": 1,
  "cases": [
    {
      "id": "synthetic-en-to-de-mfa",
      "query": "Require multifactor authentication",
      "queryLanguage": "en",
      "filters": { "corpusId": "synthetic-framework" },
      "expectedControls": [
        {
          "framework": "Synthetic Framework",
          "frameworkVersion": "1",
          "controlId": "CTRL-MFA",
          "language": "de"
        }
      ]
    }
  ]
}
```

Run it after ingestion with:

```text
npm run rag:evaluate -- path/to/evaluation-cases.json 10
```

The CLI prints aggregate metrics only; it does not print queries or retrieved control text. Unit tests verify the metric contract with synthetic rankings, but those values are not evidence of real embedding quality. A meaningful baseline requires an approved representative corpus, independently labeled cases, and the configured multilingual model running against Qdrant.

The checked-in [`tests/fixtures/regulatory-evaluation.synthetic.json`](../tests/fixtures/regulatory-evaluation.synthetic.json) provides three explicitly cross-language smoke cases for the synthetic corpus.

Live smoke result on 2026-09-15 with local llama.cpp build 10472, BGE-M3 FP16, and Qdrant 1.19.1: six EN/DE documents produced six 1,024-dimensional chunks. All three cross-language cases returned their expected control at rank 1, giving HitRate@3 `1.0`, Recall@3 `1.0`, and MRR `1.0`. This verifies wiring and basic language crossing only; three controls are far too small and distinct to estimate production retrieval quality.

## Extension seams

- `RegulatoryCandidateRetriever` is the replaceable candidate stage. The current implementation is dense Qdrant retrieval; a later hybrid implementation can combine dense and sparse/BM25 candidates here.
- `RegulatoryReranker` is optional and receives only the bounded candidate set. Without one, dense ordering is retained.
- `RegulatoryRetriever` is the consumer-facing contract. Mapping code should depend on it, not on Qdrant.

The filtered manual Result DB batch was activated by explicit user decision before the earlier quality-gate sequence. That decision changes orchestration only: broaden public-signal coverage through reviewed catalogs, establish a representative independently labeled EN/DE baseline, and evaluate sparse/hybrid retrieval, reranking, and confidence calibration before claiming production retrieval quality or adding automatic decisions. Single and batch proposals remain human-reviewed.

## Reference basis

- Qdrant collections and named vectors: https://qdrant.tech/documentation/manage-data/collections/
- Qdrant payload indexing and filtering: https://qdrant.tech/documentation/manage-data/indexing/
- Qdrant hybrid and multi-stage Query API: https://qdrant.tech/documentation/search/hybrid-queries/
- Self-hosted Qdrant security: https://qdrant.tech/documentation/tutorials-operations/secure-qdrant/
- LiteLLM OpenAI-compatible endpoints: https://docs.litellm.ai/
- llama.cpp server embeddings and authentication: https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md
- BGE-M3 model contract: https://huggingface.co/BAAI/bge-m3
- OpenAI embedding dimensions parameter: https://developers.openai.com/api/reference/resources/embeddings/methods/create