# Regulatory-control RAG with Qdrant

Status: Phase 13D one-finding vertical slice. The server prepares the licensed German ISO 27001/27002 corpus, indexes section-safe ISO 27002 semantic chunks plus ISO 27001 references, retrieves and groups candidates, requests a grounded LiteLLM proposal, persists runs and reviewer decisions, and exposes the workflow inline in Work. Batch mapping, sparse retrieval, reranking, and confidence calibration remain out of scope.

## Boundaries

- PostgreSQL remains the only system of record for ASSESSOR projects, assessments, scans, findings, results, reviews, mapping runs, and mapping decisions.
- Qdrant is a derived, rebuildable search index. Losing or deleting its volume must not lose an ASSESSOR scan, finding, result, or review.
- The versioned regulatory corpus JSON is the rebuild source for Qdrant. It must be retained and governed separately from the index.
- React does not connect to Qdrant or the embedding provider. Both are backend-only boundaries.
- Regulatory retrieval produces candidates, not compliance decisions. The constrained mapping stage retains retrieved chunk IDs, scores, corpus/source hashes, model, prompt/schema versions, and append-only reviewer decisions in PostgreSQL.

```text
versioned regulatory corpus JSON
        | validate and chunk
        v
approved multilingual embedding service
        | dense vectors
        v
Qdrant assessor_regulatory_chunks_v2
        ^
scanner finding -> public-signal query builder -> same embedding model -> dense candidates
        -> group top 32 chunks into top 8 controls
        -> optional future hybrid fusion -> optional reranker
        -> constrained mapper -> human decision -> PostgreSQL audit record
```

## Collection contract

The active collection is `assessor_regulatory_chunks_v2`. It has two named vector slots:

- `dense`: the configured embedding dimension with cosine distance.
- `sparse`: reserved now so a later sparse/BM25-style encoder can be added without changing the retrieval abstractions.

Qdrant payloads use schema version 2 and contain:

| Field | Purpose |
|---|---|
| `corpus_id`, `corpus_version` | Versioned rebuild source |
| `document_id` | Stable control-language document identity |
| `framework`, `framework_version` | Regulatory framework scope |
| `control_id` | Native control identifier |
| `chunk_id`, `chunk_index` | Deterministic chunk identity and order |
| `language` | Lower-case BCP 47 language tag |
| `title`, `text`, `text_sha256` | Retrieval content and integrity metadata |
| `embedding_model` | Prevents querying vectors from a different model space |
| `source_uri` | Optional public/source reference |
| `annex_a_control_id`, `control_theme` | Shared `5.1`–`8.34` identity, Annex A alias, and theme |
| `source_standard`, `source_edition`, `national_edition` | ISO/base and DIN source identity |
| `section_kind`, `source_page_start`, `source_page_end` | Section-safe retrieval and page provenance |
| `retrieval_role` | `semantic` for ISO 27002 or `reference` for ISO 27001 |
| `parser_version`, `source_sha256` | Reproducible parser/source provenance |

All fields used by filters have Qdrant payload indexes. An existing collection with a different dense dimension, distance, or missing sparse slot fails closed and must be rebuilt from the corpus.

## ISO corpus preparation and ingestion

The server-only PDF.js CLI accepts explicit ISO 27001 and ISO 27002 paths and writes corpus-v2 JSON below the ignored `.assessor-data/regulatory-corpora/` directory:

```text
npm run rag:prepare-iso -- "ISO27001 2024.PDF" "ISO27002 2024.PDF"
npm run rag:ingest -- .assessor-data/regulatory-corpora/iso-27001-27002-de.json
```

The checked-in [`tests/fixtures/regulatory-corpus.synthetic.json`](../tests/fixtures/regulatory-corpus.synthetic.json) is an unlicensed synthetic EN/DE smoke corpus. It exists to verify rebuilding and retrieval mechanics only; it is not a regulatory source and must never be presented as compliance guidance.

Configure `.env.postgres` from the commented variables in `.env.postgres.example`, then run:

```text
npm run db:up
npm run rag:ingest -- path/to/regulatory-corpus.json
```

The parser accepts encrypted documents only when no password is required and copying is permitted. It rejects scanned/empty, oversized, copy-prohibited, structurally unexpected, incomplete, duplicate, or mismatched catalogs. The build must contain the matching 93 controls with theme counts 37/8/14/34. It retains hashes, editions, control/section metadata, and page provenance without committing licensed text. Ingestion splits only inside a control section, embeds all chunks in bounded batches, and then replaces indexed chunks per `(corpusId, documentId)`. If Qdrant fails after deletion, rerun ingestion from the unchanged corpus JSON.

## Configuration and trust

Qdrant 1.19.1 runs in Docker with a persistent named volume and outbound usage telemetry disabled. Local REST access is loopback-only; deployment access stays on the Compose network. An API key is optional for an unsecured loopback/internal self-host and required for remote hosts. Remote plain HTTP Qdrant URLs are rejected.

The embedding adapter uses an OpenAI-compatible `/v1/embeddings` endpoint. Remote endpoints must use HTTPS. Plain HTTP is accepted only for `localhost`, `127.0.0.1`, or `::1`, enabling a same-machine llama.cpp development server without weakening remote transport. Configuration must explicitly provide the model name and its exact native/requested dimensions. Redirects are rejected, response size and time are bounded, and API keys remain in backend environment variables.

### Recommended local development model

Use BGE-M3 for the first local dense baseline. Its published model contract is multilingual, 1,024-dimensional, and supports up to 8,192 tokens. Those dimensions match the current Qdrant collection. llama.cpp supports dedicated embedding mode, model/default or explicit pooling, bearer API keys, and the OpenAI-compatible `/v1/embeddings` endpoint.

Example command, using a locally obtained compatible BGE-M3 GGUF file:

```text
llama-server.exe -m path/to/bge-m3.gguf --embedding --pooling cls --host 127.0.0.1 --port 8080 --api-key <local-secret> -c 8192
```

Use matching ASSESSOR settings:

```text
ASSESSOR_RAG_EMBEDDING_BASE_URL=http://127.0.0.1:8080/v1
ASSESSOR_RAG_EMBEDDING_MODEL=bge-m3
ASSESSOR_RAG_EMBEDDING_API_KEY=<same-local-secret>
ASSESSOR_RAG_EMBEDDING_DIMENSIONS=1024
ASSESSOR_RAG_EMBEDDING_REQUEST_DIMENSIONS=false
ASSESSOR_RAG_ACTIVE_CORPUS_ID=iso-27001-27002-de
ASSESSOR_RAG_ACTIVE_CORPUS_VERSION=<generated-corpus-version>
```

For a later LiteLLM comparison with `text-embedding-3-large`, use the approved HTTPS gateway and set `ASSESSOR_RAG_EMBEDDING_REQUEST_DIMENSIONS=true` if requesting 1,024-dimensional output. OpenAI supports the `dimensions` parameter for `text-embedding-3` models; llama.cpp currently returns the model's native dimension and should leave this flag false. Changing model identity or dimensions requires rebuilding the derived Qdrant corpus before comparing results.

Embedding is an external processing boundary. Regulatory text is sent during ingestion. The finding retrieval service sends only the privacy-safe check query described below; the result table, paths, tenant identities, reviews, raw evidence, and local provenance IDs remain outside the embedding and completion requests.

## Finding query privacy boundary

`buildRegulatoryFindingQuery` accepts the existing `UnifiedResultProjection` and builds the query from the **shared privacy-safe finding projection** (`src/domain/privacy-safe-finding.ts`, `privacySafeFinding`) — the identical boundary used by the AI overview:

- the rule-level **check text** — title, plus description and recommendation when present — which comes from the deduplicated finding *definition* (canonical key `tool:ruleId`) and describes the check, not the tenant; email addresses are redacted;
- a catalog Maester/PingCastle rule ID, a fixed public scanner name, and recognized public technical topics, all as optional query **enrichment**.

The query never reads the **result table / occurrence data**: affected asset, source locator/path, per-object values, project/tenant identity, review owner, review text, original values, or provenance IDs. Because the query is built from the rule-level check text, **every scanner finding with a check title maps**; only non-scanner records and findings without a check title fail closed as `insufficient-public-context` and do not call the retriever. The local `sourceResultId` is returned only for in-process correlation and is not included in the embedding request.

This is a technical minimization boundary. Activation still depends on the approved embedding/LiteLLM destinations, catalog coverage, retention terms, and the precise finding classes allowed to cross that boundary. Residual note: `privacySafeFinding` redacts emails only; it relies on definitions being rule-generic, so a scanner that folds an identifier (host, IP, path) into a *definition* title/description would carry it into the query. Strengthening that redaction is a shared decision affecting both AI features (see implementation plan).

## One-finding mapping workflow

The Work finding reader loads mapping state only after `Map ISO controls` is opened. Mappings are **global**: the finding contributes only its non-personal rule identity and check context, and the mapping is keyed by that identity plus the pipeline versions, so one rule is mapped once and every occurrence in every project reuses it. The API returns HTTP 202 and the UI polls while queued/running. The service retrieves 32 eligible ISO 27002 chunks, groups by maximum dense score, keeps eight controls, attaches up to two semantic passages and the matching ISO 27001 statement, and bounds the LiteLLM payload to 48,000 characters.

The completion schema permits at most five `primary` or `supporting` proposals, advisory low/medium/high confidence, a bounded rationale, grounded evidence IDs, or an explicit `no_relevant_control`/`insufficient_context` outcome. Unknown controls, foreign evidence IDs, duplicates, malformed JSON, and oversized content fail closed. A reviewer can accept or reject any grounded top-eight candidate with an optional comment; each change appends a **global** decision event keyed by rule identity and control, using optimistic concurrency, shared across every project. Acceptance means relevant control, never compliance.

Routes:

- `GET /api/v1/projects/{projectId}/regulatory-mappings/{resultProjectionId}`
- `POST /api/v1/projects/{projectId}/regulatory-mapping-runs`
- `POST /api/v1/projects/{projectId}/regulatory-mapping-decisions`

Migration 025 originally added occurrence-bound, project-scoped run and decision tables. Migration 026 supersedes it: it drops those tables and creates the global `assessor_regulatory_mappings` (keyed by `rule_key` + pipeline versions, `UNIQUE(rule_key, corpus_version, retrieval_version, prompt_version, model)`) and a global, rule-keyed `assessor_regulatory_mapping_decisions`; neither carries `project_id` or occurrence identity, and both survive project/finding deletion. Before 026 was applied, the 025 tables held only regenerable local test data (19 runs, 10 decisions); a verified backup of both tables was taken at `backups/regulatory_mapping_025_backup_20260915-155410.sql` and the data was then intentionally discarded per the user's decision. A live development backend applied 026 through its own migration runner (checksum recorded and re-verified against the source file); no reimport is required because mapping runs are regenerable and no scanner data changed.

### Local Qdrant integration check

`tests/server/qdrant-regulatory-index.integration.test.ts` is opt-in and uses deterministic synthetic vectors. With `ASSESSOR_QDRANT_INTEGRATION=1` and the normal Qdrant/model identity environment variables, it verifies the real collection, payload indexes, document replacement, filters, and retrieval without calling an embedding provider. The standard suite skips it so tests do not unexpectedly mutate a developer's derived index.

On 2026-09-15 this check passed against Qdrant 1.19.1 in WSL: the collection was green, Dense used 1,024-dimensional cosine vectors, the Sparse slot existed, all 12 filter payload indexes existed, and one synthetic integration point was retrieved and then removed.

## Dense retrieval evaluation

The evaluation CLI runs the configured dense retriever against a bounded, versioned labeled case file and reports control-level HitRate@K, Recall@K, and mean reciprocal rank. Queries can declare a language different from the expected control language so English-to-German and German-to-English cases remain explicit.

```json
{
  "schemaVersion": 1,
  "cases": [
    {
      "id": "synthetic-en-to-de-mfa",
      "query": "Require multifactor authentication",
      "queryLanguage": "en",
      "filters": { "corpusId": "synthetic-framework" },
      "expectedControls": [
        {
          "framework": "Synthetic Framework",
          "frameworkVersion": "1",
          "controlId": "CTRL-MFA",
          "language": "de"
        }
      ]
    }
  ]
}
```

Run it after ingestion with:

```text
npm run rag:evaluate -- path/to/evaluation-cases.json 10
```

The CLI prints aggregate metrics only; it does not print queries or retrieved control text. Unit tests verify the metric contract with synthetic rankings, but those values are not evidence of real embedding quality. A meaningful baseline requires an approved representative corpus, independently labeled cases, and the configured multilingual model running against Qdrant.

The checked-in [`tests/fixtures/regulatory-evaluation.synthetic.json`](../tests/fixtures/regulatory-evaluation.synthetic.json) provides three explicitly cross-language smoke cases for the synthetic corpus.

Live smoke result on 2026-09-15 with local llama.cpp build 10472, BGE-M3 FP16, and Qdrant 1.19.1: six EN/DE documents produced six 1,024-dimensional chunks. All three cross-language cases returned their expected control at rank 1, giving HitRate@3 `1.0`, Recall@3 `1.0`, and MRR `1.0`. This verifies wiring and basic language crossing only; three controls are far too small and distinct to estimate production retrieval quality.

## Extension seams

- `RegulatoryCandidateRetriever` is the replaceable candidate stage. The current implementation is dense Qdrant retrieval; a later hybrid implementation can combine dense and sparse/BM25 candidates here.
- `RegulatoryReranker` is optional and receives only the bounded candidate set. Without one, dense ordering is retained.
- `RegulatoryRetriever` is the consumer-facing contract. Mapping code should depend on it, not on Qdrant.

Before assessment-wide batch mapping, broaden public-signal coverage only through reviewed catalogs, establish a representative independently labeled EN/DE baseline, and evaluate sparse/hybrid retrieval, reranking, and confidence calibration. The one-finding workflow must remain human-reviewed.

## Reference basis

- Qdrant collections and named vectors: https://qdrant.tech/documentation/manage-data/collections/
- Qdrant payload indexing and filtering: https://qdrant.tech/documentation/manage-data/indexing/
- Qdrant hybrid and multi-stage Query API: https://qdrant.tech/documentation/search/hybrid-queries/
- Self-hosted Qdrant security: https://qdrant.tech/documentation/tutorials-operations/secure-qdrant/
- LiteLLM OpenAI-compatible endpoints: https://docs.litellm.ai/
- llama.cpp server embeddings and authentication: https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md
- BGE-M3 model contract: https://huggingface.co/BAAI/bge-m3
- OpenAI embedding dimensions parameter: https://developers.openai.com/api/reference/resources/embeddings/methods/create