# Regulatory-control RAG with Qdrant

Status: Phase 13D one-finding and filtered Result DB batch vertical slice. The server prepares the licensed German ISO 27001/27002 corpus, indexes section-safe ISO 27002 semantic chunks plus ISO 27001 references, derives problem and remediation searches only from privacy-safe finding text, combines dense and hashed-BM25 sparse rankings at control level, requests a grounded assessment for every shortlisted control, deterministically selects proposals, persists runs and reviewer decisions, and exposes manual proposal generation in Work and Result DB. Learned/model-based reranking and confidence calibration remain out of scope.

## Boundaries

- PostgreSQL remains the only system of record for ASSESSOR projects, assessments, scans, findings, results, reviews, mapping runs, and mapping decisions.
- Qdrant is a derived, rebuildable search index. Losing or deleting its volume must not lose an ASSESSOR scan, finding, result, or review.
- The versioned regulatory corpus JSON is the rebuild source for Qdrant. It must be retained and governed separately from the index.
- React does not connect to Qdrant or the embedding provider. Both are backend-only boundaries.
- Regulatory retrieval produces candidates, not compliance decisions. The constrained mapping stage retains retrieved chunk IDs, scores, corpus/source hashes, model, prompt/schema versions, and append-only reviewer decisions in PostgreSQL.

```text
versioned regulatory corpus JSON
        | validate and chunk
        v
approved multilingual embedding service + local BM25 encoder
        | dense + sparse vectors
        v
Qdrant assessor_regulatory_chunks_v2
        ^
scanner finding -> privacy-safe title + description + recommendation
        -> problem query + optional remediation query + full-context section searches
        -> dense/sparse control-level RRF per request
        -> collapse control/purpose/guidance into one full-context control ranking
        -> three-family control-level RRF -> 12-control shortlist
        -> per-candidate assessment -> deterministic 0-1/0-3 selection
        -> human decision -> PostgreSQL audit record
```

## Collection contract

The active collection is `assessor_regulatory_chunks_v2`. It has two named vector slots:

- `dense`: the configured embedding dimension with cosine distance.
- `sparse`: deterministic hashed BM25 weights generated locally from control titles and text.

Qdrant payloads use schema version 2 and contain:

| Field | Purpose |
|---|---|
| `corpus_id`, `corpus_version` | Versioned rebuild source |
| `document_id` | Stable control-language document identity |
| `framework`, `framework_version` | Regulatory framework scope |
| `control_id` | Native control identifier |
| `chunk_id`, `chunk_index` | Deterministic chunk identity and order |
| `language` | Lower-case BCP 47 language tag |
| `title`, `text`, `text_sha256` | Retrieval content and integrity metadata |
| `embedding_model` | Prevents querying vectors from a different model space |
| `source_uri` | Optional public/source reference |
| `annex_a_control_id`, `control_theme` | Shared `5.1`–`8.34` identity, Annex A alias, and theme |
| `source_standard`, `source_edition`, `national_edition` | ISO/base and DIN source identity |
| `section_kind`, `source_page_start`, `source_page_end` | Section-safe retrieval and page provenance |
| `retrieval_role` | `semantic` for ISO 27002 or `reference` for ISO 27001 |
| `parser_version`, `source_sha256` | Reproducible parser/source provenance |

All fields used by filters have Qdrant payload indexes. An existing collection with a different dense dimension, distance, or missing sparse slot fails closed and must be rebuilt from the corpus.

## ISO corpus preparation and ingestion

The server-only PDF.js CLI accepts explicit ISO 27001 and ISO 27002 paths and writes corpus-v2 JSON below the ignored `.assessor-data/regulatory-corpora/` directory:

```text
npm run rag:prepare-iso -- "ISO27001 2024.PDF" "ISO27002 2024.PDF"
npm run rag:ingest -- .assessor-data/regulatory-corpora/iso-27001-27002-de.json
```

The checked-in [`tests/fixtures/regulatory-corpus.synthetic.json`](../tests/fixtures/regulatory-corpus.synthetic.json) is an unlicensed synthetic EN/DE smoke corpus. It exists to verify rebuilding and retrieval mechanics only; it is not a regulatory source and must never be presented as compliance guidance.

Configure `.env.postgres` from the commented variables in `.env.postgres.example`, then run:

```text
npm run db:up
npm run rag:ingest -- path/to/regulatory-corpus.json
```

The parser accepts encrypted documents only when no password is required and copying is permitted. It rejects scanned/empty, oversized, copy-prohibited, structurally unexpected, incomplete, duplicate, or mismatched catalogs. The build must contain the matching 93 controls with theme counts 37/8/14/34. It retains hashes, editions, control/section metadata, and page provenance without committing licensed text. Ingestion splits only inside a control section, embeds all chunks in bounded batches, derives corpus-wide hashed BM25 vectors without a maintained topic vocabulary, and then replaces indexed chunks per `(corpusId, documentId)`. Collections populated before retrieval 1.4 must be re-ingested once so their existing points receive sparse vectors; dense retrieval remains available until then. If Qdrant fails after deletion, rerun ingestion from the unchanged corpus JSON.

## Configuration and trust

Qdrant 1.19.1 runs in Docker with a persistent named volume and outbound usage telemetry disabled. Local REST access is loopback-only; deployment access stays on the Compose network. An API key is optional for an unsecured loopback/internal self-host and required for remote hosts. Remote plain HTTP Qdrant URLs are rejected.

The embedding adapter uses an OpenAI-compatible `/v1/embeddings` endpoint. Remote endpoints must use HTTPS. Plain HTTP is accepted only for `localhost`, `127.0.0.1`, or `::1`, enabling a same-machine llama.cpp development server without weakening remote transport. Configuration must explicitly provide the model name and its exact native/requested dimensions. Redirects are rejected, response size and time are bounded, and API keys remain in backend environment variables.

### Recommended local development model

Use BGE-M3 for the first local dense baseline. Its published model contract is multilingual, 1,024-dimensional, and supports up to 8,192 tokens. Those dimensions match the current Qdrant collection. llama.cpp supports dedicated embedding mode, model/default or explicit pooling, bearer API keys, and the OpenAI-compatible `/v1/embeddings` endpoint.

Example command, using a locally obtained compatible BGE-M3 GGUF file:

```text
llama-server.exe -m path/to/bge-m3.gguf --embedding --pooling cls --host 127.0.0.1 --port 8080 --api-key <local-secret> -c 8192
```

Use matching ASSESSOR settings:

```text
ASSESSOR_RAG_EMBEDDING_BASE_URL=http://127.0.0.1:8080/v1
ASSESSOR_RAG_EMBEDDING_MODEL=bge-m3
ASSESSOR_RAG_EMBEDDING_API_KEY=<same-local-secret>
ASSESSOR_RAG_EMBEDDING_DIMENSIONS=1024
ASSESSOR_RAG_EMBEDDING_REQUEST_DIMENSIONS=false
ASSESSOR_RAG_ACTIVE_CORPUS_ID=iso-27001-27002-de
ASSESSOR_RAG_ACTIVE_CORPUS_VERSION=<generated-corpus-version>
```

For a later LiteLLM comparison with `text-embedding-3-large`, use the approved HTTPS gateway and set `ASSESSOR_RAG_EMBEDDING_REQUEST_DIMENSIONS=true` if requesting 1,024-dimensional output. OpenAI supports the `dimensions` parameter for `text-embedding-3` models; llama.cpp currently returns the model's native dimension and should leave this flag false. Changing model identity or dimensions requires rebuilding the derived Qdrant corpus before comparing results.

Embedding is an external processing boundary. Regulatory text is sent during ingestion. The finding retrieval service sends only the privacy-safe check query described below; the result table, paths, tenant identities, reviews, raw evidence, and local provenance IDs remain outside the embedding and completion requests.

## Finding query privacy boundary

`buildRegulatoryFindingQuery` accepts the existing `UnifiedResultProjection` and builds related values from the **shared privacy-safe finding projection** (`src/domain/privacy-safe-finding.ts`, `privacySafeFinding`) — the identical boundary used by the AI overview:

- the rule-level **check text** — title, plus description and recommendation when present — which comes from the deduplicated finding *definition* (canonical key `tool:ruleId`) and describes the check, not the tenant; email addresses are redacted;
- a problem query containing the bounded title and description;
- an optional remediation query containing the same title and bounded recommendation;
- a full finding-text query used for section-specific candidate ranking and evidence retrieval.

Scanner name, source rule ID, rule catalogs and `public-security-topics.ts` do not enter regulatory retrieval or mapper context. The topic list remains a separate concern of the public web-search feature. Consequently, identical privacy-safe finding text produces identical regulatory queries regardless of whether the scanner is known, newly added or private.

The query never reads the **result table / occurrence data**: affected asset, source locator/path, per-object values, project/tenant identity, review owner, review text, original values, or provenance IDs. Because the query is built from the rule-level check text, **every scanner finding with a check title maps**; only non-scanner records and findings without a check title fail closed as `insufficient-public-context` and do not call the retriever. The local `sourceResultId` is returned only for in-process correlation and is not included in the embedding request.

This is a technical minimization boundary. Activation still depends on the approved embedding/LiteLLM destinations, catalog coverage, retention terms, and the precise finding classes allowed to cross that boundary. Residual note: `privacySafeFinding` redacts emails only; it relies on definitions being rule-generic, so a scanner that folds an identifier (host, IP, path) into a *definition* title/description would carry it into the query. Strengthening that redaction is a shared decision affecting both AI features (see implementation plan).

## One-finding mapping workflow

The Work finding reader loads mapping state only after `Map ISO controls` is opened. Mappings are **global**: the finding contributes only its non-personal rule identity and check context, and the mapping is keyed by that identity plus the pipeline versions, so one rule is mapped once and every occurrence in every project reuses it. The API returns HTTP 202 and the UI polls while queued/running.

Each problem/remediation request obtains independent dense and BM25 sparse rankings and combines them by reciprocal-rank fusion (`k=60`). Fusion is keyed by control, not chunk: a control contributes at most once to each ranking and multiple chunks cannot crowd out other controls. Separate `control`, `purpose`, and `guidance` searches use the complete finding text; their control rankings are first collapsed into one full-context family ranking. The problem, optional remediation and full-context family rankings are then fused at control level to choose at most twelve controls. Thus section coverage can improve the order within the full-context family, but the three correlated sections contribute at most one final vote per control. The mapper receives the available semantic passages plus the matching ISO 27001 statement, while the LiteLLM payload remains bounded to 48,000 characters. Rank-derived values remain deterministic ordering signals; the reviewer UI shows ordinal retrieval rank because a value such as `0.03` is neither a percentage nor calibrated confidence. No learned or model-based reranker runs in this pipeline.

The model must assess every shortlisted control exactly once as `primary`, `supporting`, or `irrelevant`, with advisory low/medium/high confidence, a bounded rationale, and grounded evidence IDs. Missing controls, duplicates, unknown controls, foreign evidence IDs, malformed JSON, and oversized content fail closed. A deterministic reducer chooses the highest-confidence, earliest-ranked primary candidate, demotes any additional primary candidates into the supporting pool, retains at most three supporting candidates, permits supporting-only output, and derives `no_relevant_control` when every candidate is irrelevant. The persisted proposal schema and reviewer UI remain unchanged. A reviewer can accept or reject any grounded shortlisted candidate with an optional comment; each change appends a **global** decision event keyed by rule identity and control, using optimistic concurrency, shared across every project. Acceptance means relevant control, never compliance.

Routes:

- `GET /api/v1/projects/{projectId}/regulatory-mappings/{resultProjectionId}`
- `POST /api/v1/projects/{projectId}/regulatory-mapping-runs`
- `POST /api/v1/projects/{projectId}/regulatory-mapping-decisions`
- `GET /api/v1/projects/{projectId}/regulatory-mapping-batches`
- `POST /api/v1/projects/{projectId}/regulatory-mapping-batches`

Migration 025 originally added occurrence-bound, project-scoped run and decision tables. Migration 026 supersedes it: it drops those tables and creates the global `assessor_regulatory_mappings` (keyed by `rule_key` + pipeline versions, `UNIQUE(rule_key, corpus_version, retrieval_version, prompt_version, model)`) and a global, rule-keyed `assessor_regulatory_mapping_decisions`; neither carries `project_id` or occurrence identity, and both survive project/finding deletion. Before 026 was applied, the 025 tables held only regenerable local test data (19 runs, 10 decisions); a verified backup of both tables was taken at `backups/regulatory_mapping_025_backup_20260915-155410.sql` and the data was then intentionally discarded per the user's decision. A live development backend applied 026 through its own migration runner (checksum recorded and re-verified against the source file); no reimport is required because mapping runs are regenerable and no scanner data changed.

## Filtered Result DB batch workflow

`Map all ISO controls` sits beside Refresh and Export JSON in Result DB. Both actions derive their scope from the same normalized `UnifiedResultQuery`: the selected assessment and scan-run IDs are the outer timeslot boundary, and every active Result DB filter narrows that scope. This includes deviation, source, severity, outcome, result type, processing state, review state, categories, and supported text/date filters. The default `deviation: true` filter therefore maps deviations only until the user changes or resets it. Offset, limit, client project IDs, and client audience values are discarded. Display pagination, Load more, and the number of mounted rows never change batch membership.

The server enumerates the complete query through stable pagination and fails the batch if the filtered total changes during enumeration. It counts non-scanner results as excluded, scanner findings without usable public check context as unsupported, and deduplicates eligible occurrences by global `ruleKey`. One deterministic representative is selected per rule. Current successful or active mappings are reused; missing mappings and previously failed mappings are atomically claimed and queued. Single-finding and batch work share the same queue with at most two concurrent retrieval/model pipelines, so simultaneous requests cannot create duplicate model calls for a deterministic mapping identity. A completed batch may still report individual rule failures.

Migration 027 adds only project-scoped batch orchestration records: the normalized timeslot/filter snapshot, canonical filter hash, idempotency key, state, aggregate counters, timestamps, and bounded failure details. Global individual mapping rows remain authoritative; there is no batch-item table. Batch records are exported with their project and removed with project deletion. Active mappings and batches are marked failed after an interrupted server process; a later batch reuses successful work and retries only missing or failed rules. The migration is additive and does not change scanner processing, so no processing-manifest bump or reimport is required. After 027 had been applied, a later ordering-only edit incorrectly changed that migration and the checksum guard stopped backend startup. The file has been restored to its applied SHA-256 state and locked by regression; no follow-up schema migration is required.

The filter snapshot is captured when Start is pressed and is never copied into embedding or LiteLLM payloads. Result DB reloads the latest batch for the currently selected normalized scope, so changing filters does not relabel an active run. Batches generate proposals only. Accept/Reject remains the existing explicit human decision per global rule/control in Work; there is no automatic acceptance or batch decision queue.

### Local Qdrant integration check

`tests/server/qdrant-regulatory-index.integration.test.ts` is opt-in and uses deterministic synthetic vectors. With `ASSESSOR_QDRANT_INTEGRATION=1` and the normal Qdrant/model identity environment variables, it verifies the real collection, payload indexes, document replacement, filters, and retrieval without calling an embedding provider. The standard suite skips it so tests do not unexpectedly mutate a developer's derived index.

On 2026-09-15 this check passed against Qdrant 1.19.1 in WSL: the collection was green, Dense used 1,024-dimensional cosine vectors, the Sparse slot existed, all 12 filter payload indexes existed, and one synthetic integration point was retrieved and then removed.

## Retrieval evaluation

The evaluation CLI runs the configured hybrid retriever against a bounded, versioned labeled case file and reports control-level HitRate@K, Recall@K, and mean reciprocal rank. Queries can declare a language different from the expected control language so English-to-German and German-to-English cases remain explicit.

```json
{
  "schemaVersion": 1,
  "cases": [
    {
      "id": "synthetic-en-to-de-mfa",
      "query": "Require multifactor authentication",
      "queryLanguage": "en",
      "filters": { "corpusId": "synthetic-framework" },
      "expectedControls": [
        {
          "framework": "Synthetic Framework",
          "frameworkVersion": "1",
          "controlId": "CTRL-MFA",
          "language": "de"
        }
      ]
    }
  ]
}
```

Run it after ingestion with:

```text
npm run rag:evaluate -- path/to/evaluation-cases.json 10
```

The CLI prints aggregate metrics only; it does not print queries or retrieved control text. Unit tests verify the metric contract with synthetic rankings, but those values are not evidence of real embedding quality. A meaningful baseline requires an approved representative corpus, independently labeled cases, and the configured multilingual model running against Qdrant.

The checked-in [`tests/fixtures/regulatory-evaluation.synthetic.json`](../tests/fixtures/regulatory-evaluation.synthetic.json) provides three explicitly cross-language smoke cases for the synthetic corpus.

Live smoke result on 2026-09-15 with local llama.cpp build 10472, BGE-M3 FP16, and Qdrant 1.19.1: six EN/DE documents produced six 1,024-dimensional chunks. All three cross-language cases returned their expected control at rank 1, giving HitRate@3 `1.0`, Recall@3 `1.0`, and MRR `1.0`. This verifies wiring and basic language crossing only; three controls are far too small and distinct to estimate production retrieval quality.

The 2026-09-17 regression review used all 37 PingCastle findings and the 78 primary/supporting control relationships independently recorded in the review workbook. Candidate Recall@12 was `61/78` for retrieval 1.1, fell to `54/78` for the topic-led section retrieval 1.2, and reached `68/78` (`87.2%`) with retrieval 1.3's five generic reservations plus seven supplemental RRF controls. The 1.3 split was `33/38` primary relationships (`86.8%`) and `35/40` supporting relationships (`87.5%`). This is close to the requested approximately 90% human-review target, but it measures candidate availability only, not final mapper precision or accepted mapping accuracy. It is also a same-tool regression set rather than a representative new-scanner evaluation, so the generic reservation and unknown-scanner contract are retained as architectural safeguards rather than claiming general quality from PingCastle alone.

### Repeatable manual comparison

After a complete mapping run, create the Excel comparison with:

```text
npm run rag:compare-mappings
```

The read-only command takes the active retrieval version from `REGULATORY_RETRIEVAL_VERSION`, selects the most recent prior version covering every baseline scanner, reads the canonical local PostgreSQL mapping history and writes one workbook below `outputs/regulatory-mapping/`. Use `--current-version`, `--previous-version`, `--baseline` or `--output` to override those defaults. It stops without creating a workbook when the active version does not contain every baseline rule.

The versioned baseline at `evaluation/regulatory-mapping/manual-baseline.json` contains only scanner rule IDs and independently reviewed primary/supporting control roles. It is evaluation data for the human review workflow, not a product mapping source or CI assertion. Update or replace it deliberately when the regulatory corpus or manual assessment changes. Permanent tests cover only generic comparison mechanics and never assert scanner-specific ISO control IDs.

## Extension seams

- `RegulatoryCandidateRetriever` is the replaceable candidate stage. The current implementation fuses dense Qdrant retrieval with locally encoded sparse/BM25 retrieval.
- `RegulatoryReranker` is optional and receives only the bounded candidate set. Without one, the fused deterministic ordering is retained.
- `RegulatoryRetriever` is the consumer-facing contract. Mapping code should depend on it, not on Qdrant.

The filtered manual Result DB batch was activated by explicit user decision before the earlier quality-gate sequence. That decision changes orchestration only: broaden public-signal coverage through reviewed catalogs, establish a representative independently labeled EN/DE baseline, and evaluate sparse/hybrid retrieval, reranking, and confidence calibration before claiming production retrieval quality or adding automatic decisions. Single and batch proposals remain human-reviewed.

## Reference basis

- Qdrant collections and named vectors: https://qdrant.tech/documentation/manage-data/collections/
- Qdrant payload indexing and filtering: https://qdrant.tech/documentation/manage-data/indexing/
- Qdrant hybrid and multi-stage Query API: https://qdrant.tech/documentation/search/hybrid-queries/
- Self-hosted Qdrant security: https://qdrant.tech/documentation/tutorials-operations/secure-qdrant/
- LiteLLM OpenAI-compatible endpoints: https://docs.litellm.ai/
- llama.cpp server embeddings and authentication: https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md
- BGE-M3 model contract: https://huggingface.co/BAAI/bge-m3
- OpenAI embedding dimensions parameter: https://developers.openai.com/api/reference/resources/embeddings/methods/create