Assessor_2.0/docs/REGULATORY_RAG.md

# Regulatory-control RAG with Qdrant

Status: Phase 13F foundation. The current slice indexes and retrieves regulatory control text and can create privacy-minimized retrieval candidates for scanner findings. It does not map findings, call an LLM for classification, expose a browser API, or persist mapping decisions.

## Boundaries

- PostgreSQL remains the only system of record for ASSESSOR projects, assessments, scans, findings, results, reviews, and future accepted mappings.
- Qdrant is a derived, rebuildable search index. Losing or deleting its volume must not lose an ASSESSOR scan, finding, result, or review.
- The versioned regulatory corpus JSON is the rebuild source for Qdrant. It must be retained and governed separately from the index.
- React does not connect to Qdrant or the embedding provider. Both are backend-only boundaries.
- Regulatory retrieval produces candidates, not compliance decisions. A later constrained mapping stage must retain the retrieved chunk IDs, scores, corpus versions, model version, prompt/schema version, and final decision in PostgreSQL.

```text
versioned regulatory corpus JSON
        | validate and chunk
        v
approved multilingual embedding service
        | dense vectors
        v
Qdrant assessor_regulatory_chunks_v1
        ^
scanner finding -> public-signal query builder -> same embedding model -> dense candidates
        -> optional future hybrid fusion -> optional reranker
        -> later constrained mapper -> PostgreSQL audit record
```

## Collection contract

The collection is `assessor_regulatory_chunks_v1`. It has two named vector slots:

- `dense`: the configured embedding dimension with cosine distance.
- `sparse`: reserved now so a later sparse/BM25-style encoder can be added without changing the retrieval abstractions.

Qdrant payloads use schema version 1 and contain:

| Field | Purpose |
|---|---|
| `corpus_id`, `corpus_version` | Versioned rebuild source |
| `document_id` | Stable control-language document identity |
| `framework`, `framework_version` | Regulatory framework scope |
| `control_id` | Native control identifier |
| `chunk_id`, `chunk_index` | Deterministic chunk identity and order |
| `language` | Lower-case BCP 47 language tag |
| `title`, `text`, `text_sha256` | Retrieval content and integrity metadata |
| `embedding_model` | Prevents querying vectors from a different model space |
| `source_uri` | Optional public/source reference |

All fields used by filters have Qdrant payload indexes. An existing collection with a different dense dimension, distance, or missing sparse slot fails closed and must be rebuilt from the corpus.

## Corpus format and ingestion

The CLI accepts a UTF-8 JSON file of no more than 64 MiB:

```json
{
  "schemaVersion": 1,
  "documents": [
    {
      "schemaVersion": 1,
      "corpusId": "synthetic-framework",
      "corpusVersion": "2026-09-15",
      "documentId": "synthetic-control-1-de",
      "framework": "Synthetic Framework",
      "frameworkVersion": "1",
      "controlId": "CTRL-1",
      "language": "de",
      "title": "Synthetic control title",
      "text": "Synthetic control text used only to demonstrate the contract.",
      "sourceUri": "https://example.invalid/framework/control-1"
    }
  ]
}
```

The checked-in [`tests/fixtures/regulatory-corpus.synthetic.json`](../tests/fixtures/regulatory-corpus.synthetic.json) is an unlicensed synthetic EN/DE smoke corpus. It exists to verify rebuilding and retrieval mechanics only; it is not a regulatory source and must never be presented as compliance guidance.

Configure `.env.postgres` from the commented variables in `.env.postgres.example`, then run:

```text
npm run db:up
npm run rag:ingest -- path/to/regulatory-corpus.json
```

Ingestion validates every document, splits text deterministically on paragraph/sentence/word boundaries, embeds all chunks in bounded batches, and only then replaces the indexed chunks for each `(corpusId, documentId)`. Replacement is not a cross-system transaction: if Qdrant fails after deletion, rerun ingestion from the unchanged corpus.

## Configuration and trust

Qdrant 1.19.1 runs in Docker with a persistent named volume, an API key, and outbound usage telemetry disabled. Local REST access is loopback-only; deployment access stays on the Compose network. Remote plain HTTP Qdrant URLs are rejected.

The embedding adapter uses an OpenAI-compatible `/v1/embeddings` endpoint. Remote endpoints must use HTTPS. Plain HTTP is accepted only for `localhost`, `127.0.0.1`, or `::1`, enabling a same-machine llama.cpp development server without weakening remote transport. Configuration must explicitly provide the model name and its exact native/requested dimensions. Redirects are rejected, response size and time are bounded, and API keys remain in backend environment variables.

### Recommended local development model

Use BGE-M3 for the first local dense baseline. Its published model contract is multilingual, 1,024-dimensional, and supports up to 8,192 tokens. Those dimensions match the current Qdrant collection. llama.cpp supports dedicated embedding mode, model/default or explicit pooling, bearer API keys, and the OpenAI-compatible `/v1/embeddings` endpoint.

Example command, using a locally obtained compatible BGE-M3 GGUF file:

```text
llama-server.exe -m path/to/bge-m3.gguf --embedding --pooling cls --host 127.0.0.1 --port 8080 --api-key <local-secret> -c 8192
```

Use matching ASSESSOR settings:

```text
ASSESSOR_RAG_EMBEDDING_BASE_URL=http://127.0.0.1:8080/v1
ASSESSOR_RAG_EMBEDDING_MODEL=bge-m3
ASSESSOR_RAG_EMBEDDING_API_KEY=<same-local-secret>
ASSESSOR_RAG_EMBEDDING_DIMENSIONS=1024
ASSESSOR_RAG_EMBEDDING_REQUEST_DIMENSIONS=false
```

For a later LiteLLM comparison with `text-embedding-3-large`, use the approved HTTPS gateway and set `ASSESSOR_RAG_EMBEDDING_REQUEST_DIMENSIONS=true` if requesting 1,024-dimensional output. OpenAI supports the `dimensions` parameter for `text-embedding-3` models; llama.cpp currently returns the model's native dimension and should leave this flag false. Changing model identity or dimensions requires rebuilding the derived Qdrant corpus before comparing results.

Embedding is an external processing boundary. Regulatory text is sent during ingestion. The finding retrieval service can now send only the minimized public-signal query described below; unrestricted report text, paths, tenant identities, reviews, raw evidence, and local provenance IDs remain outside the embedding request. No browser endpoint or automatic mapping workflow activates this path.

## Finding query privacy boundary

`buildRegulatoryFindingQuery` accepts the existing `UnifiedResultProjection` but emits only fixed public concepts:

- exact Maester or PingCastle rule IDs that already exist in the checked-in public catalog;
- fixed public technical topic labels recognized locally from title, description, or recommendation; and
- a fixed public scanner name for known profiles.

The query never copies raw title, description, recommendation, asset, source path, project/tenant identity, review owner, review text, original values, or provenance IDs. Unsupported result kinds and scanner findings without a recognized public rule or topic fail closed as `insufficient-public-context` and do not call the retriever. The local `sourceResultId` is returned only for in-process correlation and is not included in the embedding request.

This is a technical minimization boundary, not organizational approval to process production findings externally. The service remains backend-only and has no API/UI route. Any production activation still requires approval of the embedding destination, catalog coverage, retention terms, and the precise finding classes allowed to cross that boundary.

### Local Qdrant integration check

`tests/server/qdrant-regulatory-index.integration.test.ts` is opt-in and uses deterministic synthetic vectors. With `ASSESSOR_QDRANT_INTEGRATION=1` and the normal Qdrant/model identity environment variables, it verifies the real collection, payload indexes, document replacement, filters, and retrieval without calling an embedding provider. The standard suite skips it so tests do not unexpectedly mutate a developer's derived index.

On 2026-09-15 this check passed against Qdrant 1.19.1 in WSL: authenticated access returned HTTP 200, unauthenticated access returned HTTP 401, the collection was green, Dense used 1,024-dimensional cosine vectors, the Sparse slot existed, all nine filter payload indexes existed, and one synthetic integration point was retrieved and then removed. PostgreSQL was not accessed or changed.

## Dense retrieval evaluation

The evaluation CLI runs the configured dense retriever against a bounded, versioned labeled case file and reports control-level HitRate@K, Recall@K, and mean reciprocal rank. Queries can declare a language different from the expected control language so English-to-German and German-to-English cases remain explicit.

```json
{
  "schemaVersion": 1,
  "cases": [
    {
      "id": "synthetic-en-to-de-mfa",
      "query": "Require multifactor authentication",
      "queryLanguage": "en",
      "filters": { "corpusId": "synthetic-framework" },
      "expectedControls": [
        {
          "framework": "Synthetic Framework",
          "frameworkVersion": "1",
          "controlId": "CTRL-MFA",
          "language": "de"
        }
      ]
    }
  ]
}
```

Run it after ingestion with:

```text
npm run rag:evaluate -- path/to/evaluation-cases.json 10
```

The CLI prints aggregate metrics only; it does not print queries or retrieved control text. Unit tests verify the metric contract with synthetic rankings, but those values are not evidence of real embedding quality. A meaningful baseline requires an approved representative corpus, independently labeled cases, and the configured multilingual model running against Qdrant.

The checked-in [`tests/fixtures/regulatory-evaluation.synthetic.json`](../tests/fixtures/regulatory-evaluation.synthetic.json) provides three explicitly cross-language smoke cases for the synthetic corpus.

Live smoke result on 2026-09-15 with local llama.cpp build 10472, BGE-M3 FP16, and Qdrant 1.19.1: six EN/DE documents produced six 1,024-dimensional chunks. All three cross-language cases returned their expected control at rank 1, giving HitRate@3 `1.0`, Recall@3 `1.0`, and MRR `1.0`. This verifies wiring and basic language crossing only; three controls are far too small and distinct to estimate production retrieval quality.

## Extension seams

- `RegulatoryCandidateRetriever` is the replaceable candidate stage. The current implementation is dense Qdrant retrieval; a later hybrid implementation can combine dense and sparse/BM25 candidates here.
- `RegulatoryReranker` is optional and receives only the bounded candidate set. Without one, dense ordering is retained.
- `RegulatoryRetriever` is the consumer-facing contract. Mapping code should depend on it, not on Qdrant.

Before enabling automatic finding mapping, complete organizational privacy approval, broaden public-signal coverage only through reviewed catalogs, establish the real dense baseline, and add control-level grouping, hybrid evaluation, reranker evaluation, calibrated confidence thresholds, a constrained output schema, human-review states, and append-only PostgreSQL mapping provenance.

## Reference basis

- Qdrant collections and named vectors: https://qdrant.tech/documentation/manage-data/collections/
- Qdrant payload indexing and filtering: https://qdrant.tech/documentation/manage-data/indexing/
- Qdrant hybrid and multi-stage Query API: https://qdrant.tech/documentation/search/hybrid-queries/
- Self-hosted Qdrant security: https://qdrant.tech/documentation/tutorials-operations/secure-qdrant/
- LiteLLM OpenAI-compatible endpoints: https://docs.litellm.ai/
- llama.cpp server embeddings and authentication: https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md
- BGE-M3 model contract: https://huggingface.co/BAAI/bge-m3
- OpenAI embedding dimensions parameter: https://developers.openai.com/api/reference/resources/embeddings/methods/create