DevSecOps
/
Assessor_2.0
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Assessor_2.0/docs/REGULATORY_RAG.md

16 KiB
Raw Blame History

Regulatory-control RAG with Qdrant

Status: Phase 13F one-finding vertical slice. The server prepares the licensed German ISO 27001/27002 corpus, indexes section-safe ISO 27002 semantic chunks plus ISO 27001 references, retrieves and groups candidates, requests a grounded LiteLLM proposal, persists runs and reviewer decisions, and exposes the workflow inline in Work. Batch mapping, sparse retrieval, reranking, and confidence calibration remain out of scope.

Boundaries

  • PostgreSQL remains the only system of record for ASSESSOR projects, assessments, scans, findings, results, reviews, mapping runs, and mapping decisions.
  • Qdrant is a derived, rebuildable search index. Losing or deleting its volume must not lose an ASSESSOR scan, finding, result, or review.
  • The versioned regulatory corpus JSON is the rebuild source for Qdrant. It must be retained and governed separately from the index.
  • React does not connect to Qdrant or the embedding provider. Both are backend-only boundaries.
  • Regulatory retrieval produces candidates, not compliance decisions. The constrained mapping stage retains retrieved chunk IDs, scores, corpus/source hashes, model, prompt/schema versions, and append-only reviewer decisions in PostgreSQL.
versioned regulatory corpus JSON
        | validate and chunk
        v
approved multilingual embedding service
        | dense vectors
        v
Qdrant assessor_regulatory_chunks_v2
        ^
scanner finding -> public-signal query builder -> same embedding model -> dense candidates
        -> group top 32 chunks into top 8 controls
        -> optional future hybrid fusion -> optional reranker
        -> constrained mapper -> human decision -> PostgreSQL audit record

Collection contract

The active collection is assessor_regulatory_chunks_v2. It has two named vector slots:

  • dense: the configured embedding dimension with cosine distance.
  • sparse: reserved now so a later sparse/BM25-style encoder can be added without changing the retrieval abstractions.

Qdrant payloads use schema version 2 and contain:

Field Purpose
corpus_id, corpus_version Versioned rebuild source
document_id Stable control-language document identity
framework, framework_version Regulatory framework scope
control_id Native control identifier
chunk_id, chunk_index Deterministic chunk identity and order
language Lower-case BCP 47 language tag
title, text, text_sha256 Retrieval content and integrity metadata
embedding_model Prevents querying vectors from a different model space
source_uri Optional public/source reference
annex_a_control_id, control_theme Shared 5.1–8.34 identity, Annex A alias, and theme
source_standard, source_edition, national_edition ISO/base and DIN source identity
section_kind, source_page_start, source_page_end Section-safe retrieval and page provenance
retrieval_role semantic for ISO 27002 or reference for ISO 27001
parser_version, source_sha256 Reproducible parser/source provenance

All fields used by filters have Qdrant payload indexes. An existing collection with a different dense dimension, distance, or missing sparse slot fails closed and must be rebuilt from the corpus.

ISO corpus preparation and ingestion

The server-only PDF.js CLI accepts explicit ISO 27001 and ISO 27002 paths and writes corpus-v2 JSON below the ignored .assessor-data/regulatory-corpora/ directory:

npm run rag:prepare-iso -- "ISO27001 2024.PDF" "ISO27002 2024.PDF"
npm run rag:ingest -- .assessor-data/regulatory-corpora/iso-27001-27002-de.json

The checked-in tests/fixtures/regulatory-corpus.synthetic.json is an unlicensed synthetic EN/DE smoke corpus. It exists to verify rebuilding and retrieval mechanics only; it is not a regulatory source and must never be presented as compliance guidance.

Configure .env.postgres from the commented variables in .env.postgres.example, then run:

npm run db:up
npm run rag:ingest -- path/to/regulatory-corpus.json

The parser accepts encrypted documents only when no password is required and copying is permitted. It rejects scanned/empty, oversized, copy-prohibited, structurally unexpected, incomplete, duplicate, or mismatched catalogs. The build must contain the matching 93 controls with theme counts 37/8/14/34. It retains hashes, editions, control/section metadata, and page provenance without committing licensed text. Ingestion splits only inside a control section, embeds all chunks in bounded batches, and then replaces indexed chunks per (corpusId, documentId). If Qdrant fails after deletion, rerun ingestion from the unchanged corpus JSON.

Configuration and trust

Qdrant 1.19.1 runs in Docker with a persistent named volume and outbound usage telemetry disabled. Local REST access is loopback-only; deployment access stays on the Compose network. An API key is optional for an unsecured loopback/internal self-host and required for remote hosts. Remote plain HTTP Qdrant URLs are rejected.

The embedding adapter uses an OpenAI-compatible /v1/embeddings endpoint. Remote endpoints must use HTTPS. Plain HTTP is accepted only for localhost, 127.0.0.1, or ::1, enabling a same-machine llama.cpp development server without weakening remote transport. Configuration must explicitly provide the model name and its exact native/requested dimensions. Redirects are rejected, response size and time are bounded, and API keys remain in backend environment variables.

Recommended local development model

Use BGE-M3 for the first local dense baseline. Its published model contract is multilingual, 1,024-dimensional, and supports up to 8,192 tokens. Those dimensions match the current Qdrant collection. llama.cpp supports dedicated embedding mode, model/default or explicit pooling, bearer API keys, and the OpenAI-compatible /v1/embeddings endpoint.

Example command, using a locally obtained compatible BGE-M3 GGUF file:

llama-server.exe -m path/to/bge-m3.gguf --embedding --pooling cls --host 127.0.0.1 --port 8080 --api-key <local-secret> -c 8192

Use matching ASSESSOR settings:

ASSESSOR_RAG_EMBEDDING_BASE_URL=http://127.0.0.1:8080/v1
ASSESSOR_RAG_EMBEDDING_MODEL=bge-m3
ASSESSOR_RAG_EMBEDDING_API_KEY=<same-local-secret>
ASSESSOR_RAG_EMBEDDING_DIMENSIONS=1024
ASSESSOR_RAG_EMBEDDING_REQUEST_DIMENSIONS=false
ASSESSOR_RAG_ACTIVE_CORPUS_ID=iso-27001-27002-de
ASSESSOR_RAG_ACTIVE_CORPUS_VERSION=<generated-corpus-version>

For a later LiteLLM comparison with text-embedding-3-large, use the approved HTTPS gateway and set ASSESSOR_RAG_EMBEDDING_REQUEST_DIMENSIONS=true if requesting 1,024-dimensional output. OpenAI supports the dimensions parameter for text-embedding-3 models; llama.cpp currently returns the model's native dimension and should leave this flag false. Changing model identity or dimensions requires rebuilding the derived Qdrant corpus before comparing results.

Embedding is an external processing boundary. Regulatory text is sent during ingestion. The finding retrieval service sends only the privacy-safe check query described below; the result table, paths, tenant identities, reviews, raw evidence, and local provenance IDs remain outside the embedding and completion requests.

Finding query privacy boundary

buildRegulatoryFindingQuery accepts the existing UnifiedResultProjection and builds the query from the shared privacy-safe finding projection (src/domain/privacy-safe-finding.ts, privacySafeFinding) — the identical boundary used by the AI overview:

  • the rule-level check text — title, plus description and recommendation when present — which comes from the deduplicated finding definition (canonical key tool:ruleId) and describes the check, not the tenant; email addresses are redacted;
  • a catalog Maester/PingCastle rule ID, a fixed public scanner name, and recognized public technical topics, all as optional query enrichment.

The query never reads the result table / occurrence data: affected asset, source locator/path, per-object values, project/tenant identity, review owner, review text, original values, or provenance IDs. Because the query is built from the rule-level check text, every scanner finding with a check title maps; only non-scanner records and findings without a check title fail closed as insufficient-public-context and do not call the retriever. The local sourceResultId is returned only for in-process correlation and is not included in the embedding request.

This is a technical minimization boundary. Activation still depends on the approved embedding/LiteLLM destinations, catalog coverage, retention terms, and the precise finding classes allowed to cross that boundary. Residual note: privacySafeFinding redacts emails only; it relies on definitions being rule-generic, so a scanner that folds an identifier (host, IP, path) into a definition title/description would carry it into the query. Strengthening that redaction is a shared decision affecting both AI features (see implementation plan).

One-finding mapping workflow

The Work finding reader loads mapping state only after Map ISO controls is opened. Mappings are global: the finding contributes only its non-personal rule identity and check context, and the mapping is keyed by that identity plus the pipeline versions, so one rule is mapped once and every occurrence in every project reuses it. The API returns HTTP 202 and the UI polls while queued/running. The service retrieves 32 eligible ISO 27002 chunks, groups by maximum dense score, keeps eight controls, attaches up to two semantic passages and the matching ISO 27001 statement, and bounds the LiteLLM payload to 48,000 characters.

The completion schema permits at most five primary or supporting proposals, advisory low/medium/high confidence, a bounded rationale, grounded evidence IDs, or an explicit no_relevant_control/insufficient_context outcome. Unknown controls, foreign evidence IDs, duplicates, malformed JSON, and oversized content fail closed. A reviewer can accept or reject any grounded top-eight candidate with an optional comment; each change appends a global decision event keyed by rule identity and control, using optimistic concurrency, shared across every project. Acceptance means relevant control, never compliance.

Routes:

  • GET /api/v1/projects/{projectId}/regulatory-mappings/{resultProjectionId}
  • POST /api/v1/projects/{projectId}/regulatory-mapping-runs
  • POST /api/v1/projects/{projectId}/regulatory-mapping-decisions

On the consolidated migration line (2026-10-02), the single migration 026 regulatory-mapping creates the global tables directly, together with the project-scoped assessor_regulatory_mapping_batches; the superseded occurrence-bound tables are never created. The remainder of this paragraph records the original history on the former experimental line, where migration 025 originally added occurrence-bound, project-scoped run and decision tables and migration 026 superseded it: it dropped those tables and created the global assessor_regulatory_mappings (keyed by rule_key + pipeline versions, UNIQUE(rule_key, corpus_version, retrieval_version, prompt_version, model)) and a global, rule-keyed assessor_regulatory_mapping_decisions; neither carries project_id or occurrence identity, and both survive project/finding deletion. Before 026 was applied, the 025 tables held only regenerable local test data (19 runs, 10 decisions); a verified backup of both tables was taken at backups/regulatory_mapping_025_backup_20260915-155410.sql and the data was then intentionally discarded per the user's decision. A live development backend applied 026 through its own migration runner (checksum recorded and re-verified against the source file); no reimport is required because mapping runs are regenerable and no scanner data changed.

Local Qdrant integration check

tests/server/qdrant-regulatory-index.integration.test.ts is opt-in and uses deterministic synthetic vectors. With ASSESSOR_QDRANT_INTEGRATION=1 and the normal Qdrant/model identity environment variables, it verifies the real collection, payload indexes, document replacement, filters, and retrieval without calling an embedding provider. The standard suite skips it so tests do not unexpectedly mutate a developer's derived index.

On 2026-09-15 this check passed against Qdrant 1.19.1 in WSL: the collection was green, Dense used 1,024-dimensional cosine vectors, the Sparse slot existed, all 12 filter payload indexes existed, and one synthetic integration point was retrieved and then removed.

Dense retrieval evaluation

The evaluation CLI runs the configured dense retriever against a bounded, versioned labeled case file and reports control-level HitRate@K, Recall@K, and mean reciprocal rank. Queries can declare a language different from the expected control language so English-to-German and German-to-English cases remain explicit.

{
  "schemaVersion": 1,
  "cases": [
    {
      "id": "synthetic-en-to-de-mfa",
      "query": "Require multifactor authentication",
      "queryLanguage": "en",
      "filters": { "corpusId": "synthetic-framework" },
      "expectedControls": [
        {
          "framework": "Synthetic Framework",
          "frameworkVersion": "1",
          "controlId": "CTRL-MFA",
          "language": "de"
        }
      ]
    }
  ]
}

Run it after ingestion with:

npm run rag:evaluate -- path/to/evaluation-cases.json 10

The CLI prints aggregate metrics only; it does not print queries or retrieved control text. Unit tests verify the metric contract with synthetic rankings, but those values are not evidence of real embedding quality. A meaningful baseline requires an approved representative corpus, independently labeled cases, and the configured multilingual model running against Qdrant.

The checked-in tests/fixtures/regulatory-evaluation.synthetic.json provides three explicitly cross-language smoke cases for the synthetic corpus.

Live smoke result on 2026-09-15 with local llama.cpp build 10472, BGE-M3 FP16, and Qdrant 1.19.1: six EN/DE documents produced six 1,024-dimensional chunks. All three cross-language cases returned their expected control at rank 1, giving HitRate@3 1.0, Recall@3 1.0, and MRR 1.0. This verifies wiring and basic language crossing only; three controls are far too small and distinct to estimate production retrieval quality.

Extension seams

  • RegulatoryCandidateRetriever is the replaceable candidate stage. The current implementation is dense Qdrant retrieval; a later hybrid implementation can combine dense and sparse/BM25 candidates here.
  • RegulatoryReranker is optional and receives only the bounded candidate set. Without one, dense ordering is retained.
  • RegulatoryRetriever is the consumer-facing contract. Mapping code should depend on it, not on Qdrant.

Before assessment-wide batch mapping, broaden public-signal coverage only through reviewed catalogs, establish a representative independently labeled EN/DE baseline, and evaluate sparse/hybrid retrieval, reranking, and confidence calibration. The one-finding workflow must remain human-reviewed.

Reference basis