DevSecOps
/
Assessor_2.0
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: enhance regulatory mapping integration tests

This commit is contained in:
Binh 2026-09-21 09:22:54 +02:00
parent c48ffb2194
commit 6600198312
22 changed files with 1618 additions and 167 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
.gitignore vendored
View File

@ -13,6 +13,7 @@ vite.config.d.ts
artifacts/objects/
Mock-Defender/
outputs/
evaluation/
.env.postgres
.assessor-data/

42
docs/IMPLEMENTATION_PLAN.md
View File

@ -3,6 +3,22 @@
Status: lebendes Projektdokument
Work-LiteLLM AI Overview (Phasen 7/13C und 11): Client Info besitzt eine eigene `AI Overview`-Registerkarte. Eine explizite Generate-/Regenerate-Aktion liest den kompletten aktiven Timeslot ueber den versionierten Privacy-Safe-JSON-Vertrag, hasht ihn deterministisch und uebergibt nur diesen datensparsamen Inhalt serverseitig an das konfigurierte Work-LiteLLM. HTTPS-Pflicht, feste Ziel-URL, Redirectablehnung, Backend-only Bearer-Secret, Timeout, Antwortgrenze, begrenzte hierarchische Chunks und lokale Schema-/Quell-ID-Pruefung sichern die Grenze. Migration 024 speichert append-only Runs und ausschliesslich validierte Endergebnisse. AI-Ausgaben bleiben beratend und veraendern keine Scores, Findings, Reviews, Imports oder Accepted-Risk-Daten.
Wiederholbarer ISO-Mappingvergleich 2026-09-18 (Phase 13D, ausdruecklicher Nutzerauftrag): Ein read-only Evaluationsskript vergleicht abgeschlossene PostgreSQL-Mappinglaeufe fuer PingCastle und Purple Knight mit einer versionierten manuellen Baseline und erzeugt eine reviewfaehige Excel-Datei mit Summary, Findingvergleich, Beziehungsdetails und Methodik. Ohne Parameter erkennt es die aktuelle Retrievalversion aus dem Produktivcode und waehlt die neueste vorherige Version mit vollstaendiger Scannerabdeckung; unvollstaendige neue Laeufe stoppen explizit. Die Baseline enthaelt ausschliesslich oeffentliche Rule-IDs und manuell bewertete Controlrollen, erlaubt null Primaercontrol und ist bewusst Evaluationsmaterial statt CI-Sollvertrag oder Produktmapping. Dauerhafte Tests verwenden nur synthetische Kontrollkennungen und pruefen Validierung, Retrieval-/Auswahl-/Rollen-/Extra-Trennung sowie Aenderungsklassifikation. Der erzeugte Smoke-Workbook bestand Import, Strukturscan, Formelpruefung und visuelle Kontrolle aller vier Blaetter. Beide Produktionsbuilds erfolgreich; der fokussierte neue Test besteht 3/3. Der Gesamtlauf erreicht 1.331 bestandene Tests und zehn opt-in-Skips, wird aber reproduzierbar durch den fachfremden vorhandenen 5-Sekunden-Timeout in `sheet-selection-cost.test.tsx` beendet; derselbe Test besteht isoliert in 0,9 Sekunden. Keine PostgreSQL-/Qdrant-Mutation, keine Mappingannahme, kein Manifestbump und kein Reimport.
ISO-Full-Context-Familienfusion 2026-09-18 (Phase 13D, ausdruecklicher Nutzerauftrag): Die bestehende Finding-only Querygrenze bleibt unveraendert. Die mit vollstaendigem Findingtext ausgefuehrten `control`-, `purpose`- und `guidance`-Suchen werden jetzt zunaechst zu genau einer Control-Rangliste fusioniert; diese Full-Context-Familie stimmt anschliessend neben Problem- und optionaler Remediationrangliste in der finalen Control-RRF ab. Damit koennen Abschnittstreffer wieder Controls in die Zwolf-Control-Shortlist einbringen, ohne drei korrelierte Schlussstimmen zu erhalten. Retrievalversion `hybrid-finding-query-section-family-rrf-1.6.0` erzeugt append-only neue Mapping-IDs; Queryschema 4, Prompt 1.5.0, vollstaendige Kandidatenbewertung, deterministische Auswahl, Human Review und fehlender produktiver Reranker bleiben unveraendert. Synthetische Mechanikregressionen pruefen Eintritt eines Abschnittstreffers und die einzelne Familienstimme, aber keine versionsabhaengigen ISO-Sollzuordnungen. 1.329 Tests bestanden, zehn opt-in Integrationen uebersprungen; beide Typechecks, Local-only-/Designpruefung und beide Produktionsbuilds erfolgreich. Die reale Recall-/Praezisionswirkung bleibt durch einen neuen manuellen PingCastle-/Purple-Knight-Lauf zu bewerten. Keine PostgreSQL-/Qdrant-Mutation, keine Scannerverarbeitung, kein Manifestbump und kein Reimport.
ISO-Finding-only-Retrieval 2026-09-18 (Phase 13D, ausdruecklicher Nutzerauftrag): Regulatory Queryschema 4 erzeugt ausschliesslich aus datenschutzsicherem Findingtitel, Beschreibung und Empfehlung eine Problemquery und optional eine Remediationquery; Scannername, Rule-ID, Regelkatalog und `public-security-topics.ts` beeinflussen weder Retrieval noch Mapperkontext. Die Themenliste bleibt nur im getrennten Public-Web-Search-Feature. Dense und BM25 werden je Query per Control-ID statt Chunk-ID fusioniert, sodass jedes Control pro Rangliste genau eine Stimme besitzt; rohe, zwischen Dense und Sparse nicht vergleichbare Scores entscheiden keine Gleichstaende mehr. Eine zweite Control-RRF waehlt hoechstens zwoelf Controls. Getrennte `control`-, `purpose`- und `guidance`-Suchen liefern danach nur Evidenz fuer bereits ausgewaehlte Controls und koennen weder Kandidaten hinzufuegen noch Stimmen vervielfachen. Retrievalversion `hybrid-finding-query-control-rrf-1.5.0` erzeugt append-only neue Mapping-IDs. Der produktive Runtimepfad aktiviert keinen Reranker; seine bestehende Schnittstelle bleibt nur als spaetere Erweiterungsgrenze. 1.328 Tests bestanden, zehn opt-in Integrationen uebersprungen; beide Typechecks, Local-only-/Designpruefung und Client-/Server-Produktionsbuild erfolgreich. Keine PostgreSQL-/Qdrant-Mutation, keine Scannerverarbeitung, kein Manifestbump und kein Reimport.
ISO-Retrievaltests entkoppelt 2026-09-18 (Phase 13D, ausdrueckliche Nutzerkorrektur): Der eingecheckte PingCastle-Gap-Satz mit fest erwarteten ISO-Control-IDs und Topics wurde entfernt. Regulatorische Fassungen, fachliche Bewertungen und Retrievalergebnisse bleiben bewusst ausserhalb der verpflichtenden Testsuite und werden bei Bedarf ueber den manuellen Review-/Excelvergleich oder eine explizit gestartete Evaluation bewertet. Dauerhafte Tests pruefen nur synthetische, regulationsunabhaengige Mechanik wie Privacy-Grenze, Rankingfusion, Deduplizierung, Shortlistlimit, Vollstaendigkeit der Kandidatenbewertung und deterministische Auswahl. Dadurch kann eine neue Regulatory-Fassung die CI nicht allein wegen geaenderter Sollzuordnungen brechen. 1.327 Tests bestanden, zehn opt-in Integrationen wurden uebersprungen; beide Typechecks, Local-only- und Designpruefung sind erfolgreich. Produktivcode, Pipelineversionen, PostgreSQL und Qdrant blieben unveraendert.
ISO-Hybridretrieval und Kandidatenbewertung 2026-09-18 (Phase 13D, ausdruecklicher Nutzerauftrag): Der Mappingpfad bewertet jetzt alle zwoelf gelieferten Controls einzeln als `primary`, `supporting` oder `irrelevant`; fehlende, doppelte, unbekannte oder nicht evidenzgebundene Bewertungen scheitern geschlossen. Ein deterministischer Reducer waehlt danach hoechstens ein Primaercontrol und drei Sekundaercontrols, erlaubt weiterhin Sekundaer-only und leitet `no_relevant_control` ausschliesslich aus vollstaendig irrelevanten Bewertungen ab. Promptversion 1.5.0 und Retrievalversion `hybrid-generic-reserved-section-rrf-1.4.0` erzeugen append-only neue Mapping-IDs; Persistenz-, Review- und UI-Vertraege bleiben unveraendert. Die Ingestion erzeugt zusaetzlich werkzeugunabhaengige gehashte BM25-Sparse-Vektoren aus Kontrolltitel und -text; der Candidate-Retriever vereinigt Dense- und Sparse-Rankings per RRF, bevor die vorhandene generische Reservierung und Abschnittsfusion die Zwolf-Control-Shortlist bilden. Der bestehende Qdrant-Korpus muss einmal aus der unveraenderten Korpusquelle neu ingestiert werden, damit Bestands-Punkte Sparse-Vektoren erhalten; Dense bleibt bis dahin funktionsfaehig. Der Aktivierungsversuch stoppte vor Qdrant-Aenderungen, weil der konfigurierte lokale Embeddingdienst nicht lief. 1.338 Tests bestanden, zehn opt-in Integrationen wurden uebersprungen; beide Typechecks, Produktionsbuild sowie Local-only- und Designpruefung sind erfolgreich. Keine PostgreSQL-Mutation, kein Scanner-Importvertrag, kein Manifestbump und kein Reimport.
ISO-Mapping-Promptbalance 2026-09-17 (Phase 13D, ausdruecklicher Nutzerauftrag): Der Vergleich des vollstaendigen v1.3-Laufs bestaetigt den Retrievalgewinn (`69/78` erwartete Controls in der Shortlist), aber auch eine Ueberkorrektur des Prompts: nur `6/40` erwartete Sekundaerbeziehungen wurden ausgegeben, obwohl 30 fehlende Sollbeziehungen als Kandidaten vorlagen. Prompt 1.4.0 erlaubt deshalb ausdruecklich null oder ein Primaercontrol und null bis drei Sekundaercontrols. Ein Mapping ohne Primaercontrol ist gueltig, wenn relevante Controls materiell beitragen, aber keines eindeutig dominiert; `no_relevant_control` bleibt dem Fall ohne relevante Kandidaten vorbehalten. Materiell unterschiedliche Sekundaerbeziehungen duerfen nicht allein zur Verkuerzung entfallen, generische Wortnaehe bleibt ausgeschlossen. Retrieval v1.3, Zwolf-Control-Shortlist, Hardlimit von fuenf, Human Review und Persistenzvertrag bleiben unveraendert. Neue Runs erhalten append-only eine neue Mapping-ID. Client-/Server-Typecheck, Local-only- und Designpruefung erfolgreich; der fokussierte Vitest-Start bleibt an der bekannten Sandbox-Verweigerung fuer das uebergeordnete Verzeichnis blockiert. Keine Datenbankmutation, kein Reimport und keine automatische Entscheidung.
ISO-Retrieval-Gesamtregression und Korrektur 2026-09-17 (Phase 13D, ausdruecklicher Nutzerauftrag): Der Abgleich aller 37 PingCastle-Findings gegen die unabhaengig erfassten 78 Primaer-/Sekundaerbeziehungen zeigte, dass die unmittelbar vorherige topic-zentrierte v1.2-Nachhaertung zwar die gezielten Luecken schloss, den Gesamt-Recall@12 aber von `61/78` auf `54/78` verschlechterte. Retrieval v1.3 schuetzt deshalb fuenf Controls aus einer werkzeugneutralen breiten Suche ueber Titel, Beschreibung und Empfehlung; eine optionale Topic-Query fuellt zusammen mit breiten sowie `control`-, `purpose`- und `guidance`-Rankings per RRF die restlichen sieben Plaetze. `public-security-topics.ts` ist damit nur Booster und keine Voraussetzung fuer neue Scanner. Der read-only BGE-M3-/Qdrant-Abgleich erreicht `68/78` (`87,2 %`; Primaer `33/38`, Sekundaer `35/40`) und liegt nahe am ausdruecklichen ungefaehren 90-%-Ziel mit Human Review. Queryschema 3, Retrievalversion `dense-generic-reserved-section-rrf-1.3.0` und Promptversion 1.3.0 erzeugen append-only neue Mapping-IDs; Accept/Reject bleibt autoritativ, keine Entscheidung wurde automatisch uebernommen. Client- und Server-Typecheck, Local-only- und Designpruefung sind erfolgreich. Der fokussierte Vitest-Lauf und der Vite-Build konnten in der Workspace-Sandbox wegen verweigertem Zugriff auf das uebergeordnete Verzeichnis nicht gestartet werden; eine Ausfuehrung ausserhalb der Sandbox war wegen erschoepftem Host-Freigabekontingent nicht moeglich. Keine Datenbankmutation, kein Importvertrag, kein Manifestbump und kein Reimport.
ISO-Retrieval-Nachhaertung 2026-09-17 (Phase 13D, ausdruecklicher Nutzerauftrag): Der Embeddingpfad verwendet jetzt eine von der vollstaendigen Modellkontextquery getrennte, kurze Public-Signal-Query aus bereinigtem Findingtitel und kuratierten oeffentlichen Sicherheitskonzepten. ISO-27002-`control`- und `purpose`-Abschnitte werden mit BGE-M3 getrennt durchsucht, auf Control-Ebene per deterministischem Reciprocal Rank Fusion (`k=60`) vereinigt und zunaechst auf 24 Controls begrenzt; das Mappingmodell erhaelt daraus hoechstens zwoelf Controls und darf weiterhin maximal fuenf evidenzgebundene Beziehungen ausgeben. `REGULATORY_FINDING_QUERY_SCHEMA_VERSION` ist 2, `REGULATORY_RETRIEVAL_VERSION` ist `dense-section-rrf-control-grouping-1.2.0`; alte Runs bleiben unveraendert und neue Vorschlaege erhalten eine neue deterministische Mapping-ID. Ein eingecheckter oeffentlicher PingCastle-Gap-Vertrag umfasst zehn Regeln und 17 unabhaengig zuvor validierte Sollbeziehungen. Der produktiv gebaute lokale BGE-M3-/Qdrant-Pfad erreichte fuer diesen begrenzten Satz Recall@12 1,0; dies behebt die beobachteten Retrievalluecken, ersetzt aber keinen unabhaengig gelabelten repraesentativen EN/DE-Gesamtdatensatz. 1.329 Standardtests bestanden, zehn optionale Integrationen wurden uebersprungen; Local-only-/Designpruefung und Client-/Server-Produktionsbuild erfolgreich. Keine Datenbankmutation, kein Importvertrag, kein Manifestbump und kein Reimport.
Gefiltertes Result-DB-ISO-Batchmapping 2026-09-16 (Phase 13D, ausdruecklicher Nutzerauftrag): Result DB bietet `Map all ISO controls` neben Refresh und Export JSON. Export und Batch leiten ihren Scope aus demselben normalisierten `UnifiedResultQuery` ab; der gewaehlte Timeslot bleibt aeussere Grenze, alle aktiven Filter einschliesslich des Standardfilters `deviation: true` grenzen weiter ein, und Pagination/geladene Zeilen haben keinen Einfluss. Der Server liest stabil alle Treffer, bricht bei veraenderter Gesamtzahl sicher ab, zaehlt Nicht-Scanner und nicht unterstuetzte Findings getrennt, dedupliziert global nach `ruleKey`, verwendet erfolgreiche/aktive Mappings wieder und plant fehlende beziehungsweise zuvor fehlgeschlagene Regeln ueber dieselbe auf zwei parallele Retrieval-/Modellpipelines begrenzte Queue wie Einzelaktionen. Atomare deterministische Claims verhindern doppelte Modellaufrufe. Migration 027 speichert nur den projektgebundenen normalisierten Filter-/Timeslot-Snapshot, Hash, Idempotenz, Status und Aggregatfortschritt; globale Mappingzeilen bleiben autoritativ. Das Dialogfenster zeigt Treffer und Fortschritt, bindet laufende Batches an den beim Start erfassten Filterstand und erzeugt ausschliesslich Vorschlaege; Accept/Reject bleibt manuell in Work. Filterwerte, Occurrence-Assets, Pfade, Mandanten- und Reviewdaten gelangen nicht in Retrieval oder LiteLLM. Die fruehere Batch-Zurueckstellung ist fuer diesen manuellen gefilterten Lauf ausdruecklich aufgehoben; die repraesentative Retrievalbewertung bleibt offenes Qualitaetsrisiko. Keine Scannerverarbeitung, kein Manifestbump und kein Reimport. Verifikation: `npm test` mit 1.315 bestandenen Tests und zehn uebersprungenen opt-in Integrationen, Local-only-/Designpruefung sowie Client-/Server-Produktionsbuild erfolgreich (bestehender Bundlehinweis). Der synthetische regulatorische End-to-End-Integrationstest blieb ohne aktivierte lokale Abhaengigkeiten uebersprungen. Korrektur nach Anwendung von Migration 027: Eine nachtraeglich in dieselbe Datei aufgenommene reine Sortiersequenz verletzte die Migrationsimmutabilitaet und der Checksum-Schutz stoppte das Backend korrekt. 027 ist exakt auf den angewandten Stand ohne Sortiersequenz zurueckgesetzt und durch eine feste SHA-256-Regression (`d89055c3e723b09d8d8be7f4225bfd08442f0d66e4e0b4ea3aa62ae6209e9ee9`) geschuetzt; Repositoryabfragen verwenden wieder den bereits angelegten stabilen `created_at, id`-Index. Es gibt keine Migration 028 und keine neue Datenbankmutation. Nach freigegebenem WSL-Neustart wurden der gesunde kanonische Container, Port `127.0.0.1:5432`, Volume `assessor-2_0_assessor-postgres-data`, der identische Ledger-Checksum und das Fehlen von 028 read-only bestaetigt; der automatisch neu gestartete Backendprozess meldet HTTP 200 und `persistence=postgresql`. Kein Backup war fuer die Korrektur erforderlich, weil keine neue Migration ausgefuehrt wurde.
ISO-27002-Mapping-Vertikalschnitt 2026-09-15 (Phase 13D, ausdruecklicher Nutzerauftrag): Die beiden lokal lizenzierten deutschen DIN-2024-PDFs werden serverseitig mit PDF.js und Positionsdaten in einen nicht eingecheckten Korpus-v2 ueberfuehrt. Passwortpflichtige, kopiergeschuetzte, textlose, uebergrosse oder strukturell unvollstaendige Dokumente scheitern geschlossen; ISO 27001 und 27002 muessen dieselben 93 Kontrollen mit 37/8/14/34 Themenkontrollen liefern. `assessor_regulatory_chunks_v2` enthaelt 705 mit lokalem BGE-M3 erzeugte, abschnitts- und kontrollsichere Chunks aus 558 Dokumenten, einschliesslich ISO-27001-Referenzen, Seiten-, Editions-, Parser- und Hashprovenienz; Sparse bleibt reserviert. Der Einzel-Finding-Dienst gruppiert 32 Dense-Treffer auf acht Kontrollen, bindet ISO-27001-Evidenz an, uebergibt nur die minimierte Public-Signal-Query und hoechstens 48.000 Zeichen ISO-Evidenz an das vorhandene Work-LiteLLM und akzeptiert ausschliesslich lokal validierte, evidenzgebundene JSON-Vorschlaege. Migration 025, beide Repositories und drei projektgebundene APIs speichern Runs und Accept/Reject-Entscheidungsereignisse append-only; Work zeigt den manuellen Ablauf inline mit Polling und Seitenbelegen. Ein realer synthetischer End-to-End-Test ueber kanonisches PostgreSQL, llama.cpp/BGE-M3, Qdrant und den freigegebenen LiteLLM-Gateway bestand einschliesslich Optimistic-Conflict und Cascade-Cleanup. Der laufende Entwicklungsserver wendete die additive Migration um 13:42 vor dem um 13:52 erstellten und verifizierten Backup automatisch an; beide neuen Tabellen waren leer. Diese Abweichung ist offen dokumentiert und wird nicht als Vorabbackup ausgegeben. Scannerverarbeitung blieb unveraendert; kein Manifestbump oder Reimport.
@ -1839,13 +1855,13 @@ Umgesetzter Grundstand vom 2026-09-15:
- Ein abgesicherter OpenAI-kompatibler Embeddingadapter verlangt fuer Remoteziele HTTPS, erlaubt HTTP nur auf Loopback fuer den lokalen llama.cpp-Testbetrieb und bindet ein ausdruecklich freigegebenes mehrsprachiges Modell an dessen exakte Dimension. Redirects, unbeschraenkte Antworten und Dimensionsdrift werden abgewiesen. Ein opt-in `dimensions`-Request unterstuetzt spaetere OpenAI-/LiteLLM-Vergleiche, bleibt fuer llama.cpp aus.
- CLI-Ingestion validiert und segmentiert einen begrenzten JSON-Korpus, bettet Chunks in Batches ein und ersetzt dokumentgebundene Qdrant-Punkte. Die Queryseite verwendet dasselbe Modell und kann Korpus, Framework, Version, Control und Sprache exakt filtern.
- `RegulatoryRetriever`, `RegulatoryCandidateRetriever` und `RegulatoryReranker` trennen Verbraucher, Candidate Retrieval und optionale Nachsortierung. Spaetere Hybridfusion kann die Candidate-Stufe ersetzen, ohne Mappingcode an Qdrant zu binden.
- Der Finding-Query-Builder akzeptiert die bestehende `UnifiedResultProjection`, gibt aber ausschliesslich katalogeigene Maester-/PingCastle-Regel-IDs, fest definierte oeffentliche Technikthemen und bekannte oeffentliche Toolnamen weiter. Rohtexte dienen nur lokaler Themenklassifikation; Titel, Beschreibung, Empfehlung, Asset, Pfade, Mandanten-/Projekt- und Reviewdaten werden nicht kopiert. Ohne oeffentliches Signal bleibt der Aufruf fail-closed und erreicht den Retriever nicht.
- Der Finding-Query-Builder akzeptiert die bestehende `UnifiedResultProjection` und trennt den vollstaendigen datenschutzsicheren regelbezogenen Modellkontext von den Retrievalsignalen. Eine generische Query aus Titel, begrenzter Beschreibung und Empfehlung funktioniert ohne Werkzeugkatalog; `public-security-topics.ts` erzeugt nur eine optionale zusaetzliche Topic-Query. Asset, Ergebnistabelle, Pfade, Mandanten-/Projekt-/Review- und Provenienzdaten werden nie gelesen oder kopiert. Nicht-Scanner und Findings ohne Titel bleiben fail-closed.
- Ein begrenzter versionierter Evaluationsvertrag und die CLI `rag:evaluate` messen HitRate@K, Recall@K und MRR auf Kontrollebene. Query- und erwartete Kontrollsprache sind getrennt deklariert; die CLI gibt nur Aggregatmetriken aus. Synthetische Unit-Rankings pruefen den Vertrag, gelten aber nicht als Qualitaetsnachweis des realen Modells.
- Ein eingecheckter synthetischer EN/DE-Smoke-Korpus und drei sprachuebergreifende Labelcases sichern den ausfuehrbaren Vertrag. Der opt-in WSL-Qdrant-Integrationstest prueft mit deterministischen synthetischen Vektoren den realen 1.19.1-Collection-/Index-/Replace-/Filter-/Retrievalpfad; er bewertet ausdruecklich keine Embeddingqualitaet.
- Der produktiv gebaute End-to-End-Pfad ist mit lokalem llama.cpp/BGE-M3 FP16 live gelaufen: sechs Dokumente/sechs Chunks; drei Cross-Language-Smoke-Cases bei K=3 jeweils Rang 1 und damit HitRate, Recall und MRR 1,0. Der Mini-Korpus ist nur ein Wiring-Smoke-Test, kein belastbarer Modellvergleich.
- Der Grundstand ist in `docs/REGULATORY_RAG.md` dokumentiert und mit synthetischen Vertrags-, Client-, Ingestion-, Retrieval- und Deploymenttests abgesichert.
- Der PDF.js-Importer erzeugt aus den explizit uebergebenen ISO-27001-/27002-Pfaden einen lokalen Korpus-v2 mit Quellhashes, Ausgaben, 93 geteilten Kontrollidentitaeten, Abschnittsarten und Seitenbereichen. Nur ISO-27002-Abschnitte werden semantisch gerankt; die entsprechende ISO-27001-Aussage wird als autoritative Referenzevidenz angehaengt.
- Ein manueller Work-Vertikalschnitt startet pro kanonischem Scanner-Finding einen idempotenten asynchronen Lauf, gruppiert die 32 Dense-Chunks auf acht Controls, validiert maximal fuenf LiteLLM-Vorschlaege gegen exakt gelieferte Evidenz und speichert Runs sowie optimistisch nebenlaeufigkeitsgeschuetzte Accept/Reject-Ereignisse append-only in PostgreSQL.
- Ein manueller Work-Vertikalschnitt startet pro kanonischem Scanner-Finding einen idempotenten asynchronen Lauf. Eine breite generische Suche reserviert fuenf Controls; die optionale Topic-Query liefert eine breite sowie getrennte `control`-, `purpose`- und `guidance`-Suche, deren Control-Rankings per Reciprocal Rank Fusion die restlichen sieben Plaetze fuellen. Ohne Topic-Treffer wird dieselbe generische Query verwendet. LiteLLM erhaelt hoechstens zwoelf Controls; maximal fuenf Vorschlaege werden gegen exakt gelieferte Evidenz validiert und Accept/Reject-Ereignisse append-only in PostgreSQL gespeichert.
- Result DB startet auf ausdrueckliche Aktion einen projektgebundenen Batch fuer exakt denselben vollstaendigen normalisierten Filter-/Timeslot-Scope wie Export JSON. Der Dienst paginiert serverseitig unabhaengig von der Anzeige, dedupliziert Scanner-Vorkommen global nach `ruleKey`, verwendet erfolgreiche und aktive Mappings wieder, versucht fehlgeschlagene Regeln einmal erneut und begrenzt Einzel- plus Batchpipelines gemeinsam auf zwei. Migration 027 speichert nur Scope, Filterhash, Idempotenz, Status und Aggregatfortschritt; Entscheidungen bleiben manuell im bestehenden Work-Panel.
Offen bis zur Phasenabnahme:
@ -1872,7 +1888,7 @@ Die naechste Umsetzung fuehrt Phase 13B fort und entwickelt parallel den ausdrue
1. Phase 13A ist abgeschlossen: Erkennung, technische Dokumenttypen, Source Extractors, Mappingprofile, Finding-Identität sowie allgemeine Artefakt-, Processing- und Jobgrenzen sind konsolidiert und die Scannerpersistenz verwendet den Candidate-Pfad.
2. Phase 13B ist in Arbeit: Der sichere Ingestion-Unterbau, technische Policy-Identitaeten, versionierte ADMX-/Baselineanalysen, deterministisches Reprocessing, Environment Profiles, konservative Snapshotaggregation, typisierte Evaluation, fail-closed Finding Factory, Unified-Result-Read-Projektion, projektgebundene Review-/Orchestrierungsoberflaechen und die aktuelle snapshotbezogene Ergebnisdeduplizierung sind umgesetzt. Als naechste vertikale Etappe folgen fachlich kuratierte Policy-Regeln und vollstaendige versionierte Windows-Inbox-ADMX-/ADML-Kataloge; danach weitere Preference-CSEs und Exportdetails. Ohne RSoP/GP Results bleibt die effektive Anwendung weiterhin ausdruecklich nicht verifiziert.
3. Phase 13D besitzt den lizenzierten ISO-Korpus-v2, die rebuildbare Qdrant-/Dense-Grundlage, den fail-closed Public-Signal-Pfad, den persistierten manuell reviewten Einzel-Finding-Workflow sowie das explizit vorgezogene gefilterte Result-DB-Batchmapping. Als naechste Etappe folgen ein unabhaengig gelabelter repraesentativer EN/DE-Evaluationssatz und die reale Dense-Baseline; Sparse/Hybrid, Reranking, Confidence-Kalibrierung und jede automatische Entscheidung bleiben bis zu dieser Bewertung ausgeschlossen.
3. Phase 13D besitzt den lizenzierten ISO-Korpus-v2, die rebuildbare Qdrant-Grundlage, werkzeugunabhaengiges Dense-/BM25-Hybridretrieval, Finding-only Problem-/Remediationqueries mit Control-level RRF, abschnittsspezifische Evidenzsuche, den persistierten manuell reviewten Einzel-Finding-Workflow sowie das explizit vorgezogene gefilterte Result-DB-Batchmapping. Der fruehere 37-Finding-PingCastle-Vergleich bleibt ein historisches Regressionssignal und kein verpflichtender CI-Vertrag oder Beleg fuer neue Werkzeuge. Als naechste Etappe folgen ein unabhaengig gelabelter repraesentativer EN/DE-Evaluationssatz mit mehreren Scannern und Negativfaellen; ein gelernter oder modellbasierter Reranker, Confidence-Kalibrierung und jede automatische Entscheidung bleiben bis zu einer gesonderten Entscheidung ausgeschlossen.
4. Erst nach verifizierter Phase 13B den kontrollierten Firmenbetrieb aus Phase 14 mit SSO, Rollen, HTTPS, Monitoring, Backup/Restore und Aufbewahrung fortsetzen.
Die technische Reihenfolge ist bis einschliesslich Phase 13A umgesetzt. Phase 13B baut auf dem verifizierten sicheren Ingestion-Unterbau auf und reicht inzwischen produktiv von der begrenzten Parser-/Workerverarbeitung ueber unveraenderliche Analyse-, Profil-, Aggregations- und Evaluationslaeufe bis zur streng evidenzgebundenen Finding Factory und ihrer gemeinsamen Unified-Result-Leseprojektion. Das Backend leitet projektgebundene Aktionsbereitschaft aus aktiven Revisionen und exakt passenden Referenzdaten ab; der Datahub fuehrt Environment-Profile-Anlage, Reprocessing, Aggregation, Evaluation und Finding Factory ueber die autoritativen idempotenten Services aus. Der produktive Regelkatalog bleibt bis fachlicher Kuratierung leer, und die vorhandenen Microsoft-Baselines bleiben ohne vollstaendige Windows-Inbox-ADMX-/ADML-Kataloge partial; dadurch erzeugt dieser reale Bestand weiterhin nur Review und keine Findings.
@ -1934,6 +1950,14 @@ Festlegung 2026-09-15 (Phase 13D, globale ISO-Mappings, ausdruecklicher Nutzerau
Festlegung 2026-09-16 (Phase 13D, gefiltertes Result-DB-Batchmapping, ausdruecklicher Nutzerauftrag; begrenzte Aufhebung der frueheren Batch-Zurueckstellung): `All` bezeichnet alle Treffer der aktuell aktiven Result-DB-Filter innerhalb des gewaehlten Timeslots und ist exakt derselbe vollstaendige `UnifiedResultQuery`-Scope wie Export JSON. Offset, Limit, Load-more-Zustand und geladene Zeilen sind reine Darstellung. Der Standardfilter `deviation: true` bleibt wirksam, bis er geaendert oder zurueckgesetzt wird. Der beim Start normalisierte Snapshot bleibt fuer den Lauf unveraendert und wird projektgebunden mit kanonischem Hash und Idempotenz in Migration 027 gespeichert. Nur kanonische Scanner-Findings sind geeignet; Nicht-Scanner und fehlender oeffentlicher Pruefkontext werden gezaehlt, aber nie versendet. Vorkommen werden global nach `ruleKey` dedupliziert, erfolgreiche oder aktive Mappings wiederverwendet und fehlende beziehungsweise zuvor fehlgeschlagene deterministisch geclaimt. Einzel- und Batchaktionen teilen zwei parallele Retrieval-/Modellpipelines. Der Batch erzeugt ausschliesslich Vorschlaege; Accept/Reject bleibt manuell und es gibt weder Auto-Accept noch Batch-Entscheidungsqueue. Die vorgezogene Orchestrierung ist keine Aussage zur Retrievalqualitaet: repraesentative EN/DE-Evaluation, Sparse/Hybrid, Reranking und Confidence-Kalibrierung bleiben offen. Kein Importvertrag geaendert, daher kein Manifestbump oder Reimport.
Festlegung 2026-09-17 (Phase 13D, Retrievalluecken, ausdruecklicher Nutzerauftrag; begrenzte Revision der Reranking-Zurueckstellung): Die vollstaendige datenschutzsichere Mappingquery bleibt Modellkontext, wird aber nicht mehr unveraendert eingebettet. Eine getrennte kurze Embeddingquery aus Titel und kuratierten oeffentlichen Konzepten erzeugt zwei Dense-Rankings fuer ISO-27002-`control` und `purpose`; diese werden ohne zusaetzliches Modell auf Control-Ebene per Reciprocal Rank Fusion vereinigt. Der Retriever behaelt 24 Controls, der Mappingprompt hoechstens zwoelf, und die fachliche Ausgabe bleibt auf fuenf manuell reviewpflichtige Beziehungen begrenzt. Der oeffentliche zehn-Finding-/17-Beziehungs-Gap-Satz erreichte im produktiv gebauten lokalen BGE-M3-/Qdrant-Pfad Recall@12 1,0. Dies erlaubt die deterministische Fusion, aber weiterhin weder Auto-Accept noch eine allgemeine Qualitaetsaussage; unabhaengige repraesentative Labels, Sparse/Hybrid und ein gelernter oder modellbasierter Reranker bleiben offen. Neue Retrieval- und Queryversionen erzeugen append-only neue Mapping-IDs; Scannerimporte und bestehende Entscheidungen werden nicht umgeschrieben.
Festlegung 2026-09-17 (Phase 13D, Gesamtregression; ersetzt die unmittelbar vorherige v1.2-Retrievalzusammensetzung, nicht deren historischen Eintrag): Eine Scanner- oder Topic-Taxonomie darf die Kandidatensuche nur ergaenzen. Die generische Query aus datenschutzsicherem Findingtitel, Beschreibung und Empfehlung erzeugt immer eine breite Rangliste; ihre ersten fuenf Controls sind im zwoelfteiligen Shortlistbudget reserviert. Eine optionale Topic-Query liefert eine breite sowie `control`-, `purpose`- und `guidance`-Rangliste; RRF fuellt daraus sieben weitere eindeutige Controls, danach dient die generische Rangliste als Fallback. Damit bleibt `public-security-topics.ts` nutzbar, aber neue Werkzeuge benoetigen fuer eine erste Zuordnung keine Pflege dieses Katalogs. Der damalige Mappingprompt fordert die kleinste hinreichende Menge, normalerweise einen Primaer- und null bis zwei Sekundaercontrols; maximal fuenf und Human Review bleiben unveraendert. Queryschema 3, Retrieval 1.3 und Prompt 1.3 versionieren die Aenderung append-only. Der vollstaendige 37-Finding-/78-Beziehungs-Abgleich verbessert Candidate Recall@12 gegen v1.2 von `54/78` auf `68/78`; wegen Benchmarknaehe und fehlender Multi-Scanner-Negativfaelle bleibt dies ein Regressionssignal, keine allgemeine 90-%-Garantie.
Festlegung 2026-09-17 (Phase 13D, Promptbalance; ersetzt Prompt 1.3, nicht Retrieval 1.3): Relevanz erzwingt kein Primaercontrol. Prompt 1.4 erlaubt null oder ein Primaercontrol und null bis drei Sekundaercontrols. Primaer wird nur vergeben, wenn ein Kandidat eindeutig die naechste Anforderung fuer das Wurzelsicherheitsthema ist. Sind mehrere Controls materiell relevant, ohne dass eines dominiert, ist ein `mapped`-Ergebnis nur mit Sekundaercontrols gueltig. Ein vorhandenes materiell eigenstaendiges Sekundaercontrol darf nicht allein zugunsten einer kurzen Liste entfallen; blosse generische Wortnaehe bleibt unzureichend. `no_relevant_control` gilt nur ohne relevante Kandidaten, `mapped` verlangt mindestens eine Beziehung. Das Validierungshardlimit von fuenf bleibt rueckwaertskompatibel, waehrend der Prompt hoechstens vier Beziehungen anfordert. Neue Runs sind append-only; Entscheidungen und Scannerimporte bleiben unveraendert.
Festlegung 2026-09-18 (Phase 13D, Finding-only Retrieval; ersetzt die v1.4-Retrievalzusammensetzung, nicht deren historischen Eintrag): Regulatorische Kandidatensuche und Mapperkontext werden allein aus dem datenschutzsicheren Findingtitel, der Beschreibung und der Empfehlung erzeugt. Problem und Remediation bilden unabhaengige Queries; Scannername, Rule-ID, Catalog und kuratierte Topics sind keine Retrievalsignale. Dense-/Sparse-RRF und die anschliessende Query-RRF arbeiten auf Control-Ebene mit hoechstens einer Stimme je Control und Rangliste. Abschnittssuchen duerfen ausgewaehlte Controls mit `control`-, `purpose`- und `guidance`-Passagen belegen, aber die Zwolf-Control-Auswahl nicht veraendern. Queryschema 4 und Retrieval 1.5 versionieren die Aenderung append-only. Der produktive Runtimepfad verwendet keinen Reranker; ein solcher wird erst nach gesonderter Untersuchung aktiviert. Human Review, vollstaendige Kandidatenbewertung und deterministische 0-1/0-3-Auswahl bleiben verbindlich.
Festlegung 2026-09-15 (Phase 13D, ausdruecklicher Nutzerauftrag): Qdrant ist der rebuildbare Suchindex fuer versionierte regulatorische Chunks und niemals fachliche Quelle fuer Scannergebnisse oder Mappingentscheidungen. PostgreSQL bleibt System of Record fuer Scans, Findings, Ergebnisse, Reviews und spaetere append-only Mappingprovenienz. Dense und Sparse sind benannte Vektorraeume; zunaechst wird nur mehrsprachiges Dense Retrieval aktiviert. Candidate Retrieval und optionaler Reranker bleiben getrennte Schnittstellen. Ein Modellwechsel darf alte Vektoren nicht still weiterverwenden; Modellname und Dimension werden explizit gebunden, Abfragen filtern auf das aktive Modell, und ein Korpus-Rebuild bleibt Pflicht. Der Embeddingdienst ist eine eigene externe Vertrauensgrenze; Findinginhalte duerfen erst nach einer getrennten Datenschutzfreigabe als minimierte Query uebertragen werden.
Festlegung 2026-09-10 (Purple-Knight-Begleitdateien): Ein nativer Score darf aus einer separaten Summary auf die Findings desselben Scanlaufs uebertragen werden, wenn genau ein nativer Traeger und eine eindeutige Ergebnispopulation denselben Werkzeug- und Verarbeitungskontrakt teilen. Datei- und Vorkommensherkunft bleiben gemeinsam erhalten; Mehrdeutigkeit verhindert die Verbindung.
@ -2319,6 +2343,10 @@ Timeslotvergleich 2026-09-11: Der bisherige Zweilaufadapter kann Quellenidentita
Accepted Risk 2026-09-10: Deklarierte Namen und Nachweisreferenzen sind keine authentifizierten Signaturen oder geprueften Berechtigungen. Bestehende Projektloeschung und Datenbankadministration begrenzen den Schutz der JSONB-Historie. Es gibt noch keinen unabhaengigen Archivierungs-/Legal-Hold-Vertrag und keinen Fristenbenachrichtigungsdienst. Diese Grenzen sind im Dialog und in `SECURITY.md` dokumentiert; der Ausbau bleibt Teil von Phase 14. Synthetische Repository-/SQL-Transaktionstests ersetzen keine freigegebene PostgreSQL-Integrationsabnahme.
Regulatory RAG 2026-09-15: Regulatorische Texte koennen urheber-, lizenz- oder versionsgebunden sein; ein technisch lesbarer Korpus ist deshalb noch kein freigegebener Produktkorpus. Lokale llama.cpp-Embeddings bleiben auf Loopback, waehrend minimierte Querytexte bei einem spaeteren Work-LiteLLM-Vergleich die ASSESSOR-Prozessgrenze verlassen. Gegenmassnahmen sind ein autorisierter versionierter Korpus, explizite Modell-/Dimensionsbindung, Backend-only Secrets, Remote-HTTPS, begrenzte Requests und der implementierte fail-closed Public-Signal-Query-Builder; organisatorische Datenschutzfreigabe bleibt fuer externe Ziele erforderlich. Dense Similarity ist nur Kandidatenranking und weder Kontrollzuordnung noch Confidence; vor automatischer Nutzung sind reale sprachuebergreifende Evaluation, Hybrid-/Reranker-Vergleich, Schwellenkalibrierung und Human Review erforderlich. Qdrant-Dokumentersatz ist nicht transaktional mit einem Embeddingdienst; Fehler koennen einen unvollstaendigen abgeleiteten Index hinterlassen, der aus dem unveraenderten Korpus neu aufgebaut werden muss. Das Volume ist Betriebsbeschleunigung, keine fachliche Sicherung. Der autorisiert geloeschte alte Qdrant ist entfernt und 1.19.1 belegt kanonisch Port 6333; das alte Volume bleibt vorsorglich erhalten. Der lokale BGE-M3-End-to-End-Smoke-Test war mit drei klar getrennten Themen perfekt, ist statistisch und fachlich aber nicht repraesentativ. Ein kuratierter lizenzierter Korpus, schwierigere Negativ-/Mehrfachkontrollfaelle, unabhaengige Labels und der LiteLLM-Vergleich bleiben offen.
Retrieval-Gap-Benchmark 2026-09-17, korrigiert 2026-09-18: Die zehn PingCastle-Faelle und 17 Sollbeziehungen stammten aus derselben manuellen Validierungsrunde, die die Luecken identifiziert hatte; Recall@12 1,0 auf diesem Satz belegte nur die konkrete Reparatur und war wegen Auswahl- und Konzeptkurationsbias kein unabhaengiger Gesamtqualitaetsnachweis. Auf ausdrueckliche Nutzerentscheidung wird dieser fachlich veraenderliche Satz nicht als feste Regression ausgefuehrt. Sollzuordnungen bleiben Human-Review-/Evaluationsdaten; die verpflichtende Suite prueft ausschliesslich regulationsunabhaengige Retrievalmechanik.
Retrieval-Gesamtbenchmark 2026-09-17: Der nachtraegliche Abgleich aller 37 Findings belegte die Auswahlverzerrung des reinen Gap-Satzes: v1.2 traf dort `17/17`, verschlechterte ueber alle 78 erwarteten Beziehungen jedoch v1.1 von `61/78` auf `54/78`. Die generisch reservierte v1.3-Zusammensetzung erreicht `68/78` (`87,2 %`), wurde aber auf demselben PingCastle-Satz ausgewaehlt und kann deshalb ueberangepasst sein. Gegenmassnahmen sind die harte Abhaengigkeitsfreiheit vom Topic-Katalog, ein Unknown-Scanner-Vertrag, unveraendertes Human Review sowie als naechster Qualitaetsschritt ein separat gelabelter Multi-Scanner-Satz mit direkten, unterstuetzenden und bewusst nicht relevanten Controls. Candidate Recall darf weder als Mapperpraezision noch als Annahmequote oder Compliancebewertung ausgegeben werden.
Work-LiteLLM 2026-09-14: Der Unternehmensgateway ist eine ausdruecklich freigegebene externe Vertrauensgrenze und kann Requests gemaess eigener Betriebsrichtlinie routen oder protokollieren; ASSESSOR kann diese Aufbewahrung nicht kontrollieren. Die API sendet deshalb nur den vorhandenen Privacy-Safe-Export nach explizitem Nutzerklick, nie Rohdateien, Reviews, Quelltabellen oder Storage Keys. Gatewayausfall, Modellwechsel oder ungueltige JSON-Ausgabe lassen den bisherigen erfolgreichen Run unveraendert und erfordern eine bewusste neue Generierung. Der synthetische Workstation-Connectivitycheck erreichte die konfigurierte URL, erhielt aber HTTP 401; Keyaktivierung, Berechtigung oder Gatewaykonfiguration muessen betrieblich geklaert werden, ohne Secrets oder Providerbody offenzulegen. Migration 024 ist in diesem Lauf mangels Docker weder auf den kanonischen lokalen Datastore angewandt noch per PostgreSQL-Integration verifiziert; vor Anwendung bleiben verifiziertes Backup und erfolgreicher synthetischer Workstation-/Server-Liveaufruf verpflichtend.
Purple-Knight-Begleitdateien 2026-09-10: Alte Importe koennen den nativen Wert noch nicht als Kontextbeobachtung enthalten. Dann kann die neue Leseprojektion ihn nicht nachtraeglich aus Rohtext erfinden und die betroffenen Summary-/Findings-Dateien muessen gemeinsam neu importiert werden. Mehrere unterschiedliche Ergebnisberichte in demselben Scanlauf werden bewusst nicht automatisch mit einem einzelnen Summarywert verbunden.
@ -2686,6 +2714,14 @@ Verifiziert 2026-09-08 (Phase 7/13B): GPO-Quellenkorrektur fuer Work und Client
| 2026-09-10 | Results-Browser zeigt fuer Defender-CVEs die aggregierte Betroffenheit als `Affected Devices`-Anzahl statt eines leeren `Affected Object`. Andere Werkzeuge behalten die `Affected Object`-Zeile unveraendert. |
| 2026-09-10 | Defender-Ergebnisfenster auf eine Device-Tabelle umgestellt: `Affected Devices` zeigt Device, Scanner und Site; maschinenlesbare `Source Observations` bleiben als Evidenz erhalten und werden in der Lesespalte ausgeblendet. Keine SQL- oder Persistenzmigration; Phase 13A bleibt abgeschlossen. |
| 2026-09-10 | Phase-13A-Defender-Projektion auf CVE-Aggregation umgestellt: Der Source Extractor emittiert genau ein Finding je CVE; betroffene Devices werden zusammengefasst, alle Sensorbeobachtungen bleiben erhalten, und der Vergleichsschluessel ist `rule-id`. Keine SQL- oder Persistenzmigration; Phase 13A bleibt abgeschlossen. |
| 2026-09-18 | Phase 13D, wiederholbarer manueller ISO-Mappingvergleich: `npm run rag:compare-mappings` liest die vollstaendigen PingCastle-/Purple-Knight-Laeufe read-only aus dem kanonischen lokalen PostgreSQL, erkennt standardmaessig aktuelle und vorherige Retrievalversion und erzeugt eine formatierte Excel-Pruefdatei mit Summary, Findingvergleich, Beziehungsdetails und Methodik. Die versionierte Baseline speichert nur oeffentliche Rule-IDs und manuelle Primaer-/Supportingrollen einschliesslich null Primaer; sie ist kein Produktmapping und kein CI-Sollvertrag. Synthetische Tests pruefen nur die generische Vergleichsmechanik. Smoke-Workbook vollstaendig importiert, alle vier Blaetter gerendert und null Formelfehler gefunden. Beide Produktionsbuilds erfolgreich; neuer Test 3/3 und der zuvor unter Last abgebrochene fachfremde `sheet-selection-cost`-Test isoliert 1/1 erfolgreich. Gesamtsuite: 1.331 bestanden, zehn uebersprungen, ein vorhandener UI-Test ueberschreitet unter Gesamtsuitelast reproduzierbar sein 5-Sekunden-Limit. Keine Datenbank-/Indexmutation, keine Mappingentscheidung, kein Manifestbump oder Reimport. |
| 2026-09-18 | Phase 13D, Full-Context-Familienfusion ohne Reranker: Die drei mit vollstaendigem Findingtext ausgefuehrten Abschnittssuchen werden zu einer Control-Rangliste verdichtet und tragen neben Problem- und optionaler Remediationrangliste genau eine Stimme zur finalen RRF bei. Abschnittstreffer duerfen damit wieder in die Zwolf-Control-Shortlist gelangen, ohne korrelierte Dreifachstimmen. Retrievalversion `hybrid-finding-query-section-family-rrf-1.6.0`; Queryschema, Prompt, Kandidatenbewertung, deterministische Auswahl und Human Review unveraendert. 1.329 Tests bestanden, zehn opt-in Integrationen uebersprungen; beide Typechecks, Local-only-/Designpruefung und beide Produktionsbuilds erfolgreich. Keine Datenbank-/Indexmutation, kein Manifestbump oder Reimport; reale Qualitaetsbewertung bleibt dem naechsten manuellen Scannervergleich vorbehalten. |
| 2026-09-18 | Phase 13D, Finding-only Retrieval ohne Reranker: Queryschema 4 baut Problem- und optionale Remediationquery nur aus privacy-safe Titel, Beschreibung und Empfehlung; Tool, Rule-ID, Katalog und `public-security-topics.ts` sind aus dem regulatorischen Pfad entfernt. Hybrid-RRF und Query-RRF deduplizieren auf Control-Ebene; Abschnittssuchen liefern nur Evidenz fuer bereits ausgewaehlte Controls. Retrievalversion `hybrid-finding-query-control-rrf-1.5.0`; produktive Runtime ohne Reranker. 1.328 Tests bestanden, zehn opt-in Integrationen uebersprungen; beide Typechecks, Local-only-/Designpruefung und beide Produktionsbuilds erfolgreich. Keine Datenbank-/Indexmutation, kein Manifestbump oder Reimport. |
| 2026-09-18 | Phase 13D, ausdrueckliche Nutzerkorrektur: Den PingCastle-spezifischen Retrieval-Gap-Fixture mit zehn Regeln, 17 erwarteten ISO-Control-IDs und festen Topic-Erwartungen sowie seine Tests entfernt. Regulatorische Sollzuordnungen werden nicht mehr als CI-Vertrag behandelt; sie bleiben Gegenstand manueller Review-/Excelvergleiche oder explizit gestarteter Evaluationen. Synthetische Mechaniktests fuer Privacy, Hybrid-/Abschnittsfusion, Deduplizierung, Shortlist und deterministische Auswahl bleiben erhalten. 1.327 Tests bestanden, zehn opt-in Integrationen uebersprungen; beide Typechecks sowie Local-only-/Designpruefung erfolgreich. Keine Produktivcode-, Pipeline-, Datenbank- oder Indexaenderung. |
| 2026-09-18 | Phase 13D, ausdruecklicher Nutzerauftrag: Mappingprompt 1.5.0 bewertet alle zwoelf Kandidaten vollstaendig als `primary`, `supporting` oder `irrelevant`; Grounding und Vollstaendigkeitsvalidierung erfolgen vor einer deterministischen Auswahl von maximal einem Primaer- und drei Sekundaercontrols, einschliesslich Supporting-only und abgeleitetem `no_relevant_control`. Retrieval 1.4.0 erzeugt bei der Korpusingestion werkzeugneutrale gehashte BM25-Sparse-Vektoren und fusioniert Sparse und Dense je Query per RRF; generische Reservierung, Abschnittsfusion und Zwolf-Control-Limit bleiben erhalten. Bestehende Qdrant-Punkte benoetigen eine einmalige Reingestion aus derselben Korpusquelle, PostgreSQL und Scannerimporte bleiben unveraendert. Der Aktivierungsversuch stoppte vor Qdrant-Aenderungen am nicht laufenden lokalen Embeddingdienst. 1.338 Tests bestanden, zehn opt-in Integrationen wurden uebersprungen; beide Typechecks, beide Produktionsbuilds sowie Local-only-/Designpruefung erfolgreich. Keine automatische Entscheidung, Migration, Manifestaenderung oder Reimport. |
| 2026-09-17 | Phase 13D, Promptbalance nach dem vollstaendigen v1.3-Lauf: Retrieval bleibt `dense-generic-reserved-section-rrf-1.3.0`; Prompt 1.4.0 erlaubt null oder ein Primaercontrol und null bis drei Sekundaercontrols. Supporting-only ist bei materieller Relevanz ohne eindeutig dominierendes Control ein gueltiges `mapped`-Ergebnis. Materiell eigenstaendige Sekundaerbeziehungen duerfen nicht allein zur Listenverkuerzung entfallen; generische Wortnaehe bleibt ausgeschlossen. Domain- und Service-Regressionen sichern Supporting-only und den expliziten Promptvertrag. Client-/Server-Typecheck, Local-only- und Designpruefung erfolgreich; fokussierter Vitest-Start durch bekannte Sandbox-Verzeichnisverweigerung blockiert. Keine Retrieval-, DB-, Import-, Reimport- oder Entscheidungsveraenderung. |
| 2026-09-17 | Phase 13D, Korrektur nach Gesamtvergleich aller 37 PingCastle-Findings: Die topic-zentrierte Retrievalversion 1.2 schloss den gezielten 17-Beziehungs-Gap-Satz, verschlechterte jedoch Candidate Recall@12 ueber alle 78 validierten Primaer-/Sekundaerbeziehungen von v1.1 `61/78` auf `54/78`. Queryschema 3 erzeugt deshalb immer eine generische Query aus Titel, Beschreibung und Empfehlung; `public-security-topics.ts` liefert nur eine optionale Topic-Query. Retrieval 1.3 reserviert fuenf generische Controls und fuellt sieben Plaetze per RRF aus breiter Topic- sowie `control`-/`purpose`-/`guidance`-Suche; ohne Topic-Katalog wird die generische Query wiederverwendet. Recall@12 erreicht read-only `68/78` (`87,2 %`, Primaer `33/38`, Sekundaer `35/40`). Prompt 1.3 fordert die kleinste hinreichende Menge und normalerweise einen Primaer- plus null bis zwei Sekundaercontrols. Human Review und maximal fuenf Vorschlaege bleiben verbindlich; kein Auto-Accept und keine Mapping-/DB-Mutation. Client-/Server-Typecheck, Local-only- und Designpruefung erfolgreich; Vitest und Vite-Build konnten wegen Sandbox-Verzeichniszugriff und erschoepftem Host-Freigabekontingent nicht erneut gestartet werden. Kein Importvertrag, Manifestbump oder Reimport. |
| 2026-09-17 | Phase 13D, ausdruecklicher Nutzerauftrag: ISO-Retrievalluecken fuer zehn PingCastle-Regeln nachgehaertet. `RegulatoryFindingQuery` v2 trennt die vollstaendige privacy-safe Modellquery von einer kurzen Embeddingquery; die oeffentliche Begriffstaxonomie deckt CVE-Remediation, Hardened Paths, LAPS, NetCease, Pre-Windows-2000-Zugriffsrechte, OU-Loeschschutz, AD-Subnetze, Kerberos Armoring und Legacy-NTLM ab. Der Finding-Retriever sucht ISO-27002-`control` und `purpose` getrennt, dedupliziert je Ranking auf Control-Ebene und fusioniert deterministisch per RRF (`k=60`). Aus 24 Controls werden hoechstens zwoelf evidenzgebundene Promptkandidaten; die Modellausgabe bleibt maximal fuenf und manuell reviewpflichtig. Retrievalversion `dense-section-rrf-control-grouping-1.2.0`; bestehende Runs bleiben append-only erhalten. Der rohe RRF-Wert bleibt als Rankingprovenienz gespeichert; die Reviewoberflaeche zeigt stattdessen den ordinalen Retrievalrang, weil ein Wert wie 0,03 weder Prozentwert noch kalibrierte Confidence ist. Ein eingecheckter oeffentlicher Vertrag prueft zehn Regeln und 17 Sollbeziehungen; read-only Livewiederholung gegen den bestehenden lokalen BGE-M3-/Qdrant-ISO-Index erreichte Recall@12 1,0. 1.329 Tests bestanden, zehn opt-in Integrationen uebersprungen; Local-only-/Designpruefung und Client-/Serverbuild erfolgreich. Keine DB-Mutation, kein Reimport und keine Mappingentscheidung angenommen. |
| 2026-09-16 | Phase 13D, ausdruecklicher Nutzerauftrag: Gefiltertes Result-DB-ISO-Batchmapping umgesetzt. Der gemeinsame Scope-Builder garantiert dieselbe vollstaendige Filter-/Timeslot-Abfrage wie Export JSON und entfernt Pagination sowie nicht autoritative Client-Scopewerte. Projektgebundene GET-/POST-APIs, Migration 027, Local-/PostgreSQL-Paritaet, Export und Loeschung speichern den normalisierten Scope, kanonischen Filterhash, Idempotenz, Lifecycle und Fortschritt; globale Regelmappings bleiben autoritativ. Stabile serverseitige Mehrseitenabfrage mit Total-Drift-Abbruch, deterministische `ruleKey`-Repraesentanten, erfolgreiche/aktive Wiederverwendung, einmaliger Failed-Retry, atomare Mappingclaims und eine gemeinsame Zwei-Pipeline-Queue sind implementiert. Result DB zeigt `Map all ISO controls`, Trefferzahl, Status und Zaehler, verhindert veraltete Filterantworten und behaelt Accept/Reject manuell in Work. Synthetische Regressionen decken alle Filterfamilien, Standardabweichungsfilter, Scopebereinigung, Mehrseitenlauf, Total-Drift, Ausschluesse, globale Deduplikation, Partial Failure, Retry/Reuse, Single-/Batch-Nebenlaeufigkeit und UI ab. `npm test`: 1.315 bestanden, zehn opt-in Integrationen uebersprungen; Local-only-/Designpruefung und beide Produktionsbuilds erfolgreich. Kein Manifestbump oder Reimport. Nach der Anwendung von 027 wurde dieselbe Datei faelschlich um eine reine Sortiersequenz erweitert; der Checksum-Schutz stoppte den Backendstart. Die Datei ist exakt auf den angewandten SHA-256-Stand `d89055c3e723b09d8d8be7f4225bfd08442f0d66e4e0b4ea3aa62ae6209e9ee9` zurueckgesetzt und mit einer Regression gesperrt, die Abfragen verwenden wieder den vorhandenen `created_at, id`-Index. Keine Migration 028 und keine weitere Datenbankmutation. Fuenf fokussierte Suites mit 16 Tests, Servertyppruefung und Serverbuild erfolgreich. Nach freigegebenem WSL-Neustart: kanonischer PostgreSQL-Container gesund, Port/Volume korrekt, Ledger 027 checksum-identisch, 028 nicht vorhanden und Backend-Health HTTP 200 mit PostgreSQL-Persistenz. |
| 2026-09-15 | Phase 12/13C, ausdruecklicher Nutzerauftrag: Nicht-personenbezogene PingCastle-Umgebungsaggregate fuer Result DB und AI Overview freigegeben. Ein reiner Ableitungsschritt (`derivePingCastleEnvironmentAggregates`) erzeugt aus bereits extrahierten Rohzellen stabil benannte Zaehl- und Konfigurationsbeobachtungen (Benutzer-/Computerkonten inkl. aktiviert/deaktiviert/aktiv und Passwort-laeuft-nie, Admin-Gruppen, Trusts, PKI-Zaehlungen, Passwortrichtlinie, LAPS-Praesenz, Entra-/krbtgt-Zustand), ohne Quellzellen, Identitaeten oder Zeitstempel zu veraendern. Die fail-closed Offenlegungsgrenze `result-db-supplemental-disclosure` erhaelt exakte Allowlist-Regeln je Property; `SAFE_ENVIRONMENT_PROPERTIES` gibt dieselben Aggregate an das Privacy-Safe-AI-JSON weiter. Verzeichnis-/Forest-/NetBIOS-Namen, bewerteter Host und alle Datums-/Zeitwerte bleiben ausdruecklich ausgeschlossen. PingCastle-Verarbeitungsvertrag 1.5.0 → 1.6.0 (`compatible`, `environment-context`, reimportRequired); 1.5.0 bleibt historisch und wird `reimport-recommended`, der freigegebene Dummy-Bericht benoetigt einen Reimport, um die Aggregate zu erhalten. 1.290 Tests bestanden, acht optionale PostgreSQL-Tests uebersprungen; Local-only-/Designpruefung und Client-/Server-Produktionsbuild erfolgreich. Neue synthetische Ableitungs-, Offenlegungs- und AI-Export-Tests; reale Werte nur inhaltsfrei bzw. aggregiert im vom Nutzer ausdruecklich freigegebenen Dummy geprueft. Zusatz: Die Umgebungsfakten im Privacy-Safe-JSON/AI-Prompt entfallen um die konstanten Felder `state` (bei offengelegten Werten stets `observed`) und `count` (nach Deduplizierung je Aggregat 1); reine Serialisierungsanpassung ueber bereits gespeicherte Felder ohne Reimportbedarf. |
| 2026-09-14 | Phasen 7/13C und 11: Work-LiteLLM-gestuetztes AI Overview umgesetzt. Der Backend-Service liest den kompletten Timeslot ueber das bestehende Privacy-Safe-Result-DB-JSON, hasht ihn deterministisch, zerlegt grosse Payloads nur zwischen vollstaendigen Eintraegen und konsolidiert sie mit maximal 64 explizit nicht wiederholten LiteLLM-Aufrufen. Der HTTPS-Client haelt URL, Modell und Bearer-Key ausschliesslich in Server-Secrets, lehnt lokale/unsichere Ziele und Redirects ab und begrenzt Zeit sowie Antwortbytes. Ausgabevalidierung prueft das vollstaendige Dashboard-Schema, Laengen, Roadmap 0-4, exakte Finding-/Gap-IDs und Environment-Properties. Same-Origin-GET/POST, Migration 024, lokale/PostgreSQL-Repositoryparitaet, Projekt-Export/-Loeschung, Idempotenz und ein aktiver Run je Scope sind ergaenzt. Client Info rendert alle Analyseabschnitte mit Generate/Regenerate, Polling nur bei aktiven Runs sowie stale/failed/disabled Zustaenden. Sicherheits- und Deploymentdokumentation beschreibt den genehmigten Corporate-Gateway-Vertrag. 1.262 synthetische Tests bestanden, acht optionale PostgreSQL-Tests uebersprungen; Local-only-/Designpruefung und beide Produktionsbuilds erfolgreich. Der inhaltsfreie Workstation-Livecheck erreichte Work-LiteLLM, erhielt aber HTTP 401; Providerbody und Secrets wurden nicht ausgegeben. Docker/kanonisches Backup, Migrationanwendung, PostgreSQL-Integration sowie erfolgreicher Workstation-/Server-Livecheck bleiben offen; keine Realdateninspektion oder Reimportaenderung. |

57
docs/REGULATORY_RAG.md
View File

@ -1,6 +1,6 @@
# Regulatory-control RAG with Qdrant
Status: Phase 13D one-finding and filtered Result DB batch vertical slice. The server prepares the licensed German ISO 27001/27002 corpus, indexes section-safe ISO 27002 semantic chunks plus ISO 27001 references, retrieves and groups candidates, requests grounded LiteLLM proposals, persists runs and reviewer decisions, and exposes manual proposal generation in Work and Result DB. Sparse retrieval, reranking, and confidence calibration remain out of scope.
Status: Phase 13D one-finding and filtered Result DB batch vertical slice. The server prepares the licensed German ISO 27001/27002 corpus, indexes section-safe ISO 27002 semantic chunks plus ISO 27001 references, derives problem and remediation searches only from privacy-safe finding text, combines dense and hashed-BM25 sparse rankings at control level, requests a grounded assessment for every shortlisted control, deterministically selects proposals, persists runs and reviewer decisions, and exposes manual proposal generation in Work and Result DB. Learned/model-based reranking and confidence calibration remain out of scope.
## Boundaries
@ -14,15 +14,18 @@ Status: Phase 13D one-finding and filtered Result DB batch vertical slice. The s
versioned regulatory corpus JSON
| validate and chunk
v
approved multilingual embedding service
| dense vectors
approved multilingual embedding service + local BM25 encoder
| dense + sparse vectors
v
Qdrant assessor_regulatory_chunks_v2
^
scanner finding -> public-signal query builder -> same embedding model -> dense candidates
-> group top 32 chunks into top 8 controls
-> optional future hybrid fusion -> optional reranker
-> constrained mapper -> human decision -> PostgreSQL audit record
scanner finding -> privacy-safe title + description + recommendation
-> problem query + optional remediation query + full-context section searches
-> dense/sparse control-level RRF per request
-> collapse control/purpose/guidance into one full-context control ranking
-> three-family control-level RRF -> 12-control shortlist
-> per-candidate assessment -> deterministic 0-1/0-3 selection
-> human decision -> PostgreSQL audit record
```
## Collection contract
@ -30,7 +33,7 @@ scanner finding -> public-signal query builder -> same embedding model -> dense
The active collection is `assessor_regulatory_chunks_v2`. It has two named vector slots:
- `dense`: the configured embedding dimension with cosine distance.
- `sparse`: reserved now so a later sparse/BM25-style encoder can be added without changing the retrieval abstractions.
- `sparse`: deterministic hashed BM25 weights generated locally from control titles and text.
Qdrant payloads use schema version 2 and contain:
@ -71,7 +74,7 @@ npm run db:up
npm run rag:ingest -- path/to/regulatory-corpus.json
```
The parser accepts encrypted documents only when no password is required and copying is permitted. It rejects scanned/empty, oversized, copy-prohibited, structurally unexpected, incomplete, duplicate, or mismatched catalogs. The build must contain the matching 93 controls with theme counts 37/8/14/34. It retains hashes, editions, control/section metadata, and page provenance without committing licensed text. Ingestion splits only inside a control section, embeds all chunks in bounded batches, and then replaces indexed chunks per `(corpusId, documentId)`. If Qdrant fails after deletion, rerun ingestion from the unchanged corpus JSON.
The parser accepts encrypted documents only when no password is required and copying is permitted. It rejects scanned/empty, oversized, copy-prohibited, structurally unexpected, incomplete, duplicate, or mismatched catalogs. The build must contain the matching 93 controls with theme counts 37/8/14/34. It retains hashes, editions, control/section metadata, and page provenance without committing licensed text. Ingestion splits only inside a control section, embeds all chunks in bounded batches, derives corpus-wide hashed BM25 vectors without a maintained topic vocabulary, and then replaces indexed chunks per `(corpusId, documentId)`. Collections populated before retrieval 1.4 must be re-ingested once so their existing points receive sparse vectors; dense retrieval remains available until then. If Qdrant fails after deletion, rerun ingestion from the unchanged corpus JSON.
## Configuration and trust
@ -107,10 +110,14 @@ Embedding is an external processing boundary. Regulatory text is sent during ing
## Finding query privacy boundary
`buildRegulatoryFindingQuery` accepts the existing `UnifiedResultProjection` and builds the query from the **shared privacy-safe finding projection** (`src/domain/privacy-safe-finding.ts`, `privacySafeFinding`) — the identical boundary used by the AI overview:
`buildRegulatoryFindingQuery` accepts the existing `UnifiedResultProjection` and builds related values from the **shared privacy-safe finding projection** (`src/domain/privacy-safe-finding.ts`, `privacySafeFinding`) — the identical boundary used by the AI overview:
- the rule-level **check text** — title, plus description and recommendation when present — which comes from the deduplicated finding *definition* (canonical key `tool:ruleId`) and describes the check, not the tenant; email addresses are redacted;
- a catalog Maester/PingCastle rule ID, a fixed public scanner name, and recognized public technical topics, all as optional query **enrichment**.
- a problem query containing the bounded title and description;
- an optional remediation query containing the same title and bounded recommendation;
- a full finding-text query used for section-specific candidate ranking and evidence retrieval.
Scanner name, source rule ID, rule catalogs and `public-security-topics.ts` do not enter regulatory retrieval or mapper context. The topic list remains a separate concern of the public web-search feature. Consequently, identical privacy-safe finding text produces identical regulatory queries regardless of whether the scanner is known, newly added or private.
The query never reads the **result table / occurrence data**: affected asset, source locator/path, per-object values, project/tenant identity, review owner, review text, original values, or provenance IDs. Because the query is built from the rule-level check text, **every scanner finding with a check title maps**; only non-scanner records and findings without a check title fail closed as `insufficient-public-context` and do not call the retriever. The local `sourceResultId` is returned only for in-process correlation and is not included in the embedding request.
@ -118,9 +125,11 @@ This is a technical minimization boundary. Activation still depends on the appro
## One-finding mapping workflow
The Work finding reader loads mapping state only after `Map ISO controls` is opened. Mappings are **global**: the finding contributes only its non-personal rule identity and check context, and the mapping is keyed by that identity plus the pipeline versions, so one rule is mapped once and every occurrence in every project reuses it. The API returns HTTP 202 and the UI polls while queued/running. The service retrieves 32 eligible ISO 27002 chunks, groups by maximum dense score, keeps eight controls, attaches up to two semantic passages and the matching ISO 27001 statement, and bounds the LiteLLM payload to 48,000 characters.
The Work finding reader loads mapping state only after `Map ISO controls` is opened. Mappings are **global**: the finding contributes only its non-personal rule identity and check context, and the mapping is keyed by that identity plus the pipeline versions, so one rule is mapped once and every occurrence in every project reuses it. The API returns HTTP 202 and the UI polls while queued/running.
The completion schema permits at most five `primary` or `supporting` proposals, advisory low/medium/high confidence, a bounded rationale, grounded evidence IDs, or an explicit `no_relevant_control`/`insufficient_context` outcome. Unknown controls, foreign evidence IDs, duplicates, malformed JSON, and oversized content fail closed. A reviewer can accept or reject any grounded top-eight candidate with an optional comment; each change appends a **global** decision event keyed by rule identity and control, using optimistic concurrency, shared across every project. Acceptance means relevant control, never compliance.
Each problem/remediation request obtains independent dense and BM25 sparse rankings and combines them by reciprocal-rank fusion (`k=60`). Fusion is keyed by control, not chunk: a control contributes at most once to each ranking and multiple chunks cannot crowd out other controls. Separate `control`, `purpose`, and `guidance` searches use the complete finding text; their control rankings are first collapsed into one full-context family ranking. The problem, optional remediation and full-context family rankings are then fused at control level to choose at most twelve controls. Thus section coverage can improve the order within the full-context family, but the three correlated sections contribute at most one final vote per control. The mapper receives the available semantic passages plus the matching ISO 27001 statement, while the LiteLLM payload remains bounded to 48,000 characters. Rank-derived values remain deterministic ordering signals; the reviewer UI shows ordinal retrieval rank because a value such as `0.03` is neither a percentage nor calibrated confidence. No learned or model-based reranker runs in this pipeline.
The model must assess every shortlisted control exactly once as `primary`, `supporting`, or `irrelevant`, with advisory low/medium/high confidence, a bounded rationale, and grounded evidence IDs. Missing controls, duplicates, unknown controls, foreign evidence IDs, malformed JSON, and oversized content fail closed. A deterministic reducer chooses the highest-confidence, earliest-ranked primary candidate, demotes any additional primary candidates into the supporting pool, retains at most three supporting candidates, permits supporting-only output, and derives `no_relevant_control` when every candidate is irrelevant. The persisted proposal schema and reviewer UI remain unchanged. A reviewer can accept or reject any grounded shortlisted candidate with an optional comment; each change appends a **global** decision event keyed by rule identity and control, using optimistic concurrency, shared across every project. Acceptance means relevant control, never compliance.
Routes:
@ -148,9 +157,9 @@ The filter snapshot is captured when Start is pressed and is never copied into e
On 2026-09-15 this check passed against Qdrant 1.19.1 in WSL: the collection was green, Dense used 1,024-dimensional cosine vectors, the Sparse slot existed, all 12 filter payload indexes existed, and one synthetic integration point was retrieved and then removed.
## Dense retrieval evaluation
## Retrieval evaluation
The evaluation CLI runs the configured dense retriever against a bounded, versioned labeled case file and reports control-level HitRate@K, Recall@K, and mean reciprocal rank. Queries can declare a language different from the expected control language so English-to-German and German-to-English cases remain explicit.
The evaluation CLI runs the configured hybrid retriever against a bounded, versioned labeled case file and reports control-level HitRate@K, Recall@K, and mean reciprocal rank. Queries can declare a language different from the expected control language so English-to-German and German-to-English cases remain explicit.
```json
{
@ -186,10 +195,24 @@ The checked-in [`tests/fixtures/regulatory-evaluation.synthetic.json`](../tests/
Live smoke result on 2026-09-15 with local llama.cpp build 10472, BGE-M3 FP16, and Qdrant 1.19.1: six EN/DE documents produced six 1,024-dimensional chunks. All three cross-language cases returned their expected control at rank 1, giving HitRate@3 `1.0`, Recall@3 `1.0`, and MRR `1.0`. This verifies wiring and basic language crossing only; three controls are far too small and distinct to estimate production retrieval quality.
The 2026-09-17 regression review used all 37 PingCastle findings and the 78 primary/supporting control relationships independently recorded in the review workbook. Candidate Recall@12 was `61/78` for retrieval 1.1, fell to `54/78` for the topic-led section retrieval 1.2, and reached `68/78` (`87.2%`) with retrieval 1.3's five generic reservations plus seven supplemental RRF controls. The 1.3 split was `33/38` primary relationships (`86.8%`) and `35/40` supporting relationships (`87.5%`). This is close to the requested approximately 90% human-review target, but it measures candidate availability only, not final mapper precision or accepted mapping accuracy. It is also a same-tool regression set rather than a representative new-scanner evaluation, so the generic reservation and unknown-scanner contract are retained as architectural safeguards rather than claiming general quality from PingCastle alone.
### Repeatable manual comparison
After a complete mapping run, create the Excel comparison with:
```text
npm run rag:compare-mappings
```
The read-only command takes the active retrieval version from `REGULATORY_RETRIEVAL_VERSION`, selects the most recent prior version covering every baseline scanner, reads the canonical local PostgreSQL mapping history and writes one workbook below `outputs/regulatory-mapping/`. Use `--current-version`, `--previous-version`, `--baseline` or `--output` to override those defaults. It stops without creating a workbook when the active version does not contain every baseline rule.
The versioned baseline at `evaluation/regulatory-mapping/manual-baseline.json` contains only scanner rule IDs and independently reviewed primary/supporting control roles. It is evaluation data for the human review workflow, not a product mapping source or CI assertion. Update or replace it deliberately when the regulatory corpus or manual assessment changes. Permanent tests cover only generic comparison mechanics and never assert scanner-specific ISO control IDs.
## Extension seams
- `RegulatoryCandidateRetriever` is the replaceable candidate stage. The current implementation is dense Qdrant retrieval; a later hybrid implementation can combine dense and sparse/BM25 candidates here.
- `RegulatoryReranker` is optional and receives only the bounded candidate set. Without one, dense ordering is retained.
- `RegulatoryCandidateRetriever` is the replaceable candidate stage. The current implementation fuses dense Qdrant retrieval with locally encoded sparse/BM25 retrieval.
- `RegulatoryReranker` is optional and receives only the bounded candidate set. Without one, the fused deterministic ordering is retained.
- `RegulatoryRetriever` is the consumer-facing contract. Mapping code should depend on it, not on Qdrant.
The filtered manual Result DB batch was activated by explicit user decision before the earlier quality-gate sequence. That decision changes orchestration only: broaden public-signal coverage through reviewed catalogs, establish a representative independently labeled EN/DE baseline, and evaluate sparse/hybrid retrieval, reranking, and confidence calibration before claiming production retrieval quality or adding automatic decisions. Single and batch proposals remain human-reviewed.

1
package.json
View File

@ -17,6 +17,7 @@
"rag:ingest": "npm run build:server && node --env-file-if-exists=.env.postgres dist-server/regulatory-ingest.mjs",
"rag:prepare-iso": "npm run build:server && node --env-file-if-exists=.env.postgres dist-server/regulatory-prepare-iso.mjs",
"rag:evaluate": "npm run build:server && node --env-file-if-exists=.env.postgres dist-server/regulatory-evaluate.mjs",
"rag:compare-mappings": "node --env-file-if-exists=.env.postgres scripts/compare-regulatory-mappings.mjs",
"test:postgres": "vitest run --config vitest.config.ts tests/postgres/postgres-repository.integration.test.ts && vitest run --config vitest.config.ts tests/postgres/gpo-backup-management.integration.test.ts tests/postgres/policy-analysis.integration.test.ts",
"test": "vitest run --config vitest.config.ts && node tests/local-only-check.mjs && node tests/design-language-check.mjs",
"test:security": "node tests/local-only-check.mjs",

594
scripts/compare-regulatory-mappings.mjs Normal file
View File

@ -0,0 +1,594 @@
import fs from 'node:fs/promises';
import os from 'node:os';
import path from 'node:path';
import { createRequire } from 'node:module';
import { fileURLToPath, pathToFileURL } from 'node:url';
import pg from 'pg';
const { Client } = pg;
const scriptDir = path.dirname(fileURLToPath(import.meta.url));
const projectRoot = path.resolve(scriptDir, '..');
const defaultBaselinePath = path.join(projectRoot, 'evaluation', 'regulatory-mapping', 'manual-baseline.json');
const retrievalVersionSourcePath = path.join(projectRoot, 'src', 'domain', 'regulatory-mapping.ts');
const colors = {
navy: '#17365D', paleBlue: '#D9EAF7', blue: '#DDEBF7', white: '#FFFFFF',
text: '#1F2937', muted: '#5B6573', line: '#D0D7DE', input: '#FFF2CC',
green: '#E2F0D9', greenText: '#2F6B2F', amber: '#FFF2CC', amberText: '#8A5A00',
red: '#F4CCCC', redText: '#9C0006', gray: '#E7E6E6'
};
const font = 'Arial';
function usage() {
return `Usage: npm run rag:compare-mappings -- [options]
Reads regulatory mapping runs from the canonical local PostgreSQL database and
creates an Excel comparison against the versioned manual evaluation baseline.
Options:
--current-version <version> Version to evaluate. Defaults to the source constant.
--previous-version <version> Comparison version. Defaults to the latest common prior version.
--baseline <path> Baseline JSON. Defaults to evaluation/regulatory-mapping/manual-baseline.json.
--output <path> Output XLSX path. Defaults to outputs/regulatory-mapping/.
--help Show this help.
`;
}
function parseArgs(argv) {
const options = {};
for (let index = 0; index < argv.length; index += 1) {
const argument = argv[index];
if (argument === '--help') options.help = true;
else if (['--current-version', '--previous-version', '--baseline', '--output'].includes(argument)) {
const value = argv[index + 1];
if (!value || value.startsWith('--')) throw new Error(`${argument} requires a value.`);
options[argument.slice(2).replace(/-([a-z])/g, (_, letter) => letter.toUpperCase())] = value;
index += 1;
} else throw new Error(`Unknown option: ${argument}`);
}
return options;
}
function controlId(value) {
return String(value ?? '').trim().replace(/^A\./i, '');
}
function displayControl(value) {
const normalized = controlId(value);
return normalized ? `A.${normalized}` : 'None';
}
function displayControls(values) {
return values.length ? values.map(displayControl).join(', ') : 'None';
}
function validateBaseline(value) {
if (!value || value.schemaVersion !== 1 || !Array.isArray(value.scanners) || !value.scanners.length) {
throw new Error('The mapping baseline must use schemaVersion 1 and contain scanners.');
}
const tools = new Set();
for (const scanner of value.scanners) {
if (!scanner.sourceTool || !scanner.label || !Array.isArray(scanner.findings) || !scanner.findings.length) {
throw new Error('Every baseline scanner requires sourceTool, label and findings.');
}
if (tools.has(scanner.sourceTool)) throw new Error(`Duplicate baseline scanner ${scanner.sourceTool}.`);
tools.add(scanner.sourceTool);
const rules = new Set();
for (const finding of scanner.findings) {
if (!finding.sourceRuleId || rules.has(finding.sourceRuleId)) {
throw new Error(`Missing or duplicate rule in ${scanner.sourceTool}.`);
}
rules.add(finding.sourceRuleId);
if (finding.primary !== null && finding.primary !== undefined && !/^\d+\.\d+$/.test(controlId(finding.primary))) {
throw new Error(`Invalid primary control for ${scanner.sourceTool}:${finding.sourceRuleId}.`);
}
if (!Array.isArray(finding.supporting)) throw new Error(`Supporting controls must be an array for ${scanner.sourceTool}:${finding.sourceRuleId}.`);
const relationships = [finding.primary, ...finding.supporting].filter(Boolean).map(controlId);
if (relationships.some(id => !/^\d+\.\d+$/.test(id)) || new Set(relationships).size !== relationships.length) {
throw new Error(`Invalid or duplicate controls for ${scanner.sourceTool}:${finding.sourceRuleId}.`);
}
finding.primary = finding.primary ? controlId(finding.primary) : null;
finding.supporting = finding.supporting.map(controlId);
}
}
return value;
}
async function configuredCurrentVersion() {
const source = await fs.readFile(retrievalVersionSourcePath, 'utf8');
const match = source.match(/REGULATORY_RETRIEVAL_VERSION\s*=\s*'([^']+)'/);
if (!match) throw new Error('Could not read REGULATORY_RETRIEVAL_VERSION from the source.');
return match[1];
}
function loopbackHost(host) {
return ['127.0.0.1', 'localhost', '::1', '[::1]'].includes(host.toLowerCase());
}
function databaseOptions() {
const explicit = process.env.ASSESSOR_DATABASE_URL?.trim();
if (explicit) {
const url = new URL(explicit);
if (!['postgres:', 'postgresql:'].includes(url.protocol) || !loopbackHost(url.hostname)) {
throw new Error('The comparison script only connects to local PostgreSQL.');
}
return { connectionString: explicit };
}
const host = process.env.ASSESSOR_POSTGRES_HOST?.trim() || '127.0.0.1';
if (!loopbackHost(host)) throw new Error('The comparison script only connects to local PostgreSQL.');
const password = process.env.ASSESSOR_POSTGRES_PASSWORD?.trim();
if (!password) throw new Error('ASSESSOR_POSTGRES_PASSWORD is required. Run through npm so .env.postgres is loaded.');
return {
host,
port: Number(process.env.ASSESSOR_POSTGRES_PORT || 5432),
database: process.env.ASSESSOR_POSTGRES_DB || 'assessor',
user: process.env.ASSESSOR_POSTGRES_USER || 'assessor',
password
};
}
async function findPreviousVersion(client, sourceTools, currentVersion) {
const result = await client.query(`
SELECT retrieval_version, MAX(created_at) AS latest
FROM assessor_regulatory_mappings
WHERE source_tool = ANY($1::text[]) AND retrieval_version <> $2
GROUP BY retrieval_version
HAVING COUNT(DISTINCT source_tool) = $3
ORDER BY latest DESC, retrieval_version DESC
LIMIT 1
`, [sourceTools, currentVersion, sourceTools.length]);
if (!result.rowCount) throw new Error('No prior retrieval version covers every baseline scanner. Pass --previous-version explicitly.');
return result.rows[0].retrieval_version;
}
async function loadRuns(client, sourceTools, versions) {
const result = await client.query(`
SELECT source_tool, source_rule_id, retrieval_version, prompt_version, run_status,
corpus_version, created_at, updated_at, data
FROM assessor_regulatory_mappings
WHERE source_tool = ANY($1::text[]) AND retrieval_version = ANY($2::text[])
ORDER BY updated_at, id
`, [sourceTools, versions]);
const runs = new Map();
for (const row of result.rows) {
runs.set(`${row.source_tool}|${row.retrieval_version}|${row.source_rule_id}`, row);
}
return { rows: result.rows, runs };
}
function expectedRoles(finding) {
return new Map([
...(finding.primary ? [[finding.primary, 'primary']] : []),
...finding.supporting.map(id => [id, 'supporting'])
]);
}
function evaluateRun(run, finding) {
const candidates = new Map();
for (const [index, candidate] of (run?.data?.candidates ?? []).entries()) {
const id = controlId(candidate.controlId ?? candidate.annexAControlId);
if (id && !candidates.has(id)) candidates.set(id, { rank: index + 1, score: candidate.score });
}
const selected = new Map();
if (run?.run_status === 'succeeded') {
for (const mapping of run?.data?.result?.mappings ?? []) {
const id = controlId(mapping.controlId);
if (id && ['primary', 'supporting'].includes(mapping.relationship)) {
selected.set(id, { role: mapping.relationship, confidence: mapping.confidence ?? '' });
}
}
}
const expected = expectedRoles(finding);
const expectedIds = [...expected.keys()];
const retrieved = expectedIds.filter(id => candidates.has(id));
const selectedExpected = expectedIds.filter(id => selected.has(id));
const exact = expectedIds.filter(id => selected.get(id)?.role === expected.get(id));
const retrievalGaps = expectedIds.filter(id => !candidates.has(id));
const selectionGaps = expectedIds.filter(id => candidates.has(id) && !selected.has(id));
const roleMismatches = selectedExpected.filter(id => selected.get(id)?.role !== expected.get(id));
const extras = [...selected.keys()].filter(id => !expected.has(id));
const primary = [...selected.entries()].find(([, value]) => value.role === 'primary')?.[0] ?? null;
const supporting = [...selected.entries()].filter(([, value]) => value.role === 'supporting').map(([id]) => id);
const status = run?.run_status ?? 'missing';
let verdict;
if (!run) verdict = 'Missing run';
else if (status !== 'succeeded') verdict = 'Failed';
else if (!retrievalGaps.length && !selectionGaps.length && !roleMismatches.length && !extras.length) verdict = 'Exact';
else if (!retrievalGaps.length && !selectionGaps.length && !roleMismatches.length) verdict = 'Exact + extras';
else if (!retrievalGaps.length && !selectionGaps.length && roleMismatches.length) verdict = extras.length ? 'Role mismatch + extras' : 'Role mismatch';
else verdict = extras.length ? 'Incomplete + extras' : 'Incomplete';
return {
run, status, candidates, selected, retrieved, selectedExpected, exact,
retrievalGaps, selectionGaps, roleMismatches, extras, primary, supporting, verdict
};
}
function changeFor(previous, current) {
const statusRank = value => value.status === 'succeeded' ? 2 : value.status === 'failed' ? 1 : 0;
const statusDelta = statusRank(current) - statusRank(previous);
if (statusDelta > 0) return 'Improved';
if (statusDelta < 0) return 'Regressed';
const deltas = [
current.retrieved.length - previous.retrieved.length,
current.selectedExpected.length - previous.selectedExpected.length,
current.exact.length - previous.exact.length,
previous.extras.length - current.extras.length
];
const improved = deltas.some(value => value > 0);
const regressed = deltas.some(value => value < 0);
return improved && regressed ? 'Mixed' : improved ? 'Improved' : regressed ? 'Regressed' : 'Unchanged';
}
function relationshipComparison(evaluation, control, manualRole) {
if (!evaluation.run) return 'Missing run';
if (!manualRole) return evaluation.selected.has(control) ? 'Extra mapping' : '—';
if (!evaluation.candidates.has(control)) return 'Retrieval gap';
if (evaluation.status !== 'succeeded') return 'Run failed after retrieval';
const selected = evaluation.selected.get(control);
if (!selected) return 'Selection gap';
return selected.role === manualRole ? 'Match' : 'Role mismatch';
}
function buildComparison(baseline, runs, previousVersion, currentVersion) {
const findings = [];
for (const scanner of baseline.scanners) {
for (const finding of scanner.findings) {
const previousRun = runs.get(`${scanner.sourceTool}|${previousVersion}|${finding.sourceRuleId}`);
const currentRun = runs.get(`${scanner.sourceTool}|${currentVersion}|${finding.sourceRuleId}`);
const previous = evaluateRun(previousRun, finding);
const current = evaluateRun(currentRun, finding);
findings.push({ scanner, finding, previous, current, change: changeFor(previous, current) });
}
}
return findings;
}
function aggregate(items, key) {
const totals = {
findings: items.length, succeeded: 0, primaryExpected: 0, primaryRetrieved: 0, primaryExact: 0,
supportingExpected: 0, supportingRetrieved: 0, supportingExact: 0,
expected: 0, retrieved: 0, selected: 0, exact: 0, extras: 0
};
for (const item of items) {
const evaluation = item[key];
const primaryExpected = item.finding.primary ? 1 : 0;
totals.succeeded += Number(evaluation.status === 'succeeded');
totals.primaryExpected += primaryExpected;
totals.primaryRetrieved += Number(Boolean(item.finding.primary && evaluation.candidates.has(item.finding.primary)));
totals.primaryExact += Number(Boolean(item.finding.primary && evaluation.selected.get(item.finding.primary)?.role === 'primary'));
totals.supportingExpected += item.finding.supporting.length;
totals.supportingRetrieved += item.finding.supporting.filter(id => evaluation.candidates.has(id)).length;
totals.supportingExact += item.finding.supporting.filter(id => evaluation.selected.get(id)?.role === 'supporting').length;
totals.expected += primaryExpected + item.finding.supporting.length;
totals.retrieved += evaluation.retrieved.length;
totals.selected += evaluation.selectedExpected.length;
totals.exact += evaluation.exact.length;
totals.extras += evaluation.extras.length;
}
return totals;
}
function versionTag(value) {
const match = value.match(/(\d+\.\d+\.\d+)$/);
return match ? `v${match[1]}` : value.replace(/[^a-z0-9.-]+/gi, '_');
}
async function loadArtifactTool() {
const candidates = [
process.env.ASSESSOR_ARTIFACT_TOOL_NODE_MODULES,
path.join(os.homedir(), '.cache', 'codex-runtimes', 'codex-primary-runtime', 'dependencies', 'node', 'node_modules')
].filter(Boolean);
for (const nodeModules of candidates) {
try {
await fs.access(path.join(nodeModules, '@oai', 'artifact-tool', 'package.json'));
const runtimeRequire = createRequire(path.join(nodeModules, '_assessor-artifact-resolver.cjs'));
return await import(pathToFileURL(runtimeRequire.resolve('@oai/artifact-tool')).href);
} catch {
// Try the next configured runtime.
}
}
throw new Error('Could not locate @oai/artifact-tool. Set ASSESSOR_ARTIFACT_TOOL_NODE_MODULES to the bundled node_modules directory.');
}
function styleHeader(range) {
range.format = {
fill: colors.navy,
font: { name: font, size: 10, bold: true, color: colors.white },
horizontalAlignment: 'center', verticalAlignment: 'center', wrapText: true,
borders: { insideVertical: { style: 'thin', color: colors.white }, bottom: { style: 'thin', color: colors.navy } }
};
range.format.rowHeight = 30;
}
function titleBlock(sheet, title, subtitle, lastColumn) {
sheet.getRange('A2').values = [[title]];
sheet.getRange('A2').format.font = { name: font, size: 14, bold: true, color: colors.navy };
sheet.getRange('A3').values = [[subtitle]];
sheet.getRange(`A3:${lastColumn}3`).format.font = { name: font, size: 10, italic: true, color: colors.muted };
sheet.getRange(`A4:${lastColumn}4`).format.borders = { bottom: { style: 'thin', color: colors.navy } };
}
function applyBase(sheet, range) {
sheet.getRange(range).format.font = { name: font, size: 10, color: colors.text };
sheet.getRange(range).format.verticalAlignment = 'center';
}
function applyWidths(sheet, widths, rowCount) {
widths.forEach((width, index) => {
sheet.getRangeByIndexes(0, index, Math.max(rowCount, 1), 1).format.columnWidth = width;
});
}
async function buildWorkbook(artifactTool, comparison, metadata) {
const { SpreadsheetFile, Workbook } = artifactTool;
const workbook = Workbook.create();
const summary = workbook.worksheets.add('Summary');
const findingSheet = workbook.worksheets.add('Finding comparison');
const relationshipSheet = workbook.worksheets.add('Relationship detail');
const method = workbook.worksheets.add('Method');
for (const sheet of [summary, findingSheet, relationshipSheet, method]) sheet.showGridLines = false;
summary.tabColor = colors.navy;
findingSheet.tabColor = '#5B9BD5';
relationshipSheet.tabColor = '#ED7D31';
method.tabColor = '#7F8C8D';
const groups = [
...metadata.baseline.scanners.map(scanner => ({ label: scanner.label, items: comparison.filter(item => item.scanner.sourceTool === scanner.sourceTool) })),
{ label: 'All scanners', items: comparison }
];
const summaryRows = [];
const metrics = [
['Runs succeeded', 'findings', 'succeeded'],
['Primary controls retrieved', 'primaryExpected', 'primaryRetrieved'],
['Primary controls exact role', 'primaryExpected', 'primaryExact'],
['Supporting controls retrieved', 'supportingExpected', 'supportingRetrieved'],
['Supporting controls exact role', 'supportingExpected', 'supportingExact'],
['All expected relationships retrieved', 'expected', 'retrieved'],
['All expected relationships selected', 'expected', 'selected'],
['All expected relationships exact role', 'expected', 'exact'],
['Extra selected relationships', null, 'extras']
];
for (const group of groups) {
const previous = aggregate(group.items, 'previous');
const current = aggregate(group.items, 'current');
for (const [label, denominatorKey, valueKey] of metrics) {
const expected = denominatorKey ? previous[denominatorKey] : null;
summaryRows.push([
group.label, label, expected, previous[valueKey], expected ? previous[valueKey] / expected : null,
current[valueKey], expected ? current[valueKey] / expected : null, current[valueKey] - previous[valueKey]
]);
}
}
titleBlock(summary, 'ISO mapping comparison', `${metadata.previousVersion} compared with ${metadata.currentVersion}.`, 'N');
summary.getRange('A6:H6').values = [[
'Scanner', 'Metric', 'Expected', 'Previous', 'Previous rate', 'Current', 'Current rate', 'Delta'
]];
styleHeader(summary.getRange('A6:H6'));
summary.getRange('A7').write(summaryRows);
const summaryEnd = 6 + summaryRows.length;
summary.getRange(`E7:E${summaryEnd}`).format.numberFormat = '0.0%';
summary.getRange(`G7:G${summaryEnd}`).format.numberFormat = '0.0%';
summary.getRange(`A7:H${summaryEnd}`).format.rowHeight = 22;
summary.getRange(`A7:H${summaryEnd}`).format.wrapText = true;
summary.tables.add(`A6:H${summaryEnd}`, true, 'MappingMetricComparison').style = 'TableStyleMedium2';
summary.getRange('J6:N6').values = [['Scanner', 'Improved', 'Regressed', 'Mixed', 'Unchanged']];
styleHeader(summary.getRange('J6:N6'));
const changeRows = groups.map(group => [
group.label,
group.items.filter(item => item.change === 'Improved').length,
group.items.filter(item => item.change === 'Regressed').length,
group.items.filter(item => item.change === 'Mixed').length,
group.items.filter(item => item.change === 'Unchanged').length
]);
summary.getRange('J7').write(changeRows);
summary.tables.add(`J6:N${6 + changeRows.length}`, true, 'MappingFindingChanges').style = 'TableStyleMedium2';
applyBase(summary, `A1:N${summaryEnd}`);
styleHeader(summary.getRange('A6:H6'));
styleHeader(summary.getRange('J6:N6'));
applyWidths(summary, [18, 42, 12, 12, 14, 12, 14, 12, 3, 18, 12, 12, 12, 12], summaryEnd);
const findingRows = comparison.map(item => [
item.scanner.label,
item.finding.sourceRuleId,
displayControl(item.finding.primary),
displayControls(item.finding.supporting),
item.previous.status,
displayControl(item.previous.primary),
displayControls(item.previous.supporting),
item.current.status,
displayControl(item.current.primary),
displayControls(item.current.supporting),
item.previous.verdict,
item.current.verdict,
item.change,
displayControls(item.previous.retrievalGaps),
displayControls(item.current.retrievalGaps),
displayControls(item.previous.selectionGaps),
displayControls(item.current.selectionGaps),
displayControls(item.previous.roleMismatches),
displayControls(item.current.roleMismatches),
displayControls(item.previous.extras),
displayControls(item.current.extras),
[...item.current.candidates.keys()].map((id, index) => `${index + 1}. ${displayControl(id)}`).join(', '),
'',
''
]);
titleBlock(findingSheet, 'Finding comparison', 'One row per baseline rule. Yellow columns are reserved for human review.', 'X');
const findingHeaders = [
'Scanner', 'Rule ID', 'Manual primary', 'Manual supporting', 'Previous status', 'Previous primary',
'Previous supporting', 'Current status', 'Current primary', 'Current supporting', 'Previous verdict',
'Current verdict', 'Change', 'Previous retrieval gaps', 'Current retrieval gaps',
'Previous selection gaps', 'Current selection gaps', 'Previous role mismatches', 'Current role mismatches',
'Previous extras', 'Current extras', 'Current 12-control shortlist', 'Your decision', 'Your notes'
];
findingSheet.getRange('A6:X6').values = [findingHeaders];
styleHeader(findingSheet.getRange('A6:X6'));
findingSheet.getRange('A7').write(findingRows);
const findingEnd = 6 + findingRows.length;
applyBase(findingSheet, `A1:X${findingEnd}`);
styleHeader(findingSheet.getRange('A6:X6'));
findingSheet.getRange(`A7:X${findingEnd}`).format.wrapText = true;
findingSheet.getRange(`A7:X${findingEnd}`).format.verticalAlignment = 'top';
findingSheet.getRange(`A7:X${findingEnd}`).format.rowHeight = 48;
findingSheet.getRange(`W7:X${findingEnd}`).format.fill = colors.input;
findingSheet.getRange(`W7:W${findingEnd}`).dataValidation = { rule: { type: 'list', values: ['Accept', 'Reject', 'Needs discussion'] } };
const changeRange = findingSheet.getRange(`M7:M${findingEnd}`);
changeRange.conditionalFormats.add('containsText', { text: 'Improved', format: { fill: colors.green, font: { color: colors.greenText, bold: true } } });
changeRange.conditionalFormats.add('containsText', { text: 'Regressed', format: { fill: colors.red, font: { color: colors.redText, bold: true } } });
changeRange.conditionalFormats.add('containsText', { text: 'Mixed', format: { fill: colors.amber, font: { color: colors.amberText, bold: true } } });
findingSheet.tables.add(`A6:X${findingEnd}`, true, 'MappingFindingComparison').style = 'TableStyleMedium2';
findingSheet.freezePanes.freezeRows(6);
findingSheet.freezePanes.freezeColumns(2);
applyWidths(findingSheet, [16, 18, 18, 30, 16, 18, 30, 16, 18, 30, 22, 22, 14, 28, 28, 28, 28, 28, 28, 28, 28, 72, 20, 40], findingEnd);
const relationshipRows = [];
for (const item of comparison) {
const manual = expectedRoles(item.finding);
const controls = [...manual.keys()];
const extras = [...new Set([...item.previous.selected.keys(), ...item.current.selected.keys()])]
.filter(id => !manual.has(id))
.sort((left, right) => left.localeCompare(right, 'en', { numeric: true }));
for (const id of [...controls, ...extras]) {
const manualRole = manual.get(id) ?? '';
const previousSelected = item.previous.selected.get(id);
const currentSelected = item.current.selected.get(id);
relationshipRows.push([
item.scanner.label,
item.finding.sourceRuleId,
displayControl(id),
manualRole || '—',
previousSelected?.role ?? '—',
currentSelected?.role ?? '—',
item.previous.candidates.get(id)?.rank ?? null,
item.current.candidates.get(id)?.rank ?? null,
relationshipComparison(item.previous, id, manualRole),
relationshipComparison(item.current, id, manualRole),
currentSelected?.confidence ?? '—',
item.change
]);
}
}
titleBlock(relationshipSheet, 'Relationship detail', 'Manual, previous and current roles with shortlist availability.', 'L');
relationshipSheet.getRange('A6:L6').values = [[
'Scanner', 'Rule ID', 'Control', 'Manual role', 'Previous role', 'Current role',
'Previous candidate rank', 'Current candidate rank', 'Previous comparison', 'Current comparison',
'Current confidence', 'Finding change'
]];
styleHeader(relationshipSheet.getRange('A6:L6'));
relationshipSheet.getRange('A7').write(relationshipRows);
const relationshipEnd = 6 + relationshipRows.length;
applyBase(relationshipSheet, `A1:L${relationshipEnd}`);
styleHeader(relationshipSheet.getRange('A6:L6'));
relationshipSheet.getRange(`A7:L${relationshipEnd}`).format.rowHeight = 24;
relationshipSheet.getRange(`A7:L${relationshipEnd}`).format.wrapText = true;
relationshipSheet.tables.add(`A6:L${relationshipEnd}`, true, 'MappingRelationshipDetail').style = 'TableStyleMedium2';
relationshipSheet.freezePanes.freezeRows(6);
relationshipSheet.freezePanes.freezeColumns(2);
applyWidths(relationshipSheet, [16, 18, 14, 16, 16, 16, 18, 18, 26, 26, 18, 16], relationshipEnd);
titleBlock(method, 'Method', 'Inputs, version selection and interpretation.', 'H');
method.getRange('A6:B6').values = [['Item', 'Value']];
styleHeader(method.getRange('A6:B6'));
const promptVersions = version => [...new Set(metadata.rows.filter(row => row.retrieval_version === version).map(row => row.prompt_version))].sort().join(', ');
const corpusVersions = version => [...new Set(metadata.rows.filter(row => row.retrieval_version === version).map(row => row.corpus_version))].sort().join(', ');
const methodRows = [
['Generated', new Date(metadata.generatedAt)],
['Manual baseline', metadata.baseline.baselineId],
['Previous retrieval version', metadata.previousVersion],
['Previous prompt version(s)', promptVersions(metadata.previousVersion)],
['Previous corpus version(s)', corpusVersions(metadata.previousVersion)],
['Current retrieval version', metadata.currentVersion],
['Current prompt version(s)', promptVersions(metadata.currentVersion)],
['Current corpus version(s)', corpusVersions(metadata.currentVersion)],
['Baseline findings', comparison.length],
['Baseline relationships', comparison.reduce((total, item) => total + Number(Boolean(item.finding.primary)) + item.finding.supporting.length, 0)],
['Comparison rule', 'Improved/regressed requires one-sided metric movement. Opposing recall, role or extra-control changes are Mixed.'],
['Retrieval interpretation', 'Candidate ranks and RRF ordering are comparison signals, not probabilities or confidence percentages.'],
['Governance', 'This workbook supports human accept/reject review. The manual baseline is not a CI assertion or product rule.'],
['Privacy', 'The baseline stores scanner rule IDs and control roles only. Finding text and tenant values are excluded from this workbook.']
];
method.getRange('A7').write(methodRows);
const methodEnd = 6 + methodRows.length;
applyBase(method, `A1:H${methodEnd}`);
styleHeader(method.getRange('A6:B6'));
method.getRange(`A7:B${methodEnd}`).format.wrapText = true;
method.getRange(`A7:B${methodEnd}`).format.rowHeight = 32;
method.getRange('B7').format.numberFormat = 'yyyy-mm-dd hh:mm';
method.tables.add(`A6:B${methodEnd}`, true, 'MappingComparisonMethod').style = 'TableStyleMedium2';
applyWidths(method, [30, 100, 14, 14, 14, 14, 14, 14], methodEnd);
workbook.recalculate();
return { workbook, SpreadsheetFile };
}
async function main() {
const options = parseArgs(process.argv.slice(2));
if (options.help) {
console.log(usage());
return;
}
const baselinePath = path.resolve(projectRoot, options.baseline ?? defaultBaselinePath);
const baseline = validateBaseline(JSON.parse(await fs.readFile(baselinePath, 'utf8')));
const currentVersion = options.currentVersion ?? await configuredCurrentVersion();
const client = new Client(databaseOptions());
await client.connect();
try {
const sourceTools = baseline.scanners.map(scanner => scanner.sourceTool);
const previousVersion = options.previousVersion ?? await findPreviousVersion(client, sourceTools, currentVersion);
if (previousVersion === currentVersion) throw new Error('Previous and current retrieval versions must differ.');
const loaded = await loadRuns(client, sourceTools, [previousVersion, currentVersion]);
const missingCurrent = baseline.scanners.flatMap(scanner => scanner.findings
.filter(finding => !loaded.runs.has(`${scanner.sourceTool}|${currentVersion}|${finding.sourceRuleId}`))
.map(finding => `${scanner.label}:${finding.sourceRuleId}`));
if (missingCurrent.length) {
const counts = baseline.scanners.map(scanner => {
const missing = scanner.findings.filter(finding => !loaded.runs.has(`${scanner.sourceTool}|${currentVersion}|${finding.sourceRuleId}`)).length;
return `${scanner.label} ${missing}/${scanner.findings.length}`;
}).join(', ');
throw new Error(`Current version ${currentVersion} is incomplete (${counts}). Run the mappings before comparing.`);
}
const comparison = buildComparison(baseline, loaded.runs, previousVersion, currentVersion);
const generatedAt = new Date().toISOString();
const defaultOutput = path.join(
projectRoot,
'outputs',
'regulatory-mapping',
`ISO_Mapping_${versionTag(currentVersion)}_vs_${versionTag(previousVersion)}_Comparison.xlsx`
);
const outputPath = path.resolve(projectRoot, options.output ?? defaultOutput);
const artifactTool = await loadArtifactTool();
const built = await buildWorkbook(artifactTool, comparison, {
baseline, previousVersion, currentVersion, generatedAt, rows: loaded.rows
});
await fs.mkdir(path.dirname(outputPath), { recursive: true });
const output = await built.SpreadsheetFile.exportXlsx(built.workbook);
await output.save(outputPath);
await fs.rm(`${outputPath}.inspect.ndjson`, { force: true });
console.log(`Created ${path.relative(projectRoot, outputPath)}.`);
for (const scanner of baseline.scanners) {
const items = comparison.filter(item => item.scanner.sourceTool === scanner.sourceTool);
const previous = aggregate(items, 'previous');
const current = aggregate(items, 'current');
console.log(`${scanner.label}: retrieved ${previous.retrieved}/${previous.expected} -> ${current.retrieved}/${current.expected}; exact roles ${previous.exact}/${previous.expected} -> ${current.exact}/${current.expected}; extras ${previous.extras} -> ${current.extras}.`);
}
} finally {
await client.end();
}
}
const isMain = process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url);
if (isMain) main().catch(error => {
console.error(error instanceof Error ? error.message : String(error));
process.exitCode = 1;
});
export {
aggregate,
buildComparison,
changeFor,
controlId,
evaluateRun,
parseArgs,
validateBaseline
};

49
server/regulatory/qdrant-regulatory-index.ts
View File

@ -18,12 +18,19 @@ export interface QdrantEnvironment {
export interface IndexedRegulatoryChunk {
chunk: RegulatoryChunk;
embedding: number[];
sparseVector?: RegulatorySparseVector;
}
export interface RegulatorySparseVector {
indices: number[];
values: number[];
}
export interface RegulatoryChunkIndex {
ensureCollection(): Promise<void>;
replaceDocuments(chunks: IndexedRegulatoryChunk[]): Promise<void>;
searchDense(vector: number[], limit: number, filters?: RegulatoryRetrievalFilters): Promise<RegulatoryRetrievalCandidate[]>;
searchSparse(vector: RegulatorySparseVector, limit: number, filters?: RegulatoryRetrievalFilters): Promise<RegulatoryRetrievalCandidate[]>;
list(filters: RegulatoryRetrievalFilters, limit: number): Promise<RegulatoryChunk[]>;
}
@ -91,7 +98,10 @@ export class QdrantRegulatoryIndex implements RegulatoryChunkIndex {
async replaceDocuments(chunks: IndexedRegulatoryChunk[]): Promise<void> {
if (!chunks.length) return;
for (const entry of chunks) validateEmbedding(entry.embedding, this.options.dimensions);
for (const entry of chunks) {
validateEmbedding(entry.embedding, this.options.dimensions);
if (entry.sparseVector) validateSparseVector(entry.sparseVector);
}
const documents = new Map<string, { corpusId: string; documentId: string }>();
for (const { chunk } of chunks) documents.set(`${chunk.corpusId}\0${chunk.documentId}`, { corpusId: chunk.corpusId, documentId: chunk.documentId });
for (const document of documents.values()) {
@ -102,9 +112,12 @@ export class QdrantRegulatoryIndex implements RegulatoryChunkIndex {
for (let offset = 0; offset < chunks.length; offset += 64) {
const batch = chunks.slice(offset, offset + 64);
await this.request('PUT', `/collections/${REGULATORY_QDRANT_COLLECTION}/points?wait=true`, {
points: batch.map(({ chunk, embedding }) => ({
points: batch.map(({ chunk, embedding, sparseVector }) => ({
id: pointId(chunk.chunkId),
vector: { [REGULATORY_DENSE_VECTOR]: embedding },
vector: {
[REGULATORY_DENSE_VECTOR]: embedding,
...(sparseVector ? { [REGULATORY_SPARSE_VECTOR]: sparseVector } : {})
},
payload: chunkPayload(chunk, this.options.embeddingModel)
}))
});
@ -133,6 +146,28 @@ export class QdrantRegulatoryIndex implements RegulatoryChunkIndex {
return points.map(point => parseCandidate(point));
}
async searchSparse(
vector: RegulatorySparseVector,
limit: number,
filters: RegulatoryRetrievalFilters = {}
): Promise<RegulatoryRetrievalCandidate[]> {
validateSparseVector(vector);
if (!Number.isInteger(limit) || limit < 1 || limit > 100) throw new Error('Regulatory retrieval limit must be between 1 and 100.');
const must = [keywordCondition('embedding_model', this.options.embeddingModel), ...filterConditions(filters)];
const response = await this.request('POST', `/collections/${REGULATORY_QDRANT_COLLECTION}/points/query`, {
query: vector,
using: REGULATORY_SPARSE_VECTOR,
filter: { must },
limit,
with_payload: true,
with_vector: false
});
const result = object(response.body)?.result;
const points = Array.isArray(object(result)?.points) ? object(result)!.points as unknown[] : Array.isArray(result) ? result : undefined;
if (!points) throw new QdrantClientError('qdrant_invalid_response', 'Qdrant returned an invalid query response.');
return points.map(point => parseCandidate(point));
}
async list(filters: RegulatoryRetrievalFilters, limit: number): Promise<RegulatoryChunk[]> {
if (!Number.isInteger(limit) || limit < 1 || limit > 100) throw new Error('Regulatory list limit must be between 1 and 100.');
const response = await this.request('POST', `/collections/${REGULATORY_QDRANT_COLLECTION}/points/scroll`, {
@ -331,6 +366,14 @@ function validateEmbedding(value: number[], dimensions: number): void {
}
}
function validateSparseVector(value: RegulatorySparseVector): void {
if (!Array.isArray(value.indices) || !Array.isArray(value.values) || value.indices.length !== value.values.length || !value.indices.length
|| value.indices.some((entry, index) => !Number.isInteger(entry) || entry < 0 || entry > 0xffffffff || index > 0 && entry <= value.indices[index - 1])
|| value.values.some(entry => typeof entry !== 'number' || !Number.isFinite(entry) || entry <= 0)) {
throw new Error('Regulatory sparse vectors must contain equally sized, ascending indices and positive finite values.');
}
}
function boundedInteger(value: string | undefined, fallback: number, minimum: number, maximum: number, name: string): number {
if (!value?.trim()) return fallback;
const parsed = Number(value);

102
server/regulatory/regulatory-finding-retrieval.ts
View File

@ -7,6 +7,12 @@ import type {
import type { UnifiedResultProjection } from '../../src/domain/unified-result';
import { buildRegulatoryFindingQuery } from '../../src/profiles/public-search/regulatory-query';
const RECIPROCAL_RANK_FUSION_K = 60;
const FINDING_QUERY_SEARCH_LIMIT = 64;
const SECTION_SEARCH_LIMIT = 32;
const FULL_CONTEXT_CONTROL_LIMIT = 100;
const CANDIDATE_SECTION_KINDS = ['control', 'purpose', 'guidance'] as const;
export interface RegulatoryFindingRetrievalResult {
sourceResultId: string;
status: 'candidates' | 'insufficient-public-context';
@ -26,7 +32,101 @@ export class RegulatoryFindingRetrievalService {
if (!preparedQuery) {
return { sourceResultId: finding.sourceResultId, status: 'insufficient-public-context', candidates: [] };
}
const candidates = await this.retriever.retrieve({ query: preparedQuery.query, limit, filters });
const query = preparedQuery.retrievalQuery ?? preparedQuery.query;
if (filters?.sectionKind) {
const candidates = await this.retriever.retrieve({ query, limit, filters });
return { sourceResultId: finding.sourceResultId, status: 'candidates', preparedQuery, candidates };
}
const selectionQueries = [...new Set([
preparedQuery.problemQuery ?? query,
preparedQuery.remediationQuery
].filter((value): value is string => Boolean(value)))];
const selectionRankings = await Promise.all(selectionQueries.map(selectionQuery => this.retriever.retrieve({
query: selectionQuery,
limit: Math.max(limit, FINDING_QUERY_SEARCH_LIMIT),
filters
})));
const sectionRankings = await Promise.all(CANDIDATE_SECTION_KINDS.map(sectionKind => this.retriever.retrieve({
query,
limit: Math.max(limit, SECTION_SEARCH_LIMIT),
filters: { ...filters, sectionKind }
})));
const candidates = mergeFindingQueryRankings(selectionRankings, sectionRankings, limit);
return { sourceResultId: finding.sourceResultId, status: 'candidates', preparedQuery, candidates };
}
}
/**
* Collapses the full-context section searches into one control ranking, then
* fuses that family with the independently phrased problem and remediation
* rankings. A control can therefore receive at most one final vote from the
* section family while its section coverage still determines that family's
* internal order.
*/
export function mergeFindingQueryRankings(
selectionRankings: RegulatoryRetrievalCandidate[][],
evidenceRankings: RegulatoryRetrievalCandidate[][],
limit: number
): RegulatoryRetrievalCandidate[] {
if (!Number.isInteger(limit) || limit < 1 || limit > 100) throw new Error('Regulatory retrieval limit must be between 1 and 100.');
const fullContextRanking = reciprocalRankFuseByControl(evidenceRankings, FULL_CONTEXT_CONTROL_LIMIT);
const selected = uniqueControlIds(reciprocalRankFuseByControl(
[...selectionRankings, fullContextRanking],
limit
));
const allEvidenceRankings = [...evidenceRankings, ...selectionRankings];
return selected.flatMap((controlId, index) => {
const seenChunks = new Set<string>();
return allEvidenceRankings.flatMap(ranking => {
const hit = ranking.find(candidate => candidate.chunk.controlId === controlId && !seenChunks.has(candidate.chunk.chunkId));
if (!hit) return [];
seenChunks.add(hit.chunk.chunkId);
return [{ ...hit, score: 1 / (RECIPROCAL_RANK_FUSION_K + index + 1) }];
});
});
}
/** Fuses independent rankings at control level, allowing one vote per control and ranking. */
export function reciprocalRankFuseByControl(
rankings: RegulatoryRetrievalCandidate[][],
limit: number
): RegulatoryRetrievalCandidate[] {
if (!Number.isInteger(limit) || limit < 1 || limit > 100) throw new Error('Regulatory retrieval limit must be between 1 and 100.');
const controls = new Map<string, {
score: number;
bestRank: number;
hits: Map<number, RegulatoryRetrievalCandidate>;
}>();
rankings.forEach((ranking, rankingIndex) => {
const seen = new Set<string>();
for (const hit of ranking) {
const controlId = hit.chunk.controlId;
if (seen.has(controlId)) continue;
seen.add(controlId);
const controlRank = seen.size;
const entry = controls.get(controlId) ?? { score: 0, bestRank: Number.MAX_SAFE_INTEGER, hits: new Map() };
entry.score += 1 / (RECIPROCAL_RANK_FUSION_K + controlRank);
entry.bestRank = Math.min(entry.bestRank, controlRank);
entry.hits.set(rankingIndex, hit);
controls.set(controlId, entry);
}
});
return [...controls.entries()]
.sort(([leftId, left], [rightId, right]) => right.score - left.score
|| left.bestRank - right.bestRank
|| leftId.localeCompare(rightId, 'en', { numeric: true }))
.slice(0, limit)
.flatMap(([, entry]) => [...entry.hits.entries()]
.sort(([left], [right]) => left - right)
.map(([, hit]) => ({ ...hit, score: entry.score })));
}
function uniqueControlIds(candidates: RegulatoryRetrievalCandidate[]): string[] {
const seen = new Set<string>();
return candidates.flatMap(candidate => {
const controlId = candidate.chunk.controlId;
if (seen.has(controlId)) return [];
seen.add(controlId);
return [controlId];
});
}

24
server/regulatory/regulatory-mapping-service.ts
View File

@ -8,9 +8,12 @@ import {
REGULATORY_MAPPING_PROMPT_VERSION,
REGULATORY_RETRIEVAL_VERSION,
emptyRegulatoryMappingBatchCounts,
groundRegulatoryMappingResult,
groundRegulatoryCandidateAssessmentResult,
normalizeRegulatoryMappingBatchScope,
regulatoryCandidateAssessmentValidationErrors,
regulatoryMappingResultValidationErrors,
selectRegulatoryMappingsFromAssessments,
type RegulatoryCandidateAssessmentResult,
type RegulatoryMappingBatchCreateRequest,
type RegulatoryMappingBatchResponse,
type RegulatoryMappingBatchRun,
@ -36,6 +39,7 @@ const POLL_AFTER_MS = 1_500;
const MAXIMUM_PROMPT_CHARACTERS = 48_000;
const BATCH_PAGE_SIZE = 500;
const DEFAULT_MAXIMUM_CONCURRENT_RUNS = 2;
const MAPPING_CONTROL_SHORTLIST_LIMIT = 12;
export interface RegulatoryMappingServiceOptions {
findingRetrieval?: RegulatoryFindingRetrievalService;
@ -409,7 +413,7 @@ export class RegulatoryMappingService {
const selected = [...byControl.entries()]
.map(([controlId, values]) => ({ controlId, values: values.sort((left, right) => right.score - left.score), score: Math.max(...values.map(value => value.score)) }))
.sort((left, right) => right.score - left.score || left.controlId.localeCompare(right.controlId, 'en', { numeric: true }))
.slice(0, 8);
.slice(0, MAPPING_CONTROL_SHORTLIST_LIMIT);
return Promise.all(selected.map(async selection => {
const semantic = selection.values.slice(0, 2).map(value => evidence(value.chunk));
const references = await this.options.index!.list({
@ -438,7 +442,7 @@ export class RegulatoryMappingService {
try {
let retrieval;
try {
retrieval = await this.options.findingRetrieval!.retrieve(finding, 32, {
retrieval = await this.options.findingRetrieval!.retrieve(finding, MAPPING_CONTROL_SHORTLIST_LIMIT, {
corpusId: this.options.corpusId,
corpusVersion: this.options.corpusVersion,
sourceStandard: 'ISO/IEC 27002',
@ -468,14 +472,20 @@ export class RegulatoryMappingService {
const allowed = new Map(run.candidates.map(candidate => [
candidate.controlId, new Set(candidate.evidence.map(item => item.chunkId))
]));
const grounded = groundRegulatoryMappingResult(completion.value, allowed);
const errors = regulatoryMappingResultValidationErrors(grounded, allowed);
const grounded = groundRegulatoryCandidateAssessmentResult(completion.value, allowed);
const errors = regulatoryCandidateAssessmentValidationErrors(grounded, allowed);
if (errors.length) throw new RegulatoryMappingServiceError('regulatory_mapping_invalid_output', errors.join(' '), 502);
const selected = selectRegulatoryMappingsFromAssessments(
grounded as RegulatoryCandidateAssessmentResult,
run.candidates.map(candidate => candidate.controlId)
);
const selectionErrors = regulatoryMappingResultValidationErrors(selected, allowed);
if (selectionErrors.length) throw new RegulatoryMappingServiceError('regulatory_mapping_invalid_output', selectionErrors.join(' '), 502);
const completedAt = this.now().toISOString();
await this.repository.saveRegulatoryMappingRun({
...run,
status: 'succeeded',
result: grounded as RegulatoryMappingResult,
result: selected as RegulatoryMappingResult,
...(completion.usage ? { tokenUsage: completion.usage } : {}),
updatedAt: completedAt,
completedAt
@ -534,7 +544,7 @@ function mappingPrompt(run: RegulatoryMappingRun): string {
return payload;
}
const SYSTEM_PROMPT = `Map one public security finding query to the supplied ISO control candidates. A mapping means the control is topically relevant to the finding, never proof of compliance. Map a candidate when it governs the same security subject as the finding (for example password or authentication management, logging, or access restriction), even when the control text does not state the finding's specific value, threshold, parameter, or configuration; ISO controls are principle-level and rarely restate such specifics. Judge relevance from the control's subject and its supplied evidence, not from whether the exact requirement text appears. Return JSON only with schemaVersion 1, outcome mapped|no_relevant_control|insufficient_context, optional bounded reason, and mappings (maximum five). Each mapping requires controlId, relationship primary|supporting, confidence low|medium|high, a concise rationale under 600 characters, and one to three evidenceChunkIds copied only from that control's supplied evidence. Use no_relevant_control only when no candidate governs the finding's subject, and insufficient_context only when the finding query itself lacks enough meaning to judge. Do not invent controls or evidence.`;
const SYSTEM_PROMPT = `Assess every supplied ISO control candidate independently against one public security finding query. Relevance means the control materially governs, prevents, limits, detects, or supports recovery from the finding; it never proves compliance. Return every supplied candidate exactly once, even when it is irrelevant. Label a candidate primary only when it directly governs the finding's root security subject or remediation. Label it supporting only when it adds a distinct material requirement, prerequisite, detection, restriction, or recovery contribution. Label it irrelevant when it merely shares a technology, protocol, account type, security domain, or broad words such as security, access, monitoring, policy, or configuration. ISO controls are principle-level and rarely repeat a finding's exact product wording, parameter, threshold, or value, so judge the governed subject from the supplied evidence. Assess candidates independently rather than stopping after one or two relevant controls; a deterministic reducer will select zero or one primary and zero to three supporting controls. Return JSON only with schemaVersion 1, outcome assessed|insufficient_context, optional bounded reason, and assessments. For outcome assessed, assessments must contain every supplied candidate exactly once. Each assessment requires controlId, relevance primary|supporting|irrelevant, confidence low|medium|high, a concise rationale under 600 characters, and one to three evidenceChunkIds copied only from that control's supplied evidence. Use insufficient_context only when the finding query itself lacks enough meaning to assess any candidate, and then return no assessments. Do not invent controls or evidence.`;
function safeFailure(error: unknown): string {
if (error instanceof RegulatoryMappingServiceError || error instanceof LiteLlmClientError) return error.code;

7
server/regulatory/regulatory-rag-runtime.ts
View File

@ -3,8 +3,10 @@ import { createQdrantRegulatoryIndexFromEnvironment, type QdrantEnvironment } fr
import { RegulatoryFindingRetrievalService } from './regulatory-finding-retrieval';
import {
DenseRegulatoryCandidateRetriever,
HybridRegulatoryCandidateRetriever,
RegulatoryIngestionService,
RegulatoryRetrievalPipeline
RegulatoryRetrievalPipeline,
SparseRegulatoryCandidateRetriever
} from './regulatory-rag-service';
export interface RegulatoryRagEnvironment extends EmbeddingEnvironment, QdrantEnvironment {
@ -36,7 +38,8 @@ export function createRegulatoryRagRuntime(
const embeddings = createEmbeddingClientFromEnvironment(environment);
const index = createQdrantRegulatoryIndexFromEnvironment(embeddings.dimensions, embeddings.model, environment);
const dense = new DenseRegulatoryCandidateRetriever(embeddings, index);
const retrieval = new RegulatoryRetrievalPipeline(dense);
const sparse = new SparseRegulatoryCandidateRetriever(index);
const retrieval = new RegulatoryRetrievalPipeline(new HybridRegulatoryCandidateRetriever(dense, sparse));
return {
ingestion: new RegulatoryIngestionService(embeddings, index, {
maximumChunkCharacters: boundedInteger(environment.ASSESSOR_RAG_CHUNK_CHARACTERS, 1_600, 200, 20_000, 'ASSESSOR_RAG_CHUNK_CHARACTERS'),

126
server/regulatory/regulatory-rag-service.ts
View File

@ -1,4 +1,5 @@
import type {
RegulatoryChunk,
RegulatoryControlDocument,
RegulatoryReranker,
RegulatoryRetrievalCandidate,
@ -7,7 +8,11 @@ import type {
} from '../../src/domain/regulatory-rag';
import type { TextEmbeddingClient } from './embedding-client';
import { chunkRegulatoryControl, regulatoryChunkEmbeddingText, validateRegulatoryControlDocument } from './regulatory-chunker';
import type { RegulatoryChunkIndex } from './qdrant-regulatory-index';
import type { RegulatoryChunkIndex, RegulatorySparseVector } from './qdrant-regulatory-index';
const HYBRID_RRF_K = 60;
const BM25_K1 = 1.2;
const BM25_B = 0.75;
export interface RegulatoryCandidateRetriever {
retrieveCandidates(request: RegulatoryRetrievalRequest): Promise<RegulatoryRetrievalCandidate[]>;
@ -25,6 +30,32 @@ export class DenseRegulatoryCandidateRetriever implements RegulatoryCandidateRet
}
}
export class SparseRegulatoryCandidateRetriever implements RegulatoryCandidateRetriever {
constructor(private readonly index: RegulatoryChunkIndex) {}
async retrieveCandidates(request: RegulatoryRetrievalRequest): Promise<RegulatoryRetrievalCandidate[]> {
const vector = regulatoryQuerySparseVector(validatedQuery(request.query));
if (!vector) return [];
return this.index.searchSparse(vector, validatedLimit(request.limit), request.filters);
}
}
export class HybridRegulatoryCandidateRetriever implements RegulatoryCandidateRetriever {
constructor(
private readonly dense: RegulatoryCandidateRetriever,
private readonly sparse: RegulatoryCandidateRetriever
) {}
async retrieveCandidates(request: RegulatoryRetrievalRequest): Promise<RegulatoryRetrievalCandidate[]> {
const limit = validatedLimit(request.limit);
const [dense, sparse] = await Promise.all([
this.dense.retrieveCandidates({ ...request, limit }),
this.sparse.retrieveCandidates({ ...request, limit })
]);
return reciprocalRankFuseCandidates([dense, sparse], limit);
}
}
export class RegulatoryRetrievalPipeline implements RegulatoryRetriever {
constructor(
private readonly candidates: RegulatoryCandidateRetriever,
@ -89,12 +120,103 @@ export class RegulatoryIngestionService {
vectors.push(...await this.embeddings.embed(batch));
}
if (vectors.length !== chunks.length) throw new Error('Embedding provider returned an unexpected vector count.');
const sparseVectors = buildRegulatorySparseVectors(chunks);
await this.index.ensureCollection();
await this.index.replaceDocuments(chunks.map((chunk, index) => ({ chunk, embedding: vectors[index] })));
await this.index.replaceDocuments(chunks.map((chunk, index) => ({
chunk,
embedding: vectors[index],
...(sparseVectors.get(chunk.chunkId) ? { sparseVector: sparseVectors.get(chunk.chunkId) } : {})
})));
return { documentCount: documents.length, chunkCount: chunks.length, embeddingModel: this.embeddings.model };
}
}
/** Builds deterministic hashed BM25 document vectors without a scanner-specific vocabulary. */
export function buildRegulatorySparseVectors(chunks: RegulatoryChunk[]): Map<string, RegulatorySparseVector> {
const documents = chunks.map(chunk => tokenFrequencies(`${chunk.title} ${chunk.title} ${chunk.text}`));
const documentFrequency = new Map<number, number>();
for (const document of documents) {
for (const index of document.frequencies.keys()) documentFrequency.set(index, (documentFrequency.get(index) ?? 0) + 1);
}
const averageLength = documents.reduce((sum, document) => sum + document.length, 0) / Math.max(documents.length, 1);
const vectors = new Map<string, RegulatorySparseVector>();
documents.forEach((document, documentIndex) => {
if (!document.frequencies.size) return;
const entries = [...document.frequencies.entries()].map(([index, frequency]) => {
const matchingDocuments = documentFrequency.get(index) ?? 1;
const inverseDocumentFrequency = Math.log(1 + (documents.length - matchingDocuments + 0.5) / (matchingDocuments + 0.5));
const normalizedFrequency = frequency * (BM25_K1 + 1)
/ (frequency + BM25_K1 * (1 - BM25_B + BM25_B * document.length / Math.max(averageLength, 1)));
return [index, inverseDocumentFrequency * normalizedFrequency] as const;
}).sort(([left], [right]) => left - right);
vectors.set(chunks[documentIndex].chunkId, {
indices: entries.map(([index]) => index),
values: entries.map(([, score]) => score)
});
});
return vectors;
}
export function regulatoryQuerySparseVector(query: string): RegulatorySparseVector | undefined {
const indices = [...new Set(regulatoryTokens(query).map(sparseIndex))].sort((left, right) => left - right);
return indices.length ? { indices, values: indices.map(() => 1) } : undefined;
}
export function reciprocalRankFuseCandidates(
rankings: RegulatoryRetrievalCandidate[][],
limit: number
): RegulatoryRetrievalCandidate[] {
const fused = new Map<string, { score: number; bestRank: number; candidate: RegulatoryRetrievalCandidate }>();
for (const ranking of rankings) {
const seen = new Set<string>();
let rank = 0;
for (const candidate of ranking) {
const controlId = candidate.chunk.controlId;
if (seen.has(controlId)) continue;
seen.add(controlId);
rank += 1;
const existing = fused.get(controlId);
const entry = existing ?? { score: 0, bestRank: Number.MAX_SAFE_INTEGER, candidate };
entry.score += 1 / (HYBRID_RRF_K + rank);
if (rank < entry.bestRank) {
entry.bestRank = rank;
entry.candidate = candidate;
}
fused.set(controlId, entry);
}
}
return [...fused.values()]
.sort((left, right) => right.score - left.score
|| left.bestRank - right.bestRank
|| left.candidate.chunk.controlId.localeCompare(right.candidate.chunk.controlId, 'en', { numeric: true }))
.slice(0, limit)
.map(entry => ({ ...entry.candidate, score: entry.score }));
}
function tokenFrequencies(value: string): { frequencies: Map<number, number>; length: number } {
const frequencies = new Map<number, number>();
const tokens = regulatoryTokens(value);
for (const token of tokens) {
const index = sparseIndex(token);
frequencies.set(index, (frequencies.get(index) ?? 0) + 1);
}
return { frequencies, length: tokens.length };
}
function regulatoryTokens(value: string): string[] {
return (value.normalize('NFKD').toLocaleLowerCase('und').replace(/\p{M}/gu, '').match(/[\p{L}\p{N}]+/gu) ?? [])
.filter(token => token.length > 1);
}
function sparseIndex(token: string): number {
let hash = 0x811c9dc5;
for (let index = 0; index < token.length; index += 1) {
hash ^= token.charCodeAt(index);
hash = Math.imul(hash, 0x01000193);
}
return hash >>> 0;
}
function validatedQuery(value: string): string {
if (typeof value !== 'string' || !value.trim()) throw new Error('Regulatory retrieval query must be non-empty.');
if (value.length > 20_000) throw new Error('Regulatory retrieval query exceeds 20000 characters.');

115
src/domain/regulatory-mapping.ts
View File

@ -2,9 +2,9 @@ import type { AiOverviewTokenUsage } from './ai-overview';
import type { RegulatoryFindingQuery, RegulatorySectionKind } from './regulatory-rag';
import type { UnifiedResultQuery } from './unified-result';
export const REGULATORY_MAPPING_PROMPT_VERSION = '1.2.0';
export const REGULATORY_MAPPING_PROMPT_VERSION = '1.5.0';
export const REGULATORY_MAPPING_OUTPUT_SCHEMA_VERSION = 1 as const;
export const REGULATORY_RETRIEVAL_VERSION = 'dense-control-grouping-1.1.0';
export const REGULATORY_RETRIEVAL_VERSION = 'hybrid-finding-query-section-family-rrf-1.6.0';
export type RegulatoryMappingRunStatus = 'queued' | 'running' | 'succeeded' | 'failed';
export type RegulatoryMappingRelationship = 'primary' | 'supporting';
@ -95,6 +95,23 @@ export interface RegulatoryMappingProposal {
evidenceChunkIds: string[];
}
export type RegulatoryCandidateAssessmentRelevance = 'primary' | 'supporting' | 'irrelevant';
export interface RegulatoryCandidateAssessment {
controlId: string;
relevance: RegulatoryCandidateAssessmentRelevance;
confidence: RegulatoryMappingConfidence;
rationale: string;
evidenceChunkIds: string[];
}
export interface RegulatoryCandidateAssessmentResult {
schemaVersion: 1;
outcome: 'assessed' | 'insufficient_context';
reason?: string;
assessments: RegulatoryCandidateAssessment[];
}
export interface RegulatoryMappingResult {
schemaVersion: 1;
outcome: 'mapped' | 'no_relevant_control' | 'insufficient_context';
@ -198,6 +215,98 @@ export function groundRegulatoryMappingResult(
return { ...value, mappings };
}
/** Grounds every candidate assessment before its complete coverage is validated. */
export function groundRegulatoryCandidateAssessmentResult(
value: unknown,
allowedCandidates: ReadonlyMap<string, ReadonlySet<string>>
): unknown {
if (!isObject(value) || !Array.isArray(value.assessments)) return value;
const assessments: unknown[] = [];
for (const assessment of value.assessments) {
const allowed = isObject(assessment) && typeof assessment.controlId === 'string'
? allowedCandidates.get(assessment.controlId)
: undefined;
if (!allowed || !Array.isArray((assessment as Record<string, unknown>).evidenceChunkIds)) {
assessments.push(assessment);
continue;
}
const grounded = [...new Set(((assessment as Record<string, unknown>).evidenceChunkIds as unknown[])
.filter((id): id is string => typeof id === 'string' && allowed.has(id)))].slice(0, 3);
if (grounded.length) assessments.push({ ...(assessment as Record<string, unknown>), evidenceChunkIds: grounded });
}
return { ...value, assessments };
}
export function regulatoryCandidateAssessmentValidationErrors(
value: unknown,
allowedCandidates: ReadonlyMap<string, ReadonlySet<string>>
): string[] {
const errors: string[] = [];
if (!isObject(value)) return ['Regulatory candidate assessment output must be a JSON object.'];
if (value.schemaVersion !== 1) errors.push('schemaVersion must be 1.');
if (!['assessed', 'insufficient_context'].includes(String(value.outcome))) errors.push('outcome is invalid.');
if (!Array.isArray(value.assessments)) errors.push('assessments must be an array.');
else {
const controls = new Set<string>();
value.assessments.forEach((assessment, index) => {
if (!isObject(assessment)) { errors.push(`assessments[${index}] must be an object.`); return; }
const controlId = boundedString(assessment.controlId, 20, `assessments[${index}].controlId`, errors);
enumeration(assessment.relevance, ['primary', 'supporting', 'irrelevant'], `assessments[${index}].relevance`, errors);
enumeration(assessment.confidence, ['low', 'medium', 'high'], `assessments[${index}].confidence`, errors);
boundedString(assessment.rationale, 600, `assessments[${index}].rationale`, errors);
if (controlId && controls.has(controlId)) errors.push(`assessments[${index}] duplicates control ${controlId}.`);
if (controlId) controls.add(controlId);
const allowedEvidence = controlId ? allowedCandidates.get(controlId) : undefined;
if (!Array.isArray(assessment.evidenceChunkIds) || assessment.evidenceChunkIds.length < 1 || assessment.evidenceChunkIds.length > 3) {
errors.push(`assessments[${index}].evidenceChunkIds must contain one to three entries.`);
} else assessment.evidenceChunkIds.forEach((chunkId, evidenceIndex) => {
if (typeof chunkId !== 'string' || !allowedEvidence?.has(chunkId)) {
errors.push(`assessments[${index}].evidenceChunkIds[${evidenceIndex}] is not grounded in the supplied control.`);
}
});
if (controlId && !allowedCandidates.has(controlId)) errors.push(`assessments[${index}] references an unknown candidate control.`);
});
if (value.outcome === 'assessed') {
for (const controlId of allowedCandidates.keys()) {
if (!controls.has(controlId)) errors.push(`assessments must include candidate control ${controlId}.`);
}
if (controls.size !== allowedCandidates.size) errors.push('assessments must contain every supplied candidate exactly once.');
}
if (value.outcome !== 'assessed' && value.assessments.length) errors.push('insufficient_context must not include assessments.');
}
if (value.outcome === 'insufficient_context') boundedString(value.reason, 600, 'reason', errors);
return [...new Set(errors)];
}
/** Reduces complete independent assessments to the stable 0–1 primary / 0–3 supporting contract. */
export function selectRegulatoryMappingsFromAssessments(
value: RegulatoryCandidateAssessmentResult,
candidateOrder: readonly string[]
): RegulatoryMappingResult {
if (value.outcome === 'insufficient_context') {
return { schemaVersion: 1, outcome: 'insufficient_context', reason: value.reason, mappings: [] };
}
const order = new Map(candidateOrder.map((controlId, index) => [controlId, index]));
const confidenceRank: Record<RegulatoryMappingConfidence, number> = { high: 3, medium: 2, low: 1 };
const compare = (left: RegulatoryCandidateAssessment, right: RegulatoryCandidateAssessment) =>
confidenceRank[right.confidence] - confidenceRank[left.confidence]
|| (order.get(left.controlId) ?? Number.MAX_SAFE_INTEGER) - (order.get(right.controlId) ?? Number.MAX_SAFE_INTEGER)
|| left.controlId.localeCompare(right.controlId, 'en', { numeric: true });
const primaryCandidates = value.assessments.filter(item => item.relevance === 'primary').sort(compare);
const primary = primaryCandidates[0];
const supporting = value.assessments
.filter(item => item.relevance !== 'irrelevant' && item.controlId !== primary?.controlId)
.sort((left, right) => Number(right.relevance === 'primary') - Number(left.relevance === 'primary') || compare(left, right))
.slice(0, 3);
const mappings: RegulatoryMappingProposal[] = [
...(primary ? [{ ...primary, relationship: 'primary' as const }] : []),
...supporting.map(item => ({ ...item, relationship: 'supporting' as const }))
].map(({ relevance: _relevance, ...mapping }) => mapping);
return mappings.length
? { schemaVersion: 1, outcome: 'mapped', mappings }
: { schemaVersion: 1, outcome: 'no_relevant_control', reason: 'No supplied candidate was assessed as materially relevant.', mappings: [] };
}
export function regulatoryMappingResultValidationErrors(
value: unknown,
allowedCandidates?: ReadonlyMap<string, ReadonlySet<string>>
@ -243,7 +352,7 @@ export function regulatoryMappingRunValidationErrors(run: RegulatoryMappingRun):
if (!/^[a-f0-9]{64}$/.test(run.queryHash)) errors.push('Regulatory mapping queryHash must be SHA-256.');
const candidatesRequired = run.status === 'running' || run.status === 'succeeded';
if ((!run.sourceHashes.length && candidatesRequired) || run.sourceHashes.some(value => !/^[a-f0-9]{64}$/.test(value))) errors.push('Regulatory mapping sourceHashes must contain SHA-256 digests.');
if (!Array.isArray(run.candidates) || (candidatesRequired && run.candidates.length < 1) || run.candidates.length > 8) errors.push('Regulatory mapping candidates must contain one to eight controls when processing has started.');
if (!Array.isArray(run.candidates) || (candidatesRequired && run.candidates.length < 1) || run.candidates.length > 12) errors.push('Regulatory mapping candidates must contain one to twelve controls when processing has started.');
else {
const controls = new Set<string>();
run.candidates.forEach(candidate => {

13
src/domain/regulatory-rag.ts
View File

@ -1,5 +1,5 @@
export const REGULATORY_CHUNK_SCHEMA_VERSION = 2 as const;
export const REGULATORY_FINDING_QUERY_SCHEMA_VERSION = 1 as const;
export const REGULATORY_FINDING_QUERY_SCHEMA_VERSION = 4 as const;
export type RegulatoryDocumentSchemaVersion = 1 | 2;
export type RegulatoryControlTheme = 'organizational' | 'people' | 'physical' | 'technological';
@ -106,12 +106,17 @@ export interface RegulatoryRetriever {
}
export interface RegulatoryFindingQuery {
schemaVersion: 1;
schemaVersion: 1 | 2 | 3 | 4;
query: string;
specificity: 'rule' | 'topic';
retrievalQuery?: string;
problemQuery?: string;
remediationQuery?: string;
// Legacy schema 1-3 fields retained for persisted historical runs.
topicQuery?: string;
specificity?: 'rule' | 'topic';
publicTool?: string;
publicRuleId?: string;
publicTopics: string[];
publicTopics?: string[];
}
export interface RegulatoryReranker {

18
src/features/finding-reader/RegulatoryMappingPanel.tsx
View File

@ -18,11 +18,13 @@ export function RegulatoryMappingPanel({ response, loading, error, onStart, onDe
const run = response?.run;
const proposals = run?.result?.mappings ?? [];
const proposalIds = new Set(proposals.map(item => item.controlId));
const rankedCandidates = (run?.candidates ?? []).map((candidate, index) => ({ candidate, rank: index + 1 }));
const candidatesByControl = new Map(rankedCandidates.map(item => [item.candidate.controlId, item]));
const proposedCandidates = proposals.flatMap(proposal => {
const candidate = run?.candidates.find(item => item.controlId === proposal.controlId);
return candidate ? [{ candidate, proposal }] : [];
const rankedCandidate = candidatesByControl.get(proposal.controlId);
return rankedCandidate ? [{ ...rankedCandidate, proposal }] : [];
});
const otherCandidates = run?.candidates.filter(item => !proposalIds.has(item.controlId)) ?? [];
const otherCandidates = rankedCandidates.filter(item => !proposalIds.has(item.candidate.controlId));
return <section className="regulatory-mapping-panel" aria-label="ISO control mapping">
<header><div><h3>ISO control mapping</h3><p>Proposals indicate relevance to this finding, not proof of compliance.</p></div>
@ -35,17 +37,19 @@ export function RegulatoryMappingPanel({ response, loading, error, onStart, onDe
{run?.status === 'failed' && <p className="regulatory-mapping-error" role="status">Mapping stopped safely ({run.errorCode ?? 'unknown error'}). You can start a new run.{run.errorDetail ? <><br/><span className="regulatory-mapping-error-detail">{run.errorDetail}</span></> : null}</p>}
{run?.status === 'succeeded' && run.result?.outcome !== 'mapped' && <p role="status">{run.result?.reason ?? (run.result?.outcome === 'insufficient_context' ? 'Insufficient public context.' : 'No relevant control was proposed.')}</p>}
{proposedCandidates.length > 0 && <div className="regulatory-control-list">
{proposedCandidates.map(item => <ControlCandidate key={item.candidate.controlId} {...item} decisions={response?.decisions ?? []} loading={loading} onDecision={onDecision}/>) }
{proposedCandidates.map(item => <ControlCandidate key={item.candidate.controlId} {...item} candidateCount={rankedCandidates.length} decisions={response?.decisions ?? []} loading={loading} onDecision={onDecision}/>) }
</div>}
{otherCandidates.length > 0 && <details className="regulatory-other-candidates"><summary>Other retrieved controls ({otherCandidates.length})</summary>
<div className="regulatory-control-list">{otherCandidates.map(candidate => <ControlCandidate key={candidate.controlId} candidate={candidate} decisions={response?.decisions ?? []} loading={loading} onDecision={onDecision}/>)}</div>
<div className="regulatory-control-list">{otherCandidates.map(item => <ControlCandidate key={item.candidate.controlId} {...item} candidateCount={rankedCandidates.length} decisions={response?.decisions ?? []} loading={loading} onDecision={onDecision}/>)}</div>
</details>}
</section>;
}
function ControlCandidate({ candidate, proposal, decisions, loading, onDecision }: {
function ControlCandidate({ candidate, proposal, rank, candidateCount, decisions, loading, onDecision }: {
candidate: RegulatoryMappingCandidate;
proposal?: RegulatoryMappingProposal;
rank: number;
candidateCount: number;
decisions: RegulatoryMappingDecision[];
loading: boolean;
onDecision(candidate: RegulatoryMappingCandidate, decision: 'accepted' | 'rejected', comment?: string): void;
@ -54,7 +58,7 @@ function ControlCandidate({ candidate, proposal, decisions, loading, onDecision
const current = decisions.find(item => item.controlId === candidate.controlId);
return <article className="regulatory-control-candidate">
<header><div><h4>{candidate.annexAControlId} — {candidate.title}</h4>
<p>{proposal ? <>{proposal.relationship} · {proposal.confidence} confidence</> : 'Retrieved candidate'} · score {candidate.score.toFixed(3)}</p></div>
<p>{proposal ? <>{proposal.relationship} · {proposal.confidence} confidence</> : 'Retrieved candidate'} · retrieval rank {rank} of {candidateCount}</p></div>
{current && <span className="status-badge">{current.decision}</span>}
</header>
{proposal?.rationale && <p>{proposal.rationale}</p>}

59
src/profiles/public-search/regulatory-query.ts
View File

@ -2,66 +2,53 @@ import type { RegulatoryFindingQuery } from '../../domain/regulatory-rag';
import { REGULATORY_FINDING_QUERY_SCHEMA_VERSION } from '../../domain/regulatory-rag';
import type { UnifiedResultProjection } from '../../domain/unified-result';
import { privacySafeFinding } from '../../domain/privacy-safe-finding';
import catalog from './catalog.json';
import { publicSecurityTopics } from './public-security-topics';
const MAXIMUM_TITLE_CHARACTERS = 500;
const MAXIMUM_DETAIL_CHARACTERS = 1_200;
const MAXIMUM_RECOMMENDATION_CHARACTERS = 600;
const publicTools: Record<string, string> = {
'cis-cat': 'CIS-CAT / CIS Benchmarks',
maester: 'Maester',
pingcastle: 'PingCastle',
'purple-knight': 'Semperis Purple Knight'
};
const publicRuleIds: Record<string, readonly string[]> = {
maester: catalog.maesterIds,
pingcastle: catalog.pingcastleIds
};
/**
* Builds an embedding query from the finding's rule-level check text — its title,
* description and recommendation projected through the shared privacy boundary
* ({@link privacySafeFinding}). That text describes the check, not the tenant: the
* result-table / occurrence data (affectedAsset, per-object values) is never read,
* and email addresses are redacted. Every scanner finding with a check title maps;
* catalog rule IDs and fixed public topics only sharpen the query when present.
* and email addresses are redacted. Scanner names, rule catalogs and maintained
* topic dictionaries never influence retrieval, so an unknown scanner follows the
* same path as an existing one.
*/
export function buildRegulatoryFindingQuery(finding: UnifiedResultProjection): RegulatoryFindingQuery | undefined {
if (finding.sourceKind !== 'scanner-finding') return undefined;
const safe = privacySafeFinding(finding);
const title = safe.title.trim().slice(0, MAXIMUM_TITLE_CHARACTERS);
if (!title) return undefined;
const toolId = finding.sourceTool?.trim().toLowerCase() ?? '';
const publicTool = publicTools[toolId];
const publicRuleId = recognizedRuleId(toolId, finding.sourceRuleId);
const detail = safe.description?.slice(0, MAXIMUM_DETAIL_CHARACTERS);
const recommendation = safe.recommendation?.slice(0, MAXIMUM_RECOMMENDATION_CHARACTERS);
const topics = publicSecurityTopics([title, detail, recommendation].filter(Boolean).join('\n'));
const detail = safe.description?.trim().slice(0, MAXIMUM_DETAIL_CHARACTERS) || undefined;
const recommendation = safe.recommendation?.trim().slice(0, MAXIMUM_RECOMMENDATION_CHARACTERS) || undefined;
const problemQuery = [
retrievalClause('Security check', title),
detail && retrievalClause('Check detail', detail)
].filter(Boolean).join(' ');
const remediationQuery = recommendation ? [
retrievalClause('Security check', title),
retrievalClause('Recommended action', recommendation)
].join(' ') : undefined;
const retrievalQuery = [
problemQuery,
recommendation && retrievalClause('Recommended action', recommendation)
].filter(Boolean).join(' ');
const query = [
'Retrieve regulatory controls whose normative requirements directly address this security finding.',
publicTool && `Public assessment tool: ${publicTool}.`,
publicRuleId && `Public rule ID: ${publicRuleId}.`,
`Security check: ${title}.`,
detail && `Check detail: ${detail}.`,
recommendation && `Recommended action: ${recommendation}.`,
topics.length && `Public technical topics: ${topics.join('; ')}.`,
retrievalQuery,
'Prefer controls with direct requirement text over merely related guidance. Do not infer customer scope, assets, configuration values, or compliance status.'
].filter(Boolean).join(' ');
return {
schemaVersion: REGULATORY_FINDING_QUERY_SCHEMA_VERSION,
query,
specificity: publicRuleId ? 'rule' : 'topic',
...(publicTool ? { publicTool } : {}),
...(publicRuleId ? { publicRuleId } : {}),
publicTopics: topics
retrievalQuery,
problemQuery,
...(remediationQuery && remediationQuery !== problemQuery ? { remediationQuery } : {})
};
}
function recognizedRuleId(toolId: string, value: string | undefined): string | undefined {
const candidate = value?.trim();
if (!candidate) return undefined;
return publicRuleIds[toolId]?.find(id => id.toLowerCase() === candidate.toLowerCase());
function retrievalClause(label: string, value: string): string {
return `${label}: ${value.trim().replace(/[.!?]+$/, '')}.`;
}

129
tests/domain/regulatory-mapping.test.ts
View File

@ -1,9 +1,75 @@
import { describe, expect, it } from 'vitest';
import { groundRegulatoryMappingResult, regulatoryMappingResultValidationErrors } from '../../src/domain/regulatory-mapping';
import {
groundRegulatoryCandidateAssessmentResult,
groundRegulatoryMappingResult,
regulatoryCandidateAssessmentValidationErrors,
regulatoryMappingResultValidationErrors,
regulatoryMappingRunValidationErrors,
selectRegulatoryMappingsFromAssessments,
type RegulatoryCandidateAssessmentResult,
type RegulatoryMappingRun
} from '../../src/domain/regulatory-mapping';
const grounded = new Map([['5.1', new Set(['semantic-5.1', 'reference-5.1'])]]);
describe('regulatory mapping output validation', () => {
it('requires one grounded assessment for every supplied candidate', () => {
const allowed = new Map([
['5.1', new Set(['semantic-5.1'])],
['8.2', new Set(['semantic-8.2'])]
]);
const valid = {
schemaVersion: 1, outcome: 'assessed', assessments: [
{ controlId: '5.1', relevance: 'primary', confidence: 'high', rationale: 'Direct.', evidenceChunkIds: ['semantic-5.1'] },
{ controlId: '8.2', relevance: 'irrelevant', confidence: 'high', rationale: 'Only broad overlap.', evidenceChunkIds: ['semantic-8.2'] }
]
};
expect(regulatoryCandidateAssessmentValidationErrors(valid, allowed)).toEqual([]);
expect(regulatoryCandidateAssessmentValidationErrors({ ...valid, assessments: valid.assessments.slice(0, 1) }, allowed))
.toEqual(expect.arrayContaining([expect.stringContaining('candidate control 8.2'), expect.stringContaining('every supplied candidate')]));
expect(regulatoryCandidateAssessmentValidationErrors({ ...valid, assessments: [...valid.assessments, valid.assessments[0]] }, allowed))
.toEqual(expect.arrayContaining([expect.stringContaining('duplicates control 5.1')]));
});
it('grounds assessments and deterministically selects at most one primary plus three supporting controls', () => {
const allowed = new Map([
['5.1', new Set(['chunk-5.1'])], ['5.2', new Set(['chunk-5.2'])], ['5.3', new Set(['chunk-5.3'])],
['5.4', new Set(['chunk-5.4'])], ['5.5', new Set(['chunk-5.5'])]
]);
const assessment = (controlId: string, relevance: 'primary' | 'supporting' | 'irrelevant', confidence: 'low' | 'medium' | 'high') => ({
controlId, relevance, confidence, rationale: `Synthetic ${controlId}.`, evidenceChunkIds: [`chunk-${controlId}`, 'foreign']
});
const groundedAssessment = groundRegulatoryCandidateAssessmentResult({
schemaVersion: 1, outcome: 'assessed', assessments: [
assessment('5.1', 'primary', 'medium'), assessment('5.2', 'primary', 'high'),
assessment('5.3', 'supporting', 'high'), assessment('5.4', 'supporting', 'medium'), assessment('5.5', 'supporting', 'low')
]
}, allowed) as RegulatoryCandidateAssessmentResult;
expect(regulatoryCandidateAssessmentValidationErrors(groundedAssessment, allowed)).toEqual([]);
const selected = selectRegulatoryMappingsFromAssessments(groundedAssessment, ['5.1', '5.2', '5.3', '5.4', '5.5']);
expect(selected.mappings.map(item => `${item.controlId}:${item.relationship}`)).toEqual([
'5.2:primary', '5.1:supporting', '5.3:supporting', '5.4:supporting'
]);
expect(selected.mappings.every(item => item.evidenceChunkIds.length === 1)).toBe(true);
});
it('derives supporting-only and no-relevant-control outcomes without forcing a primary', () => {
const supporting = selectRegulatoryMappingsFromAssessments({
schemaVersion: 1, outcome: 'assessed', assessments: [{
controlId: '5.1', relevance: 'supporting', confidence: 'medium', rationale: 'Material support.', evidenceChunkIds: ['semantic-5.1']
}]
}, ['5.1']);
expect(supporting).toMatchObject({ outcome: 'mapped', mappings: [{ controlId: '5.1', relationship: 'supporting' }] });
const irrelevant = selectRegulatoryMappingsFromAssessments({
schemaVersion: 1, outcome: 'assessed', assessments: [{
controlId: '5.1', relevance: 'irrelevant', confidence: 'high', rationale: 'Only topic overlap.', evidenceChunkIds: ['semantic-5.1']
}]
}, ['5.1']);
expect(irrelevant).toMatchObject({ outcome: 'no_relevant_control', mappings: [] });
});
it('accepts a grounded proposal and rejects unknown controls, duplicate controls, and foreign evidence', () => {
const valid = {
schemaVersion: 1, outcome: 'mapped', mappings: [{
@ -49,4 +115,65 @@ describe('regulatory mapping output validation', () => {
}]
}, grounded)).toContain('A refusal outcome must not include mappings.');
});
it('accepts a mapped result with supporting controls and no primary control', () => {
expect(regulatoryMappingResultValidationErrors({
schemaVersion: 1, outcome: 'mapped', mappings: [{
controlId: '5.1', relationship: 'supporting', confidence: 'medium', rationale: 'Relevant, but not uniquely primary.',
evidenceChunkIds: ['semantic-5.1']
}]
}, grounded)).toEqual([]);
});
it('accepts a twelve-control shortlist and rejects a thirteenth control', () => {
const candidate = (index: number) => {
const controlId = `8.${index + 1}`;
return {
controlId,
annexAControlId: `A.${controlId}`,
title: `Synthetic control ${controlId}`,
score: 1 - index / 100,
evidence: [{
chunkId: `chunk-${controlId}`,
textSha256: 'a'.repeat(64),
sourceSha256: 'b'.repeat(64),
sourceStandard: 'ISO/IEC 27002' as const,
sectionKind: 'control' as const,
pageStart: 1,
pageEnd: 1,
text: 'Synthetic evidence.'
}]
};
};
const run: RegulatoryMappingRun = {
version: 1,
id: 'mapping-1',
ruleKey: 'pingcastle:synthetic',
sourceTool: 'pingcastle',
status: 'running',
queryHash: 'c'.repeat(64),
preparedQuery: {
schemaVersion: 2,
query: 'Synthetic full context.',
retrievalQuery: 'Synthetic concise query.',
specificity: 'rule',
publicTopics: []
},
corpusId: 'synthetic',
corpusVersion: '1',
sourceHashes: ['b'.repeat(64)],
embeddingModel: 'synthetic',
retrievalVersion: 'synthetic',
promptVersion: 'synthetic',
outputSchemaVersion: 1,
model: 'synthetic',
createdAt: '2026-09-17T00:00:00.000Z',
updatedAt: '2026-09-17T00:00:00.000Z',
candidates: Array.from({ length: 12 }, (_, index) => candidate(index))
};
expect(regulatoryMappingRunValidationErrors(run)).toEqual([]);
expect(regulatoryMappingRunValidationErrors({ ...run, candidates: [...run.candidates, candidate(12)] }))
.toContain('Regulatory mapping candidates must contain one to twelve controls when processing has started.');
});
});

7
tests/features/regulatory-mapping-panel.test.tsx
View File

@ -13,11 +13,11 @@ const response: RegulatoryMappingResponse = {
retrievalVersion: 'test', promptVersion: 'test', outputSchemaVersion: 1, model: 'work-model',
createdAt: '2026-09-15T10:00:00.000Z', updatedAt: '2026-09-15T10:00:00.000Z',
candidates: [{
controlId: '5.1', annexAControlId: 'A.5.1', title: 'Synthetic title', score: 0.91,
controlId: '5.1', annexAControlId: 'A.5.1', title: 'Synthetic title', score: 0.0328,
evidence: [{ chunkId: 'chunk-1', textSha256: 'c'.repeat(64), sourceSha256: 'b'.repeat(64), sourceStandard: 'ISO/IEC 27002',
sectionKind: 'purpose', pageStart: 12, pageEnd: 13, text: 'Synthetic evidence text.' }]
}, {
controlId: '5.2', annexAControlId: 'A.5.2', title: 'Other synthetic title', score: 0.72,
controlId: '5.2', annexAControlId: 'A.5.2', title: 'Other synthetic title', score: 0.0161,
evidence: [{ chunkId: 'chunk-2', textSha256: 'd'.repeat(64), sourceSha256: 'b'.repeat(64), sourceStandard: 'ISO/IEC 27001',
sectionKind: 'control', pageStart: 14, pageEnd: 14, text: 'Other synthetic evidence.' }]
}],
@ -32,6 +32,9 @@ describe('regulatory mapping panel', () => {
expect(html.indexOf('A.5.1 — Synthetic title')).toBeLessThan(html.indexOf('Other retrieved controls'));
expect(html).not.toContain('5.1 · A.5.1');
expect(html).toContain('primary · high confidence');
expect(html).toContain('retrieval rank 1 of 2');
expect(html).toContain('retrieval rank 2 of 2');
expect(html).not.toContain('score 0.033');
expect(html).toContain('pages 12–13');
expect(html).toContain('ISO/IEC 27002 · purpose');
expect(html).not.toContain('Synthetic evidence text.');

9
tests/server/qdrant-regulatory-index.test.ts
View File

@ -63,15 +63,19 @@ describe('Qdrant regulatory index', () => {
embeddingModel: 'multilingual-model', timeoutMs: 1_000, fetcher
});
await index.replaceDocuments([{ chunk, embedding: [0.2, 0.8] }]);
await index.replaceDocuments([{ chunk, embedding: [0.2, 0.8], sparseVector: { indices: [7, 11], values: [1.2, 0.5] } }]);
await expect(index.searchDense([0.4, 0.6], 5, { framework: 'ISO/IEC 27001', language: 'DE' }))
.resolves.toEqual([{ score: 0.91, chunk }]);
await expect(index.searchSparse({ indices: [7], values: [1] }, 5, { framework: 'ISO/IEC 27001' }))
.resolves.toEqual([{ score: 0.91, chunk }]);
expect(requests[0].body.filter.must).toEqual([
{ key: 'corpus_id', match: { value: 'iso-27001' } },
{ key: 'document_id', match: { value: 'a-5-1-de' } }
]);
expect(requests[1].body.points[0]).toMatchObject({ vector: { dense: [0.2, 0.8] }, payload: {
expect(requests[1].body.points[0]).toMatchObject({ vector: {
dense: [0.2, 0.8], sparse: { indices: [7, 11], values: [1.2, 0.5] }
}, payload: {
framework: 'ISO/IEC 27001', control_id: 'A.5.1', language: 'de', embedding_model: 'multilingual-model'
} });
expect(requests[2].body.filter.must).toEqual(expect.arrayContaining([
@ -79,6 +83,7 @@ describe('Qdrant regulatory index', () => {
{ key: 'framework', match: { value: 'ISO/IEC 27001' } },
{ key: 'language', match: { value: 'de' } }
]));
expect(requests[3].body).toMatchObject({ query: { indices: [7], values: [1] }, using: 'sparse' });
});
it('fails closed for an incompatible collection and unsafe remote HTTP', async () => {

181
tests/server/regulatory-finding-retrieval.test.ts
View File

@ -1,11 +1,15 @@
import { describe, expect, it, vi } from 'vitest';
import { RegulatoryFindingRetrievalService } from '../../server/regulatory/regulatory-finding-retrieval';
import {
RegulatoryFindingRetrievalService,
mergeFindingQueryRankings,
reciprocalRankFuseByControl
} from '../../server/regulatory/regulatory-finding-retrieval';
import type { RegulatoryRetrievalCandidate, RegulatoryRetrievalRequest } from '../../src/domain/regulatory-rag';
import type { UnifiedResultProjection } from '../../src/domain/unified-result';
import { buildRegulatoryFindingQuery } from '../../src/profiles/public-search/regulatory-query';
// Result-table / occurrence data and raw addresses that must never reach the query.
const resultTableAsset = 'DC01.customer.local';
const resultTableAsset = 'host.customer.invalid';
const reporterEmail = 'ops@acme.invalid';
const ownerEmail = 'analyst@acme.invalid';
@ -13,10 +17,10 @@ function finding(overrides: Partial<UnifiedResultProjection> = {}): UnifiedResul
return {
resultProjectionId: 'projection-1', sourceKind: 'scanner-finding', sourceResultId: 'source-1',
projectId: 'tenant-1', assessmentId: 'assessment-1', collectionRunId: 'run-1',
sourceTool: 'purple-knight',
title: 'SMB signing is not required on domain controllers',
description: `The SMB server permits unsigned sessions, allowing relay attacks. Reported by ${reporterEmail}.`,
recommendation: 'Require SMB signing through Group Policy on all domain controllers.',
sourceTool: 'scanner-one', sourceRuleId: 'RULE-1',
title: 'Unsigned protocol sessions are permitted',
description: `The service permits unsigned sessions. Reported by ${reporterEmail}.`,
recommendation: 'Require signed sessions through the platform policy.',
affectedAsset: resultTableAsset,
severityOriginal: 'high', severityNormalized: 'high', outcomeOriginal: 'fail', outcomeNormalized: 'fail',
technicalExecutionState: 'completed', resultType: 'compliance_check', isIoe: false, isDeviation: true,
@ -26,49 +30,60 @@ function finding(overrides: Partial<UnifiedResultProjection> = {}): UnifiedResul
};
}
const candidate: RegulatoryRetrievalCandidate = {
score: 0.8,
function hit(
controlId: string,
chunkId: string,
score: number,
sectionKind?: RegulatoryRetrievalCandidate['chunk']['sectionKind']
): RegulatoryRetrievalCandidate {
return {
score,
chunk: {
schemaVersion: 1, corpusId: 'synthetic', corpusVersion: '1', documentId: 'control-de',
framework: 'Synthetic Framework', frameworkVersion: '1', controlId: 'CTRL-1', chunkId: 'chunk-1',
schemaVersion: 1, corpusId: 'synthetic', corpusVersion: '1', documentId: `${controlId}-document`,
framework: 'Synthetic Framework', frameworkVersion: '1', controlId, chunkId,
chunkIndex: 0, language: 'de', title: 'Synthetic control', text: 'Synthetic regulatory text.',
textSha256: 'a'.repeat(64)
textSha256: 'a'.repeat(64),
...(sectionKind ? { sectionKind } : {})
}
};
}
describe('regulatory finding retrieval boundary', () => {
it('builds the query from the non-personal check text and never from the result table', () => {
it('builds problem and remediation queries only from privacy-safe finding text', () => {
const query = buildRegulatoryFindingQuery(finding());
expect(query).toMatchObject({
schemaVersion: 1,
specificity: 'topic',
publicTool: 'Semperis Purple Knight',
publicTopics: ['SMB signing requirements']
expect(query).toEqual({
schemaVersion: 4,
problemQuery: 'Security check: Unsigned protocol sessions are permitted. Check detail: The service permits unsigned sessions. Reported by [redacted-email].',
remediationQuery: 'Security check: Unsigned protocol sessions are permitted. Recommended action: Require signed sessions through the platform policy.',
retrievalQuery: 'Security check: Unsigned protocol sessions are permitted. Check detail: The service permits unsigned sessions. Reported by [redacted-email]. Recommended action: Require signed sessions through the platform policy.',
query: 'Retrieve regulatory controls whose normative requirements directly address this security finding. Security check: Unsigned protocol sessions are permitted. Check detail: The service permits unsigned sessions. Reported by [redacted-email]. Recommended action: Require signed sessions through the platform policy. Prefer controls with direct requirement text over merely related guidance. Do not infer customer scope, assets, configuration values, or compliance status.'
});
// The rule-level check text is the intended source.
expect(query?.query).toContain('SMB signing is not required on domain controllers');
expect(query?.query).toContain('unsigned sessions');
// Result-table / occurrence data and raw email addresses never appear.
expect(query?.query).not.toContain(resultTableAsset);
expect(query?.query).not.toContain(reporterEmail);
expect(query?.query).not.toContain(ownerEmail);
expect(query?.query).toContain('[redacted-email]');
for (const marker of [resultTableAsset, reporterEmail, ownerEmail, 'scanner-one', 'RULE-1']) {
expect(JSON.stringify(query)).not.toContain(marker);
}
});
it('maps every scanner finding, sharpening with a catalog rule ID when available', () => {
expect(buildRegulatoryFindingQuery(finding({
sourceTool: 'pingcastle', sourceRuleId: 'a-krbtgt'
}))).toMatchObject({ specificity: 'rule', publicRuleId: 'A-Krbtgt' });
it('produces identical retrieval input for identical findings from unknown or known scanners', () => {
const first = buildRegulatoryFindingQuery(finding({ sourceTool: 'scanner-one', sourceRuleId: 'RULE-1' }));
const second = buildRegulatoryFindingQuery(finding({ sourceTool: 'brand-new-scanner', sourceRuleId: 'NEW-999' }));
// Unknown tool, non-catalog rule ID, no topic match: still mapped from the check title.
const generic = buildRegulatoryFindingQuery(finding({
sourceTool: 'customer-private-scanner', sourceRuleId: 'A-Custom',
title: 'Weak service account configuration', description: undefined, recommendation: undefined
expect(second).toEqual(first);
});
it('uses one problem query when no recommendation is available', () => {
const query = buildRegulatoryFindingQuery(finding({
title: 'Excessive service privileges',
description: undefined,
recommendation: undefined
}));
expect(generic).toMatchObject({ specificity: 'topic' });
expect(generic?.query).toContain('Weak service account configuration');
expect(generic?.publicRuleId).toBeUndefined();
expect(query).toMatchObject({
schemaVersion: 4,
problemQuery: 'Security check: Excessive service privileges.',
retrievalQuery: 'Security check: Excessive service privileges.'
});
expect(query?.remediationQuery).toBeUndefined();
});
it('fails closed only for non-scanner records or findings without a check title', () => {
@ -76,24 +91,94 @@ describe('regulatory finding retrieval boundary', () => {
expect(buildRegulatoryFindingQuery(finding({ title: ' ' }))).toBeUndefined();
});
it('passes only the prepared query and explicit filters to the retriever', async () => {
const retrieve = vi.fn(async (_request: RegulatoryRetrievalRequest) => [candidate]);
it('runs problem, remediation and section-specific full-context queries', async () => {
const retrieve = vi.fn(async (request: RegulatoryRetrievalRequest) => [hit(
'C-1',
`chunk-${request.filters?.sectionKind ?? (request.query.includes('Recommended action') ? 'remediation' : 'problem')}`,
0.8,
request.filters?.sectionKind
)]);
const service = new RegulatoryFindingRetrievalService({ retrieve });
const result = await service.retrieve(finding(), 5, { framework: 'Synthetic Framework' });
const result = await service.retrieve(finding(), 12, { framework: 'Synthetic Framework' });
expect(result).toMatchObject({ sourceResultId: 'source-1', status: 'candidates', candidates: [candidate] });
expect(retrieve).toHaveBeenCalledOnce();
expect(retrieve.mock.calls[0][0]).toEqual({
query: result.preparedQuery?.query,
limit: 5,
filters: { framework: 'Synthetic Framework' }
});
for (const marker of [resultTableAsset, reporterEmail, ownerEmail]) {
expect(JSON.stringify(retrieve.mock.calls[0][0])).not.toContain(marker);
expect(result).toMatchObject({ sourceResultId: 'source-1', status: 'candidates' });
expect(retrieve.mock.calls.map(call => call[0])).toEqual([
{ query: result.preparedQuery?.problemQuery, limit: 64, filters: { framework: 'Synthetic Framework' } },
{ query: result.preparedQuery?.remediationQuery, limit: 64, filters: { framework: 'Synthetic Framework' } },
...(['control', 'purpose', 'guidance'] as const).map(sectionKind => ({
query: result.preparedQuery?.retrievalQuery,
limit: 32,
filters: { framework: 'Synthetic Framework', sectionKind }
}))
]);
expect(result.candidates.map(item => item.chunk.sectionKind).filter(Boolean))
.toEqual(['control', 'purpose', 'guidance']);
for (const marker of [resultTableAsset, reporterEmail, ownerEmail, 'scanner-one', 'RULE-1']) {
expect(JSON.stringify(retrieve.mock.calls)).not.toContain(marker);
}
});
it('does not duplicate the broad search when no remediation text is present', async () => {
const retrieve = vi.fn(async () => [hit('C-1', 'chunk-1', 0.8)]);
const service = new RegulatoryFindingRetrievalService({ retrieve });
await service.retrieve(finding({ recommendation: undefined }), 12);
expect(retrieve).toHaveBeenCalledTimes(4);
});
it('fuses each control once per independent finding query', () => {
const fused = reciprocalRankFuseByControl([
[hit('C-1', 'problem-c1-a', 0.9), hit('C-1', 'problem-c1-b', 0.89), hit('C-2', 'problem-c2', 0.8)],
[hit('C-2', 'remediation-c2', 0.85), hit('C-3', 'remediation-c3', 0.75), hit('C-1', 'remediation-c1', 0.65)]
], 2);
expect(fused.map(item => item.chunk.chunkId)).toEqual([
'problem-c2', 'remediation-c2', 'problem-c1-a', 'remediation-c1'
]);
expect(new Set(fused.map(item => item.chunk.controlId))).toEqual(new Set(['C-1', 'C-2']));
expect(fused[0].score).toBeGreaterThan(fused[2].score);
});
it('lets the collapsed full-context section family contribute a shortlist control', () => {
const selected = mergeFindingQueryRankings(
[
[hit('C-1', 'problem-c1', 0.9), hit('C-2', 'problem-c2', 0.8)],
[hit('C-1', 'remediation-c1', 0.9)]
],
[
[hit('C-3', 'control-c3', 1, 'control'), hit('C-1', 'control-c1', 0.7, 'control')],
[hit('C-3', 'purpose-c3', 1, 'purpose'), hit('C-1', 'purpose-c1', 0.7, 'purpose')],
[hit('C-3', 'guidance-c3', 1, 'guidance'), hit('C-1', 'guidance-c1', 0.7, 'guidance')]
],
2
);
expect(new Set(selected.map(item => item.chunk.controlId))).toEqual(new Set(['C-1', 'C-3']));
expect(selected.some(item => item.chunk.controlId === 'C-2')).toBe(false);
expect(selected.filter(item => item.chunk.controlId === 'C-1').map(item => item.chunk.chunkId))
.toEqual(['control-c1', 'purpose-c1', 'guidance-c1', 'problem-c1', 'remediation-c1']);
});
it('counts the three full-context sections as one final ranking vote', () => {
const selected = mergeFindingQueryRankings(
[
[hit('C-1', 'problem-c1', 0.9), hit('C-2', 'problem-c2', 0.8)],
[hit('C-2', 'remediation-c2', 0.9)]
],
[
[hit('C-3', 'control-c3', 1, 'control')],
[hit('C-3', 'purpose-c3', 1, 'purpose')],
[hit('C-3', 'guidance-c3', 1, 'guidance')]
],
2
);
expect(new Set(selected.map(item => item.chunk.controlId))).toEqual(new Set(['C-1', 'C-2']));
expect(selected.some(item => item.chunk.controlId === 'C-3')).toBe(false);
});
it('does not call retrieval for a finding without a check title', async () => {
const retrieve = vi.fn();
const service = new RegulatoryFindingRetrievalService({ retrieve });

57
tests/server/regulatory-mapping-comparison-script.test.ts Normal file
View File

@ -0,0 +1,57 @@
import { describe, expect, it } from 'vitest';
// The reusable evaluation command is plain ESM so it can run directly with Node.
// @ts-expect-error The script intentionally has no separate declaration file.
import { changeFor, evaluateRun, validateBaseline } from '../../scripts/compare-regulatory-mappings.mjs';
function run(candidates: string[], mappings: Array<{ controlId: string; relationship: 'primary' | 'supporting' }>, status = 'succeeded') {
return {
run_status: status,
data: {
candidates: candidates.map((controlId, index) => ({ controlId, score: 1 / (61 + index) })),
result: { mappings: mappings.map(mapping => ({ ...mapping, confidence: 'high' })) }
}
};
}
const finding = { sourceRuleId: 'RULE-1', primary: '1.1', supporting: ['1.2'] };
describe('regulatory mapping comparison script', () => {
it('allows a manual baseline with no primary and rejects duplicate relationships', () => {
expect(() => validateBaseline({
schemaVersion: 1,
scanners: [{ sourceTool: 'scanner', label: 'Scanner', findings: [{ sourceRuleId: 'RULE-1', primary: null, supporting: ['1.1'] }] }]
})).not.toThrow();
expect(() => validateBaseline({
schemaVersion: 1,
scanners: [{ sourceTool: 'scanner', label: 'Scanner', findings: [{ sourceRuleId: 'RULE-1', primary: '1.1', supporting: ['1.1'] }] }]
})).toThrow(/duplicate controls/i);
});
it('separates retrieval, selection, role and extra-control results', () => {
const evaluation = evaluateRun(run(
['1.1', '1.2', '2.1'],
[{ controlId: '1.1', relationship: 'supporting' }, { controlId: '2.1', relationship: 'primary' }]
), finding);
expect(evaluation.retrievalGaps).toEqual([]);
expect(evaluation.selectionGaps).toEqual(['1.2']);
expect(evaluation.roleMismatches).toEqual(['1.1']);
expect(evaluation.extras).toEqual(['2.1']);
expect(evaluation.verdict).toBe('Incomplete + extras');
});
it('classifies opposing recall and extra-control changes as mixed', () => {
const previous = evaluateRun(run(['1.1'], [{ controlId: '1.1', relationship: 'primary' }]), finding);
const current = evaluateRun(run(
['1.1', '1.2', '2.1'],
[
{ controlId: '1.1', relationship: 'primary' },
{ controlId: '1.2', relationship: 'supporting' },
{ controlId: '2.1', relationship: 'supporting' }
]
), finding);
expect(changeFor(previous, current)).toBe('Mixed');
});
});

132
tests/server/regulatory-mapping-service.test.ts
View File

@ -101,20 +101,33 @@ function mockRepository(findings: UnifiedResultProjection[]) {
function service(repository: AssessorRepository, complete: (system: string, input: string) => Promise<unknown>) {
return new RegulatoryMappingService(repository, {
findingRetrieval: new RegulatoryFindingRetrievalService({ retrieve: vi.fn(async (_r: RegulatoryRetrievalRequest) => [{ score: 0.9, chunk: chunk('ISO/IEC 27002') }]) }),
index: { ensureCollection: vi.fn(), replaceDocuments: vi.fn(), searchDense: vi.fn(), list: vi.fn(async () => [chunk('ISO/IEC 27001')]) },
index: { ensureCollection: vi.fn(), replaceDocuments: vi.fn(), searchDense: vi.fn(), searchSparse: vi.fn(), list: vi.fn(async () => [chunk('ISO/IEC 27001')]) },
client: { model: 'work-model', complete: complete as never }, corpusId: 'iso-test', corpusVersion: 'v2', embeddingModel: 'bge-m3',
now: () => new Date(stamp)
});
}
function assessed(
relevance: 'primary' | 'supporting' | 'irrelevant' = 'irrelevant',
overrides: Record<string, unknown> = {}
) {
return {
schemaVersion: 1,
outcome: 'assessed',
assessments: [{
controlId: '5.1', relevance, confidence: relevance === 'irrelevant' ? 'low' : 'high',
rationale: 'Synthetic assessment.', evidenceChunkIds: ['semantic-5.1'], ...overrides
}]
};
}
describe('regulatory mapping service', () => {
it('shares one global mapping and decision across projects and sends no result-table data', async () => {
const findingB = finding({ resultProjectionId: 'scanner-finding:occurrence-2', sourceResultId: 'occurrence-2', projectId: 'project-2', affectedAsset: 'DC02.other.local' });
const repository = mockRepository([finding(), findingB]);
const complete = vi.fn(async (_system: string, _input: string) => ({ value: {
schemaVersion: 1, outcome: 'mapped', mappings: [{ controlId: '5.1', relationship: 'primary', confidence: 'high',
rationale: 'Synthetic rationale.', evidenceChunkIds: ['semantic-5.1', 'reference-5.1'] }]
}, usage: { totalTokens: 12 } }));
const complete = vi.fn(async (_system: string, _input: string) => ({ value: assessed('primary', {
evidenceChunkIds: ['semantic-5.1', 'reference-5.1']
}), usage: { totalTokens: 12 } }));
const svc = service(repository, complete);
await svc.start('project-1', { resultProjectionId: 'scanner-finding:occurrence-1' });
@ -125,10 +138,12 @@ describe('regulatory mapping service', () => {
expect(shared).toMatchObject({ state: 'succeeded', run: { ruleKey: 'purple-knight:A-SmbSigning', sourceTool: 'purple-knight', embeddingModel: 'bge-m3', tokenUsage: { totalTokens: 12 } } });
expect(shared.run?.result?.mappings[0].controlId).toBe('5.1');
expect(complete).toHaveBeenCalledOnce();
expect(complete.mock.calls[0][0]).toContain('Assess every supplied ISO control candidate independently');
expect(complete.mock.calls[0][0]).toContain('zero or one primary and zero to three supporting controls');
// The copyrighted ISO clause text is stripped from the response but still reaches the model.
expect(shared.run?.candidates.every(c => c.evidence.every(item => item.text === undefined))).toBe(true);
expect(complete.mock.calls[0][0]).toContain('governs the same security subject');
expect(complete.mock.calls[0][0]).toContain('root security subject');
expect(complete.mock.calls[0][1]).toContain('Synthetic semantic evidence.');
expect(complete.mock.calls[0][1]).toContain('SMB signing is not required');
expect(complete.mock.calls[0][1]).not.toContain(resultTableAsset);
@ -142,19 +157,25 @@ describe('regulatory mapping service', () => {
});
it('drops one ungrounded evidence id but keeps the otherwise-valid mapping', async () => {
const { svc } = await runWith({
schemaVersion: 1, outcome: 'mapped', mappings: [{ controlId: '5.1', relationship: 'primary', confidence: 'high',
rationale: 'Synthetic rationale.', evidenceChunkIds: ['semantic-5.1', 'not-a-supplied-chunk'] }]
});
const { svc } = await runWith(assessed('primary', { evidenceChunkIds: ['semantic-5.1', 'not-a-supplied-chunk'] }));
const completed = await svc.get('project-1', 'scanner-finding:occurrence-1');
expect(completed.run?.status).toBe('succeeded');
expect(completed.run?.result?.mappings[0].evidenceChunkIds).toEqual(['semantic-5.1']);
expect(completed.run?.errorCode).toBeUndefined();
});
it('accepts a grounded supporting-only mapping when no candidate is clearly primary', async () => {
const { svc } = await runWith(assessed('supporting', {
confidence: 'medium', rationale: 'Relevant, but not uniquely primary.'
}));
const completed = await svc.get('project-1', 'scanner-finding:occurrence-1');
expect(completed.run?.status).toBe('succeeded');
expect(completed.run?.result?.mappings).toEqual([expect.objectContaining({ controlId: '5.1', relationship: 'supporting' })]);
});
it('fails safely with a structural detail when the model maps to an unknown control', async () => {
const { svc } = await runWith({
schemaVersion: 1, outcome: 'mapped', mappings: [{ controlId: 'A.5.1', relationship: 'primary', confidence: 'high',
schemaVersion: 1, outcome: 'assessed', assessments: [{ controlId: 'A.5.1', relevance: 'primary', confidence: 'high',
rationale: 'Synthetic rationale.', evidenceChunkIds: ['semantic-5.1'] }]
});
const completed = await svc.get('project-1', 'scanner-finding:occurrence-1');
@ -164,6 +185,78 @@ describe('regulatory mapping service', () => {
expect(completed.run?.errorDetail).not.toContain(resultTableAsset);
});
it('retrieves a broad control pool and sends only the best twelve controls to the model', async () => {
const repository = mockRepository([finding()]);
const semanticCandidates = Array.from({ length: 13 }, (_, index) => {
const controlId = `8.${index + 1}`;
return {
score: 1 - index / 100,
chunk: {
...chunk('ISO/IEC 27002'),
controlId,
annexAControlId: `A.${controlId}`,
documentId: `27002-${controlId}`,
chunkId: `semantic-${controlId}`,
title: `Synthetic control ${controlId}`,
sectionKind: 'control' as const
}
};
});
const retrieve = vi.fn(async (_finding: UnifiedResultProjection, limit: number) => {
expect(limit).toBe(12);
return {
sourceResultId: 'occurrence-1', status: 'candidates' as const,
preparedQuery: {
schemaVersion: 3 as const,
query: 'Synthetic public mapping context.',
retrievalQuery: 'Synthetic concise retrieval query.',
specificity: 'rule' as const,
publicTool: 'Semperis Purple Knight',
publicRuleId: 'A-SmbSigning',
publicTopics: ['SMB signing requirements']
},
candidates: semanticCandidates
};
});
const list = vi.fn(async (filters: { controlId?: string }) => {
const controlId = filters.controlId ?? '8.1';
return [{
...chunk('ISO/IEC 27001'),
controlId,
annexAControlId: `A.${controlId}`,
documentId: `27001-${controlId}`,
chunkId: `reference-${controlId}`,
title: `Synthetic control ${controlId}`
}];
});
const complete = vi.fn(async (_system: string, input: string) => {
const payload = JSON.parse(input) as { candidates: Array<{ controlId: string }> };
expect(payload.candidates).toHaveLength(12);
expect(payload.candidates.map(candidate => candidate.controlId)).toEqual(Array.from({ length: 12 }, (_, index) => `8.${index + 1}`));
return { value: {
schemaVersion: 1,
outcome: 'assessed',
assessments: payload.candidates.map(candidate => ({
controlId: candidate.controlId, relevance: 'irrelevant', confidence: 'low', rationale: 'Synthetic assessment.',
evidenceChunkIds: [`semantic-${candidate.controlId}`]
}))
} };
});
const svc = new RegulatoryMappingService(repository, {
findingRetrieval: { retrieve } as never,
index: { ensureCollection: vi.fn(), replaceDocuments: vi.fn(), searchDense: vi.fn(), searchSparse: vi.fn(), list },
client: { model: 'work-model', complete: complete as never },
corpusId: 'iso-test', corpusVersion: 'v2', embeddingModel: 'bge-m3', now: () => new Date(stamp)
});
await svc.start('project-1', { resultProjectionId: 'scanner-finding:occurrence-1' });
await svc.close();
const completed = await svc.get('project-1', 'scanner-finding:occurrence-1');
expect(completed.run?.candidates).toHaveLength(12);
expect(complete).toHaveBeenCalledOnce();
});
it('maps the complete filtered timeslot, excludes non-scanner results and deduplicates global rules', async () => {
const findings = [
finding(),
@ -174,10 +267,9 @@ describe('regulatory mapping service', () => {
finding({ resultProjectionId: 'scanner-finding:not-deviation', sourceResultId: 'not-deviation', sourceRuleId: 'A-Ignored', isDeviation: false })
];
const repository = mockRepository(findings);
const complete = vi.fn(async (_system: string, _input: string) => ({ value: {
schemaVersion: 1, outcome: 'mapped', mappings: [{ controlId: '5.1', relationship: 'primary', confidence: 'high',
rationale: 'Synthetic rationale.', evidenceChunkIds: ['semantic-5.1', 'reference-5.1'] }]
} }));
const complete = vi.fn(async (_system: string, _input: string) => ({ value: assessed('primary', {
evidenceChunkIds: ['semantic-5.1', 'reference-5.1']
}) }));
const svc = service(repository, complete);
await svc.startBatch('project-1', {
@ -217,7 +309,7 @@ describe('regulatory mapping service', () => {
maximum = Math.max(maximum, active);
await new Promise(resolve => setTimeout(resolve, 5));
active -= 1;
return { value: { schemaVersion: 1, outcome: 'no_relevant_control', reason: 'Synthetic.', mappings: [] } };
return { value: assessed() };
});
const svc = service(mockRepository(findings), complete);
await svc.startBatch('project-1', {
@ -234,9 +326,7 @@ describe('regulatory mapping service', () => {
sourceResultId: `occurrence-${index}`
}));
const repository = mockRepository(findings);
const complete = vi.fn(async () => ({ value: {
schemaVersion: 1, outcome: 'no_relevant_control', reason: 'Synthetic.', mappings: []
} }));
const complete = vi.fn(async () => ({ value: assessed() }));
const svc = service(repository, complete);
await svc.startBatch('project-1', {
assessmentIds: ['assessment-1'], scanRunIds: ['run-1'], filters: { deviation: true }, idempotencyKey: 'batch-many-pages-1'
@ -282,7 +372,7 @@ describe('regulatory mapping service', () => {
let secondAttempts = 0;
const complete = vi.fn(async (_system: string, input: string) => {
if (input.includes('Second synthetic check') && ++secondAttempts === 1) throw new Error('synthetic provider failure');
return { value: { schemaVersion: 1, outcome: 'no_relevant_control', reason: 'Synthetic.', mappings: [] } };
return { value: assessed() };
});
const svc = service(mockRepository(findings), complete);
const scope = { assessmentIds: ['assessment-1'], scanRunIds: ['run-1'], filters: { deviation: true } };
@ -307,7 +397,7 @@ describe('regulatory mapping service', () => {
const gate = new Promise<void>(resolve => { release = resolve; });
const complete = vi.fn(async () => {
await gate;
return { value: { schemaVersion: 1, outcome: 'no_relevant_control', reason: 'Synthetic.', mappings: [] } };
return { value: assessed() };
});
const svc = service(mockRepository([finding()]), complete);
await svc.start('project-1', { resultProjectionId: 'scanner-finding:occurrence-1' });

2
tests/server/regulatory-mapping.integration.test.ts
View File

@ -60,7 +60,7 @@ describeIntegration('regulatory mapping end-to-end integration', () => {
await service.close();
const result = await service.get(project.id, resultProjectionId);
expect(result.run?.status, result.run?.errorCode).toBe('succeeded');
expect(result.run?.candidates).toHaveLength(8);
expect(result.run?.candidates).toHaveLength(12);
expect(result.run?.candidates.every(candidate => candidate.evidence.some(item => item.sourceStandard === 'ISO/IEC 27001'))).toBe(true);
expect(result.run?.result?.mappings.length).toBeLessThanOrEqual(5);
expect(result.run?.result?.mappings.every(mapping => {

52
tests/server/regulatory-rag-service.test.ts
View File

@ -3,9 +3,12 @@ import type { RegulatoryControlDocument, RegulatoryRetrievalCandidate } from '..
import type { TextEmbeddingClient } from '../../server/regulatory/embedding-client';
import type { IndexedRegulatoryChunk, RegulatoryChunkIndex } from '../../server/regulatory/qdrant-regulatory-index';
import {
buildRegulatorySparseVectors,
DenseRegulatoryCandidateRetriever,
HybridRegulatoryCandidateRetriever,
RegulatoryIngestionService,
RegulatoryRetrievalPipeline
RegulatoryRetrievalPipeline,
SparseRegulatoryCandidateRetriever
} from '../../server/regulatory/regulatory-rag-service';
const document: RegulatoryControlDocument = {
@ -26,7 +29,7 @@ describe('regulatory RAG services', () => {
};
const ensureCollection = vi.fn(async () => {});
const replaceDocuments = vi.fn(async (_chunks: IndexedRegulatoryChunk[]) => {});
const index: RegulatoryChunkIndex = { ensureCollection, replaceDocuments, searchDense: vi.fn(), list: vi.fn() };
const index: RegulatoryChunkIndex = { ensureCollection, replaceDocuments, searchDense: vi.fn(), searchSparse: vi.fn(), list: vi.fn() };
const service = new RegulatoryIngestionService(embeddings, index, { maximumChunkCharacters: 220, embeddingBatchSize: 2 });
const result = await service.ingest([document]);
@ -37,6 +40,7 @@ describe('regulatory RAG services', () => {
expect(ensureCollection).toHaveBeenCalledOnce();
expect(replaceDocuments).toHaveBeenCalledOnce();
expect(replaceDocuments.mock.calls[0][0]).toHaveLength(result.chunkCount);
expect(replaceDocuments.mock.calls[0][0].every(entry => entry.sparseVector?.indices.length)).toBe(true);
});
it('supports multilingual dense candidates and an optional later reranker stage', async () => {
@ -48,7 +52,7 @@ describe('regulatory RAG services', () => {
const searchDense = vi.fn(async () => candidates);
const dense = new DenseRegulatoryCandidateRetriever(
{ model: 'multilingual-model', dimensions: 2, embed },
{ ensureCollection: vi.fn(), replaceDocuments: vi.fn(), searchDense, list: vi.fn() }
{ ensureCollection: vi.fn(), replaceDocuments: vi.fn(), searchDense, searchSparse: vi.fn(), list: vi.fn() }
);
const rerank = vi.fn(async (_query: string, values: RegulatoryRetrievalCandidate[]) => values.slice().reverse());
const retrieval = new RegulatoryRetrievalPipeline(dense, { rerank }, 4);
@ -60,4 +64,46 @@ describe('regulatory RAG services', () => {
expect(rerank).toHaveBeenCalledOnce();
expect(result.map(item => item.chunk.chunkId)).toEqual(['chunk-7', 'chunk-6']);
});
it('combines dense and lexical rankings once per control with reciprocal-rank fusion', async () => {
const candidate = (controlId: string, chunkId: string, score: number): RegulatoryRetrievalCandidate => ({
score,
chunk: { ...document, controlId, chunkId, chunkIndex: 0, textSha256: chunkId.padEnd(64, '0') }
});
const dense = { retrieveCandidates: vi.fn(async () => [
candidate('C-1', 'dense-c1-a', 0.9),
candidate('C-1', 'dense-c1-b', 0.89),
candidate('C-2', 'dense-c2', 0.8)
]) };
const sparse = { retrieveCandidates: vi.fn(async () => [
candidate('C-2', 'sparse-c2', 4.2),
candidate('C-3', 'sparse-c3', 3.1)
]) };
const hybrid = new HybridRegulatoryCandidateRetriever(dense, sparse);
const result = await hybrid.retrieveCandidates({ query: 'synthetic access control', limit: 3 });
expect(result.map(item => item.chunk.controlId)).toEqual(['C-2', 'C-1', 'C-3']);
expect(result.map(item => item.chunk.chunkId)).toEqual(['sparse-c2', 'dense-c1-a', 'sparse-c3']);
expect(dense.retrieveCandidates).toHaveBeenCalledOnce();
expect(sparse.retrieveCandidates).toHaveBeenCalledOnce();
});
it('builds deterministic BM25 vectors and uses the sparse Qdrant slot for lexical retrieval', async () => {
const chunks = [
{ ...document, chunkId: 'chunk-a', chunkIndex: 0, textSha256: 'a'.repeat(64), title: 'Privileged access', text: 'Restrict privileged access rights.' },
{ ...document, chunkId: 'chunk-b', chunkIndex: 1, textSha256: 'b'.repeat(64), title: 'Backup', text: 'Protect backup copies.' }
];
const vectors = buildRegulatorySparseVectors(chunks);
expect(vectors.get('chunk-a')?.indices).toEqual(vectors.get('chunk-a')?.indices.slice().sort((left, right) => left - right));
expect(vectors.get('chunk-a')?.values.every(value => value > 0)).toBe(true);
const searchSparse = vi.fn(async () => [{ score: 2.4, chunk: chunks[0] }]);
const sparse = new SparseRegulatoryCandidateRetriever({
ensureCollection: vi.fn(), replaceDocuments: vi.fn(), searchDense: vi.fn(), searchSparse, list: vi.fn()
});
await expect(sparse.retrieveCandidates({ query: 'privileged access', limit: 5, filters: { language: 'de' } }))
.resolves.toEqual([{ score: 2.4, chunk: chunks[0] }]);
expect(searchSparse).toHaveBeenCalledWith(expect.objectContaining({ indices: expect.any(Array), values: expect.any(Array) }), 5, { language: 'de' });
});
});