DevSecOps
/
Assessor_2.0
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add regulatory mapping batch functionality

This commit is contained in:
Binh 2026-09-17 09:56:17 +02:00
parent 527deabb03
commit 55da7867b8
21 changed files with 1202 additions and 75 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

10
docs/IMPLEMENTATION_PLAN.md
View File

@ -473,6 +473,8 @@ Referenzerkennung 2026-09-26 (Phase 13D, erster Umfang umgesetzt und verifiziert
Status: lebendes Projektdokument Status: lebendes Projektdokument
Gefiltertes Result-DB-ISO-Batchmapping 2026-09-16 (Phase 13F, ausdruecklicher Nutzerauftrag): Result DB bietet `Map all ISO controls` neben Refresh und Export JSON. Export und Batch leiten ihren Scope aus demselben normalisierten `UnifiedResultQuery` ab; der gewaehlte Timeslot bleibt aeussere Grenze, alle aktiven Filter einschliesslich des Standardfilters `deviation: true` grenzen weiter ein, und Pagination/geladene Zeilen haben keinen Einfluss. Der Server liest stabil alle Treffer, bricht bei veraenderter Gesamtzahl sicher ab, zaehlt Nicht-Scanner und nicht unterstuetzte Findings getrennt, dedupliziert global nach `ruleKey`, verwendet erfolgreiche/aktive Mappings wieder und plant fehlende beziehungsweise zuvor fehlgeschlagene Regeln ueber dieselbe auf zwei parallele Retrieval-/Modellpipelines begrenzte Queue wie Einzelaktionen. Atomare deterministische Claims verhindern doppelte Modellaufrufe. Migration 027 speichert nur den projektgebundenen normalisierten Filter-/Timeslot-Snapshot, Hash, Idempotenz, Status und Aggregatfortschritt; globale Mappingzeilen bleiben autoritativ. Das Dialogfenster zeigt Treffer und Fortschritt, bindet laufende Batches an den beim Start erfassten Filterstand und erzeugt ausschliesslich Vorschlaege; Accept/Reject bleibt manuell in Work. Filterwerte, Occurrence-Assets, Pfade, Mandanten- und Reviewdaten gelangen nicht in Retrieval oder LiteLLM. Die fruehere Batch-Zurueckstellung ist fuer diesen manuellen gefilterten Lauf ausdruecklich aufgehoben; die repraesentative Retrievalbewertung bleibt offenes Qualitaetsrisiko. Keine Scannerverarbeitung, kein Manifestbump und kein Reimport. Verifikation: `npm test` mit 1.315 bestandenen Tests und zehn uebersprungenen opt-in Integrationen, Local-only-/Designpruefung sowie Client-/Server-Produktionsbuild erfolgreich (bestehender Bundlehinweis). Der synthetische regulatorische End-to-End-Integrationstest blieb ohne aktivierte lokale Abhaengigkeiten uebersprungen. Korrektur nach Anwendung von Migration 027: Eine nachtraeglich in dieselbe Datei aufgenommene reine Sortiersequenz verletzte die Migrationsimmutabilitaet und der Checksum-Schutz stoppte das Backend korrekt. 027 ist exakt auf den angewandten Stand ohne Sortiersequenz zurueckgesetzt und durch eine feste SHA-256-Regression (`d89055c3e723b09d8d8be7f4225bfd08442f0d66e4e0b4ea3aa62ae6209e9ee9`) geschuetzt; Repositoryabfragen verwenden wieder den bereits angelegten stabilen `created_at, id`-Index. Es gibt keine Migration 028 und keine neue Datenbankmutation. Nach freigegebenem WSL-Neustart wurden der gesunde kanonische Container, Port `127.0.0.1:5432`, Volume `assessor-2_0_assessor-postgres-data`, der identische Ledger-Checksum und das Fehlen von 028 read-only bestaetigt; der automatisch neu gestartete Backendprozess meldet HTTP 200 und `persistence=postgresql`. Kein Backup war fuer die Korrektur erforderlich, weil keine neue Migration ausgefuehrt wurde.
ISO-27002-Mapping-Vertikalschnitt 2026-09-15 (Phase 13F, ausdruecklicher Nutzerauftrag): Die beiden lokal lizenzierten deutschen DIN-2024-PDFs werden serverseitig mit PDF.js und Positionsdaten in einen nicht eingecheckten Korpus-v2 ueberfuehrt. Passwortpflichtige, kopiergeschuetzte, textlose, uebergrosse oder strukturell unvollstaendige Dokumente scheitern geschlossen; ISO 27001 und 27002 muessen dieselben 93 Kontrollen mit 37/8/14/34 Themenkontrollen liefern. `assessor_regulatory_chunks_v2` enthaelt 705 mit lokalem BGE-M3 erzeugte, abschnitts- und kontrollsichere Chunks aus 558 Dokumenten, einschliesslich ISO-27001-Referenzen, Seiten-, Editions-, Parser- und Hashprovenienz; Sparse bleibt reserviert. Der Einzel-Finding-Dienst gruppiert 32 Dense-Treffer auf acht Kontrollen, bindet ISO-27001-Evidenz an, uebergibt nur die minimierte Public-Signal-Query und hoechstens 48.000 Zeichen ISO-Evidenz an das vorhandene Work-LiteLLM und akzeptiert ausschliesslich lokal validierte, evidenzgebundene JSON-Vorschlaege. Migration 025, beide Repositories und drei projektgebundene APIs speichern Runs und Accept/Reject-Entscheidungsereignisse append-only; Work zeigt den manuellen Ablauf inline mit Polling und Seitenbelegen. Ein realer synthetischer End-to-End-Test ueber kanonisches PostgreSQL, llama.cpp/BGE-M3, Qdrant und den freigegebenen LiteLLM-Gateway bestand einschliesslich Optimistic-Conflict und Cascade-Cleanup. Der laufende Entwicklungsserver wendete die additive Migration um 13:42 vor dem um 13:52 erstellten und verifizierten Backup automatisch an; beide neuen Tabellen waren leer. Diese Abweichung ist offen dokumentiert und wird nicht als Vorabbackup ausgegeben. Scannerverarbeitung blieb unveraendert; kein Manifestbump oder Reimport. ISO-27002-Mapping-Vertikalschnitt 2026-09-15 (Phase 13F, ausdruecklicher Nutzerauftrag): Die beiden lokal lizenzierten deutschen DIN-2024-PDFs werden serverseitig mit PDF.js und Positionsdaten in einen nicht eingecheckten Korpus-v2 ueberfuehrt. Passwortpflichtige, kopiergeschuetzte, textlose, uebergrosse oder strukturell unvollstaendige Dokumente scheitern geschlossen; ISO 27001 und 27002 muessen dieselben 93 Kontrollen mit 37/8/14/34 Themenkontrollen liefern. `assessor_regulatory_chunks_v2` enthaelt 705 mit lokalem BGE-M3 erzeugte, abschnitts- und kontrollsichere Chunks aus 558 Dokumenten, einschliesslich ISO-27001-Referenzen, Seiten-, Editions-, Parser- und Hashprovenienz; Sparse bleibt reserviert. Der Einzel-Finding-Dienst gruppiert 32 Dense-Treffer auf acht Kontrollen, bindet ISO-27001-Evidenz an, uebergibt nur die minimierte Public-Signal-Query und hoechstens 48.000 Zeichen ISO-Evidenz an das vorhandene Work-LiteLLM und akzeptiert ausschliesslich lokal validierte, evidenzgebundene JSON-Vorschlaege. Migration 025, beide Repositories und drei projektgebundene APIs speichern Runs und Accept/Reject-Entscheidungsereignisse append-only; Work zeigt den manuellen Ablauf inline mit Polling und Seitenbelegen. Ein realer synthetischer End-to-End-Test ueber kanonisches PostgreSQL, llama.cpp/BGE-M3, Qdrant und den freigegebenen LiteLLM-Gateway bestand einschliesslich Optimistic-Conflict und Cascade-Cleanup. Der laufende Entwicklungsserver wendete die additive Migration um 13:42 vor dem um 13:52 erstellten und verifizierten Backup automatisch an; beide neuen Tabellen waren leer. Diese Abweichung ist offen dokumentiert und wird nicht als Vorabbackup ausgegeben. Scannerverarbeitung blieb unveraendert; kein Manifestbump oder Reimport.
Regulatory-RAG-Llama-Livebaseline 2026-09-15 (Phase 13F, Nutzerstart): Der lokal gestartete llama.cpp-Server Build 10472 meldet Health 200 und bedient BGE-M3 FP16 ueber den OpenAI-kompatiblen Endpoint. Die produktiv gebauten CLIs indexierten den eingecheckten ausschliesslich synthetischen EN/DE-Korpus mit sechs Dokumenten als sechs 1.024-dimensionale Chunks in den kanonischen Qdrant 1.19.1. Drei ausdruecklich sprachuebergreifende Smoke-Cases erreichten bei K=3 HitRate 1,0, Recall 1,0 und MRR 1,0; alle erwarteten Kontrollen standen auf Rang 1. Das belegt End-to-End-Verkabelung und grundlegende Sprachueberquerung, wegen nur drei klar getrennten Kontrollthemen aber keine produktive Retrievalqualitaet. Qdrant enthaelt sechs synthetische Punkte; PostgreSQL und Realdaten wurden nicht beruehrt. 1.285 Standardtests bestanden, acht optionale PostgreSQL- und ein optionaler Qdrant-Test uebersprungen; Local-only-/Designpruefung und beide Produktionsbuilds erfolgreich (bestehender Bundlehinweis). Ein repraesentativer autorisierter Korpus und unabhaengig gelabelter EN/DE-Satz bleiben fuer die reale Baseline erforderlich. Regulatory-RAG-Llama-Livebaseline 2026-09-15 (Phase 13F, Nutzerstart): Der lokal gestartete llama.cpp-Server Build 10472 meldet Health 200 und bedient BGE-M3 FP16 ueber den OpenAI-kompatiblen Endpoint. Die produktiv gebauten CLIs indexierten den eingecheckten ausschliesslich synthetischen EN/DE-Korpus mit sechs Dokumenten als sechs 1.024-dimensionale Chunks in den kanonischen Qdrant 1.19.1. Drei ausdruecklich sprachuebergreifende Smoke-Cases erreichten bei K=3 HitRate 1,0, Recall 1,0 und MRR 1,0; alle erwarteten Kontrollen standen auf Rang 1. Das belegt End-to-End-Verkabelung und grundlegende Sprachueberquerung, wegen nur drei klar getrennten Kontrollthemen aber keine produktive Retrievalqualitaet. Qdrant enthaelt sechs synthetische Punkte; PostgreSQL und Realdaten wurden nicht beruehrt. 1.285 Standardtests bestanden, acht optionale PostgreSQL- und ein optionaler Qdrant-Test uebersprungen; Local-only-/Designpruefung und beide Produktionsbuilds erfolgreich (bestehender Bundlehinweis). Ein repraesentativer autorisierter Korpus und unabhaengig gelabelter EN/DE-Satz bleiben fuer die reale Baseline erforderlich.
@ -2583,6 +2585,7 @@ Umgesetzter Grundstand vom 2026-09-15:
- Der Grundstand ist in `docs/REGULATORY_RAG.md` dokumentiert und mit synthetischen Vertrags-, Client-, Ingestion-, Retrieval- und Deploymenttests abgesichert. - Der Grundstand ist in `docs/REGULATORY_RAG.md` dokumentiert und mit synthetischen Vertrags-, Client-, Ingestion-, Retrieval- und Deploymenttests abgesichert.
- Der PDF.js-Importer erzeugt aus den explizit uebergebenen ISO-27001-/27002-Pfaden einen lokalen Korpus-v2 mit Quellhashes, Ausgaben, 93 geteilten Kontrollidentitaeten, Abschnittsarten und Seitenbereichen. Nur ISO-27002-Abschnitte werden semantisch gerankt; die entsprechende ISO-27001-Aussage wird als autoritative Referenzevidenz angehaengt. - Der PDF.js-Importer erzeugt aus den explizit uebergebenen ISO-27001-/27002-Pfaden einen lokalen Korpus-v2 mit Quellhashes, Ausgaben, 93 geteilten Kontrollidentitaeten, Abschnittsarten und Seitenbereichen. Nur ISO-27002-Abschnitte werden semantisch gerankt; die entsprechende ISO-27001-Aussage wird als autoritative Referenzevidenz angehaengt.
- Ein manueller Work-Vertikalschnitt startet pro kanonischem Scanner-Finding einen idempotenten asynchronen Lauf, gruppiert die 32 Dense-Chunks auf acht Controls, validiert maximal fuenf LiteLLM-Vorschlaege gegen exakt gelieferte Evidenz und speichert Runs sowie optimistisch nebenlaeufigkeitsgeschuetzte Accept/Reject-Ereignisse append-only in PostgreSQL. - Ein manueller Work-Vertikalschnitt startet pro kanonischem Scanner-Finding einen idempotenten asynchronen Lauf, gruppiert die 32 Dense-Chunks auf acht Controls, validiert maximal fuenf LiteLLM-Vorschlaege gegen exakt gelieferte Evidenz und speichert Runs sowie optimistisch nebenlaeufigkeitsgeschuetzte Accept/Reject-Ereignisse append-only in PostgreSQL.
- Result DB startet auf ausdrueckliche Aktion einen projektgebundenen Batch fuer exakt denselben vollstaendigen normalisierten Filter-/Timeslot-Scope wie Export JSON. Der Dienst paginiert serverseitig unabhaengig von der Anzeige, dedupliziert Scanner-Vorkommen global nach `ruleKey`, verwendet erfolgreiche und aktive Mappings wieder, versucht fehlgeschlagene Regeln einmal erneut und begrenzt Einzel- plus Batchpipelines gemeinsam auf zwei. Migration 027 speichert nur Scope, Filterhash, Idempotenz, Status und Aggregatfortschritt; Entscheidungen bleiben manuell im bestehenden Work-Panel.
Offen bis zur Phasenabnahme: Offen bis zur Phasenabnahme:
@ -2590,7 +2593,7 @@ Offen bis zur Phasenabnahme:
- kuratierte Erweiterung der oeffentlichen Signal-/Regelkataloge - kuratierte Erweiterung der oeffentlichen Signal-/Regelkataloge
- repraesentativer freigegebener EN/DE-Evaluationssatz und dokumentierte reale Dense-Baseline gegen den autorisierten Korpus - repraesentativer freigegebener EN/DE-Evaluationssatz und dokumentierte reale Dense-Baseline gegen den autorisierten Korpus
- aktive Sparse-/BM25-Erzeugung, Hybridfusion und evaluierter Reranker - aktive Sparse-/BM25-Erzeugung, Hybridfusion und evaluierter Reranker
- Confidence-Kalibrierung, Assessment-Batchlauf, Mandanten-/Rollenabgrenzung, Betriebsmonitoring und Lasttest - Confidence-Kalibrierung, Mandanten-/Rollenabgrenzung, Betriebsmonitoring und Lasttest
- Betriebsprozess, der vor jeder kuenftigen Migration einen laufenden Hot-Reload-Backendprozess stoppt oder die Migration bis nach dem verifizierten Backup sperrt - Betriebsprozess, der vor jeder kuenftigen Migration einen laufenden Hot-Reload-Backendprozess stoppt oder die Migration bis nach dem verifizierten Backup sperrt
Abnahmekriterien: Abnahmekriterien:
@ -2618,7 +2621,7 @@ Die naechste Umsetzung fuehrt Phase 13B auf dem konsolidierten Import-, Policy-A
1. Phase 13A ist abgeschlossen: Erkennung, technische Dokumenttypen, Source Extractors, Mappingprofile, Finding-Identität sowie allgemeine Artefakt-, Processing- und Jobgrenzen sind konsolidiert und die Scannerpersistenz verwendet den Candidate-Pfad. 1. Phase 13A ist abgeschlossen: Erkennung, technische Dokumenttypen, Source Extractors, Mappingprofile, Finding-Identität sowie allgemeine Artefakt-, Processing- und Jobgrenzen sind konsolidiert und die Scannerpersistenz verwendet den Candidate-Pfad.
2. Phase 13B ist in Arbeit: Der sichere Ingestion-Unterbau, technische Policy-Identitaeten, versionierte ADMX-/Baselineanalysen, deterministisches Reprocessing, Environment Profiles, konservative Snapshotaggregation, typisierte Evaluation, fail-closed Finding Factory, Unified-Result-Read-Projektion, projektgebundene Review-/Orchestrierungsoberflaechen und die aktuelle snapshotbezogene Ergebnisdeduplizierung sind umgesetzt. Als naechste vertikale Etappe folgen fachlich kuratierte Policy-Regeln und vollstaendige versionierte Windows-Inbox-ADMX-/ADML-Kataloge; danach weitere Preference-CSEs und Exportdetails. Ohne RSoP/GP Results bleibt die effektive Anwendung weiterhin ausdruecklich nicht verifiziert. 2. Phase 13B ist in Arbeit: Der sichere Ingestion-Unterbau, technische Policy-Identitaeten, versionierte ADMX-/Baselineanalysen, deterministisches Reprocessing, Environment Profiles, konservative Snapshotaggregation, typisierte Evaluation, fail-closed Finding Factory, Unified-Result-Read-Projektion, projektgebundene Review-/Orchestrierungsoberflaechen und die aktuelle snapshotbezogene Ergebnisdeduplizierung sind umgesetzt. Als naechste vertikale Etappe folgen fachlich kuratierte Policy-Regeln und vollstaendige versionierte Windows-Inbox-ADMX-/ADML-Kataloge; danach weitere Preference-CSEs und Exportdetails. Ohne RSoP/GP Results bleibt die effektive Anwendung weiterhin ausdruecklich nicht verifiziert.
3. Phase 13F besitzt den lizenzierten ISO-Korpus-v2, die rebuildbare Qdrant-/Dense-Grundlage, den fail-closed Public-Signal-Pfad sowie den persistierten manuell reviewten Einzel-Finding-Workflow. Als naechste Etappe folgen ein unabhaengig gelabelter repraesentativer EN/DE-Evaluationssatz und die reale Dense-Baseline; Sparse/Hybrid, Reranking und Assessment-Batchmapping werden erst danach aktiviert. 3. Phase 13F besitzt den lizenzierten ISO-Korpus-v2, die rebuildbare Qdrant-/Dense-Grundlage, den fail-closed Public-Signal-Pfad, den persistierten manuell reviewten Einzel-Finding-Workflow sowie das explizit vorgezogene gefilterte Result-DB-Batchmapping. Als naechste Etappe folgen ein unabhaengig gelabelter repraesentativer EN/DE-Evaluationssatz und die reale Dense-Baseline; Sparse/Hybrid, Reranking, Confidence-Kalibrierung und jede automatische Entscheidung bleiben bis zu dieser Bewertung ausgeschlossen.
4. Erst nach verifizierter Phase 13B den kontrollierten Firmenbetrieb aus Phase 14 mit SSO, Rollen, HTTPS, Monitoring, Backup/Restore und Aufbewahrung fortsetzen. 4. Erst nach verifizierter Phase 13B den kontrollierten Firmenbetrieb aus Phase 14 mit SSO, Rollen, HTTPS, Monitoring, Backup/Restore und Aufbewahrung fortsetzen.
Die technische Reihenfolge ist bis einschliesslich Phase 13A umgesetzt. Phase 13B baut auf dem verifizierten sicheren Ingestion-Unterbau auf und reicht inzwischen produktiv von der begrenzten Parser-/Workerverarbeitung ueber unveraenderliche Analyse-, Profil-, Aggregations- und Evaluationslaeufe bis zur streng evidenzgebundenen Finding Factory und ihrer gemeinsamen Unified-Result-Leseprojektion. Das Backend leitet projektgebundene Aktionsbereitschaft aus aktiven Revisionen und exakt passenden Referenzdaten ab; der Datahub fuehrt Environment-Profile-Anlage, Reprocessing, Aggregation, Evaluation und Finding Factory ueber die autoritativen idempotenten Services aus. Der produktive Regelkatalog bleibt bis fachlicher Kuratierung leer, und die vorhandenen Microsoft-Baselines bleiben ohne vollstaendige Windows-Inbox-ADMX-/ADML-Kataloge partial; dadurch erzeugt dieser reale Bestand weiterhin nur Review und keine Findings. Die technische Reihenfolge ist bis einschliesslich Phase 13A umgesetzt. Phase 13B baut auf dem verifizierten sicheren Ingestion-Unterbau auf und reicht inzwischen produktiv von der begrenzten Parser-/Workerverarbeitung ueber unveraenderliche Analyse-, Profil-, Aggregations- und Evaluationslaeufe bis zur streng evidenzgebundenen Finding Factory und ihrer gemeinsamen Unified-Result-Leseprojektion. Das Backend leitet projektgebundene Aktionsbereitschaft aus aktiven Revisionen und exakt passenden Referenzdaten ab; der Datahub fuehrt Environment-Profile-Anlage, Reprocessing, Aggregation, Evaluation und Finding Factory ueber die autoritativen idempotenten Services aus. Der produktive Regelkatalog bleibt bis fachlicher Kuratierung leer, und die vorhandenen Microsoft-Baselines bleiben ohne vollstaendige Windows-Inbox-ADMX-/ADML-Kataloge partial; dadurch erzeugt dieser reale Bestand weiterhin nur Review und keine Findings.
@ -2693,6 +2696,8 @@ Praezisierung 2026-09-15 (Phase 13F, Mapping-Relevanz): Der Mapping-Systemprompt
Festlegung 2026-09-15 (Phase 13F, globale ISO-Mappings, ausdruecklicher Nutzerauftrag): ISO-Mappings und Accept/Reject-Entscheidungen sind jetzt global und mandantenunabhaengig, analog zur CVE-Anreicherung. Migration `026` ersetzt die per-Occurrence-Tabellen `assessor_regulatory_mapping_runs`/`_decisions` (nur regenerierbare Testdaten, vor dem Drop gesichert) durch `assessor_regulatory_mappings` und ein globales `assessor_regulatory_mapping_decisions`. Schluessel ist die Regelidentitaet `ruleKey` (`tool:ruleId`, ersatzweise ein stabiler Titel-Hash) plus die Pipelineversionen (`UNIQUE(rule_key, corpus_version, retrieval_version, prompt_version, model)`); die Mapping-ID ist ein deterministisches Digest dieses Tupels, sodass ein Regel-Mapping genau einmal existiert und jede Occurrence in jedem Projekt es wiederverwendet. Entscheidungen sind append-only, per `(ruleKey, controlId)` gefuehrt, versionsunabhaengig und ueberleben Scan-, Assessment- und Projektloeschung. Der Dienst leitet nur die non-personale Regelidentitaet und den Pruefkontext aus dem Finding ab; Ergebnistabelle/Occurrence fliesst nie ein. Kein Reimport (kein Scanner-Import veraendert). Naechste Etappe: assessment-weites Batchmapping ueber distinct Regeln. Festlegung 2026-09-15 (Phase 13F, globale ISO-Mappings, ausdruecklicher Nutzerauftrag): ISO-Mappings und Accept/Reject-Entscheidungen sind jetzt global und mandantenunabhaengig, analog zur CVE-Anreicherung. Migration `026` ersetzt die per-Occurrence-Tabellen `assessor_regulatory_mapping_runs`/`_decisions` (nur regenerierbare Testdaten, vor dem Drop gesichert) durch `assessor_regulatory_mappings` und ein globales `assessor_regulatory_mapping_decisions`. Schluessel ist die Regelidentitaet `ruleKey` (`tool:ruleId`, ersatzweise ein stabiler Titel-Hash) plus die Pipelineversionen (`UNIQUE(rule_key, corpus_version, retrieval_version, prompt_version, model)`); die Mapping-ID ist ein deterministisches Digest dieses Tupels, sodass ein Regel-Mapping genau einmal existiert und jede Occurrence in jedem Projekt es wiederverwendet. Entscheidungen sind append-only, per `(ruleKey, controlId)` gefuehrt, versionsunabhaengig und ueberleben Scan-, Assessment- und Projektloeschung. Der Dienst leitet nur die non-personale Regelidentitaet und den Pruefkontext aus dem Finding ab; Ergebnistabelle/Occurrence fliesst nie ein. Kein Reimport (kein Scanner-Import veraendert). Naechste Etappe: assessment-weites Batchmapping ueber distinct Regeln.
Festlegung 2026-09-16 (Phase 13F, gefiltertes Result-DB-Batchmapping, ausdruecklicher Nutzerauftrag; begrenzte Aufhebung der frueheren Batch-Zurueckstellung): `All` bezeichnet alle Treffer der aktuell aktiven Result-DB-Filter innerhalb des gewaehlten Timeslots und ist exakt derselbe vollstaendige `UnifiedResultQuery`-Scope wie Export JSON. Offset, Limit, Load-more-Zustand und geladene Zeilen sind reine Darstellung. Der Standardfilter `deviation: true` bleibt wirksam, bis er geaendert oder zurueckgesetzt wird. Der beim Start normalisierte Snapshot bleibt fuer den Lauf unveraendert und wird projektgebunden mit kanonischem Hash und Idempotenz in Migration 027 gespeichert. Nur kanonische Scanner-Findings sind geeignet; Nicht-Scanner und fehlender oeffentlicher Pruefkontext werden gezaehlt, aber nie versendet. Vorkommen werden global nach `ruleKey` dedupliziert, erfolgreiche oder aktive Mappings wiederverwendet und fehlende beziehungsweise zuvor fehlgeschlagene deterministisch geclaimt. Einzel- und Batchaktionen teilen zwei parallele Retrieval-/Modellpipelines. Der Batch erzeugt ausschliesslich Vorschlaege; Accept/Reject bleibt manuell und es gibt weder Auto-Accept noch Batch-Entscheidungsqueue. Die vorgezogene Orchestrierung ist keine Aussage zur Retrievalqualitaet: repraesentative EN/DE-Evaluation, Sparse/Hybrid, Reranking und Confidence-Kalibrierung bleiben offen. Kein Importvertrag geaendert, daher kein Manifestbump oder Reimport.
Festlegung 2026-09-10 (Purple-Knight-Begleitdateien): Ein nativer Score darf aus einer separaten Summary auf die Findings desselben Scanlaufs uebertragen werden, wenn genau ein nativer Traeger und eine eindeutige Ergebnispopulation denselben Werkzeug- und Verarbeitungskontrakt teilen. Datei- und Vorkommensherkunft bleiben gemeinsam erhalten; Mehrdeutigkeit verhindert die Verbindung. Festlegung 2026-09-10 (Purple-Knight-Begleitdateien): Ein nativer Score darf aus einer separaten Summary auf die Findings desselben Scanlaufs uebertragen werden, wenn genau ein nativer Traeger und eine eindeutige Ergebnispopulation denselben Werkzeug- und Verarbeitungskontrakt teilen. Datei- und Vorkommensherkunft bleiben gemeinsam erhalten; Mehrdeutigkeit verhindert die Verbindung.
Festlegung 2026-09-10 (weitere Scanner): Native Gesamtwerte werden bei jeder Profilerweiterung zuerst geprueft und deklarativ mit ihrer Bedeutung erfasst. Die Scoredefinition ist Pflicht, die Berechnung gemeinsam; ihre Version gehoert zur Vergleichskompatibilitaet. Extraktion und Originalherkunft bleiben im generischen Importpfad. AGENTS.md und docs/SCORING_MODEL.md halten die Erweiterungsregel fest. Festlegung 2026-09-10 (weitere Scanner): Native Gesamtwerte werden bei jeder Profilerweiterung zuerst geprueft und deklarativ mit ihrer Bedeutung erfasst. Die Scoredefinition ist Pflicht, die Berechnung gemeinsam; ihre Version gehoert zur Vergleichskompatibilitaet. Extraktion und Originalherkunft bleiben im generischen Importpfad. AGENTS.md und docs/SCORING_MODEL.md halten die Erweiterungsregel fest.
@ -3677,6 +3682,7 @@ Verifiziert 2026-09-08 (Phase 7/13B): GPO-Quellenkorrektur fuer Work und Client
| 2026-09-15 | Phase 7/13B: GPO Analysis verwendet fuer seine Baseline-Ergebniszaehler jetzt dieselben backendautoritativen aktuellen Aggregationslaeufe wie Work und Client Info. Historische Aggregationen bleiben in der Evaluationshistorie, werden aber nicht mehr zu aktuellen Pass-/Fail-/Not-configured-Zaehlern addiert. Review required ist in der Zusammenfassung sichtbar. Exakte synthetische Mehrlauf-/Ein-Timeslot-Komponentenregression, 26 fokussierte Tests, 1.267 bestandene Gesamttests plus isoliert bestandener bekannter Last-Timeout und beide Produktionsbuilds erfolgreich; acht optionale PostgreSQL-Tests uebersprungen. Keine Datenmutation, Neuauswertung, Import-, Persistenz-, Schema-, Manifest- oder Reimportaenderung. | | 2026-09-15 | Phase 7/13B: GPO Analysis verwendet fuer seine Baseline-Ergebniszaehler jetzt dieselben backendautoritativen aktuellen Aggregationslaeufe wie Work und Client Info. Historische Aggregationen bleiben in der Evaluationshistorie, werden aber nicht mehr zu aktuellen Pass-/Fail-/Not-configured-Zaehlern addiert. Review required ist in der Zusammenfassung sichtbar. Exakte synthetische Mehrlauf-/Ein-Timeslot-Komponentenregression, 26 fokussierte Tests, 1.267 bestandene Gesamttests plus isoliert bestandener bekannter Last-Timeout und beide Produktionsbuilds erfolgreich; acht optionale PostgreSQL-Tests uebersprungen. Keine Datenmutation, Neuauswertung, Import-, Persistenz-, Schema-, Manifest- oder Reimportaenderung. |
| 2026-09-15 | Phase 13/9-11: Timeslot-Importlast korrigiert. Inaktive GPO-Importsteuerelemente in Timeslotzeilen erzeugen keine eigenen Poller mehr; sichtbare Statusansichten und der Data Hub fragen nur waehrend queued/running ab, bewahren unveraenderte Zustandsobjekte und berechnen Result-Facetten nur bei einer echten Statusrevision neu. Datei- und Definitionszuordnung im PostgreSQL-Commit ist je Import vorindiziert. Gemessener Ausgangszustand: 4,4 s Vorschau, 200,3 s Commit unter 270 Facetten- und 326 Statusabfragen in fuenf Minuten. Synthetische Interaktionsregression, 30 fokussierte Tests, 1.266 bestandene Gesamttests plus isoliert bestandener bekannter Last-Timeout, Local-only-/Designpruefung und beide Builds erfolgreich; acht optionale PostgreSQL-Integrationstests uebersprungen. Keine Schema-, Daten-, Importsemantik-, Manifest- oder Reimportaenderung. Restrisiko serieller PostgreSQL-Einzelwrites dokumentiert. | | 2026-09-15 | Phase 13/9-11: Timeslot-Importlast korrigiert. Inaktive GPO-Importsteuerelemente in Timeslotzeilen erzeugen keine eigenen Poller mehr; sichtbare Statusansichten und der Data Hub fragen nur waehrend queued/running ab, bewahren unveraenderte Zustandsobjekte und berechnen Result-Facetten nur bei einer echten Statusrevision neu. Datei- und Definitionszuordnung im PostgreSQL-Commit ist je Import vorindiziert. Gemessener Ausgangszustand: 4,4 s Vorschau, 200,3 s Commit unter 270 Facetten- und 326 Statusabfragen in fuenf Minuten. Synthetische Interaktionsregression, 30 fokussierte Tests, 1.266 bestandene Gesamttests plus isoliert bestandener bekannter Last-Timeout, Local-only-/Designpruefung und beide Builds erfolgreich; acht optionale PostgreSQL-Integrationstests uebersprungen. Keine Schema-, Daten-, Importsemantik-, Manifest- oder Reimportaenderung. Restrisiko serieller PostgreSQL-Einzelwrites dokumentiert. |
| 2026-09-14 | Phase 7/13B: Regression im Work-Filterkontext korrigiert. `Source: Scan = GPOs` reduziert auf dem All-Records-Blatt nun vor Priorisierung und Facettenlimit auf den GPO-Vertrag; das Statusmenue wird unabhaengig von einem separaten GPO-Blatt mit All, Deviation, Passed, Not configured, Conflicts und Duplicates vollstaendig erzeugt, fachliche Nullwerte bleiben waehlbar. GPO-Statuswerte werden ausschliesslich gegen das aktuelle Statusfeld geprueft; ein Pass-Vergleichswert kann einen Not-configured-Datensatz nicht mehr als Passed filtern. Fremde Scannerfacetten und ungueltige gesetzte Altfilter werden entfernt, `Source: Scan` bleibt vollstaendig und rechts; ohne Quellrestriktion kehrt der Mischkontext zurueck. Exakte synthetische Interaktionsregressionen pruefen Anzeige, tatsaechliche Filterwirkung und den gemeldeten Gegenfall; 51 fokussierte Tests sowie der Gesamtlauf mit 1.243 bestandenen und acht uebersprungenen Tests, Local-only-/Designpruefung und Client-/Serverbuild erfolgreich. `AGENTS.md` bindet dokumentierte Nutzerentscheidungen als Abnahmekriterien und verlangt transparente Meldung vor regelwidrigen Aenderungen. Keine Daten-, Import-, API-, Persistenz-, Score- oder Reimportaenderung; ausschliesslich synthetische Inhaltspruefung. | | 2026-09-14 | Phase 7/13B: Regression im Work-Filterkontext korrigiert. `Source: Scan = GPOs` reduziert auf dem All-Records-Blatt nun vor Priorisierung und Facettenlimit auf den GPO-Vertrag; das Statusmenue wird unabhaengig von einem separaten GPO-Blatt mit All, Deviation, Passed, Not configured, Conflicts und Duplicates vollstaendig erzeugt, fachliche Nullwerte bleiben waehlbar. GPO-Statuswerte werden ausschliesslich gegen das aktuelle Statusfeld geprueft; ein Pass-Vergleichswert kann einen Not-configured-Datensatz nicht mehr als Passed filtern. Fremde Scannerfacetten und ungueltige gesetzte Altfilter werden entfernt, `Source: Scan` bleibt vollstaendig und rechts; ohne Quellrestriktion kehrt der Mischkontext zurueck. Exakte synthetische Interaktionsregressionen pruefen Anzeige, tatsaechliche Filterwirkung und den gemeldeten Gegenfall; 51 fokussierte Tests sowie der Gesamtlauf mit 1.243 bestandenen und acht uebersprungenen Tests, Local-only-/Designpruefung und Client-/Serverbuild erfolgreich. `AGENTS.md` bindet dokumentierte Nutzerentscheidungen als Abnahmekriterien und verlangt transparente Meldung vor regelwidrigen Aenderungen. Keine Daten-, Import-, API-, Persistenz-, Score- oder Reimportaenderung; ausschliesslich synthetische Inhaltspruefung. |
| 2026-09-16 | Phase 13F, ausdruecklicher Nutzerauftrag: Gefiltertes Result-DB-ISO-Batchmapping umgesetzt. Der gemeinsame Scope-Builder garantiert dieselbe vollstaendige Filter-/Timeslot-Abfrage wie Export JSON und entfernt Pagination sowie nicht autoritative Client-Scopewerte. Projektgebundene GET-/POST-APIs, Migration 027, Local-/PostgreSQL-Paritaet, Export und Loeschung speichern den normalisierten Scope, kanonischen Filterhash, Idempotenz, Lifecycle und Fortschritt; globale Regelmappings bleiben autoritativ. Stabile serverseitige Mehrseitenabfrage mit Total-Drift-Abbruch, deterministische `ruleKey`-Repraesentanten, erfolgreiche/aktive Wiederverwendung, einmaliger Failed-Retry, atomare Mappingclaims und eine gemeinsame Zwei-Pipeline-Queue sind implementiert. Result DB zeigt `Map all ISO controls`, Trefferzahl, Status und Zaehler, verhindert veraltete Filterantworten und behaelt Accept/Reject manuell in Work. Synthetische Regressionen decken alle Filterfamilien, Standardabweichungsfilter, Scopebereinigung, Mehrseitenlauf, Total-Drift, Ausschluesse, globale Deduplikation, Partial Failure, Retry/Reuse, Single-/Batch-Nebenlaeufigkeit und UI ab. `npm test`: 1.315 bestanden, zehn opt-in Integrationen uebersprungen; Local-only-/Designpruefung und beide Produktionsbuilds erfolgreich. Kein Manifestbump oder Reimport. Nach der Anwendung von 027 wurde dieselbe Datei faelschlich um eine reine Sortiersequenz erweitert; der Checksum-Schutz stoppte den Backendstart. Die Datei ist exakt auf den angewandten SHA-256-Stand `d89055c3e723b09d8d8be7f4225bfd08442f0d66e4e0b4ea3aa62ae6209e9ee9` zurueckgesetzt und mit einer Regression gesperrt, die Abfragen verwenden wieder den vorhandenen `created_at, id`-Index. Keine Migration 028 und keine weitere Datenbankmutation. Fuenf fokussierte Suites mit 16 Tests, Servertyppruefung und Serverbuild erfolgreich. Nach freigegebenem WSL-Neustart: kanonischer PostgreSQL-Container gesund, Port/Volume korrekt, Ledger 027 checksum-identisch, 028 nicht vorhanden und Backend-Health HTTP 200 mit PostgreSQL-Persistenz. |
| 2026-09-14 | Phasen 7/13C und 11: Work-LiteLLM-gestuetztes AI Overview umgesetzt. Der Backend-Service liest den kompletten Timeslot ueber das bestehende Privacy-Safe-Result-DB-JSON, hasht ihn deterministisch, zerlegt grosse Payloads nur zwischen vollstaendigen Eintraegen und konsolidiert sie mit maximal 64 explizit nicht wiederholten LiteLLM-Aufrufen. Der HTTPS-Client haelt URL, Modell und Bearer-Key ausschliesslich in Server-Secrets, lehnt lokale/unsichere Ziele und Redirects ab und begrenzt Zeit sowie Antwortbytes. Ausgabevalidierung prueft das vollstaendige Dashboard-Schema, Laengen, Roadmap 0-4, exakte Finding-/Gap-IDs und Environment-Properties. Same-Origin-GET/POST, Migration 024, lokale/PostgreSQL-Repositoryparitaet, Projekt-Export/-Loeschung, Idempotenz und ein aktiver Run je Scope sind ergaenzt. Client Info rendert alle Analyseabschnitte mit Generate/Regenerate, Polling nur bei aktiven Runs sowie stale/failed/disabled Zustaenden. Sicherheits- und Deploymentdokumentation beschreibt den genehmigten Corporate-Gateway-Vertrag. 1.262 synthetische Tests bestanden, acht optionale PostgreSQL-Tests uebersprungen; Local-only-/Designpruefung und beide Produktionsbuilds erfolgreich. Der inhaltsfreie Workstation-Livecheck erreichte Work-LiteLLM, erhielt aber HTTP 401; Providerbody und Secrets wurden nicht ausgegeben. Docker/kanonisches Backup, Migrationanwendung, PostgreSQL-Integration sowie erfolgreicher Workstation-/Server-Livecheck bleiben offen; keine Realdateninspektion oder Reimportaenderung. | | 2026-09-14 | Phasen 7/13C und 11: Work-LiteLLM-gestuetztes AI Overview umgesetzt. Der Backend-Service liest den kompletten Timeslot ueber das bestehende Privacy-Safe-Result-DB-JSON, hasht ihn deterministisch, zerlegt grosse Payloads nur zwischen vollstaendigen Eintraegen und konsolidiert sie mit maximal 64 explizit nicht wiederholten LiteLLM-Aufrufen. Der HTTPS-Client haelt URL, Modell und Bearer-Key ausschliesslich in Server-Secrets, lehnt lokale/unsichere Ziele und Redirects ab und begrenzt Zeit sowie Antwortbytes. Ausgabevalidierung prueft das vollstaendige Dashboard-Schema, Laengen, Roadmap 0-4, exakte Finding-/Gap-IDs und Environment-Properties. Same-Origin-GET/POST, Migration 024, lokale/PostgreSQL-Repositoryparitaet, Projekt-Export/-Loeschung, Idempotenz und ein aktiver Run je Scope sind ergaenzt. Client Info rendert alle Analyseabschnitte mit Generate/Regenerate, Polling nur bei aktiven Runs sowie stale/failed/disabled Zustaenden. Sicherheits- und Deploymentdokumentation beschreibt den genehmigten Corporate-Gateway-Vertrag. 1.262 synthetische Tests bestanden, acht optionale PostgreSQL-Tests uebersprungen; Local-only-/Designpruefung und beide Produktionsbuilds erfolgreich. Der inhaltsfreie Workstation-Livecheck erreichte Work-LiteLLM, erhielt aber HTTP 401; Providerbody und Secrets wurden nicht ausgegeben. Docker/kanonisches Backup, Migrationanwendung, PostgreSQL-Integration sowie erfolgreicher Workstation-/Server-Livecheck bleiben offen; keine Realdateninspektion oder Reimportaenderung. |
| 2026-09-14 | Rebase der AI-Integration auf defender-iot: Die einzelne AI-Commit-Aenderung wurde neu aufgesetzt und die kollidierende Result-DB-LLM-Uebergabe zusammengefuehrt. Das versionierte, datensparsame Privacy-Safe-JSON-Schema dieses Branches ist nun das Result-DB-Exportformat und wird als einzelne Aktion `Export JSON` ueber den gemeinsamen Browser-Download-Adapter als `.json`-Datei geladen; damit ersetzt es sowohl den zuvor auf defender-iot vorhandenen `Export JSONL`-Download als auch die fluechtige `Copy JSON`-Zwischenablageaktion. Der Builder wurde zu `resultsToPrivacySafeExportJson` mit begleitendem Dateinamens-Helfer `privacySafeResultJsonFileName` umbenannt; Entscheidungs-, Risiko- und Testeintraege wurden entsprechend angeglichen. Alle uebrigen defender-iot-Inhalte (CVE-Enrichment, GPO-/Policy-Ergebnisse, Timeslot-Scope) bleiben unveraendert. 1248 Tests bestanden, 8 optionale PostgreSQL-Tests uebersprungen; Local-only-, Designpruefung und Client-/Server-Produktionsbuild erfolgreich. | | 2026-09-14 | Rebase der AI-Integration auf defender-iot: Die einzelne AI-Commit-Aenderung wurde neu aufgesetzt und die kollidierende Result-DB-LLM-Uebergabe zusammengefuehrt. Das versionierte, datensparsame Privacy-Safe-JSON-Schema dieses Branches ist nun das Result-DB-Exportformat und wird als einzelne Aktion `Export JSON` ueber den gemeinsamen Browser-Download-Adapter als `.json`-Datei geladen; damit ersetzt es sowohl den zuvor auf defender-iot vorhandenen `Export JSONL`-Download als auch die fluechtige `Copy JSON`-Zwischenablageaktion. Der Builder wurde zu `resultsToPrivacySafeExportJson` mit begleitendem Dateinamens-Helfer `privacySafeResultJsonFileName` umbenannt; Entscheidungs-, Risiko- und Testeintraege wurden entsprechend angeglichen. Alle uebrigen defender-iot-Inhalte (CVE-Enrichment, GPO-/Policy-Ergebnisse, Timeslot-Scope) bleiben unveraendert. 1248 Tests bestanden, 8 optionale PostgreSQL-Tests uebersprungen; Local-only-, Designpruefung und Client-/Server-Produktionsbuild erfolgreich. |
| 2026-09-08 | Phase 7: Verbindungskurven der Client Info treffen ihre Domainknoten wieder. Feste Zielkoordinaten in einem verzerrten SVG-Koordinatensystem durch eine aus dem Layout gemessene Geometrie ersetzt, Neumessung ueber Groessenbeobachter und Fensterereignis. Am Dummy bei drei Breiten geprueft: Abweichung 0 Pixel statt bis zu 80. Keine Aenderung an Werten oder Kachelgeometrie. 1.011 Tests, Local-only-/Designpruefung und Builds erfolgreich. | | 2026-09-08 | Phase 7: Verbindungskurven der Client Info treffen ihre Domainknoten wieder. Feste Zielkoordinaten in einem verzerrten SVG-Koordinatensystem durch eine aus dem Layout gemessene Geometrie ersetzt, Neumessung ueber Groessenbeobachter und Fensterereignis. Am Dummy bei drei Breiten geprueft: Abweichung 0 Pixel statt bis zu 80. Keine Aenderung an Werten oder Kachelgeometrie. 1.011 Tests, Local-only-/Designpruefung und Builds erfolgreich. |

2
docs/PROJECT_OVERVIEW.md
View File

@ -307,7 +307,7 @@ opt-in, and off by default. Trust boundaries:
fields are captured losslessly). Full-text `tsvector` + GIN indexes back search. fields are captured losslessly). Full-text `tsvector` + GIN indexes back search.
- **Migrations** are versioned SQL files in - **Migrations** are versioned SQL files in
[`server/postgres/migrations/`](../server/postgres/migrations/) (currently through [`server/postgres/migrations/`](../server/postgres/migrations/) (currently through
`026_*`). On start the server takes an advisory lock, verifies each migration's `027_*`). On start the server takes an advisory lock, verifies each migration's
version + SHA-256, applies new ones in their own transaction, and refuses to version + SHA-256, applies new ones in their own transaction, and refuses to
start if an already-applied migration was edited. Never edit an applied start if an already-applied migration was edited. Never edit an applied
migration — add a new numbered file. migration — add a new numbered file.

16
docs/REGULATORY_RAG.md
View File

@ -1,6 +1,6 @@
# Regulatory-control RAG with Qdrant # Regulatory-control RAG with Qdrant
Status: Phase 13F one-finding vertical slice. The server prepares the licensed German ISO 27001/27002 corpus, indexes section-safe ISO 27002 semantic chunks plus ISO 27001 references, retrieves and groups candidates, requests a grounded LiteLLM proposal, persists runs and reviewer decisions, and exposes the workflow inline in Work. Batch mapping, sparse retrieval, reranking, and confidence calibration remain out of scope. Status: Phase 13F one-finding and filtered Result DB batch vertical slice. The server prepares the licensed German ISO 27001/27002 corpus, indexes section-safe ISO 27002 semantic chunks plus ISO 27001 references, retrieves and groups candidates, requests grounded LiteLLM proposals, persists runs and reviewer decisions, and exposes manual proposal generation in Work and Result DB. Sparse retrieval, reranking, and confidence calibration remain out of scope.
## Boundaries ## Boundaries
@ -127,9 +127,21 @@ Routes:
- `GET /api/v1/projects/{projectId}/regulatory-mappings/{resultProjectionId}` - `GET /api/v1/projects/{projectId}/regulatory-mappings/{resultProjectionId}`
- `POST /api/v1/projects/{projectId}/regulatory-mapping-runs` - `POST /api/v1/projects/{projectId}/regulatory-mapping-runs`
- `POST /api/v1/projects/{projectId}/regulatory-mapping-decisions` - `POST /api/v1/projects/{projectId}/regulatory-mapping-decisions`
- `GET /api/v1/projects/{projectId}/regulatory-mapping-batches`
- `POST /api/v1/projects/{projectId}/regulatory-mapping-batches`
On the consolidated migration line (2026-10-02), the single migration 026 `regulatory-mapping` creates the global tables directly, together with the project-scoped `assessor_regulatory_mapping_batches`; the superseded occurrence-bound tables are never created. The remainder of this paragraph records the original history on the former experimental line, where migration 025 originally added occurrence-bound, project-scoped run and decision tables and migration 026 superseded it: it dropped those tables and created the global `assessor_regulatory_mappings` (keyed by `rule_key` + pipeline versions, `UNIQUE(rule_key, corpus_version, retrieval_version, prompt_version, model)`) and a global, rule-keyed `assessor_regulatory_mapping_decisions`; neither carries `project_id` or occurrence identity, and both survive project/finding deletion. Before 026 was applied, the 025 tables held only regenerable local test data (19 runs, 10 decisions); a verified backup of both tables was taken at `backups/regulatory_mapping_025_backup_20260915-155410.sql` and the data was then intentionally discarded per the user's decision. A live development backend applied 026 through its own migration runner (checksum recorded and re-verified against the source file); no reimport is required because mapping runs are regenerable and no scanner data changed. On the consolidated migration line (2026-10-02), the single migration 026 `regulatory-mapping` creates the global tables directly, together with the project-scoped `assessor_regulatory_mapping_batches`; the superseded occurrence-bound tables are never created. The remainder of this paragraph records the original history on the former experimental line, where migration 025 originally added occurrence-bound, project-scoped run and decision tables and migration 026 superseded it: it dropped those tables and created the global `assessor_regulatory_mappings` (keyed by `rule_key` + pipeline versions, `UNIQUE(rule_key, corpus_version, retrieval_version, prompt_version, model)`) and a global, rule-keyed `assessor_regulatory_mapping_decisions`; neither carries `project_id` or occurrence identity, and both survive project/finding deletion. Before 026 was applied, the 025 tables held only regenerable local test data (19 runs, 10 decisions); a verified backup of both tables was taken at `backups/regulatory_mapping_025_backup_20260915-155410.sql` and the data was then intentionally discarded per the user's decision. A live development backend applied 026 through its own migration runner (checksum recorded and re-verified against the source file); no reimport is required because mapping runs are regenerable and no scanner data changed.
## Filtered Result DB batch workflow
`Map all ISO controls` sits beside Refresh and Export JSON in Result DB. Both actions derive their scope from the same normalized `UnifiedResultQuery`: the selected assessment and scan-run IDs are the outer timeslot boundary, and every active Result DB filter narrows that scope. This includes deviation, source, severity, outcome, result type, processing state, review state, categories, and supported text/date filters. The default `deviation: true` filter therefore maps deviations only until the user changes or resets it. Offset, limit, client project IDs, and client audience values are discarded. Display pagination, Load more, and the number of mounted rows never change batch membership.
The server enumerates the complete query through stable pagination and fails the batch if the filtered total changes during enumeration. It counts non-scanner results as excluded, scanner findings without usable public check context as unsupported, and deduplicates eligible occurrences by global `ruleKey`. One deterministic representative is selected per rule. Current successful or active mappings are reused; missing mappings and previously failed mappings are atomically claimed and queued. Single-finding and batch work share the same queue with at most two concurrent retrieval/model pipelines, so simultaneous requests cannot create duplicate model calls for a deterministic mapping identity. A completed batch may still report individual rule failures.
The batch table (part of migration 026 on the consolidated line, originally migration 027) holds only project-scoped batch orchestration records: the normalized timeslot/filter snapshot, canonical filter hash, idempotency key, state, aggregate counters, timestamps, and bounded failure details. Global individual mapping rows remain authoritative; there is no batch-item table. Batch records are exported with their project and removed with project deletion. Active mappings and batches are marked failed after an interrupted server process; a later batch reuses successful work and retries only missing or failed rules. The migration is additive and does not change scanner processing, so no processing-manifest bump or reimport is required. After 027 had been applied, a later ordering-only edit incorrectly changed that migration and the checksum guard stopped backend startup. The file has been restored to its applied SHA-256 state and locked by regression; no follow-up schema migration is required.
The filter snapshot is captured when Start is pressed and is never copied into embedding or LiteLLM payloads. Result DB reloads the latest batch for the currently selected normalized scope, so changing filters does not relabel an active run. Batches generate proposals only. Accept/Reject remains the existing explicit human decision per global rule/control in Work; there is no automatic acceptance or batch decision queue.
### Local Qdrant integration check ### Local Qdrant integration check
`tests/server/qdrant-regulatory-index.integration.test.ts` is opt-in and uses deterministic synthetic vectors. With `ASSESSOR_QDRANT_INTEGRATION=1` and the normal Qdrant/model identity environment variables, it verifies the real collection, payload indexes, document replacement, filters, and retrieval without calling an embedding provider. The standard suite skips it so tests do not unexpectedly mutate a developer's derived index. `tests/server/qdrant-regulatory-index.integration.test.ts` is opt-in and uses deterministic synthetic vectors. With `ASSESSOR_QDRANT_INTEGRATION=1` and the normal Qdrant/model identity environment variables, it verifies the real collection, payload indexes, document replacement, filters, and retrieval without calling an embedding provider. The standard suite skips it so tests do not unexpectedly mutate a developer's derived index.
@ -180,7 +192,7 @@ Live smoke result on 2026-09-15 with local llama.cpp build 10472, BGE-M3 FP16, a
- `RegulatoryReranker` is optional and receives only the bounded candidate set. Without one, dense ordering is retained. - `RegulatoryReranker` is optional and receives only the bounded candidate set. Without one, dense ordering is retained.
- `RegulatoryRetriever` is the consumer-facing contract. Mapping code should depend on it, not on Qdrant. - `RegulatoryRetriever` is the consumer-facing contract. Mapping code should depend on it, not on Qdrant.
Before assessment-wide batch mapping, broaden public-signal coverage only through reviewed catalogs, establish a representative independently labeled EN/DE baseline, and evaluate sparse/hybrid retrieval, reranking, and confidence calibration. The one-finding workflow must remain human-reviewed. The filtered manual Result DB batch was activated by explicit user decision before the earlier quality-gate sequence. That decision changes orchestration only: broaden public-signal coverage through reviewed catalogs, establish a representative independently labeled EN/DE baseline, and evaluate sparse/hybrid retrieval, reranking, and confidence calibration before claiming production retrieval quality or adding automatic decisions. Single and batch proposals remain human-reviewed.
## Reference basis ## Reference basis

24
server/assessor-http-server.ts
View File

@ -31,7 +31,12 @@ import type { GpoPolicyAnalysisOverviewService } from './gpo/gpo-policy-analysis
import type { AiOverviewService } from './ai/ai-overview-service'; import type { AiOverviewService } from './ai/ai-overview-service';
import { AiOverviewServiceError } from './ai/ai-overview-service'; import { AiOverviewServiceError } from './ai/ai-overview-service';
import type { AiOverviewCreateRequest, AiOverviewScope } from '../src/domain/ai-overview'; import type { AiOverviewCreateRequest, AiOverviewScope } from '../src/domain/ai-overview';
import type { RegulatoryMappingCreateRequest, RegulatoryMappingDecisionCreateRequest } from '../src/domain/regulatory-mapping'; import type {
RegulatoryMappingBatchCreateRequest,
RegulatoryMappingBatchScope,
RegulatoryMappingCreateRequest,
RegulatoryMappingDecisionCreateRequest
} from '../src/domain/regulatory-mapping';
import type { RegulatoryMappingService } from './regulatory/regulatory-mapping-service'; import type { RegulatoryMappingService } from './regulatory/regulatory-mapping-service';
import { RegulatoryMappingServiceError } from './regulatory/regulatory-mapping-service'; import { RegulatoryMappingServiceError } from './regulatory/regulatory-mapping-service';
import type { GpoAutomaticAnalysisService } from './gpo/gpo-automatic-analysis-service'; import type { GpoAutomaticAnalysisService } from './gpo/gpo-automatic-analysis-service';
@ -304,6 +309,14 @@ function unifiedQueryFrom(url: URL): UnifiedResultQuery {
}; };
} }
function regulatoryMappingBatchScopeFrom(url: URL): RegulatoryMappingBatchScope {
const query = unifiedQueryFrom(url);
const { audience: _audience, projectId: _projectId, assessmentId: _assessmentId,
assessmentIds = [], scanRunId: _scanRunId, scanRunIds = [], offset: _offset, limit: _limit,
...filters } = query;
return { assessmentIds, scanRunIds, filters };
}
function validateImportSource(value: ImportSourceValidationRequest): ImportSourceValidationResponse { function validateImportSource(value: ImportSourceValidationRequest): ImportSourceValidationResponse {
if (!value || typeof value.fileName !== 'string' || !value.fileName.trim()) { if (!value || typeof value.fileName !== 'string' || !value.fileName.trim()) {
throw new HttpError(400, 'invalid_file_metadata', 'Ein Dateiname ist erforderlich.'); throw new HttpError(400, 'invalid_file_metadata', 'Ein Dateiname ist erforderlich.');
@ -1120,6 +1133,15 @@ export function createAssessorHttpServer(options: AssessorHttpServerOptions): Se
const body = await readJson<RegulatoryMappingDecisionCreateRequest>(request, maximumBodyBytes); const body = await readJson<RegulatoryMappingDecisionCreateRequest>(request, maximumBodyBytes);
status = 200; sendJson(response, status, await options.regulatoryMappingService.decide(segments[1], body), requestId); return; status = 200; sendJson(response, status, await options.regulatoryMappingService.decide(segments[1], body), requestId); return;
} }
if (segments[0] === 'projects' && segments[2] === 'regulatory-mapping-batches' && segments.length === 3 && method === 'GET') {
if (!options.regulatoryMappingService) throw new HttpError(503, 'regulatory_mapping_disabled', 'Regulatory mapping is not configured.');
status = 200; sendJson(response, status, await options.regulatoryMappingService.getBatch(segments[1], regulatoryMappingBatchScopeFrom(url)), requestId); return;
}
if (segments[0] === 'projects' && segments[2] === 'regulatory-mapping-batches' && segments.length === 3 && method === 'POST') {
if (!options.regulatoryMappingService) throw new HttpError(503, 'regulatory_mapping_disabled', 'Regulatory mapping is not configured.');
const body = await readJson<RegulatoryMappingBatchCreateRequest>(request, maximumBodyBytes);
status = 202; sendJson(response, status, await options.regulatoryMappingService.startBatch(segments[1], body), requestId); return;
}
if (segments.length === 1 && segments[0] === 'assessments' && method === 'POST') { if (segments.length === 1 && segments[0] === 'assessments' && method === 'POST') {
const assessment = await readJson<Assessment>(request, maximumBodyBytes); const assessment = await readJson<Assessment>(request, maximumBodyBytes);
await repository.saveAssessment(assessment); await repository.saveAssessment(assessment);

91
server/postgres/postgres-assessor-repository.ts
View File

@ -27,8 +27,8 @@ import type { EnvironmentObservation, EnvironmentSnapshot, EnvironmentSnapshotQu
import { environmentCoverage } from '../../src/domain/environment-context'; import { environmentCoverage } from '../../src/domain/environment-context';
import type { AiOverviewRun } from '../../src/domain/ai-overview'; import type { AiOverviewRun } from '../../src/domain/ai-overview';
import { aiOverviewRunValidationErrors } from '../../src/domain/ai-overview'; import { aiOverviewRunValidationErrors } from '../../src/domain/ai-overview';
import type { RegulatoryMappingDecision, RegulatoryMappingRun } from '../../src/domain/regulatory-mapping'; import type { RegulatoryMappingBatchRun, RegulatoryMappingDecision, RegulatoryMappingRun } from '../../src/domain/regulatory-mapping';
import { regulatoryMappingDecisionValidationErrors, regulatoryMappingRunValidationErrors } from '../../src/domain/regulatory-mapping'; import { regulatoryMappingBatchRunValidationErrors, regulatoryMappingDecisionValidationErrors, regulatoryMappingRunValidationErrors } from '../../src/domain/regulatory-mapping';
import { storedPolicyFindingRunValidationErrors } from '../../src/domain/policy-finding'; import { storedPolicyFindingRunValidationErrors } from '../../src/domain/policy-finding';
import { latestReferenceRuns, type ReferenceRecognitionRun, type ReferenceCatalogSnapshot } from '../../src/domain/reference-recognition'; import { latestReferenceRuns, type ReferenceRecognitionRun, type ReferenceCatalogSnapshot } from '../../src/domain/reference-recognition';
import { validateReferenceRun } from '../../src/persistence/reference-run-validation'; import { validateReferenceRun } from '../../src/persistence/reference-run-validation';
@ -378,6 +378,7 @@ const DB_STRUCTURE_LAYER_BY_TABLE: Record<string, DbStructureLayerId> = {
assessor_finding_definitions: 'result', assessor_finding_definitions: 'result',
assessor_regulatory_mappings: 'result', assessor_regulatory_mappings: 'result',
assessor_regulatory_mapping_decisions: 'governance', assessor_regulatory_mapping_decisions: 'governance',
assessor_regulatory_mapping_batches: 'governance',
assessor_ai_overview_runs: 'governance', assessor_ai_overview_runs: 'governance',
assessor_finding_occurrences: 'result', assessor_finding_occurrences: 'result',
assessor_result_categories: 'result', assessor_result_categories: 'result',
@ -704,6 +705,37 @@ export class PostgresAssessorRepository implements AssessorRepository {
)); ));
} }
async claimRegulatoryMappingRun(run: RegulatoryMappingRun, forceRegeneration: boolean): Promise<{ run: RegulatoryMappingRun; claimed: boolean }> {
const errors = regulatoryMappingRunValidationErrors(run);
if (errors.length) throw new RepositoryValidationError(errors.join(' '));
return this.mutate(async client => {
const inserted = await client.query(`
INSERT INTO assessor_regulatory_mappings
(id, rule_key, source_tool, source_rule_id, run_status, query_hash, corpus_id, corpus_version,
embedding_model, retrieval_version, prompt_version, output_schema_version, model, created_at, updated_at, data)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16::jsonb)
ON CONFLICT DO NOTHING
RETURNING id
`, [run.id, run.ruleKey, run.sourceTool, run.sourceRuleId ?? null, run.status, run.queryHash, run.corpusId,
run.corpusVersion, run.embeddingModel, run.retrievalVersion, run.promptVersion, run.outputSchemaVersion,
run.model, run.createdAt, run.updatedAt, json(run)]);
if (inserted.rowCount) return { run: clone(run), claimed: true };
const existing = dataRow(await client.query<DataRow<RegulatoryMappingRun>>(
'SELECT data FROM assessor_regulatory_mappings WHERE id = $1 FOR UPDATE', [run.id]
));
if (!existing) throw new RepositoryConflictError('Regulatory mapping identity conflicted with another pipeline tuple.');
if (['queued', 'running'].includes(existing.status) || existing.status === 'succeeded' && !forceRegeneration) {
return { run: existing, claimed: false };
}
await client.query(`
UPDATE assessor_regulatory_mappings SET
run_status = $2, query_hash = $3, created_at = $4, updated_at = $5, data = $6::jsonb
WHERE id = $1
`, [run.id, run.status, run.queryHash, run.createdAt, run.updatedAt, json(run)]);
return { run: clone(run), claimed: true };
});
}
async saveRegulatoryMappingRun(run: RegulatoryMappingRun): Promise<RegulatoryMappingRun> { async saveRegulatoryMappingRun(run: RegulatoryMappingRun): Promise<RegulatoryMappingRun> {
const errors = regulatoryMappingRunValidationErrors(run); const errors = regulatoryMappingRunValidationErrors(run);
if (errors.length) throw new RepositoryValidationError(errors.join(' ')); if (errors.length) throw new RepositoryValidationError(errors.join(' '));
@ -728,6 +760,60 @@ export class PostgresAssessorRepository implements AssessorRepository {
)); ));
} }
async createRegulatoryMappingBatchRun(run: RegulatoryMappingBatchRun): Promise<RegulatoryMappingBatchRun> {
const errors = regulatoryMappingBatchRunValidationErrors(run);
if (errors.length) throw new RepositoryValidationError(errors.join(' '));
return this.mutate(async client => {
const inserted = await client.query(`
INSERT INTO assessor_regulatory_mapping_batches
(id, project_id, scope_key, filter_hash, idempotency_key, run_status, created_at, updated_at, data)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9::jsonb)
ON CONFLICT DO NOTHING
RETURNING data
`, [run.id, run.projectId, run.scopeKey, run.filterHash, run.idempotencyKey, run.status,
run.createdAt, run.updatedAt, json(run)]);
if (inserted.rowCount) return clone(run);
const existing = dataRow(await client.query<DataRow<RegulatoryMappingBatchRun>>(`
SELECT data FROM assessor_regulatory_mapping_batches
WHERE project_id = $1 AND (idempotency_key = $2 OR (scope_key = $3 AND run_status IN ('queued', 'running')))
ORDER BY (idempotency_key = $2) DESC, created_at DESC, id DESC
LIMIT 1
`, [run.projectId, run.idempotencyKey, run.scopeKey]));
if (!existing) throw new RepositoryConflictError('Regulatory mapping batch conflicted with another active request.');
return existing;
});
}
async updateRegulatoryMappingBatchRun(run: RegulatoryMappingBatchRun): Promise<RegulatoryMappingBatchRun> {
const errors = regulatoryMappingBatchRunValidationErrors(run);
if (errors.length) throw new RepositoryValidationError(errors.join(' '));
return this.mutate(async client => {
const existing = dataRow(await client.query<DataRow<RegulatoryMappingBatchRun>>(
'SELECT data FROM assessor_regulatory_mapping_batches WHERE id = $1 FOR UPDATE', [run.id]
));
if (!existing) throw new RepositoryNotFoundError('Regulatory mapping batch does not exist.');
if (existing.projectId !== run.projectId || existing.scopeKey !== run.scopeKey || existing.idempotencyKey !== run.idempotencyKey) {
throw new RepositoryConflictError('Regulatory mapping batch identity is immutable.');
}
if (['completed', 'failed'].includes(existing.status)) throw new RepositoryConflictError('A completed regulatory mapping batch is immutable.');
const allowed = existing.status === 'queued' ? ['running', 'failed'] : ['running', 'completed', 'failed'];
if (!allowed.includes(run.status)) throw new RepositoryConflictError(`Invalid regulatory mapping batch transition from ${existing.status} to ${run.status}.`);
await client.query(`
UPDATE assessor_regulatory_mapping_batches SET run_status = $2, updated_at = $3, data = $4::jsonb
WHERE id = $1
`, [run.id, run.status, run.updatedAt, json(run)]);
return clone(run);
});
}
async listRegulatoryMappingBatchRuns(projectId: string, scopeKey?: string): Promise<RegulatoryMappingBatchRun[]> {
return dataRows(await this.pool.query<DataRow<RegulatoryMappingBatchRun>>(`
SELECT data FROM assessor_regulatory_mapping_batches
WHERE project_id = $1 AND ($2::text IS NULL OR scope_key = $2)
ORDER BY created_at DESC, id DESC
`, [projectId, scopeKey ?? null]));
}
async createRegulatoryMappingDecision( async createRegulatoryMappingDecision(
decision: RegulatoryMappingDecision, decision: RegulatoryMappingDecision,
expectedLatestDecisionId?: string expectedLatestDecisionId?: string
@ -2760,6 +2846,7 @@ export class PostgresAssessorRepository implements AssessorRepository {
policyEvaluations: await this.listPolicyEvaluations(projectId), policyEvaluations: await this.listPolicyEvaluations(projectId),
policyFindingRuns, policyFindingRuns,
aiOverviewRuns: (await this.listAiOverviewRuns(projectId)).filter(item => item.status === 'succeeded'), aiOverviewRuns: (await this.listAiOverviewRuns(projectId)).filter(item => item.status === 'succeeded'),
regulatoryMappingBatchRuns: await this.listRegulatoryMappingBatchRuns(projectId),
// Regulatory mappings and decisions are global reference data, not project-scoped. // Regulatory mappings and decisions are global reference data, not project-scoped.
regulatoryMappingRuns: (await this.listRegulatoryMappingRuns()).filter(item => item.status === 'succeeded'), regulatoryMappingRuns: (await this.listRegulatoryMappingRuns()).filter(item => item.status === 'succeeded'),
regulatoryMappingDecisions: dataRows(await this.pool.query<DataRow<RegulatoryMappingDecision>>( regulatoryMappingDecisions: dataRows(await this.pool.query<DataRow<RegulatoryMappingDecision>>(

326
server/regulatory/regulatory-mapping-service.ts
View File

@ -2,13 +2,19 @@ import { createHash, randomUUID } from 'node:crypto';
import type { AiCompletionClient } from '../ai/litellm-client'; import type { AiCompletionClient } from '../ai/litellm-client';
import { LiteLlmClientError } from '../ai/litellm-client'; import { LiteLlmClientError } from '../ai/litellm-client';
import type { AssessorRepository } from '../../src/persistence/assessor-repository'; import type { AssessorRepository } from '../../src/persistence/assessor-repository';
import { RepositoryNotFoundError, RepositoryValidationError } from '../../src/persistence/repository-errors'; import { RepositoryConflictError, RepositoryNotFoundError, RepositoryValidationError } from '../../src/persistence/repository-errors';
import { import {
REGULATORY_MAPPING_OUTPUT_SCHEMA_VERSION, REGULATORY_MAPPING_OUTPUT_SCHEMA_VERSION,
REGULATORY_MAPPING_PROMPT_VERSION, REGULATORY_MAPPING_PROMPT_VERSION,
REGULATORY_RETRIEVAL_VERSION, REGULATORY_RETRIEVAL_VERSION,
emptyRegulatoryMappingBatchCounts,
groundRegulatoryMappingResult, groundRegulatoryMappingResult,
normalizeRegulatoryMappingBatchScope,
regulatoryMappingResultValidationErrors, regulatoryMappingResultValidationErrors,
type RegulatoryMappingBatchCreateRequest,
type RegulatoryMappingBatchResponse,
type RegulatoryMappingBatchRun,
type RegulatoryMappingBatchScope,
type RegulatoryMappingCandidate, type RegulatoryMappingCandidate,
type RegulatoryMappingCreateRequest, type RegulatoryMappingCreateRequest,
type RegulatoryMappingDecision, type RegulatoryMappingDecision,
@ -24,9 +30,12 @@ import { privacySafeFinding } from '../../src/domain/privacy-safe-finding';
import type { RegulatoryFindingRetrievalService } from './regulatory-finding-retrieval'; import type { RegulatoryFindingRetrievalService } from './regulatory-finding-retrieval';
import type { RegulatoryChunkIndex } from './qdrant-regulatory-index'; import type { RegulatoryChunkIndex } from './qdrant-regulatory-index';
import { buildRegulatoryFindingQuery } from '../../src/profiles/public-search/regulatory-query'; import { buildRegulatoryFindingQuery } from '../../src/profiles/public-search/regulatory-query';
import { canonicalJson } from '../../src/shared/canonical-json';
const POLL_AFTER_MS = 1_500; const POLL_AFTER_MS = 1_500;
const MAXIMUM_PROMPT_CHARACTERS = 48_000; const MAXIMUM_PROMPT_CHARACTERS = 48_000;
const BATCH_PAGE_SIZE = 500;
const DEFAULT_MAXIMUM_CONCURRENT_RUNS = 2;
export interface RegulatoryMappingServiceOptions { export interface RegulatoryMappingServiceOptions {
findingRetrieval?: RegulatoryFindingRetrievalService; findingRetrieval?: RegulatoryFindingRetrievalService;
@ -37,14 +46,25 @@ export interface RegulatoryMappingServiceOptions {
embeddingModel?: string; embeddingModel?: string;
disabledReason?: string; disabledReason?: string;
now?: () => Date; now?: () => Date;
maximumConcurrentRuns?: number;
} }
export class RegulatoryMappingService { export class RegulatoryMappingService {
private readonly activeTasks = new Set<Promise<void>>(); private readonly activeTasks = new Set<Promise<void>>();
private readonly now: () => Date; private readonly now: () => Date;
private readonly maximumConcurrentRuns: number;
private readonly workQueue: Array<{
task: () => Promise<'succeeded' | 'failed'>;
resolve: (status: 'succeeded' | 'failed') => void;
}> = [];
private activeWorkerCount = 0;
constructor(private readonly repository: AssessorRepository, private readonly options: RegulatoryMappingServiceOptions = {}) { constructor(private readonly repository: AssessorRepository, private readonly options: RegulatoryMappingServiceOptions = {}) {
this.now = options.now ?? (() => new Date()); this.now = options.now ?? (() => new Date());
this.maximumConcurrentRuns = options.maximumConcurrentRuns ?? DEFAULT_MAXIMUM_CONCURRENT_RUNS;
if (!Number.isInteger(this.maximumConcurrentRuns) || this.maximumConcurrentRuns < 1 || this.maximumConcurrentRuns > 8) {
throw new Error('Regulatory mapping concurrency must be an integer between 1 and 8.');
}
} }
get generationEnabled(): boolean { get generationEnabled(): boolean {
@ -63,6 +83,18 @@ export class RegulatoryMappingService {
errorCode: 'regulatory_mapping_interrupted' errorCode: 'regulatory_mapping_interrupted'
}); });
} }
for (const project of await this.repository.listProjects()) {
for (const run of (await this.repository.listRegulatoryMappingBatchRuns(project.id)).filter(item => ['queued', 'running'].includes(item.status))) {
await this.repository.updateRegulatoryMappingBatchRun({
...run,
status: 'failed',
updatedAt: completedAt,
completedAt,
errorCode: 'regulatory_mapping_batch_interrupted',
errorDetail: 'Batch processing was interrupted before completion. Start a new batch to retry remaining rules.'
});
}
}
} }
async get(projectId: string, resultProjectionId: string): Promise<RegulatoryMappingResponse> { async get(projectId: string, resultProjectionId: string): Promise<RegulatoryMappingResponse> {
@ -93,55 +125,60 @@ export class RegulatoryMappingService {
if (!this.generationEnabled) throw new RegulatoryMappingServiceError('regulatory_mapping_disabled', this.options.disabledReason ?? 'Regulatory mapping is not configured.', 503); if (!this.generationEnabled) throw new RegulatoryMappingServiceError('regulatory_mapping_disabled', this.options.disabledReason ?? 'Regulatory mapping is not configured.', 503);
const finding = await this.finding(projectId, request.resultProjectionId); const finding = await this.finding(projectId, request.resultProjectionId);
if (finding.sourceKind !== 'scanner-finding') throw new RegulatoryMappingServiceError('unsupported_finding_kind', 'Only canonical scanner findings can be mapped in this version.', 422); if (finding.sourceKind !== 'scanner-finding') throw new RegulatoryMappingServiceError('unsupported_finding_kind', 'Only canonical scanner findings can be mapped in this version.', 422);
const ruleKey = this.ruleKey(finding); if (!buildRegulatoryFindingQuery(finding)) throw new RegulatoryMappingServiceError('insufficient_public_context', 'This finding has insufficient public context for regulatory mapping.', 422);
const id = this.mappingId(ruleKey); const claimed = await this.claim(finding, true);
const existing = await this.repository.getRegulatoryMappingRun(id); if (claimed.claimed) this.track(this.enqueue(() => this.process(claimed.run, finding)).then(() => undefined));
if (existing && ['queued', 'running'].includes(existing.status)) return this.get(projectId, request.resultProjectionId);
let retrieval;
try {
retrieval = await this.options.findingRetrieval!.retrieve(finding, 32, {
corpusId: this.options.corpusId,
corpusVersion: this.options.corpusVersion,
sourceStandard: 'ISO/IEC 27002',
retrievalRole: 'semantic'
});
} catch {
throw new RegulatoryMappingServiceError('regulatory_retrieval_unavailable', 'Regulatory retrieval is currently unavailable.', 503);
}
if (retrieval.status !== 'candidates' || !retrieval.preparedQuery) {
throw new RegulatoryMappingServiceError('insufficient_public_context', 'This finding has insufficient public context for regulatory mapping.', 422);
}
const candidates = await this.groupCandidates(retrieval.candidates);
if (!candidates.length) throw new RegulatoryMappingServiceError('regulatory_corpus_empty', 'The active regulatory corpus returned no eligible controls.', 503);
const timestamp = this.now().toISOString();
const queryHash = createHash('sha256').update(JSON.stringify(retrieval.preparedQuery), 'utf8').digest('hex');
const proposed: RegulatoryMappingRun = {
version: 1,
id,
ruleKey,
sourceTool: finding.sourceTool ?? 'unknown',
...(finding.sourceRuleId ? { sourceRuleId: finding.sourceRuleId } : {}),
status: 'queued',
queryHash,
preparedQuery: retrieval.preparedQuery,
corpusId: this.options.corpusId!,
corpusVersion: this.options.corpusVersion!,
sourceHashes: [...new Set(candidates.flatMap(candidate => candidate.evidence.map(evidence => evidence.sourceSha256)))].sort(),
embeddingModel: this.options.embeddingModel!,
retrievalVersion: REGULATORY_RETRIEVAL_VERSION,
promptVersion: REGULATORY_MAPPING_PROMPT_VERSION,
outputSchemaVersion: REGULATORY_MAPPING_OUTPUT_SCHEMA_VERSION,
model: this.options.client!.model,
createdAt: timestamp,
updatedAt: timestamp,
candidates
};
await this.repository.saveRegulatoryMappingRun(proposed);
const task = this.process(proposed).finally(() => this.activeTasks.delete(task));
this.activeTasks.add(task);
return this.get(projectId, request.resultProjectionId); return this.get(projectId, request.resultProjectionId);
} }
async getBatch(projectId: string, requestedScope: RegulatoryMappingBatchScope): Promise<RegulatoryMappingBatchResponse> {
const scope = await this.validateBatchScope(projectId, requestedScope);
const scopeKey = this.batchScopeKey(projectId, scope);
const run = (await this.repository.listRegulatoryMappingBatchRuns(projectId, scopeKey))[0];
const availability = await this.availability();
return {
version: 1,
projectId,
scope,
generationEnabled: availability.enabled,
state: run?.status ?? 'empty',
...(!availability.enabled ? { disabledReason: availability.reason ?? this.options.disabledReason ?? 'Regulatory mapping is not configured.' } : {}),
...(run && ['queued', 'running'].includes(run.status) ? { pollAfterMs: POLL_AFTER_MS } : {}),
...(run ? { run } : {})
};
}
async startBatch(projectId: string, request: RegulatoryMappingBatchCreateRequest): Promise<RegulatoryMappingBatchResponse> {
if (!this.generationEnabled) throw new RegulatoryMappingServiceError('regulatory_mapping_disabled', this.options.disabledReason ?? 'Regulatory mapping is not configured.', 503);
if (typeof request.idempotencyKey !== 'string' || !/^[A-Za-z0-9._:-]{8,128}$/.test(request.idempotencyKey)) {
throw new RepositoryValidationError('idempotencyKey must contain 8 to 128 safe characters.');
}
const scope = await this.validateBatchScope(projectId, request);
const timestamp = this.now().toISOString();
const candidate: RegulatoryMappingBatchRun = {
version: 1,
id: `regulatory-mapping-batch-${randomUUID()}`,
projectId,
scopeKey: this.batchScopeKey(projectId, scope),
filterHash: sha256(canonicalJson(scope.filters)),
assessmentIds: scope.assessmentIds,
scanRunIds: scope.scanRunIds,
filters: scope.filters,
idempotencyKey: request.idempotencyKey,
status: 'queued',
counts: emptyRegulatoryMappingBatchCounts(),
createdAt: timestamp,
updatedAt: timestamp
};
const run = await this.repository.createRegulatoryMappingBatchRun(candidate);
if (run.id === candidate.id) this.track(this.processBatch(run));
return this.getBatch(projectId, {
assessmentIds: run.assessmentIds,
scanRunIds: run.scanRunIds,
filters: run.filters
});
}
async decide(projectId: string, request: RegulatoryMappingDecisionCreateRequest): Promise<RegulatoryMappingResponse> { async decide(projectId: string, request: RegulatoryMappingDecisionCreateRequest): Promise<RegulatoryMappingResponse> {
if (!this.generationEnabled) throw new RegulatoryMappingServiceError('regulatory_mapping_disabled', this.options.disabledReason ?? 'Regulatory mapping is not configured.', 503); if (!this.generationEnabled) throw new RegulatoryMappingServiceError('regulatory_mapping_disabled', this.options.disabledReason ?? 'Regulatory mapping is not configured.', 503);
const finding = await this.finding(projectId, request.resultProjectionId); const finding = await this.finding(projectId, request.resultProjectionId);
@ -182,7 +219,162 @@ export class RegulatoryMappingService {
return `regulatory-mapping-${createHash('sha256').update(tuple, 'utf8').digest('hex').slice(0, 40)}`; return `regulatory-mapping-${createHash('sha256').update(tuple, 'utf8').digest('hex').slice(0, 40)}`;
} }
async close(): Promise<void> { await Promise.allSettled([...this.activeTasks]); } async close(): Promise<void> {
while (this.activeTasks.size) await Promise.allSettled([...this.activeTasks]);
}
private track(task: Promise<void>): void {
this.activeTasks.add(task);
void task.finally(() => this.activeTasks.delete(task));
}
private enqueue(task: () => Promise<'succeeded' | 'failed'>): Promise<'succeeded' | 'failed'> {
const promise = new Promise<'succeeded' | 'failed'>(resolve => this.workQueue.push({ task, resolve }));
this.drainQueue();
return promise;
}
private drainQueue(): void {
while (this.activeWorkerCount < this.maximumConcurrentRuns && this.workQueue.length) {
const work = this.workQueue.shift()!;
this.activeWorkerCount += 1;
void work.task().then(work.resolve, () => work.resolve('failed')).finally(() => {
this.activeWorkerCount -= 1;
this.drainQueue();
});
}
}
private queuedRun(finding: UnifiedResultProjection): RegulatoryMappingRun {
const preparedQuery = buildRegulatoryFindingQuery(finding);
if (!preparedQuery) throw new RegulatoryMappingServiceError('insufficient_public_context', 'This finding has insufficient public context for regulatory mapping.', 422);
const timestamp = this.now().toISOString();
const ruleKey = this.ruleKey(finding);
return {
version: 1,
id: this.mappingId(ruleKey),
ruleKey,
sourceTool: finding.sourceTool ?? 'unknown',
...(finding.sourceRuleId ? { sourceRuleId: finding.sourceRuleId } : {}),
status: 'queued',
queryHash: sha256(JSON.stringify(preparedQuery)),
preparedQuery,
corpusId: this.options.corpusId!,
corpusVersion: this.options.corpusVersion!,
sourceHashes: [],
embeddingModel: this.options.embeddingModel!,
retrievalVersion: REGULATORY_RETRIEVAL_VERSION,
promptVersion: REGULATORY_MAPPING_PROMPT_VERSION,
outputSchemaVersion: REGULATORY_MAPPING_OUTPUT_SCHEMA_VERSION,
model: this.options.client!.model,
createdAt: timestamp,
updatedAt: timestamp,
candidates: []
};
}
private async claim(finding: UnifiedResultProjection, forceRegeneration: boolean) {
return this.repository.claimRegulatoryMappingRun(this.queuedRun(finding), forceRegeneration);
}
private async validateBatchScope(projectId: string, requested: RegulatoryMappingBatchScope): Promise<RegulatoryMappingBatchScope> {
if (!await this.repository.getProject(projectId)) throw new RepositoryNotFoundError('The selected project does not exist.');
const scope = normalizeRegulatoryMappingBatchScope(requested);
if (!scope.assessmentIds.length && !scope.scanRunIds.length) throw new RepositoryValidationError('A selected timeslot scope is required.');
const assessments = await this.repository.listAssessments(projectId);
const allowedAssessmentIds = new Set(assessments.map(item => item.id));
if (scope.assessmentIds.some(id => !allowedAssessmentIds.has(id))) throw new RepositoryValidationError('The batch scope contains an assessment outside this project.');
const scanRuns = await this.repository.listProjectScanRuns(projectId);
const allowedRuns = new Map(scanRuns.filter(item => !item.supersededByScanRunId).map(item => [item.id, item]));
if (scope.scanRunIds.some(id => !allowedRuns.has(id))) throw new RepositoryValidationError('The batch scope contains an unavailable scan run.');
if (scope.assessmentIds.length && scope.scanRunIds.some(id => !scope.assessmentIds.includes(allowedRuns.get(id)!.assessmentId))) {
throw new RepositoryValidationError('The assessment and scan-run scope do not describe the same timeslot.');
}
return scope;
}
private batchScopeKey(projectId: string, scope: RegulatoryMappingBatchScope): string {
return sha256(canonicalJson({ projectId, ...scope }));
}
private async queryBatchResults(projectId: string, scope: RegulatoryMappingBatchScope): Promise<UnifiedResultProjection[]> {
const query = {
...scope.filters,
audience: 'result-db' as const,
projectId,
assessmentIds: scope.assessmentIds,
scanRunIds: scope.scanRunIds
};
const items: UnifiedResultProjection[] = [];
let expectedTotal: number | undefined;
for (let offset = 0; ; offset += BATCH_PAGE_SIZE) {
const page = await this.repository.queryUnifiedResults({ ...query, offset, limit: BATCH_PAGE_SIZE });
expectedTotal ??= page.total;
if (page.total !== expectedTotal) throw new RepositoryConflictError('The filtered result set changed while the ISO mapping batch was being prepared.');
items.push(...page.items);
if (items.length >= page.total || page.items.length === 0) break;
}
if (items.length !== expectedTotal) throw new RepositoryConflictError('The complete filtered result set could not be read consistently.');
return items;
}
private async processBatch(queued: RegulatoryMappingBatchRun): Promise<void> {
const startedAt = this.now().toISOString();
let run = await this.repository.updateRegulatoryMappingBatchRun({ ...queued, status: 'running', startedAt, updatedAt: startedAt });
try {
const results = (await this.queryBatchResults(run.projectId, run)).slice()
.sort((left, right) => left.resultProjectionId.localeCompare(right.resultProjectionId));
const representatives = new Map<string, UnifiedResultProjection>();
const counts = { ...emptyRegulatoryMappingBatchCounts(), filteredResultCount: results.length };
for (const finding of results) {
if (finding.sourceKind !== 'scanner-finding') { counts.excludedNonScannerCount += 1; continue; }
if (!buildRegulatoryFindingQuery(finding)) { counts.unsupportedFindingCount += 1; continue; }
counts.eligibleFindingCount += 1;
const ruleKey = this.ruleKey(finding);
if (!representatives.has(ruleKey)) representatives.set(ruleKey, finding);
}
counts.distinctRuleCount = representatives.size;
const scheduled: Array<Promise<'succeeded' | 'failed'>> = [];
for (const finding of representatives.values()) {
const claimed = await this.claim(finding, false);
if (!claimed.claimed) {
if (claimed.run.status === 'succeeded') counts.reusedSucceededRuleCount += 1;
else counts.reusedActiveRuleCount += 1;
continue;
}
counts.scheduledRuleCount += 1;
const completion = this.enqueue(() => this.process(claimed.run, finding));
this.track(completion.then(() => undefined));
scheduled.push(completion);
}
run = await this.repository.updateRegulatoryMappingBatchRun({ ...run, counts, updatedAt: this.now().toISOString() });
let progress = Promise.resolve();
const completions = scheduled.map(completion => completion.then(status => {
progress = progress.then(async () => {
if (status === 'succeeded') counts.succeededRuleCount += 1;
else counts.failedRuleCount += 1;
run = await this.repository.updateRegulatoryMappingBatchRun({ ...run, counts: { ...counts }, updatedAt: this.now().toISOString() });
});
return progress;
}));
await Promise.all(completions);
await progress;
const completedAt = this.now().toISOString();
await this.repository.updateRegulatoryMappingBatchRun({ ...run, status: 'completed', counts, updatedAt: completedAt, completedAt });
} catch (error) {
const completedAt = this.now().toISOString();
try {
await this.repository.updateRegulatoryMappingBatchRun({
...run,
status: 'failed',
updatedAt: completedAt,
completedAt,
errorCode: error instanceof RepositoryConflictError ? 'regulatory_mapping_batch_scope_changed' : 'regulatory_mapping_batch_failed',
errorDetail: error instanceof RepositoryConflictError ? error.message : 'The batch stopped safely before all rules were scheduled.'
});
} catch { /* Preserve the original safe batch failure. */ }
}
}
private async availability(): Promise<{ enabled: boolean; reason?: string }> { private async availability(): Promise<{ enabled: boolean; reason?: string }> {
if (!this.generationEnabled) return { enabled: false, reason: this.options.disabledReason ?? 'Regulatory mapping is not configured.' }; if (!this.generationEnabled) return { enabled: false, reason: this.options.disabledReason ?? 'Regulatory mapping is not configured.' };
@ -241,10 +433,36 @@ export class RegulatoryMappingService {
})); }));
} }
private async process(queued: RegulatoryMappingRun): Promise<void> { private async process(queued: RegulatoryMappingRun, finding: UnifiedResultProjection): Promise<'succeeded' | 'failed'> {
const startedAt = this.now().toISOString(); let run = queued;
const run = await this.repository.saveRegulatoryMappingRun({ ...queued, status: 'running', startedAt, updatedAt: startedAt });
try { try {
let retrieval;
try {
retrieval = await this.options.findingRetrieval!.retrieve(finding, 32, {
corpusId: this.options.corpusId,
corpusVersion: this.options.corpusVersion,
sourceStandard: 'ISO/IEC 27002',
retrievalRole: 'semantic'
});
} catch {
throw new RegulatoryMappingServiceError('regulatory_retrieval_unavailable', 'Regulatory retrieval is currently unavailable.', 503);
}
if (retrieval.status !== 'candidates' || !retrieval.preparedQuery) {
throw new RegulatoryMappingServiceError('insufficient_public_context', 'This finding has insufficient public context for regulatory mapping.', 422);
}
const candidates = await this.groupCandidates(retrieval.candidates);
if (!candidates.length) throw new RegulatoryMappingServiceError('regulatory_corpus_empty', 'The active regulatory corpus returned no eligible controls.', 503);
const startedAt = this.now().toISOString();
run = await this.repository.saveRegulatoryMappingRun({
...queued,
status: 'running',
preparedQuery: retrieval.preparedQuery,
queryHash: sha256(JSON.stringify(retrieval.preparedQuery)),
sourceHashes: [...new Set(candidates.flatMap(candidate => candidate.evidence.map(evidence => evidence.sourceSha256)))].sort(),
candidates,
startedAt,
updatedAt: startedAt
});
const prompt = mappingPrompt(run); const prompt = mappingPrompt(run);
const completion = await this.options.client!.complete(SYSTEM_PROMPT, prompt); const completion = await this.options.client!.complete(SYSTEM_PROMPT, prompt);
const allowed = new Map(run.candidates.map(candidate => [ const allowed = new Map(run.candidates.map(candidate => [
@ -262,6 +480,7 @@ export class RegulatoryMappingService {
updatedAt: completedAt, updatedAt: completedAt,
completedAt completedAt
}); });
return 'succeeded';
} catch (error) { } catch (error) {
const completedAt = this.now().toISOString(); const completedAt = this.now().toISOString();
const failure = safeFailure(error); const failure = safeFailure(error);
@ -272,6 +491,7 @@ export class RegulatoryMappingService {
...(detail ? { errorDetail: detail } : {}) ...(detail ? { errorDetail: detail } : {})
}); });
} catch { /* Preserve the original safe failure code. */ } } catch { /* Preserve the original safe failure code. */ }
return 'failed';
} }
} }
} }
@ -329,3 +549,7 @@ function safeFailureDetail(error: unknown): string | undefined {
const detail = error.message.replace(/<\/?[a-z][^>]*>/gi, ' ').replace(/\s+/g, ' ').trim().slice(0, 600); const detail = error.message.replace(/<\/?[a-z][^>]*>/gi, ' ').replace(/\s+/g, ' ').trim().slice(0, 600);
return detail || undefined; return detail || undefined;
} }
function sha256(value: string): string {
return createHash('sha256').update(value, 'utf8').digest('hex');
}

17
src/api/assessor-api-client.ts
View File

@ -55,6 +55,9 @@ import {
type MicrosoftSecurityBaselineAnalysisListResponse, type MicrosoftSecurityBaselineAnalysisListResponse,
type AiOverviewCreateRequest, type AiOverviewCreateRequest,
type AiOverviewResponse, type AiOverviewResponse,
type RegulatoryMappingBatchCreateRequest,
type RegulatoryMappingBatchResponse,
type RegulatoryMappingBatchScope,
type RegulatoryMappingCreateRequest, type RegulatoryMappingCreateRequest,
type RegulatoryMappingDecisionCreateRequest, type RegulatoryMappingDecisionCreateRequest,
type RegulatoryMappingResponse type RegulatoryMappingResponse
@ -246,6 +249,20 @@ export class AssessorApiClient {
}); });
} }
getRegulatoryMappingBatch(projectId: string, scope: RegulatoryMappingBatchScope): Promise<RegulatoryMappingBatchResponse> {
return this.request(unifiedQueryPath(`/projects/${encodeURIComponent(projectId)}/regulatory-mapping-batches`, {
...scope.filters,
assessmentIds: scope.assessmentIds,
scanRunIds: scope.scanRunIds
}));
}
startRegulatoryMappingBatch(projectId: string, request: RegulatoryMappingBatchCreateRequest): Promise<RegulatoryMappingBatchResponse> {
return this.request(`/projects/${encodeURIComponent(projectId)}/regulatory-mapping-batches`, {
method: 'POST', body: JSON.stringify(request)
});
}
listProjectImportFiles(projectId: string): Promise<ImportFileRecord[]> { listProjectImportFiles(projectId: string): Promise<ImportFileRecord[]> {
return this.request(`/projects/${encodeURIComponent(projectId)}/import-files`); return this.request(`/projects/${encodeURIComponent(projectId)}/import-files`);
} }

3
src/api/assessor-api-contract.ts
View File

@ -19,6 +19,9 @@ import type { MitreGraphData } from '../features/mitre-graph/mitre-graph';
import type { AssessmentContext } from '../assessment-context/context-engine'; import type { AssessmentContext } from '../assessment-context/context-engine';
export type { AiOverviewCreateRequest, AiOverviewResponse } from '../domain/ai-overview'; export type { AiOverviewCreateRequest, AiOverviewResponse } from '../domain/ai-overview';
export type { export type {
RegulatoryMappingBatchCreateRequest,
RegulatoryMappingBatchResponse,
RegulatoryMappingBatchScope,
RegulatoryMappingCreateRequest, RegulatoryMappingCreateRequest,
RegulatoryMappingDecisionCreateRequest, RegulatoryMappingDecisionCreateRequest,
RegulatoryMappingResponse RegulatoryMappingResponse

140
src/domain/regulatory-mapping.ts
View File

@ -1,5 +1,6 @@
import type { AiOverviewTokenUsage } from './ai-overview'; import type { AiOverviewTokenUsage } from './ai-overview';
import type { RegulatoryFindingQuery, RegulatorySectionKind } from './regulatory-rag'; import type { RegulatoryFindingQuery, RegulatorySectionKind } from './regulatory-rag';
import type { UnifiedResultQuery } from './unified-result';
export const REGULATORY_MAPPING_PROMPT_VERSION = '1.2.0'; export const REGULATORY_MAPPING_PROMPT_VERSION = '1.2.0';
export const REGULATORY_MAPPING_OUTPUT_SCHEMA_VERSION = 1 as const; export const REGULATORY_MAPPING_OUTPUT_SCHEMA_VERSION = 1 as const;
@ -9,6 +10,61 @@ export type RegulatoryMappingRunStatus = 'queued' | 'running' | 'succeeded' | 'f
export type RegulatoryMappingRelationship = 'primary' | 'supporting'; export type RegulatoryMappingRelationship = 'primary' | 'supporting';
export type RegulatoryMappingConfidence = 'low' | 'medium' | 'high'; export type RegulatoryMappingConfidence = 'low' | 'medium' | 'high';
export type RegulatoryMappingDecisionValue = 'accepted' | 'rejected'; export type RegulatoryMappingDecisionValue = 'accepted' | 'rejected';
export type RegulatoryMappingBatchStatus = 'queued' | 'running' | 'completed' | 'failed';
export type RegulatoryMappingBatchFilters = Omit<UnifiedResultQuery,
'audience' | 'projectId' | 'assessmentId' | 'assessmentIds' | 'scanRunId' | 'scanRunIds' | 'offset' | 'limit'>;
export interface RegulatoryMappingBatchScope {
assessmentIds: string[];
scanRunIds: string[];
filters: RegulatoryMappingBatchFilters;
}
export interface RegulatoryMappingBatchCounts {
filteredResultCount: number;
excludedNonScannerCount: number;
eligibleFindingCount: number;
unsupportedFindingCount: number;
distinctRuleCount: number;
reusedSucceededRuleCount: number;
reusedActiveRuleCount: number;
scheduledRuleCount: number;
succeededRuleCount: number;
failedRuleCount: number;
}
export interface RegulatoryMappingBatchRun extends RegulatoryMappingBatchScope {
version: 1;
id: string;
projectId: string;
scopeKey: string;
filterHash: string;
idempotencyKey: string;
status: RegulatoryMappingBatchStatus;
counts: RegulatoryMappingBatchCounts;
createdAt: string;
updatedAt: string;
startedAt?: string;
completedAt?: string;
errorCode?: string;
errorDetail?: string;
}
export interface RegulatoryMappingBatchResponse {
version: 1;
projectId: string;
scope: RegulatoryMappingBatchScope;
generationEnabled: boolean;
state: 'empty' | RegulatoryMappingBatchStatus;
disabledReason?: string;
pollAfterMs?: number;
run?: RegulatoryMappingBatchRun;
}
export interface RegulatoryMappingBatchCreateRequest extends RegulatoryMappingBatchScope {
idempotencyKey: string;
}
export interface RegulatoryMappingEvidence { export interface RegulatoryMappingEvidence {
chunkId: string; chunkId: string;
@ -185,8 +241,9 @@ export function regulatoryMappingRunValidationErrors(run: RegulatoryMappingRun):
if (run.sourceRuleId !== undefined && (!run.sourceRuleId.trim() || run.sourceRuleId.length > 200)) errors.push('Regulatory mapping sourceRuleId is invalid.'); if (run.sourceRuleId !== undefined && (!run.sourceRuleId.trim() || run.sourceRuleId.length > 200)) errors.push('Regulatory mapping sourceRuleId is invalid.');
if (!['queued', 'running', 'succeeded', 'failed'].includes(run.status)) errors.push('Regulatory mapping run status is invalid.'); if (!['queued', 'running', 'succeeded', 'failed'].includes(run.status)) errors.push('Regulatory mapping run status is invalid.');
if (!/^[a-f0-9]{64}$/.test(run.queryHash)) errors.push('Regulatory mapping queryHash must be SHA-256.'); if (!/^[a-f0-9]{64}$/.test(run.queryHash)) errors.push('Regulatory mapping queryHash must be SHA-256.');
if (!run.sourceHashes.length || run.sourceHashes.some(value => !/^[a-f0-9]{64}$/.test(value))) errors.push('Regulatory mapping sourceHashes must contain SHA-256 digests.'); const candidatesRequired = run.status === 'running' || run.status === 'succeeded';
if (!Array.isArray(run.candidates) || run.candidates.length < 1 || run.candidates.length > 8) errors.push('Regulatory mapping candidates must contain one to eight controls.'); if ((!run.sourceHashes.length && candidatesRequired) || run.sourceHashes.some(value => !/^[a-f0-9]{64}$/.test(value))) errors.push('Regulatory mapping sourceHashes must contain SHA-256 digests.');
if (!Array.isArray(run.candidates) || (candidatesRequired && run.candidates.length < 1) || run.candidates.length > 8) errors.push('Regulatory mapping candidates must contain one to eight controls when processing has started.');
else { else {
const controls = new Set<string>(); const controls = new Set<string>();
run.candidates.forEach(candidate => { run.candidates.forEach(candidate => {
@ -207,6 +264,85 @@ export function regulatoryMappingRunValidationErrors(run: RegulatoryMappingRun):
return [...new Set(errors)]; return [...new Set(errors)];
} }
export function normalizeRegulatoryMappingBatchScope(scope: RegulatoryMappingBatchScope): RegulatoryMappingBatchScope {
const values = (input: readonly string[] | undefined) => [...new Set((input ?? []).map(value => value.trim()).filter(Boolean))]
.sort((left, right) => left.localeCompare(right));
const text = (input: string | undefined) => input?.trim() || undefined;
const filters = scope.filters ?? {};
return {
assessmentIds: values(scope.assessmentIds),
scanRunIds: values(scope.scanRunIds),
filters: {
evaluationRunId: text(filters.evaluationRunId),
sourceKind: values(filters.sourceKind) as RegulatoryMappingBatchFilters['sourceKind'],
sourceTool: values(filters.sourceTool),
severity: values(filters.severity) as RegulatoryMappingBatchFilters['severity'],
outcome: values(filters.outcome) as RegulatoryMappingBatchFilters['outcome'],
technicalExecutionState: values(filters.technicalExecutionState) as RegulatoryMappingBatchFilters['technicalExecutionState'],
resultType: values(filters.resultType) as RegulatoryMappingBatchFilters['resultType'],
policyClassification: values(filters.policyClassification) as RegulatoryMappingBatchFilters['policyClassification'],
isIoe: filters.isIoe,
deviation: filters.deviation,
mappingRuleVersion: values(filters.mappingRuleVersion),
sourceProfileVersion: values(filters.sourceProfileVersion),
processingStatus: values(filters.processingStatus) as RegulatoryMappingBatchFilters['processingStatus'],
reviewDisposition: values(filters.reviewDisposition) as RegulatoryMappingBatchFilters['reviewDisposition'],
lifecycleState: values(filters.lifecycleState) as RegulatoryMappingBatchFilters['lifecycleState'],
categoryNamespace: values(filters.categoryNamespace),
categoryValue: values(filters.categoryValue),
affectedAsset: text(filters.affectedAsset),
environmentState: values(filters.environmentState) as RegulatoryMappingBatchFilters['environmentState'],
environmentProperty: values(filters.environmentProperty),
environmentSubjectType: values(filters.environmentSubjectType),
search: text(filters.search),
observedFrom: text(filters.observedFrom),
observedTo: text(filters.observedTo)
}
};
}
export function emptyRegulatoryMappingBatchCounts(): RegulatoryMappingBatchCounts {
return {
filteredResultCount: 0,
excludedNonScannerCount: 0,
eligibleFindingCount: 0,
unsupportedFindingCount: 0,
distinctRuleCount: 0,
reusedSucceededRuleCount: 0,
reusedActiveRuleCount: 0,
scheduledRuleCount: 0,
succeededRuleCount: 0,
failedRuleCount: 0
};
}
export function regulatoryMappingBatchRunValidationErrors(run: RegulatoryMappingBatchRun): string[] {
const errors: string[] = [];
for (const value of [run.id, run.projectId, run.scopeKey, run.filterHash, run.idempotencyKey]) {
if (!value?.trim()) errors.push('Regulatory mapping batch identity is incomplete.');
}
if (!/^[a-f0-9]{64}$/.test(run.filterHash)) errors.push('Regulatory mapping batch filterHash must be SHA-256.');
if (!/^[A-Za-z0-9._:-]{8,128}$/.test(run.idempotencyKey)) errors.push('Regulatory mapping batch idempotencyKey is invalid.');
if (!['queued', 'running', 'completed', 'failed'].includes(run.status)) errors.push('Regulatory mapping batch status is invalid.');
const normalized = normalizeRegulatoryMappingBatchScope(run);
if (!normalized.assessmentIds.length && !normalized.scanRunIds.length) errors.push('Regulatory mapping batch requires a selected timeslot.');
const counts = Object.values(run.counts);
if (counts.some(value => !Number.isInteger(value) || value < 0)) errors.push('Regulatory mapping batch counts must be non-negative integers.');
if (run.counts.filteredResultCount !== run.counts.excludedNonScannerCount + run.counts.eligibleFindingCount + run.counts.unsupportedFindingCount) {
errors.push('Regulatory mapping batch result counts are inconsistent.');
}
if (run.counts.distinctRuleCount !== run.counts.reusedSucceededRuleCount + run.counts.reusedActiveRuleCount + run.counts.scheduledRuleCount) {
errors.push('Regulatory mapping batch rule counts are inconsistent.');
}
if (run.counts.succeededRuleCount + run.counts.failedRuleCount > run.counts.scheduledRuleCount) {
errors.push('Regulatory mapping batch completion counts exceed scheduled rules.');
}
if (run.errorDetail !== undefined && (run.errorDetail.length > 600 || /<\/?[a-z][^>]*>/i.test(run.errorDetail))) {
errors.push('Regulatory mapping batch errorDetail must be plain text with at most 600 characters.');
}
return [...new Set(errors)];
}
export function regulatoryMappingDecisionValidationErrors(decision: RegulatoryMappingDecision): string[] { export function regulatoryMappingDecisionValidationErrors(decision: RegulatoryMappingDecision): string[] {
const errors: string[] = []; const errors: string[] = [];
for (const value of [decision.id, decision.ruleKey, decision.sourceTool, decision.mappingId, decision.controlId]) { for (const value of [decision.id, decision.ruleKey, decision.sourceTool, decision.mappingId, decision.controlId]) {

27
src/domain/result-db-query.ts Normal file
View File

@ -0,0 +1,27 @@
import type { RegulatoryMappingBatchFilters } from './regulatory-mapping';
import type { UnifiedResultQuery } from './unified-result';
export function buildScopedResultDbQuery(
filters: UnifiedResultQuery,
projectId: string,
scanRunIds: readonly string[] = [],
assessmentIds: readonly string[] = []
): UnifiedResultQuery {
const { audience: _audience, projectId: _projectId, assessmentId: _assessmentId,
assessmentIds: _assessmentIds, scanRunId: _scanRunId, scanRunIds: _scanRunIds,
offset: _offset, limit: _limit, ...filterValues } = filters;
return {
...filterValues,
audience: 'result-db',
projectId,
assessmentIds: [...assessmentIds],
scanRunIds: [...scanRunIds]
};
}
export function resultDbBatchFilters(query: UnifiedResultQuery): RegulatoryMappingBatchFilters {
const { audience: _audience, projectId: _projectId, assessmentId: _assessmentId,
assessmentIds: _assessmentIds, scanRunId: _scanRunId, scanRunIds: _scanRunIds,
offset: _offset, limit: _limit, ...filters } = query;
return filters;
}

124
src/features/results-browser/RegulatoryMappingBatchDialog.tsx Normal file
View File

@ -0,0 +1,124 @@
import { useCallback, useEffect, useRef, useState } from 'react';
import { AssessorApiClient } from '../../api/assessor-api-client';
import { Dialog } from '../../components/Dialog/Dialog';
import type { RegulatoryMappingBatchResponse, RegulatoryMappingBatchScope } from '../../domain/regulatory-mapping';
const assessorApi = new AssessorApiClient();
export interface RegulatoryMappingBatchController {
response?: RegulatoryMappingBatchResponse;
loading: boolean;
starting: boolean;
error?: string;
start(): Promise<void>;
}
export function useRegulatoryMappingBatch(projectId: string | undefined, scope: RegulatoryMappingBatchScope | undefined): RegulatoryMappingBatchController {
const [response, setResponse] = useState<RegulatoryMappingBatchResponse>();
const [loading, setLoading] = useState(false);
const [starting, setStarting] = useState(false);
const [error, setError] = useState<string>();
const scopeKey = JSON.stringify(scope ?? null);
const requestKey = `${projectId ?? ''}\u0000${scopeKey}`;
const activeRequestKey = useRef(requestKey);
activeRequestKey.current = requestKey;
const refresh = useCallback(async () => {
if (!projectId || !scope || (!scope.assessmentIds.length && !scope.scanRunIds.length)) {
setResponse(undefined);
return;
}
try {
const next = await assessorApi.getRegulatoryMappingBatch(projectId, scope);
if (activeRequestKey.current !== requestKey) return;
setResponse(next);
setError(undefined);
} catch (reason) {
if (activeRequestKey.current !== requestKey) return;
setError(reason instanceof Error ? reason.message : 'ISO mapping batch status could not be loaded.');
}
}, [projectId, requestKey, scopeKey]); // eslint-disable-line react-hooks/exhaustive-deps
useEffect(() => {
let active = true;
setLoading(true);
setStarting(false);
setError(undefined);
setResponse(undefined);
void refresh().finally(() => { if (active) setLoading(false); });
return () => { active = false; };
}, [refresh, scopeKey]);
useEffect(() => {
if (!response || !['queued', 'running'].includes(response.state)) return;
const timer = window.setTimeout(() => void refresh(), response.pollAfterMs ?? 1_500);
return () => window.clearTimeout(timer);
}, [refresh, response]);
const start = useCallback(async () => {
if (!projectId || !scope) return;
setStarting(true);
setError(undefined);
try {
const next = await assessorApi.startRegulatoryMappingBatch(projectId, {
...scope,
idempotencyKey: `regulatory-batch-${crypto.randomUUID()}`
});
if (activeRequestKey.current !== requestKey) return;
setResponse(next);
} catch (reason) {
if (activeRequestKey.current !== requestKey) return;
setError(reason instanceof Error ? reason.message : 'ISO mapping batch could not be started.');
} finally {
if (activeRequestKey.current === requestKey) setStarting(false);
}
}, [projectId, requestKey, scopeKey]); // eslint-disable-line react-hooks/exhaustive-deps
return { response, loading, starting, error, start };
}
export function RegulatoryMappingBatchDialog({ controller, filteredResultCount, onClose }: {
controller: RegulatoryMappingBatchController;
filteredResultCount: number;
onClose(): void;
}) {
const { response, loading, starting, error, start } = controller;
const run = response?.run;
const active = response && ['queued', 'running'].includes(response.state);
const counts = run?.counts;
const actionLabel = run ? 'Run again for current filters' : 'Start mapping batch';
return <Dialog title="ISO mapping batch" onClose={onClose}>
<div className="regulatory-batch-dialog">
<p>Generate ISO control proposals for all <strong>{filteredResultCount.toLocaleString('en-US')}</strong> results matching the current Result DB filters.</p>
<p>Pagination, “Load more,” and loaded-row limits do not restrict this batch. Proposals remain subject to manual Accept/Reject review.</p>
{loading && !response && <p role="status">Loading batch status…</p>}
{error && <p className="regulatory-mapping-error" role="alert">{error}</p>}
{response && !response.generationEnabled && <p role="status">{response.disabledReason ?? 'Regulatory mapping is unavailable.'}</p>}
{run && <>
<p className="regulatory-batch-state" role="status">Status: <strong>{run.status === 'completed' && run.counts.failedRuleCount ? 'completed with failures' : run.status}</strong></p>
<dl className="regulatory-batch-counts">
<BatchCount label="Filtered results" value={counts!.filteredResultCount}/>
<BatchCount label="Excluded non-scanner" value={counts!.excludedNonScannerCount}/>
<BatchCount label="Eligible findings" value={counts!.eligibleFindingCount}/>
<BatchCount label="Unsupported findings" value={counts!.unsupportedFindingCount}/>
<BatchCount label="Distinct rules" value={counts!.distinctRuleCount}/>
<BatchCount label="Reused mappings" value={counts!.reusedSucceededRuleCount + counts!.reusedActiveRuleCount}/>
<BatchCount label="Completed" value={counts!.succeededRuleCount}/>
<BatchCount label="Failed" value={counts!.failedRuleCount}/>
</dl>
{run.errorDetail && <p className="regulatory-mapping-error-detail">{run.errorDetail}</p>}
</>}
{active && <p role="status">Mapping is running on the server. You can close this dialog and return later.</p>}
<div className="dialog-actions">
<button className="btn" type="button" onClick={onClose}>Close</button>
{response?.generationEnabled && <button className="btn primary" type="button"
disabled={Boolean(active) || starting || filteredResultCount === 0}
onClick={() => void start()}>{starting ? 'Starting…' : actionLabel}</button>}
</div>
</div>
</Dialog>;
}
function BatchCount({ label, value }: { label: string; value: number }) {
return <div><dt>{label}</dt><dd>{value.toLocaleString('en-US')}</dd></div>;
}

31
src/features/results-browser/ResultsBrowser.tsx
View File

@ -1,6 +1,6 @@
import { beginActivity, showActivityFrame } from '../../platform/activity-store'; import { beginActivity, showActivityFrame } from '../../platform/activity-store';
import { buildUnifiedResultCsv, unifiedResultCsvFileName } from '../../exports/unified-result-csv'; import { buildUnifiedResultCsv, unifiedResultCsvFileName } from '../../exports/unified-result-csv';
import { useCallback, useEffect, useState } from 'react'; import { useCallback, useEffect, useMemo, useState } from 'react';
import { AssessorApiClient } from '../../api/assessor-api-client'; import { AssessorApiClient } from '../../api/assessor-api-client';
import { resultSourceToolLabel, type UnifiedResultFacetResult, type UnifiedResultProjection, type UnifiedResultQuery, type UnifiedResultQueryResult } from '../../domain/unified-result'; import { resultSourceToolLabel, type UnifiedResultFacetResult, type UnifiedResultProjection, type UnifiedResultQuery, type UnifiedResultQueryResult } from '../../domain/unified-result';
import { discloseResultDbProjection, removeResultDbScopeIdentifiers } from '../../domain/result-db-disclosure'; import { discloseResultDbProjection, removeResultDbScopeIdentifiers } from '../../domain/result-db-disclosure';
@ -17,6 +17,8 @@ import { POLICY_CLASSIFICATION_LABELS } from '../../domain/policy-status-labels'
import { BackendConnectionDialog } from '../../components/BackendConnectionDialog/BackendConnectionDialog'; import { BackendConnectionDialog } from '../../components/BackendConnectionDialog/BackendConnectionDialog';
import { formatPolicyComparisonValue } from '../../profiles/group-policy/presentation'; import { formatPolicyComparisonValue } from '../../profiles/group-policy/presentation';
import type { ResultDbMaterializationStatus } from '../../domain/result-db-materialization'; import type { ResultDbMaterializationStatus } from '../../domain/result-db-materialization';
import { buildScopedResultDbQuery, resultDbBatchFilters } from '../../domain/result-db-query';
import { RegulatoryMappingBatchDialog, useRegulatoryMappingBatch } from './RegulatoryMappingBatchDialog';
const assessorApi = new AssessorApiClient(); const assessorApi = new AssessorApiClient();
const PAGE_SIZE = 50; const PAGE_SIZE = 50;
@ -45,7 +47,17 @@ export function ResultsBrowser({ backendStatus, projectId, scanRunIds, assessmen
const [exporting, setExporting] = useState<'jsonl' | 'excel' | 'csv' | 'json' | null>(null); const [exporting, setExporting] = useState<'jsonl' | 'excel' | 'csv' | 'json' | null>(null);
const [error, setError] = useState<string | null>(null); const [error, setError] = useState<string | null>(null);
const [expandedId, setExpandedId] = useState<string | null>(null); const [expandedId, setExpandedId] = useState<string | null>(null);
const [batchDialogOpen, setBatchDialogOpen] = useState(false);
const hasTimeslotScope = hasSelectedTimeslotScope(scanRunIds, assessmentIds); const hasTimeslotScope = hasSelectedTimeslotScope(scanRunIds, assessmentIds);
const scopedQuery = useMemo(() => projectId && hasTimeslotScope
? buildScopedResultDbQuery(filters, projectId, scanRunIds, assessmentIds)
: undefined, [assessmentIds, filters, hasTimeslotScope, projectId, scanRunIds]);
const batchScope = useMemo(() => scopedQuery ? {
assessmentIds: scopedQuery.assessmentIds ?? [],
scanRunIds: scopedQuery.scanRunIds ?? [],
filters: resultDbBatchFilters(scopedQuery)
} : undefined, [scopedQuery]);
const regulatoryBatch = useRegulatoryMappingBatch(backendStatus === 'connected' ? projectId : undefined, batchScope);
const load = useCallback(async () => { const load = useCallback(async () => {
if (backendStatus !== 'connected' || !projectId || !hasTimeslotScope) { if (backendStatus !== 'connected' || !projectId || !hasTimeslotScope) {
@ -57,12 +69,7 @@ export function ResultsBrowser({ backendStatus, projectId, scanRunIds, assessmen
setLoading(true); setLoading(true);
setError(null); setError(null);
try { try {
const scopedQuery: UnifiedResultQuery = { if (!scopedQuery) return;
...filters,
projectId,
scanRunIds,
assessmentIds
};
const [facets, firstPage] = await Promise.all([ const [facets, firstPage] = await Promise.all([
assessorApi.queryUnifiedResultFacets(scopedQuery), assessorApi.queryUnifiedResultFacets(scopedQuery),
assessorApi.queryUnifiedResults({ ...scopedQuery, offset: 0, limit }) assessorApi.queryUnifiedResults({ ...scopedQuery, offset: 0, limit })
@ -86,7 +93,7 @@ export function ResultsBrowser({ backendStatus, projectId, scanRunIds, assessmen
} finally { } finally {
setLoading(false); setLoading(false);
} }
}, [backendStatus, projectId, scanRunIds, assessmentIds, hasTimeslotScope, filters, limit, loadAll]); }, [backendStatus, projectId, hasTimeslotScope, scopedQuery, limit, loadAll]);
useEffect(() => { void load(); }, [load]); useEffect(() => { void load(); }, [load]);
@ -102,7 +109,7 @@ export function ResultsBrowser({ backendStatus, projectId, scanRunIds, assessmen
setExporting(format); setExporting(format);
setError(null); setError(null);
try { try {
const scopedQuery: UnifiedResultQuery = { ...filters, projectId, scanRunIds, assessmentIds }; if (!scopedQuery) return;
const results = await queryAllUnifiedResults(query => assessorApi.queryUnifiedResults(query), scopedQuery); const results = await queryAllUnifiedResults(query => assessorApi.queryUnifiedResults(query), scopedQuery);
activity.update('Preparing the filtered export file'); activity.update('Preparing the filtered export file');
await showActivityFrame(); await showActivityFrame();
@ -171,6 +178,10 @@ export function ResultsBrowser({ backendStatus, projectId, scanRunIds, assessmen
<p className="db-structure-meta">{(queryResult?.total ?? 0).toLocaleString('en-US')} hits{queryResult && queryResult.total > queryResult.items.length ? ` · ${queryResult.items.length.toLocaleString('en-US')} loaded` : ''}{resultDbStatus && <><br/><span className="results-db-status" data-state={resultDbStatus.state}>{resultDbStatusText(resultDbStatus)}</span></>}</p> <p className="db-structure-meta">{(queryResult?.total ?? 0).toLocaleString('en-US')} hits{queryResult && queryResult.total > queryResult.items.length ? ` · ${queryResult.items.length.toLocaleString('en-US')} loaded` : ''}{resultDbStatus && <><br/><span className="results-db-status" data-state={resultDbStatus.state}>{resultDbStatusText(resultDbStatus)}</span></>}</p>
<div className="page-action-buttons"> <div className="page-action-buttons">
<button className="btn" type="button" disabled={loading} onClick={() => void load()}>{loading ? 'Loading …' : 'Refresh'}</button> <button className="btn" type="button" disabled={loading} onClick={() => void load()}>{loading ? 'Loading …' : 'Refresh'}</button>
<button className="btn" type="button"
disabled={loading || regulatoryBatch.loading || Boolean(regulatoryBatch.error) || !queryResult?.total || regulatoryBatch.response?.generationEnabled === false}
title={regulatoryBatch.error ?? regulatoryBatch.response?.disabledReason ?? 'Generate ISO control proposals for every result matching the current filters'}
onClick={() => setBatchDialogOpen(true)}>Map all ISO controls</button>
<button className="btn primary" type="button" disabled={loading || exporting !== null || !queryResult?.total} <button className="btn primary" type="button" disabled={loading || exporting !== null || !queryResult?.total}
onClick={() => void exportFilteredResults('jsonl')}>{exporting === 'jsonl' ? 'Exporting …' : 'Export JSONL'}</button> onClick={() => void exportFilteredResults('jsonl')}>{exporting === 'jsonl' ? 'Exporting …' : 'Export JSONL'}</button>
<button className="btn primary" type="button" disabled={loading || exporting !== null || !queryResult?.total} <button className="btn primary" type="button" disabled={loading || exporting !== null || !queryResult?.total}
@ -197,6 +208,8 @@ export function ResultsBrowser({ backendStatus, projectId, scanRunIds, assessmen
</div>} </div>}
</div> </div>
</div>} </div>}
{batchDialogOpen && <RegulatoryMappingBatchDialog controller={regulatoryBatch}
filteredResultCount={queryResult?.total ?? 0} onClose={() => setBatchDialogOpen(false)}/>}
</div>; </div>;
} }

7
src/persistence/assessor-repository.ts
View File

@ -35,7 +35,7 @@ import type {
} from '../domain/unified-result'; } from '../domain/unified-result';
import type { EnvironmentObservation, EnvironmentSnapshot, EnvironmentSnapshotQuery, ScannerReportSection } from '../domain/environment-context'; import type { EnvironmentObservation, EnvironmentSnapshot, EnvironmentSnapshotQuery, ScannerReportSection } from '../domain/environment-context';
import type { AiOverviewRun } from '../domain/ai-overview'; import type { AiOverviewRun } from '../domain/ai-overview';
import type { RegulatoryMappingDecision, RegulatoryMappingRun } from '../domain/regulatory-mapping'; import type { RegulatoryMappingBatchRun, RegulatoryMappingDecision, RegulatoryMappingRun } from '../domain/regulatory-mapping';
export interface FindingOccurrenceQuery { export interface FindingOccurrenceQuery {
projectId?: string; projectId?: string;
@ -147,6 +147,7 @@ export interface ProjectDataExport {
environmentProfiles: EnvironmentProfile[]; environmentProfiles: EnvironmentProfile[];
policySnapshotAggregations: StoredPolicySnapshotAggregation[]; policySnapshotAggregations: StoredPolicySnapshotAggregation[];
aiOverviewRuns: AiOverviewRun[]; aiOverviewRuns: AiOverviewRun[];
regulatoryMappingBatchRuns: RegulatoryMappingBatchRun[];
regulatoryMappingRuns: RegulatoryMappingRun[]; regulatoryMappingRuns: RegulatoryMappingRun[];
regulatoryMappingDecisions: RegulatoryMappingDecision[]; regulatoryMappingDecisions: RegulatoryMappingDecision[];
} }
@ -226,8 +227,12 @@ export interface AssessorRepository {
updateAiOverviewRun(run: AiOverviewRun): Promise<AiOverviewRun>; updateAiOverviewRun(run: AiOverviewRun): Promise<AiOverviewRun>;
listAiOverviewRuns(projectId: string, scopeKey?: string): Promise<AiOverviewRun[]>; listAiOverviewRuns(projectId: string, scopeKey?: string): Promise<AiOverviewRun[]>;
getRegulatoryMappingRun(id: string): Promise<RegulatoryMappingRun | null>; getRegulatoryMappingRun(id: string): Promise<RegulatoryMappingRun | null>;
claimRegulatoryMappingRun(run: RegulatoryMappingRun, forceRegeneration: boolean): Promise<{ run: RegulatoryMappingRun; claimed: boolean }>;
saveRegulatoryMappingRun(run: RegulatoryMappingRun): Promise<RegulatoryMappingRun>; saveRegulatoryMappingRun(run: RegulatoryMappingRun): Promise<RegulatoryMappingRun>;
listRegulatoryMappingRuns(): Promise<RegulatoryMappingRun[]>; listRegulatoryMappingRuns(): Promise<RegulatoryMappingRun[]>;
createRegulatoryMappingBatchRun(run: RegulatoryMappingBatchRun): Promise<RegulatoryMappingBatchRun>;
updateRegulatoryMappingBatchRun(run: RegulatoryMappingBatchRun): Promise<RegulatoryMappingBatchRun>;
listRegulatoryMappingBatchRuns(projectId: string, scopeKey?: string): Promise<RegulatoryMappingBatchRun[]>;
createRegulatoryMappingDecision(decision: RegulatoryMappingDecision, expectedLatestDecisionId?: string): Promise<RegulatoryMappingDecision>; createRegulatoryMappingDecision(decision: RegulatoryMappingDecision, expectedLatestDecisionId?: string): Promise<RegulatoryMappingDecision>;
listRegulatoryMappingDecisions(ruleKey: string): Promise<RegulatoryMappingDecision[]>; listRegulatoryMappingDecisions(ruleKey: string): Promise<RegulatoryMappingDecision[]>;

55
src/persistence/local-assessor-repository.ts
View File

@ -27,8 +27,8 @@ import type { EnvironmentObservation, EnvironmentSnapshot, EnvironmentSnapshotQu
import { environmentCoverage } from '../domain/environment-context'; import { environmentCoverage } from '../domain/environment-context';
import type { AiOverviewRun } from '../domain/ai-overview'; import type { AiOverviewRun } from '../domain/ai-overview';
import { aiOverviewRunValidationErrors } from '../domain/ai-overview'; import { aiOverviewRunValidationErrors } from '../domain/ai-overview';
import type { RegulatoryMappingDecision, RegulatoryMappingRun } from '../domain/regulatory-mapping'; import type { RegulatoryMappingBatchRun, RegulatoryMappingDecision, RegulatoryMappingRun } from '../domain/regulatory-mapping';
import { regulatoryMappingDecisionValidationErrors, regulatoryMappingRunValidationErrors } from '../domain/regulatory-mapping'; import { regulatoryMappingBatchRunValidationErrors, regulatoryMappingDecisionValidationErrors, regulatoryMappingRunValidationErrors } from '../domain/regulatory-mapping';
import { storedPolicyFindingRunValidationErrors } from '../domain/policy-finding'; import { storedPolicyFindingRunValidationErrors } from '../domain/policy-finding';
import { import {
environmentProfileValidationErrors, environmentProfileValidationErrors,
@ -105,6 +105,7 @@ interface LocalRepositoryState {
findingDefinitions: FindingDefinition[]; findingDefinitions: FindingDefinition[];
findingOccurrences: FindingOccurrence[]; findingOccurrences: FindingOccurrence[];
aiOverviewRuns: AiOverviewRun[]; aiOverviewRuns: AiOverviewRun[];
regulatoryMappingBatchRuns: RegulatoryMappingBatchRun[];
regulatoryMappingRuns: RegulatoryMappingRun[]; regulatoryMappingRuns: RegulatoryMappingRun[];
regulatoryMappingDecisions: RegulatoryMappingDecision[]; regulatoryMappingDecisions: RegulatoryMappingDecision[];
evidence: EvidenceRecord[]; evidence: EvidenceRecord[];
@ -142,6 +143,7 @@ const emptyState = (): LocalRepositoryState => ({
findingDefinitions: [], findingDefinitions: [],
findingOccurrences: [], findingOccurrences: [],
aiOverviewRuns: [], aiOverviewRuns: [],
regulatoryMappingBatchRuns: [],
regulatoryMappingRuns: [], regulatoryMappingRuns: [],
regulatoryMappingDecisions: [], regulatoryMappingDecisions: [],
evidence: [], evidence: [],
@ -437,6 +439,17 @@ export class LocalAssessorRepository implements AssessorRepository {
return run ? clone(run) : null; return run ? clone(run) : null;
} }
async claimRegulatoryMappingRun(run: RegulatoryMappingRun, forceRegeneration: boolean): Promise<{ run: RegulatoryMappingRun; claimed: boolean }> {
const errors = regulatoryMappingRunValidationErrors(run);
if (errors.length) throw new RepositoryValidationError(errors.join(' '));
const existing = this.state.regulatoryMappingRuns.find(item => item.id === run.id);
if (existing && (['queued', 'running'].includes(existing.status) || existing.status === 'succeeded' && !forceRegeneration)) {
return { run: clone(existing), claimed: false };
}
const stored = await this.saveRegulatoryMappingRun(run);
return { run: stored, claimed: true };
}
async saveRegulatoryMappingRun(run: RegulatoryMappingRun): Promise<RegulatoryMappingRun> { async saveRegulatoryMappingRun(run: RegulatoryMappingRun): Promise<RegulatoryMappingRun> {
const errors = regulatoryMappingRunValidationErrors(run); const errors = regulatoryMappingRunValidationErrors(run);
if (errors.length) throw new RepositoryValidationError(errors.join(' ')); if (errors.length) throw new RepositoryValidationError(errors.join(' '));
@ -453,6 +466,42 @@ export class LocalAssessorRepository implements AssessorRepository {
.sort((left, right) => right.createdAt.localeCompare(left.createdAt) || right.id.localeCompare(left.id)); .sort((left, right) => right.createdAt.localeCompare(left.createdAt) || right.id.localeCompare(left.id));
} }
async createRegulatoryMappingBatchRun(run: RegulatoryMappingBatchRun): Promise<RegulatoryMappingBatchRun> {
const errors = regulatoryMappingBatchRunValidationErrors(run);
if (errors.length) throw new RepositoryValidationError(errors.join(' '));
const idempotent = this.state.regulatoryMappingBatchRuns.find(item => item.projectId === run.projectId && item.idempotencyKey === run.idempotencyKey);
if (idempotent) return clone(idempotent);
const active = this.state.regulatoryMappingBatchRuns.find(item => item.projectId === run.projectId && item.scopeKey === run.scopeKey
&& ['queued', 'running'].includes(item.status));
if (active) return clone(active);
const next = clone(this.state);
next.regulatoryMappingBatchRuns.push(clone(run));
this.replaceState(next);
return clone(run);
}
async updateRegulatoryMappingBatchRun(run: RegulatoryMappingBatchRun): Promise<RegulatoryMappingBatchRun> {
const errors = regulatoryMappingBatchRunValidationErrors(run);
if (errors.length) throw new RepositoryValidationError(errors.join(' '));
const existing = this.state.regulatoryMappingBatchRuns.find(item => item.id === run.id);
if (!existing) throw new RepositoryNotFoundError('Regulatory mapping batch does not exist.');
if (existing.projectId !== run.projectId || existing.scopeKey !== run.scopeKey || existing.idempotencyKey !== run.idempotencyKey) {
throw new RepositoryConflictError('Regulatory mapping batch identity is immutable.');
}
if (['completed', 'failed'].includes(existing.status)) throw new RepositoryConflictError('A completed regulatory mapping batch is immutable.');
const allowed = existing.status === 'queued' ? ['running', 'failed'] : ['running', 'completed', 'failed'];
if (!allowed.includes(run.status)) throw new RepositoryConflictError(`Invalid regulatory mapping batch transition from ${existing.status} to ${run.status}.`);
const next = clone(this.state);
next.regulatoryMappingBatchRuns[next.regulatoryMappingBatchRuns.findIndex(item => item.id === run.id)] = clone(run);
this.replaceState(next);
return clone(run);
}
async listRegulatoryMappingBatchRuns(projectId: string, scopeKey?: string): Promise<RegulatoryMappingBatchRun[]> {
return this.state.regulatoryMappingBatchRuns.filter(item => item.projectId === projectId && (!scopeKey || item.scopeKey === scopeKey))
.map(clone).reverse();
}
async createRegulatoryMappingDecision(decision: RegulatoryMappingDecision, expectedLatestDecisionId?: string): Promise<RegulatoryMappingDecision> { async createRegulatoryMappingDecision(decision: RegulatoryMappingDecision, expectedLatestDecisionId?: string): Promise<RegulatoryMappingDecision> {
const errors = regulatoryMappingDecisionValidationErrors(decision); const errors = regulatoryMappingDecisionValidationErrors(decision);
if (errors.length) throw new RepositoryValidationError(errors.join(' ')); if (errors.length) throw new RepositoryValidationError(errors.join(' '));
@ -1591,6 +1640,7 @@ export class LocalAssessorRepository implements AssessorRepository {
policyEvaluations: this.state.policyEvaluations.filter(item => item.run.projectId === projectId).map(clone), policyEvaluations: this.state.policyEvaluations.filter(item => item.run.projectId === projectId).map(clone),
policyFindingRuns: policyFindingRuns.map(clone), policyFindingRuns: policyFindingRuns.map(clone),
aiOverviewRuns: this.state.aiOverviewRuns.filter(item => item.projectId === projectId && item.status === 'succeeded').map(clone), aiOverviewRuns: this.state.aiOverviewRuns.filter(item => item.projectId === projectId && item.status === 'succeeded').map(clone),
regulatoryMappingBatchRuns: this.state.regulatoryMappingBatchRuns.filter(item => item.projectId === projectId).map(clone),
// Regulatory mappings and decisions are global reference data, not project-scoped. // Regulatory mappings and decisions are global reference data, not project-scoped.
regulatoryMappingRuns: this.state.regulatoryMappingRuns.filter(item => item.status === 'succeeded').map(clone), regulatoryMappingRuns: this.state.regulatoryMappingRuns.filter(item => item.status === 'succeeded').map(clone),
regulatoryMappingDecisions: this.state.regulatoryMappingDecisions.map(clone) regulatoryMappingDecisions: this.state.regulatoryMappingDecisions.map(clone)
@ -1781,6 +1831,7 @@ export class LocalAssessorRepository implements AssessorRepository {
next.policySnapshotAggregations = next.policySnapshotAggregations.filter(item => item.run.projectId !== projectId); next.policySnapshotAggregations = next.policySnapshotAggregations.filter(item => item.run.projectId !== projectId);
next.policyEvaluations = next.policyEvaluations.filter(item => item.run.projectId !== projectId); next.policyEvaluations = next.policyEvaluations.filter(item => item.run.projectId !== projectId);
next.aiOverviewRuns = next.aiOverviewRuns.filter(item => item.projectId !== projectId); next.aiOverviewRuns = next.aiOverviewRuns.filter(item => item.projectId !== projectId);
next.regulatoryMappingBatchRuns = next.regulatoryMappingBatchRuns.filter(item => item.projectId !== projectId);
// Regulatory mappings and decisions are global and survive project deletion. // Regulatory mappings and decisions are global and survive project deletion.
this.replaceState(next); this.replaceState(next);
return true; return true;

9
src/styles/datahub.css
View File

@ -958,12 +958,21 @@ button.client-overall-ring:focus-visible { outline: var(--bw-emphasis) solid var
.results-environment-value p { margin: var(--legacy-space-1) 0 var(--legacy-space-3); overflow-wrap: anywhere; white-space: pre-wrap; } .results-environment-value p { margin: var(--legacy-space-1) 0 var(--legacy-space-3); overflow-wrap: anywhere; white-space: pre-wrap; }
.results-processing-alert { margin-bottom: var(--legacy-space-3); } .results-processing-alert { margin-bottom: var(--legacy-space-3); }
.results-browser-load-actions { display: flex; gap: var(--legacy-space-2); margin-top: var(--legacy-space-3); } .results-browser-load-actions { display: flex; gap: var(--legacy-space-2); margin-top: var(--legacy-space-3); }
.regulatory-batch-dialog { display: grid; gap: var(--legacy-space-4); }
.regulatory-batch-dialog p { margin: 0; font-size: var(--ui-reading-size); line-height: 1.5; }
.regulatory-batch-state { padding-top: var(--legacy-space-3); border-top: var(--border-hair) solid var(--line); }
.regulatory-batch-counts { display: grid; grid-template-columns: repeat(2, minmax(0, 1fr)); gap: var(--legacy-space-2); margin: 0; }
.regulatory-batch-counts > div { display: flex; align-items: center; justify-content: space-between; gap: var(--legacy-space-3); min-width: 0; padding: var(--legacy-space-2) var(--legacy-space-3); background: var(--panel-soft); }
.regulatory-batch-counts dt { color: var(--muted); font-size: var(--ui-label-size); font-weight: var(--ui-label-weight); }
.regulatory-batch-counts dd { margin: 0; color: var(--heading); font-variant-numeric: tabular-nums; font-weight: var(--ui-strong-weight); }
.regulatory-batch-dialog .dialog-actions { display: flex; justify-content: flex-end; gap: var(--legacy-space-2); padding-top: var(--legacy-space-2); }
@media (max-width: 1150px) { @media (max-width: 1150px) {
.results-browser-layout { grid-template-columns: 1fr; } .results-browser-layout { grid-template-columns: 1fr; }
.results-filter-panel { position: static; max-height: none; } .results-filter-panel { position: static; max-height: none; }
.results-browser-row { grid-template-columns: 1fr 1fr; gap: var(--legacy-space-2) var(--legacy-space-4); } .results-browser-row { grid-template-columns: 1fr 1fr; gap: var(--legacy-space-2) var(--legacy-space-4); }
.results-browser-row span::before { content: attr(data-label); display: block; margin-bottom: 2px; color: var(--muted); font-size: var(--ui-label-size); font-weight: var(--ui-label-weight); letter-spacing: var(--ui-label-tracking); text-transform: uppercase; } .results-browser-row span::before { content: attr(data-label); display: block; margin-bottom: 2px; color: var(--muted); font-size: var(--ui-label-size); font-weight: var(--ui-label-weight); letter-spacing: var(--ui-label-tracking); text-transform: uppercase; }
.results-browser-head { display: none; } .results-browser-head { display: none; }
.regulatory-batch-counts { grid-template-columns: 1fr; }
} }
.datahub-mitre-management { .datahub-mitre-management {

16
tests/api/assessor-api-client.test.ts
View File

@ -113,6 +113,22 @@ describe('AssessorApiClient', () => {
expect(new Headers(fetcher.mock.calls[1][1]?.headers).get('X-Assessor-Request')).toBe('1'); expect(new Headers(fetcher.mock.calls[1][1]?.headers).get('X-Assessor-Request')).toBe('1');
expect(new Headers(fetcher.mock.calls[2][1]?.headers).get('X-Assessor-Request')).toBe('1'); expect(new Headers(fetcher.mock.calls[2][1]?.headers).get('X-Assessor-Request')).toBe('1');
}); });
it('uses the complete filtered Result DB scope for regulatory mapping batches', async () => {
const fetcher = vi.fn<typeof fetch>().mockResolvedValue(new Response(JSON.stringify({
version: 1, projectId: 'project/one', scope: { assessmentIds: ['assessment/one'], scanRunIds: ['run/one'], filters: { deviation: true } },
generationEnabled: true, state: 'empty'
}), { status: 200, headers: { 'Content-Type': 'application/json' } }));
const client = new AssessorApiClient({ fetcher });
const scope = { assessmentIds: ['assessment/one'], scanRunIds: ['run/one'], filters: { deviation: true, severity: ['high' as const] } };
await client.getRegulatoryMappingBatch('project/one', scope);
await client.startRegulatoryMappingBatch('project/one', { ...scope, idempotencyKey: 'batch-request-1' });
expect(fetcher.mock.calls[0][0]).toBe('/api/v1/projects/project%2Fone/regulatory-mapping-batches?deviation=true&severity=high&assessmentIds=assessment%2Fone&scanRunIds=run%2Fone');
expect(fetcher.mock.calls[1][0]).toBe('/api/v1/projects/project%2Fone/regulatory-mapping-batches');
expect(new Headers(fetcher.mock.calls[0][1]?.headers).has('X-Assessor-Request')).toBe(false);
expect(new Headers(fetcher.mock.calls[1][1]?.headers).get('X-Assessor-Request')).toBe('1');
});
it('submits GPO backups only through the project-bound mutation endpoint', async () => { it('submits GPO backups only through the project-bound mutation endpoint', async () => {
const fetcher = vi.fn<typeof fetch>().mockResolvedValue(new Response(JSON.stringify({ const fetcher = vi.fn<typeof fetch>().mockResolvedValue(new Response(JSON.stringify({
version: 1, artifactId: 'artifact-1', jobId: 'job-1', sha256: 'a'.repeat(64), sizeBytes: 3, version: 1, artifactId: 'artifact-1', jobId: 'job-1', sha256: 'a'.repeat(64), sizeBytes: 3,

50
tests/domain/result-db-query.test.ts Normal file
View File

@ -0,0 +1,50 @@
import { describe, expect, it } from 'vitest';
import { buildScopedResultDbQuery, resultDbBatchFilters } from '../../src/domain/result-db-query';
import type { UnifiedResultQuery } from '../../src/domain/unified-result';
describe('Result DB scoped query', () => {
it('uses one pagination-free query for export and regulatory batch filters', () => {
const filters: UnifiedResultQuery = {
evaluationRunId: 'evaluation-1',
sourceKind: ['scanner-finding'],
deviation: true,
isIoe: false,
severity: ['high'],
outcome: ['fail'],
technicalExecutionState: ['completed'],
resultType: ['compliance_check'],
policyClassification: ['misconfigured'],
sourceTool: ['purple-knight'],
mappingRuleVersion: ['mapping-1'],
sourceProfileVersion: ['profile-1'],
processingStatus: ['current'],
reviewDisposition: ['accepted'],
lifecycleState: ['new'],
categoryNamespace: ['framework'],
categoryValue: ['iso'],
affectedAsset: 'synthetic asset',
environmentState: ['observed'],
environmentProperty: ['domain.count'],
environmentSubjectType: ['domain'],
search: 'synthetic',
observedFrom: '2026-09-01T00:00:00.000Z',
observedTo: '2026-09-16T00:00:00.000Z'
};
const query = buildScopedResultDbQuery({
...filters,
offset: 100,
limit: 50,
projectId: 'untrusted-project',
scanRunIds: ['untrusted-run']
}, 'project-1', ['run-1'], ['assessment-1']);
expect(query).toEqual({
audience: 'result-db',
projectId: 'project-1',
assessmentIds: ['assessment-1'],
scanRunIds: ['run-1'],
...filters
});
expect(resultDbBatchFilters(query)).toEqual(filters);
});
});

66
tests/features/regulatory-mapping-batch.test.tsx Normal file
View File

@ -0,0 +1,66 @@
// @vitest-environment jsdom
import { readFileSync } from 'node:fs';
import { resolve } from 'node:path';
import { act } from 'react';
import { createRoot } from 'react-dom/client';
import { describe, expect, it, vi } from 'vitest';
import { RegulatoryMappingBatchDialog } from '../../src/features/results-browser/RegulatoryMappingBatchDialog';
(globalThis as typeof globalThis & { IS_REACT_ACT_ENVIRONMENT: boolean }).IS_REACT_ACT_ENVIRONMENT = true;
describe('Result DB regulatory mapping batch', () => {
it('places the action with the Result DB page actions and shares its scoped query with export', () => {
const source = readFileSync(resolve(process.cwd(), 'src/features/results-browser/ResultsBrowser.tsx'), 'utf8');
expect(source.indexOf('Map all ISO controls')).toBeGreaterThan(source.indexOf("'Refresh'"));
expect(source.indexOf('Map all ISO controls')).toBeLessThan(source.indexOf("'Exporting …'"));
expect(source).toContain('buildScopedResultDbQuery');
expect(source).toContain('queryAllUnifiedResults');
expect(source).toContain('resultDbBatchFilters(scopedQuery)');
});
it('explains complete filtered membership and displays durable progress counts', async () => {
const container = document.createElement('div');
document.body.appendChild(container);
const root = createRoot(container);
await act(async () => root.render(<RegulatoryMappingBatchDialog filteredResultCount={75} onClose={() => {}} controller={{
loading: false,
starting: false,
start: vi.fn(),
response: {
version: 1,
projectId: 'project-1',
scope: { assessmentIds: ['assessment-1'], scanRunIds: ['run-1'], filters: { deviation: true } },
generationEnabled: true,
state: 'running',
pollAfterMs: 1_500,
run: {
version: 1,
id: 'batch-1',
projectId: 'project-1',
scopeKey: 'scope-1',
filterHash: 'a'.repeat(64),
idempotencyKey: 'batch-request-1',
assessmentIds: ['assessment-1'],
scanRunIds: ['run-1'],
filters: { deviation: true },
status: 'running',
counts: {
filteredResultCount: 75, excludedNonScannerCount: 5, eligibleFindingCount: 68, unsupportedFindingCount: 2,
distinctRuleCount: 20, reusedSucceededRuleCount: 8, reusedActiveRuleCount: 1, scheduledRuleCount: 11,
succeededRuleCount: 4, failedRuleCount: 1
},
createdAt: '2026-09-16T08:00:00.000Z',
updatedAt: '2026-09-16T08:01:00.000Z'
}
}
}}/>));
const text = document.body.textContent ?? '';
expect(text).toContain('all 75 results matching the current Result DB filters');
expect(text).toContain('Pagination, “Load more,” and loaded-row limits do not restrict this batch.');
expect(text).toContain('Distinct rules20');
expect(text).toContain('Completed4');
expect(text).toContain('Failed1');
await act(async () => root.unmount());
container.remove();
});
});

50
tests/persistence/regulatory-mapping-batch.test.ts Normal file
View File

@ -0,0 +1,50 @@
import { describe, expect, it } from 'vitest';
import type { RegulatoryMappingBatchRun } from '../../src/domain/regulatory-mapping';
import { LocalAssessorRepository } from '../../src/persistence/local-assessor-repository';
const stamp = '2026-09-16T08:00:00.000Z';
function batch(overrides: Partial<RegulatoryMappingBatchRun> = {}): RegulatoryMappingBatchRun {
return {
version: 1,
id: 'batch-1',
projectId: 'project-1',
scopeKey: 'scope-1',
filterHash: 'a'.repeat(64),
idempotencyKey: 'batch-request-1',
assessmentIds: ['assessment-1'],
scanRunIds: [],
filters: { deviation: true },
status: 'queued',
counts: {
filteredResultCount: 0, excludedNonScannerCount: 0, eligibleFindingCount: 0, unsupportedFindingCount: 0,
distinctRuleCount: 0, reusedSucceededRuleCount: 0, reusedActiveRuleCount: 0, scheduledRuleCount: 0,
succeededRuleCount: 0, failedRuleCount: 0
},
createdAt: stamp,
updatedAt: stamp,
...overrides
};
}
describe('local regulatory mapping batch persistence', () => {
it('preserves idempotency and active-scope uniqueness, exports batches and deletes them with the project', async () => {
const repository = new LocalAssessorRepository();
await repository.saveProject({ id: 'project-1', name: 'Synthetic', createdAt: stamp, updatedAt: stamp, attributes: {} });
await repository.saveAssessment({ id: 'assessment-1', projectId: 'project-1', name: 'Synthetic', createdAt: stamp, updatedAt: stamp });
const first = await repository.createRegulatoryMappingBatchRun(batch());
await expect(repository.createRegulatoryMappingBatchRun(batch({ id: 'batch-idempotent' }))).resolves.toEqual(first);
await expect(repository.createRegulatoryMappingBatchRun(batch({ id: 'batch-active', idempotencyKey: 'batch-request-2' }))).resolves.toEqual(first);
expect((await repository.exportProject('project-1')).regulatoryMappingBatchRuns).toEqual([first]);
const running = await repository.updateRegulatoryMappingBatchRun({ ...first, status: 'running', startedAt: stamp });
await repository.updateRegulatoryMappingBatchRun({ ...running, status: 'completed', completedAt: stamp });
const retry = await repository.createRegulatoryMappingBatchRun(batch({ id: 'batch-2', idempotencyKey: 'batch-request-2' }));
expect((await repository.listRegulatoryMappingBatchRuns('project-1', 'scope-1')).map(item => item.id)).toEqual(['batch-2', 'batch-1']);
expect(retry.id).toBe('batch-2');
await repository.deleteProject('project-1', { projectId: 'project-1', cascade: true });
await expect(repository.listRegulatoryMappingBatchRuns('project-1')).resolves.toEqual([]);
});
});

16
tests/postgres/regulatory-mapping-batch-schema.test.ts Normal file
View File

@ -0,0 +1,16 @@
import { readFileSync } from 'node:fs';
import { describe, expect, it } from 'vitest';
const sql = readFileSync(new URL('../../server/postgres/migrations/026_regulatory_mapping.sql', import.meta.url), 'utf8');
describe('regulatory mapping batch persistence schema', () => {
it('keeps orchestration project-scoped and protects idempotency and active scope', () => {
expect(sql).toContain('CREATE TABLE assessor_regulatory_mapping_batches');
expect(sql).toContain('project_id text NOT NULL REFERENCES assessor_projects(id) ON DELETE CASCADE');
expect(sql).toContain('filter_hash text NOT NULL');
expect(sql).toContain('UNIQUE (project_id, idempotency_key)');
expect(sql).toMatch(/UNIQUE INDEX assessor_regulatory_mapping_batches_active_scope_uq[\s\S]*WHERE run_status IN \('queued', 'running'\)/);
expect(sql).toContain('(project_id, scope_key, created_at DESC, id DESC)');
expect(sql).not.toContain('source_result_id');
});
});

197
tests/server/regulatory-mapping-service.test.ts
View File

@ -1,9 +1,9 @@
import { describe, expect, it, vi } from 'vitest'; import { describe, expect, it, vi } from 'vitest';
import { RegulatoryMappingService } from '../../server/regulatory/regulatory-mapping-service'; import { RegulatoryMappingService } from '../../server/regulatory/regulatory-mapping-service';
import { RegulatoryFindingRetrievalService } from '../../server/regulatory/regulatory-finding-retrieval'; import { RegulatoryFindingRetrievalService } from '../../server/regulatory/regulatory-finding-retrieval';
import type { RegulatoryMappingDecision, RegulatoryMappingRun } from '../../src/domain/regulatory-mapping'; import type { RegulatoryMappingBatchRun, RegulatoryMappingDecision, RegulatoryMappingRun } from '../../src/domain/regulatory-mapping';
import type { RegulatoryChunk, RegulatoryRetrievalRequest } from '../../src/domain/regulatory-rag'; import type { RegulatoryChunk, RegulatoryRetrievalRequest } from '../../src/domain/regulatory-rag';
import type { UnifiedResultProjection } from '../../src/domain/unified-result'; import { filterUnifiedResults, type UnifiedResultProjection } from '../../src/domain/unified-result';
import type { AssessorRepository } from '../../src/persistence/assessor-repository'; import type { AssessorRepository } from '../../src/persistence/assessor-repository';
const stamp = '2026-09-15T10:00:00.000Z'; const stamp = '2026-09-15T10:00:00.000Z';
@ -40,12 +40,48 @@ function chunk(sourceStandard: 'ISO/IEC 27001' | 'ISO/IEC 27002'): RegulatoryChu
function mockRepository(findings: UnifiedResultProjection[]) { function mockRepository(findings: UnifiedResultProjection[]) {
const byProjection = new Map(findings.map(item => [item.resultProjectionId, item])); const byProjection = new Map(findings.map(item => [item.resultProjectionId, item]));
const runs = new Map<string, RegulatoryMappingRun>(); const runs = new Map<string, RegulatoryMappingRun>();
const batches: RegulatoryMappingBatchRun[] = [];
const decisions: RegulatoryMappingDecision[] = []; const decisions: RegulatoryMappingDecision[] = [];
return { return {
getProject: vi.fn(async (projectId: string) => findings.some(item => item.projectId === projectId) ? { id: projectId, name: 'Synthetic project' } : null),
listProjects: vi.fn(async () => [...new Set(findings.map(item => item.projectId))].map(id => ({ id, name: 'Synthetic project' }))),
listAssessments: vi.fn(async (projectId: string) => [...new Set(findings.filter(item => item.projectId === projectId).map(item => item.assessmentId).filter(Boolean))]
.map(id => ({ id: id!, projectId, label: 'Synthetic assessment', createdAt: stamp }))),
listProjectScanRuns: vi.fn(async (projectId: string) => [...new Map(findings.filter(item => item.projectId === projectId && item.scanRunId)
.map(item => [item.scanRunId!, { id: item.scanRunId!, assessmentId: item.assessmentId!, sequence: 1, startedAt: stamp }])).values()]),
queryUnifiedResults: vi.fn(async (query = {}) => {
const filtered = filterUnifiedResults(findings, query);
const offset = query.offset ?? 0;
return { total: filtered.length, items: filtered.slice(offset, offset + (query.limit ?? filtered.length)) };
}),
getUnifiedResultById: vi.fn(async (id: string) => byProjection.get(id) ?? null), getUnifiedResultById: vi.fn(async (id: string) => byProjection.get(id) ?? null),
getRegulatoryMappingRun: vi.fn(async (id: string) => runs.get(id) ?? null), getRegulatoryMappingRun: vi.fn(async (id: string) => runs.get(id) ?? null),
claimRegulatoryMappingRun: vi.fn(async (run: RegulatoryMappingRun, force: boolean) => {
const existing = runs.get(run.id);
if (existing && (['queued', 'running'].includes(existing.status) || existing.status === 'succeeded' && !force)) {
return { run: existing, claimed: false };
}
runs.set(run.id, run);
return { run, claimed: true };
}),
saveRegulatoryMappingRun: vi.fn(async (run: RegulatoryMappingRun) => { runs.set(run.id, run); return run; }), saveRegulatoryMappingRun: vi.fn(async (run: RegulatoryMappingRun) => { runs.set(run.id, run); return run; }),
listRegulatoryMappingRuns: vi.fn(async () => [...runs.values()]), listRegulatoryMappingRuns: vi.fn(async () => [...runs.values()]),
createRegulatoryMappingBatchRun: vi.fn(async (run: RegulatoryMappingBatchRun) => {
const existing = batches.find(item => item.projectId === run.projectId && (item.idempotencyKey === run.idempotencyKey
|| item.scopeKey === run.scopeKey && ['queued', 'running'].includes(item.status)));
if (existing) return existing;
batches.push(run);
return run;
}),
updateRegulatoryMappingBatchRun: vi.fn(async (run: RegulatoryMappingBatchRun) => {
const index = batches.findIndex(item => item.id === run.id);
if (index < 0) throw new Error('missing batch');
batches[index] = run;
return run;
}),
listRegulatoryMappingBatchRuns: vi.fn(async (projectId: string, scopeKey?: string) => batches
.filter(item => item.projectId === projectId && (!scopeKey || item.scopeKey === scopeKey))
.slice().reverse()),
createRegulatoryMappingDecision: vi.fn(async (decision: RegulatoryMappingDecision, expected?: string) => { createRegulatoryMappingDecision: vi.fn(async (decision: RegulatoryMappingDecision, expected?: string) => {
const run = runs.get(decision.mappingId); const run = runs.get(decision.mappingId);
if (!run || run.ruleKey !== decision.ruleKey || run.status !== 'succeeded') throw new Error('missing mapping'); if (!run || run.ruleKey !== decision.ruleKey || run.status !== 'succeeded') throw new Error('missing mapping');
@ -127,6 +163,163 @@ describe('regulatory mapping service', () => {
expect(completed.run?.errorDetail).toContain('unknown candidate control'); expect(completed.run?.errorDetail).toContain('unknown candidate control');
expect(completed.run?.errorDetail).not.toContain(resultTableAsset); expect(completed.run?.errorDetail).not.toContain(resultTableAsset);
}); });
it('maps the complete filtered timeslot, excludes non-scanner results and deduplicates global rules', async () => {
const findings = [
finding(),
finding({ resultProjectionId: 'scanner-finding:occurrence-2', sourceResultId: 'occurrence-2' }),
finding({ resultProjectionId: 'scanner-finding:occurrence-3', sourceResultId: 'occurrence-3', sourceRuleId: 'A-Second', title: 'Second synthetic check' }),
finding({ resultProjectionId: 'policy-evaluation:one', sourceResultId: 'policy-one', sourceKind: 'policy-evaluation', sourceRuleId: undefined }),
finding({ resultProjectionId: 'scanner-finding:unsupported', sourceResultId: 'unsupported', sourceRuleId: 'A-Unsupported', title: '' }),
finding({ resultProjectionId: 'scanner-finding:not-deviation', sourceResultId: 'not-deviation', sourceRuleId: 'A-Ignored', isDeviation: false })
];
const repository = mockRepository(findings);
const complete = vi.fn(async (_system: string, _input: string) => ({ value: {
schemaVersion: 1, outcome: 'mapped', mappings: [{ controlId: '5.1', relationship: 'primary', confidence: 'high',
rationale: 'Synthetic rationale.', evidenceChunkIds: ['semantic-5.1', 'reference-5.1'] }]
} }));
const svc = service(repository, complete);
await svc.startBatch('project-1', {
assessmentIds: ['assessment-1'], scanRunIds: ['run-1'], filters: { deviation: true, affectedAsset: resultTableAsset },
idempotencyKey: 'batch-filtered-1'
});
await svc.close();
const response = await svc.getBatch('project-1', {
assessmentIds: ['assessment-1'], scanRunIds: ['run-1'], filters: { deviation: true, affectedAsset: resultTableAsset }
});
expect(response.run).toMatchObject({ status: 'completed', counts: {
filteredResultCount: 5,
excludedNonScannerCount: 1,
eligibleFindingCount: 3,
unsupportedFindingCount: 1,
distinctRuleCount: 2,
scheduledRuleCount: 2,
succeededRuleCount: 2,
failedRuleCount: 0
} });
expect(complete).toHaveBeenCalledTimes(2);
expect(complete.mock.calls.every(call => !String(call[1]).includes(resultTableAsset))).toBe(true);
});
it('bounds concurrent mapping completions across a batch', async () => {
const findings = Array.from({ length: 5 }, (_, index) => finding({
resultProjectionId: `scanner-finding:occurrence-${index}`,
sourceResultId: `occurrence-${index}`,
sourceRuleId: `A-Rule-${index}`,
title: `Synthetic check ${index}`
}));
let active = 0;
let maximum = 0;
const complete = vi.fn(async () => {
active += 1;
maximum = Math.max(maximum, active);
await new Promise(resolve => setTimeout(resolve, 5));
active -= 1;
return { value: { schemaVersion: 1, outcome: 'no_relevant_control', reason: 'Synthetic.', mappings: [] } };
});
const svc = service(mockRepository(findings), complete);
await svc.startBatch('project-1', {
assessmentIds: ['assessment-1'], scanRunIds: ['run-1'], filters: { deviation: true }, idempotencyKey: 'batch-concurrency-1'
});
await svc.close();
expect(complete).toHaveBeenCalledTimes(5);
expect(maximum).toBe(2);
});
it('reads every server page independently of loaded rows', async () => {
const findings = Array.from({ length: 501 }, (_, index) => finding({
resultProjectionId: `scanner-finding:occurrence-${index}`,
sourceResultId: `occurrence-${index}`
}));
const repository = mockRepository(findings);
const complete = vi.fn(async () => ({ value: {
schemaVersion: 1, outcome: 'no_relevant_control', reason: 'Synthetic.', mappings: []
} }));
const svc = service(repository, complete);
await svc.startBatch('project-1', {
assessmentIds: ['assessment-1'], scanRunIds: ['run-1'], filters: { deviation: true }, idempotencyKey: 'batch-many-pages-1'
});
await svc.close();
const response = await svc.getBatch('project-1', {
assessmentIds: ['assessment-1'], scanRunIds: ['run-1'], filters: { deviation: true }
});
expect(response.run?.counts).toMatchObject({ filteredResultCount: 501, eligibleFindingCount: 501, distinctRuleCount: 1 });
expect(repository.queryUnifiedResults).toHaveBeenCalledTimes(2);
expect(complete).toHaveBeenCalledOnce();
});
it('fails the batch safely when the filtered total changes between pages', async () => {
const findings = Array.from({ length: 501 }, (_, index) => finding({
resultProjectionId: `scanner-finding:occurrence-${index}`,
sourceResultId: `occurrence-${index}`
}));
const repository = mockRepository(findings);
let page = 0;
vi.mocked(repository.queryUnifiedResults).mockImplementation(async (query = {}) => {
page += 1;
const offset = query.offset ?? 0;
return { total: page === 1 ? 501 : 500, items: findings.slice(offset, offset + (query.limit ?? findings.length)) };
});
const complete = vi.fn();
const svc = service(repository, complete);
await svc.startBatch('project-1', {
assessmentIds: ['assessment-1'], scanRunIds: ['run-1'], filters: { deviation: true }, idempotencyKey: 'batch-changing-total-1'
});
await svc.close();
const response = await svc.getBatch('project-1', {
assessmentIds: ['assessment-1'], scanRunIds: ['run-1'], filters: { deviation: true }
});
expect(response.run).toMatchObject({ status: 'failed', errorCode: 'regulatory_mapping_batch_scope_changed' });
expect(complete).not.toHaveBeenCalled();
});
it('completes with partial failures, then reuses success and retries only the failed rule', async () => {
const findings = [finding(), finding({
resultProjectionId: 'scanner-finding:occurrence-2', sourceResultId: 'occurrence-2', sourceRuleId: 'A-Second', title: 'Second synthetic check'
})];
let secondAttempts = 0;
const complete = vi.fn(async (_system: string, input: string) => {
if (input.includes('Second synthetic check') && ++secondAttempts === 1) throw new Error('synthetic provider failure');
return { value: { schemaVersion: 1, outcome: 'no_relevant_control', reason: 'Synthetic.', mappings: [] } };
});
const svc = service(mockRepository(findings), complete);
const scope = { assessmentIds: ['assessment-1'], scanRunIds: ['run-1'], filters: { deviation: true } };
await svc.startBatch('project-1', { ...scope, idempotencyKey: 'batch-partial-1' });
await svc.close();
expect((await svc.getBatch('project-1', scope)).run).toMatchObject({
status: 'completed', counts: { scheduledRuleCount: 2, succeededRuleCount: 1, failedRuleCount: 1 }
});
await svc.startBatch('project-1', { ...scope, idempotencyKey: 'batch-partial-2' });
await svc.close();
expect((await svc.getBatch('project-1', scope)).run).toMatchObject({
status: 'completed', counts: {
distinctRuleCount: 2, reusedSucceededRuleCount: 1, scheduledRuleCount: 1, succeededRuleCount: 1, failedRuleCount: 0
}
});
expect(complete).toHaveBeenCalledTimes(3);
});
it('reuses an active single-finding mapping without a duplicate model call', async () => {
let release!: () => void;
const gate = new Promise<void>(resolve => { release = resolve; });
const complete = vi.fn(async () => {
await gate;
return { value: { schemaVersion: 1, outcome: 'no_relevant_control', reason: 'Synthetic.', mappings: [] } };
});
const svc = service(mockRepository([finding()]), complete);
await svc.start('project-1', { resultProjectionId: 'scanner-finding:occurrence-1' });
const scope = { assessmentIds: ['assessment-1'], scanRunIds: ['run-1'], filters: { deviation: true } };
await svc.startBatch('project-1', { ...scope, idempotencyKey: 'batch-active-reuse-1' });
release();
await svc.close();
expect((await svc.getBatch('project-1', scope)).run?.counts).toMatchObject({
distinctRuleCount: 1, reusedActiveRuleCount: 1, scheduledRuleCount: 0
});
expect(complete).toHaveBeenCalledOnce();
});
}); });
async function runWith(value: unknown): Promise<{ svc: RegulatoryMappingService }> { async function runWith(value: unknown): Promise<{ svc: RegulatoryMappingService }> {