diff --git a/docs/IMPLEMENTATION_PLAN.md b/docs/IMPLEMENTATION_PLAN.md index f265fb3..6d09197 100644 --- a/docs/IMPLEMENTATION_PLAN.md +++ b/docs/IMPLEMENTATION_PLAN.md @@ -1047,6 +1047,8 @@ Referenzerkennung 2026-09-26 (Phase 13D, erster Umfang umgesetzt und verifiziert Status: lebendes Projektdokument +Gefiltertes Result-DB-ISO-Batchmapping 2026-09-16 (Phase 13F, ausdruecklicher Nutzerauftrag): Result DB bietet `Map all ISO controls` neben Refresh und Export JSON. Export und Batch leiten ihren Scope aus demselben normalisierten `UnifiedResultQuery` ab; der gewaehlte Timeslot bleibt aeussere Grenze, alle aktiven Filter einschliesslich des Standardfilters `deviation: true` grenzen weiter ein, und Pagination/geladene Zeilen haben keinen Einfluss. Der Server liest stabil alle Treffer, bricht bei veraenderter Gesamtzahl sicher ab, zaehlt Nicht-Scanner und nicht unterstuetzte Findings getrennt, dedupliziert global nach `ruleKey`, verwendet erfolgreiche/aktive Mappings wieder und plant fehlende beziehungsweise zuvor fehlgeschlagene Regeln ueber dieselbe auf zwei parallele Retrieval-/Modellpipelines begrenzte Queue wie Einzelaktionen. Atomare deterministische Claims verhindern doppelte Modellaufrufe. Migration 027 speichert nur den projektgebundenen normalisierten Filter-/Timeslot-Snapshot, Hash, Idempotenz, Status und Aggregatfortschritt; globale Mappingzeilen bleiben autoritativ. Das Dialogfenster zeigt Treffer und Fortschritt, bindet laufende Batches an den beim Start erfassten Filterstand und erzeugt ausschliesslich Vorschlaege; Accept/Reject bleibt manuell in Work. Filterwerte, Occurrence-Assets, Pfade, Mandanten- und Reviewdaten gelangen nicht in Retrieval oder LiteLLM. Die fruehere Batch-Zurueckstellung ist fuer diesen manuellen gefilterten Lauf ausdruecklich aufgehoben; die repraesentative Retrievalbewertung bleibt offenes Qualitaetsrisiko. Keine Scannerverarbeitung, kein Manifestbump und kein Reimport. Verifikation: `npm test` mit 1.315 bestandenen Tests und zehn uebersprungenen opt-in Integrationen, Local-only-/Designpruefung sowie Client-/Server-Produktionsbuild erfolgreich (bestehender Bundlehinweis). Der synthetische regulatorische End-to-End-Integrationstest blieb ohne aktivierte lokale Abhaengigkeiten uebersprungen. Korrektur nach Anwendung von Migration 027: Eine nachtraeglich in dieselbe Datei aufgenommene reine Sortiersequenz verletzte die Migrationsimmutabilitaet und der Checksum-Schutz stoppte das Backend korrekt. 027 ist exakt auf den angewandten Stand ohne Sortiersequenz zurueckgesetzt und durch eine feste SHA-256-Regression (`d89055c3e723b09d8d8be7f4225bfd08442f0d66e4e0b4ea3aa62ae6209e9ee9`) geschuetzt; Repositoryabfragen verwenden wieder den bereits angelegten stabilen `created_at, id`-Index. Es gibt keine Migration 028 und keine neue Datenbankmutation. Nach freigegebenem WSL-Neustart wurden der gesunde kanonische Container, Port `127.0.0.1:5432`, Volume `assessor-2_0_assessor-postgres-data`, der identische Ledger-Checksum und das Fehlen von 028 read-only bestaetigt; der automatisch neu gestartete Backendprozess meldet HTTP 200 und `persistence=postgresql`. Kein Backup war fuer die Korrektur erforderlich, weil keine neue Migration ausgefuehrt wurde. + ISO-27002-Mapping-Vertikalschnitt 2026-09-15 (Phase 13F, ausdruecklicher Nutzerauftrag): Die beiden lokal lizenzierten deutschen DIN-2024-PDFs werden serverseitig mit PDF.js und Positionsdaten in einen nicht eingecheckten Korpus-v2 ueberfuehrt. Passwortpflichtige, kopiergeschuetzte, textlose, uebergrosse oder strukturell unvollstaendige Dokumente scheitern geschlossen; ISO 27001 und 27002 muessen dieselben 93 Kontrollen mit 37/8/14/34 Themenkontrollen liefern. `assessor_regulatory_chunks_v2` enthaelt 705 mit lokalem BGE-M3 erzeugte, abschnitts- und kontrollsichere Chunks aus 558 Dokumenten, einschliesslich ISO-27001-Referenzen, Seiten-, Editions-, Parser- und Hashprovenienz; Sparse bleibt reserviert. Der Einzel-Finding-Dienst gruppiert 32 Dense-Treffer auf acht Kontrollen, bindet ISO-27001-Evidenz an, uebergibt nur die minimierte Public-Signal-Query und hoechstens 48.000 Zeichen ISO-Evidenz an das vorhandene Work-LiteLLM und akzeptiert ausschliesslich lokal validierte, evidenzgebundene JSON-Vorschlaege. Migration 025, beide Repositories und drei projektgebundene APIs speichern Runs und Accept/Reject-Entscheidungsereignisse append-only; Work zeigt den manuellen Ablauf inline mit Polling und Seitenbelegen. Ein realer synthetischer End-to-End-Test ueber kanonisches PostgreSQL, llama.cpp/BGE-M3, Qdrant und den freigegebenen LiteLLM-Gateway bestand einschliesslich Optimistic-Conflict und Cascade-Cleanup. Der laufende Entwicklungsserver wendete die additive Migration um 13:42 vor dem um 13:52 erstellten und verifizierten Backup automatisch an; beide neuen Tabellen waren leer. Diese Abweichung ist offen dokumentiert und wird nicht als Vorabbackup ausgegeben. Scannerverarbeitung blieb unveraendert; kein Manifestbump oder Reimport. Regulatory-RAG-Llama-Livebaseline 2026-09-15 (Phase 13F, Nutzerstart): Der lokal gestartete llama.cpp-Server Build 10472 meldet Health 200 und bedient BGE-M3 FP16 ueber den OpenAI-kompatiblen Endpoint. Die produktiv gebauten CLIs indexierten den eingecheckten ausschliesslich synthetischen EN/DE-Korpus mit sechs Dokumenten als sechs 1.024-dimensionale Chunks in den kanonischen Qdrant 1.19.1. Drei ausdruecklich sprachuebergreifende Smoke-Cases erreichten bei K=3 HitRate 1,0, Recall 1,0 und MRR 1,0; alle erwarteten Kontrollen standen auf Rang 1. Das belegt End-to-End-Verkabelung und grundlegende Sprachueberquerung, wegen nur drei klar getrennten Kontrollthemen aber keine produktive Retrievalqualitaet. Qdrant enthaelt sechs synthetische Punkte; PostgreSQL und Realdaten wurden nicht beruehrt. 1.285 Standardtests bestanden, acht optionale PostgreSQL- und ein optionaler Qdrant-Test uebersprungen; Local-only-/Designpruefung und beide Produktionsbuilds erfolgreich (bestehender Bundlehinweis). Ein repraesentativer autorisierter Korpus und unabhaengig gelabelter EN/DE-Satz bleiben fuer die reale Baseline erforderlich. @@ -3157,6 +3159,7 @@ Umgesetzter Grundstand vom 2026-09-15: - Der Grundstand ist in `docs/REGULATORY_RAG.md` dokumentiert und mit synthetischen Vertrags-, Client-, Ingestion-, Retrieval- und Deploymenttests abgesichert. - Der PDF.js-Importer erzeugt aus den explizit uebergebenen ISO-27001-/27002-Pfaden einen lokalen Korpus-v2 mit Quellhashes, Ausgaben, 93 geteilten Kontrollidentitaeten, Abschnittsarten und Seitenbereichen. Nur ISO-27002-Abschnitte werden semantisch gerankt; die entsprechende ISO-27001-Aussage wird als autoritative Referenzevidenz angehaengt. - Ein manueller Work-Vertikalschnitt startet pro kanonischem Scanner-Finding einen idempotenten asynchronen Lauf, gruppiert die 32 Dense-Chunks auf acht Controls, validiert maximal fuenf LiteLLM-Vorschlaege gegen exakt gelieferte Evidenz und speichert Runs sowie optimistisch nebenlaeufigkeitsgeschuetzte Accept/Reject-Ereignisse append-only in PostgreSQL. +- Result DB startet auf ausdrueckliche Aktion einen projektgebundenen Batch fuer exakt denselben vollstaendigen normalisierten Filter-/Timeslot-Scope wie Export JSON. Der Dienst paginiert serverseitig unabhaengig von der Anzeige, dedupliziert Scanner-Vorkommen global nach `ruleKey`, verwendet erfolgreiche und aktive Mappings wieder, versucht fehlgeschlagene Regeln einmal erneut und begrenzt Einzel- plus Batchpipelines gemeinsam auf zwei. Migration 027 speichert nur Scope, Filterhash, Idempotenz, Status und Aggregatfortschritt; Entscheidungen bleiben manuell im bestehenden Work-Panel. Offen bis zur Phasenabnahme: @@ -3164,7 +3167,7 @@ Offen bis zur Phasenabnahme: - kuratierte Erweiterung der oeffentlichen Signal-/Regelkataloge - repraesentativer freigegebener EN/DE-Evaluationssatz und dokumentierte reale Dense-Baseline gegen den autorisierten Korpus - aktive Sparse-/BM25-Erzeugung, Hybridfusion und evaluierter Reranker -- Confidence-Kalibrierung, Assessment-Batchlauf, Mandanten-/Rollenabgrenzung, Betriebsmonitoring und Lasttest +- Confidence-Kalibrierung, Mandanten-/Rollenabgrenzung, Betriebsmonitoring und Lasttest - Betriebsprozess, der vor jeder kuenftigen Migration einen laufenden Hot-Reload-Backendprozess stoppt oder die Migration bis nach dem verifizierten Backup sperrt Abnahmekriterien: @@ -3192,7 +3195,7 @@ Die naechste Umsetzung fuehrt Phase 13B auf dem konsolidierten Import-, Policy-A 1. Phase 13A ist abgeschlossen: Erkennung, technische Dokumenttypen, Source Extractors, Mappingprofile, Finding-Identität sowie allgemeine Artefakt-, Processing- und Jobgrenzen sind konsolidiert und die Scannerpersistenz verwendet den Candidate-Pfad. 2. Phase 13B ist in Arbeit: Der sichere Ingestion-Unterbau, technische Policy-Identitaeten, versionierte ADMX-/Baselineanalysen, deterministisches Reprocessing, Environment Profiles, konservative Snapshotaggregation, typisierte Evaluation, fail-closed Finding Factory, Unified-Result-Read-Projektion, projektgebundene Review-/Orchestrierungsoberflaechen und die aktuelle snapshotbezogene Ergebnisdeduplizierung sind umgesetzt. Als naechste vertikale Etappe folgen fachlich kuratierte Policy-Regeln und vollstaendige versionierte Windows-Inbox-ADMX-/ADML-Kataloge; danach weitere Preference-CSEs und Exportdetails. Ohne RSoP/GP Results bleibt die effektive Anwendung weiterhin ausdruecklich nicht verifiziert. -3. Phase 13F besitzt den lizenzierten ISO-Korpus-v2, die rebuildbare Qdrant-/Dense-Grundlage, den fail-closed Public-Signal-Pfad sowie den persistierten manuell reviewten Einzel-Finding-Workflow. Als naechste Etappe folgen ein unabhaengig gelabelter repraesentativer EN/DE-Evaluationssatz und die reale Dense-Baseline; Sparse/Hybrid, Reranking und Assessment-Batchmapping werden erst danach aktiviert. +3. Phase 13F besitzt den lizenzierten ISO-Korpus-v2, die rebuildbare Qdrant-/Dense-Grundlage, den fail-closed Public-Signal-Pfad, den persistierten manuell reviewten Einzel-Finding-Workflow sowie das explizit vorgezogene gefilterte Result-DB-Batchmapping. Als naechste Etappe folgen ein unabhaengig gelabelter repraesentativer EN/DE-Evaluationssatz und die reale Dense-Baseline; Sparse/Hybrid, Reranking, Confidence-Kalibrierung und jede automatische Entscheidung bleiben bis zu dieser Bewertung ausgeschlossen. 4. Erst nach verifizierter Phase 13B den kontrollierten Firmenbetrieb aus Phase 14 mit SSO, Rollen, HTTPS, Monitoring, Backup/Restore und Aufbewahrung fortsetzen. Die technische Reihenfolge ist bis einschliesslich Phase 13A umgesetzt. Phase 13B baut auf dem verifizierten sicheren Ingestion-Unterbau auf und reicht inzwischen produktiv von der begrenzten Parser-/Workerverarbeitung ueber unveraenderliche Analyse-, Profil-, Aggregations- und Evaluationslaeufe bis zur streng evidenzgebundenen Finding Factory und ihrer gemeinsamen Unified-Result-Leseprojektion. Das Backend leitet projektgebundene Aktionsbereitschaft aus aktiven Revisionen und exakt passenden Referenzdaten ab; der Datahub fuehrt Environment-Profile-Anlage, Reprocessing, Aggregation, Evaluation und Finding Factory ueber die autoritativen idempotenten Services aus. Der produktive Regelkatalog bleibt bis fachlicher Kuratierung leer, und die vorhandenen Microsoft-Baselines bleiben ohne vollstaendige Windows-Inbox-ADMX-/ADML-Kataloge partial; dadurch erzeugt dieser reale Bestand weiterhin nur Review und keine Findings. @@ -3267,6 +3270,8 @@ Praezisierung 2026-09-15 (Phase 13F, Mapping-Relevanz): Der Mapping-Systemprompt Festlegung 2026-09-15 (Phase 13F, globale ISO-Mappings, ausdruecklicher Nutzerauftrag): ISO-Mappings und Accept/Reject-Entscheidungen sind jetzt global und mandantenunabhaengig, analog zur CVE-Anreicherung. Migration `026` ersetzt die per-Occurrence-Tabellen `assessor_regulatory_mapping_runs`/`_decisions` (nur regenerierbare Testdaten, vor dem Drop gesichert) durch `assessor_regulatory_mappings` und ein globales `assessor_regulatory_mapping_decisions`. Schluessel ist die Regelidentitaet `ruleKey` (`tool:ruleId`, ersatzweise ein stabiler Titel-Hash) plus die Pipelineversionen (`UNIQUE(rule_key, corpus_version, retrieval_version, prompt_version, model)`); die Mapping-ID ist ein deterministisches Digest dieses Tupels, sodass ein Regel-Mapping genau einmal existiert und jede Occurrence in jedem Projekt es wiederverwendet. Entscheidungen sind append-only, per `(ruleKey, controlId)` gefuehrt, versionsunabhaengig und ueberleben Scan-, Assessment- und Projektloeschung. Der Dienst leitet nur die non-personale Regelidentitaet und den Pruefkontext aus dem Finding ab; Ergebnistabelle/Occurrence fliesst nie ein. Kein Reimport (kein Scanner-Import veraendert). Naechste Etappe: assessment-weites Batchmapping ueber distinct Regeln. +Festlegung 2026-09-16 (Phase 13F, gefiltertes Result-DB-Batchmapping, ausdruecklicher Nutzerauftrag; begrenzte Aufhebung der frueheren Batch-Zurueckstellung): `All` bezeichnet alle Treffer der aktuell aktiven Result-DB-Filter innerhalb des gewaehlten Timeslots und ist exakt derselbe vollstaendige `UnifiedResultQuery`-Scope wie Export JSON. Offset, Limit, Load-more-Zustand und geladene Zeilen sind reine Darstellung. Der Standardfilter `deviation: true` bleibt wirksam, bis er geaendert oder zurueckgesetzt wird. Der beim Start normalisierte Snapshot bleibt fuer den Lauf unveraendert und wird projektgebunden mit kanonischem Hash und Idempotenz in Migration 027 gespeichert. Nur kanonische Scanner-Findings sind geeignet; Nicht-Scanner und fehlender oeffentlicher Pruefkontext werden gezaehlt, aber nie versendet. Vorkommen werden global nach `ruleKey` dedupliziert, erfolgreiche oder aktive Mappings wiederverwendet und fehlende beziehungsweise zuvor fehlgeschlagene deterministisch geclaimt. Einzel- und Batchaktionen teilen zwei parallele Retrieval-/Modellpipelines. Der Batch erzeugt ausschliesslich Vorschlaege; Accept/Reject bleibt manuell und es gibt weder Auto-Accept noch Batch-Entscheidungsqueue. Die vorgezogene Orchestrierung ist keine Aussage zur Retrievalqualitaet: repraesentative EN/DE-Evaluation, Sparse/Hybrid, Reranking und Confidence-Kalibrierung bleiben offen. Kein Importvertrag geaendert, daher kein Manifestbump oder Reimport. + Festlegung 2026-09-10 (Purple-Knight-Begleitdateien): Ein nativer Score darf aus einer separaten Summary auf die Findings desselben Scanlaufs uebertragen werden, wenn genau ein nativer Traeger und eine eindeutige Ergebnispopulation denselben Werkzeug- und Verarbeitungskontrakt teilen. Datei- und Vorkommensherkunft bleiben gemeinsam erhalten; Mehrdeutigkeit verhindert die Verbindung. Festlegung 2026-09-10 (weitere Scanner): Native Gesamtwerte werden bei jeder Profilerweiterung zuerst geprueft und deklarativ mit ihrer Bedeutung erfasst. Die Scoredefinition ist Pflicht, die Berechnung gemeinsam; ihre Version gehoert zur Vergleichskompatibilitaet. Extraktion und Originalherkunft bleiben im generischen Importpfad. AGENTS.md und docs/SCORING_MODEL.md halten die Erweiterungsregel fest. @@ -4256,6 +4261,7 @@ Verifiziert 2026-09-08 (Phase 7/13B): GPO-Quellenkorrektur fuer Work und Client | 2026-09-15 | Phase 7/13B: GPO Analysis verwendet fuer seine Baseline-Ergebniszaehler jetzt dieselben backendautoritativen aktuellen Aggregationslaeufe wie Work und Client Info. Historische Aggregationen bleiben in der Evaluationshistorie, werden aber nicht mehr zu aktuellen Pass-/Fail-/Not-configured-Zaehlern addiert. Review required ist in der Zusammenfassung sichtbar. Exakte synthetische Mehrlauf-/Ein-Timeslot-Komponentenregression, 26 fokussierte Tests, 1.267 bestandene Gesamttests plus isoliert bestandener bekannter Last-Timeout und beide Produktionsbuilds erfolgreich; acht optionale PostgreSQL-Tests uebersprungen. Keine Datenmutation, Neuauswertung, Import-, Persistenz-, Schema-, Manifest- oder Reimportaenderung. | | 2026-09-15 | Phase 13/9-11: Timeslot-Importlast korrigiert. Inaktive GPO-Importsteuerelemente in Timeslotzeilen erzeugen keine eigenen Poller mehr; sichtbare Statusansichten und der Data Hub fragen nur waehrend queued/running ab, bewahren unveraenderte Zustandsobjekte und berechnen Result-Facetten nur bei einer echten Statusrevision neu. Datei- und Definitionszuordnung im PostgreSQL-Commit ist je Import vorindiziert. Gemessener Ausgangszustand: 4,4 s Vorschau, 200,3 s Commit unter 270 Facetten- und 326 Statusabfragen in fuenf Minuten. Synthetische Interaktionsregression, 30 fokussierte Tests, 1.266 bestandene Gesamttests plus isoliert bestandener bekannter Last-Timeout, Local-only-/Designpruefung und beide Builds erfolgreich; acht optionale PostgreSQL-Integrationstests uebersprungen. Keine Schema-, Daten-, Importsemantik-, Manifest- oder Reimportaenderung. Restrisiko serieller PostgreSQL-Einzelwrites dokumentiert. | | 2026-09-14 | Phase 7/13B: Regression im Work-Filterkontext korrigiert. `Source: Scan = GPOs` reduziert auf dem All-Records-Blatt nun vor Priorisierung und Facettenlimit auf den GPO-Vertrag; das Statusmenue wird unabhaengig von einem separaten GPO-Blatt mit All, Deviation, Passed, Not configured, Conflicts und Duplicates vollstaendig erzeugt, fachliche Nullwerte bleiben waehlbar. GPO-Statuswerte werden ausschliesslich gegen das aktuelle Statusfeld geprueft; ein Pass-Vergleichswert kann einen Not-configured-Datensatz nicht mehr als Passed filtern. Fremde Scannerfacetten und ungueltige gesetzte Altfilter werden entfernt, `Source: Scan` bleibt vollstaendig und rechts; ohne Quellrestriktion kehrt der Mischkontext zurueck. Exakte synthetische Interaktionsregressionen pruefen Anzeige, tatsaechliche Filterwirkung und den gemeldeten Gegenfall; 51 fokussierte Tests sowie der Gesamtlauf mit 1.243 bestandenen und acht uebersprungenen Tests, Local-only-/Designpruefung und Client-/Serverbuild erfolgreich. `AGENTS.md` bindet dokumentierte Nutzerentscheidungen als Abnahmekriterien und verlangt transparente Meldung vor regelwidrigen Aenderungen. Keine Daten-, Import-, API-, Persistenz-, Score- oder Reimportaenderung; ausschliesslich synthetische Inhaltspruefung. | +| 2026-09-16 | Phase 13F, ausdruecklicher Nutzerauftrag: Gefiltertes Result-DB-ISO-Batchmapping umgesetzt. Der gemeinsame Scope-Builder garantiert dieselbe vollstaendige Filter-/Timeslot-Abfrage wie Export JSON und entfernt Pagination sowie nicht autoritative Client-Scopewerte. Projektgebundene GET-/POST-APIs, Migration 027, Local-/PostgreSQL-Paritaet, Export und Loeschung speichern den normalisierten Scope, kanonischen Filterhash, Idempotenz, Lifecycle und Fortschritt; globale Regelmappings bleiben autoritativ. Stabile serverseitige Mehrseitenabfrage mit Total-Drift-Abbruch, deterministische `ruleKey`-Repraesentanten, erfolgreiche/aktive Wiederverwendung, einmaliger Failed-Retry, atomare Mappingclaims und eine gemeinsame Zwei-Pipeline-Queue sind implementiert. Result DB zeigt `Map all ISO controls`, Trefferzahl, Status und Zaehler, verhindert veraltete Filterantworten und behaelt Accept/Reject manuell in Work. Synthetische Regressionen decken alle Filterfamilien, Standardabweichungsfilter, Scopebereinigung, Mehrseitenlauf, Total-Drift, Ausschluesse, globale Deduplikation, Partial Failure, Retry/Reuse, Single-/Batch-Nebenlaeufigkeit und UI ab. `npm test`: 1.315 bestanden, zehn opt-in Integrationen uebersprungen; Local-only-/Designpruefung und beide Produktionsbuilds erfolgreich. Kein Manifestbump oder Reimport. Nach der Anwendung von 027 wurde dieselbe Datei faelschlich um eine reine Sortiersequenz erweitert; der Checksum-Schutz stoppte den Backendstart. Die Datei ist exakt auf den angewandten SHA-256-Stand `d89055c3e723b09d8d8be7f4225bfd08442f0d66e4e0b4ea3aa62ae6209e9ee9` zurueckgesetzt und mit einer Regression gesperrt, die Abfragen verwenden wieder den vorhandenen `created_at, id`-Index. Keine Migration 028 und keine weitere Datenbankmutation. Fuenf fokussierte Suites mit 16 Tests, Servertyppruefung und Serverbuild erfolgreich. Nach freigegebenem WSL-Neustart: kanonischer PostgreSQL-Container gesund, Port/Volume korrekt, Ledger 027 checksum-identisch, 028 nicht vorhanden und Backend-Health HTTP 200 mit PostgreSQL-Persistenz. | | 2026-09-14 | Phasen 7/13C und 11: Work-LiteLLM-gestuetztes AI Overview umgesetzt. Der Backend-Service liest den kompletten Timeslot ueber das bestehende Privacy-Safe-Result-DB-JSON, hasht ihn deterministisch, zerlegt grosse Payloads nur zwischen vollstaendigen Eintraegen und konsolidiert sie mit maximal 64 explizit nicht wiederholten LiteLLM-Aufrufen. Der HTTPS-Client haelt URL, Modell und Bearer-Key ausschliesslich in Server-Secrets, lehnt lokale/unsichere Ziele und Redirects ab und begrenzt Zeit sowie Antwortbytes. Ausgabevalidierung prueft das vollstaendige Dashboard-Schema, Laengen, Roadmap 0-4, exakte Finding-/Gap-IDs und Environment-Properties. Same-Origin-GET/POST, Migration 024, lokale/PostgreSQL-Repositoryparitaet, Projekt-Export/-Loeschung, Idempotenz und ein aktiver Run je Scope sind ergaenzt. Client Info rendert alle Analyseabschnitte mit Generate/Regenerate, Polling nur bei aktiven Runs sowie stale/failed/disabled Zustaenden. Sicherheits- und Deploymentdokumentation beschreibt den genehmigten Corporate-Gateway-Vertrag. 1.262 synthetische Tests bestanden, acht optionale PostgreSQL-Tests uebersprungen; Local-only-/Designpruefung und beide Produktionsbuilds erfolgreich. Der inhaltsfreie Workstation-Livecheck erreichte Work-LiteLLM, erhielt aber HTTP 401; Providerbody und Secrets wurden nicht ausgegeben. Docker/kanonisches Backup, Migrationanwendung, PostgreSQL-Integration sowie erfolgreicher Workstation-/Server-Livecheck bleiben offen; keine Realdateninspektion oder Reimportaenderung. | | 2026-09-14 | Rebase der AI-Integration auf defender-iot: Die einzelne AI-Commit-Aenderung wurde neu aufgesetzt und die kollidierende Result-DB-LLM-Uebergabe zusammengefuehrt. Das versionierte, datensparsame Privacy-Safe-JSON-Schema dieses Branches ist nun das Result-DB-Exportformat und wird als einzelne Aktion `Export JSON` ueber den gemeinsamen Browser-Download-Adapter als `.json`-Datei geladen; damit ersetzt es sowohl den zuvor auf defender-iot vorhandenen `Export JSONL`-Download als auch die fluechtige `Copy JSON`-Zwischenablageaktion. Der Builder wurde zu `resultsToPrivacySafeExportJson` mit begleitendem Dateinamens-Helfer `privacySafeResultJsonFileName` umbenannt; Entscheidungs-, Risiko- und Testeintraege wurden entsprechend angeglichen. Alle uebrigen defender-iot-Inhalte (CVE-Enrichment, GPO-/Policy-Ergebnisse, Timeslot-Scope) bleiben unveraendert. 1248 Tests bestanden, 8 optionale PostgreSQL-Tests uebersprungen; Local-only-, Designpruefung und Client-/Server-Produktionsbuild erfolgreich. | | 2026-09-08 | Phase 7: Verbindungskurven der Client Info treffen ihre Domainknoten wieder. Feste Zielkoordinaten in einem verzerrten SVG-Koordinatensystem durch eine aus dem Layout gemessene Geometrie ersetzt, Neumessung ueber Groessenbeobachter und Fensterereignis. Am Dummy bei drei Breiten geprueft: Abweichung 0 Pixel statt bis zu 80. Keine Aenderung an Werten oder Kachelgeometrie. 1.011 Tests, Local-only-/Designpruefung und Builds erfolgreich. | diff --git a/docs/PROJECT_OVERVIEW.md b/docs/PROJECT_OVERVIEW.md index 346c44d..6ea89c8 100644 --- a/docs/PROJECT_OVERVIEW.md +++ b/docs/PROJECT_OVERVIEW.md @@ -307,7 +307,7 @@ opt-in, and off by default. Trust boundaries: fields are captured losslessly). Full-text `tsvector` + GIN indexes back search. - **Migrations** are versioned SQL files in [`server/postgres/migrations/`](../server/postgres/migrations/) (currently through - `026_*`). On start the server takes an advisory lock, verifies each migration's + `027_*`). On start the server takes an advisory lock, verifies each migration's version + SHA-256, applies new ones in their own transaction, and refuses to start if an already-applied migration was edited. Never edit an applied migration — add a new numbered file. diff --git a/docs/REGULATORY_RAG.md b/docs/REGULATORY_RAG.md index 1231cf3..0a57526 100644 --- a/docs/REGULATORY_RAG.md +++ b/docs/REGULATORY_RAG.md @@ -1,6 +1,6 @@ # Regulatory-control RAG with Qdrant -Status: Phase 13F one-finding vertical slice. The server prepares the licensed German ISO 27001/27002 corpus, indexes section-safe ISO 27002 semantic chunks plus ISO 27001 references, retrieves and groups candidates, requests a grounded LiteLLM proposal, persists runs and reviewer decisions, and exposes the workflow inline in Work. Batch mapping, sparse retrieval, reranking, and confidence calibration remain out of scope. +Status: Phase 13F one-finding and filtered Result DB batch vertical slice. The server prepares the licensed German ISO 27001/27002 corpus, indexes section-safe ISO 27002 semantic chunks plus ISO 27001 references, retrieves and groups candidates, requests grounded LiteLLM proposals, persists runs and reviewer decisions, and exposes manual proposal generation in Work and Result DB. Sparse retrieval, reranking, and confidence calibration remain out of scope. ## Boundaries @@ -127,9 +127,21 @@ Routes: - `GET /api/v1/projects/{projectId}/regulatory-mappings/{resultProjectionId}` - `POST /api/v1/projects/{projectId}/regulatory-mapping-runs` - `POST /api/v1/projects/{projectId}/regulatory-mapping-decisions` +- `GET /api/v1/projects/{projectId}/regulatory-mapping-batches` +- `POST /api/v1/projects/{projectId}/regulatory-mapping-batches` On the consolidated migration line (2026-10-02), the single migration 026 `regulatory-mapping` creates the global tables directly, together with the project-scoped `assessor_regulatory_mapping_batches`; the superseded occurrence-bound tables are never created. The remainder of this paragraph records the original history on the former experimental line, where migration 025 originally added occurrence-bound, project-scoped run and decision tables and migration 026 superseded it: it dropped those tables and created the global `assessor_regulatory_mappings` (keyed by `rule_key` + pipeline versions, `UNIQUE(rule_key, corpus_version, retrieval_version, prompt_version, model)`) and a global, rule-keyed `assessor_regulatory_mapping_decisions`; neither carries `project_id` or occurrence identity, and both survive project/finding deletion. Before 026 was applied, the 025 tables held only regenerable local test data (19 runs, 10 decisions); a verified backup of both tables was taken at `backups/regulatory_mapping_025_backup_20260915-155410.sql` and the data was then intentionally discarded per the user's decision. A live development backend applied 026 through its own migration runner (checksum recorded and re-verified against the source file); no reimport is required because mapping runs are regenerable and no scanner data changed. +## Filtered Result DB batch workflow + +`Map all ISO controls` sits beside Refresh and Export JSON in Result DB. Both actions derive their scope from the same normalized `UnifiedResultQuery`: the selected assessment and scan-run IDs are the outer timeslot boundary, and every active Result DB filter narrows that scope. This includes deviation, source, severity, outcome, result type, processing state, review state, categories, and supported text/date filters. The default `deviation: true` filter therefore maps deviations only until the user changes or resets it. Offset, limit, client project IDs, and client audience values are discarded. Display pagination, Load more, and the number of mounted rows never change batch membership. + +The server enumerates the complete query through stable pagination and fails the batch if the filtered total changes during enumeration. It counts non-scanner results as excluded, scanner findings without usable public check context as unsupported, and deduplicates eligible occurrences by global `ruleKey`. One deterministic representative is selected per rule. Current successful or active mappings are reused; missing mappings and previously failed mappings are atomically claimed and queued. Single-finding and batch work share the same queue with at most two concurrent retrieval/model pipelines, so simultaneous requests cannot create duplicate model calls for a deterministic mapping identity. A completed batch may still report individual rule failures. + +The batch table (part of migration 026 on the consolidated line, originally migration 027) holds only project-scoped batch orchestration records: the normalized timeslot/filter snapshot, canonical filter hash, idempotency key, state, aggregate counters, timestamps, and bounded failure details. Global individual mapping rows remain authoritative; there is no batch-item table. Batch records are exported with their project and removed with project deletion. Active mappings and batches are marked failed after an interrupted server process; a later batch reuses successful work and retries only missing or failed rules. The migration is additive and does not change scanner processing, so no processing-manifest bump or reimport is required. After 027 had been applied, a later ordering-only edit incorrectly changed that migration and the checksum guard stopped backend startup. The file has been restored to its applied SHA-256 state and locked by regression; no follow-up schema migration is required. + +The filter snapshot is captured when Start is pressed and is never copied into embedding or LiteLLM payloads. Result DB reloads the latest batch for the currently selected normalized scope, so changing filters does not relabel an active run. Batches generate proposals only. Accept/Reject remains the existing explicit human decision per global rule/control in Work; there is no automatic acceptance or batch decision queue. + ### Local Qdrant integration check `tests/server/qdrant-regulatory-index.integration.test.ts` is opt-in and uses deterministic synthetic vectors. With `ASSESSOR_QDRANT_INTEGRATION=1` and the normal Qdrant/model identity environment variables, it verifies the real collection, payload indexes, document replacement, filters, and retrieval without calling an embedding provider. The standard suite skips it so tests do not unexpectedly mutate a developer's derived index. @@ -180,7 +192,7 @@ Live smoke result on 2026-09-15 with local llama.cpp build 10472, BGE-M3 FP16, a - `RegulatoryReranker` is optional and receives only the bounded candidate set. Without one, dense ordering is retained. - `RegulatoryRetriever` is the consumer-facing contract. Mapping code should depend on it, not on Qdrant. -Before assessment-wide batch mapping, broaden public-signal coverage only through reviewed catalogs, establish a representative independently labeled EN/DE baseline, and evaluate sparse/hybrid retrieval, reranking, and confidence calibration. The one-finding workflow must remain human-reviewed. +The filtered manual Result DB batch was activated by explicit user decision before the earlier quality-gate sequence. That decision changes orchestration only: broaden public-signal coverage through reviewed catalogs, establish a representative independently labeled EN/DE baseline, and evaluate sparse/hybrid retrieval, reranking, and confidence calibration before claiming production retrieval quality or adding automatic decisions. Single and batch proposals remain human-reviewed. ## Reference basis diff --git a/server/assessor-http-server.ts b/server/assessor-http-server.ts index 75c8fc2..4ffd0e6 100644 --- a/server/assessor-http-server.ts +++ b/server/assessor-http-server.ts @@ -31,7 +31,12 @@ import type { GpoPolicyAnalysisOverviewService } from './gpo/gpo-policy-analysis import type { AiOverviewService } from './ai/ai-overview-service'; import { AiOverviewServiceError } from './ai/ai-overview-service'; import type { AiOverviewCreateRequest, AiOverviewScope } from '../src/domain/ai-overview'; -import type { RegulatoryMappingCreateRequest, RegulatoryMappingDecisionCreateRequest } from '../src/domain/regulatory-mapping'; +import type { + RegulatoryMappingBatchCreateRequest, + RegulatoryMappingBatchScope, + RegulatoryMappingCreateRequest, + RegulatoryMappingDecisionCreateRequest +} from '../src/domain/regulatory-mapping'; import type { RegulatoryMappingService } from './regulatory/regulatory-mapping-service'; import { RegulatoryMappingServiceError } from './regulatory/regulatory-mapping-service'; import type { GpoAutomaticAnalysisService } from './gpo/gpo-automatic-analysis-service'; @@ -311,6 +316,14 @@ function unifiedQueryFrom(url: URL): UnifiedResultQuery { }; } +function regulatoryMappingBatchScopeFrom(url: URL): RegulatoryMappingBatchScope { + const query = unifiedQueryFrom(url); + const { audience: _audience, projectId: _projectId, assessmentId: _assessmentId, + assessmentIds = [], scanRunId: _scanRunId, scanRunIds = [], offset: _offset, limit: _limit, + ...filters } = query; + return { assessmentIds, scanRunIds, filters }; +} + function validateImportSource(value: ImportSourceValidationRequest): ImportSourceValidationResponse { if (!value || typeof value.fileName !== 'string' || !value.fileName.trim()) { throw new HttpError(400, 'invalid_file_metadata', 'Ein Dateiname ist erforderlich.'); @@ -1186,6 +1199,15 @@ export function createAssessorHttpServer(options: AssessorHttpServerOptions): Se const body = await readJson(request, maximumBodyBytes); status = 200; sendJson(response, status, await options.regulatoryMappingService.decide(segments[1], body), requestId); return; } + if (segments[0] === 'projects' && segments[2] === 'regulatory-mapping-batches' && segments.length === 3 && method === 'GET') { + if (!options.regulatoryMappingService) throw new HttpError(503, 'regulatory_mapping_disabled', 'Regulatory mapping is not configured.'); + status = 200; sendJson(response, status, await options.regulatoryMappingService.getBatch(segments[1], regulatoryMappingBatchScopeFrom(url)), requestId); return; + } + if (segments[0] === 'projects' && segments[2] === 'regulatory-mapping-batches' && segments.length === 3 && method === 'POST') { + if (!options.regulatoryMappingService) throw new HttpError(503, 'regulatory_mapping_disabled', 'Regulatory mapping is not configured.'); + const body = await readJson(request, maximumBodyBytes); + status = 202; sendJson(response, status, await options.regulatoryMappingService.startBatch(segments[1], body), requestId); return; + } if (segments.length === 1 && segments[0] === 'assessments' && method === 'POST') { const assessment = await readJson(request, maximumBodyBytes); await repository.saveAssessment(assessment); diff --git a/server/postgres/postgres-assessor-repository.ts b/server/postgres/postgres-assessor-repository.ts index c433da3..e227c5e 100644 --- a/server/postgres/postgres-assessor-repository.ts +++ b/server/postgres/postgres-assessor-repository.ts @@ -27,8 +27,8 @@ import type { EnvironmentObservation, EnvironmentSnapshot, EnvironmentSnapshotQu import { environmentCoverage } from '../../src/domain/environment-context'; import type { AiOverviewRun } from '../../src/domain/ai-overview'; import { aiOverviewRunValidationErrors } from '../../src/domain/ai-overview'; -import type { RegulatoryMappingDecision, RegulatoryMappingRun } from '../../src/domain/regulatory-mapping'; -import { regulatoryMappingDecisionValidationErrors, regulatoryMappingRunValidationErrors } from '../../src/domain/regulatory-mapping'; +import type { RegulatoryMappingBatchRun, RegulatoryMappingDecision, RegulatoryMappingRun } from '../../src/domain/regulatory-mapping'; +import { regulatoryMappingBatchRunValidationErrors, regulatoryMappingDecisionValidationErrors, regulatoryMappingRunValidationErrors } from '../../src/domain/regulatory-mapping'; import { storedPolicyFindingRunValidationErrors } from '../../src/domain/policy-finding'; import { latestReferenceRuns, type ReferenceRecognitionRun, type ReferenceCatalogSnapshot } from '../../src/domain/reference-recognition'; import { validateReferenceRun } from '../../src/persistence/reference-run-validation'; @@ -378,6 +378,7 @@ const DB_STRUCTURE_LAYER_BY_TABLE: Record = { assessor_finding_definitions: 'result', assessor_regulatory_mappings: 'result', assessor_regulatory_mapping_decisions: 'governance', + assessor_regulatory_mapping_batches: 'governance', assessor_ai_overview_runs: 'governance', assessor_finding_occurrences: 'result', assessor_result_categories: 'result', @@ -704,6 +705,37 @@ export class PostgresAssessorRepository implements AssessorRepository { )); } + async claimRegulatoryMappingRun(run: RegulatoryMappingRun, forceRegeneration: boolean): Promise<{ run: RegulatoryMappingRun; claimed: boolean }> { + const errors = regulatoryMappingRunValidationErrors(run); + if (errors.length) throw new RepositoryValidationError(errors.join(' ')); + return this.mutate(async client => { + const inserted = await client.query(` + INSERT INTO assessor_regulatory_mappings + (id, rule_key, source_tool, source_rule_id, run_status, query_hash, corpus_id, corpus_version, + embedding_model, retrieval_version, prompt_version, output_schema_version, model, created_at, updated_at, data) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16::jsonb) + ON CONFLICT DO NOTHING + RETURNING id + `, [run.id, run.ruleKey, run.sourceTool, run.sourceRuleId ?? null, run.status, run.queryHash, run.corpusId, + run.corpusVersion, run.embeddingModel, run.retrievalVersion, run.promptVersion, run.outputSchemaVersion, + run.model, run.createdAt, run.updatedAt, json(run)]); + if (inserted.rowCount) return { run: clone(run), claimed: true }; + const existing = dataRow(await client.query>( + 'SELECT data FROM assessor_regulatory_mappings WHERE id = $1 FOR UPDATE', [run.id] + )); + if (!existing) throw new RepositoryConflictError('Regulatory mapping identity conflicted with another pipeline tuple.'); + if (['queued', 'running'].includes(existing.status) || existing.status === 'succeeded' && !forceRegeneration) { + return { run: existing, claimed: false }; + } + await client.query(` + UPDATE assessor_regulatory_mappings SET + run_status = $2, query_hash = $3, created_at = $4, updated_at = $5, data = $6::jsonb + WHERE id = $1 + `, [run.id, run.status, run.queryHash, run.createdAt, run.updatedAt, json(run)]); + return { run: clone(run), claimed: true }; + }); + } + async saveRegulatoryMappingRun(run: RegulatoryMappingRun): Promise { const errors = regulatoryMappingRunValidationErrors(run); if (errors.length) throw new RepositoryValidationError(errors.join(' ')); @@ -728,6 +760,60 @@ export class PostgresAssessorRepository implements AssessorRepository { )); } + async createRegulatoryMappingBatchRun(run: RegulatoryMappingBatchRun): Promise { + const errors = regulatoryMappingBatchRunValidationErrors(run); + if (errors.length) throw new RepositoryValidationError(errors.join(' ')); + return this.mutate(async client => { + const inserted = await client.query(` + INSERT INTO assessor_regulatory_mapping_batches + (id, project_id, scope_key, filter_hash, idempotency_key, run_status, created_at, updated_at, data) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9::jsonb) + ON CONFLICT DO NOTHING + RETURNING data + `, [run.id, run.projectId, run.scopeKey, run.filterHash, run.idempotencyKey, run.status, + run.createdAt, run.updatedAt, json(run)]); + if (inserted.rowCount) return clone(run); + const existing = dataRow(await client.query>(` + SELECT data FROM assessor_regulatory_mapping_batches + WHERE project_id = $1 AND (idempotency_key = $2 OR (scope_key = $3 AND run_status IN ('queued', 'running'))) + ORDER BY (idempotency_key = $2) DESC, created_at DESC, id DESC + LIMIT 1 + `, [run.projectId, run.idempotencyKey, run.scopeKey])); + if (!existing) throw new RepositoryConflictError('Regulatory mapping batch conflicted with another active request.'); + return existing; + }); + } + + async updateRegulatoryMappingBatchRun(run: RegulatoryMappingBatchRun): Promise { + const errors = regulatoryMappingBatchRunValidationErrors(run); + if (errors.length) throw new RepositoryValidationError(errors.join(' ')); + return this.mutate(async client => { + const existing = dataRow(await client.query>( + 'SELECT data FROM assessor_regulatory_mapping_batches WHERE id = $1 FOR UPDATE', [run.id] + )); + if (!existing) throw new RepositoryNotFoundError('Regulatory mapping batch does not exist.'); + if (existing.projectId !== run.projectId || existing.scopeKey !== run.scopeKey || existing.idempotencyKey !== run.idempotencyKey) { + throw new RepositoryConflictError('Regulatory mapping batch identity is immutable.'); + } + if (['completed', 'failed'].includes(existing.status)) throw new RepositoryConflictError('A completed regulatory mapping batch is immutable.'); + const allowed = existing.status === 'queued' ? ['running', 'failed'] : ['running', 'completed', 'failed']; + if (!allowed.includes(run.status)) throw new RepositoryConflictError(`Invalid regulatory mapping batch transition from ${existing.status} to ${run.status}.`); + await client.query(` + UPDATE assessor_regulatory_mapping_batches SET run_status = $2, updated_at = $3, data = $4::jsonb + WHERE id = $1 + `, [run.id, run.status, run.updatedAt, json(run)]); + return clone(run); + }); + } + + async listRegulatoryMappingBatchRuns(projectId: string, scopeKey?: string): Promise { + return dataRows(await this.pool.query>(` + SELECT data FROM assessor_regulatory_mapping_batches + WHERE project_id = $1 AND ($2::text IS NULL OR scope_key = $2) + ORDER BY created_at DESC, id DESC + `, [projectId, scopeKey ?? null])); + } + async createRegulatoryMappingDecision( decision: RegulatoryMappingDecision, expectedLatestDecisionId?: string @@ -2760,6 +2846,7 @@ export class PostgresAssessorRepository implements AssessorRepository { policyEvaluations: await this.listPolicyEvaluations(projectId), policyFindingRuns, aiOverviewRuns: (await this.listAiOverviewRuns(projectId)).filter(item => item.status === 'succeeded'), + regulatoryMappingBatchRuns: await this.listRegulatoryMappingBatchRuns(projectId), // Regulatory mappings and decisions are global reference data, not project-scoped. regulatoryMappingRuns: (await this.listRegulatoryMappingRuns()).filter(item => item.status === 'succeeded'), regulatoryMappingDecisions: dataRows(await this.pool.query>( diff --git a/server/regulatory/regulatory-mapping-service.ts b/server/regulatory/regulatory-mapping-service.ts index 827f3a8..4fc5a77 100644 --- a/server/regulatory/regulatory-mapping-service.ts +++ b/server/regulatory/regulatory-mapping-service.ts @@ -2,13 +2,19 @@ import { createHash, randomUUID } from 'node:crypto'; import type { AiCompletionClient } from '../ai/litellm-client'; import { LiteLlmClientError } from '../ai/litellm-client'; import type { AssessorRepository } from '../../src/persistence/assessor-repository'; -import { RepositoryNotFoundError, RepositoryValidationError } from '../../src/persistence/repository-errors'; +import { RepositoryConflictError, RepositoryNotFoundError, RepositoryValidationError } from '../../src/persistence/repository-errors'; import { REGULATORY_MAPPING_OUTPUT_SCHEMA_VERSION, REGULATORY_MAPPING_PROMPT_VERSION, REGULATORY_RETRIEVAL_VERSION, + emptyRegulatoryMappingBatchCounts, groundRegulatoryMappingResult, + normalizeRegulatoryMappingBatchScope, regulatoryMappingResultValidationErrors, + type RegulatoryMappingBatchCreateRequest, + type RegulatoryMappingBatchResponse, + type RegulatoryMappingBatchRun, + type RegulatoryMappingBatchScope, type RegulatoryMappingCandidate, type RegulatoryMappingCreateRequest, type RegulatoryMappingDecision, @@ -24,9 +30,12 @@ import { privacySafeFinding } from '../../src/domain/privacy-safe-finding'; import type { RegulatoryFindingRetrievalService } from './regulatory-finding-retrieval'; import type { RegulatoryChunkIndex } from './qdrant-regulatory-index'; import { buildRegulatoryFindingQuery } from '../../src/profiles/public-search/regulatory-query'; +import { canonicalJson } from '../../src/shared/canonical-json'; const POLL_AFTER_MS = 1_500; const MAXIMUM_PROMPT_CHARACTERS = 48_000; +const BATCH_PAGE_SIZE = 500; +const DEFAULT_MAXIMUM_CONCURRENT_RUNS = 2; export interface RegulatoryMappingServiceOptions { findingRetrieval?: RegulatoryFindingRetrievalService; @@ -37,14 +46,25 @@ export interface RegulatoryMappingServiceOptions { embeddingModel?: string; disabledReason?: string; now?: () => Date; + maximumConcurrentRuns?: number; } export class RegulatoryMappingService { private readonly activeTasks = new Set>(); private readonly now: () => Date; + private readonly maximumConcurrentRuns: number; + private readonly workQueue: Array<{ + task: () => Promise<'succeeded' | 'failed'>; + resolve: (status: 'succeeded' | 'failed') => void; + }> = []; + private activeWorkerCount = 0; constructor(private readonly repository: AssessorRepository, private readonly options: RegulatoryMappingServiceOptions = {}) { this.now = options.now ?? (() => new Date()); + this.maximumConcurrentRuns = options.maximumConcurrentRuns ?? DEFAULT_MAXIMUM_CONCURRENT_RUNS; + if (!Number.isInteger(this.maximumConcurrentRuns) || this.maximumConcurrentRuns < 1 || this.maximumConcurrentRuns > 8) { + throw new Error('Regulatory mapping concurrency must be an integer between 1 and 8.'); + } } get generationEnabled(): boolean { @@ -63,6 +83,18 @@ export class RegulatoryMappingService { errorCode: 'regulatory_mapping_interrupted' }); } + for (const project of await this.repository.listProjects()) { + for (const run of (await this.repository.listRegulatoryMappingBatchRuns(project.id)).filter(item => ['queued', 'running'].includes(item.status))) { + await this.repository.updateRegulatoryMappingBatchRun({ + ...run, + status: 'failed', + updatedAt: completedAt, + completedAt, + errorCode: 'regulatory_mapping_batch_interrupted', + errorDetail: 'Batch processing was interrupted before completion. Start a new batch to retry remaining rules.' + }); + } + } } async get(projectId: string, resultProjectionId: string): Promise { @@ -93,55 +125,60 @@ export class RegulatoryMappingService { if (!this.generationEnabled) throw new RegulatoryMappingServiceError('regulatory_mapping_disabled', this.options.disabledReason ?? 'Regulatory mapping is not configured.', 503); const finding = await this.finding(projectId, request.resultProjectionId); if (finding.sourceKind !== 'scanner-finding') throw new RegulatoryMappingServiceError('unsupported_finding_kind', 'Only canonical scanner findings can be mapped in this version.', 422); - const ruleKey = this.ruleKey(finding); - const id = this.mappingId(ruleKey); - const existing = await this.repository.getRegulatoryMappingRun(id); - if (existing && ['queued', 'running'].includes(existing.status)) return this.get(projectId, request.resultProjectionId); - let retrieval; - try { - retrieval = await this.options.findingRetrieval!.retrieve(finding, 32, { - corpusId: this.options.corpusId, - corpusVersion: this.options.corpusVersion, - sourceStandard: 'ISO/IEC 27002', - retrievalRole: 'semantic' - }); - } catch { - throw new RegulatoryMappingServiceError('regulatory_retrieval_unavailable', 'Regulatory retrieval is currently unavailable.', 503); - } - if (retrieval.status !== 'candidates' || !retrieval.preparedQuery) { - throw new RegulatoryMappingServiceError('insufficient_public_context', 'This finding has insufficient public context for regulatory mapping.', 422); - } - const candidates = await this.groupCandidates(retrieval.candidates); - if (!candidates.length) throw new RegulatoryMappingServiceError('regulatory_corpus_empty', 'The active regulatory corpus returned no eligible controls.', 503); - const timestamp = this.now().toISOString(); - const queryHash = createHash('sha256').update(JSON.stringify(retrieval.preparedQuery), 'utf8').digest('hex'); - const proposed: RegulatoryMappingRun = { - version: 1, - id, - ruleKey, - sourceTool: finding.sourceTool ?? 'unknown', - ...(finding.sourceRuleId ? { sourceRuleId: finding.sourceRuleId } : {}), - status: 'queued', - queryHash, - preparedQuery: retrieval.preparedQuery, - corpusId: this.options.corpusId!, - corpusVersion: this.options.corpusVersion!, - sourceHashes: [...new Set(candidates.flatMap(candidate => candidate.evidence.map(evidence => evidence.sourceSha256)))].sort(), - embeddingModel: this.options.embeddingModel!, - retrievalVersion: REGULATORY_RETRIEVAL_VERSION, - promptVersion: REGULATORY_MAPPING_PROMPT_VERSION, - outputSchemaVersion: REGULATORY_MAPPING_OUTPUT_SCHEMA_VERSION, - model: this.options.client!.model, - createdAt: timestamp, - updatedAt: timestamp, - candidates - }; - await this.repository.saveRegulatoryMappingRun(proposed); - const task = this.process(proposed).finally(() => this.activeTasks.delete(task)); - this.activeTasks.add(task); + if (!buildRegulatoryFindingQuery(finding)) throw new RegulatoryMappingServiceError('insufficient_public_context', 'This finding has insufficient public context for regulatory mapping.', 422); + const claimed = await this.claim(finding, true); + if (claimed.claimed) this.track(this.enqueue(() => this.process(claimed.run, finding)).then(() => undefined)); return this.get(projectId, request.resultProjectionId); } + async getBatch(projectId: string, requestedScope: RegulatoryMappingBatchScope): Promise { + const scope = await this.validateBatchScope(projectId, requestedScope); + const scopeKey = this.batchScopeKey(projectId, scope); + const run = (await this.repository.listRegulatoryMappingBatchRuns(projectId, scopeKey))[0]; + const availability = await this.availability(); + return { + version: 1, + projectId, + scope, + generationEnabled: availability.enabled, + state: run?.status ?? 'empty', + ...(!availability.enabled ? { disabledReason: availability.reason ?? this.options.disabledReason ?? 'Regulatory mapping is not configured.' } : {}), + ...(run && ['queued', 'running'].includes(run.status) ? { pollAfterMs: POLL_AFTER_MS } : {}), + ...(run ? { run } : {}) + }; + } + + async startBatch(projectId: string, request: RegulatoryMappingBatchCreateRequest): Promise { + if (!this.generationEnabled) throw new RegulatoryMappingServiceError('regulatory_mapping_disabled', this.options.disabledReason ?? 'Regulatory mapping is not configured.', 503); + if (typeof request.idempotencyKey !== 'string' || !/^[A-Za-z0-9._:-]{8,128}$/.test(request.idempotencyKey)) { + throw new RepositoryValidationError('idempotencyKey must contain 8 to 128 safe characters.'); + } + const scope = await this.validateBatchScope(projectId, request); + const timestamp = this.now().toISOString(); + const candidate: RegulatoryMappingBatchRun = { + version: 1, + id: `regulatory-mapping-batch-${randomUUID()}`, + projectId, + scopeKey: this.batchScopeKey(projectId, scope), + filterHash: sha256(canonicalJson(scope.filters)), + assessmentIds: scope.assessmentIds, + scanRunIds: scope.scanRunIds, + filters: scope.filters, + idempotencyKey: request.idempotencyKey, + status: 'queued', + counts: emptyRegulatoryMappingBatchCounts(), + createdAt: timestamp, + updatedAt: timestamp + }; + const run = await this.repository.createRegulatoryMappingBatchRun(candidate); + if (run.id === candidate.id) this.track(this.processBatch(run)); + return this.getBatch(projectId, { + assessmentIds: run.assessmentIds, + scanRunIds: run.scanRunIds, + filters: run.filters + }); + } + async decide(projectId: string, request: RegulatoryMappingDecisionCreateRequest): Promise { if (!this.generationEnabled) throw new RegulatoryMappingServiceError('regulatory_mapping_disabled', this.options.disabledReason ?? 'Regulatory mapping is not configured.', 503); const finding = await this.finding(projectId, request.resultProjectionId); @@ -182,7 +219,162 @@ export class RegulatoryMappingService { return `regulatory-mapping-${createHash('sha256').update(tuple, 'utf8').digest('hex').slice(0, 40)}`; } - async close(): Promise { await Promise.allSettled([...this.activeTasks]); } + async close(): Promise { + while (this.activeTasks.size) await Promise.allSettled([...this.activeTasks]); + } + + private track(task: Promise): void { + this.activeTasks.add(task); + void task.finally(() => this.activeTasks.delete(task)); + } + + private enqueue(task: () => Promise<'succeeded' | 'failed'>): Promise<'succeeded' | 'failed'> { + const promise = new Promise<'succeeded' | 'failed'>(resolve => this.workQueue.push({ task, resolve })); + this.drainQueue(); + return promise; + } + + private drainQueue(): void { + while (this.activeWorkerCount < this.maximumConcurrentRuns && this.workQueue.length) { + const work = this.workQueue.shift()!; + this.activeWorkerCount += 1; + void work.task().then(work.resolve, () => work.resolve('failed')).finally(() => { + this.activeWorkerCount -= 1; + this.drainQueue(); + }); + } + } + + private queuedRun(finding: UnifiedResultProjection): RegulatoryMappingRun { + const preparedQuery = buildRegulatoryFindingQuery(finding); + if (!preparedQuery) throw new RegulatoryMappingServiceError('insufficient_public_context', 'This finding has insufficient public context for regulatory mapping.', 422); + const timestamp = this.now().toISOString(); + const ruleKey = this.ruleKey(finding); + return { + version: 1, + id: this.mappingId(ruleKey), + ruleKey, + sourceTool: finding.sourceTool ?? 'unknown', + ...(finding.sourceRuleId ? { sourceRuleId: finding.sourceRuleId } : {}), + status: 'queued', + queryHash: sha256(JSON.stringify(preparedQuery)), + preparedQuery, + corpusId: this.options.corpusId!, + corpusVersion: this.options.corpusVersion!, + sourceHashes: [], + embeddingModel: this.options.embeddingModel!, + retrievalVersion: REGULATORY_RETRIEVAL_VERSION, + promptVersion: REGULATORY_MAPPING_PROMPT_VERSION, + outputSchemaVersion: REGULATORY_MAPPING_OUTPUT_SCHEMA_VERSION, + model: this.options.client!.model, + createdAt: timestamp, + updatedAt: timestamp, + candidates: [] + }; + } + + private async claim(finding: UnifiedResultProjection, forceRegeneration: boolean) { + return this.repository.claimRegulatoryMappingRun(this.queuedRun(finding), forceRegeneration); + } + + private async validateBatchScope(projectId: string, requested: RegulatoryMappingBatchScope): Promise { + if (!await this.repository.getProject(projectId)) throw new RepositoryNotFoundError('The selected project does not exist.'); + const scope = normalizeRegulatoryMappingBatchScope(requested); + if (!scope.assessmentIds.length && !scope.scanRunIds.length) throw new RepositoryValidationError('A selected timeslot scope is required.'); + const assessments = await this.repository.listAssessments(projectId); + const allowedAssessmentIds = new Set(assessments.map(item => item.id)); + if (scope.assessmentIds.some(id => !allowedAssessmentIds.has(id))) throw new RepositoryValidationError('The batch scope contains an assessment outside this project.'); + const scanRuns = await this.repository.listProjectScanRuns(projectId); + const allowedRuns = new Map(scanRuns.filter(item => !item.supersededByScanRunId).map(item => [item.id, item])); + if (scope.scanRunIds.some(id => !allowedRuns.has(id))) throw new RepositoryValidationError('The batch scope contains an unavailable scan run.'); + if (scope.assessmentIds.length && scope.scanRunIds.some(id => !scope.assessmentIds.includes(allowedRuns.get(id)!.assessmentId))) { + throw new RepositoryValidationError('The assessment and scan-run scope do not describe the same timeslot.'); + } + return scope; + } + + private batchScopeKey(projectId: string, scope: RegulatoryMappingBatchScope): string { + return sha256(canonicalJson({ projectId, ...scope })); + } + + private async queryBatchResults(projectId: string, scope: RegulatoryMappingBatchScope): Promise { + const query = { + ...scope.filters, + audience: 'result-db' as const, + projectId, + assessmentIds: scope.assessmentIds, + scanRunIds: scope.scanRunIds + }; + const items: UnifiedResultProjection[] = []; + let expectedTotal: number | undefined; + for (let offset = 0; ; offset += BATCH_PAGE_SIZE) { + const page = await this.repository.queryUnifiedResults({ ...query, offset, limit: BATCH_PAGE_SIZE }); + expectedTotal ??= page.total; + if (page.total !== expectedTotal) throw new RepositoryConflictError('The filtered result set changed while the ISO mapping batch was being prepared.'); + items.push(...page.items); + if (items.length >= page.total || page.items.length === 0) break; + } + if (items.length !== expectedTotal) throw new RepositoryConflictError('The complete filtered result set could not be read consistently.'); + return items; + } + + private async processBatch(queued: RegulatoryMappingBatchRun): Promise { + const startedAt = this.now().toISOString(); + let run = await this.repository.updateRegulatoryMappingBatchRun({ ...queued, status: 'running', startedAt, updatedAt: startedAt }); + try { + const results = (await this.queryBatchResults(run.projectId, run)).slice() + .sort((left, right) => left.resultProjectionId.localeCompare(right.resultProjectionId)); + const representatives = new Map(); + const counts = { ...emptyRegulatoryMappingBatchCounts(), filteredResultCount: results.length }; + for (const finding of results) { + if (finding.sourceKind !== 'scanner-finding') { counts.excludedNonScannerCount += 1; continue; } + if (!buildRegulatoryFindingQuery(finding)) { counts.unsupportedFindingCount += 1; continue; } + counts.eligibleFindingCount += 1; + const ruleKey = this.ruleKey(finding); + if (!representatives.has(ruleKey)) representatives.set(ruleKey, finding); + } + counts.distinctRuleCount = representatives.size; + const scheduled: Array> = []; + for (const finding of representatives.values()) { + const claimed = await this.claim(finding, false); + if (!claimed.claimed) { + if (claimed.run.status === 'succeeded') counts.reusedSucceededRuleCount += 1; + else counts.reusedActiveRuleCount += 1; + continue; + } + counts.scheduledRuleCount += 1; + const completion = this.enqueue(() => this.process(claimed.run, finding)); + this.track(completion.then(() => undefined)); + scheduled.push(completion); + } + run = await this.repository.updateRegulatoryMappingBatchRun({ ...run, counts, updatedAt: this.now().toISOString() }); + let progress = Promise.resolve(); + const completions = scheduled.map(completion => completion.then(status => { + progress = progress.then(async () => { + if (status === 'succeeded') counts.succeededRuleCount += 1; + else counts.failedRuleCount += 1; + run = await this.repository.updateRegulatoryMappingBatchRun({ ...run, counts: { ...counts }, updatedAt: this.now().toISOString() }); + }); + return progress; + })); + await Promise.all(completions); + await progress; + const completedAt = this.now().toISOString(); + await this.repository.updateRegulatoryMappingBatchRun({ ...run, status: 'completed', counts, updatedAt: completedAt, completedAt }); + } catch (error) { + const completedAt = this.now().toISOString(); + try { + await this.repository.updateRegulatoryMappingBatchRun({ + ...run, + status: 'failed', + updatedAt: completedAt, + completedAt, + errorCode: error instanceof RepositoryConflictError ? 'regulatory_mapping_batch_scope_changed' : 'regulatory_mapping_batch_failed', + errorDetail: error instanceof RepositoryConflictError ? error.message : 'The batch stopped safely before all rules were scheduled.' + }); + } catch { /* Preserve the original safe batch failure. */ } + } + } private async availability(): Promise<{ enabled: boolean; reason?: string }> { if (!this.generationEnabled) return { enabled: false, reason: this.options.disabledReason ?? 'Regulatory mapping is not configured.' }; @@ -241,10 +433,36 @@ export class RegulatoryMappingService { })); } - private async process(queued: RegulatoryMappingRun): Promise { - const startedAt = this.now().toISOString(); - const run = await this.repository.saveRegulatoryMappingRun({ ...queued, status: 'running', startedAt, updatedAt: startedAt }); + private async process(queued: RegulatoryMappingRun, finding: UnifiedResultProjection): Promise<'succeeded' | 'failed'> { + let run = queued; try { + let retrieval; + try { + retrieval = await this.options.findingRetrieval!.retrieve(finding, 32, { + corpusId: this.options.corpusId, + corpusVersion: this.options.corpusVersion, + sourceStandard: 'ISO/IEC 27002', + retrievalRole: 'semantic' + }); + } catch { + throw new RegulatoryMappingServiceError('regulatory_retrieval_unavailable', 'Regulatory retrieval is currently unavailable.', 503); + } + if (retrieval.status !== 'candidates' || !retrieval.preparedQuery) { + throw new RegulatoryMappingServiceError('insufficient_public_context', 'This finding has insufficient public context for regulatory mapping.', 422); + } + const candidates = await this.groupCandidates(retrieval.candidates); + if (!candidates.length) throw new RegulatoryMappingServiceError('regulatory_corpus_empty', 'The active regulatory corpus returned no eligible controls.', 503); + const startedAt = this.now().toISOString(); + run = await this.repository.saveRegulatoryMappingRun({ + ...queued, + status: 'running', + preparedQuery: retrieval.preparedQuery, + queryHash: sha256(JSON.stringify(retrieval.preparedQuery)), + sourceHashes: [...new Set(candidates.flatMap(candidate => candidate.evidence.map(evidence => evidence.sourceSha256)))].sort(), + candidates, + startedAt, + updatedAt: startedAt + }); const prompt = mappingPrompt(run); const completion = await this.options.client!.complete(SYSTEM_PROMPT, prompt); const allowed = new Map(run.candidates.map(candidate => [ @@ -262,6 +480,7 @@ export class RegulatoryMappingService { updatedAt: completedAt, completedAt }); + return 'succeeded'; } catch (error) { const completedAt = this.now().toISOString(); const failure = safeFailure(error); @@ -272,6 +491,7 @@ export class RegulatoryMappingService { ...(detail ? { errorDetail: detail } : {}) }); } catch { /* Preserve the original safe failure code. */ } + return 'failed'; } } } @@ -329,3 +549,7 @@ function safeFailureDetail(error: unknown): string | undefined { const detail = error.message.replace(/<\/?[a-z][^>]*>/gi, ' ').replace(/\s+/g, ' ').trim().slice(0, 600); return detail || undefined; } + +function sha256(value: string): string { + return createHash('sha256').update(value, 'utf8').digest('hex'); +} diff --git a/src/api/assessor-api-client.ts b/src/api/assessor-api-client.ts index 04b7013..3ff19c3 100644 --- a/src/api/assessor-api-client.ts +++ b/src/api/assessor-api-client.ts @@ -55,6 +55,9 @@ import { type MicrosoftSecurityBaselineAnalysisListResponse, type AiOverviewCreateRequest, type AiOverviewResponse, + type RegulatoryMappingBatchCreateRequest, + type RegulatoryMappingBatchResponse, + type RegulatoryMappingBatchScope, type RegulatoryMappingCreateRequest, type RegulatoryMappingDecisionCreateRequest, type RegulatoryMappingResponse @@ -253,6 +256,20 @@ export class AssessorApiClient { }); } + getRegulatoryMappingBatch(projectId: string, scope: RegulatoryMappingBatchScope): Promise { + return this.request(unifiedQueryPath(`/projects/${encodeURIComponent(projectId)}/regulatory-mapping-batches`, { + ...scope.filters, + assessmentIds: scope.assessmentIds, + scanRunIds: scope.scanRunIds + })); + } + + startRegulatoryMappingBatch(projectId: string, request: RegulatoryMappingBatchCreateRequest): Promise { + return this.request(`/projects/${encodeURIComponent(projectId)}/regulatory-mapping-batches`, { + method: 'POST', body: JSON.stringify(request) + }); + } + listProjectImportFiles(projectId: string): Promise { return this.request(`/projects/${encodeURIComponent(projectId)}/import-files`); } diff --git a/src/api/assessor-api-contract.ts b/src/api/assessor-api-contract.ts index c72d12a..2d8ed7b 100644 --- a/src/api/assessor-api-contract.ts +++ b/src/api/assessor-api-contract.ts @@ -21,6 +21,9 @@ import type { RemediationResult } from '../remediation/remediation-engine'; export type WorkPackagesResponse = { status: 'ready'; result: RemediationResult } | { status: 'context-required' }; export type { AiOverviewCreateRequest, AiOverviewResponse } from '../domain/ai-overview'; export type { + RegulatoryMappingBatchCreateRequest, + RegulatoryMappingBatchResponse, + RegulatoryMappingBatchScope, RegulatoryMappingCreateRequest, RegulatoryMappingDecisionCreateRequest, RegulatoryMappingResponse diff --git a/src/domain/regulatory-mapping.ts b/src/domain/regulatory-mapping.ts index c1a0cca..d31de1e 100644 --- a/src/domain/regulatory-mapping.ts +++ b/src/domain/regulatory-mapping.ts @@ -1,5 +1,6 @@ import type { AiOverviewTokenUsage } from './ai-overview'; import type { RegulatoryFindingQuery, RegulatorySectionKind } from './regulatory-rag'; +import type { UnifiedResultQuery } from './unified-result'; export const REGULATORY_MAPPING_PROMPT_VERSION = '1.2.0'; export const REGULATORY_MAPPING_OUTPUT_SCHEMA_VERSION = 1 as const; @@ -9,6 +10,61 @@ export type RegulatoryMappingRunStatus = 'queued' | 'running' | 'succeeded' | 'f export type RegulatoryMappingRelationship = 'primary' | 'supporting'; export type RegulatoryMappingConfidence = 'low' | 'medium' | 'high'; export type RegulatoryMappingDecisionValue = 'accepted' | 'rejected'; +export type RegulatoryMappingBatchStatus = 'queued' | 'running' | 'completed' | 'failed'; + +export type RegulatoryMappingBatchFilters = Omit; + +export interface RegulatoryMappingBatchScope { + assessmentIds: string[]; + scanRunIds: string[]; + filters: RegulatoryMappingBatchFilters; +} + +export interface RegulatoryMappingBatchCounts { + filteredResultCount: number; + excludedNonScannerCount: number; + eligibleFindingCount: number; + unsupportedFindingCount: number; + distinctRuleCount: number; + reusedSucceededRuleCount: number; + reusedActiveRuleCount: number; + scheduledRuleCount: number; + succeededRuleCount: number; + failedRuleCount: number; +} + +export interface RegulatoryMappingBatchRun extends RegulatoryMappingBatchScope { + version: 1; + id: string; + projectId: string; + scopeKey: string; + filterHash: string; + idempotencyKey: string; + status: RegulatoryMappingBatchStatus; + counts: RegulatoryMappingBatchCounts; + createdAt: string; + updatedAt: string; + startedAt?: string; + completedAt?: string; + errorCode?: string; + errorDetail?: string; +} + +export interface RegulatoryMappingBatchResponse { + version: 1; + projectId: string; + scope: RegulatoryMappingBatchScope; + generationEnabled: boolean; + state: 'empty' | RegulatoryMappingBatchStatus; + disabledReason?: string; + pollAfterMs?: number; + run?: RegulatoryMappingBatchRun; +} + +export interface RegulatoryMappingBatchCreateRequest extends RegulatoryMappingBatchScope { + idempotencyKey: string; +} export interface RegulatoryMappingEvidence { chunkId: string; @@ -185,8 +241,9 @@ export function regulatoryMappingRunValidationErrors(run: RegulatoryMappingRun): if (run.sourceRuleId !== undefined && (!run.sourceRuleId.trim() || run.sourceRuleId.length > 200)) errors.push('Regulatory mapping sourceRuleId is invalid.'); if (!['queued', 'running', 'succeeded', 'failed'].includes(run.status)) errors.push('Regulatory mapping run status is invalid.'); if (!/^[a-f0-9]{64}$/.test(run.queryHash)) errors.push('Regulatory mapping queryHash must be SHA-256.'); - if (!run.sourceHashes.length || run.sourceHashes.some(value => !/^[a-f0-9]{64}$/.test(value))) errors.push('Regulatory mapping sourceHashes must contain SHA-256 digests.'); - if (!Array.isArray(run.candidates) || run.candidates.length < 1 || run.candidates.length > 8) errors.push('Regulatory mapping candidates must contain one to eight controls.'); + const candidatesRequired = run.status === 'running' || run.status === 'succeeded'; + if ((!run.sourceHashes.length && candidatesRequired) || run.sourceHashes.some(value => !/^[a-f0-9]{64}$/.test(value))) errors.push('Regulatory mapping sourceHashes must contain SHA-256 digests.'); + if (!Array.isArray(run.candidates) || (candidatesRequired && run.candidates.length < 1) || run.candidates.length > 8) errors.push('Regulatory mapping candidates must contain one to eight controls when processing has started.'); else { const controls = new Set(); run.candidates.forEach(candidate => { @@ -207,6 +264,85 @@ export function regulatoryMappingRunValidationErrors(run: RegulatoryMappingRun): return [...new Set(errors)]; } +export function normalizeRegulatoryMappingBatchScope(scope: RegulatoryMappingBatchScope): RegulatoryMappingBatchScope { + const values = (input: readonly string[] | undefined) => [...new Set((input ?? []).map(value => value.trim()).filter(Boolean))] + .sort((left, right) => left.localeCompare(right)); + const text = (input: string | undefined) => input?.trim() || undefined; + const filters = scope.filters ?? {}; + return { + assessmentIds: values(scope.assessmentIds), + scanRunIds: values(scope.scanRunIds), + filters: { + evaluationRunId: text(filters.evaluationRunId), + sourceKind: values(filters.sourceKind) as RegulatoryMappingBatchFilters['sourceKind'], + sourceTool: values(filters.sourceTool), + severity: values(filters.severity) as RegulatoryMappingBatchFilters['severity'], + outcome: values(filters.outcome) as RegulatoryMappingBatchFilters['outcome'], + technicalExecutionState: values(filters.technicalExecutionState) as RegulatoryMappingBatchFilters['technicalExecutionState'], + resultType: values(filters.resultType) as RegulatoryMappingBatchFilters['resultType'], + policyClassification: values(filters.policyClassification) as RegulatoryMappingBatchFilters['policyClassification'], + isIoe: filters.isIoe, + deviation: filters.deviation, + mappingRuleVersion: values(filters.mappingRuleVersion), + sourceProfileVersion: values(filters.sourceProfileVersion), + processingStatus: values(filters.processingStatus) as RegulatoryMappingBatchFilters['processingStatus'], + reviewDisposition: values(filters.reviewDisposition) as RegulatoryMappingBatchFilters['reviewDisposition'], + lifecycleState: values(filters.lifecycleState) as RegulatoryMappingBatchFilters['lifecycleState'], + categoryNamespace: values(filters.categoryNamespace), + categoryValue: values(filters.categoryValue), + affectedAsset: text(filters.affectedAsset), + environmentState: values(filters.environmentState) as RegulatoryMappingBatchFilters['environmentState'], + environmentProperty: values(filters.environmentProperty), + environmentSubjectType: values(filters.environmentSubjectType), + search: text(filters.search), + observedFrom: text(filters.observedFrom), + observedTo: text(filters.observedTo) + } + }; +} + +export function emptyRegulatoryMappingBatchCounts(): RegulatoryMappingBatchCounts { + return { + filteredResultCount: 0, + excludedNonScannerCount: 0, + eligibleFindingCount: 0, + unsupportedFindingCount: 0, + distinctRuleCount: 0, + reusedSucceededRuleCount: 0, + reusedActiveRuleCount: 0, + scheduledRuleCount: 0, + succeededRuleCount: 0, + failedRuleCount: 0 + }; +} + +export function regulatoryMappingBatchRunValidationErrors(run: RegulatoryMappingBatchRun): string[] { + const errors: string[] = []; + for (const value of [run.id, run.projectId, run.scopeKey, run.filterHash, run.idempotencyKey]) { + if (!value?.trim()) errors.push('Regulatory mapping batch identity is incomplete.'); + } + if (!/^[a-f0-9]{64}$/.test(run.filterHash)) errors.push('Regulatory mapping batch filterHash must be SHA-256.'); + if (!/^[A-Za-z0-9._:-]{8,128}$/.test(run.idempotencyKey)) errors.push('Regulatory mapping batch idempotencyKey is invalid.'); + if (!['queued', 'running', 'completed', 'failed'].includes(run.status)) errors.push('Regulatory mapping batch status is invalid.'); + const normalized = normalizeRegulatoryMappingBatchScope(run); + if (!normalized.assessmentIds.length && !normalized.scanRunIds.length) errors.push('Regulatory mapping batch requires a selected timeslot.'); + const counts = Object.values(run.counts); + if (counts.some(value => !Number.isInteger(value) || value < 0)) errors.push('Regulatory mapping batch counts must be non-negative integers.'); + if (run.counts.filteredResultCount !== run.counts.excludedNonScannerCount + run.counts.eligibleFindingCount + run.counts.unsupportedFindingCount) { + errors.push('Regulatory mapping batch result counts are inconsistent.'); + } + if (run.counts.distinctRuleCount !== run.counts.reusedSucceededRuleCount + run.counts.reusedActiveRuleCount + run.counts.scheduledRuleCount) { + errors.push('Regulatory mapping batch rule counts are inconsistent.'); + } + if (run.counts.succeededRuleCount + run.counts.failedRuleCount > run.counts.scheduledRuleCount) { + errors.push('Regulatory mapping batch completion counts exceed scheduled rules.'); + } + if (run.errorDetail !== undefined && (run.errorDetail.length > 600 || /<\/?[a-z][^>]*>/i.test(run.errorDetail))) { + errors.push('Regulatory mapping batch errorDetail must be plain text with at most 600 characters.'); + } + return [...new Set(errors)]; +} + export function regulatoryMappingDecisionValidationErrors(decision: RegulatoryMappingDecision): string[] { const errors: string[] = []; for (const value of [decision.id, decision.ruleKey, decision.sourceTool, decision.mappingId, decision.controlId]) { diff --git a/src/domain/result-db-query.ts b/src/domain/result-db-query.ts new file mode 100644 index 0000000..0c956a5 --- /dev/null +++ b/src/domain/result-db-query.ts @@ -0,0 +1,27 @@ +import type { RegulatoryMappingBatchFilters } from './regulatory-mapping'; +import type { UnifiedResultQuery } from './unified-result'; + +export function buildScopedResultDbQuery( + filters: UnifiedResultQuery, + projectId: string, + scanRunIds: readonly string[] = [], + assessmentIds: readonly string[] = [] +): UnifiedResultQuery { + const { audience: _audience, projectId: _projectId, assessmentId: _assessmentId, + assessmentIds: _assessmentIds, scanRunId: _scanRunId, scanRunIds: _scanRunIds, + offset: _offset, limit: _limit, ...filterValues } = filters; + return { + ...filterValues, + audience: 'result-db', + projectId, + assessmentIds: [...assessmentIds], + scanRunIds: [...scanRunIds] + }; +} + +export function resultDbBatchFilters(query: UnifiedResultQuery): RegulatoryMappingBatchFilters { + const { audience: _audience, projectId: _projectId, assessmentId: _assessmentId, + assessmentIds: _assessmentIds, scanRunId: _scanRunId, scanRunIds: _scanRunIds, + offset: _offset, limit: _limit, ...filters } = query; + return filters; +} diff --git a/src/features/results-browser/RegulatoryMappingBatchDialog.tsx b/src/features/results-browser/RegulatoryMappingBatchDialog.tsx new file mode 100644 index 0000000..b1f2997 --- /dev/null +++ b/src/features/results-browser/RegulatoryMappingBatchDialog.tsx @@ -0,0 +1,124 @@ +import { useCallback, useEffect, useRef, useState } from 'react'; +import { AssessorApiClient } from '../../api/assessor-api-client'; +import { Dialog } from '../../components/Dialog/Dialog'; +import type { RegulatoryMappingBatchResponse, RegulatoryMappingBatchScope } from '../../domain/regulatory-mapping'; + +const assessorApi = new AssessorApiClient(); + +export interface RegulatoryMappingBatchController { + response?: RegulatoryMappingBatchResponse; + loading: boolean; + starting: boolean; + error?: string; + start(): Promise; +} + +export function useRegulatoryMappingBatch(projectId: string | undefined, scope: RegulatoryMappingBatchScope | undefined): RegulatoryMappingBatchController { + const [response, setResponse] = useState(); + const [loading, setLoading] = useState(false); + const [starting, setStarting] = useState(false); + const [error, setError] = useState(); + const scopeKey = JSON.stringify(scope ?? null); + const requestKey = `${projectId ?? ''}\u0000${scopeKey}`; + const activeRequestKey = useRef(requestKey); + activeRequestKey.current = requestKey; + + const refresh = useCallback(async () => { + if (!projectId || !scope || (!scope.assessmentIds.length && !scope.scanRunIds.length)) { + setResponse(undefined); + return; + } + try { + const next = await assessorApi.getRegulatoryMappingBatch(projectId, scope); + if (activeRequestKey.current !== requestKey) return; + setResponse(next); + setError(undefined); + } catch (reason) { + if (activeRequestKey.current !== requestKey) return; + setError(reason instanceof Error ? reason.message : 'ISO mapping batch status could not be loaded.'); + } + }, [projectId, requestKey, scopeKey]); // eslint-disable-line react-hooks/exhaustive-deps + + useEffect(() => { + let active = true; + setLoading(true); + setStarting(false); + setError(undefined); + setResponse(undefined); + void refresh().finally(() => { if (active) setLoading(false); }); + return () => { active = false; }; + }, [refresh, scopeKey]); + + useEffect(() => { + if (!response || !['queued', 'running'].includes(response.state)) return; + const timer = window.setTimeout(() => void refresh(), response.pollAfterMs ?? 1_500); + return () => window.clearTimeout(timer); + }, [refresh, response]); + + const start = useCallback(async () => { + if (!projectId || !scope) return; + setStarting(true); + setError(undefined); + try { + const next = await assessorApi.startRegulatoryMappingBatch(projectId, { + ...scope, + idempotencyKey: `regulatory-batch-${crypto.randomUUID()}` + }); + if (activeRequestKey.current !== requestKey) return; + setResponse(next); + } catch (reason) { + if (activeRequestKey.current !== requestKey) return; + setError(reason instanceof Error ? reason.message : 'ISO mapping batch could not be started.'); + } finally { + if (activeRequestKey.current === requestKey) setStarting(false); + } + }, [projectId, requestKey, scopeKey]); // eslint-disable-line react-hooks/exhaustive-deps + + return { response, loading, starting, error, start }; +} + +export function RegulatoryMappingBatchDialog({ controller, filteredResultCount, onClose }: { + controller: RegulatoryMappingBatchController; + filteredResultCount: number; + onClose(): void; +}) { + const { response, loading, starting, error, start } = controller; + const run = response?.run; + const active = response && ['queued', 'running'].includes(response.state); + const counts = run?.counts; + const actionLabel = run ? 'Run again for current filters' : 'Start mapping batch'; + return +
+

Generate ISO control proposals for all {filteredResultCount.toLocaleString('en-US')} results matching the current Result DB filters.

+

Pagination, “Load more,” and loaded-row limits do not restrict this batch. Proposals remain subject to manual Accept/Reject review.

+ {loading && !response &&

Loading batch status…

} + {error &&

{error}

} + {response && !response.generationEnabled &&

{response.disabledReason ?? 'Regulatory mapping is unavailable.'}

} + {run && <> +

Status: {run.status === 'completed' && run.counts.failedRuleCount ? 'completed with failures' : run.status}

+
+ + + + + + + + +
+ {run.errorDetail &&

{run.errorDetail}

} + } + {active &&

Mapping is running on the server. You can close this dialog and return later.

} +
+ + {response?.generationEnabled && } +
+
+ ; +} + +function BatchCount({ label, value }: { label: string; value: number }) { + return
{label}
{value.toLocaleString('en-US')}
; +} diff --git a/src/features/results-browser/ResultsBrowser.tsx b/src/features/results-browser/ResultsBrowser.tsx index 37b39dc..95f6f33 100644 --- a/src/features/results-browser/ResultsBrowser.tsx +++ b/src/features/results-browser/ResultsBrowser.tsx @@ -1,6 +1,6 @@ import { beginActivity, showActivityFrame } from '../../platform/activity-store'; import { buildUnifiedResultCsv, unifiedResultCsvFileName } from '../../exports/unified-result-csv'; -import { useCallback, useEffect, useState } from 'react'; +import { useCallback, useEffect, useMemo, useState } from 'react'; import { AssessorApiClient } from '../../api/assessor-api-client'; import { resultSourceToolLabel, type UnifiedResultFacetResult, type UnifiedResultProjection, type UnifiedResultQuery, type UnifiedResultQueryResult } from '../../domain/unified-result'; import { discloseResultDbProjection, removeResultDbScopeIdentifiers } from '../../domain/result-db-disclosure'; @@ -17,6 +17,8 @@ import { POLICY_CLASSIFICATION_LABELS } from '../../domain/policy-status-labels' import { BackendConnectionDialog } from '../../components/BackendConnectionDialog/BackendConnectionDialog'; import { formatPolicyComparisonValue } from '../../profiles/group-policy/presentation'; import type { ResultDbMaterializationStatus } from '../../domain/result-db-materialization'; +import { buildScopedResultDbQuery, resultDbBatchFilters } from '../../domain/result-db-query'; +import { RegulatoryMappingBatchDialog, useRegulatoryMappingBatch } from './RegulatoryMappingBatchDialog'; const assessorApi = new AssessorApiClient(); const PAGE_SIZE = 50; @@ -45,7 +47,17 @@ export function ResultsBrowser({ backendStatus, projectId, scanRunIds, assessmen const [exporting, setExporting] = useState<'jsonl' | 'excel' | 'csv' | 'json' | null>(null); const [error, setError] = useState(null); const [expandedId, setExpandedId] = useState(null); + const [batchDialogOpen, setBatchDialogOpen] = useState(false); const hasTimeslotScope = hasSelectedTimeslotScope(scanRunIds, assessmentIds); + const scopedQuery = useMemo(() => projectId && hasTimeslotScope + ? buildScopedResultDbQuery(filters, projectId, scanRunIds, assessmentIds) + : undefined, [assessmentIds, filters, hasTimeslotScope, projectId, scanRunIds]); + const batchScope = useMemo(() => scopedQuery ? { + assessmentIds: scopedQuery.assessmentIds ?? [], + scanRunIds: scopedQuery.scanRunIds ?? [], + filters: resultDbBatchFilters(scopedQuery) + } : undefined, [scopedQuery]); + const regulatoryBatch = useRegulatoryMappingBatch(backendStatus === 'connected' ? projectId : undefined, batchScope); const load = useCallback(async () => { if (backendStatus !== 'connected' || !projectId || !hasTimeslotScope) { @@ -57,12 +69,7 @@ export function ResultsBrowser({ backendStatus, projectId, scanRunIds, assessmen setLoading(true); setError(null); try { - const scopedQuery: UnifiedResultQuery = { - ...filters, - projectId, - scanRunIds, - assessmentIds - }; + if (!scopedQuery) return; const [facets, firstPage] = await Promise.all([ assessorApi.queryUnifiedResultFacets(scopedQuery), assessorApi.queryUnifiedResults({ ...scopedQuery, offset: 0, limit }) @@ -86,7 +93,7 @@ export function ResultsBrowser({ backendStatus, projectId, scanRunIds, assessmen } finally { setLoading(false); } - }, [backendStatus, projectId, scanRunIds, assessmentIds, hasTimeslotScope, filters, limit, loadAll]); + }, [backendStatus, projectId, hasTimeslotScope, scopedQuery, limit, loadAll]); useEffect(() => { void load(); }, [load]); @@ -102,7 +109,7 @@ export function ResultsBrowser({ backendStatus, projectId, scanRunIds, assessmen setExporting(format); setError(null); try { - const scopedQuery: UnifiedResultQuery = { ...filters, projectId, scanRunIds, assessmentIds }; + if (!scopedQuery) return; const results = await queryAllUnifiedResults(query => assessorApi.queryUnifiedResults(query), scopedQuery); activity.update('Preparing the filtered export file'); await showActivityFrame(); @@ -171,6 +178,10 @@ export function ResultsBrowser({ backendStatus, projectId, scanRunIds, assessmen

{(queryResult?.total ?? 0).toLocaleString('en-US')} hits{queryResult && queryResult.total > queryResult.items.length ? ` · ${queryResult.items.length.toLocaleString('en-US')} loaded` : ''}{resultDbStatus && <>
{resultDbStatusText(resultDbStatus)}}

+
} + {batchDialogOpen && setBatchDialogOpen(false)}/>} ; } diff --git a/src/persistence/assessor-repository.ts b/src/persistence/assessor-repository.ts index 53e7e35..c2d61a4 100644 --- a/src/persistence/assessor-repository.ts +++ b/src/persistence/assessor-repository.ts @@ -35,7 +35,7 @@ import type { } from '../domain/unified-result'; import type { EnvironmentObservation, EnvironmentSnapshot, EnvironmentSnapshotQuery, ScannerReportSection } from '../domain/environment-context'; import type { AiOverviewRun } from '../domain/ai-overview'; -import type { RegulatoryMappingDecision, RegulatoryMappingRun } from '../domain/regulatory-mapping'; +import type { RegulatoryMappingBatchRun, RegulatoryMappingDecision, RegulatoryMappingRun } from '../domain/regulatory-mapping'; export interface FindingOccurrenceQuery { projectId?: string; @@ -147,6 +147,7 @@ export interface ProjectDataExport { environmentProfiles: EnvironmentProfile[]; policySnapshotAggregations: StoredPolicySnapshotAggregation[]; aiOverviewRuns: AiOverviewRun[]; + regulatoryMappingBatchRuns: RegulatoryMappingBatchRun[]; regulatoryMappingRuns: RegulatoryMappingRun[]; regulatoryMappingDecisions: RegulatoryMappingDecision[]; } @@ -226,8 +227,12 @@ export interface AssessorRepository { updateAiOverviewRun(run: AiOverviewRun): Promise; listAiOverviewRuns(projectId: string, scopeKey?: string): Promise; getRegulatoryMappingRun(id: string): Promise; + claimRegulatoryMappingRun(run: RegulatoryMappingRun, forceRegeneration: boolean): Promise<{ run: RegulatoryMappingRun; claimed: boolean }>; saveRegulatoryMappingRun(run: RegulatoryMappingRun): Promise; listRegulatoryMappingRuns(): Promise; + createRegulatoryMappingBatchRun(run: RegulatoryMappingBatchRun): Promise; + updateRegulatoryMappingBatchRun(run: RegulatoryMappingBatchRun): Promise; + listRegulatoryMappingBatchRuns(projectId: string, scopeKey?: string): Promise; createRegulatoryMappingDecision(decision: RegulatoryMappingDecision, expectedLatestDecisionId?: string): Promise; listRegulatoryMappingDecisions(ruleKey: string): Promise; diff --git a/src/persistence/local-assessor-repository.ts b/src/persistence/local-assessor-repository.ts index 1508ac1..b867c1a 100644 --- a/src/persistence/local-assessor-repository.ts +++ b/src/persistence/local-assessor-repository.ts @@ -27,8 +27,8 @@ import type { EnvironmentObservation, EnvironmentSnapshot, EnvironmentSnapshotQu import { environmentCoverage } from '../domain/environment-context'; import type { AiOverviewRun } from '../domain/ai-overview'; import { aiOverviewRunValidationErrors } from '../domain/ai-overview'; -import type { RegulatoryMappingDecision, RegulatoryMappingRun } from '../domain/regulatory-mapping'; -import { regulatoryMappingDecisionValidationErrors, regulatoryMappingRunValidationErrors } from '../domain/regulatory-mapping'; +import type { RegulatoryMappingBatchRun, RegulatoryMappingDecision, RegulatoryMappingRun } from '../domain/regulatory-mapping'; +import { regulatoryMappingBatchRunValidationErrors, regulatoryMappingDecisionValidationErrors, regulatoryMappingRunValidationErrors } from '../domain/regulatory-mapping'; import { storedPolicyFindingRunValidationErrors } from '../domain/policy-finding'; import { environmentProfileValidationErrors, @@ -105,6 +105,7 @@ interface LocalRepositoryState { findingDefinitions: FindingDefinition[]; findingOccurrences: FindingOccurrence[]; aiOverviewRuns: AiOverviewRun[]; + regulatoryMappingBatchRuns: RegulatoryMappingBatchRun[]; regulatoryMappingRuns: RegulatoryMappingRun[]; regulatoryMappingDecisions: RegulatoryMappingDecision[]; evidence: EvidenceRecord[]; @@ -142,6 +143,7 @@ const emptyState = (): LocalRepositoryState => ({ findingDefinitions: [], findingOccurrences: [], aiOverviewRuns: [], + regulatoryMappingBatchRuns: [], regulatoryMappingRuns: [], regulatoryMappingDecisions: [], evidence: [], @@ -437,6 +439,17 @@ export class LocalAssessorRepository implements AssessorRepository { return run ? clone(run) : null; } + async claimRegulatoryMappingRun(run: RegulatoryMappingRun, forceRegeneration: boolean): Promise<{ run: RegulatoryMappingRun; claimed: boolean }> { + const errors = regulatoryMappingRunValidationErrors(run); + if (errors.length) throw new RepositoryValidationError(errors.join(' ')); + const existing = this.state.regulatoryMappingRuns.find(item => item.id === run.id); + if (existing && (['queued', 'running'].includes(existing.status) || existing.status === 'succeeded' && !forceRegeneration)) { + return { run: clone(existing), claimed: false }; + } + const stored = await this.saveRegulatoryMappingRun(run); + return { run: stored, claimed: true }; + } + async saveRegulatoryMappingRun(run: RegulatoryMappingRun): Promise { const errors = regulatoryMappingRunValidationErrors(run); if (errors.length) throw new RepositoryValidationError(errors.join(' ')); @@ -453,6 +466,42 @@ export class LocalAssessorRepository implements AssessorRepository { .sort((left, right) => right.createdAt.localeCompare(left.createdAt) || right.id.localeCompare(left.id)); } + async createRegulatoryMappingBatchRun(run: RegulatoryMappingBatchRun): Promise { + const errors = regulatoryMappingBatchRunValidationErrors(run); + if (errors.length) throw new RepositoryValidationError(errors.join(' ')); + const idempotent = this.state.regulatoryMappingBatchRuns.find(item => item.projectId === run.projectId && item.idempotencyKey === run.idempotencyKey); + if (idempotent) return clone(idempotent); + const active = this.state.regulatoryMappingBatchRuns.find(item => item.projectId === run.projectId && item.scopeKey === run.scopeKey + && ['queued', 'running'].includes(item.status)); + if (active) return clone(active); + const next = clone(this.state); + next.regulatoryMappingBatchRuns.push(clone(run)); + this.replaceState(next); + return clone(run); + } + + async updateRegulatoryMappingBatchRun(run: RegulatoryMappingBatchRun): Promise { + const errors = regulatoryMappingBatchRunValidationErrors(run); + if (errors.length) throw new RepositoryValidationError(errors.join(' ')); + const existing = this.state.regulatoryMappingBatchRuns.find(item => item.id === run.id); + if (!existing) throw new RepositoryNotFoundError('Regulatory mapping batch does not exist.'); + if (existing.projectId !== run.projectId || existing.scopeKey !== run.scopeKey || existing.idempotencyKey !== run.idempotencyKey) { + throw new RepositoryConflictError('Regulatory mapping batch identity is immutable.'); + } + if (['completed', 'failed'].includes(existing.status)) throw new RepositoryConflictError('A completed regulatory mapping batch is immutable.'); + const allowed = existing.status === 'queued' ? ['running', 'failed'] : ['running', 'completed', 'failed']; + if (!allowed.includes(run.status)) throw new RepositoryConflictError(`Invalid regulatory mapping batch transition from ${existing.status} to ${run.status}.`); + const next = clone(this.state); + next.regulatoryMappingBatchRuns[next.regulatoryMappingBatchRuns.findIndex(item => item.id === run.id)] = clone(run); + this.replaceState(next); + return clone(run); + } + + async listRegulatoryMappingBatchRuns(projectId: string, scopeKey?: string): Promise { + return this.state.regulatoryMappingBatchRuns.filter(item => item.projectId === projectId && (!scopeKey || item.scopeKey === scopeKey)) + .map(clone).reverse(); + } + async createRegulatoryMappingDecision(decision: RegulatoryMappingDecision, expectedLatestDecisionId?: string): Promise { const errors = regulatoryMappingDecisionValidationErrors(decision); if (errors.length) throw new RepositoryValidationError(errors.join(' ')); @@ -1591,6 +1640,7 @@ export class LocalAssessorRepository implements AssessorRepository { policyEvaluations: this.state.policyEvaluations.filter(item => item.run.projectId === projectId).map(clone), policyFindingRuns: policyFindingRuns.map(clone), aiOverviewRuns: this.state.aiOverviewRuns.filter(item => item.projectId === projectId && item.status === 'succeeded').map(clone), + regulatoryMappingBatchRuns: this.state.regulatoryMappingBatchRuns.filter(item => item.projectId === projectId).map(clone), // Regulatory mappings and decisions are global reference data, not project-scoped. regulatoryMappingRuns: this.state.regulatoryMappingRuns.filter(item => item.status === 'succeeded').map(clone), regulatoryMappingDecisions: this.state.regulatoryMappingDecisions.map(clone) @@ -1781,6 +1831,7 @@ export class LocalAssessorRepository implements AssessorRepository { next.policySnapshotAggregations = next.policySnapshotAggregations.filter(item => item.run.projectId !== projectId); next.policyEvaluations = next.policyEvaluations.filter(item => item.run.projectId !== projectId); next.aiOverviewRuns = next.aiOverviewRuns.filter(item => item.projectId !== projectId); + next.regulatoryMappingBatchRuns = next.regulatoryMappingBatchRuns.filter(item => item.projectId !== projectId); // Regulatory mappings and decisions are global and survive project deletion. this.replaceState(next); return true; diff --git a/src/styles/datahub.css b/src/styles/datahub.css index bc5fc8b..f90857f 100644 --- a/src/styles/datahub.css +++ b/src/styles/datahub.css @@ -985,12 +985,21 @@ button.client-overall-ring:focus-visible { outline: var(--bw-emphasis) solid var .results-environment-value p { margin: var(--legacy-space-1) 0 var(--legacy-space-3); overflow-wrap: anywhere; white-space: pre-wrap; } .results-processing-alert { margin-bottom: var(--legacy-space-3); } .results-browser-load-actions { display: flex; gap: var(--legacy-space-2); margin-top: var(--legacy-space-3); } +.regulatory-batch-dialog { display: grid; gap: var(--legacy-space-4); } +.regulatory-batch-dialog p { margin: 0; font-size: var(--ui-reading-size); line-height: 1.5; } +.regulatory-batch-state { padding-top: var(--legacy-space-3); border-top: var(--border-hair) solid var(--line); } +.regulatory-batch-counts { display: grid; grid-template-columns: repeat(2, minmax(0, 1fr)); gap: var(--legacy-space-2); margin: 0; } +.regulatory-batch-counts > div { display: flex; align-items: center; justify-content: space-between; gap: var(--legacy-space-3); min-width: 0; padding: var(--legacy-space-2) var(--legacy-space-3); background: var(--panel-soft); } +.regulatory-batch-counts dt { color: var(--muted); font-size: var(--ui-label-size); font-weight: var(--ui-label-weight); } +.regulatory-batch-counts dd { margin: 0; color: var(--heading); font-variant-numeric: tabular-nums; font-weight: var(--ui-strong-weight); } +.regulatory-batch-dialog .dialog-actions { display: flex; justify-content: flex-end; gap: var(--legacy-space-2); padding-top: var(--legacy-space-2); } @media (max-width: 1150px) { .results-browser-layout { grid-template-columns: 1fr; } .results-filter-panel { position: static; max-height: none; } .results-browser-row { grid-template-columns: 1fr 1fr; gap: var(--legacy-space-2) var(--legacy-space-4); } .results-browser-row span::before { content: attr(data-label); display: block; margin-bottom: 2px; color: var(--muted); font-size: var(--ui-label-size); font-weight: var(--ui-label-weight); letter-spacing: var(--ui-label-tracking); text-transform: uppercase; } .results-browser-head { display: none; } + .regulatory-batch-counts { grid-template-columns: 1fr; } } .datahub-mitre-management { diff --git a/tests/api/assessor-api-client.test.ts b/tests/api/assessor-api-client.test.ts index c604b01..2417dfe 100644 --- a/tests/api/assessor-api-client.test.ts +++ b/tests/api/assessor-api-client.test.ts @@ -113,6 +113,22 @@ describe('AssessorApiClient', () => { expect(new Headers(fetcher.mock.calls[1][1]?.headers).get('X-Assessor-Request')).toBe('1'); expect(new Headers(fetcher.mock.calls[2][1]?.headers).get('X-Assessor-Request')).toBe('1'); }); + it('uses the complete filtered Result DB scope for regulatory mapping batches', async () => { + const fetcher = vi.fn().mockResolvedValue(new Response(JSON.stringify({ + version: 1, projectId: 'project/one', scope: { assessmentIds: ['assessment/one'], scanRunIds: ['run/one'], filters: { deviation: true } }, + generationEnabled: true, state: 'empty' + }), { status: 200, headers: { 'Content-Type': 'application/json' } })); + const client = new AssessorApiClient({ fetcher }); + const scope = { assessmentIds: ['assessment/one'], scanRunIds: ['run/one'], filters: { deviation: true, severity: ['high' as const] } }; + + await client.getRegulatoryMappingBatch('project/one', scope); + await client.startRegulatoryMappingBatch('project/one', { ...scope, idempotencyKey: 'batch-request-1' }); + + expect(fetcher.mock.calls[0][0]).toBe('/api/v1/projects/project%2Fone/regulatory-mapping-batches?deviation=true&severity=high&assessmentIds=assessment%2Fone&scanRunIds=run%2Fone'); + expect(fetcher.mock.calls[1][0]).toBe('/api/v1/projects/project%2Fone/regulatory-mapping-batches'); + expect(new Headers(fetcher.mock.calls[0][1]?.headers).has('X-Assessor-Request')).toBe(false); + expect(new Headers(fetcher.mock.calls[1][1]?.headers).get('X-Assessor-Request')).toBe('1'); + }); it('submits GPO backups only through the project-bound mutation endpoint', async () => { const fetcher = vi.fn().mockResolvedValue(new Response(JSON.stringify({ version: 1, artifactId: 'artifact-1', jobId: 'job-1', sha256: 'a'.repeat(64), sizeBytes: 3, diff --git a/tests/domain/result-db-query.test.ts b/tests/domain/result-db-query.test.ts new file mode 100644 index 0000000..29fbe60 --- /dev/null +++ b/tests/domain/result-db-query.test.ts @@ -0,0 +1,50 @@ +import { describe, expect, it } from 'vitest'; +import { buildScopedResultDbQuery, resultDbBatchFilters } from '../../src/domain/result-db-query'; +import type { UnifiedResultQuery } from '../../src/domain/unified-result'; + +describe('Result DB scoped query', () => { + it('uses one pagination-free query for export and regulatory batch filters', () => { + const filters: UnifiedResultQuery = { + evaluationRunId: 'evaluation-1', + sourceKind: ['scanner-finding'], + deviation: true, + isIoe: false, + severity: ['high'], + outcome: ['fail'], + technicalExecutionState: ['completed'], + resultType: ['compliance_check'], + policyClassification: ['misconfigured'], + sourceTool: ['purple-knight'], + mappingRuleVersion: ['mapping-1'], + sourceProfileVersion: ['profile-1'], + processingStatus: ['current'], + reviewDisposition: ['accepted'], + lifecycleState: ['new'], + categoryNamespace: ['framework'], + categoryValue: ['iso'], + affectedAsset: 'synthetic asset', + environmentState: ['observed'], + environmentProperty: ['domain.count'], + environmentSubjectType: ['domain'], + search: 'synthetic', + observedFrom: '2026-09-01T00:00:00.000Z', + observedTo: '2026-09-16T00:00:00.000Z' + }; + const query = buildScopedResultDbQuery({ + ...filters, + offset: 100, + limit: 50, + projectId: 'untrusted-project', + scanRunIds: ['untrusted-run'] + }, 'project-1', ['run-1'], ['assessment-1']); + + expect(query).toEqual({ + audience: 'result-db', + projectId: 'project-1', + assessmentIds: ['assessment-1'], + scanRunIds: ['run-1'], + ...filters + }); + expect(resultDbBatchFilters(query)).toEqual(filters); + }); +}); diff --git a/tests/features/regulatory-mapping-batch.test.tsx b/tests/features/regulatory-mapping-batch.test.tsx new file mode 100644 index 0000000..d532b8b --- /dev/null +++ b/tests/features/regulatory-mapping-batch.test.tsx @@ -0,0 +1,66 @@ +// @vitest-environment jsdom +import { readFileSync } from 'node:fs'; +import { resolve } from 'node:path'; +import { act } from 'react'; +import { createRoot } from 'react-dom/client'; +import { describe, expect, it, vi } from 'vitest'; +import { RegulatoryMappingBatchDialog } from '../../src/features/results-browser/RegulatoryMappingBatchDialog'; + +(globalThis as typeof globalThis & { IS_REACT_ACT_ENVIRONMENT: boolean }).IS_REACT_ACT_ENVIRONMENT = true; + +describe('Result DB regulatory mapping batch', () => { + it('places the action with the Result DB page actions and shares its scoped query with export', () => { + const source = readFileSync(resolve(process.cwd(), 'src/features/results-browser/ResultsBrowser.tsx'), 'utf8'); + expect(source.indexOf('Map all ISO controls')).toBeGreaterThan(source.indexOf("'Refresh'")); + expect(source.indexOf('Map all ISO controls')).toBeLessThan(source.indexOf("'Exporting …'")); + expect(source).toContain('buildScopedResultDbQuery'); + expect(source).toContain('queryAllUnifiedResults'); + expect(source).toContain('resultDbBatchFilters(scopedQuery)'); + }); + + it('explains complete filtered membership and displays durable progress counts', async () => { + const container = document.createElement('div'); + document.body.appendChild(container); + const root = createRoot(container); + await act(async () => root.render( {}} controller={{ + loading: false, + starting: false, + start: vi.fn(), + response: { + version: 1, + projectId: 'project-1', + scope: { assessmentIds: ['assessment-1'], scanRunIds: ['run-1'], filters: { deviation: true } }, + generationEnabled: true, + state: 'running', + pollAfterMs: 1_500, + run: { + version: 1, + id: 'batch-1', + projectId: 'project-1', + scopeKey: 'scope-1', + filterHash: 'a'.repeat(64), + idempotencyKey: 'batch-request-1', + assessmentIds: ['assessment-1'], + scanRunIds: ['run-1'], + filters: { deviation: true }, + status: 'running', + counts: { + filteredResultCount: 75, excludedNonScannerCount: 5, eligibleFindingCount: 68, unsupportedFindingCount: 2, + distinctRuleCount: 20, reusedSucceededRuleCount: 8, reusedActiveRuleCount: 1, scheduledRuleCount: 11, + succeededRuleCount: 4, failedRuleCount: 1 + }, + createdAt: '2026-09-16T08:00:00.000Z', + updatedAt: '2026-09-16T08:01:00.000Z' + } + } + }}/>)); + const text = document.body.textContent ?? ''; + expect(text).toContain('all 75 results matching the current Result DB filters'); + expect(text).toContain('Pagination, “Load more,” and loaded-row limits do not restrict this batch.'); + expect(text).toContain('Distinct rules20'); + expect(text).toContain('Completed4'); + expect(text).toContain('Failed1'); + await act(async () => root.unmount()); + container.remove(); + }); +}); diff --git a/tests/persistence/regulatory-mapping-batch.test.ts b/tests/persistence/regulatory-mapping-batch.test.ts new file mode 100644 index 0000000..f80734a --- /dev/null +++ b/tests/persistence/regulatory-mapping-batch.test.ts @@ -0,0 +1,50 @@ +import { describe, expect, it } from 'vitest'; +import type { RegulatoryMappingBatchRun } from '../../src/domain/regulatory-mapping'; +import { LocalAssessorRepository } from '../../src/persistence/local-assessor-repository'; + +const stamp = '2026-09-16T08:00:00.000Z'; + +function batch(overrides: Partial = {}): RegulatoryMappingBatchRun { + return { + version: 1, + id: 'batch-1', + projectId: 'project-1', + scopeKey: 'scope-1', + filterHash: 'a'.repeat(64), + idempotencyKey: 'batch-request-1', + assessmentIds: ['assessment-1'], + scanRunIds: [], + filters: { deviation: true }, + status: 'queued', + counts: { + filteredResultCount: 0, excludedNonScannerCount: 0, eligibleFindingCount: 0, unsupportedFindingCount: 0, + distinctRuleCount: 0, reusedSucceededRuleCount: 0, reusedActiveRuleCount: 0, scheduledRuleCount: 0, + succeededRuleCount: 0, failedRuleCount: 0 + }, + createdAt: stamp, + updatedAt: stamp, + ...overrides + }; +} + +describe('local regulatory mapping batch persistence', () => { + it('preserves idempotency and active-scope uniqueness, exports batches and deletes them with the project', async () => { + const repository = new LocalAssessorRepository(); + await repository.saveProject({ id: 'project-1', name: 'Synthetic', createdAt: stamp, updatedAt: stamp, attributes: {} }); + await repository.saveAssessment({ id: 'assessment-1', projectId: 'project-1', name: 'Synthetic', createdAt: stamp, updatedAt: stamp }); + const first = await repository.createRegulatoryMappingBatchRun(batch()); + + await expect(repository.createRegulatoryMappingBatchRun(batch({ id: 'batch-idempotent' }))).resolves.toEqual(first); + await expect(repository.createRegulatoryMappingBatchRun(batch({ id: 'batch-active', idempotencyKey: 'batch-request-2' }))).resolves.toEqual(first); + expect((await repository.exportProject('project-1')).regulatoryMappingBatchRuns).toEqual([first]); + + const running = await repository.updateRegulatoryMappingBatchRun({ ...first, status: 'running', startedAt: stamp }); + await repository.updateRegulatoryMappingBatchRun({ ...running, status: 'completed', completedAt: stamp }); + const retry = await repository.createRegulatoryMappingBatchRun(batch({ id: 'batch-2', idempotencyKey: 'batch-request-2' })); + expect((await repository.listRegulatoryMappingBatchRuns('project-1', 'scope-1')).map(item => item.id)).toEqual(['batch-2', 'batch-1']); + expect(retry.id).toBe('batch-2'); + + await repository.deleteProject('project-1', { projectId: 'project-1', cascade: true }); + await expect(repository.listRegulatoryMappingBatchRuns('project-1')).resolves.toEqual([]); + }); +}); diff --git a/tests/postgres/regulatory-mapping-batch-schema.test.ts b/tests/postgres/regulatory-mapping-batch-schema.test.ts new file mode 100644 index 0000000..b0caa8b --- /dev/null +++ b/tests/postgres/regulatory-mapping-batch-schema.test.ts @@ -0,0 +1,16 @@ +import { readFileSync } from 'node:fs'; +import { describe, expect, it } from 'vitest'; + +const sql = readFileSync(new URL('../../server/postgres/migrations/026_regulatory_mapping.sql', import.meta.url), 'utf8'); + +describe('regulatory mapping batch persistence schema', () => { + it('keeps orchestration project-scoped and protects idempotency and active scope', () => { + expect(sql).toContain('CREATE TABLE assessor_regulatory_mapping_batches'); + expect(sql).toContain('project_id text NOT NULL REFERENCES assessor_projects(id) ON DELETE CASCADE'); + expect(sql).toContain('filter_hash text NOT NULL'); + expect(sql).toContain('UNIQUE (project_id, idempotency_key)'); + expect(sql).toMatch(/UNIQUE INDEX assessor_regulatory_mapping_batches_active_scope_uq[\s\S]*WHERE run_status IN \('queued', 'running'\)/); + expect(sql).toContain('(project_id, scope_key, created_at DESC, id DESC)'); + expect(sql).not.toContain('source_result_id'); + }); +}); diff --git a/tests/server/regulatory-mapping-service.test.ts b/tests/server/regulatory-mapping-service.test.ts index 16a702b..d1af663 100644 --- a/tests/server/regulatory-mapping-service.test.ts +++ b/tests/server/regulatory-mapping-service.test.ts @@ -1,9 +1,9 @@ import { describe, expect, it, vi } from 'vitest'; import { RegulatoryMappingService } from '../../server/regulatory/regulatory-mapping-service'; import { RegulatoryFindingRetrievalService } from '../../server/regulatory/regulatory-finding-retrieval'; -import type { RegulatoryMappingDecision, RegulatoryMappingRun } from '../../src/domain/regulatory-mapping'; +import type { RegulatoryMappingBatchRun, RegulatoryMappingDecision, RegulatoryMappingRun } from '../../src/domain/regulatory-mapping'; import type { RegulatoryChunk, RegulatoryRetrievalRequest } from '../../src/domain/regulatory-rag'; -import type { UnifiedResultProjection } from '../../src/domain/unified-result'; +import { filterUnifiedResults, type UnifiedResultProjection } from '../../src/domain/unified-result'; import type { AssessorRepository } from '../../src/persistence/assessor-repository'; const stamp = '2026-09-15T10:00:00.000Z'; @@ -40,12 +40,48 @@ function chunk(sourceStandard: 'ISO/IEC 27001' | 'ISO/IEC 27002'): RegulatoryChu function mockRepository(findings: UnifiedResultProjection[]) { const byProjection = new Map(findings.map(item => [item.resultProjectionId, item])); const runs = new Map(); + const batches: RegulatoryMappingBatchRun[] = []; const decisions: RegulatoryMappingDecision[] = []; return { + getProject: vi.fn(async (projectId: string) => findings.some(item => item.projectId === projectId) ? { id: projectId, name: 'Synthetic project' } : null), + listProjects: vi.fn(async () => [...new Set(findings.map(item => item.projectId))].map(id => ({ id, name: 'Synthetic project' }))), + listAssessments: vi.fn(async (projectId: string) => [...new Set(findings.filter(item => item.projectId === projectId).map(item => item.assessmentId).filter(Boolean))] + .map(id => ({ id: id!, projectId, label: 'Synthetic assessment', createdAt: stamp }))), + listProjectScanRuns: vi.fn(async (projectId: string) => [...new Map(findings.filter(item => item.projectId === projectId && item.scanRunId) + .map(item => [item.scanRunId!, { id: item.scanRunId!, assessmentId: item.assessmentId!, sequence: 1, startedAt: stamp }])).values()]), + queryUnifiedResults: vi.fn(async (query = {}) => { + const filtered = filterUnifiedResults(findings, query); + const offset = query.offset ?? 0; + return { total: filtered.length, items: filtered.slice(offset, offset + (query.limit ?? filtered.length)) }; + }), getUnifiedResultById: vi.fn(async (id: string) => byProjection.get(id) ?? null), getRegulatoryMappingRun: vi.fn(async (id: string) => runs.get(id) ?? null), + claimRegulatoryMappingRun: vi.fn(async (run: RegulatoryMappingRun, force: boolean) => { + const existing = runs.get(run.id); + if (existing && (['queued', 'running'].includes(existing.status) || existing.status === 'succeeded' && !force)) { + return { run: existing, claimed: false }; + } + runs.set(run.id, run); + return { run, claimed: true }; + }), saveRegulatoryMappingRun: vi.fn(async (run: RegulatoryMappingRun) => { runs.set(run.id, run); return run; }), listRegulatoryMappingRuns: vi.fn(async () => [...runs.values()]), + createRegulatoryMappingBatchRun: vi.fn(async (run: RegulatoryMappingBatchRun) => { + const existing = batches.find(item => item.projectId === run.projectId && (item.idempotencyKey === run.idempotencyKey + || item.scopeKey === run.scopeKey && ['queued', 'running'].includes(item.status))); + if (existing) return existing; + batches.push(run); + return run; + }), + updateRegulatoryMappingBatchRun: vi.fn(async (run: RegulatoryMappingBatchRun) => { + const index = batches.findIndex(item => item.id === run.id); + if (index < 0) throw new Error('missing batch'); + batches[index] = run; + return run; + }), + listRegulatoryMappingBatchRuns: vi.fn(async (projectId: string, scopeKey?: string) => batches + .filter(item => item.projectId === projectId && (!scopeKey || item.scopeKey === scopeKey)) + .slice().reverse()), createRegulatoryMappingDecision: vi.fn(async (decision: RegulatoryMappingDecision, expected?: string) => { const run = runs.get(decision.mappingId); if (!run || run.ruleKey !== decision.ruleKey || run.status !== 'succeeded') throw new Error('missing mapping'); @@ -127,6 +163,163 @@ describe('regulatory mapping service', () => { expect(completed.run?.errorDetail).toContain('unknown candidate control'); expect(completed.run?.errorDetail).not.toContain(resultTableAsset); }); + + it('maps the complete filtered timeslot, excludes non-scanner results and deduplicates global rules', async () => { + const findings = [ + finding(), + finding({ resultProjectionId: 'scanner-finding:occurrence-2', sourceResultId: 'occurrence-2' }), + finding({ resultProjectionId: 'scanner-finding:occurrence-3', sourceResultId: 'occurrence-3', sourceRuleId: 'A-Second', title: 'Second synthetic check' }), + finding({ resultProjectionId: 'policy-evaluation:one', sourceResultId: 'policy-one', sourceKind: 'policy-evaluation', sourceRuleId: undefined }), + finding({ resultProjectionId: 'scanner-finding:unsupported', sourceResultId: 'unsupported', sourceRuleId: 'A-Unsupported', title: '' }), + finding({ resultProjectionId: 'scanner-finding:not-deviation', sourceResultId: 'not-deviation', sourceRuleId: 'A-Ignored', isDeviation: false }) + ]; + const repository = mockRepository(findings); + const complete = vi.fn(async (_system: string, _input: string) => ({ value: { + schemaVersion: 1, outcome: 'mapped', mappings: [{ controlId: '5.1', relationship: 'primary', confidence: 'high', + rationale: 'Synthetic rationale.', evidenceChunkIds: ['semantic-5.1', 'reference-5.1'] }] + } })); + const svc = service(repository, complete); + + await svc.startBatch('project-1', { + assessmentIds: ['assessment-1'], scanRunIds: ['run-1'], filters: { deviation: true, affectedAsset: resultTableAsset }, + idempotencyKey: 'batch-filtered-1' + }); + await svc.close(); + + const response = await svc.getBatch('project-1', { + assessmentIds: ['assessment-1'], scanRunIds: ['run-1'], filters: { deviation: true, affectedAsset: resultTableAsset } + }); + expect(response.run).toMatchObject({ status: 'completed', counts: { + filteredResultCount: 5, + excludedNonScannerCount: 1, + eligibleFindingCount: 3, + unsupportedFindingCount: 1, + distinctRuleCount: 2, + scheduledRuleCount: 2, + succeededRuleCount: 2, + failedRuleCount: 0 + } }); + expect(complete).toHaveBeenCalledTimes(2); + expect(complete.mock.calls.every(call => !String(call[1]).includes(resultTableAsset))).toBe(true); + }); + + it('bounds concurrent mapping completions across a batch', async () => { + const findings = Array.from({ length: 5 }, (_, index) => finding({ + resultProjectionId: `scanner-finding:occurrence-${index}`, + sourceResultId: `occurrence-${index}`, + sourceRuleId: `A-Rule-${index}`, + title: `Synthetic check ${index}` + })); + let active = 0; + let maximum = 0; + const complete = vi.fn(async () => { + active += 1; + maximum = Math.max(maximum, active); + await new Promise(resolve => setTimeout(resolve, 5)); + active -= 1; + return { value: { schemaVersion: 1, outcome: 'no_relevant_control', reason: 'Synthetic.', mappings: [] } }; + }); + const svc = service(mockRepository(findings), complete); + await svc.startBatch('project-1', { + assessmentIds: ['assessment-1'], scanRunIds: ['run-1'], filters: { deviation: true }, idempotencyKey: 'batch-concurrency-1' + }); + await svc.close(); + expect(complete).toHaveBeenCalledTimes(5); + expect(maximum).toBe(2); + }); + + it('reads every server page independently of loaded rows', async () => { + const findings = Array.from({ length: 501 }, (_, index) => finding({ + resultProjectionId: `scanner-finding:occurrence-${index}`, + sourceResultId: `occurrence-${index}` + })); + const repository = mockRepository(findings); + const complete = vi.fn(async () => ({ value: { + schemaVersion: 1, outcome: 'no_relevant_control', reason: 'Synthetic.', mappings: [] + } })); + const svc = service(repository, complete); + await svc.startBatch('project-1', { + assessmentIds: ['assessment-1'], scanRunIds: ['run-1'], filters: { deviation: true }, idempotencyKey: 'batch-many-pages-1' + }); + await svc.close(); + const response = await svc.getBatch('project-1', { + assessmentIds: ['assessment-1'], scanRunIds: ['run-1'], filters: { deviation: true } + }); + expect(response.run?.counts).toMatchObject({ filteredResultCount: 501, eligibleFindingCount: 501, distinctRuleCount: 1 }); + expect(repository.queryUnifiedResults).toHaveBeenCalledTimes(2); + expect(complete).toHaveBeenCalledOnce(); + }); + + it('fails the batch safely when the filtered total changes between pages', async () => { + const findings = Array.from({ length: 501 }, (_, index) => finding({ + resultProjectionId: `scanner-finding:occurrence-${index}`, + sourceResultId: `occurrence-${index}` + })); + const repository = mockRepository(findings); + let page = 0; + vi.mocked(repository.queryUnifiedResults).mockImplementation(async (query = {}) => { + page += 1; + const offset = query.offset ?? 0; + return { total: page === 1 ? 501 : 500, items: findings.slice(offset, offset + (query.limit ?? findings.length)) }; + }); + const complete = vi.fn(); + const svc = service(repository, complete); + await svc.startBatch('project-1', { + assessmentIds: ['assessment-1'], scanRunIds: ['run-1'], filters: { deviation: true }, idempotencyKey: 'batch-changing-total-1' + }); + await svc.close(); + const response = await svc.getBatch('project-1', { + assessmentIds: ['assessment-1'], scanRunIds: ['run-1'], filters: { deviation: true } + }); + expect(response.run).toMatchObject({ status: 'failed', errorCode: 'regulatory_mapping_batch_scope_changed' }); + expect(complete).not.toHaveBeenCalled(); + }); + + it('completes with partial failures, then reuses success and retries only the failed rule', async () => { + const findings = [finding(), finding({ + resultProjectionId: 'scanner-finding:occurrence-2', sourceResultId: 'occurrence-2', sourceRuleId: 'A-Second', title: 'Second synthetic check' + })]; + let secondAttempts = 0; + const complete = vi.fn(async (_system: string, input: string) => { + if (input.includes('Second synthetic check') && ++secondAttempts === 1) throw new Error('synthetic provider failure'); + return { value: { schemaVersion: 1, outcome: 'no_relevant_control', reason: 'Synthetic.', mappings: [] } }; + }); + const svc = service(mockRepository(findings), complete); + const scope = { assessmentIds: ['assessment-1'], scanRunIds: ['run-1'], filters: { deviation: true } }; + await svc.startBatch('project-1', { ...scope, idempotencyKey: 'batch-partial-1' }); + await svc.close(); + expect((await svc.getBatch('project-1', scope)).run).toMatchObject({ + status: 'completed', counts: { scheduledRuleCount: 2, succeededRuleCount: 1, failedRuleCount: 1 } + }); + + await svc.startBatch('project-1', { ...scope, idempotencyKey: 'batch-partial-2' }); + await svc.close(); + expect((await svc.getBatch('project-1', scope)).run).toMatchObject({ + status: 'completed', counts: { + distinctRuleCount: 2, reusedSucceededRuleCount: 1, scheduledRuleCount: 1, succeededRuleCount: 1, failedRuleCount: 0 + } + }); + expect(complete).toHaveBeenCalledTimes(3); + }); + + it('reuses an active single-finding mapping without a duplicate model call', async () => { + let release!: () => void; + const gate = new Promise(resolve => { release = resolve; }); + const complete = vi.fn(async () => { + await gate; + return { value: { schemaVersion: 1, outcome: 'no_relevant_control', reason: 'Synthetic.', mappings: [] } }; + }); + const svc = service(mockRepository([finding()]), complete); + await svc.start('project-1', { resultProjectionId: 'scanner-finding:occurrence-1' }); + const scope = { assessmentIds: ['assessment-1'], scanRunIds: ['run-1'], filters: { deviation: true } }; + await svc.startBatch('project-1', { ...scope, idempotencyKey: 'batch-active-reuse-1' }); + release(); + await svc.close(); + expect((await svc.getBatch('project-1', scope)).run?.counts).toMatchObject({ + distinctRuleCount: 1, reusedActiveRuleCount: 1, scheduledRuleCount: 0 + }); + expect(complete).toHaveBeenCalledOnce(); + }); }); async function runWith(value: unknown): Promise<{ svc: RegulatoryMappingService }> {