feat: extend Compliance Engine and ISO workflows for v2.19.0
- Move AI overview and privacy-safe JSON export into Compliance Engine - Add ISO control and theme facets to Work, including GPO settings - Enrich ISO queries with baseline context, Windows code meanings and technical explanations - Count GPO settings once and preserve editable decisions across mapping runs - Fix GPO mapping version conflicts and backup domain identity resolution - Add regression coverage and update architecture and reimport documentation
This commit is contained in:
parent
a1f5a59299
commit
32c67bacce
|
|
@ -36,7 +36,7 @@
|
||||||
## Core Engine and engines
|
## Core Engine and engines
|
||||||
|
|
||||||
- Explicit user decision 2026-10-01: ASSESSOR with all its existing functions is the **Core Engine**. The context enrichment (Phase 13E, "context engine"/"Kontext-Engine" in earlier text) is the **Context Engine**. The read-only Report Engine (Phase 7, `src/reporting/`, `server/reporting/`) is an engine in the same sense. Future features arrive as further engines that provide a capability without changing the Core. The UI labels `Context`/`Kontext` stay unchanged.
|
- Explicit user decision 2026-10-01: ASSESSOR with all its existing functions is the **Core Engine**. The context enrichment (Phase 13E, "context engine"/"Kontext-Engine" in earlier text) is the **Context Engine**. The read-only Report Engine (Phase 7, `src/reporting/`, `server/reporting/`) is an engine in the same sense. Future features arrive as further engines that provide a capability without changing the Core. The UI labels `Context`/`Kontext` stay unchanged.
|
||||||
- Explicit user decision 2026-10-05: the regulatory control mapping (Phase 13F, ISO mapping) is the **Compliance Engine** (`src/compliance/`, `server/compliance/`, `tests/compliance/`). It is framework-neutral by intent: ISO/IEC 27001/27002 is its first corpus, later frameworks such as BSI or DORA arrive inside this engine without a Core change. It reads the Core only through `ComplianceReadPort` (single Result DB row by project, filtered Result DB pages, project, assessment and scan-run lists) and stores mapping runs, accept/reject decisions and batches only through its own `ComplianceStore`. The existing tables of migration 026 keep their names and data and are owned by the engine; its writes never advance the Core repository revision. The Core project export receives the engine data beside the Core export at the HTTP route. Its extension points are the Finding Reader `Map ISO controls` action and its window, the Finding Reader `ISO controls` section, the Result DB `Map all ISO controls` batch action, the Data Hub ISO corpus tile and the Client Info `ISO controls` section. Explicit user decision 2026-10-05: `Map ISO controls` opens a window like `Accepted Risk` with Accept and Reject per control (no third decision value). The Finding Reader lists the controls whose latest decision is accepted in an `ISO controls` section read like `Description`, after the summary fields. Client Info shows the accepted controls of the selected timeslot directly below the Context heatmap in the same tile layout without severity colour, each with its ISO 27002 theme and the number of deviation findings (a GPO setting counts once); a tile opens exactly those findings in Work. Decisions stay global per scanner rule. Provisional explicit user decision 2026-10-05 (may change later): after a mapping succeeds, single or batch, the engine accepts its first two proposals in the model's order automatically, only the one if there is a single proposal, with the comment `Accepted automatically: one of the first two proposals of the mapping.`; a control that already has a decision keeps it, so a manual reject is never overridden, and a failure of this step never fails the mapping.
|
- Explicit user decision 2026-10-05: the regulatory control mapping (Phase 13F, ISO mapping) is the **Compliance Engine** (`src/compliance/`, `server/compliance/`, `tests/compliance/`). It is framework-neutral by intent: ISO/IEC 27001/27002 is its first corpus, later frameworks such as BSI or DORA arrive inside this engine without a Core change. It reads the Core only through `ComplianceReadPort` (single Result DB row by project, filtered Result DB pages, project, assessment and scan-run lists; explicit user decision 2026-10-05: additionally the internal-analysis `policy-evaluation` rows of the same project and timeslot, used solely to read the GPO technical setting ID that the Result DB withholds, so a GPO setting counts once like its rolled-up Work record and the Work ISO facets match it; that ID is never displayed or exported and no other internal field is read) and stores mapping runs, accept/reject decisions, batches and AI overview runs only through its own `ComplianceStore`. Explicit user decision 2026-10-05: all AI integration belongs to this engine. The AI overview (service, model, Client Info panel) moved from the Core into it, and the Result DB `Export JSON` is the engine's 1:1 copy of what its AI features receive: the exact AI overview input built by the same function the overview sends to the model, and every distinct ISO mapping query with the Result DB rows that produce it (the retrieved candidate controls depend on the index and are not exported). The shared model client `server/ai/litellm-client.ts` stays Core platform code. The existing tables of migrations 025 (AI overview runs) and 026 keep their names and data and are owned by the engine; its writes never advance the Core repository revision. Because the Core deletes scan runs and assessments without knowing the engine, the engine removes AI overview runs of a deleted timeslot from its own table on start and on read. The Core project export receives the engine data, including succeeded AI overview runs, beside the Core export at the HTTP route. Its extension points are the Finding Reader `Map ISO controls` action and its window, the Finding Reader `ISO controls` section, the Result DB `Map all ISO controls` batch action and `Export JSON` action, the Data Hub ISO corpus tile, the Client Info `ISO controls` section and `AI Overview` tab, and the Work `ISO control` and `ISO theme` facets. Switched off, the engine also removes the AI overview tab and its requests and the `Export JSON` action. Explicit user decision 2026-10-05: `Map ISO controls` opens a window like `Accepted Risk` with Accept and Reject per control (no third decision value). The Finding Reader lists the controls whose latest decision is accepted in an `ISO controls` section read like `Description`, after the summary fields. Client Info shows the accepted controls of the selected timeslot directly below the Context heatmap in the same tile layout without severity colour, each with its ISO 27002 theme and the number of deviation findings (a GPO setting counts once); a tile opens Work with the `Deviation` status and that control selected in the `ISO control` facet, like a Context heatmap cell. Explicit user decision 2026-10-05: Work offers the Compliance Engine facets `ISO control` (one value per accepted control, `A.8.20 · Networks security`) and `ISO theme` (Organizational, People, Physical, Technological) in mixed Work and in both GPO entry paths, directly before the permanent final `Source: Scan`; this extends the GPO-only facet list. They read the accepted controls of every mappable finding of the timeslot (scanner findings by occurrence, GPO settings by scope and technical identity), offer only values with hits, count against the other active filters, keep an active selection while loading, and disappear with their filter cleared when the engine is off. Decisions stay global per scanner rule. Provisional explicit user decision 2026-10-05 (may change later): after a mapping succeeds, single or batch, the engine accepts its first two proposals in the model's order automatically, only the one if there is a single proposal, with the comment `Accepted automatically: one of the first two proposals of the mapping.`; a control that already has a decision keeps it, so a manual reject is never overridden, and a failure of this step never fails the mapping.
|
||||||
- Engines may read the Core without restriction but never write to, change or override it. They read only through a declared read-only port and the read-only database access, never through Core tables or internal modules directly. What an engine shows or exports still obeys the Core's privacy and Result DB disclosure rules.
|
- Engines may read the Core without restriction but never write to, change or override it. They read only through a declared read-only port and the read-only database access, never through Core tables or internal modules directly. What an engine shows or exports still obeys the Core's privacy and Result DB disclosure rules.
|
||||||
- Dependencies point one way: an engine depends on the Core, the Core never depends on an engine. The Core works unchanged with every engine switched off; each engine keeps a permanent on/off invariance test like the Context layer's. Engines do not build on each other except through another engine's published results.
|
- Dependencies point one way: an engine depends on the Core, the Core never depends on an engine. The Core works unchanged with every engine switched off; each engine keeps a permanent on/off invariance test like the Context layer's. Engines do not build on each other except through another engine's published results.
|
||||||
- An engine that needs storage gets its own schema or tables after separate approval with backup and migration, never columns in Core tables. An engine cannot create findings: new scanner profiles, importers and parsers write Core data and are always Core changes.
|
- An engine that needs storage gets its own schema or tables after separate approval with backup and migration, never columns in Core tables. An engine cannot create findings: new scanner profiles, importers and parsers write Core data and are always Core changes.
|
||||||
|
|
@ -51,7 +51,7 @@
|
||||||
|
|
||||||
## Context layer (Phase 13E)
|
## Context layer (Phase 13E)
|
||||||
|
|
||||||
- Latest explicit user decision 2026-10-01 supersedes the preceding permanent-Context decision: the Work toolbar has an `Engines` button at the former Context-button position. It stays marked blue, including on hover and while the popup is closed (an explicit colour-role exception for this control only). The shared popup lists Core Engine, Context Engine, Report Engine and, since the explicit user decision of 2026-10-05, Compliance Engine with checkboxes. All default to active; Core Engine is always checked and cannot be disabled. User choices for optional engines are stored locally under the shared Core-owned engine-activation contract. Disabling Context removes its UI, stops new context requests and clears the active Context Cluster filter; disabling Report blocks new report generation from the UI; disabling Compliance removes the Finding Reader ISO action and panel, the Result DB ISO batch action and the Data Hub ISO corpus tile and stops their requests, and re-enabling starts the Finding Reader ISO panel closed. Already submitted backend work is not cancelled by this local preference; its late UI output is suppressed. Cluster badges still select the exact Work filter, and the Context detail section still starts collapsed. Engine independence and Core invariance remain binding.
|
- Latest explicit user decision 2026-10-01 supersedes the preceding permanent-Context decision: the Work toolbar has an `Engines` button at the former Context-button position. It stays marked blue, including on hover and while the popup is closed (an explicit colour-role exception for this control only). The shared popup lists Core Engine, Context Engine, Report Engine and, since the explicit user decision of 2026-10-05, Compliance Engine with checkboxes. All default to active; Core Engine is always checked and cannot be disabled. User choices for optional engines are stored locally under the shared Core-owned engine-activation contract. Disabling Context removes its UI, stops new context requests and clears the active Context Cluster filter; disabling Report blocks new report generation from the UI; disabling Compliance removes the Finding Reader ISO action and panel, the Result DB ISO batch action and `Export JSON`, the Client Info `AI Overview` tab and the Data Hub ISO corpus tile and stops their requests, and re-enabling starts the Finding Reader ISO panel closed. Already submitted backend work is not cancelled by this local preference; its late UI output is suppressed. Cluster badges still select the exact Work filter, and the Context detail section still starts collapsed. Engine independence and Core invariance remain binding.
|
||||||
|
|
||||||
- Explicit user decision 2026-10-01 supersedes the earlier Context on/off, default-off, locally saved switch, hover-button and badge-scroll agreements: Context is permanently active and the global toolbar Context button is removed. Ignore obsolete locally saved on/off preferences. Clicking a Finding Reader Context cluster badge selects that exact `Context Cluster` Work facet value, replacing the previous cluster selection while preserving other filters. The Context detail section still starts collapsed for each finding and opens through its own summary. All read-only, provenance and verdict/score boundaries remain binding; only an explicit cluster-filter selection restricts the Work record set.
|
- Explicit user decision 2026-10-01 supersedes the earlier Context on/off, default-off, locally saved switch, hover-button and badge-scroll agreements: Context is permanently active and the global toolbar Context button is removed. Ignore obsolete locally saved on/off preferences. Clicking a Finding Reader Context cluster badge selects that exact `Context Cluster` Work facet value, replacing the previous cluster selection while preserving other filters. The Context detail section still starts collapsed for each finding and opens through its own summary. All read-only, provenance and verdict/score boundaries remain binding; only an explicit cluster-filter selection restricts the Work record set.
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -3404,6 +3404,18 @@ Festgelegtes Betriebsrisiko vom 2026-09-06: Verzeichnisabhaengige Compose-Projek
|
||||||
|
|
||||||
## 12. Änderungsprotokoll
|
## 12. Änderungsprotokoll
|
||||||
|
|
||||||
|
2026-10-05 - Phase 13F / AI-Integration in die Compliance Engine verlegt (Nutzerentscheidung; Core-Freigabe Platform, Presentation, Disclosure, Store, Normalization, Core Tests, Governance; per Test verifiziert; in Produktversion 2.19.0 enthalten, keine weitere Versionserhoehung): Die AI-Uebersicht (Dienst `server/compliance/ai-overview-service.ts`, Modell `src/compliance/ai-overview.ts`, Panel und CSS unter `src/compliance/ui/`) und der Privacy-Safe-Export (`src/compliance/privacy-safe-export.ts`, vorher `src/exports/result-db-copy.ts`) gehoeren jetzt zur Compliance Engine. Der Dienst liest ueber `ComplianceReadPort` und speichert ueber `ComplianceStore` (Postgres und lokal); die Tabelle aus Migration 025 behaelt Namen und Daten und gehoert der Engine, ohne Migration oder Schemaaenderung. Das Core-Repository verliert die AI-Methoden, das Feld im Core-Projektexport und die Loeschkaskade; die Engine liefert die erfolgreichen Laeufe neben dem Core-Export und entfernt Laeufe geloeschter Timeslots beim Start und beim Lesen selbst. `server/ai/litellm-client.ts` bleibt Core und hat einen eigenen Tokenzaehler-Typ. Neue Erweiterungspunkte: Client-Info-Tab `AI Overview` und Result-DB-Aktion `Export JSON`, beide nur bei eingeschalteter Engine. `Export JSON` (Schema 2) enthaelt 1:1, was die KI erhaelt: `aiOverviewInput` aus derselben Funktion, die die AI-Uebersicht hasht und sendet (deterministische Reihenfolge), und `isoMappingQueries` mit jeder verschiedenen ISO-Anfrage und ihren Result-DB-Zeilen; die abgerufenen Kandidatenkontrollen haengen vom Index ab und sind nicht enthalten. Tests verschoben nach `tests/compliance/`; neu: 1:1-Export, Engine-Schalter fuer AI-Tab und Export, Bereinigung geloeschter Timeslots, Core-Export ohne AI-Laeufe. Gesamtsuite gruen bis auf einen lastbedingten Timeout von `sheet-selection-cost` (einzeln 0,7 s gruen); beide Builds, Local-only- und Designpruefung bestanden.
|
||||||
|
|
||||||
|
2026-10-05 - Phase 13F / GPO-Einstellungen in den Work-ISO-Facetten (Engine-Arbeit, Nutzerentscheidung Option 1 mit Core-Freigabe Governance, per Test verifiziert): Die Engine liest ueber `ComplianceReadPort` zusaetzlich die internen `policy-evaluation`-Zeilen desselben Projekts und Timeslots, ausschliesslich um die von der Result DB zurueckgehaltene technische GPO-Einstellungs-ID zu erhalten. Damit zaehlt eine GPO-Einstellung exakt wie ihr Work-Datensatz (auch bei gleichem Titel verschiedener Einstellungen), und die Work-Facetten `ISO control`/`ISO theme` finden GPO-Einstellungen. Die ID wird nie angezeigt oder exportiert; Mapping und Entscheidungen bleiben je offengelegtem Titel. Die zugehoerige AGENTS.md-Regel wurde am selben Tag mit erneuter Governance-Freigabe nachgetragen.
|
||||||
|
|
||||||
|
2026-10-05 - Phase 13F / GPO-Kontext in der ISO-Mapping-Anfrage (Engine-Arbeit, per Test verifiziert): Die Anfrage einer GPO-Abweichung bestand bisher praktisch nur aus Einstellungstitel und einem festen Beschreibungssatz. Sie enthaelt jetzt aus offengelegten Result-DB-Feldern Einstellungsart (Scope, Auslieferungsmechanismus), Policy-Abschnitt bzw. Registry-Schluessel, den in Woerter zerlegten Einstellungsnamen (`MinimumPasswordLength` -> `Minimum Password Length`, damit auch die Themenerkennung greift), Baseline-Familie, empfohlenen Baselinewert und den bewiesenen Befund (nicht konfiguriert, widerspruechlich oder abweichend). Eine Richtung (schwaecher/staerker) wird nur genannt, wenn der Vergleich sie festhaelt; alle aktuellen Vergleiche sind Gleichheitsvergleiche und beweisen nur `different`, daher wird z. B. nicht behauptet, eine Passwortlaenge liege unter 14. Der Kundenwert wird nie gelesen. Neue Version `REGULATORY_GPO_QUERY_VERSION` (gpo-baseline-context-1.0.0) geht nur in die Mapping-ID von GPO-Regeln ein; der naechste Batch erzeugt GPO-Mappings neu, Scanner-Mappings bleiben wiederverwendet. Bestehende Entscheidungen bleiben je Regel erhalten; die vorlaeufige Auto-Akzeptanz nimmt bei neuen Laeufen bis zu zwei noch unentschiedene Vorschlaege hinzu.
|
||||||
|
|
||||||
|
2026-10-05 - Phase 13F / Compliance Engine zaehlt eine GPO-Einstellung einmal (Engine-Arbeit, per Test verifiziert): Die Result DB fuehrt je Baselinevergleich eine Zeile (Beispiel: 127 Zeilen fuer 25 abweichende Einstellungen) und haelt die interne Kennung `gpo-policy-identity-*` zurueck. Die Engine fiel dadurch fuer GPO auf die vergleichsbezogene `sourceResultId` zurueck und zaehlte im Client-Info-Bereich `ISO controls` und im Batch (`Eligible/Unsupported findings`) je Vergleich. Neu identifiziert die Engine eine GPO-Einstellung ueber Scope und offengelegten Titel (die technische Einstellungsidentitaet), wenn keine oeffentliche Kennung vorliegt; `Filtered results` bleibt die Result-DB-Zeilenzahl. Keine Core-Aenderung. Offener Befund: Die Work-Facetten `ISO control`/`ISO theme` ordnen GPO-Einstellungen ueber `sourceRuleId` zu, die in Result-DB-Zeilen fehlt; GPO-Einstellungen erscheinen dort deshalb nicht. Eine Loesung erfordert eine Nutzerentscheidung (Read-Port-Vertrag oder Result-DB-Offenlegung).
|
||||||
|
|
||||||
|
2026-10-05 - Phase 13B / GPO-Domaenenschluessel aus Backup-GPO-Ordnern (Core-Freigabe Normalization, Ingest, Core Tests; per Test und am gespeicherten Artefakt inhaltsfrei verifiziert): Ein Standard-`Backup-GPO`-Ordner nennt die Domaene in `bkupInfo.xml` per `GPODomainGuid` und in `Backup.xml`/`GPReport.xml` per DNS-Name. `aggregateGpoBackupMetadata` verglich `domain-guid:` und `dns:` als einen Wert und meldete `gpo-metadata-domainKey-conflict`; jede GPO blieb mit `metadata-conflict` unaufgeloest und blockiert, Data Hub zeigte `Limited analysis`. Seit dd92aa2 (2026-09-06, DNS auch aus `Backup.xml`/`GPReport.xml`) betraf das jeden Standardexport. Korrektur: GUID und DNS-Name sind zwei Kennungsarten derselben Domaene; ein Konflikt besteht nur bei zwei verschiedenen Werten derselben Art, die GUID hat Vorrang. Append-only `gpo-backup-policy-analysis` und Jobprozessor 2.2.5 -> 2.2.6; betroffene Backups zeigen `Reprocessing required` (kein Upload, keine SQL-Migration, Metadatenparser 1.4.0 unveraendert). Lokale Backup-Zeiten ohne Zeitzone bleiben wie bisher unaufgeloest. Zwei synthetische Regressionstests (Standardlayout aufgeloest; abweichende DNS-Namen neben GUID weiterhin blockiert). Gesamtsuite: ein nicht zugehoeriger Fehlschlag (`unified-result-jsonl` erwartet Disclosure-Policy 3.5.0, paralleler Arbeitsstand 3.6.0). Produktversion nicht angehoben, da der Bereich Platform nicht freigegeben war.
|
||||||
|
|
||||||
|
2026-10-06 - Phase 13F / Windows-Code-Bedeutungen in der GPO-ISO-Mapping-Anfrage (Engine-Arbeit, per Test verifiziert): Die Anfrage nennt zu Einstellungsname und empfohlenem Baselinewert jetzt die dokumentierte Bedeutung standardisierter Windows-Codes aus derselben Liste, die der Finding Reader anzeigt (`src/shared/windows-codes.ts`): `Setting name: Se Backup Privilege (Back up files and directories)`, `*S-1-5-32-544 (Administrators)`. Nur feste oeffentliche Microsoft-Namen; Domaenen-SIDs werden wie bisher nur ueber ihre Standard-RID benannt, der Kundenwert wird nie gelesen. `REGULATORY_GPO_QUERY_VERSION` 1.0.0 -> 1.1.0; der naechste Batch erzeugt GPO-Mappings neu, Scanner-Mappings und bestehende Entscheidungen bleiben. Keine Core-Aenderung, keine Produktversionsanhebung (Engine-Verfeinerung im laufenden Stand 2.19.0). Korrektur desselben Tages: Die GPO-Query-Version ging nur in die Mapping-ID ein, nicht in das eindeutige Identitaetstupel der Tabelle `assessor_regulatory_mappings` (rule_key, corpus_version, retrieval_version, prompt_version, model). Ein Neulauf alter GPO-Regeln scheiterte daher mit `Regulatory mapping identity conflicted with another pipeline tuple.`. GPO-Laeufe speichern ihre `promptVersion` jetzt als `1.2.0+gpo-baseline-context-1.1.0`; Scanner-Laeufe und alle IDs bleiben unveraendert, keine Migration. Zweite Korrektur: Entscheidungen gelten global je Regel, das Fenster `Map ISO controls` listete aber nur Kontrollen des aktuellen Laufs. Eine von einem frueheren Lauf automatisch akzeptierte Kontrolle, die der neue Lauf nicht mehr abruft, blieb in der Finding-Reader-Sektion `ISO controls` (ohne Titel) und liess sich nicht ablehnen. Die Antwort traegt jetzt `earlierControls` (entschiedene, im aktuellen Lauf nicht abgerufene Kontrollen, Titel aus aelteren Laeufen der Regel); das Fenster zeigt sie unter `Earlier decisions` mit Accept/Reject, die Sektion uebernimmt ihren Titel. Synthetischer Interaktionstest (Reject auf einer frueheren Kontrolle) und Servicetest. Dritte Korrektur: Der Store verlangt, dass eine Entscheidung auf einen Lauf zeigt, der die Kontrolle abgerufen hat (`Decision control was not a grounded candidate.`); eine Entscheidung zu einer frueheren Kontrolle verweist deshalb auf den neuesten aelteren erfolgreichen Lauf der Regel, der sie abgerufen hat. Der Servicetest lehnt die fruehere Kontrolle jetzt ab.
|
||||||
|
|
||||||
2026-10-02 - Nutzerwunsch fuer spaeter dokumentiert: automatische Security-Baseline-Updates als zurueckgestelltes Arbeitspaket in Phase 13B, mit Phase-10-Backend und Phase-7-Anzeige. Datenqualitaet, Herkunft/Version/Format/Vollstaendigkeit, Erhalt der letzten gueltigen Baseline und unveraenderte historische Auswertungen sind Abnahmekriterien; Quellen und Intervall bleiben offen. Ausschliesslich Planaufnahme, keine Implementierung, Automatisierung oder Produktversionsaenderung.
|
2026-10-02 - Nutzerwunsch fuer spaeter dokumentiert: automatische Security-Baseline-Updates als zurueckgestelltes Arbeitspaket in Phase 13B, mit Phase-10-Backend und Phase-7-Anzeige. Datenqualitaet, Herkunft/Version/Format/Vollstaendigkeit, Erhalt der letzten gueltigen Baseline und unveraenderte historische Auswertungen sind Abnahmekriterien; Quellen und Intervall bleiben offen. Ausschliesslich Planaufnahme, keine Implementierung, Automatisierung oder Produktversionsaenderung.
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -3657,6 +3669,8 @@ Verifiziert 2026-09-09 (Phase 7): Bereichswechsel beschleunigt und Performance-V
|
||||||
Verifiziert 2026-09-08 (Phase 7/13B): GPO-Quellenkorrektur fuer Work und Client Info. Aktuelle Evaluationen aus dem vorhandenen autoritativen Readmodell statt ergebnisabhaengiger Backupgruppierung; Import- und Auswertungszeit getrennt, Source-Scan-Provenienz auch auf GPO-only-Blaettern. 1.014 Tests bestanden, acht optionale PostgreSQL-Tests uebersprungen; Local-only-/Designpruefung und beide Builds erfolgreich. Dummy lesend bestaetigt: eine statt drei GPO-Quellenoptionen, 3.805 aktuelle Zeilen, elf historische Evaluationen unveraendert. Keine Persistenzmutation oder Neuanalyse.
|
Verifiziert 2026-09-08 (Phase 7/13B): GPO-Quellenkorrektur fuer Work und Client Info. Aktuelle Evaluationen aus dem vorhandenen autoritativen Readmodell statt ergebnisabhaengiger Backupgruppierung; Import- und Auswertungszeit getrennt, Source-Scan-Provenienz auch auf GPO-only-Blaettern. 1.014 Tests bestanden, acht optionale PostgreSQL-Tests uebersprungen; Local-only-/Designpruefung und beide Builds erfolgreich. Dummy lesend bestaetigt: eine statt drei GPO-Quellenoptionen, 3.805 aktuelle Zeilen, elf historische Evaluationen unveraendert. Keine Persistenzmutation oder Neuanalyse.
|
||||||
|
|
||||||
| Datum | Änderung |
|
| Datum | Änderung |
|
||||||
|
| 2026-10-05 | Phasen 6/13F, in Produktversion 2.19.0 enthalten, Disclosure Policy 3.6.0 (Core-Freigabe Normalization, Disclosure, Presentation, Core Tests): Neues allgemeines Result-DB-Feld `technicalExplanation` fuer die technische Erklaerung einer Regel, wo ein Scanner sie liefert (heute PingCastle `Technical explanation`; andere Werkzeuge lassen es leer). Es wird beim Aufbau der Result DB aus dem bereits importierten Originalfeld gelesen (Schluessel `technical-explanation` oder Bezeichnung `Technical explanation`), daher ohne Reimport, in Local- und PostgreSQL-Repository ueber dieselbe Funktion. Es ist oeffentliche Regelanleitung wie `Description`, wird genauso streng redigiert (nur bekannte Scanner, unbekannte Quellen fail-closed, GPO ohne Feld), vom Identitaetsschutz geprueft und im inhaltsfreien Trace gefuehrt; `Details` mit Umgebungswerten bleibt zurueckgehalten. Die Policy-Erhoehung baut die gespeicherte Result DB neu auf. Result-DB-Details zeigen das Feld, Excel und CSV fuehren die Spalte nach `Description`. Die ISO-Zuordnung nimmt die redigierte technische Erklaerung in Suche und Modellanfrage auf; bestehende Zuordnungen behalten ihre ID. Tests: beide Importwege, Redaktion, Trace, Identitaetsschutz, fail-closed, interne Analyse mit Original, ISO-Abfrage. |
|
||||||
|
| 2026-10-05 | Phase 13F, Produktversion 2.19.0 (Minor: neue Work-Faehigkeit; Core-Freigabe Governance, Presentation, Platform, Core Tests): Work erhaelt die Compliance-Engine-Facetten `ISO control` (je akzeptierter Kontrolle, `A.8.20 · Networks security`) und `ISO theme` (Organizational, People, Physical, Technological) wie die Context-Cluster-Facette, direkt vor dem permanenten `Source: Scan`, in gemischtem Work und beiden GPO-Einstiegen (Erweiterung der GPO-Facettenliste per Nutzerentscheidung). Grundlage ist der neue Endpunkt `GET /projects/:id/regulatory-control-assignments` mit den akzeptierten Kontrollen aller zuordenbaren Findings des Timeslots (nicht nur Abweichungen), damit der Work-Statusfilter wie gewohnt wirkt; Scanner-Findings ueber das Vorkommen, GPO-Einstellungen ueber Scope und technische Identitaet (gemeinsamer Helfer `gpoWorkSettingIdentity`). Eine Anfrage je Timeslot fuer alle fuenf Work-Verbraucher, Neuladen nach jeder Entscheidung, aktive Auswahl bleibt waehrend des Ladens; ausgeschaltet entfallen die Facetten und ein aktiver ISO-Filter wird entfernt. Themenwerte werden kleingeschrieben verglichen, die Anzeige bleibt ISO-Schreibweise. Die Client-Info-Kachel oeffnet Work jetzt mit Status `Deviation` und der Kontrolle im `ISO control`-Filter statt einer festen Record-Liste. Aktivitaetslabel `Loading ISO control filters`. Tests: Werte, Zaehlung gegen andere Filter, Scanner- und GPO-Abgleich, Thema, Position vor `Source: Scan`, geteilte Anfrage, Neuladen, Engine aus mit Filterentfernung, Kachel-Sprung; Version-UI/API-Tests und beide Builds. |
|
||||||
| 2026-10-05 | Phase 13F, in Produktversion 2.18.0 enthalten (nur Engine-Code), vorlaeufige Nutzerentscheidung: Nach einer erfolgreichen Zuordnung werden die ersten zwei Vorschlaege in der Reihenfolge des Modells automatisch akzeptiert (bei nur einem Vorschlag nur dieser), mit dem Kommentar `Accepted automatically: one of the first two proposals of the mapping.`; eine Kontrolle mit bestehender Entscheidung behaelt sie, ein manuelles Reject wird also nie ueberschrieben, und ein Fehler dabei laesst die Zuordnung erfolgreich. Gilt fuer Einzel- und Batchzuordnung. Einmalige Nachholung ueber die normale Entscheidungs-API fuer die bestehenden 130 Regeln mit erfolgreichem Lauf (129 mit Vorschlaegen, 1 ohne): 210 Entscheidungen fuer 112 Regeln angelegt (98 mit zwei, 14 mit einer akzeptierten Kontrolle), 33 bereits entschiedene Kontrollen uebersprungen, keine Fehler. Die Regel steht seit der Governance-Freigabe vom selben Tag als vorlaeufige Nutzerentscheidung in den verbindlichen Projektregeln. |
|
| 2026-10-05 | Phase 13F, in Produktversion 2.18.0 enthalten (nur Engine-Code), vorlaeufige Nutzerentscheidung: Nach einer erfolgreichen Zuordnung werden die ersten zwei Vorschlaege in der Reihenfolge des Modells automatisch akzeptiert (bei nur einem Vorschlag nur dieser), mit dem Kommentar `Accepted automatically: one of the first two proposals of the mapping.`; eine Kontrolle mit bestehender Entscheidung behaelt sie, ein manuelles Reject wird also nie ueberschrieben, und ein Fehler dabei laesst die Zuordnung erfolgreich. Gilt fuer Einzel- und Batchzuordnung. Einmalige Nachholung ueber die normale Entscheidungs-API fuer die bestehenden 130 Regeln mit erfolgreichem Lauf (129 mit Vorschlaegen, 1 ohne): 210 Entscheidungen fuer 112 Regeln angelegt (98 mit zwei, 14 mit einer akzeptierten Kontrolle), 33 bereits entschiedene Kontrollen uebersprungen, keine Fehler. Die Regel steht seit der Governance-Freigabe vom selben Tag als vorlaeufige Nutzerentscheidung in den verbindlichen Projektregeln. |
|
||||||
| 2026-10-05 | Phase 13F, in Produktversion 2.18.0 enthalten (nur Engine-Code): Das Aktivitaetspanel sprang waehrend laufender ISO-Arbeit, weil jede Statusabfrage (alle 1,5 s bei Zuordnung und Batch, alle 2 s beim Korpus-Embedding) einen neuen Eintrag startete; das Panel oeffnet sich bei jedem neuen Eintrag und klappt die Historie zu, kurz sichtbar wurden dabei die Core-Gruppen der Historie. Wiederholte Statusabfragen laufen jetzt ueber einen eigenen Client ohne Protokolleintrag (`describeActivity`); der eine Eintrag je laufender Arbeit bleibt, das erste Laden jeder Ansicht wird weiter protokolliert. Tests ersetzen die Anfrage unterhalb der Protokollierung und pruefen, dass waehrend des Laufs kein neuer Eintrag startet; ohne die Korrektur scheitern sie. |
|
| 2026-10-05 | Phase 13F, in Produktversion 2.18.0 enthalten (nur Engine-Code): Das Aktivitaetspanel sprang waehrend laufender ISO-Arbeit, weil jede Statusabfrage (alle 1,5 s bei Zuordnung und Batch, alle 2 s beim Korpus-Embedding) einen neuen Eintrag startete; das Panel oeffnet sich bei jedem neuen Eintrag und klappt die Historie zu, kurz sichtbar wurden dabei die Core-Gruppen der Historie. Wiederholte Statusabfragen laufen jetzt ueber einen eigenen Client ohne Protokolleintrag (`describeActivity`); der eine Eintrag je laufender Arbeit bleibt, das erste Laden jeder Ansicht wird weiter protokolliert. Tests ersetzen die Anfrage unterhalb der Protokollierung und pruefen, dass waehrend des Laufs kein neuer Eintrag startet; ohne die Korrektur scheitern sie. |
|
||||||
| 2026-10-05 | Phase 13F, in Produktversion 2.18.0 enthalten (nur Engine-Code): Seit `Map ISO controls` ein Fenster ist, endete der Protokolleintrag `Mapping ISO controls` beim Schliessen des Fensters mit `continues in the backend`, und das Protokoll meldete `All tasks completed`, waehrend das Modell noch lief. Ein Lauf-Tracker der Engine haelt den Eintrag jetzt je laufender Zuordnung bis zum Endzustand des Backends, unabhaengig von Fenster, Finding und Ansicht; Fenster und Reader-Abschnitt uebernehmen seine Statusaktualisierungen, sodass nur einmal gepollt wird. Ausschalten der Engine beendet nur die Beobachtung. Tests fuer geschlossenes Fenster bis Abschluss und fuer einen fehlgeschlagenen Lauf. |
|
| 2026-10-05 | Phase 13F, in Produktversion 2.18.0 enthalten (nur Engine-Code): Seit `Map ISO controls` ein Fenster ist, endete der Protokolleintrag `Mapping ISO controls` beim Schliessen des Fensters mit `continues in the backend`, und das Protokoll meldete `All tasks completed`, waehrend das Modell noch lief. Ein Lauf-Tracker der Engine haelt den Eintrag jetzt je laufender Zuordnung bis zum Endzustand des Backends, unabhaengig von Fenster, Finding und Ansicht; Fenster und Reader-Abschnitt uebernehmen seine Statusaktualisierungen, sodass nur einmal gepollt wird. Ausschalten der Engine beendet nur die Beobachtung. Tests fuer geschlossenes Fenster bis Abschluss und fuer einen fehlgeschlagenen Lauf. |
|
||||||
|
|
|
||||||
|
|
@ -39,6 +39,8 @@ Clarification 2026-09-17 (baseline analyses of the written form): The Microsoft
|
||||||
|
|
||||||
Clarification 2026-09-17 (written security-template section and key): The Security Template parser folded the section name to lower case and discarded the written form, so `[Privilege Rights]` was retained only as `privilege rights`. The parser now keeps the written section and key; the case-folded forms stay internal to grammar selection, grouping and the identity digest, which keeps existing observation and identity IDs stable. The native target resolver stores the written section and key in the retained target, matching what the registry resolver already did, so Work shows `Privilege Rights - SeCreateTokenPrivilege` and the value stays searchable against Microsoft documentation and external tables. This changes persisted parsing output, so it is an append-only processing-version change, not a presentation change: `gpo-security-template-inf` 1.1.0 -> 1.2.0, `gpo-backup-policy-analysis` 2.2.3 -> 2.2.4 and the GPO job processor 2.2.3 -> 2.2.4. Historical imports keep their retained values unchanged and are marked for reprocessing: Data Hub reports an affected GPO backup as `Reprocessing required` with the analyzer versions and offers the existing reprocess action on the stored source, so no upload is needed. Reprocessing precedes re-evaluation, and prior runs remain retained as history. No SQL migration and no evaluator change.
|
Clarification 2026-09-17 (written security-template section and key): The Security Template parser folded the section name to lower case and discarded the written form, so `[Privilege Rights]` was retained only as `privilege rights`. The parser now keeps the written section and key; the case-folded forms stay internal to grammar selection, grouping and the identity digest, which keeps existing observation and identity IDs stable. The native target resolver stores the written section and key in the retained target, matching what the registry resolver already did, so Work shows `Privilege Rights - SeCreateTokenPrivilege` and the value stays searchable against Microsoft documentation and external tables. This changes persisted parsing output, so it is an append-only processing-version change, not a presentation change: `gpo-security-template-inf` 1.1.0 -> 1.2.0, `gpo-backup-policy-analysis` 2.2.3 -> 2.2.4 and the GPO job processor 2.2.3 -> 2.2.4. Historical imports keep their retained values unchanged and are marked for reprocessing: Data Hub reports an affected GPO backup as `Reprocessing required` with the analyzer versions and offers the existing reprocess action on the stored source, so no upload is needed. Reprocessing precedes re-evaluation, and prior runs remain retained as history. No SQL migration and no evaluator change.
|
||||||
|
|
||||||
|
Clarification 2026-10-05 (GPO domain key): A standard `Backup-GPO` folder names the domain by GUID in `bkupInfo.xml` and by DNS name in `Backup.xml`/`GPReport.xml`. The metadata aggregate treated these two identifier kinds as conflicting values, so every such backup stayed unresolved and blocked (`Limited analysis`). A GUID and a DNS name now count as two kinds of the same domain; only two different values of one kind conflict, and the GUID is preferred. This changes persisted identity resolution, so it is an append-only processing-version change: `gpo-backup-policy-analysis` and the GPO job processor 2.2.5 -> 2.2.6 (metadata parser 1.4.0 unchanged). Affected backups show `Reprocessing required`; the existing reprocess action on the stored source applies the fix. No upload, SQL migration or evaluator change.
|
||||||
|
|
||||||
Clarification 2026-09-25 (GPO link facts): The GPO metadata parser retains, from each backup's own `GPReport.xml`, the number of `LinksTo` entries and how many are enabled (never the linked OU or domain names). A backup with no enabled link is presented as `Not linked`: the Finding Reader marks its source sections and states `No effect` in the problem summary when every contributing GPO is unlinked, and GPO Processing shows the count. This is a backup-report fact at backup time, not a claim about effective device state, and verdicts and score are unchanged. Because parsing now retains a new fact, it is an append-only processing-version change: `gpo-backup-metadata-xml` 1.2.0 -> 1.4.0 (the Microsoft baseline template variant keeps 1.3.0, so baseline packages need no re-analysis), `gpo-backup-policy-analysis` 2.2.4 -> 2.2.5 and the GPO job processor 2.2.4 -> 2.2.5. Historical imports are marked `Reprocessing required` and the existing reprocess action on the stored source adds the link facts; no upload, SQL migration or evaluator change is required. Until then no link status is shown.
|
Clarification 2026-09-25 (GPO link facts): The GPO metadata parser retains, from each backup's own `GPReport.xml`, the number of `LinksTo` entries and how many are enabled (never the linked OU or domain names). A backup with no enabled link is presented as `Not linked`: the Finding Reader marks its source sections and states `No effect` in the problem summary when every contributing GPO is unlinked, and GPO Processing shows the count. This is a backup-report fact at backup time, not a claim about effective device state, and verdicts and score are unchanged. Because parsing now retains a new fact, it is an append-only processing-version change: `gpo-backup-metadata-xml` 1.2.0 -> 1.4.0 (the Microsoft baseline template variant keeps 1.3.0, so baseline packages need no re-analysis), `gpo-backup-policy-analysis` 2.2.4 -> 2.2.5 and the GPO job processor 2.2.4 -> 2.2.5. Historical imports are marked `Reprocessing required` and the existing reprocess action on the stored source adds the link facts; no upload, SQL migration or evaluator change is required. Until then no link status is shown.
|
||||||
|
|
||||||
Clarification 2026-09-17 (Not comparable verdict): A retained customer setting whose delivery mechanism has no comparable technical policy identity - today always a Group Policy Preference - reaches Work under the score-neutral verdict `Not comparable` instead of being dropped. It is never resolved into a policy identity and never compared with a Microsoft baseline. The unified read projection additionally exposes the already retained preference CSE, action and item-level targeting status so that context stays readable in the agreed GPO columns; no new column, facet or persisted field is introduced. Snapshot-aggregation review hints that restate an unresolved identity already reported by their source analysis are no longer emitted a second time. All of this projects facts the importer already retained, so no processing-manifest version, source reimport, evaluator change or SQL migration is required.
|
Clarification 2026-09-17 (Not comparable verdict): A retained customer setting whose delivery mechanism has no comparable technical policy identity - today always a Group Policy Preference - reaches Work under the score-neutral verdict `Not comparable` instead of being dropped. It is never resolved into a policy identity and never compared with a Microsoft baseline. The unified read projection additionally exposes the already retained preference CSE, action and item-level targeting status so that context stays readable in the agreed GPO columns; no new column, facet or persisted field is introduced. Snapshot-aggregation review hints that restate an unresolved identity already reported by their source analysis are no longer emitted a second time. All of this projects facts the importer already retained, so no processing-manifest version, source reimport, evaluator change or SQL migration is required.
|
||||||
|
|
|
||||||
|
|
@ -1,12 +1,12 @@
|
||||||
{
|
{
|
||||||
"name": "assessor",
|
"name": "assessor",
|
||||||
"version": "2.18.0",
|
"version": "2.19.0",
|
||||||
"lockfileVersion": 3,
|
"lockfileVersion": 3,
|
||||||
"requires": true,
|
"requires": true,
|
||||||
"packages": {
|
"packages": {
|
||||||
"": {
|
"": {
|
||||||
"name": "assessor",
|
"name": "assessor",
|
||||||
"version": "2.18.0",
|
"version": "2.19.0",
|
||||||
"dependencies": {
|
"dependencies": {
|
||||||
"@fontsource-variable/inter": "^5.3.0",
|
"@fontsource-variable/inter": "^5.3.0",
|
||||||
"jsdom": "^29.1.1",
|
"jsdom": "^29.1.1",
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,6 @@
|
||||||
{
|
{
|
||||||
"name": "assessor",
|
"name": "assessor",
|
||||||
"version": "2.18.0",
|
"version": "2.19.0",
|
||||||
"private": true,
|
"private": true,
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"scripts": {
|
"scripts": {
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,9 @@
|
||||||
import type { AiOverviewTokenUsage } from '../../src/domain/ai-overview';
|
/** Token counts the provider reports for one completion. */
|
||||||
|
export interface AiCompletionTokenUsage {
|
||||||
|
promptTokens?: number;
|
||||||
|
completionTokens?: number;
|
||||||
|
totalTokens?: number;
|
||||||
|
}
|
||||||
|
|
||||||
export interface LiteLlmEnvironment {
|
export interface LiteLlmEnvironment {
|
||||||
ASSESSOR_AI_MODE?: string;
|
ASSESSOR_AI_MODE?: string;
|
||||||
|
|
@ -11,7 +16,7 @@ export interface LiteLlmEnvironment {
|
||||||
|
|
||||||
export interface AiCompletion {
|
export interface AiCompletion {
|
||||||
value: unknown;
|
value: unknown;
|
||||||
usage?: AiOverviewTokenUsage;
|
usage?: AiCompletionTokenUsage;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface AiCompletionClient {
|
export interface AiCompletionClient {
|
||||||
|
|
@ -206,7 +211,7 @@ function completionContent(value: unknown): string {
|
||||||
return value.choices[0].message.content;
|
return value.choices[0].message.content;
|
||||||
}
|
}
|
||||||
|
|
||||||
function completionUsage(value: unknown): AiOverviewTokenUsage | undefined {
|
function completionUsage(value: unknown): AiCompletionTokenUsage | undefined {
|
||||||
if (!isObject(value) || !isObject(value.usage)) return undefined;
|
if (!isObject(value) || !isObject(value.usage)) return undefined;
|
||||||
const number = (entry: unknown) => typeof entry === 'number' && Number.isFinite(entry) ? entry : undefined;
|
const number = (entry: unknown) => typeof entry === 'number' && Number.isFinite(entry) ? entry : undefined;
|
||||||
const usage = {
|
const usage = {
|
||||||
|
|
|
||||||
|
|
@ -29,9 +29,9 @@ import type { GpoPolicySnapshotService } from './gpo/gpo-policy-snapshot-service
|
||||||
import type { GpoPolicyEvaluationService } from './gpo/gpo-policy-evaluation-service';
|
import type { GpoPolicyEvaluationService } from './gpo/gpo-policy-evaluation-service';
|
||||||
import type { GpoPolicyFindingService } from './gpo/gpo-policy-finding-service';
|
import type { GpoPolicyFindingService } from './gpo/gpo-policy-finding-service';
|
||||||
import type { GpoPolicyAnalysisOverviewService } from './gpo/gpo-policy-analysis-overview-service';
|
import type { GpoPolicyAnalysisOverviewService } from './gpo/gpo-policy-analysis-overview-service';
|
||||||
import type { AiOverviewService } from './ai/ai-overview-service';
|
import type { AiOverviewService } from './compliance/ai-overview-service';
|
||||||
import { AiOverviewServiceError } from './ai/ai-overview-service';
|
import { AiOverviewServiceError } from './compliance/ai-overview-service';
|
||||||
import type { AiOverviewCreateRequest, AiOverviewScope } from '../src/domain/ai-overview';
|
import type { AiOverviewCreateRequest, AiOverviewScope } from '../src/compliance/ai-overview';
|
||||||
import type {
|
import type {
|
||||||
RegulatoryMappingBatchCreateRequest,
|
RegulatoryMappingBatchCreateRequest,
|
||||||
RegulatoryMappingBatchScope,
|
RegulatoryMappingBatchScope,
|
||||||
|
|
@ -1175,6 +1175,10 @@ export function createAssessorHttpServer(options: AssessorHttpServerOptions): Se
|
||||||
const body = await readJson<RegulatoryMappingDecisionCreateRequest>(request, maximumBodyBytes);
|
const body = await readJson<RegulatoryMappingDecisionCreateRequest>(request, maximumBodyBytes);
|
||||||
status = 200; sendJson(response, status, await options.regulatoryMappingService.decide(segments[1], body), requestId); return;
|
status = 200; sendJson(response, status, await options.regulatoryMappingService.decide(segments[1], body), requestId); return;
|
||||||
}
|
}
|
||||||
|
if (segments[0] === 'projects' && segments[2] === 'regulatory-control-assignments' && segments.length === 3 && method === 'GET') {
|
||||||
|
if (!options.regulatoryMappingService) throw new HttpError(503, 'regulatory_mapping_disabled', 'Regulatory mapping is not configured.');
|
||||||
|
status = 200; sendJson(response, status, await options.regulatoryMappingService.controlAssignments(segments[1], regulatoryMappingBatchScopeFrom(url)), requestId); return;
|
||||||
|
}
|
||||||
if (segments[0] === 'projects' && segments[2] === 'regulatory-control-overview' && segments.length === 3 && method === 'GET') {
|
if (segments[0] === 'projects' && segments[2] === 'regulatory-control-overview' && segments.length === 3 && method === 'GET') {
|
||||||
if (!options.regulatoryMappingService) throw new HttpError(503, 'regulatory_mapping_disabled', 'Regulatory mapping is not configured.');
|
if (!options.regulatoryMappingService) throw new HttpError(503, 'regulatory_mapping_disabled', 'Regulatory mapping is not configured.');
|
||||||
status = 200; sendJson(response, status, await options.regulatoryMappingService.controlOverview(segments[1], regulatoryMappingBatchScopeFrom(url)), requestId); return;
|
status = 200; sendJson(response, status, await options.regulatoryMappingService.controlOverview(segments[1], regulatoryMappingBatchScopeFrom(url)), requestId); return;
|
||||||
|
|
|
||||||
|
|
@ -1,7 +1,7 @@
|
||||||
import { createHash, randomUUID } from 'node:crypto';
|
import { createHash, randomUUID } from 'node:crypto';
|
||||||
import type { AssessorRepository } from '../../src/persistence/assessor-repository';
|
import type { ComplianceReadPort, ComplianceStore } from '../../src/compliance/compliance-store';
|
||||||
import { RepositoryConflictError, RepositoryNotFoundError, RepositoryValidationError } from '../../src/persistence/repository-errors';
|
import { RepositoryConflictError, RepositoryNotFoundError, RepositoryValidationError } from '../../src/persistence/repository-errors';
|
||||||
import { resultsToPrivacySafeExportJson } from '../../src/exports/result-db-copy';
|
import { aiOverviewInputJson } from '../../src/compliance/privacy-safe-export';
|
||||||
import type { UnifiedResultProjection } from '../../src/domain/unified-result';
|
import type { UnifiedResultProjection } from '../../src/domain/unified-result';
|
||||||
import {
|
import {
|
||||||
AI_OVERVIEW_OUTPUT_SCHEMA_VERSION,
|
AI_OVERVIEW_OUTPUT_SCHEMA_VERSION,
|
||||||
|
|
@ -15,9 +15,9 @@ import {
|
||||||
type AiOverviewRun,
|
type AiOverviewRun,
|
||||||
type AiOverviewScope,
|
type AiOverviewScope,
|
||||||
type AiOverviewTokenUsage
|
type AiOverviewTokenUsage
|
||||||
} from '../../src/domain/ai-overview';
|
} from '../../src/compliance/ai-overview';
|
||||||
import type { AiCompletionClient } from './litellm-client';
|
import type { AiCompletionClient } from '../ai/litellm-client';
|
||||||
import { LiteLlmClientError } from './litellm-client';
|
import { LiteLlmClientError } from '../ai/litellm-client';
|
||||||
|
|
||||||
const PAGE_SIZE = 500;
|
const PAGE_SIZE = 500;
|
||||||
const ACTIVE_POLL_MS = 1_500;
|
const ACTIVE_POLL_MS = 1_500;
|
||||||
|
|
@ -57,7 +57,7 @@ export class AiOverviewService {
|
||||||
private readonly workerId: string;
|
private readonly workerId: string;
|
||||||
private readonly activeTasks = new Set<Promise<void>>();
|
private readonly activeTasks = new Set<Promise<void>>();
|
||||||
|
|
||||||
constructor(private readonly repository: AssessorRepository, options: AiOverviewServiceOptions = {}) {
|
constructor(private readonly read: ComplianceReadPort, private readonly store: ComplianceStore, options: AiOverviewServiceOptions = {}) {
|
||||||
this.client = options.client;
|
this.client = options.client;
|
||||||
this.now = options.now ?? (() => new Date());
|
this.now = options.now ?? (() => new Date());
|
||||||
this.maximumChunkCharacters = options.maximumChunkCharacters ?? 80_000;
|
this.maximumChunkCharacters = options.maximumChunkCharacters ?? 80_000;
|
||||||
|
|
@ -76,10 +76,11 @@ export class AiOverviewService {
|
||||||
|
|
||||||
async recoverInterruptedRuns(): Promise<void> {
|
async recoverInterruptedRuns(): Promise<void> {
|
||||||
const now = this.now().toISOString();
|
const now = this.now().toISOString();
|
||||||
for (const project of await this.repository.listProjects()) {
|
for (const project of await this.read.listProjects()) {
|
||||||
const runs = await this.repository.listAiOverviewRuns(project.id);
|
await this.pruneDeletedTimeslots(project.id);
|
||||||
|
const runs = await this.store.listAiOverviewRuns(project.id);
|
||||||
for (const run of runs.filter(item => ['queued', 'running'].includes(item.status))) {
|
for (const run of runs.filter(item => ['queued', 'running'].includes(item.status))) {
|
||||||
await this.repository.updateAiOverviewRun({
|
await this.store.updateAiOverviewRun({
|
||||||
...run,
|
...run,
|
||||||
status: 'failed',
|
status: 'failed',
|
||||||
updatedAt: now,
|
updatedAt: now,
|
||||||
|
|
@ -93,10 +94,27 @@ export class AiOverviewService {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The Core deletes scan runs and assessments without knowing this engine. Their AI overviews are
|
||||||
|
* derived from deleted data, so the engine removes them from its own table on start and on read.
|
||||||
|
*/
|
||||||
|
async pruneDeletedTimeslots(projectId: string): Promise<void> {
|
||||||
|
const [assessments, scanRuns, runs] = await Promise.all([
|
||||||
|
this.read.listAssessments(projectId), this.read.listProjectScanRuns(projectId), this.store.listAiOverviewRuns(projectId)
|
||||||
|
]);
|
||||||
|
const assessmentIds = new Set(assessments.map(item => item.id));
|
||||||
|
const scanRunIds = new Set(scanRuns.map(item => item.id));
|
||||||
|
const orphaned = runs.filter(run => !['queued', 'running'].includes(run.status)
|
||||||
|
&& (run.assessmentIds.some(id => !assessmentIds.has(id)) || run.scanRunIds.some(id => !scanRunIds.has(id))));
|
||||||
|
await this.store.deleteAiOverviewRuns(projectId, orphaned.map(run => run.id));
|
||||||
|
}
|
||||||
|
|
||||||
async get(projectId: string, requestedScope: AiOverviewScope): Promise<AiOverviewResponse> {
|
async get(projectId: string, requestedScope: AiOverviewScope): Promise<AiOverviewResponse> {
|
||||||
|
if (!await this.read.getProject(projectId)) throw new RepositoryNotFoundError(`Projekt ${projectId} ist nicht vorhanden.`);
|
||||||
|
await this.pruneDeletedTimeslots(projectId);
|
||||||
const scope = await this.validateScope(projectId, requestedScope);
|
const scope = await this.validateScope(projectId, requestedScope);
|
||||||
const scopeKey = aiOverviewScopeKey(projectId, scope);
|
const scopeKey = aiOverviewScopeKey(projectId, scope);
|
||||||
const runs = await this.repository.listAiOverviewRuns(projectId, scopeKey);
|
const runs = await this.store.listAiOverviewRuns(projectId, scopeKey);
|
||||||
const run = runs[0];
|
const run = runs[0];
|
||||||
const previousSuccessfulRun = run?.status === 'succeeded' ? undefined : runs.find(item => item.status === 'succeeded');
|
const previousSuccessfulRun = run?.status === 'succeeded' ? undefined : runs.find(item => item.status === 'succeeded');
|
||||||
let stale = false;
|
let stale = false;
|
||||||
|
|
@ -136,13 +154,13 @@ export class AiOverviewService {
|
||||||
updatedAt: timestamp,
|
updatedAt: timestamp,
|
||||||
callCount: 0
|
callCount: 0
|
||||||
};
|
};
|
||||||
const run = await this.repository.createAiOverviewRun(candidate);
|
const run = await this.store.createAiOverviewRun(candidate);
|
||||||
if (run.id === candidate.id) {
|
if (run.id === candidate.id) {
|
||||||
const task = this.process(run, prepared).finally(() => this.activeTasks.delete(task));
|
const task = this.process(run, prepared).finally(() => this.activeTasks.delete(task));
|
||||||
this.activeTasks.add(task);
|
this.activeTasks.add(task);
|
||||||
}
|
}
|
||||||
const previousSuccessfulRun = run.status === 'succeeded' ? undefined
|
const previousSuccessfulRun = run.status === 'succeeded' ? undefined
|
||||||
: (await this.repository.listAiOverviewRuns(projectId, run.scopeKey)).find(item => item.status === 'succeeded');
|
: (await this.store.listAiOverviewRuns(projectId, run.scopeKey)).find(item => item.status === 'succeeded');
|
||||||
return responseFor(projectId, scope, true, run, false, previousSuccessfulRun);
|
return responseFor(projectId, scope, true, run, false, previousSuccessfulRun);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -161,7 +179,7 @@ export class AiOverviewService {
|
||||||
leaseExpiresAt: new Date(this.now().getTime() + this.leaseDurationMs).toISOString()
|
leaseExpiresAt: new Date(this.now().getTime() + this.leaseDurationMs).toISOString()
|
||||||
};
|
};
|
||||||
try {
|
try {
|
||||||
run = await this.repository.updateAiOverviewRun(run);
|
run = await this.store.updateAiOverviewRun(run);
|
||||||
const completion = await generateAiOverview(prepared.parsed, {
|
const completion = await generateAiOverview(prepared.parsed, {
|
||||||
client: this.client!,
|
client: this.client!,
|
||||||
maximumChunkCharacters: this.maximumChunkCharacters,
|
maximumChunkCharacters: this.maximumChunkCharacters,
|
||||||
|
|
@ -171,7 +189,7 @@ export class AiOverviewService {
|
||||||
factProperties: prepared.factProperties,
|
factProperties: prepared.factProperties,
|
||||||
onCall: async usage => {
|
onCall: async usage => {
|
||||||
const updatedAt = this.now().toISOString();
|
const updatedAt = this.now().toISOString();
|
||||||
run = await this.repository.updateAiOverviewRun({
|
run = await this.store.updateAiOverviewRun({
|
||||||
...run,
|
...run,
|
||||||
callCount: run.callCount + 1,
|
callCount: run.callCount + 1,
|
||||||
tokenUsage: addUsage(run.tokenUsage, usage),
|
tokenUsage: addUsage(run.tokenUsage, usage),
|
||||||
|
|
@ -181,7 +199,7 @@ export class AiOverviewService {
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
const completedAt = this.now().toISOString();
|
const completedAt = this.now().toISOString();
|
||||||
await this.repository.updateAiOverviewRun({
|
await this.store.updateAiOverviewRun({
|
||||||
...run,
|
...run,
|
||||||
status: 'succeeded',
|
status: 'succeeded',
|
||||||
result: completion,
|
result: completion,
|
||||||
|
|
@ -194,7 +212,7 @@ export class AiOverviewService {
|
||||||
const completedAt = this.now().toISOString();
|
const completedAt = this.now().toISOString();
|
||||||
const failure = safeGenerationFailure(error);
|
const failure = safeGenerationFailure(error);
|
||||||
try {
|
try {
|
||||||
await this.repository.updateAiOverviewRun({
|
await this.store.updateAiOverviewRun({
|
||||||
...run,
|
...run,
|
||||||
status: 'failed',
|
status: 'failed',
|
||||||
updatedAt: completedAt,
|
updatedAt: completedAt,
|
||||||
|
|
@ -211,13 +229,13 @@ export class AiOverviewService {
|
||||||
}
|
}
|
||||||
|
|
||||||
private async validateScope(projectId: string, requested: AiOverviewScope): Promise<AiOverviewScope> {
|
private async validateScope(projectId: string, requested: AiOverviewScope): Promise<AiOverviewScope> {
|
||||||
if (!await this.repository.getProject(projectId)) throw new RepositoryNotFoundError(`Projekt ${projectId} ist nicht vorhanden.`);
|
if (!await this.read.getProject(projectId)) throw new RepositoryNotFoundError(`Projekt ${projectId} ist nicht vorhanden.`);
|
||||||
const scope = normalizeAiOverviewScope(requested);
|
const scope = normalizeAiOverviewScope(requested);
|
||||||
if (!scope.assessmentIds.length && !scope.scanRunIds.length) throw new RepositoryValidationError('A selected timeslot scope is required.');
|
if (!scope.assessmentIds.length && !scope.scanRunIds.length) throw new RepositoryValidationError('A selected timeslot scope is required.');
|
||||||
const assessments = await this.repository.listAssessments(projectId);
|
const assessments = await this.read.listAssessments(projectId);
|
||||||
const allowedAssessmentIds = new Set(assessments.map(item => item.id));
|
const allowedAssessmentIds = new Set(assessments.map(item => item.id));
|
||||||
if (scope.assessmentIds.some(id => !allowedAssessmentIds.has(id))) throw new RepositoryValidationError('The AI overview scope contains an assessment outside this project.');
|
if (scope.assessmentIds.some(id => !allowedAssessmentIds.has(id))) throw new RepositoryValidationError('The AI overview scope contains an assessment outside this project.');
|
||||||
const scanRuns = await this.repository.listProjectScanRuns(projectId);
|
const scanRuns = await this.read.listProjectScanRuns(projectId);
|
||||||
const allowedRuns = new Map(scanRuns.filter(item => !item.supersededByScanRunId).map(item => [item.id, item]));
|
const allowedRuns = new Map(scanRuns.filter(item => !item.supersededByScanRunId).map(item => [item.id, item]));
|
||||||
if (scope.scanRunIds.some(id => !allowedRuns.has(id))) throw new RepositoryValidationError('The AI overview scope contains an unavailable scan run.');
|
if (scope.scanRunIds.some(id => !allowedRuns.has(id))) throw new RepositoryValidationError('The AI overview scope contains an unavailable scan run.');
|
||||||
if (scope.assessmentIds.length && scope.scanRunIds.some(id => !scope.assessmentIds.includes(allowedRuns.get(id)!.assessmentId))) {
|
if (scope.assessmentIds.length && scope.scanRunIds.some(id => !scope.assessmentIds.includes(allowedRuns.get(id)!.assessmentId))) {
|
||||||
|
|
@ -227,9 +245,9 @@ export class AiOverviewService {
|
||||||
}
|
}
|
||||||
|
|
||||||
private async prepareInput(projectId: string, scope: AiOverviewScope): Promise<PreparedInput> {
|
private async prepareInput(projectId: string, scope: AiOverviewScope): Promise<PreparedInput> {
|
||||||
const items = await queryAllUnifiedResults(this.repository, projectId, scope);
|
const items = await queryAllUnifiedResults(this.read, projectId, scope);
|
||||||
const ordered = items.slice().sort((left, right) => stableResultKey(left).localeCompare(stableResultKey(right)));
|
// The same function builds the Result DB `Export JSON`, so the export is exactly what the model receives.
|
||||||
const json = resultsToPrivacySafeExportJson(ordered);
|
const json = aiOverviewInputJson(items);
|
||||||
const parsed = parsePrivacySafeInput(json);
|
const parsed = parsePrivacySafeInput(json);
|
||||||
const findingIds = new Set(parsed.findings.flatMap(item => typeof item.id === 'string' ? [item.id] : []));
|
const findingIds = new Set(parsed.findings.flatMap(item => typeof item.id === 'string' ? [item.id] : []));
|
||||||
const gapIds = new Set([
|
const gapIds = new Set([
|
||||||
|
|
@ -309,7 +327,7 @@ export function splitPrivacySafeInput(input: PrivacySafeInput, maximumCharacters
|
||||||
return chunks;
|
return chunks;
|
||||||
}
|
}
|
||||||
|
|
||||||
async function queryAllUnifiedResults(repository: AssessorRepository, projectId: string, scope: AiOverviewScope): Promise<UnifiedResultProjection[]> {
|
async function queryAllUnifiedResults(repository: ComplianceReadPort, projectId: string, scope: AiOverviewScope): Promise<UnifiedResultProjection[]> {
|
||||||
const items: UnifiedResultProjection[] = [];
|
const items: UnifiedResultProjection[] = [];
|
||||||
let expectedTotal: number | undefined;
|
let expectedTotal: number | undefined;
|
||||||
for (let offset = 0; ; offset += PAGE_SIZE) {
|
for (let offset = 0; ; offset += PAGE_SIZE) {
|
||||||
|
|
@ -351,10 +369,6 @@ function parsePrivacySafeInput(json: string): PrivacySafeInput {
|
||||||
return value as PrivacySafeInput;
|
return value as PrivacySafeInput;
|
||||||
}
|
}
|
||||||
|
|
||||||
function stableResultKey(item: UnifiedResultProjection): string {
|
|
||||||
return [item.sourceKind, item.sourceTool ?? '', item.sourceRuleId ?? '', item.resultProjectionId].join('\u0000');
|
|
||||||
}
|
|
||||||
|
|
||||||
function responseFor(projectId: string, scope: AiOverviewScope, enabled: boolean, run?: AiOverviewRun, stale = false, previousSuccessfulRun?: AiOverviewRun): AiOverviewResponse {
|
function responseFor(projectId: string, scope: AiOverviewScope, enabled: boolean, run?: AiOverviewRun, stale = false, previousSuccessfulRun?: AiOverviewRun): AiOverviewResponse {
|
||||||
return {
|
return {
|
||||||
version: 1,
|
version: 1,
|
||||||
|
|
@ -9,10 +9,12 @@ import { PostgresComplianceStore } from './postgres-compliance-store';
|
||||||
import { createRegulatoryRagRuntime } from './regulatory-rag-runtime';
|
import { createRegulatoryRagRuntime } from './regulatory-rag-runtime';
|
||||||
import { RegulatoryMappingService } from './regulatory-mapping-service';
|
import { RegulatoryMappingService } from './regulatory-mapping-service';
|
||||||
import { RegulatoryCorpusUploadService } from './regulatory-corpus-upload-service';
|
import { RegulatoryCorpusUploadService } from './regulatory-corpus-upload-service';
|
||||||
|
import { AiOverviewService, type AiOverviewServiceOptions } from './ai-overview-service';
|
||||||
|
|
||||||
export interface ComplianceEngineRuntime {
|
export interface ComplianceEngineRuntime {
|
||||||
mappingService: RegulatoryMappingService;
|
mappingService: RegulatoryMappingService;
|
||||||
corpusUploadService: RegulatoryCorpusUploadService;
|
corpusUploadService: RegulatoryCorpusUploadService;
|
||||||
|
aiOverviewService: AiOverviewService;
|
||||||
store: ComplianceStore;
|
store: ComplianceStore;
|
||||||
/** Stops running work; ASSESSOR keeps its own repository open. */
|
/** Stops running work; ASSESSOR keeps its own repository open. */
|
||||||
stop(): Promise<void>;
|
stop(): Promise<void>;
|
||||||
|
|
@ -25,6 +27,7 @@ export interface ComplianceEngineRuntimeOptions {
|
||||||
aiClient?: AiCompletionClient;
|
aiClient?: AiCompletionClient;
|
||||||
artifactRoot: string;
|
artifactRoot: string;
|
||||||
environment?: RepositoryEnvironment;
|
environment?: RepositoryEnvironment;
|
||||||
|
aiOverview?: Omit<AiOverviewServiceOptions, 'client'>;
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|
@ -65,11 +68,14 @@ export async function createComplianceEngineRuntime(options: ComplianceEngineRun
|
||||||
disabledReason
|
disabledReason
|
||||||
});
|
});
|
||||||
await corpusUploadService.initialize();
|
await corpusUploadService.initialize();
|
||||||
|
const aiOverviewService = new AiOverviewService(read, store, { ...options.aiOverview, client: options.aiClient });
|
||||||
|
await aiOverviewService.recoverInterruptedRuns();
|
||||||
return {
|
return {
|
||||||
mappingService,
|
mappingService,
|
||||||
corpusUploadService,
|
corpusUploadService,
|
||||||
|
aiOverviewService,
|
||||||
store,
|
store,
|
||||||
stop: () => corpusUploadService.stop().then(() => mappingService.close()),
|
stop: () => Promise.all([corpusUploadService.stop().then(() => mappingService.close()), aiOverviewService.close()]).then(() => undefined),
|
||||||
close: async () => { await pool?.end(); }
|
close: async () => { await pool?.end(); }
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -10,6 +10,7 @@ import {
|
||||||
type RegulatoryMappingDecision,
|
type RegulatoryMappingDecision,
|
||||||
type RegulatoryMappingRun
|
type RegulatoryMappingRun
|
||||||
} from '../../src/compliance/regulatory-mapping';
|
} from '../../src/compliance/regulatory-mapping';
|
||||||
|
import { aiOverviewRunValidationErrors, assertAiOverviewTransition, type AiOverviewRun } from '../../src/compliance/ai-overview';
|
||||||
|
|
||||||
interface DataRow<T> { data: T }
|
interface DataRow<T> { data: T }
|
||||||
|
|
||||||
|
|
@ -30,7 +31,7 @@ function dataRow<T>(result: QueryResult<DataRow<T>>): T | null {
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* The Compliance Engine's own tables (migration 026). Writes run in the engine's own transactions:
|
* The Compliance Engine's own tables (migration 026, and the AI overview runs of migration 025). Writes run in the engine's own transactions:
|
||||||
* they never advance the ASSESSOR repository revision and never trigger a Result DB rebuild.
|
* they never advance the ASSESSOR repository revision and never trigger a Result DB rebuild.
|
||||||
*/
|
*/
|
||||||
export class PostgresComplianceStore implements ComplianceStore {
|
export class PostgresComplianceStore implements ComplianceStore {
|
||||||
|
|
@ -211,13 +212,68 @@ export class PostgresComplianceStore implements ComplianceStore {
|
||||||
`, [ruleKey]));
|
`, [ruleKey]));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async createAiOverviewRun(run: AiOverviewRun): Promise<AiOverviewRun> {
|
||||||
|
const errors = aiOverviewRunValidationErrors(run);
|
||||||
|
if (errors.length) throw new RepositoryValidationError(errors.join(' '));
|
||||||
|
return this.transaction(async client => {
|
||||||
|
const idempotent = dataRow(await client.query<DataRow<AiOverviewRun>>(`
|
||||||
|
SELECT data FROM assessor_ai_overview_runs
|
||||||
|
WHERE project_id = $1 AND scope_key = $2 AND idempotency_key = $3
|
||||||
|
`, [run.projectId, run.scopeKey, run.idempotencyKey]));
|
||||||
|
if (idempotent) return idempotent;
|
||||||
|
await client.query(`
|
||||||
|
INSERT INTO assessor_ai_overview_runs
|
||||||
|
(id, project_id, scope_key, assessment_ids, scan_run_ids, idempotency_key, run_status,
|
||||||
|
input_hash, prompt_version, output_schema_version, model, created_at, updated_at, data)
|
||||||
|
VALUES ($1, $2, $3, $4::text[], $5::text[], $6, $7, $8, $9, $10, $11, $12, $13, $14::jsonb)
|
||||||
|
`, [run.id, run.projectId, run.scopeKey, run.assessmentIds, run.scanRunIds, run.idempotencyKey,
|
||||||
|
run.status, run.inputHash, run.promptVersion, run.outputSchemaVersion, run.model,
|
||||||
|
run.createdAt, run.updatedAt, json(run)]);
|
||||||
|
return clone(run);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
async updateAiOverviewRun(run: AiOverviewRun): Promise<AiOverviewRun> {
|
||||||
|
const errors = aiOverviewRunValidationErrors(run);
|
||||||
|
if (errors.length) throw new RepositoryValidationError(errors.join(' '));
|
||||||
|
return this.transaction(async client => {
|
||||||
|
const existing = dataRow(await client.query<DataRow<AiOverviewRun>>(
|
||||||
|
'SELECT data FROM assessor_ai_overview_runs WHERE id = $1 FOR UPDATE', [run.id]
|
||||||
|
));
|
||||||
|
if (!existing) throw new RepositoryNotFoundError(`AI overview run ${run.id} does not exist.`);
|
||||||
|
assertAiOverviewTransition(existing, run);
|
||||||
|
await client.query(`
|
||||||
|
UPDATE assessor_ai_overview_runs SET
|
||||||
|
run_status = $2, updated_at = $3, data = $4::jsonb
|
||||||
|
WHERE id = $1
|
||||||
|
`, [run.id, run.status, run.updatedAt, json(run)]);
|
||||||
|
return clone(run);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
async listAiOverviewRuns(projectId: string, scopeKey?: string): Promise<AiOverviewRun[]> {
|
||||||
|
return dataRows(await this.pool.query<DataRow<AiOverviewRun>>(`
|
||||||
|
SELECT data FROM assessor_ai_overview_runs
|
||||||
|
WHERE project_id = $1 AND ($2::text IS NULL OR scope_key = $2)
|
||||||
|
ORDER BY created_at DESC, id DESC
|
||||||
|
`, [projectId, scopeKey ?? null]));
|
||||||
|
}
|
||||||
|
|
||||||
|
async deleteAiOverviewRuns(projectId: string, runIds: string[]): Promise<void> {
|
||||||
|
if (!runIds.length) return;
|
||||||
|
await this.transaction(client => client.query(
|
||||||
|
'DELETE FROM assessor_ai_overview_runs WHERE project_id = $1 AND id = ANY($2::text[])', [projectId, runIds]
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
async exportProject(projectId: string): Promise<ComplianceProjectExport> {
|
async exportProject(projectId: string): Promise<ComplianceProjectExport> {
|
||||||
return {
|
return {
|
||||||
regulatoryMappingBatchRuns: await this.listRegulatoryMappingBatchRuns(projectId),
|
regulatoryMappingBatchRuns: await this.listRegulatoryMappingBatchRuns(projectId),
|
||||||
regulatoryMappingRuns: (await this.listRegulatoryMappingRuns()).filter(item => item.status === 'succeeded'),
|
regulatoryMappingRuns: (await this.listRegulatoryMappingRuns()).filter(item => item.status === 'succeeded'),
|
||||||
regulatoryMappingDecisions: dataRows(await this.pool.query<DataRow<RegulatoryMappingDecision>>(
|
regulatoryMappingDecisions: dataRows(await this.pool.query<DataRow<RegulatoryMappingDecision>>(
|
||||||
'SELECT data FROM assessor_regulatory_mapping_decisions ORDER BY created_at, id'
|
'SELECT data FROM assessor_regulatory_mapping_decisions ORDER BY created_at, id'
|
||||||
))
|
)),
|
||||||
|
aiOverviewRuns: (await this.listAiOverviewRuns(projectId)).filter(item => item.status === 'succeeded')
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -7,6 +7,7 @@ import {
|
||||||
REGULATORY_MAPPING_OUTPUT_SCHEMA_VERSION,
|
REGULATORY_MAPPING_OUTPUT_SCHEMA_VERSION,
|
||||||
REGULATORY_MAPPING_PROMPT_VERSION,
|
REGULATORY_MAPPING_PROMPT_VERSION,
|
||||||
REGULATORY_RETRIEVAL_VERSION,
|
REGULATORY_RETRIEVAL_VERSION,
|
||||||
|
REGULATORY_GPO_QUERY_VERSION,
|
||||||
emptyRegulatoryMappingBatchCounts,
|
emptyRegulatoryMappingBatchCounts,
|
||||||
groundRegulatoryMappingResult,
|
groundRegulatoryMappingResult,
|
||||||
normalizeRegulatoryMappingBatchScope,
|
normalizeRegulatoryMappingBatchScope,
|
||||||
|
|
@ -18,6 +19,7 @@ import {
|
||||||
type RegulatoryMappingCandidate,
|
type RegulatoryMappingCandidate,
|
||||||
type RegulatoryMappingCreateRequest,
|
type RegulatoryMappingCreateRequest,
|
||||||
type RegulatoryMappingDecision,
|
type RegulatoryMappingDecision,
|
||||||
|
type RegulatoryEarlierControl,
|
||||||
type RegulatoryMappingDecisionCreateRequest,
|
type RegulatoryMappingDecisionCreateRequest,
|
||||||
type RegulatoryMappingEvidence,
|
type RegulatoryMappingEvidence,
|
||||||
type RegulatoryMappingResponse,
|
type RegulatoryMappingResponse,
|
||||||
|
|
@ -130,6 +132,7 @@ export class RegulatoryMappingService {
|
||||||
const ruleKey = supported ? this.ruleKey(finding) : undefined;
|
const ruleKey = supported ? this.ruleKey(finding) : undefined;
|
||||||
const run = ruleKey && this.options.client ? await this.store.getRegulatoryMappingRun(this.mappingId(ruleKey)) : null;
|
const run = ruleKey && this.options.client ? await this.store.getRegulatoryMappingRun(this.mappingId(ruleKey)) : null;
|
||||||
const decisions = ruleKey ? await this.store.listRegulatoryMappingDecisions(ruleKey) : [];
|
const decisions = ruleKey ? await this.store.listRegulatoryMappingDecisions(ruleKey) : [];
|
||||||
|
const earlierControls = ruleKey ? await this.earlierControls(ruleKey, run, decisions) : [];
|
||||||
return {
|
return {
|
||||||
version: 1,
|
version: 1,
|
||||||
projectId,
|
projectId,
|
||||||
|
|
@ -142,10 +145,40 @@ export class RegulatoryMappingService {
|
||||||
: !availability.enabled ? { disabledReason: availability.reason ?? this.options.disabledReason ?? 'Regulatory mapping is not configured.' } : {}),
|
: !availability.enabled ? { disabledReason: availability.reason ?? this.options.disabledReason ?? 'Regulatory mapping is not configured.' } : {}),
|
||||||
...(run && ['queued', 'running'].includes(run.status) ? { pollAfterMs: POLL_AFTER_MS } : {}),
|
...(run && ['queued', 'running'].includes(run.status) ? { pollAfterMs: POLL_AFTER_MS } : {}),
|
||||||
...(run ? { run: withoutEvidenceText(run) } : {}),
|
...(run ? { run: withoutEvidenceText(run) } : {}),
|
||||||
decisions
|
decisions,
|
||||||
|
...(earlierControls.length ? { earlierControls } : {})
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The run that grounds a decision: the current one, or for an earlier decision the newest older succeeded
|
||||||
|
* run of the rule that retrieved the control, so it stays reviewable after a pipeline version change.
|
||||||
|
*/
|
||||||
|
private async groundingMappingId(ruleKey: string, controlId: string): Promise<string> {
|
||||||
|
const currentId = this.mappingId(ruleKey);
|
||||||
|
const current = await this.store.getRegulatoryMappingRun(currentId);
|
||||||
|
if (!current || current.candidates.some(candidate => candidate.controlId === controlId)) return currentId;
|
||||||
|
const older = (await this.store.listRegulatoryMappingRuns()).find(run => run.ruleKey === ruleKey && run.status === 'succeeded'
|
||||||
|
&& run.candidates.some(candidate => candidate.controlId === controlId));
|
||||||
|
return older?.id ?? currentId;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Decided controls the current run did not retrieve, titled from the newest older run of the rule that did. */
|
||||||
|
private async earlierControls(ruleKey: string, run: RegulatoryMappingRun | null, decisions: RegulatoryMappingDecision[]): Promise<RegulatoryEarlierControl[]> {
|
||||||
|
const current = new Set(run?.candidates.map(candidate => candidate.controlId) ?? []);
|
||||||
|
const missing = [...new Set(decisions.map(decision => decision.controlId))].filter(controlId => !current.has(controlId));
|
||||||
|
if (!missing.length) return [];
|
||||||
|
const known = new Map<string, RegulatoryMappingCandidate>();
|
||||||
|
for (const older of await this.store.listRegulatoryMappingRuns()) {
|
||||||
|
if (older.ruleKey !== ruleKey) continue;
|
||||||
|
for (const candidate of older.candidates) if (!known.has(candidate.controlId)) known.set(candidate.controlId, candidate);
|
||||||
|
}
|
||||||
|
return missing.sort((left, right) => left.localeCompare(right, 'en', { numeric: true })).map(controlId => {
|
||||||
|
const candidate = known.get(controlId);
|
||||||
|
return { controlId, annexAControlId: candidate?.annexAControlId ?? `A.${controlId}`, ...(candidate?.title ? { title: candidate.title } : {}) };
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
async start(projectId: string, request: RegulatoryMappingCreateRequest): Promise<RegulatoryMappingResponse> {
|
async start(projectId: string, request: RegulatoryMappingCreateRequest): Promise<RegulatoryMappingResponse> {
|
||||||
if (!this.generationEnabled) throw new RegulatoryMappingServiceError('regulatory_mapping_disabled', this.options.disabledReason ?? 'Regulatory mapping is not configured.', 503);
|
if (!this.generationEnabled) throw new RegulatoryMappingServiceError('regulatory_mapping_disabled', this.options.disabledReason ?? 'Regulatory mapping is not configured.', 503);
|
||||||
const finding = await this.finding(projectId, request.resultProjectionId);
|
const finding = await this.finding(projectId, request.resultProjectionId);
|
||||||
|
|
@ -210,13 +243,14 @@ export class RegulatoryMappingService {
|
||||||
if (!isRegulatoryMappableResult(finding)) throw new RegulatoryMappingServiceError('unsupported_finding_kind', UNSUPPORTED_KIND_MESSAGE, 422);
|
if (!isRegulatoryMappableResult(finding)) throw new RegulatoryMappingServiceError('unsupported_finding_kind', UNSUPPORTED_KIND_MESSAGE, 422);
|
||||||
const ruleKey = this.ruleKey(finding);
|
const ruleKey = this.ruleKey(finding);
|
||||||
const comment = request.comment?.trim();
|
const comment = request.comment?.trim();
|
||||||
|
const mappingId = await this.groundingMappingId(ruleKey, request.controlId);
|
||||||
const decision: RegulatoryMappingDecision = {
|
const decision: RegulatoryMappingDecision = {
|
||||||
version: 1,
|
version: 1,
|
||||||
id: `regulatory-decision-${randomUUID()}`,
|
id: `regulatory-decision-${randomUUID()}`,
|
||||||
ruleKey,
|
ruleKey,
|
||||||
sourceTool: finding.sourceTool ?? 'unknown',
|
sourceTool: finding.sourceTool ?? 'unknown',
|
||||||
...(finding.sourceRuleId ? { sourceRuleId: finding.sourceRuleId } : {}),
|
...(finding.sourceRuleId ? { sourceRuleId: finding.sourceRuleId } : {}),
|
||||||
mappingId: this.mappingId(ruleKey),
|
mappingId,
|
||||||
controlId: request.controlId,
|
controlId: request.controlId,
|
||||||
decision: request.decision,
|
decision: request.decision,
|
||||||
...(comment ? { comment } : {}),
|
...(comment ? { comment } : {}),
|
||||||
|
|
@ -226,18 +260,29 @@ export class RegulatoryMappingService {
|
||||||
return this.get(projectId, request.resultProjectionId);
|
return this.get(projectId, request.resultProjectionId);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** Accepted controls of every mappable finding in a client timeslot, for the Work `ISO control` and `ISO theme` filters. */
|
||||||
|
async controlAssignments(projectId: string, requested: Pick<RegulatoryMappingBatchScope, 'assessmentIds' | 'scanRunIds'>): Promise<RegulatoryControlOverviewResponse> {
|
||||||
|
return this.controlOverview(projectId, requested, { deviationsOnly: false });
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Accepted controls of the deviations in one client timeslot. Decisions are global per scanner rule;
|
* Accepted controls of the deviations in one client timeslot. Decisions are global per scanner rule;
|
||||||
* a GPO setting counts once, like its rolled-up Work record, however many baseline comparisons it has.
|
* a GPO setting counts once, like its rolled-up Work record, however many baseline comparisons it has.
|
||||||
*/
|
*/
|
||||||
async controlOverview(projectId: string, requested: Pick<RegulatoryMappingBatchScope, 'assessmentIds' | 'scanRunIds'>): Promise<RegulatoryControlOverviewResponse> {
|
async controlOverview(projectId: string, requested: Pick<RegulatoryMappingBatchScope, 'assessmentIds' | 'scanRunIds'>,
|
||||||
const scope = await this.validateBatchScope(projectId, { assessmentIds: requested.assessmentIds, scanRunIds: requested.scanRunIds, filters: { deviation: true } });
|
{ deviationsOnly = true }: { deviationsOnly?: boolean } = {}): Promise<RegulatoryControlOverviewResponse> {
|
||||||
|
const scope = await this.validateBatchScope(projectId, { assessmentIds: requested.assessmentIds, scanRunIds: requested.scanRunIds,
|
||||||
|
filters: deviationsOnly ? { deviation: true } : {} });
|
||||||
const byRule = new Map<string, Map<string, RegulatoryControlFinding>>();
|
const byRule = new Map<string, Map<string, RegulatoryControlFinding>>();
|
||||||
for (const finding of (await this.queryBatchResults(projectId, scope)).filter(isRegulatoryMappableResult)) {
|
const results = (await this.queryBatchResults(projectId, scope)).filter(isRegulatoryMappableResult);
|
||||||
const identity = finding.sourceKind === 'scanner-finding' ? `scanner:${finding.sourceResultId}` : `gpo:${finding.policyScope ?? ''}:${finding.sourceRuleId ?? finding.sourceResultId}`;
|
const settingIds = await this.gpoSettingIds(projectId, scope, results);
|
||||||
|
for (const finding of results) {
|
||||||
|
const settingId = settingIds.get(finding.resultProjectionId);
|
||||||
|
const identity = findingIdentity(finding, settingId);
|
||||||
const findings = byRule.get(this.ruleKey(finding)) ?? new Map<string, RegulatoryControlFinding>();
|
const findings = byRule.get(this.ruleKey(finding)) ?? new Map<string, RegulatoryControlFinding>();
|
||||||
|
// The setting ID only lets Work match its rolled-up GPO setting; it is never displayed or exported.
|
||||||
if (!findings.has(identity)) findings.set(identity, { resultProjectionId: finding.resultProjectionId, sourceKind: finding.sourceKind,
|
if (!findings.has(identity)) findings.set(identity, { resultProjectionId: finding.resultProjectionId, sourceKind: finding.sourceKind,
|
||||||
sourceResultId: finding.sourceResultId, sourceRuleId: finding.sourceRuleId, policyScope: finding.policyScope });
|
sourceResultId: finding.sourceResultId, sourceRuleId: settingId ?? finding.sourceRuleId, policyScope: finding.policyScope });
|
||||||
byRule.set(this.ruleKey(finding), findings);
|
byRule.set(this.ruleKey(finding), findings);
|
||||||
}
|
}
|
||||||
// Control titles come from the newest succeeded run of each rule that retrieved the control.
|
// Control titles come from the newest succeeded run of each rule that retrieved the control.
|
||||||
|
|
@ -303,10 +348,18 @@ export class RegulatoryMappingService {
|
||||||
// Deterministic id per (ruleKey, pipeline versions) so one row exists per rule per
|
// Deterministic id per (ruleKey, pipeline versions) so one row exists per rule per
|
||||||
// version tuple: regeneration reuses it, and a version bump yields a new mapping.
|
// version tuple: regeneration reuses it, and a version bump yields a new mapping.
|
||||||
private mappingId(ruleKey: string): string {
|
private mappingId(ruleKey: string): string {
|
||||||
const tuple = [ruleKey, this.options.corpusVersion, REGULATORY_RETRIEVAL_VERSION, REGULATORY_MAPPING_PROMPT_VERSION, this.options.client!.model].join(' | |||||||