diff --git a/AGENTS.md b/AGENTS.md index bc970f3..883b12b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -52,8 +52,8 @@ ## Core Engine and engines - Explicit user decision 2026-10-01: ASSESSOR with all its existing functions is the **Core Engine**. The context enrichment (Phase 13E, "context engine"/"Kontext-Engine" in earlier text) is the **Context Engine**. The read-only Report Engine (Phase 7, `src/reporting/`, `server/reporting/`) is an engine in the same sense. Future features arrive as further engines that provide a capability without changing the Core. The UI labels `Context`/`Kontext` stay unchanged. -- Explicit user decision 2026-10-05: the regulatory control mapping (Phase 13F, ISO mapping) is the **Compliance Engine** (`src/compliance/`, `server/compliance/`, `tests/compliance/`). It is framework-neutral by intent: ISO/IEC 27001/27002 is its first corpus, later frameworks such as BSI or DORA arrive inside this engine without a Core change. It reads the Core only through `ComplianceReadPort` (single Result DB row by project, filtered Result DB pages, project, assessment and scan-run lists; explicit user decision 2026-10-05: additionally the internal-analysis `policy-evaluation` rows of the same project and timeslot, used solely to read the GPO technical setting ID that the Result DB withholds, so a GPO setting counts once like its rolled-up Work record and the Work ISO facets match it; that ID is never displayed or exported and no other internal field is read) and stores mapping runs, accept/reject decisions, batches and AI overview runs only through its own `ComplianceStore`. Explicit user decision 2026-10-05: all AI integration belongs to this engine. The AI overview (service, model, Client Info panel) moved from the Core into it, and the Result DB `Export JSON` is the engine's 1:1 copy of what its AI features receive: the exact AI overview input built by the same function the overview sends to the model, and every distinct ISO mapping query with the Result DB rows that produce it (the retrieved candidate controls depend on the index and are not exported). The shared model client `server/ai/litellm-client.ts` stays Core platform code. The existing tables of migrations 025 (AI overview runs) and 026 keep their names and data and are owned by the engine; its writes never advance the Core repository revision. Because the Core deletes scan runs and assessments without knowing the engine, the engine removes AI overview runs of a deleted timeslot from its own table on start and on read. The Core project export receives the engine data, including succeeded AI overview runs, beside the Core export at the HTTP route. Its extension points are the Finding Reader `Map ISO controls` action and its window, the Finding Reader `ISO controls` section, the Result DB `Map all ISO controls` batch action and `Export JSON` action, the Data Hub ISO corpus tile, the Client Info `Compliance` tab (formerly the `Compliance` row and before that the `ISO controls` section) and `AI Overview` tab, and the Work `ISO control` and `ISO theme` facets, and for BSI the Finding Reader `BSI requirements` section, the Client Info `BSI Grundschutz++` tile, the Work `BSI requirement` and `BSI practice` facets and the Data Hub BSI catalog tile. Switched off, the engine also removes the AI overview tab and its requests and the `Export JSON` action. Explicit user decision 2026-10-05: `Map ISO controls` opens a window like `Accepted Risk` with Accept and Reject per control (no third decision value). The Finding Reader lists the controls whose latest decision is accepted in an `ISO controls` section read like `Description`, after the summary fields. Client Info shows the accepted controls of the selected timeslot directly below the Context heatmap in the same tile layout without severity colour, each with its ISO 27002 theme and the number of deviation findings (a GPO setting counts once); a tile opens Work with the `Deviation` status and that control selected in the `ISO control` facet, like a Context heatmap cell. Explicit user decision 2026-10-06 supersedes only the tile layout of that Client Info section, after feedback that it grows too large: the accepted controls are grouped by ISO 27002 theme in ISO order (Organizational, People, Physical, Technological; a theme without hits is omitted). Each full-width theme heading states its number of controls and of distinct findings (a finding counts once per theme) and opens Work with the `Deviation` status and that theme selected in the `ISO theme` facet. Below it, its controls flow in columns of twice the compact tile width, one compact row per control sorted by finding count, showing the Annex A ID, the complete title (wrapped, never truncated) and the authoritative count, without an amount bar (removed the same day because it cost title width); a row opens Work exactly like the former tile. A theme shows its top six controls; a keyboard-accessible `Show all N controls` button below expands that theme and `Show top 6` collapses it again, and every theme starts collapsed for a new timeslot. The theme heading still opens all of its controls in Work. Explicit user decision 2026-10-06, later the same day, supersedes the in-page list and its top-six cap: Client Info shows a `Compliance` row below the Context heatmap with one tile per framework, in the exact environment tile geometry and fact grid (its own `client-compliance-grid` with the same tile rules; the environment row keeps exactly its six tiles), so later frameworks such as BSI or DORA add a tile instead of a section. The `ISO 27001/27002` tile shows `Controls` (accepted controls), `Findings` (distinct findings) and the two themes with the most findings, each with its finding count (fewer when fewer themes have hits; an em dash while loading, never zero). Its `Results` button opens the shared Results window with the complete grouped list described above, without a cap; a theme heading or row there closes the window and opens Work as described. Explicit user decision 2026-10-06, again later the same day, supersedes the two-theme selection of the tile: after `Controls` and `Findings` the tile shows all four themes in ISO order (Organizational, People, Physical, Technological), each with its finding count (`0 findings` without hits, an em dash while loading). A theme with hits is one click target over its whole fact cell and opens Work with the `Deviation` status and that theme selected in the `ISO theme` facet; a theme without hits is not clickable, because the facet offers only values with hits. Hover changes only the cell surface; keyboard focus shows the blue focus ring. All six facts stay visible in this tile at every width, unlike the environment tiles, which show two facts below their measured four-fact width. Explicit user decision 2026-10-05: Work offers the Compliance Engine facets `ISO control` (one value per accepted control, `A.8.20 · Networks security`) and `ISO theme` (Organizational, People, Physical, Technological) in mixed Work and in both GPO entry paths, directly before the permanent final `Source: Scan`; this extends the GPO-only facet list. They read the accepted controls of every mappable finding of the timeslot (scanner findings by occurrence, GPO settings by scope and technical identity), offer only values with hits, count against the other active filters, keep an active selection while loading, and disappear with their filter cleared when the engine is off. Decisions stay global per scanner rule. Provisional explicit user decision 2026-10-05 (may change later): after a mapping succeeds, single or batch, the engine accepts its first two proposals in the model's order automatically, only the one if there is a single proposal, with the comment `Accepted automatically: one of the first two proposals of the mapping.`; a control that already has a decision keeps it, so a manual reject is never overridden, and a failure of this step never fails the mapping. Explicit user decision 2026-10-08: the Data Hub ISO corpus tile carries the switch `Map new scans automatically` (default off), stored only in the engine's own settings file `regulatory-corpora/auto-mapping.json`, never in a Core table. While it is on, an engine-side backend watcher (`server/compliance/regulatory-auto-mapping.ts`) checks every 10 seconds through `ComplianceReadPort` only: the active (not superseded) scan runs and, because GPO backups create no scan run, the GPO deviation rows of every timeslot as filtered Result DB pages. For each new scan run it starts the existing ISO batch over that run's deviations, and for each timeslot with new GPO deviation rows a batch over that timeslot's GPO deviations, each with a stable idempotency key. Rules with a succeeded or active mapping of the current pipeline versions are reused, so only new rules reach the model; failed mappings are retried, and the provisional auto-acceptance applies unchanged. What exists when the watcher starts or is switched on is the baseline and is not mapped retroactively; new imports wait while mapping is unavailable. Switching the Compliance Engine off in the Engines popup also switches this server-wide setting off (once, from any view); switching the engine on again leaves it off. Already started batches are never cancelled. Explicit user decision 2026-10-08 supersedes only the placement of the 2026-10-06 Client Info `Compliance` row: it is its own Client Info tab `Compliance`, after Overview (and Microsoft Defender for IoT) and before `AI Overview`, shown while the engine is on and the backend can deliver the timeslot; the Overview no longer contains it. The tab keeps the framework tile row exactly as decided, including each tile's `Results` button and window, so later frameworks add a tile there. Explicit user decision 2026-10-08, later the same day (supersedes the timeslot-only, batch-only history first agreed): because mappings and decisions are global per rule, the tab heading carries a `Last mapping` button showing the newest mapping of any client (`Last mapping: · Automatic|Manual|Single finding`, followed by `Other timeslot`, `Other client` or `Unknown` when it is not this timeslot and by the status while not completed, or plain `No mapping yet`). It opens the window `ISO mapping history` with at most 50 entries, newest first: every mapping batch of every client and every single finding mapping (`Map ISO controls` in the Finding Reader). Columns: Started, Trigger, Scope, Status and the content-free counts Findings (a batch's own eligible findings; an em dash for a single mapping), Sent to model, Reused, Succeeded and Failed. Entries carry no client, rule, scope ID, filter or key. A batch is Automatic when the watcher started it (idempotency key prefixes `auto:`/`auto-gpo:`), otherwise Manual. Explicit user correction 2026-10-08: Scope names where an entry came from relative to the shown timeslot: `This timeslot` (a batch whose scope shares an assessment or scan run with it, or a single mapping started there), `Other timeslot` (same client, another timeslot), `Other client`, or `Unknown` (a single mapping without a recorded origin). `Map ISO controls` sends the timeslot the Finding Reader shows; the server validates it against the client and stores only SHA-256 origin tokens of the client and of each assessment and scan run, never their ids, because mapping runs are global and exported with every client; an invalid timeslot leaves only the client as origin. Every mapping run records its trigger (`single` or `batch`) in the engine's own run data; runs from before 2.21.0 record none and are not listed as single mappings, because they cannot be told apart from batch runs. While a listed entry runs, the tab refreshes every five seconds without falling back to em dashes. Returning to the tab shows the last overview of that client and timeslot at once from a bounded in-memory cache of at most eight timeslots and always requests a fresh one, which replaces it; a failed refresh keeps the shown values. -- Explicit user decision 2026-10-08: BSI Grundschutz++ is the second framework of the Compliance Engine, limited to the 901 Kernel requirements (class `BSI-Stand-der-Technik-Kernel-G0`) of the BSI Stand-der-Technik-Bibliothek; Methodik and risk-management requirements stay out. Like the MITRE catalogs, the catalog and the official BSI mapping ISO/IEC 27001:2022 Annex A -> Grundschutz++ are compact, commit-pinned snapshots in `server/compliance/data/` with source URL, source SHA-256 and the CC BY-SA 4.0 attribution, generated by `server/compliance/update-bsi-assets.mjs`. Explicit user decision 2026-10-08, later the same day: like the MITRE matrix tile, a Data Hub tile `BSI GRUNDSCHUTZ++ KERNEL` (below the ISO corpus tile, shown while the engine is on) names the active catalog date, mapping version and origin (`Bundled` or `Updated`) and offers `Update from BSI`, which fetches only the two fixed official files from the BSI repository `main` branch in the backend, validates them fail-closed, never activates a state older than the active or bundled one, and stores the compact result with source URL and SHA-256 only in the engine folder `regulatory-corpora/bsi/` (see `SECURITY.md`); `Use bundled catalog`, offered only after an update, returns to the pinned snapshots. Derived requirements follow the active catalog at once. Stage 1 derives BSI requirements from the accepted ISO controls of a rule through that mapping, without a model call or migration: each derived requirement names its ISO control and the mapping relationship, is labelled as derived, is never stored as a decision and never changes an ISO decision, and a mapping target missing from the catalog is reported, never reinterpreted. The Finding Reader `BSI requirements` section follows `ISO controls`; the Client Info `Compliance` tab shows a `BSI Grundschutz++` tile beside the ISO tile (`Requirements`, `Findings` and the four practices with most findings; its `Results` window lists every requirement by practice in catalog order), whose rows and practices open Work with the `Deviation` status and the `BSI requirement` or `BSI practice` facet; Work offers these two facets after the ISO facets and before `Source: Scan`, only while a requirement is derived. BSI texts may be shown with attribution. Stage 2, a direct manual BSI mapping by the model with its own corpus, needs a separately approved migration; the automatic mapping of new scans does not cover BSI. +- Explicit user decision 2026-10-05: the regulatory control mapping (Phase 13F, ISO mapping) is the **Compliance Engine** (`src/compliance/`, `server/compliance/`, `tests/compliance/`). It is framework-neutral by intent: ISO/IEC 27001/27002 is its first corpus, later frameworks such as BSI or DORA arrive inside this engine without a Core change. It reads the Core only through `ComplianceReadPort` (single Result DB row by project, filtered Result DB pages, project, assessment and scan-run lists; explicit user decision 2026-10-05: additionally the internal-analysis `policy-evaluation` rows of the same project and timeslot, used solely to read the GPO technical setting ID that the Result DB withholds, so a GPO setting counts once like its rolled-up Work record and the Work ISO facets match it; that ID is never displayed or exported and no other internal field is read) and stores mapping runs, accept/reject decisions, batches and AI overview runs only through its own `ComplianceStore`. Explicit user decision 2026-10-05: all AI integration belongs to this engine. The AI overview (service, model, Client Info panel) moved from the Core into it, and the Result DB `Export JSON` is the engine's 1:1 copy of what its AI features receive: the exact AI overview input built by the same function the overview sends to the model, and every distinct ISO mapping query with the Result DB rows that produce it (the retrieved candidate controls depend on the index and are not exported). The shared model client `server/ai/litellm-client.ts` stays Core platform code. The existing tables of migrations 025 (AI overview runs) and 026 keep their names and data and are owned by the engine; its writes never advance the Core repository revision. Because the Core deletes scan runs and assessments without knowing the engine, the engine removes AI overview runs of a deleted timeslot from its own table on start and on read. The Core project export receives the engine data, including succeeded AI overview runs, beside the Core export at the HTTP route. Its extension points are the Finding Reader `Map ISO controls` action and its window, the Finding Reader `ISO controls` section, the Result DB `Compliance mapping` batch action (formerly `Map all ISO controls`; ISO and BSI) and `Export JSON` action, the Data Hub ISO corpus tile, the Client Info `Compliance` tab (formerly the `Compliance` row and before that the `ISO controls` section) and `AI Overview` tab, and the Work `ISO control` and `ISO theme` facets, and for BSI the Finding Reader `BSI requirements` section, the Client Info `BSI Grundschutz++` tile, the Work `BSI requirement` and `BSI practice` facets and the Data Hub BSI catalog tile. Switched off, the engine also removes the AI overview tab and its requests and the `Export JSON` action. Explicit user decision 2026-10-05: `Map ISO controls` opens a window like `Accepted Risk` with Accept and Reject per control (no third decision value). The Finding Reader lists the controls whose latest decision is accepted in an `ISO controls` section read like `Description`, after the summary fields. Client Info shows the accepted controls of the selected timeslot directly below the Context heatmap in the same tile layout without severity colour, each with its ISO 27002 theme and the number of deviation findings (a GPO setting counts once); a tile opens Work with the `Deviation` status and that control selected in the `ISO control` facet, like a Context heatmap cell. Explicit user decision 2026-10-06 supersedes only the tile layout of that Client Info section, after feedback that it grows too large: the accepted controls are grouped by ISO 27002 theme in ISO order (Organizational, People, Physical, Technological; a theme without hits is omitted). Each full-width theme heading states its number of controls and of distinct findings (a finding counts once per theme) and opens Work with the `Deviation` status and that theme selected in the `ISO theme` facet. Below it, its controls flow in columns of twice the compact tile width, one compact row per control sorted by finding count, showing the Annex A ID, the complete title (wrapped, never truncated) and the authoritative count, without an amount bar (removed the same day because it cost title width); a row opens Work exactly like the former tile. A theme shows its top six controls; a keyboard-accessible `Show all N controls` button below expands that theme and `Show top 6` collapses it again, and every theme starts collapsed for a new timeslot. The theme heading still opens all of its controls in Work. Explicit user decision 2026-10-06, later the same day, supersedes the in-page list and its top-six cap: Client Info shows a `Compliance` row below the Context heatmap with one tile per framework, in the exact environment tile geometry and fact grid (its own `client-compliance-grid` with the same tile rules; the environment row keeps exactly its six tiles), so later frameworks such as BSI or DORA add a tile instead of a section. The `ISO 27001/27002` tile shows `Controls` (accepted controls), `Findings` (distinct findings) and the two themes with the most findings, each with its finding count (fewer when fewer themes have hits; an em dash while loading, never zero). Its `Results` button opens the shared Results window with the complete grouped list described above, without a cap; a theme heading or row there closes the window and opens Work as described. Explicit user decision 2026-10-06, again later the same day, supersedes the two-theme selection of the tile: after `Controls` and `Findings` the tile shows all four themes in ISO order (Organizational, People, Physical, Technological), each with its finding count (`0 findings` without hits, an em dash while loading). A theme with hits is one click target over its whole fact cell and opens Work with the `Deviation` status and that theme selected in the `ISO theme` facet; a theme without hits is not clickable, because the facet offers only values with hits. Hover changes only the cell surface; keyboard focus shows the blue focus ring. All six facts stay visible in this tile at every width, unlike the environment tiles, which show two facts below their measured four-fact width. Explicit user decision 2026-10-05: Work offers the Compliance Engine facets `ISO control` (one value per accepted control, `A.8.20 · Networks security`) and `ISO theme` (Organizational, People, Physical, Technological) in mixed Work and in both GPO entry paths, directly before the permanent final `Source: Scan`; this extends the GPO-only facet list. They read the accepted controls of every mappable finding of the timeslot (scanner findings by occurrence, GPO settings by scope and technical identity), offer only values with hits, count against the other active filters, keep an active selection while loading, and disappear with their filter cleared when the engine is off. Decisions stay global per scanner rule. Provisional explicit user decision 2026-10-05 (may change later), narrowed by explicit user decision 2026-10-09 for ISO and BSI: after a mapping succeeds, single or batch, the engine accepts at most the first two strong proposals in the model's order automatically, a strong proposal being `primary` with `high` or `medium` confidence; weaker proposals wait for a person and without a strong proposal nothing is accepted. The comment is `Accepted automatically: one of the first two strong proposals (primary, high or medium confidence) of the mapping.`; a control that already has a decision keeps it, so a manual reject is never overridden, and a failure of this step never fails the mapping. Explicit user decision 2026-10-08: the Data Hub ISO corpus tile carries the switch `Map new scans automatically` (default off), stored only in the engine's own settings file `regulatory-corpora/auto-mapping.json`, never in a Core table. While it is on, an engine-side backend watcher (`server/compliance/regulatory-auto-mapping.ts`) checks every 10 seconds through `ComplianceReadPort` only: the active (not superseded) scan runs and, because GPO backups create no scan run, the GPO deviation rows of every timeslot as filtered Result DB pages. For each new scan run it starts the existing ISO batch over that run's deviations, and for each timeslot with new GPO deviation rows a batch over that timeslot's GPO deviations, each with a stable idempotency key. Rules with a succeeded or active mapping of the current pipeline versions are reused, so only new rules reach the model; failed mappings are retried, and the provisional auto-acceptance applies unchanged. What exists when the watcher starts or is switched on is the baseline and is not mapped retroactively; new imports wait while mapping is unavailable. Switching the Compliance Engine off in the Engines popup also switches this server-wide setting off (once, from any view); switching the engine on again leaves it off. Already started batches are never cancelled. Explicit user decision 2026-10-08 supersedes only the placement of the 2026-10-06 Client Info `Compliance` row: it is its own Client Info tab `Compliance`, after Overview (and Microsoft Defender for IoT) and before `AI Overview`, shown while the engine is on and the backend can deliver the timeslot; the Overview no longer contains it. The tab keeps the framework tile row exactly as decided, including each tile's `Results` button and window, so later frameworks add a tile there. Explicit user decision 2026-10-08, later the same day (supersedes the timeslot-only, batch-only history first agreed): because mappings and decisions are global per rule, the tab heading carries a `Last mapping` button showing the newest mapping of any client (`Last mapping: · Automatic|Manual|Single finding`, followed by `Other timeslot`, `Other client` or `Unknown` when it is not this timeslot and by the status while not completed, or plain `No mapping yet`). It opens the window `Mapping history` (formerly `ISO mapping history`) with at most 50 entries, newest first: every mapping batch of every client and every single finding mapping (`Map ISO controls` and `Map BSI requirements` in the Finding Reader); explicit user decision 2026-10-09 adds the direct BSI mappings, so the button names a BSI mapping with `· BSI` after the date and an ISO mapping keeps its label. Columns: Started, Framework (`ISO 27001/27002` or `BSI Grundschutz++`), Trigger, Scope, Status and the content-free counts Findings (a batch's own eligible findings; an em dash for a single mapping), Sent to model, Reused, Succeeded and Failed. Entries carry no client, rule, scope ID, filter or key. A batch is Automatic when the watcher started it (idempotency key prefixes `auto:`/`auto-gpo:`), otherwise Manual. Explicit user correction 2026-10-08: Scope names where an entry came from relative to the shown timeslot: `This timeslot` (a batch whose scope shares an assessment or scan run with it, or a single mapping started there), `Other timeslot` (same client, another timeslot), `Other client`, or `Unknown` (a single mapping without a recorded origin). `Map ISO controls` sends the timeslot the Finding Reader shows; the server validates it against the client and stores only SHA-256 origin tokens of the client and of each assessment and scan run, never their ids, because mapping runs are global and exported with every client; an invalid timeslot leaves only the client as origin. Every mapping run records its trigger (`single` or `batch`) in the engine's own run data; runs from before 2.21.0 record none and are not listed as single mappings, because they cannot be told apart from batch runs. While a listed entry runs, the tab refreshes every five seconds without falling back to em dashes. Returning to the tab shows the last overview of that client and timeslot at once from a bounded in-memory cache of at most eight timeslots and always requests a fresh one, which replaces it; a failed refresh keeps the shown values. +- Explicit user decision 2026-10-08: BSI Grundschutz++ is the second framework of the Compliance Engine, limited to the 901 Kernel requirements (class `BSI-Stand-der-Technik-Kernel-G0`) of the BSI Stand-der-Technik-Bibliothek; Methodik and risk-management requirements stay out. Like the MITRE catalogs, the catalog and the official BSI mapping ISO/IEC 27001:2022 Annex A -> Grundschutz++ are compact, commit-pinned snapshots in `server/compliance/data/` with source URL, source SHA-256 and the CC BY-SA 4.0 attribution, generated by `server/compliance/update-bsi-assets.mjs`. Explicit user decision 2026-10-08, later the same day: like the MITRE matrix tile, a Data Hub tile `BSI GRUNDSCHUTZ++ KERNEL` (below the ISO corpus tile, shown while the engine is on) names the active catalog date, mapping version and origin (`Bundled` or `Updated`) and offers `Update from BSI`, which fetches only the two fixed official files from the BSI repository `main` branch in the backend, validates them fail-closed, never activates a state older than the active or bundled one, and stores the compact result with source URL and SHA-256 only in the engine folder `regulatory-corpora/bsi/` (see `SECURITY.md`); `Use bundled catalog`, offered only after an update, returns to the pinned snapshots. Derived requirements follow the active catalog at once. Stage 1 derives BSI requirements from the accepted ISO controls of a rule through that mapping, without a model call or migration: each derived requirement names its ISO control and the mapping relationship, is labelled as derived, is never stored as a decision and never changes an ISO decision, and a mapping target missing from the catalog is reported, never reinterpreted. The Finding Reader `BSI requirements` section follows `ISO controls`; the Client Info `Compliance` tab shows a `BSI Grundschutz++` tile beside the ISO tile (`Requirements`, `Findings` and the four practices with most findings; its `Results` window lists every requirement by practice in catalog order), whose rows and practices open Work with the `Deviation` status and the `BSI requirement` or `BSI practice` facet; Work offers these two facets after the ISO facets and before `Source: Scan`, only while a requirement is derived. BSI texts may be shown with attribution. Explicit user decision 2026-10-09 (stage 2, migration 029 after a verified backup): the model also maps a rule directly to Kernel requirements from its own BSI corpus (`corpusId` `bsi-grundschutz-kernel` in the shared Qdrant/BGE-M3 path, built from the active catalog from the Data Hub BSI tile via `Build search index`/`Rebuild search index` or on the first BSI mapping), started from `Map BSI requirements` in the Finding Reader `BSI requirements` section or from the BSI part of the Result DB `Compliance mapping` window. Mappings and decisions carry a framework (`iso27001`, `bsi-kernel`; ISO objects keep their earlier shape) and the same auto-acceptance rule applies. The Client Info `Compliance` tab tile `ISO ↔ BSI review` compares per rule the directly accepted BSI requirements with those derived from the accepted ISO controls: `Consistent` (one direct requirement in the derived set), `Inconsistent` (reachable but none derived) or `No ISO reference` (none reachable through the BSI mapping); its Results window lists the rules by a Result DB row, offers Accept/Reject through the normal decision path and `Mark reviewed` for one fingerprint of the compared sets (engine table `assessor_regulatory_consistency_reviews`). The comparison is derived only and never changes, rejects or blocks a decision. Explicit user decision 2026-10-09: ISO and BSI retrieval are hybrid: the model restates the public query once in German, and dense English, dense German and German keyword (BM25) rankings are fused by reciprocal rank; BSI leaves out the BES, DLS, GEB and PERS practices and sends 16 candidates, and each ISO candidate carries its full ISO/IEC 27002 control text (never sent to the browser). The automatic mapping of new scans does not cover BSI. - Engines may read the Core without restriction but never write to, change or override it. They read only through a declared read-only port and the read-only database access, never through Core tables or internal modules directly. What an engine shows or exports still obeys the Core's privacy and Result DB disclosure rules. - Dependencies point one way: an engine depends on the Core, the Core never depends on an engine. The Core works unchanged with every engine switched off; each engine keeps a permanent on/off invariance test like the Context layer's. Engines do not build on each other except through another engine's published results. - An engine that needs storage gets its own schema or tables after separate approval with backup and migration, never columns in Core tables. An engine cannot create findings: new scanner profiles, importers and parsers write Core data and are always Core changes. @@ -76,7 +76,7 @@ ## Context layer (Phase 13E) -- Latest explicit user decision 2026-10-01 supersedes the preceding permanent-Context decision: the Work toolbar has an `Engines` button at the former Context-button position. It stays marked blue, including on hover and while the popup is closed (an explicit colour-role exception for this control only). The shared popup lists Core Engine, Context Engine, Report Engine and, since the explicit user decision of 2026-10-05, Compliance Engine with checkboxes. All default to active; Core Engine is always checked and cannot be disabled. User choices for optional engines are stored locally under the shared Core-owned engine-activation contract. Disabling Context removes its UI, stops new context requests and clears the active Context Cluster filter; disabling Report blocks new report generation from the UI; disabling Compliance removes the Finding Reader ISO action and panel, the Result DB ISO batch action and `Export JSON`, the Client Info `Compliance` and `AI Overview` tabs and the Data Hub ISO corpus tile and stops their requests, and re-enabling starts the Finding Reader ISO panel closed. As the only server-side effect of this local preference (explicit user decision 2026-10-08), disabling Compliance also switches the automatic ISO mapping of new uploads off for the whole server; re-enabling does not switch it on again. Already submitted backend work is not cancelled by this local preference; its late UI output is suppressed. Cluster badges still select the exact Work filter, and the Context detail section still starts collapsed. Engine independence and Core invariance remain binding. +- Latest explicit user decision 2026-10-01 supersedes the preceding permanent-Context decision: the Work toolbar has an `Engines` button at the former Context-button position. It stays marked blue, including on hover and while the popup is closed (an explicit colour-role exception for this control only). The shared popup lists Core Engine, Context Engine, Report Engine and, since the explicit user decision of 2026-10-05, Compliance Engine with checkboxes. All default to active; Core Engine is always checked and cannot be disabled. User choices for optional engines are stored locally under the shared Core-owned engine-activation contract. Disabling Context removes its UI, stops new context requests and clears the active Context Cluster filter; disabling Report blocks new report generation from the UI; disabling Compliance removes the Finding Reader ISO action and panel, the Result DB `Compliance mapping` batch action and `Export JSON`, the Client Info `Compliance` and `AI Overview` tabs and the Data Hub ISO corpus tile and stops their requests, and re-enabling starts the Finding Reader ISO panel closed. As the only server-side effect of this local preference (explicit user decision 2026-10-08), disabling Compliance also switches the automatic ISO mapping of new uploads off for the whole server; re-enabling does not switch it on again. Already submitted backend work is not cancelled by this local preference; its late UI output is suppressed. Cluster badges still select the exact Work filter, and the Context detail section still starts collapsed. Engine independence and Core invariance remain binding. - Explicit user decision 2026-10-01 supersedes the earlier Context on/off, default-off, locally saved switch, hover-button and badge-scroll agreements: Context is permanently active and the global toolbar Context button is removed. Ignore obsolete locally saved on/off preferences. Clicking a Finding Reader Context cluster badge selects that exact `Context Cluster` Work facet value, replacing the previous cluster selection while preserving other filters. The Context detail section still starts collapsed for each finding and opens through its own summary. All read-only, provenance and verdict/score boundaries remain binding; only an explicit cluster-filter selection restricts the Work record set. diff --git a/docs/IMPLEMENTATION_PLAN.md b/docs/IMPLEMENTATION_PLAN.md index ffd7139..395cb6d 100644 --- a/docs/IMPLEMENTATION_PLAN.md +++ b/docs/IMPLEMENTATION_PLAN.md @@ -3203,9 +3203,53 @@ Status: - [x] Stufe 1 Datenschicht: Konverter `src/compliance/bsi-grundschutz.ts`, Generator `server/compliance/update-bsi-assets.mjs`, gepinnte Snapshots in `server/compliance/data/` (Katalog 2026-09-24, Zuordnung 1.0.1-qa, BSI-Commit `a128311`), Ableitung mit Diagnosen, synthetische Tests (2026-10-08) - [x] Stufe 1 Darstellung: Client-Info-Kachel, Finding Reader, Work-Facetten (Produktversion 2.22.0, 2026-10-08; Sichtpruefung beider Themes in der laufenden App steht aus) -- [ ] Stufe 2: BSI-Korpus, manuelle Zuordnung, Migration (separate Freigabe) +- [x] Stufe 2: BSI-Korpus, direkte Zuordnung, Migration 029 (Produktversion 2.23.0, 2026-10-09; Sichtpruefung beider Themes und ein erster echter BSI-Lauf in der laufenden App stehen aus) +- [x] Stufe 2 Konsistenzpruefung ISO <-> BSI mit Review im Tab `Compliance` (Produktversion 2.23.0, 2026-10-09) - [x] `Update from BSI` mit Sicherheitsmodell-Aktualisierung (Data-Hub-Kachel, in Produktversion 2.22.0 enthalten, 2026-10-08) +##### Stufe 2 mit Konsistenzpruefung (Nutzerentscheidung 2026-10-09, umgesetzt in 2.23.0) + +Ziel: Die ISO-Zuordnungen verbessern, indem das Modell Befunde zusaetzlich direkt BSI-Kernel-Anforderungen zuordnet und das Ergebnis je Regel mit der Stufe-1-Ableitung (akzeptierte ISO-Kontrollen -> offizielle BSI-Zuordnung) verglichen wird. Abweichungen erscheinen zur menschlichen Pruefung. Explizite Nutzerentscheidungen vom 2026-10-09: + +- Umfang: vollstaendige Stufe 2 (eigener BSI-Korpus, direkte Zuordnung, Migration), keine reine Verifikation der abgeleiteten Anforderungen. +- Die vorlaeufige automatische Annahme der ersten zwei ISO-Vorschlaege bleibt unveraendert; eine Inkonsistenz haelt sie nicht zurueck. +- Die Pruefung findet im Client-Info-Tab `Compliance` statt (Engine-Code), nicht in einer neuen Work-Facette. +- Vorlaeufig (Nutzerentscheidung 2026-10-09, kann spaeter geaendert werden): auch direkte BSI-Zuordnungen nehmen wie ISO die ersten zwei Vorschlaege automatisch an, mit demselben Kommentar; eine bestehende Entscheidung bleibt erhalten. + +Vergleichsregel (`src/compliance/bsi-consistency.ts`): Je Regel werden die direkt akzeptierten BSI-Anforderungen (juengste Entscheidung je Anforderung, also auch die automatisch angenommenen) mit den Zielen der akzeptierten ISO-Kontrollen in der BSI-Zuordnung verglichen. Ohne akzeptierte BSI-Anforderung gibt es nichts zu vergleichen; eine Regel mit akzeptierter ISO-Kontrolle zaehlt dann als `Not compared`. Drei Zustaende: + +- `Consistent`: mindestens eine direkte Anforderung liegt in der Zielmenge einer akzeptierten ISO-Kontrolle. Ein Treffer genuegt, weil `subset-of` mehrere Anforderungen umfasst und ein Befund selten alle trifft. +- `Inconsistent`: mindestens eine direkte Anforderung ist ueber die BSI-Zuordnung erreichbar, aber keine liegt in der abgeleiteten Menge (auch ohne akzeptierte ISO-Kontrolle). Pruefpflichtig. +- `No reference`: keine direkte Anforderung ist ueber die BSI-Zuordnung von einer ISO-Kontrolle erreichbar (629 der 901 Kernel-Anforderungen). Keine Abweichung, sondern die Luecke, die die direkte Zuordnung schliesst. + +Zusaetzlich als Hinweis die Gegenrichtung: Fuehrt eine direkte Anforderung rueckwaerts ueber die BSI-Zuordnung zu einer nicht akzeptierten ISO-Kontrolle, nennt die Pruefansicht diese Kontrolle als Kandidaten. Der Zustand wird deterministisch aus Entscheidungen, Laeufen und der aktiven Zuordnungsversion berechnet, nicht gespeichert. Uebereinstimmung belegt Konsistenz, nicht Richtigkeit: beide Zuordnungen stammen vom selben Modell und Befundtext. + +Grenzen: Eine Inkonsistenz aendert, verwirft oder blockiert keine ISO-Entscheidung; abgeleitete Anforderungen werden weiterhin nie als Entscheidung gespeichert. Nur der Mensch entscheidet. `Map new scans automatically` umfasst BSI weiterhin nicht. BSI-Texte mit Namensnennung (CC BY-SA 4.0), ISO-Texte weiterhin nicht im Browser. + +Umsetzung 2.23.0 (urspruenglicher Entwurf der Pruefansicht: Liste der Regeln mit Abweichungen des Timeslots, gruppiert nach Zustand (`Inconsistent` zuerst), je Regel akzeptierte ISO-Kontrollen mit ihrer abgeleiteten BSI-Menge, direkte BSI-Vorschlaege und die Kandidaten der Gegenrichtung. Aktionen: Accept/Reject der direkten BSI-Vorschlaege und der ISO-Kontrollen ueber den bestehenden Entscheidungsweg, `Mark reviewed` fuer eine bewusst belassene Inkonsistenz. `Mark reviewed` gilt fuer einen Fingerabdruck der verglichenen Mengen; aendern sich Entscheidungen oder Zuordnungsversion, erscheint die Regel erneut. Befundzahl und Sprung nach Work wie die bestehenden ISO-Drilldowns.) + +- Datenhaltung: Migration 029 (`framework` in `assessor_regulatory_mappings` mit Eindeutigkeit einschliesslich Framework und in `assessor_regulatory_mapping_decisions`, Werte `iso27001`/`bsi-kernel`; neue Tabelle `assessor_regulatory_consistency_reviews`). Vor der Migration Backup `backups/assessor-pre-029-20261009.dump`, per Wiederherstellung in eine Pruefdatenbank verifiziert (Zaehler identisch); danach alle 162 Zuordnungen und 325 Entscheidungen `iso27001`. ISO-Objekte tragen im JSON kein Framework-Feld, ISO-Zuordnungs-IDs, Batch-Scope-Schluessel und URLs bleiben unveraendert. +- BSI-Korpus (`server/compliance/bsi-corpus.ts`): je Kernel-Anforderung ein deutsches Dokument (Aussage und Erlaeuterung mit eingesetzten Parametern) unter `corpusId` `bsi-grundschutz-kernel`, Version aus Katalogdatum und Quell-Hash. Er wird beim ersten BSI-Lauf aus dem aktiven Katalog gebaut (auch nach `Update from BSI`), aeltere Katalogstaende werden danach entfernt. +- BSI-Zuordnung: derselbe Dienst mit Framework-Parameter, Retrieval `bsi-dense-requirement-1.0.0` (40 Treffer, bis zu 12 Anforderungen, jeweils alle Chunks der Anforderung als Evidenz), Prompt `bsi-1.0.0`, Evidenz ohne Seitenangaben. Explizite Nutzerentscheidung 2026-10-09, spaeter am Tag (ersetzt die reine ISO-Historie der Entscheidung vom 2026-10-08): `Last mapping` und das Fenster, jetzt `Mapping history`, listen ISO- und direkte BSI-Laeufe (Batches und Einzelzuordnungen) gemeinsam; die Tabelle hat nach `Started` die Spalte `Framework` (`ISO 27001/27002`, `BSI Grundschutz++`), der Knopf nennt einen BSI-Lauf mit `· BSI` nach dem Datum, ein ISO-Lauf behaelt seine bisherige Beschriftung. `Map new scans automatically` umfasst BSI nicht. +- Startpunkte: Finding Reader, Abschnitt `BSI requirements` mit eigenem Teil `Mapped directly`, Knopf `Map BSI requirements` (Fenster `BSI requirement mapping` wie das ISO-Fenster) und dem Konsistenzzustand in der Zusammenfassung; der Abschnitt bleibt je Befund eingeklappt und erscheint fuer den direkten Teil erst, wenn das Backend den BSI-Zustand geliefert hat. Result-DB-Fenster `ISO mapping batch` mit zweitem Teil `BSI Grundschutz++ requirements` und `Start BSI mapping batch` fuer dieselben Filter. +- Pruefansicht: Kachel `ISO ↔ BSI review` in der Frameworkkachelzeile des Tabs `Compliance` (Compared rules, Inconsistent, No ISO reference, Consistent, Not compared, Reviewed; Gedankenstrich waehrend des Ladens). Ihr Results-Fenster listet die Regeln nach Zustand (Start `Inconsistent`), benannt nur ueber eine Result-DB-Zeile des Timeslots, nie ueber den internen Regelschluessel. Je Regel: akzeptierte ISO-Kontrollen (Reject), direkte BSI-Anforderungen mit Herkunft (Reject), ISO-Kandidaten der Gegenrichtung (Accept nur, wenn der aktuelle ISO-Lauf sie abgerufen hat), `Mark reviewed` mit optionalem Kommentar fuer den aktuellen Fingerabdruck. +- Core-Aenderungen mit Core-Freigabe Store, Presentation, Platform, Core Tests vom 2026-10-09: Migration und Tabellenklassifikation, HTTP-Routen (`framework`-Parameter, `regulatory-mapping-consistency`, `regulatory-mapping-consistency-reviews`), API-Client, Finding-Reader-Einbindung des BSI-Abschnitts, Batch-Ueberschriftenstil, Versionsvertragstests. AGENTS.md beschreibt Stufe 2 noch als ausstehend; die Aktualisierung braucht eine Governance-Freigabe. +- Verifikation: synthetische Tests fuer Vergleichsregel, BSI-Lauf und Entscheidungstrennung, Pruefliste mit `Mark reviewed`, Batch, Korpusaufbau, UI-Interaktionen (Kachel, Fenster, Batchdialog, Finding Reader), erweiterter Invarianztest; `npm test` (alle bestanden) und beide Builds. Ausstehend: Sichtpruefung beider Themes und ein erster echter BSI-Lauf in der laufenden App. + +Urspruengliche Umsetzungsschritte (Entwurf): + +1. Migration (Core `Store`, separate Freigabe, vorher verifiziertes Backup): `framework` (`iso27001`, `bsi-grundschutz-kernel`) in `assessor_regulatory_mappings` (Eindeutigkeit einschliesslich Framework) und `assessor_regulatory_mapping_decisions`; bestehende Zeilen erhalten `iso27001`. Neue Engine-Tabelle fuer `Mark reviewed` (Regel, Fingerabdruck, Kommentar, Zeit). Tabellenklassifikation im Repository. +2. BSI-Korpus (Engine): Anforderung und Erlaeuterung der 901 Kernel-Anforderungen aus dem aktiven Snapshot unter eigenem `corpusId` im Qdrant-/BGE-M3-Pfad, Neuaufbau bei `Update from BSI`. +3. BSI-Zuordnung (Engine): eigene Prompt- und Schemaversion fuer deutsche `MUSS`/`SOLLTE`-Anforderungen, Einzel- und Batchlauf ueber den bestehenden Dienst mit Framework-Parameter; Wiederverwendung erfolgreicher Laeufe je Regel wie bei ISO. +4. Startpunkte (Core `Presentation`, `Platform`): Framework-Wahl oder zweite Aktion bei `Map ISO controls` im Finding Reader und beim Result-DB-Batch; Framework-Parameter in den HTTP-Routen. +5. Konsistenzberechnung und Pruefansicht (Engine): reine Funktion mit synthetischen Tests fuer alle drei Zustaende, Gegenrichtung und Fingerabdruck; Pruefliste im Tab `Compliance`. +6. Finding Reader `BSI requirements` (Engine-Inhalt): direkte und abgeleitete Anforderungen getrennt kenntlich. +7. Invarianztest der Compliance Engine erweitern; `npm test`, beide Builds, Produktversion (Minor). + +Vor der Umsetzung offen, am 2026-10-09 entschieden: Migration mit Backup und Core-Freigabe erteilt; direkte BSI-Vorschlaege werden vorlaeufig wie ISO automatisch angenommen (Nutzerentscheidung entgegen dem Vorschlag, nur manuell anzunehmen). + +Risiko: Weil beide Seiten automatisch angenommen werden und vom selben Modell stammen, zeigt `Consistent` zunaechst nur die Uebereinstimmung zweier Automatiken; die Entscheidung ist ausdruecklich vorlaeufig. + #### Client-Info-Tab `Compliance`: Uebersichtszeile (Nutzerentscheidung 2026-10-08, umgesetzt) Nach Freigabe eines lokalen, nicht versionierten Mockups mit synthetischen Daten steht ueber der unveraenderten Frameworkkachelzeile eine Uebersichtszeile in denselben sechs Spuren. Reiner Engine-Code (`src/compliance/ui/ComplianceOverview.tsx`, `compliance.css`, `server/compliance/regulatory-mapping-service.ts`); kein Erweiterungspunkt und keine Core-Datei geaendert. @@ -3357,6 +3401,7 @@ Festlegung 2026-09-06 (Phase 13B/13C, ausdruecklicher Nutzerwunsch): Vor Rueckfr Festlegung vom 2026-09-06 (Phase 11, lokaler Persistenzbetrieb): Der vorhandene Docker-Compose-Bestand `assessor-2_0` mit der Datenbank `assessor`, dem Container `assessor-2_0-postgres-1` und dem persistenten Volume `assessor-2_0_assessor-postgres-data` ist der einzige kanonische lokale ASSESSOR-Datenbestand fuer diesen Workspace. Lokale Starts verwenden ausschliesslich `127.0.0.1:5432`. Die Compose-Konfiguration bindet dieses Volume deshalb namentlich als `external`; bei fehlendem Bestand oder Portkonflikt muss der Start fehlschlagen, statt eine weitere ASSESSOR-Datenbank oder einen Ersatzport anzulegen. Diese ausdrueckliche Nutzerfestlegung ersetzt die bisherige implizite, verzeichnisabhaengige Compose-Projektidentitaet. Der interne Browser-/Backend-/PostgreSQL-Zielaufbau bleibt unveraendert. | Datum | Entscheidung | Begründung | +| 2026-10-09 | BSI Stufe 2 wird vollstaendig umgesetzt und um eine Konsistenzpruefung ISO <-> BSI erweitert: direkte BSI-Zuordnung je Regel gegen die Ableitung aus akzeptierten ISO-Kontrollen, Zustaende `Consistent`, `Inconsistent`, `No reference`, Pruefung im Tab `Compliance`; die automatische Annahme der ISO-Vorschlaege bleibt unveraendert (Phase 13F, Abschnitt Stufe 2 mit Konsistenzpruefung, Entwurf). | Nutzerentscheidung; Abweichungen zwischen beiden Wegen zeigen pruefwuerdige ISO-Zuordnungen, ohne eine Entscheidung automatisch zu aendern. | | 2026-10-08 | BSI Grundschutz++ wird zweites Framework der Compliance Engine: direkte Zuordnung mit der offiziellen BSI-Zuordnung ISO 27001 -> Grundschutz++ als erster Stufe, nur Kernel-Anforderungen, Katalogbeschaffung wie bei MITRE, zunaechst nur manuelle Zuordnung (Phase 13F, Abschnitt BSI Grundschutz++). | Nutzerentscheidung; die BSI-Zuordnung liefert ohne Modellaufruf eine erste Einordnung, die direkte Zuordnung die anforderungsgenaue Praezision. | | 2026-09-23 | Der globale ISO-Korpus wird bis zur Phase-14-Nutzerverwaltung nur mit einem separaten Operator-Upload-Token aus Data Hub erneuert. Die privaten versionierten JSON-Korpusquellen bleiben im Artefakt-Volume, Original-PDFs nur temporaer; der freigegebene Embedding-Endpunkt darf den lizenzierten Kontrolltext erhalten. | Der Nutzer hat Token, private Wiederaufbauquelle und Embedding-Ziel ausdruecklich bestaetigt. Eine fuer alle Projekte wirksame Korpusaktivierung darf nicht allein von der allgemeinen VPN-/App-Erreichbarkeit abhangen. | | 2026-09-14 | AI Overview verwendet ausschliesslich das per Backend konfigurierte, unternehmensverwaltete LiteLLM ueber HTTPS; ein lokales Modell oder Browserzugriff auf Gateway/Key ist nicht vorgesehen. Der vollstaendige aktive Timeslot wird ueber den bestehenden Privacy-Safe-JSON-Vertrag uebertragen, grosse Eingaben werden ohne Record-Trunkierung hierarchisch zusammengefuehrt, und nur validierte Endergebnisse werden je Timeslot persistiert. | Der Nutzer hat Work-LiteLLM samt URL, Modellalias und API-Key bereit. Die bestehende Exportgrenze verhindert einen zweiten, weiter gefassten AI-Datenvertrag; serverseitige Secrets, explizite Ausloesung, lokale Validierung und append-only Historie halten Netz-, Kosten-, Datenschutz- und Auditgrenzen nachvollziehbar. | @@ -4302,6 +4347,13 @@ Verifiziert 2026-09-08 (Phase 7/13B): GPO-Quellenkorrektur fuer Work und Client | 2026-10-06 | Phase 7: Data Hub Deviation/Passed now use the same assessed-percentage helper as Client Info; all six donuts state their population basis. 29 focused tests and synthetic Chrome Dark/Light checks passed; absolute counts, other population shares and assessment semantics retained. Client build, design and UI/API version checks passed. Nine out-of-scope report test failures, a report Local-only failure and report server type errors prevent full release verification; attempted 2.16.9 reverted to 2.16.8 pending successful production builds. | | 2026-10-06 | Phase 7: Client Info Deviations percentage now divides by Passed + Deviations and labels Of assessed checks; zero assessed shows a dash while preserving absolute zero. Permanent UI/click invariance tests in both themes and synthetic Chrome presentation verified; unchanged counts, verdicts, scores and other donuts. Patch 2.16.8: both builds and version contracts passed; full run 1,936 passed/15 skipped with one stale API-version assertion corrected and passed in the 43-test focused rerun; Local-only/design checks passed. | | 2026-10-06 | Phase 13B: GPO folder read diagnostics retain only safe error names, stages and counts, distinguish empty/denied/unreadable selections, preserve cancellation and wrap complete timeslot messages. 27 focused GPO UI tests and synthetic Chrome Dark/Light checks passed; both builds and version contracts verified for patch 2.16.7. Full run: 1,932 passed, 15 skipped, one general import timeout passed on separate rerun; Local-only/design checks passed. Customer-specific read failure remains unconfirmed; no customer path/data used, Phase 13B remains in progress. | +| 2026-10-09 | Governance (Core-Freigabe Governance): AGENTS.md uebernimmt die Entscheidungen des Tages, BSI Stufe 2 mit Migration 029, Konsistenzpruefung und Kachel `ISO ↔ BSI review`, Hybridsuche, volle ISO-27002-Kontrolltexte, `Compliance mapping` statt `Map all ISO controls`, `Mapping history` mit ISO und BSI, BSI-Suchindex in der Data-Hub-Kachel und die automatische Annahme nur starker Vorschlaege. | +| 2026-10-09 | Phase 13F, in Produktversion 2.23.0 enthalten (unveroeffentlicht; Core-Freigabe Presentation, Platform), Nutzerauftrag: Die Result-DB-Aktion `Map all ISO controls` heisst `Compliance mapping` (Fenster `Compliance mapping`, Aktivitaeten `Starting/Loading compliance mapping batch`), weil sie ISO und BSI startet. Die Data-Hub-Kachel `BSI GRUNDSCHUTZ++ KERNEL` zeigt den Suchindex des aktiven Katalogs (`Built`, `Building` mit Fortschritt, `Not built`, `Outdated`, `Build failed`, `Unavailable`) und bietet `Build search index` beziehungsweise `Rebuild search index`; neue Routen `GET /reference-data/bsi-grundschutz/search-index` und `POST .../search-index/build` (202, `force` fuer den Neuaufbau). Eine BSI-Zuordnung baut einen fehlenden Index weiterhin selbst. Mit Datenbankbereinigung: auf Nutzerwunsch wurden alle Zuordnungsdaten (242 Laeufe, 355 Entscheidungen, 5 Batches) nach verifizierter Sicherung `backups/assessor-pre-clear-mappings-20261009.dump` geloescht; neu zugeordnet wird durch den Nutzer. | +| 2026-10-09 | Phase 13F, in Produktversion 2.23.0 enthalten (unveroeffentlicht, nur Engine-Code), Nutzerentscheidung (ersetzt die vorlaeufige Annahme der ersten zwei Vorschlaege vom 2026-10-05 fuer ISO und BSI): automatisch angenommen werden hoechstens die ersten zwei starken Vorschlaege in Modellreihenfolge, also `primary` mit Konfidenz `high` oder `medium`; schwaechere Vorschlaege warten auf eine Person, ohne starken Vorschlag wird nichts angenommen. Bestehende Entscheidungen bleiben unveraendert. Auf den 20 Purple-Knight-Validierungslaeufen: ISO 21 von 25 automatischen Annahmen in der Referenz (vorher 29 von 39), BSI 16 von 26 (vorher 18 von 37). AGENTS.md beschreibt noch die alte Regel; die Anpassung braucht eine Governance-Freigabe. | +| 2026-10-09 | Phase 13F, in Produktversion 2.23.0 enthalten (unveroeffentlicht, nur Engine-Code), Nutzerauftrag: Hybridsuche fuer ISO und BSI. Je Lauf formuliert das Modell die oeffentliche Abfrage einmal auf Deutsch (`de-expansion-1.0.0`, je Abfrage geteilt zwischen ISO und BSI, im Lauf als `expansion` gespeichert); drei Ranglisten (dichte Suche Englisch, dichte Suche Deutsch, BM25 ueber die deutschen Begriffe im Speicher, `server/compliance/regulatory-lexical-index.ts`) werden per Reciprocal Rank Fusion auf Kontrollebene vereint. BSI laesst BES, DLS, GEB und PERS aus und sendet 16 statt 12 Kandidaten. Ohne deutsche Umformulierung bleibt es bei der englischen dichten Suche. Retrieval-Versionen `hybrid-rrf-de-1.0.0` und `bsi-hybrid-rrf-de-1.0.0`. Abfrage-Korrektur: Purple Knight wird unter `purple-knight-ad/-entra/-combined` erkannt, Indikator-IDs `SI` plus sechs Ziffern zaehlen als oeffentliche Regel-ID. Validierung an den 20 Purple-Knight-Regeln gegen die Referenztabelle: ISO-Referenz in den Kandidaten 42/50 (vorher 33), BSI 21/45 (vorher 4); automatisch angenommene Vorschlaege in der Referenz ISO 29/39 (vorher 28/39), BSI 18/37 (vorher 5/25). Die Laeufe nahmen 19 ISO- und 28 BSI-Entscheidungen automatisch an. | +| 2026-10-09 | Phase 13F, in Produktversion 2.23.0 enthalten (unveroeffentlicht, nur Engine-Code), Nutzerentscheidung: ISO-Prompt 1.3.0. Das Retrieval waehlt weiterhin die 8 Kontrollen; das Modell erhaelt je Kontrolle den 27001-Kontrolltext und den vollstaendigen 27002-Text (Kontrolle, Zweck, Leitlinien in Dokumentreihenfolge, danach weitere Informationen, soweit Platz), hoechstens 8.000 Zeichen je Kontrolle bei stets enthaltenen Treffern; Promptgrenze 96.000 Zeichen, bis zu 24 Evidenzabschnitte je Kandidat. Der Browser erhaelt weiterhin nur Zitate. Validierung an 20 Purple-Knight-Regeln (Client mit Nutzerfreigabe): erfolgreich, Prompt-Tokens Median 7.225 -> 17.994; die ersten zwei Vorschlaege bleiben bei 12 Regeln gleich, bei 8 aendern sie sich ohne messbare Verbesserung gegen die Referenz; Engpass bleibt die Kandidatenauswahl des Retrievals. Der Lauf nahm 9 Kontrollen automatisch neu an. | +| 2026-10-09 | Phase 13F, in Produktversion 2.23.0 enthalten (noch unveroeffentlicht, daher kein eigener Versionssprung; nur Engine-Code), Nutzerentscheidung: `Last mapping` und das Fenster `Mapping history` zeigen auch direkte BSI-Laeufe mit Spalte `Framework`; ein BSI-Lauf heisst auf dem Knopf `· BSI`. Synthetische Dienst- und UI-Tests angepasst und ergaenzt. | +| 2026-10-09 | Phase 13F, Produktversion 2.23.0 (Minor: neue sichtbare Faehigkeit; Core-Freigabe Store, Presentation, Platform, Core Tests): BSI Grundschutz++ Stufe 2. Direkte BSI-Zuordnung mit eigenem Korpus und Prompt, Migration 029 nach verifiziertem Backup, Konsistenzpruefung ISO <-> BSI mit Kachel und Pruefliste im Tab `Compliance`, BSI-Teil im Finding Reader und im Batchdialog; vorlaeufig automatische Annahme auch fuer BSI. Details im Abschnitt Stufe 2 mit Konsistenzpruefung. | | 2026-10-08 | Phase 13F, in Produktversion 2.22.0 enthalten (nur Engine-Code), Nutzerentscheidung: Der Finding-Reader-Abschnitt `BSI requirements` war in Work zu gross. Er startet jetzt wie `Context` eingeklappt (Zusammenfassung mit Anzahl), gruppiert die Anforderungen unter ihrer ISO-Kontrolle (`A.5.18 · subset-of`, die Beziehung gilt fuer die ganze BSI-Gruppe) als einzeilige Eintraege und zeigt den Anforderungstext erst beim Aufklappen eines Eintrags; die Quellenangabe ist eine kurze Zeile, der vollstaendige Lizenzhinweis bleibt im Client-Info-Results-Fenster. Ein ueber zwei ISO-Kontrollen erreichte Anforderung steht unter beiden. Ein gemeinsamer `Compliance`-Kasten fuer ISO und BSI wurde erwogen und vom Nutzer verworfen; `ISO controls` bleibt unveraendert offen. Test: eingeklappt, Gruppen, Aufklappen eines Eintrags, neues Finding wieder eingeklappt, Engine aus. | | 2026-10-08 | Phase 13F, in Produktversion 2.22.0 enthalten (noch unveroeffentlicht, daher kein eigener Versionssprung; Core-Freigabe Presentation, Platform, Governance, Core Tests), Nutzerauftrag: Data-Hub-Kachel `BSI GRUNDSCHUTZ++ KERNEL` wie die MITRE-Matrix unter der ISO-Korpus-Kachel, nur bei eingeschalteter Engine. Plakette mit Katalogdatum, Zuordnungsversion und Herkunft (`Bundled`/`Updated`); `Update from BSI` laedt im Backend genau die zwei festen offiziellen Dateien vom BSI-Branch `main` (HTTPS, keine Weiterleitung, 120 s, 32 MiB je Datei), prueft sie fail-closed mit dem Konverter, aktiviert nie einen aelteren Stand und speichert den kompakten Stand mit Quell-URL und SHA-256 atomar und privat in `regulatory-corpora/bsi/active-catalog.json`; `Use bundled catalog` erscheint nur danach und kehrt zu den gepinnten Snapshots zurueck. Engine-Dienst `BsiCatalogService`, die Ableitung liest den aktiven Stand; Routen `GET /reference-data/bsi-grundschutz`, `POST .../refresh` und `POST .../reset` (Fehler 502 mit Code, ohne Dienst 503); Aktivitaetslabels; nach Update oder Ruecksetzen laden Client Info und Work neu. Sicherheitsmodell in `SECURITY.md` fuer das neue ausgehende Ziel ergaenzt (Nutzerfreigabe), `AGENTS.md` fortgeschrieben. Tests: Dienst (Standard, Update, unveraendert, kein Rueckschritt, Fehlerfaelle ohne Aenderung, Ruecksetzen, beschaedigter und veralteter Speicher), Routen, Kachelinteraktion, Engine aus, Platzierung, Ableitung aus dem aktiven Stand. Kein Live-Abruf gegen GitHub in Tests. | | 2026-10-08 | Phase 13F, Produktversion 2.22.0 (Minor: neue sichtbare Faehigkeit; Core-Freigabe Presentation, Platform, Core Tests, Governance): BSI Grundschutz++ Stufe 1 sichtbar. Der Engine-Dienst haengt an Einzelzuordnung (`GET regulatory-mappings`), Client-Info-Uebersicht und Work-Zuordnungen ein optionales `bsi`-Feld mit den aus akzeptierten ISO-Kontrollen ueber die gepinnte BSI-Zuordnung abgeleiteten Kernel-Anforderungen (Anforderungstext mit aufgeloesten Parametern, ISO-Route und Beziehung, Katalog-/Zuordnungsversion, CC-BY-SA-Hinweis, fehlende Ziele); ohne Ableitung entfaellt das Feld, ISO-Antworten bleiben unveraendert; keine neue Route. Finding Reader: Abschnitt `BSI requirements` direkt nach `ISO controls`. Client Info `Compliance`: Kachel `BSI Grundschutz++` neben ISO mit `Requirements`, `Findings` und den vier Praktiken mit den meisten Findings (ein Finding zaehlt je Praktik einmal); `Results` listet alle Anforderungen nach Praktik in Katalogreihenfolge mit ISO-Route, Quelle und fehlenden Zielen; Zeile und Praktik oeffnen Work mit `Deviation` und der Facette `BSI requirement` bzw. `BSI practice`. Work: beide Facetten nach den ISO-Facetten und vor `Source: Scan`, nur wenn eine Anforderung abgeleitet ist; Werte in Kleinschreibung wie `ISO theme`. `AGENTS.md` nennt Entscheidung und Erweiterungspunkte. Tests: Dienst (Ableitung, Ablehnung entfernt sie), Kachel und Results-Fenster, Finding-Reader-Abschnitt mit Engine aus, Work-Facetten und Filterung, Client-Info-Drilldown; `npm test` 2165 bestanden, 18 uebersprungen; beide Produktionsbuilds bestanden. Sichtpruefung in der laufenden App (beide Themes) steht aus. | diff --git a/server/assessor-http-server.ts b/server/assessor-http-server.ts index 54247da..5bc2257 100644 --- a/server/assessor-http-server.ts +++ b/server/assessor-http-server.ts @@ -36,13 +36,16 @@ import type { RegulatoryMappingBatchCreateRequest, RegulatoryMappingBatchScope, RegulatoryMappingCreateRequest, - RegulatoryMappingDecisionCreateRequest + RegulatoryMappingDecisionCreateRequest, + RegulatoryFramework } from '../src/compliance/regulatory-mapping'; import type { RegulatoryMappingService } from './compliance/regulatory-mapping-service'; import type { ComplianceStore } from '../src/compliance/compliance-store'; import { RegulatoryMappingServiceError } from './compliance/regulatory-mapping-service'; import { RegulatoryCorpusUploadError, type RegulatoryCorpusUploadService } from './compliance/regulatory-corpus-upload-service'; import { BsiCatalogError, type BsiCatalogService } from './compliance/bsi-catalog-service'; +import type { BsiCorpus } from './compliance/bsi-corpus'; +import type { RegulatoryConsistencyReviewCreateRequest } from '../src/compliance/bsi-consistency'; import type { GpoAutomaticAnalysisService } from './gpo/gpo-automatic-analysis-service'; import type { MicrosoftSecurityBaselineSelectionContext } from '../src/domain/microsoft-security-baseline'; import { listProjectGpoBackupStatus } from './gpo/gpo-backup-status-service'; @@ -189,6 +192,8 @@ export interface AssessorHttpServerOptions { complianceExport?: Pick; /** Compliance Engine: active BSI Kernel catalog and ISO mapping with Update from BSI. */ bsiCatalogService?: Pick; + /** Compliance Engine: the BSI Kernel search index the Data Hub BSI tile builds. */ + bsiSearchIndex?: Pick; isoUploadToken?: string; } @@ -334,6 +339,11 @@ function unifiedQueryFrom(url: URL): UnifiedResultQuery { }; } +/** The Compliance Engine framework of a mapping request; the service validates the value and defaults to ISO. */ +function regulatoryFrameworkFrom(url: URL): RegulatoryFramework | undefined { + return (url.searchParams.get('framework') ?? undefined) as RegulatoryFramework | undefined; +} + function regulatoryMappingBatchScopeFrom(url: URL): RegulatoryMappingBatchScope { const query = unifiedQueryFrom(url); const { audience: _audience, projectId: _projectId, assessmentId: _assessmentId, @@ -797,6 +807,15 @@ export function createAssessorHttpServer(options: AssessorHttpServerOptions): Se await readJson>(request, maximumBodyBytes); status = 200; sendJson(response, status, await options.bsiCatalogService.refresh(), requestId); return; } + if (segments.join('/') === 'reference-data/bsi-grundschutz/search-index' && method === 'GET') { + if (!options.bsiSearchIndex) throw new HttpError(503, 'bsi_search_index_unavailable', 'The BSI search index is not configured.'); + status = 200; sendJson(response, status, await options.bsiSearchIndex.status(), requestId); return; + } + if (segments.join('/') === 'reference-data/bsi-grundschutz/search-index/build' && method === 'POST') { + if (!options.bsiSearchIndex) throw new HttpError(503, 'bsi_search_index_unavailable', 'The BSI search index is not configured.'); + const body = await readJson<{ force?: boolean }>(request, maximumBodyBytes); + status = 202; sendJson(response, status, await options.bsiSearchIndex.startBuild(body?.force === true), requestId); return; + } if (segments.join('/') === 'reference-data/bsi-grundschutz/reset' && method === 'POST') { if (!options.bsiCatalogService) throw new HttpError(503, 'bsi_catalog_unavailable', 'The BSI catalog is not configured.'); await readJson>(request, maximumBodyBytes); @@ -1253,7 +1272,7 @@ export function createAssessorHttpServer(options: AssessorHttpServerOptions): Se } if (segments[0] === 'projects' && segments[2] === 'regulatory-mappings' && segments.length === 4 && method === 'GET') { if (!options.regulatoryMappingService) throw new HttpError(503, 'regulatory_mapping_disabled', 'Regulatory mapping is not configured.'); - status = 200; sendJson(response, status, await options.regulatoryMappingService.get(segments[1], segments[3]), requestId); return; + status = 200; sendJson(response, status, await options.regulatoryMappingService.get(segments[1], segments[3], regulatoryFrameworkFrom(url)), requestId); return; } if (segments[0] === 'projects' && segments[2] === 'regulatory-mapping-runs' && segments.length === 3 && method === 'POST') { if (!options.regulatoryMappingService) throw new HttpError(503, 'regulatory_mapping_disabled', 'Regulatory mapping is not configured.'); @@ -1275,7 +1294,16 @@ export function createAssessorHttpServer(options: AssessorHttpServerOptions): Se } if (segments[0] === 'projects' && segments[2] === 'regulatory-mapping-batches' && segments.length === 3 && method === 'GET') { if (!options.regulatoryMappingService) throw new HttpError(503, 'regulatory_mapping_disabled', 'Regulatory mapping is not configured.'); - status = 200; sendJson(response, status, await options.regulatoryMappingService.getBatch(segments[1], regulatoryMappingBatchScopeFrom(url)), requestId); return; + status = 200; sendJson(response, status, await options.regulatoryMappingService.getBatch(segments[1], regulatoryMappingBatchScopeFrom(url), regulatoryFrameworkFrom(url)), requestId); return; + } + if (segments[0] === 'projects' && segments[2] === 'regulatory-mapping-consistency' && segments.length === 3 && method === 'GET') { + if (!options.regulatoryMappingService) throw new HttpError(503, 'regulatory_mapping_disabled', 'Regulatory mapping is not configured.'); + status = 200; sendJson(response, status, await options.regulatoryMappingService.consistency(segments[1], regulatoryMappingBatchScopeFrom(url)), requestId); return; + } + if (segments[0] === 'projects' && segments[2] === 'regulatory-mapping-consistency-reviews' && segments.length === 3 && method === 'POST') { + if (!options.regulatoryMappingService) throw new HttpError(503, 'regulatory_mapping_disabled', 'Regulatory mapping is not configured.'); + const body = await readJson(request, maximumBodyBytes); + status = 200; sendJson(response, status, await options.regulatoryMappingService.markConsistencyReviewed(segments[1], body), requestId); return; } if (segments[0] === 'projects' && segments[2] === 'regulatory-mapping-batches' && segments.length === 3 && method === 'POST') { if (!options.regulatoryMappingService) throw new HttpError(503, 'regulatory_mapping_disabled', 'Regulatory mapping is not configured.'); diff --git a/server/compliance/bsi-corpus.ts b/server/compliance/bsi-corpus.ts new file mode 100644 index 0000000..09bbe34 --- /dev/null +++ b/server/compliance/bsi-corpus.ts @@ -0,0 +1,127 @@ +import type { RegulatoryControlDocument } from '../../src/compliance/regulatory-rag'; +import { BSI_CORPUS_ID, BSI_SOURCE_STANDARD } from '../../src/compliance/regulatory-mapping'; +import { bsiRequirementText, type BsiKernelCatalog } from '../../src/compliance/bsi-grundschutz'; +import type { BsiSearchIndexStatus } from '../../src/compliance/bsi-catalog-management'; +import type { BsiCatalogs } from './bsi-snapshots'; +import type { RegulatoryIngestionService } from './regulatory-rag-service'; + +export interface BsiCorpusOptions { + catalogs: () => Promise; + ingestion?: RegulatoryIngestionService; + countIndexed?: (corpusId: string, corpusVersion: string) => Promise; + pruneIndexed?: (corpusId: string, keepCorpusVersion: string) => Promise; + /** Chunks of every catalog state, so the Data Hub can tell an outdated index from a missing one. */ + countCorpus?: (corpusId: string) => Promise; +} + +/** The corpus version of a Kernel catalog: its date and the start of its source hash, so every catalog state indexes once. */ +export function bsiCorpusVersion(catalog: Pick): string { + return `kernel-${catalog.version.slice(0, 10)}-${catalog.sourceSha256.slice(0, 12)}`; +} + +/** + * One retrieval document per Kernel requirement: its statement and guidance with the parameters filled in, in the + * published German. The BSI text may be indexed and shown under CC BY-SA 4.0, unlike the ISO text. + */ +export function bsiCorpusDocuments(catalog: BsiKernelCatalog): RegulatoryControlDocument[] { + const corpusVersion = bsiCorpusVersion(catalog); + return catalog.requirements.map(requirement => ({ + schemaVersion: 1, + corpusId: BSI_CORPUS_ID, + corpusVersion, + documentId: `bsi-kernel:${requirement.id}`, + framework: BSI_SOURCE_STANDARD, + frameworkVersion: catalog.version, + controlId: requirement.id, + language: 'de', + title: requirement.title, + text: [bsiRequirementText(requirement.statement, requirement.params), + requirement.guidance ? bsiRequirementText(requirement.guidance, requirement.params) : ''].filter(Boolean).join('\n\n'), + sourceUri: catalog.sourceUrl + })); +} + +/** + * The BSI Kernel corpus in the shared Qdrant/BGE-M3 path under its own corpus id (Phase 13F, BSI stage 2). It is + * built from the active catalog from the Data Hub BSI tile or, when nobody did, when a BSI mapping first needs it, + * once per catalog state; older catalog states are pruned afterwards. Concurrent callers share one build. + */ +export class BsiCorpus { + private building?: { corpusVersion: string; task: Promise; done: boolean; progress?: { completed: number; total: number } }; + private failure?: { corpusVersion: string; error: string }; + private builtAt?: { corpusVersion: string; at: string }; + + constructor(private readonly options: BsiCorpusOptions) {} + + get configured(): boolean { + return Boolean(this.options.ingestion && this.options.countIndexed); + } + + async corpusVersion(): Promise { + return bsiCorpusVersion((await this.options.catalogs()).catalog); + } + + /** The indexed corpus version of the active catalog, building it first when it is missing. */ + async ensure(): Promise { + if (!this.configured) throw new Error('Embedding or Qdrant is not configured.'); + const { catalog } = await this.options.catalogs(); + const corpusVersion = bsiCorpusVersion(catalog); + if (this.building?.corpusVersion !== corpusVersion) this.track(catalog, corpusVersion, false); + return this.building!.task; + } + + /** + * Starts building the index of the active catalog from the Data Hub (user decision 2026-10-09) without waiting for + * it; `force` embeds every requirement again even when the index is complete. A running build is not started twice. + */ + async startBuild(force: boolean): Promise { + if (!this.configured) return this.status(); + const { catalog } = await this.options.catalogs(); + const corpusVersion = bsiCorpusVersion(catalog); + if (!(this.building?.corpusVersion === corpusVersion && !this.building.done)) this.track(catalog, corpusVersion, force).catch(() => undefined); + return this.status(); + } + + async status(): Promise { + const { catalog } = await this.options.catalogs(); + const corpusVersion = bsiCorpusVersion(catalog); + const base = { version: 1 as const, catalogVersion: catalog.version, corpusVersion, requirementCount: catalog.requirements.length }; + if (!this.configured) return { ...base, state: 'unavailable', indexedChunks: 0, disabledReason: 'Embedding or Qdrant is not configured.' }; + const indexedChunks = await this.options.countIndexed!(BSI_CORPUS_ID, corpusVersion).catch(() => 0); + const building = this.building?.corpusVersion === corpusVersion ? this.building : undefined; + if (building && !building.done) return { ...base, state: 'building', indexedChunks, ...(building.progress ? { progress: { ...building.progress } } : {}) }; + if (this.failure?.corpusVersion === corpusVersion) return { ...base, state: 'failed', indexedChunks, error: this.failure.error }; + if (indexedChunks >= catalog.requirements.length) { + return { ...base, state: 'built', indexedChunks, ...(this.builtAt?.corpusVersion === corpusVersion ? { builtAt: this.builtAt.at } : {}) }; + } + const anyVersion = await (this.options.countCorpus?.(BSI_CORPUS_ID) ?? Promise.resolve(0)).catch(() => 0); + return { ...base, state: anyVersion > indexedChunks ? 'outdated' : 'not-built', indexedChunks }; + } + + private track(catalog: BsiKernelCatalog, corpusVersion: string, force: boolean): Promise { + const entry: NonNullable = { corpusVersion, task: Promise.resolve(corpusVersion), done: false }; + entry.task = this.build(catalog, corpusVersion, force, (completed, total) => { entry.progress = { completed, total }; }); + this.building = entry; + this.failure = undefined; + entry.task.then(() => { + entry.done = true; + this.builtAt = { corpusVersion, at: new Date().toISOString() }; + }, (error: unknown) => { + // A failed build is retried by the next caller and reported in the Data Hub until then. + entry.done = true; + if (this.building === entry) this.building = undefined; + this.failure = { corpusVersion, error: error instanceof Error ? error.message.slice(0, 300) : 'The BSI search index could not be built.' }; + }); + return entry.task; + } + + private async build(catalog: BsiKernelCatalog, corpusVersion: string, force: boolean, + onProgress: (completed: number, total: number) => void): Promise { + const indexed = await this.options.countIndexed!(BSI_CORPUS_ID, corpusVersion).catch(() => 0); + if (force || indexed < catalog.requirements.length) { + await this.options.ingestion!.ingest(bsiCorpusDocuments(catalog), onProgress); + } + await this.options.pruneIndexed?.(BSI_CORPUS_ID, corpusVersion).catch(() => undefined); + return corpusVersion; + } +} diff --git a/server/compliance/compliance-engine-runtime.ts b/server/compliance/compliance-engine-runtime.ts index 60ddf23..43009f1 100644 --- a/server/compliance/compliance-engine-runtime.ts +++ b/server/compliance/compliance-engine-runtime.ts @@ -12,12 +12,15 @@ import { RegulatoryCorpusUploadService } from './regulatory-corpus-upload-servic import { RegulatoryAutoMapping } from './regulatory-auto-mapping'; import { AiOverviewService, type AiOverviewServiceOptions } from './ai-overview-service'; import { BsiCatalogService } from './bsi-catalog-service'; +import { BsiCorpus } from './bsi-corpus'; export interface ComplianceEngineRuntime { mappingService: RegulatoryMappingService; corpusUploadService: RegulatoryCorpusUploadService; aiOverviewService: AiOverviewService; bsiCatalogService: BsiCatalogService; + /** The BSI Kernel search index the Data Hub BSI tile builds and reports. */ + bsiCorpus: BsiCorpus; store: ComplianceStore; /** Stops running work; ASSESSOR keeps its own repository open. */ stop(): Promise; @@ -47,10 +50,18 @@ export async function createComplianceEngineRuntime(options: ComplianceEngineRun const bsiCatalogService = new BsiCatalogService(join(corpusRoot, 'bsi', 'active-catalog.json')); const bsiCatalogs = () => bsiCatalogService.catalogs(); let mappingService: RegulatoryMappingService; + // Without embedding or Qdrant the index reports itself unavailable. + let bsiCorpus = new BsiCorpus({ catalogs: bsiCatalogs }); let rag: ReturnType | undefined; let disabledReason: string | undefined; try { rag = createRegulatoryRagRuntime(); + const indexed = rag; + // Direct BSI mappings (Phase 13F, stage 2) retrieve from a BSI Kernel corpus built from the active catalog on first use. + bsiCorpus = new BsiCorpus({ catalogs: bsiCatalogs, ingestion: indexed.ingestion, + countIndexed: (corpusId, corpusVersion) => indexed.index.count({ corpusId, corpusVersion }), + pruneIndexed: (corpusId, corpusVersion) => indexed.index.deleteOtherCorpusVersions(corpusId, corpusVersion), + countCorpus: corpusId => indexed.index.count({ corpusId }) }); mappingService = new RegulatoryMappingService(read, store, { findingRetrieval: rag.findingRetrieval, index: rag.index, @@ -59,6 +70,9 @@ export async function createComplianceEngineRuntime(options: ComplianceEngineRun corpusId: process.env.ASSESSOR_RAG_ACTIVE_CORPUS_ID?.trim(), corpusVersion: process.env.ASSESSOR_RAG_ACTIVE_CORPUS_VERSION?.trim(), bsiCatalogs, + bsiCorpus, + // Hybrid search: the same model restates each query in German for the German norm texts. + queryExpansionClient: options.aiClient, ...(!options.aiClient ? { disabledReason: 'Work LiteLLM is not configured.' } : {}) }); } catch (error) { @@ -86,6 +100,7 @@ export async function createComplianceEngineRuntime(options: ComplianceEngineRun corpusUploadService, aiOverviewService, bsiCatalogService, + bsiCorpus, store, stop: () => Promise.all([autoMapping.stop(), corpusUploadService.stop().then(() => mappingService.close()), aiOverviewService.close()]).then(() => undefined), close: async () => { await pool?.end(); } diff --git a/server/compliance/postgres-compliance-store.ts b/server/compliance/postgres-compliance-store.ts index 2c2dae7..8a4ac8a 100644 --- a/server/compliance/postgres-compliance-store.ts +++ b/server/compliance/postgres-compliance-store.ts @@ -6,11 +6,14 @@ import { regulatoryMappingBatchRunValidationErrors, regulatoryMappingDecisionValidationErrors, regulatoryMappingRunValidationErrors, + mappingFramework, + type RegulatoryFramework, type RegulatoryMappingBatchRun, type RegulatoryMappingDecision, type RegulatoryMappingRun } from '../../src/compliance/regulatory-mapping'; import { aiOverviewRunValidationErrors, assertAiOverviewTransition, type AiOverviewRun } from '../../src/compliance/ai-overview'; +import { regulatoryConsistencyReviewValidationErrors, type RegulatoryConsistencyReview } from '../../src/compliance/bsi-consistency'; interface DataRow { data: T } @@ -31,7 +34,7 @@ function dataRow(result: QueryResult>): T | null { } /** - * The Compliance Engine's own tables (migration 026, and the AI overview runs of migration 025). Writes run in the engine's own transactions: + * The Compliance Engine's own tables (migrations 026 and 029, and the AI overview runs of migration 025). Writes run in the engine's own transactions: * they never advance the ASSESSOR repository revision and never trigger a Result DB rebuild. */ export class PostgresComplianceStore implements ComplianceStore { @@ -65,13 +68,13 @@ export class PostgresComplianceStore implements ComplianceStore { const inserted = await client.query(` INSERT INTO assessor_regulatory_mappings (id, rule_key, source_tool, source_rule_id, run_status, query_hash, corpus_id, corpus_version, - embedding_model, retrieval_version, prompt_version, output_schema_version, model, created_at, updated_at, data) - VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16::jsonb) + embedding_model, retrieval_version, prompt_version, output_schema_version, model, created_at, updated_at, data, framework) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16::jsonb, $17) ON CONFLICT DO NOTHING RETURNING id `, [run.id, run.ruleKey, run.sourceTool, run.sourceRuleId ?? null, run.status, run.queryHash, run.corpusId, run.corpusVersion, run.embeddingModel, run.retrievalVersion, run.promptVersion, run.outputSchemaVersion, - run.model, run.createdAt, run.updatedAt, json(run)]); + run.model, run.createdAt, run.updatedAt, json(run), mappingFramework(run)]); if (inserted.rowCount) return { run: clone(run), claimed: true }; const existing = dataRow(await client.query>( 'SELECT data FROM assessor_regulatory_mappings WHERE id = $1 FOR UPDATE', [run.id] @@ -96,13 +99,13 @@ export class PostgresComplianceStore implements ComplianceStore { await client.query(` INSERT INTO assessor_regulatory_mappings (id, rule_key, source_tool, source_rule_id, run_status, query_hash, corpus_id, corpus_version, - embedding_model, retrieval_version, prompt_version, output_schema_version, model, created_at, updated_at, data) - VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16::jsonb) + embedding_model, retrieval_version, prompt_version, output_schema_version, model, created_at, updated_at, data, framework) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16::jsonb, $17) ON CONFLICT (id) DO UPDATE SET run_status = excluded.run_status, updated_at = excluded.updated_at, data = excluded.data `, [run.id, run.ruleKey, run.sourceTool, run.sourceRuleId ?? null, run.status, run.queryHash, run.corpusId, run.corpusVersion, run.embeddingModel, run.retrievalVersion, run.promptVersion, run.outputSchemaVersion, - run.model, run.createdAt, run.updatedAt, json(run)]); + run.model, run.createdAt, run.updatedAt, json(run), mappingFramework(run)]); return clone(run); }); } @@ -179,14 +182,14 @@ export class PostgresComplianceStore implements ComplianceStore { const run = dataRow(await client.query>( 'SELECT data FROM assessor_regulatory_mappings WHERE id = $1 FOR UPDATE', [decision.mappingId] )); - if (!run || run.ruleKey !== decision.ruleKey || run.status !== 'succeeded') { + if (!run || run.ruleKey !== decision.ruleKey || run.status !== 'succeeded' || mappingFramework(run) !== mappingFramework(decision)) { throw new RepositoryNotFoundError('Completed regulatory mapping does not exist for this rule.'); } const latest = dataRow(await client.query>(` SELECT data FROM assessor_regulatory_mapping_decisions - WHERE rule_key = $1 AND control_id = $2 + WHERE rule_key = $1 AND control_id = $2 AND framework = $3 ORDER BY created_at DESC, id DESC LIMIT 1 - `, [decision.ruleKey, decision.controlId])); + `, [decision.ruleKey, decision.controlId, mappingFramework(decision)])); // A control with an earlier decision stays decidable after a remap no longer retrieves it; the chain of // previous decisions leads back to the run that grounded it. if (!latest && !run.candidates.some(candidate => candidate.controlId === decision.controlId)) { @@ -198,20 +201,38 @@ export class PostgresComplianceStore implements ComplianceStore { const stored = { ...decision, ...(latest ? { previousDecisionId: latest.id } : {}) }; await client.query(` INSERT INTO assessor_regulatory_mapping_decisions - (id, rule_key, source_tool, source_rule_id, mapping_id, control_id, decision, previous_decision_id, created_at, data) - VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10::jsonb) + (id, rule_key, source_tool, source_rule_id, mapping_id, control_id, decision, previous_decision_id, created_at, data, framework) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10::jsonb, $11) `, [stored.id, stored.ruleKey, stored.sourceTool, stored.sourceRuleId ?? null, stored.mappingId, stored.controlId, - stored.decision, stored.previousDecisionId ?? null, stored.createdAt, json(stored)]); + stored.decision, stored.previousDecisionId ?? null, stored.createdAt, json(stored), mappingFramework(stored)]); return clone(stored); }); } - async listRegulatoryMappingDecisions(ruleKey: string): Promise { + async listRegulatoryMappingDecisions(ruleKey: string, framework: RegulatoryFramework = 'iso27001'): Promise { return dataRows(await this.pool.query>(` SELECT data FROM assessor_regulatory_mapping_decisions - WHERE rule_key = $1 + WHERE rule_key = $1 AND framework = $2 ORDER BY created_at DESC, id DESC - `, [ruleKey])); + `, [ruleKey, framework])); + } + + async createRegulatoryConsistencyReview(review: RegulatoryConsistencyReview): Promise { + const errors = regulatoryConsistencyReviewValidationErrors(review); + if (errors.length) throw new RepositoryValidationError(errors.join(' ')); + return this.transaction(async client => { + await client.query(` + INSERT INTO assessor_regulatory_consistency_reviews (id, rule_key, fingerprint, created_at, data) + VALUES ($1, $2, $3, $4, $5::jsonb) + `, [review.id, review.ruleKey, review.fingerprint, review.createdAt, json(review)]); + return clone(review); + }); + } + + async listRegulatoryConsistencyReviews(): Promise { + return dataRows(await this.pool.query>( + 'SELECT data FROM assessor_regulatory_consistency_reviews ORDER BY created_at DESC, id DESC' + )); } async createAiOverviewRun(run: AiOverviewRun): Promise { @@ -275,6 +296,9 @@ export class PostgresComplianceStore implements ComplianceStore { regulatoryMappingDecisions: dataRows(await this.pool.query>( 'SELECT data FROM assessor_regulatory_mapping_decisions ORDER BY created_at, id' )), + regulatoryConsistencyReviews: dataRows(await this.pool.query>( + 'SELECT data FROM assessor_regulatory_consistency_reviews ORDER BY created_at, id' + )), aiOverviewRuns: (await this.listAiOverviewRuns(projectId)).filter(item => item.status === 'succeeded') }; } diff --git a/server/compliance/qdrant-regulatory-index.ts b/server/compliance/qdrant-regulatory-index.ts index 113248d..199e1ca 100644 --- a/server/compliance/qdrant-regulatory-index.ts +++ b/server/compliance/qdrant-regulatory-index.ts @@ -25,6 +25,8 @@ export interface RegulatoryChunkIndex { replaceDocuments(chunks: IndexedRegulatoryChunk[]): Promise; searchDense(vector: number[], limit: number, filters?: RegulatoryRetrievalFilters): Promise; list(filters: RegulatoryRetrievalFilters, limit: number): Promise; + /** Every chunk matching the filters, page by page; the keyword index of the hybrid search is built from it. */ + listAll?(filters: RegulatoryRetrievalFilters): Promise; } export class QdrantClientError extends Error { @@ -147,6 +149,23 @@ export class QdrantRegulatoryIndex implements RegulatoryChunkIndex { return points.map(point => parseChunkPayload(object(point)?.payload)); } + async listAll(filters: RegulatoryRetrievalFilters): Promise { + const chunks: RegulatoryChunk[] = []; + let offset: unknown; + do { + const response = await this.request('POST', `/collections/${REGULATORY_QDRANT_COLLECTION}/points/scroll`, { + filter: { must: [keywordCondition('embedding_model', this.options.embeddingModel), ...filterConditions(filters)] }, + limit: 256, with_payload: true, with_vector: false, ...(offset === undefined ? {} : { offset }) + }); + const result = object(response.body)?.result; + if (!Array.isArray(result?.points)) throw new QdrantClientError('qdrant_invalid_response', 'Qdrant returned an invalid scroll response.'); + chunks.push(...result.points.map((point: unknown) => parseChunkPayload(object(point)?.payload))); + offset = result.next_page_offset ?? undefined; + if (chunks.length > 100_000) throw new QdrantClientError('qdrant_response_too_large', 'The regulatory corpus exceeds the keyword index limit.'); + } while (offset !== undefined); + return chunks; + } + async count(filters: RegulatoryRetrievalFilters): Promise { const response = await this.request('POST', `/collections/${REGULATORY_QDRANT_COLLECTION}/points/count`, { filter: { must: [keywordCondition('embedding_model', this.options.embeddingModel), ...filterConditions(filters)] }, diff --git a/server/compliance/regulatory-finding-retrieval.ts b/server/compliance/regulatory-finding-retrieval.ts index b5d9c0b..5a5f1ea 100644 --- a/server/compliance/regulatory-finding-retrieval.ts +++ b/server/compliance/regulatory-finding-retrieval.ts @@ -29,4 +29,9 @@ export class RegulatoryFindingRetrievalService { const candidates = await this.retriever.retrieve({ query: preparedQuery.query, limit, filters }); return { sourceResultId: finding.sourceResultId, status: 'candidates', preparedQuery, candidates }; } + + /** Dense search for a further query text of the same finding, such as its German restatement. */ + retrieveText(query: string, limit: number, filters?: RegulatoryRetrievalFilters): Promise { + return this.retriever.retrieve({ query, limit, filters }); + } } diff --git a/server/compliance/regulatory-lexical-index.ts b/server/compliance/regulatory-lexical-index.ts new file mode 100644 index 0000000..ac48415 --- /dev/null +++ b/server/compliance/regulatory-lexical-index.ts @@ -0,0 +1,87 @@ +import type { RegulatoryChunk, RegulatoryRetrievalCandidate } from '../../src/compliance/regulatory-rag'; + +/** + * In-memory BM25 over the chunks of one indexed corpus, the keyword half of the hybrid search (Phase 13F, user + * decision 2026-10-09). The corpora are small (some hundred to some thousand chunks), so the index is built once per + * corpus version from Qdrant and kept in memory; nothing is stored. Tokens are lower-cased words with a light German + * and English suffix stemming, so `Berechtigungen` and `Berechtigung` meet. + */ +const STOP_WORDS = new Set(('der die das und oder ein eine einer eines zu zur zum von mit für auf in im ist sind werden wird sollte ' + + 'muss kann nicht den dem des als bei auch über durch wie the and or of to a an in on for is are be with this that by as it its ' + + 'from not should must can').split(' ')); + +export function lexicalTokens(text: string): string[] { + return (text.toLowerCase().match(/[\p{L}\p{N}]+/gu) ?? []) + .filter(word => word.length > 2 && !STOP_WORDS.has(word)) + .map(word => word.replace(/(ungen|ung|en|er|es|e|n|s)$/u, '')); +} + +export class RegulatoryLexicalIndex { + private readonly documents: Array>; + private readonly lengths: number[]; + private readonly documentFrequency = new Map(); + private readonly averageLength: number; + + constructor(private readonly chunks: readonly RegulatoryChunk[]) { + this.documents = chunks.map(chunk => { + const counts = new Map(); + for (const token of lexicalTokens(`${chunk.title} ${chunk.text}`)) counts.set(token, (counts.get(token) ?? 0) + 1); + return counts; + }); + this.lengths = this.documents.map(counts => [...counts.values()].reduce((total, count) => total + count, 0)); + for (const counts of this.documents) for (const token of counts.keys()) this.documentFrequency.set(token, (this.documentFrequency.get(token) ?? 0) + 1); + this.averageLength = this.lengths.reduce((total, length) => total + length, 0) / Math.max(1, this.lengths.length); + } + + get size(): number { return this.chunks.length; } + + /** Chunks with at least one query term, best first; equal scores keep corpus order, so results are deterministic. */ + search(query: string, limit: number): RegulatoryRetrievalCandidate[] { + const terms = [...new Set(lexicalTokens(query))]; + const total = this.documents.length; + const hits: RegulatoryRetrievalCandidate[] = []; + this.documents.forEach((counts, index) => { + let score = 0; + for (const term of terms) { + const frequency = counts.get(term); + if (!frequency) continue; + const documents = this.documentFrequency.get(term)!; + const idf = Math.log(1 + (total - documents + 0.5) / (documents + 0.5)); + score += idf * (frequency * 2.2) / (frequency + 1.2 * (0.25 + 0.75 * this.lengths[index] / this.averageLength)); + } + if (score > 0) hits.push({ score, chunk: this.chunks[index] }); + }); + return hits.sort((left, right) => right.score - left.score).slice(0, limit); + } +} + +export interface FusedControl { + controlId: string; + /** Reciprocal rank fusion score over the ranked lists; it orders candidates and is no relevance probability. */ + score: number; + /** Every distinct chunk of the control any list returned, in list order. */ + chunks: RegulatoryChunk[]; +} + +/** + * Reciprocal rank fusion (k = 60) of ranked chunk lists at control level: a control ranks in each list by its best + * chunk. Ties fall back to the control id, so identical input yields identical candidates. + */ +export function fuseControlRankings(lists: ReadonlyArray, exclude?: (controlId: string) => boolean): FusedControl[] { + const fused = new Map }>(); + for (const list of lists) { + const order: string[] = []; + for (const hit of list) { + const controlId = hit.chunk.controlId; + if (exclude?.(controlId)) continue; + const entry = fused.get(controlId) ?? { score: 0, chunks: new Map() }; + if (!entry.chunks.has(hit.chunk.chunkId)) entry.chunks.set(hit.chunk.chunkId, hit.chunk); + fused.set(controlId, entry); + if (!order.includes(controlId)) order.push(controlId); + } + order.forEach((controlId, rank) => { fused.get(controlId)!.score += 1 / (60 + rank + 1); }); + } + return [...fused.entries()] + .map(([controlId, entry]) => ({ controlId, score: entry.score, chunks: [...entry.chunks.values()] })) + .sort((left, right) => right.score - left.score || left.controlId.localeCompare(right.controlId, 'en', { numeric: true })); +} diff --git a/server/compliance/regulatory-mapping-service.ts b/server/compliance/regulatory-mapping-service.ts index cd0ce44..d9309ba 100644 --- a/server/compliance/regulatory-mapping-service.ts +++ b/server/compliance/regulatory-mapping-service.ts @@ -25,6 +25,7 @@ import { type RegulatoryMappingResponse, type RegulatoryMappingResult, type RegulatoryMappingRun, + type RegulatoryMappingProposal, type RegulatoryControlFinding, type RegulatoryControlOverviewItem, type RegulatoryControlOverviewResponse, @@ -32,26 +33,58 @@ import { regulatoryControlTheme, AUTO_MAPPING_BATCH_KEY_PREFIXES, MAPPING_HISTORY_LIMIT, + BSI_CORPUS_ID, + BSI_MAPPING_PROMPT_VERSION, + BSI_RETRIEVAL_VERSION, + BSI_SOURCE_STANDARD, + frameworkField, + mappingFramework, + type RegulatoryFramework, type RegulatoryMappingHistoryEntry } from '../../src/compliance/regulatory-mapping'; -import type { RegulatoryRetrievalCandidate } from '../../src/compliance/regulatory-rag'; +import type { + RegulatoryChunk, + RegulatoryFindingQuery, + RegulatoryQueryExpansion, + RegulatoryRetrievalCandidate, + RegulatoryRetrievalFilters +} from '../../src/compliance/regulatory-rag'; +import { fuseControlRankings, RegulatoryLexicalIndex, type FusedControl } from './regulatory-lexical-index'; import type { UnifiedResultProjection } from '../../src/domain/unified-result'; import { privacySafeFinding } from '../../src/domain/privacy-safe-finding'; import type { RegulatoryFindingRetrievalService } from './regulatory-finding-retrieval'; import { bundledBsiCatalogs, type BsiCatalogs } from './bsi-snapshots'; import { bsiFootprint, bsiOverview, type BsiOverview } from '../../src/compliance/bsi-grundschutz'; +import { + bsiConsistency, + bsiConsistencyFingerprintSource, + type RegulatoryConsistencyItem, + type RegulatoryConsistencyResponse, + type RegulatoryConsistencyReviewCreateRequest +} from '../../src/compliance/bsi-consistency'; +import { bsiCorpusVersion, type BsiCorpus } from './bsi-corpus'; import type { RegulatoryChunkIndex } from './qdrant-regulatory-index'; import { buildRegulatoryFindingQuery, isRegulatoryMappableResult } from '../../src/compliance/regulatory-query'; import { canonicalJson } from '../../src/shared/canonical-json'; const POLL_AFTER_MS = 1_500; const UNSUPPORTED_KIND_MESSAGE = 'Only canonical scanner findings and GPO baseline comparisons can be mapped in this version.'; -const MAXIMUM_PROMPT_CHARACTERS = 48_000; +// Prompt 1.3.0 sends each candidate's full control text: about 3,700 characters per control in the median, at most 8,000 +// each, so eight long controls with their JSON escaping still fit. +const MAXIMUM_PROMPT_CHARACTERS = 96_000; +const MAXIMUM_CONTROL_TEXT_CHARACTERS = 8_000; +const SECTION_ORDER: readonly string[] = ['control', 'purpose', 'guidance', 'other-information', 'attributes']; +const CORE_SECTIONS = ['control', 'purpose', 'guidance'] as const; const BATCH_PAGE_SIZE = 500; const DEFAULT_MAXIMUM_CONCURRENT_RUNS = 2; -// User decision 2026-10-05 (provisional): the first proposals of a finished mapping are accepted automatically. +// User decision 2026-10-05 (provisional), narrowed 2026-10-09: of a finished mapping, at most the first two strong +// proposals (primary with high or medium confidence) are accepted automatically; weaker ones wait for a person. export const AUTO_ACCEPTED_PROPOSALS = 2; -export const AUTO_ACCEPT_COMMENT = 'Accepted automatically: one of the first two proposals of the mapping.'; +export const AUTO_ACCEPT_COMMENT = 'Accepted automatically: one of the first two strong proposals (primary, high or medium confidence) of the mapping.'; + +export function isStrongProposal(proposal: Pick): boolean { + return proposal.relationship === 'primary' && (proposal.confidence === 'high' || proposal.confidence === 'medium'); +} export interface RegulatoryMappingServiceOptions { findingRetrieval?: RegulatoryFindingRetrievalService; @@ -65,6 +98,49 @@ export interface RegulatoryMappingServiceOptions { maximumConcurrentRuns?: number; /** The active BSI catalog and mapping (`Update from BSI`); the pinned snapshots by default. */ bsiCatalogs?: () => Promise; + /** The BSI Kernel retrieval corpus for direct BSI mappings (stage 2); without it BSI mapping is unavailable. */ + bsiCorpus?: BsiCorpus; + /** The model that restates a query in German for the hybrid search; without it only the English dense search runs. */ + queryExpansionClient?: AiCompletionClient; +} + +/** The versions that identify one mapping of a rule in one framework. */ +interface MappingPipeline { + framework: RegulatoryFramework; + corpusId: string; + corpusVersion: string; + retrievalVersion: string; + promptVersion: string; +} + +const FRAMEWORKS: readonly RegulatoryFramework[] = ['iso27001', 'bsi-kernel']; +const ISO_CANDIDATE_LIMIT = 8; +// Hybrid search (user decision 2026-10-09): 16 BSI candidates found the reference requirement for 12 of 20 test rules, 12 for 10. +const BSI_CANDIDATE_LIMIT = 16; +const BSI_EXCLUDED_PRACTICES = /^(BES|DLS|GEB|PERS)\./; +const MAXIMUM_CACHED_EXPANSIONS = 500; +const QUERY_EXPANSION_VERSION = 'de-expansion-1.0.0'; +const QUERY_EXPANSION_PROMPT = 'Rewrite one public security finding as a German search query for the German editions of ISO/IEC 27001/27002 and the BSI Grundschutz++ Kernel. Return JSON only: {"query": "", "terms": ["<8 to 15 German terms a standard would use for this subject, plus technical terms that stay English>"]}. Do not add customer data.'; + +function retrievalFilters(pipeline: MappingPipeline): RegulatoryRetrievalFilters { + return pipeline.framework === 'bsi-kernel' ? { corpusId: pipeline.corpusId, corpusVersion: pipeline.corpusVersion } + : { corpusId: pipeline.corpusId, corpusVersion: pipeline.corpusVersion, sourceStandard: 'ISO/IEC 27002', retrievalRole: 'semantic' }; +} + +/** The model's German restatement, bounded plain text only; anything else means no hybrid search for this run. */ +function queryExpansion(value: unknown): Omit | undefined { + if (!value || typeof value !== 'object') return undefined; + const item = value as { query?: unknown; terms?: unknown }; + const plain = (text: unknown, maximum: number) => typeof text === 'string' && text.trim() && text.length <= maximum && !/<\/?[a-z][^>]*>/i.test(text) ? text.trim() : undefined; + const query = plain(item.query, 600); + const terms = Array.isArray(item.terms) ? item.terms.map(term => plain(term, 80)).filter((term): term is string => Boolean(term)).slice(0, 20) : []; + return query ? { language: 'de', version: QUERY_EXPANSION_VERSION, query, terms } : undefined; +} + +function requestedFramework(value: unknown): RegulatoryFramework { + if (value === undefined || value === null || value === '') return 'iso27001'; + if (!FRAMEWORKS.includes(value as RegulatoryFramework)) throw new RepositoryValidationError('framework must be iso27001 or bsi-kernel.'); + return value as RegulatoryFramework; } export class RegulatoryMappingService { @@ -78,6 +154,8 @@ export class RegulatoryMappingService { private activeWorkerCount = 0; private activationPending = false; private titleCache?: { key: string; titles: Record }; + private readonly expansions = new Map | undefined>>(); + private readonly lexicalIndexes = new Map>(); constructor( private readonly read: ComplianceReadPort, @@ -132,20 +210,40 @@ export class RegulatoryMappingService { } } - async get(projectId: string, resultProjectionId: string): Promise { + /** True when mapping can run for the framework; ISO also needs its active licensed corpus. */ + private frameworkEnabled(framework: RegulatoryFramework): boolean { + if (framework === 'iso27001') return this.generationEnabled; + return !this.activationPending && Boolean(this.options.findingRetrieval && this.options.index && this.options.client + && this.options.embeddingModel && this.options.bsiCorpus?.configured); + } + + private async pipeline(framework: RegulatoryFramework): Promise { + if (framework === 'iso27001') return { framework, corpusId: this.options.corpusId ?? '', corpusVersion: this.options.corpusVersion ?? '', + retrievalVersion: REGULATORY_RETRIEVAL_VERSION, promptVersion: REGULATORY_MAPPING_PROMPT_VERSION }; + const { catalog } = await this.bsiCatalogs(); + return { framework, corpusId: BSI_CORPUS_ID, corpusVersion: bsiCorpusVersion(catalog), retrievalVersion: BSI_RETRIEVAL_VERSION, + promptVersion: BSI_MAPPING_PROMPT_VERSION }; + } + + async get(projectId: string, resultProjectionId: string, requested?: RegulatoryFramework): Promise { + const framework = requestedFramework(requested); const finding = await this.finding(projectId, resultProjectionId); const supported = Boolean(buildRegulatoryFindingQuery(finding)); - const availability = supported ? await this.availability() : { enabled: false }; + const availability = supported ? await this.availability(framework) : { enabled: false }; const enabled = availability.enabled && supported; const ruleKey = supported ? this.ruleKey(finding) : undefined; - const run = ruleKey && this.options.client ? await this.store.getRegulatoryMappingRun(this.mappingId(ruleKey)) : null; - const decisions = ruleKey ? await this.store.listRegulatoryMappingDecisions(ruleKey) : []; - const earlierControls = ruleKey ? await this.earlierControls(ruleKey, run, decisions) : []; - const bsi = await this.bsiOverview(acceptedControlIds(decisions)); + const pipeline = await this.pipeline(framework); + const run = ruleKey && this.options.client ? await this.store.getRegulatoryMappingRun(this.mappingId(ruleKey, pipeline)) : null; + const decisions = ruleKey ? await this.store.listRegulatoryMappingDecisions(ruleKey, framework) : []; + const earlierControls = ruleKey ? await this.earlierControls(ruleKey, run, decisions, framework) : []; + const isoDecisions = framework === 'iso27001' ? decisions : ruleKey ? await this.store.listRegulatoryMappingDecisions(ruleKey) : []; + const bsi = await this.bsiOverview(acceptedControlIds(isoDecisions)); + const consistency = framework === 'bsi-kernel' && ruleKey ? await this.ruleConsistency(ruleKey, acceptedControlIds(isoDecisions), acceptedControlIds(decisions)) : undefined; return { version: 1, projectId, resultProjectionId, + framework, generationEnabled: enabled, state: run?.status ?? (enabled ? 'empty' : 'unsupported'), ...(!supported ? { disabledReason: !isRegulatoryMappableResult(finding) @@ -156,7 +254,8 @@ export class RegulatoryMappingService { ...(run ? { run: withoutEvidenceText(run) } : {}), decisions, ...(earlierControls.length ? { earlierControls } : {}), - ...(bsi ? { bsi } : {}) + ...(bsi ? { bsi } : {}), + ...(consistency ? { consistency: { state: consistency.state, reviewed: consistency.reviewed } } : {}) }; } @@ -164,46 +263,56 @@ export class RegulatoryMappingService { * The run that grounds a decision: the current one, or for an earlier decision the newest older succeeded * run of the rule that retrieved the control, so it stays reviewable after a pipeline version change. */ - private async groundingMappingId(ruleKey: string, controlId: string): Promise { - const currentId = this.mappingId(ruleKey); + private async groundingMappingId(ruleKey: string, controlId: string, pipeline: MappingPipeline): Promise { + const currentId = this.mappingId(ruleKey, pipeline); const current = await this.store.getRegulatoryMappingRun(currentId); if (!current || current.candidates.some(candidate => candidate.controlId === controlId)) return currentId; const older = (await this.store.listRegulatoryMappingRuns()).find(run => run.ruleKey === ruleKey && run.status === 'succeeded' - && run.candidates.some(candidate => candidate.controlId === controlId)); + && mappingFramework(run) === pipeline.framework && run.candidates.some(candidate => candidate.controlId === controlId)); return older?.id ?? currentId; } /** Decided controls the current run did not retrieve, titled from the newest older run of the rule that did. */ - private async earlierControls(ruleKey: string, run: RegulatoryMappingRun | null, decisions: RegulatoryMappingDecision[]): Promise { + private async earlierControls(ruleKey: string, run: RegulatoryMappingRun | null, decisions: RegulatoryMappingDecision[], + framework: RegulatoryFramework): Promise { const current = new Set(run?.candidates.map(candidate => candidate.controlId) ?? []); const missing = [...new Set(decisions.map(decision => decision.controlId))].filter(controlId => !current.has(controlId)); if (!missing.length) return []; const known = new Map(); for (const older of await this.store.listRegulatoryMappingRuns()) { - if (older.ruleKey !== ruleKey) continue; + if (older.ruleKey !== ruleKey || mappingFramework(older) !== framework) continue; for (const candidate of older.candidates) if (!known.has(candidate.controlId)) known.set(candidate.controlId, candidate); } return missing.sort((left, right) => left.localeCompare(right, 'en', { numeric: true })).map(controlId => { const candidate = known.get(controlId); - return { controlId, annexAControlId: candidate?.annexAControlId ?? `A.${controlId}`, ...(candidate?.title ? { title: candidate.title } : {}) }; + return { controlId, annexAControlId: candidate?.annexAControlId ?? (framework === 'iso27001' ? `A.${controlId}` : controlId), + ...(candidate?.title ? { title: candidate.title } : {}) }; }); } + private disabledError(framework: RegulatoryFramework): RegulatoryMappingServiceError { + return new RegulatoryMappingServiceError('regulatory_mapping_disabled', framework === 'bsi-kernel' && this.generationEnabled + ? 'BSI mapping is not configured.' : this.options.disabledReason ?? 'Regulatory mapping is not configured.', 503); + } + async start(projectId: string, request: RegulatoryMappingCreateRequest): Promise { - if (!this.generationEnabled) throw new RegulatoryMappingServiceError('regulatory_mapping_disabled', this.options.disabledReason ?? 'Regulatory mapping is not configured.', 503); + const framework = requestedFramework(request.framework); + if (!this.frameworkEnabled(framework)) throw this.disabledError(framework); const finding = await this.finding(projectId, request.resultProjectionId); if (!isRegulatoryMappableResult(finding)) throw new RegulatoryMappingServiceError('unsupported_finding_kind', UNSUPPORTED_KIND_MESSAGE, 422); if (!buildRegulatoryFindingQuery(finding)) throw new RegulatoryMappingServiceError('insufficient_public_context', 'This finding has insufficient public context for regulatory mapping.', 422); - const claimed = await this.claim(finding, true, 'single', await this.origin(projectId, request.origin)); - if (claimed.claimed) this.track(this.enqueue(() => this.process(claimed.run, finding)).then(() => undefined)); - return this.get(projectId, request.resultProjectionId); + const pipeline = await this.pipeline(framework); + const claimed = await this.claim(finding, true, 'single', pipeline, await this.origin(projectId, request.origin)); + if (claimed.claimed) this.track(this.enqueue(() => this.process(claimed.run, finding, pipeline)).then(() => undefined)); + return this.get(projectId, request.resultProjectionId, framework); } - async getBatch(projectId: string, requestedScope: RegulatoryMappingBatchScope): Promise { + async getBatch(projectId: string, requestedScope: RegulatoryMappingBatchScope, requested?: RegulatoryFramework): Promise { + const framework = requestedFramework(requested); const scope = await this.validateBatchScope(projectId, requestedScope); - const scopeKey = this.batchScopeKey(projectId, scope); + const scopeKey = this.batchScopeKey(projectId, scope, framework); const run = (await this.store.listRegulatoryMappingBatchRuns(projectId, scopeKey))[0]; - const availability = await this.availability(); + const availability = await this.availability(framework); return { version: 1, projectId, @@ -217,7 +326,8 @@ export class RegulatoryMappingService { } async startBatch(projectId: string, request: RegulatoryMappingBatchCreateRequest): Promise { - if (!this.generationEnabled) throw new RegulatoryMappingServiceError('regulatory_mapping_disabled', this.options.disabledReason ?? 'Regulatory mapping is not configured.', 503); + const framework = requestedFramework(request.framework); + if (!this.frameworkEnabled(framework)) throw this.disabledError(framework); if (typeof request.idempotencyKey !== 'string' || !/^[A-Za-z0-9._:-]{8,128}$/.test(request.idempotencyKey)) { throw new RepositoryValidationError('idempotencyKey must contain 8 to 128 safe characters.'); } @@ -226,8 +336,9 @@ export class RegulatoryMappingService { const candidate: RegulatoryMappingBatchRun = { version: 1, id: `regulatory-mapping-batch-${randomUUID()}`, + ...frameworkField(framework), projectId, - scopeKey: this.batchScopeKey(projectId, scope), + scopeKey: this.batchScopeKey(projectId, scope, framework), filterHash: sha256(canonicalJson(scope.filters)), assessmentIds: scope.assessmentIds, scanRunIds: scope.scanRunIds, @@ -244,19 +355,21 @@ export class RegulatoryMappingService { assessmentIds: run.assessmentIds, scanRunIds: run.scanRunIds, filters: run.filters - }); + }, framework); } async decide(projectId: string, request: RegulatoryMappingDecisionCreateRequest): Promise { - if (!this.generationEnabled) throw new RegulatoryMappingServiceError('regulatory_mapping_disabled', this.options.disabledReason ?? 'Regulatory mapping is not configured.', 503); + const framework = requestedFramework(request.framework); + if (!this.frameworkEnabled(framework)) throw this.disabledError(framework); const finding = await this.finding(projectId, request.resultProjectionId); if (!isRegulatoryMappableResult(finding)) throw new RegulatoryMappingServiceError('unsupported_finding_kind', UNSUPPORTED_KIND_MESSAGE, 422); const ruleKey = this.ruleKey(finding); const comment = request.comment?.trim(); - const mappingId = await this.groundingMappingId(ruleKey, request.controlId); + const mappingId = await this.groundingMappingId(ruleKey, request.controlId, await this.pipeline(framework)); const decision: RegulatoryMappingDecision = { version: 1, id: `regulatory-decision-${randomUUID()}`, + ...frameworkField(framework), ruleKey, sourceTool: finding.sourceTool ?? 'unknown', ...(finding.sourceRuleId ? { sourceRuleId: finding.sourceRuleId } : {}), @@ -267,7 +380,102 @@ export class RegulatoryMappingService { createdAt: this.now().toISOString() }; await this.store.createRegulatoryMappingDecision(decision, request.expectedLatestDecisionId); - return this.get(projectId, request.resultProjectionId); + return this.get(projectId, request.resultProjectionId, framework); + } + + /** The consistency of one rule with whether exactly this result was kept by `Mark reviewed`. */ + private async ruleConsistency(ruleKey: string, acceptedIso: string[], acceptedBsi: string[], reviews?: Map>) { + const { crosswalk } = await this.bsiCatalogs(); + const consistency = bsiConsistency(acceptedIso, acceptedBsi, crosswalk); + if (!consistency) return undefined; + const fingerprint = sha256(bsiConsistencyFingerprintSource(acceptedIso, acceptedBsi, crosswalk.version)); + const reviewed = reviews ? reviews.get(ruleKey)?.has(fingerprint) ?? false + : (await this.store.listRegulatoryConsistencyReviews()).some(review => review.ruleKey === ruleKey && review.fingerprint === fingerprint); + return { ...consistency, fingerprint, reviewed }; + } + + /** + * The Compliance tab review list (Phase 13F, BSI stage 2, user decision 2026-10-09): every rule of the + * timeslot's mappable deviations whose accepted ISO controls and directly accepted BSI requirements can be + * compared. Derived only; nothing here changes a decision. Rules are named by one Result DB row of the timeslot. + */ + async consistency(projectId: string, requested: Pick): Promise { + const scope = await this.validateBatchScope(projectId, { assessmentIds: requested.assessmentIds, scanRunIds: requested.scanRunIds, filters: { deviation: true } }); + const results = (await this.queryBatchResults(projectId, scope)).filter(isRegulatoryMappableResult) + .sort((left, right) => left.resultProjectionId.localeCompare(right.resultProjectionId)); + const settingIds = await this.gpoSettingIds(projectId, scope, results); + const byRule = new Map }>(); + for (const finding of results) { + const ruleKey = this.ruleKey(finding); + const entry = byRule.get(ruleKey) ?? { representative: finding, identities: new Set() }; + entry.identities.add(findingIdentity(finding, settingIds.get(finding.resultProjectionId))); + byRule.set(ruleKey, entry); + } + const { catalog, crosswalk } = await this.bsiCatalogs(); + const requirements = new Map(catalog.requirements.map(requirement => [requirement.id, requirement])); + const reachable = new Set(crosswalk.entries.flatMap(entry => entry.targets)); + const reviews = new Map>(); + for (const review of await this.store.listRegulatoryConsistencyReviews()) reviews.set(review.ruleKey, (reviews.get(review.ruleKey) ?? new Set()).add(review.fingerprint)); + // ISO control titles and retrieved candidates from the latest succeeded ISO run of each rule. + const isoRuns = new Map(); + for (const run of await this.store.listRegulatoryMappingRuns()) { + if (!byRule.has(run.ruleKey) || mappingFramework(run) !== 'iso27001' || run.status !== 'succeeded') continue; + const latest = isoRuns.get(run.ruleKey); + if (!latest || run.createdAt > latest.createdAt || run.createdAt === latest.createdAt && run.id > latest.id) isoRuns.set(run.ruleKey, run); + } + const isoPipeline = await this.pipeline('iso27001'); + const counts = { consistent: 0, inconsistent: 0, noReference: 0, notCompared: 0, reviewed: 0 }; + const items: RegulatoryConsistencyItem[] = []; + for (const [ruleKey, { representative, identities }] of byRule) { + const acceptedIso = acceptedControlIds(await this.store.listRegulatoryMappingDecisions(ruleKey)); + const acceptedBsi = acceptedControlIds(await this.store.listRegulatoryMappingDecisions(ruleKey, 'bsi-kernel')); + const result = await this.ruleConsistency(ruleKey, acceptedIso, acceptedBsi, reviews); + if (!result) { if (acceptedIso.length) counts.notCompared += 1; continue; } + counts[result.state === 'no-reference' ? 'noReference' : result.state] += 1; + if (result.reviewed) counts.reviewed += 1; + // A reverse candidate is decidable here only where the current ISO run retrieved it, as the decision needs grounding. + const currentIso = this.options.client && isoPipeline.corpusVersion + ? await this.store.getRegulatoryMappingRun(this.mappingId(ruleKey, isoPipeline)) : null; + const titles = new Map((isoRuns.get(ruleKey)?.candidates ?? []).map(candidate => [candidate.controlId, candidate.title])); + const derived = new Set(result.derivedRequirementIds); + items.push({ + resultProjectionId: representative.resultProjectionId, + title: representative.title, + sourceTool: representative.sourceTool ?? 'unknown', + ...(representative.sourceRuleId ? { sourceRuleId: representative.sourceRuleId } : {}), + findingCount: identities.size, + state: result.state, + reviewed: result.reviewed, + iso: [...acceptedIso].sort(byNumber).map(controlId => ({ controlId, ...(titles.get(controlId) ? { title: titles.get(controlId) } : {}) })), + direct: result.directRequirementIds.map(requirementId => ({ requirementId, + ...(requirements.get(requirementId) ? { title: requirements.get(requirementId)!.title, practice: requirements.get(requirementId)!.practice } : {}), + derived: derived.has(requirementId), reachable: reachable.has(requirementId) })), + derivedCount: result.derivedRequirementIds.length, + reverse: result.reverseCandidates.map(candidate => ({ controlId: candidate.isoControlId, + ...(titles.get(candidate.isoControlId) ? { title: titles.get(candidate.isoControlId) } : {}), requirementIds: candidate.requirementIds, + decidable: currentIso?.status === 'succeeded' && currentIso.candidates.some(item => item.controlId === candidate.isoControlId) })) + }); + } + const order = { inconsistent: 0, 'no-reference': 1, consistent: 2 } as const; + items.sort((left, right) => Number(left.reviewed) - Number(right.reviewed) || order[left.state] - order[right.state] + || right.findingCount - left.findingCount || left.title.localeCompare(right.title) || left.resultProjectionId.localeCompare(right.resultProjectionId)); + return { version: 1, projectId, scope: { assessmentIds: scope.assessmentIds, scanRunIds: scope.scanRunIds }, + crosswalkVersion: crosswalk.version, catalogVersion: catalog.version, counts, items }; + } + + /** Records that a person keeps the current ISO <-> BSI result of the finding's rule as it is. */ + async markConsistencyReviewed(projectId: string, request: RegulatoryConsistencyReviewCreateRequest): Promise<{ reviewed: true }> { + const finding = await this.finding(projectId, request?.resultProjectionId); + if (!isRegulatoryMappableResult(finding)) throw new RegulatoryMappingServiceError('unsupported_finding_kind', UNSUPPORTED_KIND_MESSAGE, 422); + const ruleKey = this.ruleKey(finding); + const acceptedIso = acceptedControlIds(await this.store.listRegulatoryMappingDecisions(ruleKey)); + const acceptedBsi = acceptedControlIds(await this.store.listRegulatoryMappingDecisions(ruleKey, 'bsi-kernel')); + const result = await this.ruleConsistency(ruleKey, acceptedIso, acceptedBsi); + if (!result) throw new RepositoryValidationError('This rule has no accepted BSI requirement to compare.'); + const comment = typeof request.comment === 'string' ? request.comment.trim() : ''; + if (!result.reviewed) await this.store.createRegulatoryConsistencyReview({ version: 1, id: `regulatory-consistency-review-${randomUUID()}`, + ruleKey, fingerprint: result.fingerprint, ...(comment ? { comment } : {}), createdAt: this.now().toISOString() }); + return { reviewed: true }; } /** BSI Kernel requirements derived from accepted ISO controls through the active BSI mapping (Phase 13F, stage 1). */ @@ -330,7 +538,7 @@ export class RegulatoryMappingService { const mapped = new Set(); const latestRun = new Map(); for (const run of await this.store.listRegulatoryMappingRuns()) { - if (!byRule.has(run.ruleKey)) continue; + if (!byRule.has(run.ruleKey) || mappingFramework(run) !== 'iso27001') continue; const latest = latestRun.get(run.ruleKey); if (!latest || run.createdAt > latest.createdAt || run.createdAt === latest.createdAt && run.id > latest.id) latestRun.set(run.ruleKey, run); if (run.status !== 'succeeded') continue; @@ -398,8 +606,9 @@ export class RegulatoryMappingService { const viewedTimeslot = new Set(viewed.timeslot); const batchRuns = (await Promise.all((await this.read.listProjects()) .map(project => this.store.listRegulatoryMappingBatchRuns(project.id)))).flat(); + // User decision 2026-10-09: the history lists ISO and direct BSI mappings, each named by its framework. const batches: RegulatoryMappingHistoryEntry[] = batchRuns.map(run => ({ - id: run.id, kind: 'batch', + id: run.id, kind: 'batch', framework: mappingFramework(run), scope: run.projectId !== projectId ? 'other-client' : run.assessmentIds.some(id => assessments.has(id)) || run.scanRunIds.some(id => scanRuns.has(id)) ? 'timeslot' : 'other-timeslot', trigger: AUTO_MAPPING_BATCH_KEY_PREFIXES.some(prefix => run.idempotencyKey.startsWith(prefix)) ? 'automatic' : 'manual', @@ -412,7 +621,7 @@ export class RegulatoryMappingService { const singles: RegulatoryMappingHistoryEntry[] = (await this.store.listRegulatoryMappingRuns()) .filter(run => run.trigger === 'single') .map(run => ({ - id: run.id, kind: 'single', trigger: 'manual', + id: run.id, kind: 'single', framework: mappingFramework(run), trigger: 'manual', scope: !run.origin ? 'unknown' : run.origin.timeslot.some(token => viewedTimeslot.has(token)) ? 'timeslot' : run.origin.project === viewed.project ? 'other-timeslot' : 'other-client', status: run.status === 'succeeded' ? 'completed' : run.status, @@ -425,17 +634,19 @@ export class RegulatoryMappingService { } /** - * Accepts the first proposals of a finished mapping, in the model's order. A control that already has a + * Accepts the first two strong proposals of a finished mapping, in the model's order. A control that already has a * decision keeps it, so a manual reject is never overridden; a failure here never fails the mapping. */ private async autoAcceptFirstProposals(run: RegulatoryMappingRun, result: RegulatoryMappingResult): Promise { if (result.outcome !== 'mapped') return; try { - const decided = new Set((await this.store.listRegulatoryMappingDecisions(run.ruleKey)).map(item => item.controlId)); - for (const proposal of result.mappings.slice(0, AUTO_ACCEPTED_PROPOSALS)) { + // User decision 2026-10-09: direct BSI mappings are accepted automatically like ISO ones, for now. + const framework = mappingFramework(run); + const decided = new Set((await this.store.listRegulatoryMappingDecisions(run.ruleKey, framework)).map(item => item.controlId)); + for (const proposal of result.mappings.filter(isStrongProposal).slice(0, AUTO_ACCEPTED_PROPOSALS)) { if (decided.has(proposal.controlId)) continue; await this.store.createRegulatoryMappingDecision({ - version: 1, id: `regulatory-decision-${randomUUID()}`, ruleKey: run.ruleKey, sourceTool: run.sourceTool, + version: 1, id: `regulatory-decision-${randomUUID()}`, ...frameworkField(framework), ruleKey: run.ruleKey, sourceTool: run.sourceTool, ...(run.sourceRuleId ? { sourceRuleId: run.sourceRuleId } : {}), mappingId: run.id, controlId: proposal.controlId, decision: 'accepted', comment: AUTO_ACCEPT_COMMENT, createdAt: this.now().toISOString() }); @@ -456,8 +667,10 @@ export class RegulatoryMappingService { // Deterministic id per (ruleKey, pipeline versions) so one row exists per rule per // version tuple: regeneration reuses it, and a version bump yields a new mapping. - private mappingId(ruleKey: string): string { - const tuple = [ruleKey, this.options.corpusVersion, REGULATORY_RETRIEVAL_VERSION, REGULATORY_MAPPING_PROMPT_VERSION, this.options.client!.model, + // ISO ids keep their earlier tuple; a BSI id also names its framework. + private mappingId(ruleKey: string, pipeline: MappingPipeline): string { + const tuple = [...(pipeline.framework === 'bsi-kernel' ? [pipeline.framework] : []), + ruleKey, pipeline.corpusVersion, pipeline.retrievalVersion, pipeline.promptVersion, this.options.client!.model, // GPO rules carry the baseline context of their query version; scanner mapping ids stay unchanged. ...(isGpoRuleKey(ruleKey) ? [REGULATORY_GPO_QUERY_VERSION] : [])].join(''); return `regulatory-mapping-${createHash('sha256').update(tuple, 'utf8').digest('hex').slice(0, 40)}`; @@ -465,8 +678,8 @@ export class RegulatoryMappingService { // The stored prompt version must differ wherever the id differs: the table's unique // identity tuple has no column for the GPO query version, so GPO runs record it here. - private promptVersion(ruleKey: string): string { - return isGpoRuleKey(ruleKey) ? `${REGULATORY_MAPPING_PROMPT_VERSION}+${REGULATORY_GPO_QUERY_VERSION}` : REGULATORY_MAPPING_PROMPT_VERSION; + private promptVersion(ruleKey: string, pipeline: MappingPipeline): string { + return isGpoRuleKey(ruleKey) ? `${pipeline.promptVersion}+${REGULATORY_GPO_QUERY_VERSION}` : pipeline.promptVersion; } async close(): Promise { @@ -495,26 +708,27 @@ export class RegulatoryMappingService { } } - private queuedRun(finding: UnifiedResultProjection, trigger: 'single' | 'batch', origin?: RegulatoryMappingRun['origin']): RegulatoryMappingRun { + private queuedRun(finding: UnifiedResultProjection, trigger: 'single' | 'batch', pipeline: MappingPipeline, origin?: RegulatoryMappingRun['origin']): RegulatoryMappingRun { const preparedQuery = buildRegulatoryFindingQuery(finding); if (!preparedQuery) throw new RegulatoryMappingServiceError('insufficient_public_context', 'This finding has insufficient public context for regulatory mapping.', 422); const timestamp = this.now().toISOString(); const ruleKey = this.ruleKey(finding); return { version: 1, - id: this.mappingId(ruleKey), + id: this.mappingId(ruleKey, pipeline), + ...frameworkField(pipeline.framework), ruleKey, sourceTool: finding.sourceTool ?? 'unknown', ...(finding.sourceRuleId ? { sourceRuleId: finding.sourceRuleId } : {}), status: 'queued', queryHash: sha256(JSON.stringify(preparedQuery)), preparedQuery, - corpusId: this.options.corpusId!, - corpusVersion: this.options.corpusVersion!, + corpusId: pipeline.corpusId, + corpusVersion: pipeline.corpusVersion, sourceHashes: [], embeddingModel: this.options.embeddingModel!, - retrievalVersion: REGULATORY_RETRIEVAL_VERSION, - promptVersion: this.promptVersion(ruleKey), + retrievalVersion: pipeline.retrievalVersion, + promptVersion: this.promptVersion(ruleKey, pipeline), outputSchemaVersion: REGULATORY_MAPPING_OUTPUT_SCHEMA_VERSION, model: this.options.client!.model, trigger, @@ -525,8 +739,9 @@ export class RegulatoryMappingService { }; } - private async claim(finding: UnifiedResultProjection, forceRegeneration: boolean, trigger: 'single' | 'batch', origin?: RegulatoryMappingRun['origin']) { - return this.store.claimRegulatoryMappingRun(this.queuedRun(finding, trigger, origin), forceRegeneration); + private async claim(finding: UnifiedResultProjection, forceRegeneration: boolean, trigger: 'single' | 'batch', pipeline: MappingPipeline, + origin?: RegulatoryMappingRun['origin']) { + return this.store.claimRegulatoryMappingRun(this.queuedRun(finding, trigger, pipeline, origin), forceRegeneration); } /** The client always; the timeslot only when it is a valid timeslot of that client. */ @@ -555,8 +770,9 @@ export class RegulatoryMappingService { return scope; } - private batchScopeKey(projectId: string, scope: RegulatoryMappingBatchScope): string { - return sha256(canonicalJson({ projectId, ...scope })); + // ISO keys keep their earlier input, so an active ISO batch and a BSI batch of the same scope never collide. + private batchScopeKey(projectId: string, scope: RegulatoryMappingBatchScope, framework: RegulatoryFramework = 'iso27001'): string { + return sha256(canonicalJson({ projectId, ...scope, ...frameworkField(framework) })); } private async queryBatchResults(projectId: string, scope: RegulatoryMappingBatchScope): Promise { @@ -628,15 +844,16 @@ export class RegulatoryMappingService { counts.unsupportedFindingCount = unsupported.size; counts.distinctRuleCount = representatives.size; const scheduled: Array> = []; + const pipeline = await this.pipeline(mappingFramework(run)); for (const finding of representatives.values()) { - const claimed = await this.claim(finding, false, 'batch'); + const claimed = await this.claim(finding, false, 'batch', pipeline); if (!claimed.claimed) { if (claimed.run.status === 'succeeded') counts.reusedSucceededRuleCount += 1; else counts.reusedActiveRuleCount += 1; continue; } counts.scheduledRuleCount += 1; - const completion = this.enqueue(() => this.process(claimed.run, finding)); + const completion = this.enqueue(() => this.process(claimed.run, finding, pipeline)); this.track(completion.then(() => undefined)); scheduled.push(completion); } @@ -669,7 +886,11 @@ export class RegulatoryMappingService { } } - private async availability(): Promise<{ enabled: boolean; reason?: string }> { + private async availability(framework: RegulatoryFramework = 'iso27001'): Promise<{ enabled: boolean; reason?: string }> { + if (framework === 'bsi-kernel') { + // The BSI corpus is built from the bundled or updated catalog when the first BSI mapping needs it. + return this.frameworkEnabled(framework) ? { enabled: true } : { enabled: false, reason: this.disabledError(framework).message }; + } if (!this.generationEnabled) return { enabled: false, reason: this.options.disabledReason ?? 'Regulatory mapping is not configured.' }; try { const sample = await this.options.index!.list({ @@ -692,20 +913,53 @@ export class RegulatoryMappingService { return finding; } - private async groupCandidates(hits: RegulatoryRetrievalCandidate[]): Promise { - const byControl = new Map(); - for (const hit of hits) { - if (!hit.chunk.annexAControlId || !hit.chunk.sectionKind || !hit.chunk.sourcePageStart || !hit.chunk.sourcePageEnd || !hit.chunk.sourceSha256) continue; - const values = byControl.get(hit.chunk.controlId) ?? []; - values.push(hit); - byControl.set(hit.chunk.controlId, values); - } - const selected = [...byControl.entries()] - .map(([controlId, values]) => ({ controlId, values: values.sort((left, right) => right.score - left.score), score: Math.max(...values.map(value => value.score)) })) - .sort((left, right) => right.score - left.score || left.controlId.localeCompare(right.controlId, 'en', { numeric: true })) - .slice(0, 8); + /** + * The further ranked lists of the hybrid search (user decision 2026-10-09): a dense search with a German restatement + * of the finding and a keyword search with its German terms. The norm texts are German, so an English query alone + * matches them weakly. Without a German restatement the mapping keeps the English dense search only. + */ + private async hybridRankings(query: RegulatoryFindingQuery, pipeline: MappingPipeline, limit: number) { + const expansion = this.options.queryExpansionClient ? await this.expandQuery(query) : undefined; + if (!expansion) return { lists: [] as RegulatoryRetrievalCandidate[][] }; + const german = `${expansion.query} ${expansion.terms.join(' ')}`; + const lists = [await this.options.findingRetrieval!.retrieveText(german, limit, retrievalFilters(pipeline))]; + const lexical = await this.lexicalIndex(pipeline); + if (lexical) lists.push(lexical.search(german, limit)); + return { expansion: { ...expansion, searches: lexical ? ['dense-en', 'dense-de', 'keyword-de'] : ['dense-en', 'dense-de'] }, lists }; + } + + /** One German restatement per public query, shared by the ISO and BSI mapping of a rule; a failure is retried next time. */ + private expandQuery(query: RegulatoryFindingQuery): Promise | undefined> { + const key = sha256(query.query); + const cached = this.expansions.get(key); + if (cached) return cached; + const task = this.options.queryExpansionClient!.complete(QUERY_EXPANSION_PROMPT, query.query) + .then(completion => queryExpansion(completion.value)).catch(() => undefined); + this.expansions.set(key, task); + void task.then(value => { if (!value) this.expansions.delete(key); }); + while (this.expansions.size > MAXIMUM_CACHED_EXPANSIONS) this.expansions.delete(this.expansions.keys().next().value!); + return task; + } + + /** The keyword index of one corpus version, built once from Qdrant; unavailable without a full listing. */ + private lexicalIndex(pipeline: MappingPipeline): Promise { + const key = `${pipeline.corpusId}\u0000${pipeline.corpusVersion}`; + const cached = this.lexicalIndexes.get(key); + if (cached) return cached; + const listAll = this.options.index!.listAll?.bind(this.options.index); + const task = listAll ? listAll(retrievalFilters(pipeline)).then(chunks => new RegulatoryLexicalIndex(chunks)).catch(() => undefined) : Promise.resolve(undefined); + this.lexicalIndexes.set(key, task); + void task.then(value => { if (!value) this.lexicalIndexes.delete(key); }); + return task; + } + + /** The eight best ISO controls of the fused ranking, each with its 27001 statement and full 27002 text. */ + private async groupCandidates(fused: FusedControl[]): Promise { + const usable = (chunk: RegulatoryChunk) => Boolean(chunk.annexAControlId && chunk.sectionKind && chunk.sourcePageStart && chunk.sourcePageEnd && chunk.sourceSha256); + const selected = fused.map(control => ({ ...control, chunks: control.chunks.filter(usable) })) + .filter(control => control.chunks.length > 0).slice(0, ISO_CANDIDATE_LIMIT); return Promise.all(selected.map(async selection => { - const semantic = selection.values.slice(0, 2).map(value => evidence(value.chunk)); + const semantic = await this.fullControlText(selection.controlId, selection.chunks); const references = await this.options.index!.list({ corpusId: this.options.corpusId, corpusVersion: this.options.corpusVersion, @@ -716,49 +970,100 @@ export class RegulatoryMappingService { if (references.length !== 1 || !references[0].sourceSha256) { throw new RegulatoryMappingServiceError('regulatory_reference_missing', `The active corpus is missing the ISO 27001 reference for ${selection.controlId}.`, 503); } - const first = selection.values[0].chunk; + const first = selection.chunks[0]; return { controlId: selection.controlId, annexAControlId: first.annexAControlId!, title: first.title, score: selection.score, - evidence: [...semantic, ...references.map(evidence)] + evidence: [...references.map(evidence), ...semantic] }; })); } - private async process(queued: RegulatoryMappingRun, finding: UnifiedResultProjection): Promise<'succeeded' | 'failed'> { + /** + * The ISO/IEC 27002 text of one candidate control for the model (prompt 1.3.0, user decision 2026-10-09): control, + * purpose and guidance in document order instead of only the two best-matching passages, then other information + * while it fits. Retrieval only selects the control. The passages that matched are always kept; the rest stops at + * a per-control cap, so a very long control cannot crowd out the other candidates. + */ + private async fullControlText(controlId: string, matched: RegulatoryChunk[]): Promise { + const listed = await this.options.index!.list({ corpusId: this.options.corpusId, corpusVersion: this.options.corpusVersion, + controlId, sourceStandard: 'ISO/IEC 27002' }, 100); + const usable = (chunk: RegulatoryChunk) => chunk.controlId === controlId && chunk.sourceStandard === 'ISO/IEC 27002' + && Boolean(chunk.sectionKind && chunk.sourcePageStart && chunk.sourcePageEnd && chunk.sourceSha256); + const byId = new Map([...listed, ...matched].filter(usable).map(chunk => [chunk.chunkId, chunk])); + const ordered = [...byId.values()].sort((left, right) => SECTION_ORDER.indexOf(left.sectionKind!) - SECTION_ORDER.indexOf(right.sectionKind!) + || left.documentId.localeCompare(right.documentId) || left.chunkIndex - right.chunkIndex); + const kept = new Set(matched.filter(usable).map(chunk => chunk.chunkId)); + let length = [...kept].reduce((total, id) => total + byId.get(id)!.text.length, 0); + // Core sections first, other information only with the space left. + for (const pass of [CORE_SECTIONS, ['other-information']] as const) { + for (const chunk of ordered) { + if (kept.has(chunk.chunkId) || !pass.includes(chunk.sectionKind as never)) continue; + if (length + chunk.text.length > MAXIMUM_CONTROL_TEXT_CHARACTERS) continue; + kept.add(chunk.chunkId); + length += chunk.text.length; + } + } + return ordered.filter(chunk => kept.has(chunk.chunkId)).map(evidence); + } + + /** + * The best Kernel requirements of the fused ranking, each with every chunk of its requirement as evidence, + * so the model always reads the statement even when only its guidance was retrieved. + */ + private async groupBsiCandidates(fused: FusedControl[], corpusVersion: string): Promise { + const { catalog } = await this.bsiCatalogs(); + const known = new Map(catalog.requirements.map(requirement => [requirement.id, requirement])); + const selected = fused.filter(control => known.has(control.controlId) && control.chunks.some(chunk => chunk.corpusId === BSI_CORPUS_ID)) + .slice(0, BSI_CANDIDATE_LIMIT).map(control => [control.controlId, control.score] as const); + return Promise.all(selected.map(async ([controlId, score]) => { + const chunks = (await this.options.index!.list({ corpusId: BSI_CORPUS_ID, corpusVersion, controlId }, 3)) + .sort((left, right) => left.chunkIndex - right.chunkIndex); + if (!chunks.length) throw new RegulatoryMappingServiceError('regulatory_reference_missing', `The BSI corpus is missing requirement ${controlId}.`, 503); + return { controlId, annexAControlId: controlId, title: known.get(controlId)!.title, score, + evidence: chunks.map(chunk => bsiEvidence(chunk, catalog.sourceSha256)) }; + })); + } + + private async process(queued: RegulatoryMappingRun, finding: UnifiedResultProjection, pipeline: MappingPipeline): Promise<'succeeded' | 'failed'> { let run = queued; + const bsi = pipeline.framework === 'bsi-kernel'; try { let retrieval; try { - retrieval = await this.options.findingRetrieval!.retrieve(finding, 32, { - corpusId: this.options.corpusId, - corpusVersion: this.options.corpusVersion, - sourceStandard: 'ISO/IEC 27002', - retrievalRole: 'semantic' - }); + // The BSI corpus is indexed from the active catalog the first time a BSI mapping needs it. + if (bsi && await this.options.bsiCorpus!.ensure() !== pipeline.corpusVersion) { + throw new Error('The BSI catalog changed while the mapping was queued.'); + } + retrieval = await this.options.findingRetrieval!.retrieve(finding, bsi ? 40 : 32, retrievalFilters(pipeline)); } catch { throw new RegulatoryMappingServiceError('regulatory_retrieval_unavailable', 'Regulatory retrieval is currently unavailable.', 503); } if (retrieval.status !== 'candidates' || !retrieval.preparedQuery) { throw new RegulatoryMappingServiceError('insufficient_public_context', 'This finding has insufficient public context for regulatory mapping.', 422); } - const candidates = await this.groupCandidates(retrieval.candidates); + const hybrid = await this.hybridRankings(retrieval.preparedQuery, pipeline, bsi ? 40 : 32); + // The run records the German restatement it searched with. + const preparedQuery: RegulatoryFindingQuery = hybrid.expansion ? { ...retrieval.preparedQuery, expansion: hybrid.expansion } : retrieval.preparedQuery; + // BSI only: procurement, service providers, buildings and personnel are not the subject of a technical finding. + const fused = fuseControlRankings([retrieval.candidates, ...hybrid.lists], bsi ? controlId => BSI_EXCLUDED_PRACTICES.test(controlId) : undefined); + const candidates = bsi ? await this.groupBsiCandidates(fused, pipeline.corpusVersion) : await this.groupCandidates(fused); if (!candidates.length) throw new RegulatoryMappingServiceError('regulatory_corpus_empty', 'The active regulatory corpus returned no eligible controls.', 503); const startedAt = this.now().toISOString(); run = await this.store.saveRegulatoryMappingRun({ ...queued, status: 'running', - preparedQuery: retrieval.preparedQuery, - queryHash: sha256(JSON.stringify(retrieval.preparedQuery)), + preparedQuery, + queryHash: sha256(JSON.stringify(preparedQuery)), sourceHashes: [...new Set(candidates.flatMap(candidate => candidate.evidence.map(evidence => evidence.sourceSha256)))].sort(), candidates, startedAt, updatedAt: startedAt }); const prompt = mappingPrompt(run); - const completion = await this.options.client!.complete(SYSTEM_PROMPT, prompt); + const completion = await this.options.client!.complete(bsi ? BSI_SYSTEM_PROMPT : SYSTEM_PROMPT, prompt); const allowed = new Map(run.candidates.map(candidate => [ candidate.controlId, new Set(candidate.evidence.map(item => item.chunkId)) ])); @@ -820,16 +1125,27 @@ function evidence(chunk: RegulatoryRetrievalCandidate['chunk']): RegulatoryMappi }; } +function bsiEvidence(chunk: RegulatoryChunk, sourceSha256: string): RegulatoryMappingEvidence { + return { chunkId: chunk.chunkId, textSha256: chunk.textSha256, sourceSha256, sourceStandard: BSI_SOURCE_STANDARD, sectionKind: 'control', text: chunk.text }; +} + function mappingPrompt(run: RegulatoryMappingRun): string { // The Annex A id is always "A." + controlId, so it is redundant for the model and // sending it invites a reply that uses the Annex A form; expose only the controlId it must return. + // A BSI candidate repeats its requirement id there, and the BSI evidence has no pages. const candidates = run.candidates.map(({ annexAControlId, ...candidate }) => candidate); const payload = JSON.stringify({ finding: run.preparedQuery, candidates }); if (payload.length > MAXIMUM_PROMPT_CHARACTERS) throw new RegulatoryMappingServiceError('regulatory_mapping_prompt_too_large', 'The bounded regulatory prompt exceeded its limit.', 422); return payload; } -const SYSTEM_PROMPT = `Map one public security finding query to the supplied ISO control candidates. A mapping means the control is topically relevant to the finding, never proof of compliance. Map a candidate when it governs the same security subject as the finding (for example password or authentication management, logging, or access restriction), even when the control text does not state the finding's specific value, threshold, parameter, or configuration; ISO controls are principle-level and rarely restate such specifics. Judge relevance from the control's subject and its supplied evidence, not from whether the exact requirement text appears. Return JSON only with schemaVersion 1, outcome mapped|no_relevant_control|insufficient_context, optional bounded reason, and mappings (maximum five). Each mapping requires controlId, relationship primary|supporting, confidence low|medium|high, a concise rationale under 600 characters, and one to three evidenceChunkIds copied only from that control's supplied evidence. Use no_relevant_control only when no candidate governs the finding's subject, and insufficient_context only when the finding query itself lacks enough meaning to judge. Do not invent controls or evidence.`; +const SYSTEM_PROMPT = `Map one public security finding query to the supplied ISO control candidates. Each candidate carries its ISO/IEC 27001 control statement and the ISO/IEC 27002 control, purpose and guidance text of that control in document order; read the whole text before judging. A mapping means the control is topically relevant to the finding, never proof of compliance. Map a candidate when it governs the same security subject as the finding (for example password or authentication management, logging, or access restriction), even when the control text does not state the finding's specific value, threshold, parameter, or configuration; ISO controls are principle-level and rarely restate such specifics. Judge relevance from the control's subject and its supplied evidence, not from whether the exact requirement text appears. Return JSON only with schemaVersion 1, outcome mapped|no_relevant_control|insufficient_context, optional bounded reason, and mappings (maximum five). Each mapping requires controlId, relationship primary|supporting, confidence low|medium|high, a concise rationale under 600 characters, and one to three evidenceChunkIds copied only from that control's supplied evidence. Use no_relevant_control only when no candidate governs the finding's subject, and insufficient_context only when the finding query itself lacks enough meaning to judge. Do not invent controls or evidence.`; + +const BSI_SYSTEM_PROMPT = `Map one public security finding query to the supplied BSI Grundschutz++ Kernel requirement candidates. The requirements are published in German; MUSS, SOLLTE and KANN state their obligation level. A mapping means the requirement is topically relevant to the finding, never proof of compliance. Map a candidate when it governs the same security subject as the finding (for example authentication, hardening of a configuration, logging, or restriction of access), even when the requirement does not restate the finding's specific value, threshold, parameter, or configuration. Prefer the requirements that address the finding's specific subject over general management or process requirements. Judge relevance from the requirement's subject and its supplied evidence. Return JSON only with schemaVersion 1, outcome mapped|no_relevant_control|insufficient_context, optional bounded reason, and mappings (maximum five). Each mapping requires controlId (the requirement id exactly as supplied, for example KONF.2.1), relationship primary|supporting, confidence low|medium|high, a concise English rationale under 600 characters, and one to three evidenceChunkIds copied only from that requirement's supplied evidence. Use no_relevant_control only when no candidate governs the finding's subject, and insufficient_context only when the finding query itself lacks enough meaning to judge. Do not invent requirements or evidence.`; + +function byNumber(left: string, right: string): number { + return left.localeCompare(right, 'en', { numeric: true }); +} function safeFailure(error: unknown): string { if (error instanceof RegulatoryMappingServiceError || error instanceof LiteLlmClientError) return error.code; diff --git a/server/index.ts b/server/index.ts index 7289f2f..4fb7bc8 100644 --- a/server/index.ts +++ b/server/index.ts @@ -158,6 +158,7 @@ async function main(): Promise { regulatoryMappingService: complianceEngine.mappingService, regulatoryCorpusUploadService: complianceEngine.corpusUploadService, bsiCatalogService: complianceEngine.bsiCatalogService, + bsiSearchIndex: complianceEngine.bsiCorpus, complianceExport: complianceEngine.store, isoUploadToken: process.env.ASSESSOR_ISO_UPLOAD_TOKEN?.trim(), automaticGpoAnalysis, diff --git a/server/postgres/migration-catalog.ts b/server/postgres/migration-catalog.ts index 6c07948..ab3e3a8 100644 --- a/server/postgres/migration-catalog.ts +++ b/server/postgres/migration-catalog.ts @@ -24,6 +24,7 @@ import aiOverviewRuns from './migrations/025_ai_overview_runs.sql?raw'; import regulatoryMapping from './migrations/026_regulatory_mapping.sql?raw'; import vulnerabilityFindingResultType from './migrations/027_vulnerability_finding_result_type.sql?raw'; import scopedRiskAcceptances from './migrations/028_scoped_risk_acceptances.sql?raw'; +import regulatoryMappingFramework from './migrations/029_regulatory_mapping_framework.sql?raw'; export interface DatabaseMigration { version: string; @@ -57,5 +58,6 @@ export const databaseMigrations: readonly DatabaseMigration[] = [ { version: '025', name: 'ai-overview-runs', sql: aiOverviewRuns }, { version: '026', name: 'regulatory-mapping', sql: regulatoryMapping }, { version: '027', name: 'vulnerability-finding-result-type', sql: vulnerabilityFindingResultType }, - { version: '028', name: 'scoped-risk-acceptances', sql: scopedRiskAcceptances } + { version: '028', name: 'scoped-risk-acceptances', sql: scopedRiskAcceptances }, + { version: '029', name: 'regulatory-mapping-framework', sql: regulatoryMappingFramework } ]; diff --git a/server/postgres/migrations/029_regulatory_mapping_framework.sql b/server/postgres/migrations/029_regulatory_mapping_framework.sql new file mode 100644 index 0000000..6feab38 --- /dev/null +++ b/server/postgres/migrations/029_regulatory_mapping_framework.sql @@ -0,0 +1,29 @@ +-- Phase 13F, BSI Grundschutz++ stage 2 (user decision 2026-10-09): the Compliance Engine maps rules to ISO +-- controls and, separately, directly to BSI Kernel requirements. Mappings and decisions name their framework; +-- every existing row is an ISO mapping. Consistency reviews record that a person kept an ISO <-> BSI +-- disagreement on purpose, for one fingerprint of the compared sets. All three tables belong to the engine. + +ALTER TABLE assessor_regulatory_mappings + ADD COLUMN framework text NOT NULL DEFAULT 'iso27001' CHECK (framework IN ('iso27001', 'bsi-kernel')); + +ALTER TABLE assessor_regulatory_mappings DROP CONSTRAINT assessor_regulatory_mappings_identity_uq; +ALTER TABLE assessor_regulatory_mappings ADD CONSTRAINT assessor_regulatory_mappings_identity_uq + UNIQUE (framework, rule_key, corpus_version, retrieval_version, prompt_version, model); + +ALTER TABLE assessor_regulatory_mapping_decisions + ADD COLUMN framework text NOT NULL DEFAULT 'iso27001' CHECK (framework IN ('iso27001', 'bsi-kernel')); + +DROP INDEX assessor_regulatory_mapping_decisions_rule_control_idx; +CREATE INDEX assessor_regulatory_mapping_decisions_rule_control_idx + ON assessor_regulatory_mapping_decisions (rule_key, framework, control_id, created_at DESC, id DESC); + +CREATE TABLE assessor_regulatory_consistency_reviews ( + id text PRIMARY KEY, + rule_key text NOT NULL, + fingerprint text NOT NULL CHECK (fingerprint ~ '^[a-f0-9]{64}$'), + created_at timestamptz NOT NULL, + data jsonb NOT NULL +); + +CREATE INDEX assessor_regulatory_consistency_reviews_rule_idx + ON assessor_regulatory_consistency_reviews (rule_key, created_at DESC, id DESC); diff --git a/server/postgres/postgres-assessor-repository.ts b/server/postgres/postgres-assessor-repository.ts index a35aa61..d587594 100644 --- a/server/postgres/postgres-assessor-repository.ts +++ b/server/postgres/postgres-assessor-repository.ts @@ -384,6 +384,7 @@ const DB_STRUCTURE_LAYER_BY_TABLE: Record = { assessor_regulatory_mappings: 'result', assessor_regulatory_mapping_decisions: 'governance', assessor_regulatory_mapping_batches: 'governance', + assessor_regulatory_consistency_reviews: 'governance', assessor_ai_overview_runs: 'governance', assessor_finding_occurrences: 'result', assessor_result_categories: 'result', diff --git a/src/api/assessor-api-client.ts b/src/api/assessor-api-client.ts index b8a5a61..c830fe2 100644 --- a/src/api/assessor-api-client.ts +++ b/src/api/assessor-api-client.ts @@ -17,7 +17,9 @@ import type { ClientInformationQuery, ClientInformationResult } from '../domain/ import type { ImportedDataset } from '../domain/dataset'; import type { MicrosoftSecurityBaselineSelectionContext } from '../domain/microsoft-security-baseline'; import type { RegulatoryCorpusStatus } from '../compliance/regulatory-corpus-management'; -import type { BsiCatalogRefreshResult, BsiCatalogState } from '../compliance/bsi-catalog-management'; +import type { RegulatoryFramework } from '../compliance/regulatory-mapping'; +import type { RegulatoryConsistencyResponse, RegulatoryConsistencyReviewCreateRequest } from '../compliance/bsi-consistency'; +import type { BsiCatalogRefreshResult, BsiCatalogState, BsiSearchIndexStatus } from '../compliance/bsi-catalog-management'; import type { ImportBatchPreview, ImportBatchSelection } from '../domain/import-workflow'; import type { ResultDbMaterializationStatus } from '../domain/result-db-materialization'; import { @@ -107,6 +109,11 @@ export interface AssessorApiClientOptions { describeActivity?: typeof requestActivity; } +/** ISO requests keep their earlier URL; only another framework is named. */ +function frameworkQuery(framework: RegulatoryFramework, separator: '?' | '&'): string { + return framework === 'iso27001' ? '' : `${separator}framework=${encodeURIComponent(framework)}`; +} + function unifiedQueryPath(path: string, query: UnifiedResultQuery = {}): string { const parameters = new URLSearchParams(); Object.entries(query).forEach(([key, value]) => { @@ -244,8 +251,8 @@ export class AssessorApiClient { }); } - getRegulatoryMappings(projectId: string, resultProjectionId: string): Promise { - return this.request(`/projects/${encodeURIComponent(projectId)}/regulatory-mappings/${encodeURIComponent(resultProjectionId)}`); + getRegulatoryMappings(projectId: string, resultProjectionId: string, framework: RegulatoryFramework = 'iso27001'): Promise { + return this.request(`/projects/${encodeURIComponent(projectId)}/regulatory-mappings/${encodeURIComponent(resultProjectionId)}${frameworkQuery(framework, '?')}`); } generateRegulatoryMappings(projectId: string, request: RegulatoryMappingCreateRequest): Promise { @@ -260,14 +267,28 @@ export class AssessorApiClient { }); } - getRegulatoryMappingBatch(projectId: string, scope: RegulatoryMappingBatchScope): Promise { - return this.request(unifiedQueryPath(`/projects/${encodeURIComponent(projectId)}/regulatory-mapping-batches`, { + getRegulatoryMappingBatch(projectId: string, scope: RegulatoryMappingBatchScope, framework: RegulatoryFramework = 'iso27001'): Promise { + const path = unifiedQueryPath(`/projects/${encodeURIComponent(projectId)}/regulatory-mapping-batches`, { ...scope.filters, assessmentIds: scope.assessmentIds, scanRunIds: scope.scanRunIds + }); + return this.request(`${path}${frameworkQuery(framework, path.includes('?') ? '&' : '?')}`); + } + + getRegulatoryMappingConsistency(projectId: string, scope: { assessmentIds: string[]; scanRunIds: string[] }): Promise { + return this.request(unifiedQueryPath(`/projects/${encodeURIComponent(projectId)}/regulatory-mapping-consistency`, { + assessmentIds: scope.assessmentIds, + scanRunIds: scope.scanRunIds })); } + markRegulatoryMappingConsistencyReviewed(projectId: string, request: RegulatoryConsistencyReviewCreateRequest): Promise<{ reviewed: true }> { + return this.request(`/projects/${encodeURIComponent(projectId)}/regulatory-mapping-consistency-reviews`, { + method: 'POST', body: JSON.stringify(request) + }); + } + getRegulatoryControlAssignments(projectId: string, scope: { assessmentIds: string[]; scanRunIds: string[] }): Promise { return this.request(unifiedQueryPath(`/projects/${encodeURIComponent(projectId)}/regulatory-control-assignments`, { assessmentIds: scope.assessmentIds, @@ -496,6 +517,14 @@ export class AssessorApiClient { return this.request('/reference-data/bsi-grundschutz/refresh', { method: 'POST', body: '{}' }); } + getBsiSearchIndex(): Promise { + return this.request('/reference-data/bsi-grundschutz/search-index'); + } + + buildBsiSearchIndex(force: boolean): Promise { + return this.request('/reference-data/bsi-grundschutz/search-index/build', { method: 'POST', body: JSON.stringify({ force }) }); + } + resetBsiCatalog(): Promise { return this.request('/reference-data/bsi-grundschutz/reset', { method: 'POST', body: '{}' }); } diff --git a/src/api/request-activity.ts b/src/api/request-activity.ts index e5fd184..cbc5931 100644 --- a/src/api/request-activity.ts +++ b/src/api/request-activity.ts @@ -31,12 +31,14 @@ function complianceActivity(route: string, method: string): string | undefined { const write = method !== 'GET'; if (route.includes('/regulatory-control-assignments')) return 'Loading ISO control filters'; if (route.includes('/regulatory-control-overview')) return 'Loading ISO control overview'; - if (route.includes('/regulatory-mapping-batches')) return write ? 'Starting ISO mapping batch' : 'Loading ISO mapping batch status'; + if (route.includes('/regulatory-mapping-batches')) return write ? 'Starting compliance mapping batch' : 'Loading compliance mapping batch status'; if (route.includes('/regulatory-mapping-runs')) return 'Generating ISO control proposals'; if (route.includes('/regulatory-mapping-decisions')) return 'Saving ISO mapping decision'; if (route.includes('/regulatory-mappings/')) return 'Loading ISO mapping'; if (route.endsWith('/ai-overview')) return write ? 'Starting AI overview' : 'Loading AI overview status'; if (route.endsWith('/reference-data/iso-corpus')) return write ? 'Uploading ISO corpus' : 'Loading ISO corpus status'; + if (route.endsWith('/reference-data/bsi-grundschutz/search-index/build')) return 'Building BSI search index'; + if (route.endsWith('/reference-data/bsi-grundschutz/search-index')) return 'Loading BSI search index status'; if (route.endsWith('/reference-data/bsi-grundschutz/refresh')) return 'Updating BSI catalog'; if (route.endsWith('/reference-data/bsi-grundschutz/reset')) return 'Restoring bundled BSI catalog'; if (route.endsWith('/reference-data/bsi-grundschutz')) return 'Loading BSI catalog'; diff --git a/src/compliance/bsi-catalog-management.ts b/src/compliance/bsi-catalog-management.ts index 3fb50a7..1aab694 100644 --- a/src/compliance/bsi-catalog-management.ts +++ b/src/compliance/bsi-catalog-management.ts @@ -17,3 +17,23 @@ export interface BsiCatalogState { export interface BsiCatalogRefreshResult extends BsiCatalogState { updated: boolean; } + +/** + * Data Hub BSI tile: the retrieval index of the active Kernel catalog for direct BSI mappings (user decision + * 2026-10-09). `outdated` means only an index of another catalog state exists; `unavailable` means embedding or + * Qdrant is not configured. A BSI mapping still builds a missing index itself. + */ +export interface BsiSearchIndexStatus { + version: 1; + state: 'unavailable' | 'not-built' | 'outdated' | 'building' | 'built' | 'failed'; + catalogVersion: string; + corpusVersion: string; + requirementCount: number; + /** Indexed chunks of the active catalog state; a requirement with long guidance has several. */ + indexedChunks: number; + /** While building: embedded chunks of all chunks. */ + progress?: { completed: number; total: number }; + builtAt?: string; + error?: string; + disabledReason?: string; +} diff --git a/src/compliance/bsi-consistency.ts b/src/compliance/bsi-consistency.ts new file mode 100644 index 0000000..1ea32c0 --- /dev/null +++ b/src/compliance/bsi-consistency.ts @@ -0,0 +1,130 @@ +import type { BsiIsoCrosswalk } from './bsi-grundschutz'; + +/** + * ISO <-> BSI consistency of one rule (Phase 13F, BSI stage 2, user decision 2026-10-09). The BSI requirements the + * model accepted directly are compared with the requirements the official BSI mapping derives from the accepted + * ISO controls. Agreement shows consistency, not correctness: both mappings come from the same model and finding + * text. The result is derived only; it never changes, rejects or blocks a decision. + * + * - `consistent`: at least one direct requirement lies in the derived set (`subset-of` covers several requirements, + * and a finding rarely hits all of them, so one hit is enough). + * - `inconsistent`: a direct requirement is reachable through the BSI mapping, but none lies in the derived set. + * - `no-reference`: no direct requirement is reachable from any ISO control; the gap the direct mapping fills. + */ +export type BsiConsistencyState = 'consistent' | 'inconsistent' | 'no-reference'; + +export interface BsiConsistency { + state: BsiConsistencyState; + /** Accepted direct BSI requirements, sorted. */ + directRequirementIds: string[]; + /** Requirements derived from the accepted ISO controls, sorted. */ + derivedRequirementIds: string[]; + /** Direct requirements that also lie in the derived set. */ + overlapRequirementIds: string[]; + /** Direct requirements the BSI mapping reaches from no ISO control. */ + unreachableRequirementIds: string[]; + /** + * The reverse direction: ISO controls that are not accepted but whose BSI mapping targets contain a direct + * requirement, as candidates the ISO mapping may have missed. + */ + reverseCandidates: Array<{ isoControlId: string; requirementIds: string[] }>; +} + +const byNumber = (left: string, right: string) => left.localeCompare(right, 'en', { numeric: true }); + +/** Undefined while no direct BSI requirement is accepted: then there is nothing to compare. */ +export function bsiConsistency(acceptedIsoControlIds: readonly string[], acceptedBsiRequirementIds: readonly string[], + crosswalk: BsiIsoCrosswalk): BsiConsistency | undefined { + const direct = [...new Set(acceptedBsiRequirementIds)].sort(byNumber); + if (!direct.length) return undefined; + const acceptedIso = new Set(acceptedIsoControlIds); + const derived = new Set(); + const reachable = new Set(); + const reverse = new Map>(); + const directSet = new Set(direct); + for (const entry of crosswalk.entries) { + for (const target of entry.targets) { + reachable.add(target); + if (acceptedIso.has(entry.isoControlId)) derived.add(target); + else if (directSet.has(target)) reverse.set(entry.isoControlId, (reverse.get(entry.isoControlId) ?? new Set()).add(target)); + } + } + const overlap = direct.filter(id => derived.has(id)); + const unreachable = direct.filter(id => !reachable.has(id)); + return { + state: overlap.length ? 'consistent' : unreachable.length === direct.length ? 'no-reference' : 'inconsistent', + directRequirementIds: direct, + derivedRequirementIds: [...derived].sort(byNumber), + overlapRequirementIds: overlap, + unreachableRequirementIds: unreachable, + reverseCandidates: [...reverse.entries()].sort(([left], [right]) => byNumber(left, right)) + .map(([isoControlId, ids]) => ({ isoControlId, requirementIds: [...ids].sort(byNumber) })) + }; +} + +/** + * The canonical text a `Mark reviewed` fingerprint is hashed from: the compared accepted sets and the BSI mapping + * version. A changed decision or mapping version yields a new fingerprint, so the rule is listed again. + */ +export function bsiConsistencyFingerprintSource(acceptedIsoControlIds: readonly string[], acceptedBsiRequirementIds: readonly string[], + crosswalkVersion: string): string { + return JSON.stringify({ + crosswalkVersion, + iso: [...new Set(acceptedIsoControlIds)].sort(byNumber), + bsi: [...new Set(acceptedBsiRequirementIds)].sort(byNumber) + }); +} + +/** A person kept the ISO <-> BSI result of a rule as it is, for one fingerprint. Global per rule like decisions. */ +export interface RegulatoryConsistencyReview { + version: 1; + id: string; + ruleKey: string; + fingerprint: string; + comment?: string; + createdAt: string; +} + +export function regulatoryConsistencyReviewValidationErrors(review: RegulatoryConsistencyReview): string[] { + const errors: string[] = []; + if (!review.id?.trim() || !review.ruleKey?.trim() || review.ruleKey.length > 200) errors.push('Consistency review identity is incomplete.'); + if (!/^[a-f0-9]{64}$/.test(review.fingerprint)) errors.push('Consistency review fingerprint must be SHA-256.'); + if (review.comment !== undefined && (review.comment.length > 1_000 || /<\/?[a-z][^>]*>/i.test(review.comment))) { + errors.push('Consistency review comment must be plain text with at most 1000 characters.'); + } + return errors; +} + +/** One rule of the Compliance tab review list, named by a Result DB row of the timeslot; never by its rule key. */ +export interface RegulatoryConsistencyItem { + resultProjectionId: string; + title: string; + sourceTool: string; + sourceRuleId?: string; + /** Deviations of the timeslot with this rule, counted like Work. */ + findingCount: number; + state: BsiConsistencyState; + /** A person kept exactly this result with `Mark reviewed`. */ + reviewed: boolean; + iso: Array<{ controlId: string; title?: string }>; + direct: Array<{ requirementId: string; title?: string; practice?: string; derived: boolean; reachable: boolean }>; + derivedCount: number; + /** `decidable` when the latest ISO mapping of the rule retrieved the control, so it can be accepted here. */ + reverse: Array<{ controlId: string; title?: string; requirementIds: string[]; decidable: boolean }>; +} + +export interface RegulatoryConsistencyResponse { + version: 1; + projectId: string; + scope: { assessmentIds: string[]; scanRunIds: string[] }; + crosswalkVersion: string; + catalogVersion: string; + /** Rules of the timeslot's deviations by state; `notCompared` has an accepted ISO control but no accepted BSI requirement. */ + counts: { consistent: number; inconsistent: number; noReference: number; notCompared: number; reviewed: number }; + items: RegulatoryConsistencyItem[]; +} + +export interface RegulatoryConsistencyReviewCreateRequest { + resultProjectionId: string; + comment?: string; +} diff --git a/src/compliance/compliance-store.ts b/src/compliance/compliance-store.ts index 8be58ef..11faf61 100644 --- a/src/compliance/compliance-store.ts +++ b/src/compliance/compliance-store.ts @@ -1,5 +1,6 @@ import type { AssessorRepository } from '../persistence/assessor-repository'; -import type { RegulatoryMappingBatchRun, RegulatoryMappingDecision, RegulatoryMappingRun } from './regulatory-mapping'; +import type { RegulatoryFramework, RegulatoryMappingBatchRun, RegulatoryMappingDecision, RegulatoryMappingRun } from './regulatory-mapping'; +import type { RegulatoryConsistencyReview } from './bsi-consistency'; import type { AiOverviewRun } from './ai-overview'; /** @@ -26,6 +27,8 @@ export interface ComplianceProjectExport { /** Mappings and decisions are global and keyed by scanner rule, not by project. */ regulatoryMappingRuns: RegulatoryMappingRun[]; regulatoryMappingDecisions: RegulatoryMappingDecision[]; + /** ISO <-> BSI consistency reviews, global per rule like decisions. */ + regulatoryConsistencyReviews: RegulatoryConsistencyReview[]; /** Succeeded AI overview runs of the project (table of migration 025, owned by the engine). */ aiOverviewRuns: AiOverviewRun[]; } @@ -45,6 +48,9 @@ export interface ComplianceStore { updateRegulatoryMappingBatchRun(run: RegulatoryMappingBatchRun): Promise; listRegulatoryMappingBatchRuns(projectId: string, scopeKey?: string): Promise; createRegulatoryMappingDecision(decision: RegulatoryMappingDecision, expectedLatestDecisionId?: string): Promise; - listRegulatoryMappingDecisions(ruleKey: string): Promise; + /** The decisions of one rule in one framework, ISO by default. */ + listRegulatoryMappingDecisions(ruleKey: string, framework?: RegulatoryFramework): Promise; + createRegulatoryConsistencyReview(review: RegulatoryConsistencyReview): Promise; + listRegulatoryConsistencyReviews(): Promise; exportProject(projectId: string): Promise; } diff --git a/src/compliance/local-compliance-store.ts b/src/compliance/local-compliance-store.ts index 6b72774..19dbf1f 100644 --- a/src/compliance/local-compliance-store.ts +++ b/src/compliance/local-compliance-store.ts @@ -4,11 +4,14 @@ import { regulatoryMappingBatchRunValidationErrors, regulatoryMappingDecisionValidationErrors, regulatoryMappingRunValidationErrors, + mappingFramework, + type RegulatoryFramework, type RegulatoryMappingBatchRun, type RegulatoryMappingDecision, type RegulatoryMappingRun } from './regulatory-mapping'; import { aiOverviewRunValidationErrors, assertAiOverviewTransition, type AiOverviewRun } from './ai-overview'; +import { regulatoryConsistencyReviewValidationErrors, type RegulatoryConsistencyReview } from './bsi-consistency'; function clone(value: T): T { return structuredClone(value); @@ -23,6 +26,7 @@ export class LocalComplianceStore implements ComplianceStore { private batches: RegulatoryMappingBatchRun[] = []; private decisions: RegulatoryMappingDecision[] = []; private aiOverviewRuns: AiOverviewRun[] = []; + private consistencyReviews: RegulatoryConsistencyReview[] = []; async getRegulatoryMappingRun(id: string): Promise { const run = this.runs.find(item => item.id === id); @@ -90,10 +94,11 @@ export class LocalComplianceStore implements ComplianceStore { const errors = regulatoryMappingDecisionValidationErrors(decision); if (errors.length) throw new RepositoryValidationError(errors.join(' ')); const run = this.runs.find(item => item.id === decision.mappingId); - if (!run || run.ruleKey !== decision.ruleKey || run.status !== 'succeeded') { + if (!run || run.ruleKey !== decision.ruleKey || run.status !== 'succeeded' || mappingFramework(run) !== mappingFramework(decision)) { throw new RepositoryNotFoundError('Completed regulatory mapping does not exist for this rule.'); } - const latest = this.decisions.filter(item => item.ruleKey === decision.ruleKey && item.controlId === decision.controlId).sort(newestFirst)[0]; + const latest = this.decisions.filter(item => item.ruleKey === decision.ruleKey && item.controlId === decision.controlId + && mappingFramework(item) === mappingFramework(decision)).sort(newestFirst)[0]; // A control with an earlier decision stays decidable after a remap no longer retrieves it; the chain of // previous decisions leads back to the run that grounded it. if (!latest && !run.candidates.some(candidate => candidate.controlId === decision.controlId)) throw new RepositoryValidationError('Decision control was not a grounded candidate.'); @@ -103,8 +108,19 @@ export class LocalComplianceStore implements ComplianceStore { return clone(stored); } - async listRegulatoryMappingDecisions(ruleKey: string): Promise { - return this.decisions.filter(item => item.ruleKey === ruleKey).map(clone).sort(newestFirst); + async listRegulatoryMappingDecisions(ruleKey: string, framework: RegulatoryFramework = 'iso27001'): Promise { + return this.decisions.filter(item => item.ruleKey === ruleKey && mappingFramework(item) === framework).map(clone).sort(newestFirst); + } + + async createRegulatoryConsistencyReview(review: RegulatoryConsistencyReview): Promise { + const errors = regulatoryConsistencyReviewValidationErrors(review); + if (errors.length) throw new RepositoryValidationError(errors.join(' ')); + this.consistencyReviews.push(clone(review)); + return clone(review); + } + + async listRegulatoryConsistencyReviews(): Promise { + return this.consistencyReviews.map(clone).sort(newestFirst); } async createAiOverviewRun(run: AiOverviewRun): Promise { @@ -147,6 +163,7 @@ export class LocalComplianceStore implements ComplianceStore { regulatoryMappingBatchRuns: this.batches.filter(item => item.projectId === projectId).map(clone), regulatoryMappingRuns: this.runs.filter(item => item.status === 'succeeded').map(clone), regulatoryMappingDecisions: this.decisions.map(clone), + regulatoryConsistencyReviews: this.consistencyReviews.map(clone), aiOverviewRuns: this.aiOverviewRuns.filter(item => item.projectId === projectId && item.status === 'succeeded') .map(clone).sort(newestFirst) }; diff --git a/src/compliance/regulatory-mapping.ts b/src/compliance/regulatory-mapping.ts index 9fcbe48..c4cb52a 100644 --- a/src/compliance/regulatory-mapping.ts +++ b/src/compliance/regulatory-mapping.ts @@ -1,15 +1,41 @@ import type { AiOverviewTokenUsage } from './ai-overview'; import type { BsiFootprint, BsiOverview } from './bsi-grundschutz'; +import type { BsiConsistencyState } from './bsi-consistency'; import type { RegulatoryFindingQuery, RegulatorySectionKind } from './regulatory-rag'; import type { ResultDbFilters } from '../domain/result-db-query'; import type { UnifiedResultProjection } from '../domain/unified-result'; -export const REGULATORY_MAPPING_PROMPT_VERSION = '1.2.0'; +// 1.3.0 (user decision 2026-10-09): each ISO candidate carries its full ISO/IEC 27002 control text, not only the matched passages. +export const REGULATORY_MAPPING_PROMPT_VERSION = '1.3.0'; export const REGULATORY_MAPPING_OUTPUT_SCHEMA_VERSION = 1 as const; -export const REGULATORY_RETRIEVAL_VERSION = 'dense-control-grouping-1.1.0'; +// Hybrid search (user decision 2026-10-09): dense English and German queries plus German keywords, fused by rank. +export const REGULATORY_RETRIEVAL_VERSION = 'hybrid-rrf-de-1.0.0'; /** Version of the GPO comparison context in the query; only GPO rules are regenerated when it changes. */ export const REGULATORY_GPO_QUERY_VERSION = 'gpo-baseline-context-1.1.0'; +/** + * Phase 13F, BSI stage 2 (user decision 2026-10-09): a rule is mapped to ISO controls and, separately, directly + * to BSI Kernel requirements. ISO objects carry no framework field, so every earlier object stays unchanged. + */ +export type RegulatoryFramework = 'iso27001' | 'bsi-kernel'; +export const BSI_MAPPING_PROMPT_VERSION = 'bsi-1.0.0'; +export const BSI_RETRIEVAL_VERSION = 'bsi-hybrid-rrf-de-1.0.0'; +export const BSI_CORPUS_ID = 'bsi-grundschutz-kernel'; +export const BSI_SOURCE_STANDARD = 'BSI Grundschutz++'; +const BSI_REQUIREMENT_ID = /^[A-Z]{2,5}(?:\.\d+)+$/; +// Since prompt 1.3.0 an ISO candidate carries every passage of its control text; runs before keep at most three. +const MAXIMUM_CANDIDATE_EVIDENCE = 24; +const MAXIMUM_CANDIDATES: Readonly> = { iso27001: 8, 'bsi-kernel': 16 }; + +export function mappingFramework(item: { framework?: RegulatoryFramework }): RegulatoryFramework { + return item.framework ?? 'iso27001'; +} + +/** The framework field an object carries: none for ISO, so ISO objects keep their earlier shape. */ +export function frameworkField(framework: RegulatoryFramework): { framework?: 'bsi-kernel' } { + return framework === 'bsi-kernel' ? { framework } : {}; +} + export type RegulatoryMappingRunStatus = 'queued' | 'running' | 'succeeded' | 'failed'; export type RegulatoryMappingRelationship = 'primary' | 'supporting'; export type RegulatoryMappingConfidence = 'low' | 'medium' | 'high'; @@ -40,6 +66,8 @@ export interface RegulatoryMappingBatchCounts { export interface RegulatoryMappingBatchRun extends RegulatoryMappingBatchScope { version: 1; id: string; + /** Absent for an ISO batch. */ + framework?: 'bsi-kernel'; projectId: string; scopeKey: string; filterHash: string; @@ -67,16 +95,18 @@ export interface RegulatoryMappingBatchResponse { export interface RegulatoryMappingBatchCreateRequest extends RegulatoryMappingBatchScope { idempotencyKey: string; + framework?: RegulatoryFramework; } export interface RegulatoryMappingEvidence { chunkId: string; textSha256: string; sourceSha256: string; - sourceStandard: 'ISO/IEC 27001' | 'ISO/IEC 27002'; + sourceStandard: 'ISO/IEC 27001' | 'ISO/IEC 27002' | typeof BSI_SOURCE_STANDARD; sectionKind: RegulatorySectionKind; - pageStart: number; - pageEnd: number; + /** ISO evidence only: the BSI catalog is one structured file without pages. */ + pageStart?: number; + pageEnd?: number; // Full ISO clause text. Required in the persisted run (the model reads it to reason), // but omitted from the API response projection so the copyrighted text never reaches the browser. text?: string; @@ -114,6 +144,8 @@ export interface RegulatoryMappingResult { export interface RegulatoryMappingRun { version: 1; id: string; + /** Absent for an ISO mapping. */ + framework?: 'bsi-kernel'; ruleKey: string; sourceTool: string; sourceRuleId?: string; @@ -150,6 +182,8 @@ export interface RegulatoryMappingRun { export interface RegulatoryMappingDecision { version: 1; id: string; + /** Absent for an ISO decision. */ + framework?: 'bsi-kernel'; ruleKey: string; sourceTool: string; sourceRuleId?: string; @@ -209,6 +243,8 @@ export const MAPPING_HISTORY_LIMIT = 50; export interface RegulatoryMappingHistoryEntry { id: string; kind: 'batch' | 'single'; + /** ISO controls or direct BSI requirements (user decision 2026-10-09); absent means ISO. */ + framework?: RegulatoryFramework; /** Where the mapping was started, relative to the shown timeslot; `unknown` for a single mapping without a recorded origin. */ scope: 'timeslot' | 'other-timeslot' | 'other-client' | 'unknown'; /** Automatic only for batches the auto-mapping watcher started. */ @@ -262,6 +298,8 @@ export interface RegulatoryMappingResponse { version: 1; projectId: string; resultProjectionId: string; + /** Absent means ISO. */ + framework?: RegulatoryFramework; generationEnabled: boolean; state: 'empty' | RegulatoryMappingRunStatus | 'unsupported'; disabledReason?: string; @@ -275,6 +313,8 @@ export interface RegulatoryMappingResponse { earlierControls?: RegulatoryEarlierControl[]; /** BSI Kernel requirements derived from this rule's accepted controls; absent when none is derived. */ bsi?: BsiOverview; + /** ISO <-> BSI consistency of this rule; only on a BSI response and only once a BSI requirement is accepted. */ + consistency?: { state: BsiConsistencyState; reviewed: boolean }; } export interface RegulatoryEarlierControl { @@ -285,12 +325,14 @@ export interface RegulatoryEarlierControl { export interface RegulatoryMappingCreateRequest { resultProjectionId: string; + framework?: RegulatoryFramework; /** The timeslot the Finding Reader shows; the run keeps it only as hashes, for the mapping history scope. */ origin?: { assessmentIds: string[]; scanRunIds: string[] }; } export interface RegulatoryMappingDecisionCreateRequest { resultProjectionId: string; + framework?: RegulatoryFramework; controlId: string; decision: RegulatoryMappingDecisionValue; comment?: string; @@ -365,18 +407,25 @@ export function regulatoryMappingRunValidationErrors(run: RegulatoryMappingRun): if (!/^[a-f0-9]{64}$/.test(run.queryHash)) errors.push('Regulatory mapping queryHash must be SHA-256.'); const candidatesRequired = run.status === 'running' || run.status === 'succeeded'; if ((!run.sourceHashes.length && candidatesRequired) || run.sourceHashes.some(value => !/^[a-f0-9]{64}$/.test(value))) errors.push('Regulatory mapping sourceHashes must contain SHA-256 digests.'); - if (!Array.isArray(run.candidates) || (candidatesRequired && run.candidates.length < 1) || run.candidates.length > 8) errors.push('Regulatory mapping candidates must contain one to eight controls when processing has started.'); + if (run.framework !== undefined && run.framework !== 'bsi-kernel') errors.push('Regulatory mapping framework is invalid.'); + const bsi = mappingFramework(run) === 'bsi-kernel'; + const maximum = MAXIMUM_CANDIDATES[mappingFramework(run)]; + if (!Array.isArray(run.candidates) || (candidatesRequired && run.candidates.length < 1) || run.candidates.length > maximum) errors.push(`Regulatory mapping candidates must contain one to ${maximum} controls when processing has started.`); else { const controls = new Set(); run.candidates.forEach(candidate => { - if (!/^[5-8]\.\d{1,2}$/.test(candidate.controlId) || candidate.annexAControlId !== `A.${candidate.controlId}`) errors.push('Regulatory mapping candidate control identity is invalid.'); + const identity = bsi ? BSI_REQUIREMENT_ID.test(candidate.controlId) && candidate.annexAControlId === candidate.controlId + : /^[5-8]\.\d{1,2}$/.test(candidate.controlId) && candidate.annexAControlId === `A.${candidate.controlId}`; + if (!identity) errors.push('Regulatory mapping candidate control identity is invalid.'); if (controls.has(candidate.controlId)) errors.push(`Regulatory mapping candidate ${candidate.controlId} is duplicated.`); controls.add(candidate.controlId); if (!candidate.title.trim() || candidate.title.length > 500 || !Number.isFinite(candidate.score)) errors.push(`Regulatory mapping candidate ${candidate.controlId} metadata is invalid.`); - if (!candidate.evidence.length || candidate.evidence.length > 3) errors.push(`Regulatory mapping candidate ${candidate.controlId} must contain one to three evidence chunks.`); + if (!candidate.evidence.length || candidate.evidence.length > MAXIMUM_CANDIDATE_EVIDENCE) errors.push(`Regulatory mapping candidate ${candidate.controlId} must contain one to ${MAXIMUM_CANDIDATE_EVIDENCE} evidence chunks.`); candidate.evidence.forEach(evidence => { if (!evidence.chunkId.trim() || !/^[a-f0-9]{64}$/.test(evidence.textSha256) || !/^[a-f0-9]{64}$/.test(evidence.sourceSha256) - || !Number.isInteger(evidence.pageStart) || !Number.isInteger(evidence.pageEnd) || evidence.pageStart < 1 || evidence.pageEnd < evidence.pageStart + || (bsi ? evidence.sourceStandard !== BSI_SOURCE_STANDARD || evidence.pageStart !== undefined || evidence.pageEnd !== undefined + : evidence.sourceStandard === BSI_SOURCE_STANDARD || !Number.isInteger(evidence.pageStart) || !Number.isInteger(evidence.pageEnd) + || evidence.pageStart! < 1 || evidence.pageEnd! < evidence.pageStart!) || typeof evidence.text !== 'string' || !evidence.text.trim() || evidence.text.length > 20_000) errors.push(`Regulatory mapping candidate ${candidate.controlId} evidence is invalid.`); }); }); @@ -446,6 +495,7 @@ export function regulatoryMappingBatchRunValidationErrors(run: RegulatoryMapping if (!/^[a-f0-9]{64}$/.test(run.filterHash)) errors.push('Regulatory mapping batch filterHash must be SHA-256.'); if (!/^[A-Za-z0-9._:-]{8,128}$/.test(run.idempotencyKey)) errors.push('Regulatory mapping batch idempotencyKey is invalid.'); if (!['queued', 'running', 'completed', 'failed'].includes(run.status)) errors.push('Regulatory mapping batch status is invalid.'); + if (run.framework !== undefined && run.framework !== 'bsi-kernel') errors.push('Regulatory mapping batch framework is invalid.'); const normalized = normalizeRegulatoryMappingBatchScope(run); if (!normalized.assessmentIds.length && !normalized.scanRunIds.length) errors.push('Regulatory mapping batch requires a selected timeslot.'); const counts = Object.values(run.counts); @@ -472,6 +522,7 @@ export function regulatoryMappingDecisionValidationErrors(decision: RegulatoryMa } if (decision.sourceRuleId !== undefined && (!decision.sourceRuleId.trim() || decision.sourceRuleId.length > 200)) errors.push('Regulatory mapping decision sourceRuleId is invalid.'); if (!['accepted', 'rejected'].includes(decision.decision)) errors.push('Regulatory mapping decision is invalid.'); + if (decision.framework !== undefined && decision.framework !== 'bsi-kernel') errors.push('Regulatory mapping decision framework is invalid.'); if (decision.comment !== undefined && (decision.comment.length > 1_000 || /<\/?[a-z][^>]*>/i.test(decision.comment))) errors.push('Regulatory mapping decision comment must be plain text with at most 1000 characters.'); return [...new Set(errors)]; } diff --git a/src/compliance/regulatory-query.ts b/src/compliance/regulatory-query.ts index 99f73ba..94b279e 100644 --- a/src/compliance/regulatory-query.ts +++ b/src/compliance/regulatory-query.ts @@ -17,7 +17,19 @@ const publicTools: Record = { gpos: 'Microsoft Group Policy', maester: 'Maester', pingcastle: 'PingCastle', - 'purple-knight': 'Semperis Purple Knight' + 'purple-knight': 'Semperis Purple Knight', + // The Result DB names Purple Knight by its assessed scope (result-db-disclosure). + 'purple-knight-ad': 'Semperis Purple Knight for Active Directory', + 'purple-knight-entra': 'Semperis Purple Knight for Entra ID', + 'purple-knight-combined': 'Semperis Purple Knight for Active Directory and Entra ID' +}; + +/** Public indicator IDs of tools without a bundled rule catalog, recognized by their published form only. */ +const publicRuleIdPatterns: Record = { + 'purple-knight': /^SI\d{6}$/, + 'purple-knight-ad': /^SI\d{6}$/, + 'purple-knight-entra': /^SI\d{6}$/, + 'purple-knight-combined': /^SI\d{6}$/ }; const publicRuleIds: Record = { @@ -157,6 +169,7 @@ function policyContext(item: UnifiedResultProjection, title: string): string[] { function recognizedRuleId(toolId: string, value: string | undefined): string | undefined { const candidate = value?.trim(); if (!candidate) return undefined; + if (publicRuleIdPatterns[toolId]?.test(candidate)) return candidate; return publicRuleIds[toolId]?.find(id => id.toLowerCase() === candidate.toLowerCase()); } diff --git a/src/compliance/regulatory-rag.ts b/src/compliance/regulatory-rag.ts index 5047a9d..2e3467b 100644 --- a/src/compliance/regulatory-rag.ts +++ b/src/compliance/regulatory-rag.ts @@ -112,6 +112,18 @@ export interface RegulatoryFindingQuery { publicTool?: string; publicRuleId?: string; publicTopics: string[]; + /** The German restatement the hybrid search used, recorded on the run for provenance; absent before it existed or when it failed. */ + expansion?: RegulatoryQueryExpansion; +} + +/** A German restatement of the public query by the model, for the German norm texts (user decision 2026-10-09). */ +export interface RegulatoryQueryExpansion { + language: 'de'; + version: string; + query: string; + terms: string[]; + /** The ranked lists fused for this run, for example `dense-en`, `dense-de`, `keyword-de`. */ + searches: string[]; } export interface RegulatoryReranker { diff --git a/src/compliance/ui/BsiCatalogLibrary.tsx b/src/compliance/ui/BsiCatalogLibrary.tsx index 9d1959f..fdb7a47 100644 --- a/src/compliance/ui/BsiCatalogLibrary.tsx +++ b/src/compliance/ui/BsiCatalogLibrary.tsx @@ -1,10 +1,11 @@ -import { useEffect, useState } from 'react'; +import { useCallback, useEffect, useState } from 'react'; import { AssessorApiClient } from '../../api/assessor-api-client'; import { useEnginePreferences } from '../../app/engine-preferences'; -import type { BsiCatalogState } from '../bsi-catalog-management'; +import type { BsiCatalogState, BsiSearchIndexStatus } from '../bsi-catalog-management'; import { markRegulatoryDecisionSaved } from './decision-revision'; const api = new AssessorApiClient(); +const INDEX_POLL_MS = 2_000; /** Data Hub BSI catalog tile, like the MITRE matrix tile; a switched-off Compliance Engine renders and requests nothing. */ export function BsiCatalogLibrary() { @@ -12,12 +13,27 @@ export function BsiCatalogLibrary() { } const day = (version: string) => version.slice(0, 10); +const number = (value: number) => value.toLocaleString('en-US'); + +function indexLabel(index: BsiSearchIndexStatus): string { + switch (index.state) { + case 'built': return `Built · ${number(index.indexedChunks)} chunks for ${number(index.requirementCount)} requirements`; + case 'building': return index.progress ? `Building · ${number(index.progress.completed)} of ${number(index.progress.total)} chunks embedded` : 'Building …'; + case 'outdated': return 'Outdated · built for another catalog state'; + case 'not-built': return 'Not built · the first BSI mapping builds it otherwise'; + case 'failed': return 'Build failed'; + case 'unavailable': return 'Unavailable'; + } +} function BsiCatalogLibraryPanel() { const [status, setStatus] = useState(null); - const [busy, setBusy] = useState<'refresh' | 'reset' | null>(null); + const [index, setIndex] = useState(null); + const [busy, setBusy] = useState<'refresh' | 'reset' | 'index' | null>(null); const [feedback, setFeedback] = useState(null); const [error, setError] = useState(null); + // The search index status is optional: an older backend or a failure leaves only the catalog part. + const loadIndex = useCallback(async () => { try { setIndex(await api.getBsiSearchIndex()); } catch { setIndex(null); } }, []); useEffect(() => { let active = true; // Any failure stays inside this tile; Data Hub keeps working. @@ -25,8 +41,15 @@ function BsiCatalogLibraryPanel() { try { const value = await api.getBsiCatalog(); if (active) setStatus(value); } catch (reason) { if (active) setError(reason instanceof Error ? reason.message : 'The BSI catalog status could not be loaded.'); } })(); + void loadIndex(); return () => { active = false; }; - }, []); + }, [loadIndex]); + // While the index builds on the server, its progress is shown without a reload. + useEffect(() => { + if (index?.state !== 'building') return; + const timer = window.setTimeout(() => void loadIndex(), INDEX_POLL_MS); + return () => window.clearTimeout(timer); + }, [index, loadIndex]); const run = async (action: 'refresh' | 'reset') => { setBusy(action); setFeedback(null); setError(null); @@ -44,9 +67,21 @@ function BsiCatalogLibraryPanel() { } } catch (reason) { setError(reason instanceof Error ? reason.message : 'The BSI catalog could not be updated.'); - } finally { setBusy(null); } + } finally { + setBusy(null); + // Another catalog state needs its own search index. + void loadIndex(); + } }; + const buildIndex = async () => { + setBusy('index'); setFeedback(null); setError(null); + try { setIndex(await api.buildBsiSearchIndex(index?.state === 'built')); } + catch (reason) { setError(reason instanceof Error ? reason.message : 'The BSI search index could not be built.'); } + finally { setBusy(null); } + }; + + const building = index?.state === 'building'; return
@@ -57,16 +92,24 @@ function BsiCatalogLibraryPanel() { : 'Loading…'}
+ {index &&
+ Search index {indexLabel(index)} +
}
- - {status?.origin === 'bsi' && } + {index && index.state !== 'unavailable' && }
{feedback &&
{feedback}
} + {index?.state === 'failed' && index.error &&
{index.error}
} + {index?.state === 'unavailable' && index.disabledReason &&
{index.disabledReason}
} {error &&
{error}
}
; } diff --git a/src/compliance/ui/ClientIsoControls.tsx b/src/compliance/ui/ClientIsoControls.tsx index b7f5e2d..b5ce698 100644 --- a/src/compliance/ui/ClientIsoControls.tsx +++ b/src/compliance/ui/ClientIsoControls.tsx @@ -9,6 +9,7 @@ import { useRegulatoryDecisionRevision } from './decision-revision'; import { bsiRequirementFindings, type BsiOverview } from '../bsi-grundschutz'; import { ComplianceFrameworkSwitch, ComplianceOverview, type ComplianceFramework } from './ComplianceOverview'; import type { ComplianceMappingState } from './compliance-work-facets'; +import { MappingReviewTile } from './MappingReviewTile'; import './compliance.css'; const assessorApi = new AssessorApiClient(); @@ -181,6 +182,8 @@ export function ClientIsoControls({ projectId, assessmentIds = [], scanRunIds = {state.status === 'error' ?

ISO controls could not be loaded.

: tileFacts} + {/* User decision 2026-10-09: ISO <-> BSI consistency review beside the framework tiles. */} + {projectId && } ; @@ -219,27 +222,30 @@ export function resetClientComplianceCacheForTests(): void { overviewCache.clear const MAPPING_POLL_MS = 5_000; const triggerLabel = (entry: RegulatoryMappingHistoryEntry) => entry.kind === 'single' ? 'Single finding' : entry.trigger === 'automatic' ? 'Automatic' : 'Manual'; +const FRAMEWORK_LABEL = { iso27001: 'ISO 27001/27002', 'bsi-kernel': 'BSI Grundschutz++' } as const; const SCOPE_LABEL = { timeslot: 'This timeslot', 'other-timeslot': 'Other timeslot', 'other-client': 'Other client', unknown: 'Unknown' } as const; const STATUS_LABEL = { queued: 'Queued', running: 'Running', completed: 'Completed', failed: 'Failed' } as const; const dateTime = (value: string) => new Date(value).toLocaleString('en-GB', { day: 'numeric', month: 'short', year: 'numeric', hour: '2-digit', minute: '2-digit' }); /** - * The latest ISO mapping of any client and what started it; mappings are global per rule. The window lists - * the latest batches and single finding mappings with their scope and rule counts. + * The latest mapping of any client and what started it; mappings are global per rule. The window lists the latest + * ISO and direct BSI batches and single finding mappings with their framework, scope and rule counts (user decision + * 2026-10-09); the button names BSI, an ISO mapping keeps its earlier label. */ function LastMapping({ history }: { history: RegulatoryMappingHistoryEntry[] }) { const latest = history[0]; if (!latest) return No mapping yet; - const label = `Last mapping: ${dateTime(latest.createdAt)} · ${triggerLabel(latest)}` + const label = `Last mapping: ${dateTime(latest.createdAt)}${latest.framework === 'bsi-kernel' ? ' · BSI' : ''} · ${triggerLabel(latest)}` + (latest.scope === 'timeslot' ? '' : ` · ${SCOPE_LABEL[latest.scope]}`) + (latest.status === 'completed' ? '' : ` · ${STATUS_LABEL[latest.status]}`); - return + return
- + {history.map(entry => + diff --git a/src/compliance/ui/FindingBsiRequirements.tsx b/src/compliance/ui/FindingBsiRequirements.tsx index 6509e24..84f7c01 100644 --- a/src/compliance/ui/FindingBsiRequirements.tsx +++ b/src/compliance/ui/FindingBsiRequirements.tsx @@ -1,25 +1,51 @@ import { TriangleIcon } from '../../components/TriangleIcon'; +import type { BackendStatus } from '../../app/app-state'; import type { BsiOverview } from '../bsi-grundschutz'; +import type { BsiConsistencyState } from '../bsi-consistency'; +import { RegulatoryMappingDialog } from './RegulatoryMappingPanel'; +import { useRegulatoryMapping } from './use-regulatory-mapping'; import './compliance.css'; const byNumber = (left: string, right: string) => left.localeCompare(right, 'en', { numeric: true }); +export const CONSISTENCY_LABELS: Readonly> = { + consistent: 'Consistent with ISO', + inconsistent: 'Inconsistent with ISO', + 'no-reference': 'No ISO reference' +}; + /** - * Finding Reader section of the BSI Grundschutz++ requirements derived from this finding's accepted ISO - * controls through the BSI mapping (Phase 13F, stage 1). Derived only; it is no decision of its own. - * User decision 2026-10-08: it starts collapsed like `Context`, groups the requirements under their ISO - * control as one-line rows and shows a statement only when its requirement is opened. + * Finding Reader section of the BSI Grundschutz++ requirements of this finding. Stage 1 (user decision + * 2026-10-08): requirements derived from the accepted ISO controls through the BSI mapping; derived only, never a + * decision. It starts collapsed like `Context`, groups them under their ISO control as one-line rows and shows a + * statement only when its requirement is opened. Stage 2 (user decision 2026-10-09): the direct BSI mapping of the + * same rule, its accepted requirements and their consistency with the ISO side; a disagreement is only shown. */ -export function FindingBsiRequirements({ bsi }: { bsi?: BsiOverview }) { - if (!bsi?.requirements.length) return null; - const isoControls = [...new Set(bsi.requirements.flatMap(requirement => requirement.via.map(via => via.isoControlId)))].sort(byNumber); - const count = bsi.requirements.length; - return
+export function FindingBsiRequirements({ bsi, projectId, resultProjectionId, backendStatus }: { + bsi?: BsiOverview; + projectId: string; + /** Result DB row of the record; empty when the record has none, then only the derived requirements are shown. */ + resultProjectionId: string; + backendStatus: BackendStatus; +}) { + const direct = useRegulatoryMapping(projectId, resultProjectionId, backendStatus, 'bsi-kernel'); + const derived = bsi?.requirements ?? []; + // Only once the backend delivered the BSI mapping state of a mappable finding: when it can map or has mapped. + const showDirect = direct.available && Boolean(resultProjectionId) && Boolean(direct.response) + && (direct.response!.generationEnabled || Boolean(direct.response!.run) || direct.acceptedCount > 0); + if (!derived.length && !showDirect) return null; + const isoControls = [...new Set(derived.flatMap(requirement => requirement.via.map(via => via.isoControlId)))].sort(byNumber); + const derivedIds = new Set(derived.map(requirement => requirement.requirementId)); + const consistency = direct.response?.consistency; + const meta = [derived.length ? `${derived.length} ${derived.length === 1 ? 'requirement' : 'requirements'} derived from the accepted ISO controls` : '', direct.acceptedCount ? `${direct.acceptedCount} mapped directly` : '', + consistency ? CONSISTENCY_LABELS[consistency.state] + (consistency.reviewed ? ' (reviewed)' : '') : ''].filter(Boolean).join(' · '); + // Keyed by finding, so every finding starts with the section collapsed. + return

BSI requirements

- {count} {count === 1 ? 'requirement' : 'requirements'} derived from the accepted ISO controls + {meta || 'Not mapped yet'}
{isoControls.map(isoControlId => { - const reached = bsi.requirements.filter(requirement => requirement.via.some(via => via.isoControlId === isoControlId)); + const reached = derived.filter(requirement => requirement.via.some(via => via.isoControlId === isoControlId)); // One BSI mapping entry per ISO control, so its relationship holds for the whole group. const relationship = reached[0].via.find(via => via.isoControlId === isoControlId)!.relationship; return
@@ -32,9 +58,26 @@ export function FindingBsiRequirements({ bsi }: { bsi?: BsiOverview }) { )}
; })} -

+ {showDirect &&

+
+

Mapped directly{consistency ? ` · ${CONSISTENCY_LABELS[consistency.state]}${consistency.reviewed ? ' (reviewed)' : ''}` : ''}

+ +
+ {direct.acceptedControls.length > 0 &&
    {direct.acceptedControls.map(requirement =>
  • +
    {requirement.controlId}{requirement.title} + {derivedIds.has(requirement.controlId) && also derived}
    +
    • )}
} +
} + {bsi &&

Via BSI mapping {bsi.crosswalkVersion} · catalog {bsi.catalogVersion.slice(0, 10)} · Source: BSI Stand-der-Technik-Bibliothek, CC BY-SA 4.0 -

+

} + {!bsi && showDirect &&

Source: BSI Stand-der-Technik-Bibliothek, CC BY-SA 4.0

}
+ {direct.open && void direct.start()} onDecision={(candidate, decision, comment) => void direct.decide(candidate, decision, comment)} + onClose={() => direct.setOpen(false)}/>}
; } diff --git a/src/compliance/ui/IsoCorpusLibrary.tsx b/src/compliance/ui/IsoCorpusLibrary.tsx index 6873b87..88494b9 100644 --- a/src/compliance/ui/IsoCorpusLibrary.tsx +++ b/src/compliance/ui/IsoCorpusLibrary.tsx @@ -88,7 +88,7 @@ function IsoCorpusLibraryPanel() { onChange={event => void toggleAutoMapping(event.target.checked)}/> Map new scans automatically } - {status?.autoMapping &&

After each upload, deviations of rules without an ISO mapping are mapped like Map all ISO controls. Rules mapped before are reused.

} + {status?.autoMapping &&

After each upload, deviations of rules without an ISO mapping are mapped like Compliance mapping in the Result DB. Rules mapped before are reused.

}
void submit(event)}> diff --git a/src/compliance/ui/MappingReviewTile.tsx b/src/compliance/ui/MappingReviewTile.tsx new file mode 100644 index 0000000..90c0aa9 --- /dev/null +++ b/src/compliance/ui/MappingReviewTile.tsx @@ -0,0 +1,161 @@ +import { useEffect, useState } from 'react'; +import { AssessorApiClient } from '../../api/assessor-api-client'; +import type { BackendStatus } from '../../app/app-state'; +import { DashboardGlyph } from '../../features/dashboard/ClientTileParts'; +import { EnvironmentResultsButton } from '../../features/dashboard/EnvironmentResultsButton'; +import type { BsiConsistencyState, RegulatoryConsistencyItem, RegulatoryConsistencyResponse } from '../bsi-consistency'; +import { useRegulatoryDecisionRevision } from './decision-revision'; +import { CONSISTENCY_LABELS } from './FindingBsiRequirements'; +import { useRegulatoryMapping } from './use-regulatory-mapping'; +import './compliance.css'; + +const assessorApi = new AssessorApiClient(); +export const MAPPING_REVIEW_TITLE = 'ISO ↔ BSI review'; +const number = (value: number) => value.toLocaleString('en-US'); +const FILTERS: Array<{ state: BsiConsistencyState; label: string }> = [ + { state: 'inconsistent', label: 'Inconsistent' }, { state: 'no-reference', label: 'No ISO reference' }, { state: 'consistent', label: 'Consistent' }]; + +type ReviewState = { status: 'loading' } | { status: 'error' } | { status: 'ready'; review: RegulatoryConsistencyResponse }; + +/** + * Client Info `Compliance` tab tile of the ISO <-> BSI consistency review (Phase 13F, BSI stage 2, user decision + * 2026-10-09): per rule of the timeslot's deviations, the directly accepted BSI requirements against the ones the BSI + * mapping derives from the accepted ISO controls. The Results window lists the rules for a person to review; a + * disagreement is only shown and changes no decision by itself. + */ +export function MappingReviewTile({ projectId, assessmentIds, scanRunIds, backendStatus }: { + projectId: string; + assessmentIds: readonly string[]; + scanRunIds: readonly string[]; + backendStatus: BackendStatus; +}) { + const revision = useRegulatoryDecisionRevision(); + const [state, setState] = useState({ status: 'loading' }); + const [reload, setReload] = useState(0); + const scopeKey = JSON.stringify([projectId, assessmentIds, scanRunIds]); + useEffect(() => { + let active = true; + // A refresh keeps the shown values until the new review arrives; only the first load shows em dashes. + setState(current => current.status === 'ready' && current.review.projectId === projectId ? current : { status: 'loading' }); + assessorApi.getRegulatoryMappingConsistency(projectId, { assessmentIds: [...assessmentIds], scanRunIds: [...scanRunIds] }) + .then(review => { if (active) setState({ status: 'ready', review }); }) + .catch(() => { if (active) setState(current => current.status === 'ready' ? current : { status: 'error' }); }); + return () => { active = false; }; + }, [scopeKey, revision, reload]); // eslint-disable-line react-hooks/exhaustive-deps + + const ready = state.status === 'ready'; + const counts = ready ? state.review.counts : undefined; + const value = (amount: number | undefined) => amount === undefined ? '—' : number(amount); + const compared = counts ? counts.consistent + counts.inconsistent + counts.noReference : undefined; + return
+
+

{MAPPING_REVIEW_TITLE}

+ {ready && + setReload(current => current + 1)}/> + } +
+ {state.status === 'error' ?

The ISO ↔ BSI review could not be loaded.

+ :
+
Compared rules
{value(compared)}
+
Inconsistent
{value(counts?.inconsistent)}
+
No ISO reference
{value(counts?.noReference)}
+
Consistent
{value(counts?.consistent)}
+
Not compared
{value(counts?.notCompared)}
+
Reviewed
{value(counts?.reviewed)}
+
} +
; +} + +function ReviewList({ review, projectId, backendStatus, onChanged }: { + review: RegulatoryConsistencyResponse; projectId: string; backendStatus: BackendStatus; onChanged(): void; +}) { + const [filter, setFilter] = useState('inconsistent'); + const items = review.items.filter(item => item.state === filter); + return
+

Each rule compares the BSI requirements accepted in its direct BSI mapping with the requirements the BSI + mapping {review.crosswalkVersion} derives from its accepted ISO controls. Agreement shows consistency, not correctness. Nothing here changes a + decision until you accept or reject one.

+
+ {FILTERS.map(option => { + const amount = review.items.filter(item => item.state === option.state).length; + return ; + })} +
+ {review.counts.notCompared > 0 &&

{number(review.counts.notCompared)} {review.counts.notCompared === 1 ? 'rule has' : 'rules have'} an + accepted ISO control but no accepted BSI requirement yet. Start the BSI mapping batch in the Result DB to compare them.

} + {!items.length ?

No rule in this state.

+ :
{items.map(item => )}
} +

BSI texts: BSI Stand-der-Technik-Bibliothek, catalog {review.catalogVersion.slice(0, 10)}, CC BY-SA 4.0.

+
; +} + +function ReviewItem({ item, projectId, backendStatus, onChanged }: { + item: RegulatoryConsistencyItem; projectId: string; backendStatus: BackendStatus; onChanged(): void; +}) { + const [open, setOpen] = useState(false); + return
setOpen(event.currentTarget.open)}> + + {CONSISTENCY_LABELS[item.state]}{item.reviewed ? ' · reviewed' : ''} + {item.title} + {item.sourceTool}{item.sourceRuleId ? ` · ${item.sourceRuleId}` : ''} · {number(item.findingCount)} {item.findingCount === 1 ? 'finding' : 'findings'} + + {/* The decision hooks load only for an opened rule. */} + {open && } +
; +} + +function ReviewDetail({ item, projectId, backendStatus, onChanged }: { + item: RegulatoryConsistencyItem; projectId: string; backendStatus: BackendStatus; onChanged(): void; +}) { + const iso = useRegulatoryMapping(projectId, item.resultProjectionId, backendStatus); + const bsi = useRegulatoryMapping(projectId, item.resultProjectionId, backendStatus, 'bsi-kernel'); + const [comment, setComment] = useState(''); + const [saving, setSaving] = useState(false); + const [error, setError] = useState(''); + const busy = saving || iso.loading || bsi.loading; + const markReviewed = async () => { + setSaving(true); + try { + await assessorApi.markRegulatoryMappingConsistencyReviewed(projectId, { resultProjectionId: item.resultProjectionId, ...(comment.trim() ? { comment: comment.trim() } : {}) }); + setError(''); + onChanged(); + } catch (reason) { + setError(reason instanceof Error ? reason.message : 'The review could not be saved.'); + } finally { setSaving(false); } + }; + // A saved decision refreshes the tab overview and this list through the shared decision revision. + return
+
+

Accepted ISO controls · {number(item.derivedCount)} derived BSI {item.derivedCount === 1 ? 'requirement' : 'requirements'}

+ {item.iso.length ?
    {item.iso.map(control =>
  • + A.{control.controlId}{control.title ?? ''} + +
    • )}
:

No accepted ISO control.

} +
+
+

Mapped directly to BSI

+
    {item.direct.map(requirement =>
  • + {requirement.requirementId}{requirement.title ?? ''} + {requirement.derived ? 'also derived from ISO' : requirement.reachable ? 'not derived from the accepted ISO controls' : 'no ISO control maps here'} + +
    • )}
+
+ {item.reverse.length > 0 &&
+

ISO controls the BSI mapping points to

+
    {item.reverse.map(control =>
  • + A.{control.controlId}{control.title ?? ''} + via {control.requirementIds.join(', ')} + {control.decidable + ? + : not retrieved by the ISO mapping} +
    • )}
+
} + {(iso.error || bsi.error || error) &&

{error || iso.error || bsi.error}

} + {item.state !== 'consistent' && !item.reviewed &&
+ + +
} +
; +} diff --git a/src/compliance/ui/RegulatoryMappingBatchDialog.tsx b/src/compliance/ui/RegulatoryMappingBatchDialog.tsx index 2830461..9ae9887 100644 --- a/src/compliance/ui/RegulatoryMappingBatchDialog.tsx +++ b/src/compliance/ui/RegulatoryMappingBatchDialog.tsx @@ -4,7 +4,7 @@ import { useEnginePreferences } from '../../app/engine-preferences'; import { useBackendWorkActivity } from './use-backend-work-activity'; import { statusPollApi } from './status-poll-client'; import { Dialog } from '../../components/Dialog/Dialog'; -import type { RegulatoryMappingBatchResponse, RegulatoryMappingBatchScope } from '../regulatory-mapping'; +import type { RegulatoryFramework, RegulatoryMappingBatchResponse, RegulatoryMappingBatchScope } from '../regulatory-mapping'; const assessorApi = new AssessorApiClient(); @@ -18,7 +18,14 @@ export interface RegulatoryMappingBatchController { start(): Promise; } -export function useRegulatoryMappingBatch(requestedProjectId: string | undefined, scope: RegulatoryMappingBatchScope | undefined): RegulatoryMappingBatchController { +/** The ISO batch, with the direct BSI batch of the same filters beside it (Phase 13F, BSI stage 2). */ +export function useRegulatoryMappingBatch(requestedProjectId: string | undefined, scope: RegulatoryMappingBatchScope | undefined): RegulatoryMappingBatchController & { bsi: RegulatoryMappingBatchController } { + const iso = useFrameworkBatch(requestedProjectId, scope, 'iso27001'); + const bsi = useFrameworkBatch(requestedProjectId, scope, 'bsi-kernel'); + return { ...iso, bsi }; +} + +function useFrameworkBatch(requestedProjectId: string | undefined, scope: RegulatoryMappingBatchScope | undefined, framework: RegulatoryFramework): RegulatoryMappingBatchController { // A switched-off Compliance Engine requests no batch status and offers no batch action. const available = useEnginePreferences().compliance; const projectId = available ? requestedProjectId : undefined; @@ -27,7 +34,7 @@ export function useRegulatoryMappingBatch(requestedProjectId: string | undefined const [starting, setStarting] = useState(false); const [error, setError] = useState(); const scopeKey = JSON.stringify(scope ?? null); - const requestKey = `${projectId ?? ''}\u0000${scopeKey}`; + const requestKey = `${projectId ?? ''}\u0000${scopeKey}\u0000${framework}`; const activeRequestKey = useRef(requestKey); activeRequestKey.current = requestKey; @@ -37,13 +44,13 @@ export function useRegulatoryMappingBatch(requestedProjectId: string | undefined return; } try { - const next = await (poll ? statusPollApi : assessorApi).getRegulatoryMappingBatch(projectId, scope); + const next = await (poll ? statusPollApi : assessorApi).getRegulatoryMappingBatch(projectId, scope, framework); if (activeRequestKey.current !== requestKey) return; setResponse(next); setError(undefined); } catch (reason) { if (activeRequestKey.current !== requestKey) return; - setError(reason instanceof Error ? reason.message : 'ISO mapping batch status could not be loaded.'); + setError(reason instanceof Error ? reason.message : 'Compliance mapping batch status could not be loaded.'); } }, [projectId, requestKey, scopeKey]); // eslint-disable-line react-hooks/exhaustive-deps @@ -70,38 +77,59 @@ export function useRegulatoryMappingBatch(requestedProjectId: string | undefined try { const next = await assessorApi.startRegulatoryMappingBatch(projectId, { ...scope, - idempotencyKey: `regulatory-batch-${crypto.randomUUID()}` + idempotencyKey: `regulatory-batch-${crypto.randomUUID()}`, + ...(framework === 'iso27001' ? {} : { framework }) }); if (activeRequestKey.current !== requestKey) return; setResponse(next); } catch (reason) { if (activeRequestKey.current !== requestKey) return; - setError(reason instanceof Error ? reason.message : 'ISO mapping batch could not be started.'); + setError(reason instanceof Error ? reason.message : `${framework === 'bsi-kernel' ? 'BSI' : 'ISO'} mapping batch could not be started.`); } finally { if (activeRequestKey.current === requestKey) setStarting(false); } }, [projectId, requestKey, scopeKey]); // eslint-disable-line react-hooks/exhaustive-deps - useBackendWorkActivity('Mapping all ISO controls', !response || response.state === 'empty' ? 'idle' + useBackendWorkActivity(framework === 'bsi-kernel' ? 'Mapping all BSI requirements' : 'Mapping all ISO controls', !response || response.state === 'empty' ? 'idle' : response.state === 'completed' ? 'finished' : response.state); return { available, response, loading, starting, error, start }; } export function RegulatoryMappingBatchDialog({ controller, filteredResultCount, onClose }: { - controller: RegulatoryMappingBatchController; + controller: RegulatoryMappingBatchController & { bsi?: RegulatoryMappingBatchController }; filteredResultCount: number; onClose(): void; +}) { + return +
+

Generate ISO control proposals for all {filteredResultCount.toLocaleString('en-US')} results matching the current Result DB filters.

+

Pagination, “Load more,” and loaded-row limits do not restrict this batch. Proposals remain subject to manual Accept/Reject review.

+ + {/* User decision 2026-10-09: the direct BSI mapping of the same filters, compared with ISO in the Client Info Compliance tab. */} + {controller.bsi && <> +

BSI Grundschutz++ requirements

+

Map the same results directly to BSI Kernel requirements. The Compliance tab compares them with the BSI requirements derived from the accepted ISO controls.

+ + } +
+ +
+
+ ; +} + +function BatchSection({ controller, filteredResultCount, startLabel = 'Start mapping batch' }: { + controller: RegulatoryMappingBatchController; + filteredResultCount: number; + startLabel?: string; }) { const { response, loading, starting, error, start } = controller; const run = response?.run; const active = response && ['queued', 'running'].includes(response.state); const counts = run?.counts; - const actionLabel = run ? 'Run again for current filters' : 'Start mapping batch'; - return -
-

Generate ISO control proposals for all {filteredResultCount.toLocaleString('en-US')} results matching the current Result DB filters.

-

Pagination, “Load more,” and loaded-row limits do not restrict this batch. Proposals remain subject to manual Accept/Reject review.

+ const actionLabel = run ? 'Run again for current filters' : startLabel; + return <> {loading && !response &&

Loading batch status…

} {error &&

{error}

} {response && !response.generationEnabled &&

{response.disabledReason ?? 'Regulatory mapping is unavailable.'}

} @@ -120,14 +148,12 @@ export function RegulatoryMappingBatchDialog({ controller, filteredResultCount, {run.errorDetail &&

{run.errorDetail}

} } {active &&

Mapping is running on the server. You can close this dialog and return later.

} -
- - {response?.generationEnabled && } -
-
- ; + onClick={() => void start()}>{starting ? 'Starting…' : actionLabel} + } + ; } function BatchCount({ label, value }: { label: string; value: number }) { diff --git a/src/compliance/ui/RegulatoryMappingPanel.tsx b/src/compliance/ui/RegulatoryMappingPanel.tsx index a7c058a..74a4936 100644 --- a/src/compliance/ui/RegulatoryMappingPanel.tsx +++ b/src/compliance/ui/RegulatoryMappingPanel.tsx @@ -4,10 +4,17 @@ import './compliance.css'; import type { RegulatoryMappingCandidate, RegulatoryMappingDecision, + RegulatoryFramework, RegulatoryMappingProposal, RegulatoryMappingResponse } from '../regulatory-mapping'; +// User decision 2026-10-09: the direct BSI mapping uses the same window with its own labels. +const LABELS = { + iso27001: { title: 'ISO control mapping', items: 'controls', evaluating: 'Retrieving and evaluating ISO controls…' }, + 'bsi-kernel': { title: 'BSI requirement mapping', items: 'requirements', evaluating: 'Retrieving and evaluating BSI requirements…' } +} as const; + interface Props { response?: RegulatoryMappingResponse; loading: boolean; @@ -16,16 +23,18 @@ interface Props { onDecision(control: Pick, decision: 'accepted' | 'rejected', comment?: string): void; /** Inside the window the dialog title already names the mapping. */ embedded?: boolean; + framework?: RegulatoryFramework; } -/** Finding Reader `Map ISO controls` window, opened like `Accepted Risk`. */ +/** Finding Reader `Map ISO controls` and `Map BSI requirements` window, opened like `Accepted Risk`. */ export function RegulatoryMappingDialog({ onClose, ...props }: Omit & { onClose(): void }) { - return + return ; } -export function RegulatoryMappingPanel({ response, loading, error, onStart, onDecision, embedded = false }: Props) { +export function RegulatoryMappingPanel({ response, loading, error, onStart, onDecision, embedded = false, framework = 'iso27001' }: Props) { + const labels = LABELS[framework]; const run = response?.run; const proposals = run?.result?.mappings ?? []; const proposalIds = new Set(proposals.map(item => item.controlId)); @@ -36,20 +45,20 @@ export function RegulatoryMappingPanel({ response, loading, error, onStart, onDe const otherCandidates = run?.candidates.filter(item => !proposalIds.has(item.controlId)) ?? []; const earlierControls = response?.earlierControls ?? []; - return
-
{!embedded &&

ISO control mapping

}

Proposals indicate relevance to this finding, not proof of compliance.

+ return
+
{!embedded &&

{labels.title}

}

Proposals indicate relevance to this finding, not proof of compliance.

{response?.generationEnabled && !['queued', 'running'].includes(response.state) && }
{loading && !response &&

Loading mapping state…

} {error &&

{error}

} {response && !response.generationEnabled &&

{response.disabledReason ?? 'Regulatory mapping is unavailable.'}

} - {response && ['queued', 'running'].includes(response.state) &&

{response.state === 'queued' ? 'Mapping queued…' : 'Retrieving and evaluating ISO controls…'}

} + {response && ['queued', 'running'].includes(response.state) &&

{response.state === 'queued' ? 'Mapping queued…' : labels.evaluating}

} {run?.status === 'failed' &&

Mapping stopped safely ({run.errorCode ?? 'unknown error'}). You can start a new run.{run.errorDetail ? <>
{run.errorDetail} : null}

} - {run?.status === 'succeeded' && run.result?.outcome !== 'mapped' &&

{run.result?.reason ?? (run.result?.outcome === 'insufficient_context' ? 'Insufficient public context.' : 'No relevant control was proposed.')}

} + {run?.status === 'succeeded' && run.result?.outcome !== 'mapped' &&

{run.result?.reason ?? (run.result?.outcome === 'insufficient_context' ? 'Insufficient public context.' : `No relevant ${framework === 'bsi-kernel' ? 'requirement' : 'control'} was proposed.`)}

} {proposedCandidates.length > 0 &&
{proposedCandidates.map(item => ) }
} - {otherCandidates.length > 0 &&
Other retrieved controls ({otherCandidates.length}) + {otherCandidates.length > 0 &&
Other retrieved {labels.items} ({otherCandidates.length})
{otherCandidates.map(candidate => )}
} {earlierControls.length > 0 &&
@@ -73,7 +82,7 @@ function ControlCandidate({ candidate, proposal, decisions, loading, onDecision {proposal?.rationale &&

{proposal.rationale}

}
Evidence ({candidate.evidence.length})
    {candidate.evidence.map(item =>
  • - {item.sourceStandard} · {item.sectionKind} · {item.pageStart === item.pageEnd ? `page ${item.pageStart}` : `pages ${item.pageStart}–${item.pageEnd}`} + {item.sourceStandard} · {item.pageStart === undefined ? 'catalog text' : <>{item.sectionKind} · {item.pageStart === item.pageEnd ? `page ${item.pageStart}` : `pages ${item.pageStart}–${item.pageEnd}`}}
    • )}
; diff --git a/src/compliance/ui/compliance.css b/src/compliance/ui/compliance.css index e160ba0..ddca5f9 100644 --- a/src/compliance/ui/compliance.css +++ b/src/compliance/ui/compliance.css @@ -20,6 +20,10 @@ .finding-bsi-requirements__item > summary:focus-visible { outline: var(--bw-emphasis) solid var(--focus-ring); outline-offset: var(--bw-emphasis); } .finding-bsi-requirements__item > summary strong { font-weight: var(--fw-semibold); font-variant-numeric: tabular-nums; white-space: nowrap; } .finding-bsi-requirements__statement { margin: var(--space-1) 0 var(--space-2) var(--space-5); font-size: var(--type-body-size); line-height: 1.5; } +/* Direct BSI mapping (stage 2): its own group below the derived ones, with the window action beside the heading. */ +.finding-bsi-requirements__direct-header { display: flex; flex-wrap: wrap; align-items: center; justify-content: space-between; gap: var(--space-2); } +div.finding-bsi-requirements__item { display: flex; flex-wrap: wrap; align-items: baseline; gap: var(--space-2); font-size: var(--type-body-size); line-height: 1.5; } +div.finding-bsi-requirements__item strong { font-weight: var(--fw-semibold); font-variant-numeric: tabular-nums; white-space: nowrap; } /* Client Info `Compliance` tab; its tiles use the environment tile geometry. */ .client-compliance-page { align-content: start; } @@ -156,3 +160,27 @@ button.compliance-footprint__cell:focus-visible { outline: var(--bw-emphasis) so .compliance-coverage__legend dd button { padding: 0; border: 0; background: transparent; color: inherit; font: inherit; font-variant-numeric: tabular-nums; cursor: pointer; } .compliance-coverage__legend dd button::after { content: ''; position: absolute; inset: 0; } .compliance-coverage__legend dd button:focus-visible { outline: none; } + +/* ISO <-> BSI review window (Phase 13F, BSI stage 2): rules as disclosure rows, no status colour. */ +.mapping-review { display: grid; gap: var(--space-3); min-width: 0; } +.mapping-review__intro, .mapping-review__note { margin: 0; font-size: var(--type-body-size); line-height: 1.5; } +.mapping-review__note { color: var(--text-secondary); } +.mapping-review__items { display: grid; min-width: 0; border-top: var(--border-hair) solid var(--tile-divider); } +.mapping-review__item { min-width: 0; border-bottom: var(--border-hair) solid var(--tile-divider); } +.mapping-review__item > summary { display: grid; grid-template-columns: minmax(150px, auto) minmax(0, 1fr); gap: var(--space-1) var(--space-3); padding: var(--space-2) 0; cursor: pointer; list-style: none; } +.mapping-review__item > summary::-webkit-details-marker { display: none; } +.mapping-review__item > summary:hover { background: var(--tile-surface-hover); } +.mapping-review__item > summary:focus-visible { outline: var(--bw-emphasis) solid var(--focus-ring); outline-offset: calc(var(--bw-emphasis) * -1); } +.mapping-review__state { font-size: var(--type-table-body-size); font-weight: var(--fw-semibold); } +.mapping-review__title { font-size: var(--type-body-size); overflow-wrap: break-word; min-width: 0; } +.mapping-review__meta { font-size: var(--type-table-body-size); color: var(--text-secondary); overflow-wrap: anywhere; } +.mapping-review__item > summary .mapping-review__meta { grid-column: 2; } +.mapping-review__detail { display: grid; gap: var(--space-3); padding: 0 0 var(--space-3); } +.mapping-review__side h4 { margin: 0 0 var(--space-1); font-size: var(--type-table-body-size); font-weight: var(--fw-semibold); } +.mapping-review__side p { margin: 0; font-size: var(--type-table-body-size); color: var(--text-secondary); } +.mapping-review__side ul { display: grid; gap: var(--space-1); margin: 0; padding: 0; list-style: none; } +.mapping-review__side li { display: flex; flex-wrap: wrap; align-items: center; gap: var(--space-2); font-size: var(--type-table-body-size); } +.mapping-review__side li strong { font-weight: var(--fw-semibold); font-variant-numeric: tabular-nums; white-space: nowrap; } +.mapping-review__side li > span:not(.mapping-review__meta) { flex: 1 1 200px; min-width: 0; overflow-wrap: break-word; } +.mapping-review__actions { display: flex; flex-wrap: wrap; align-items: end; gap: var(--space-2); } +.mapping-review__actions label { display: grid; flex: 1 1 260px; gap: var(--space-1); font-size: var(--type-table-body-size); } diff --git a/src/compliance/ui/mapping-run-tracker.ts b/src/compliance/ui/mapping-run-tracker.ts index b968604..eecc9d6 100644 --- a/src/compliance/ui/mapping-run-tracker.ts +++ b/src/compliance/ui/mapping-run-tracker.ts @@ -1,6 +1,6 @@ import { readEnginePreferences } from '../../app/engine-preferences'; import { beginActivity } from '../../platform/activity-store'; -import type { RegulatoryMappingResponse } from '../regulatory-mapping'; +import type { RegulatoryFramework, RegulatoryMappingResponse } from '../regulatory-mapping'; import { statusPollApi } from './status-poll-client'; const tracked = new Set(); @@ -8,7 +8,9 @@ const listeners = new Set<(key: string, response: RegulatoryMappingResponse) => const isActive = (response: RegulatoryMappingResponse) => response.state === 'queued' || response.state === 'running'; const wait = (milliseconds: number) => new Promise(resolve => setTimeout(resolve, milliseconds)); -export const mappingRunKey = (projectId: string, resultProjectionId: string) => `${projectId}\u0000${resultProjectionId}`; +// ISO keys keep their earlier form; a BSI mapping of the same finding is tracked separately. +export const mappingRunKey = (projectId: string, resultProjectionId: string, framework: RegulatoryFramework = 'iso27001') => + `${projectId}\u0000${resultProjectionId}${framework === 'iso27001' ? '' : `\u0000${framework}`}`; /** Views showing a mapping receive the tracker's status updates instead of polling themselves. */ export function subscribeRegulatoryMappingRuns(listener: (key: string, response: RegulatoryMappingResponse) => void): () => void { @@ -22,10 +24,12 @@ export function subscribeRegulatoryMappingRuns(listener: (key: string, response: * Engine off stops observing; the backend work itself is never cancelled. */ export function trackRegulatoryMappingRun(projectId: string, resultProjectionId: string, response: RegulatoryMappingResponse): void { - const key = mappingRunKey(projectId, resultProjectionId); + const framework = response.framework ?? 'iso27001'; + const key = mappingRunKey(projectId, resultProjectionId, framework); if (!isActive(response) || tracked.has(key)) return; tracked.add(key); - const activity = beginActivity('Compliance Engine', 'Mapping ISO controls', `compliance-mapping:${key}`); + const label = framework === 'bsi-kernel' ? 'Mapping BSI requirements' : 'Mapping ISO controls'; + const activity = beginActivity('Compliance Engine', label, `compliance-mapping:${key}`); if (response.state === 'queued') activity.queue(); void (async () => { let current = response; @@ -33,18 +37,18 @@ export function trackRegulatoryMappingRun(projectId: string, resultProjectionId: while (isActive(current)) { await wait(current.pollAfterMs ?? 1_500); if (!readEnginePreferences().compliance) { - activity.update('Mapping ISO controls continues in the backend'); + activity.update(`${label} continues in the backend`); activity.complete(); return; } - current = await statusPollApi.getRegulatoryMappings(projectId, resultProjectionId); + current = await statusPollApi.getRegulatoryMappings(projectId, resultProjectionId, framework); if (current.state === 'running') activity.run(); listeners.forEach(listener => listener(key, current)); } if (current.state === 'failed') activity.fail(); else activity.complete(); } catch { - activity.update('Mapping ISO controls: status could not be checked'); + activity.update(`${label}: status could not be checked`); activity.fail(); } finally { tracked.delete(key); diff --git a/src/compliance/ui/use-regulatory-mapping.ts b/src/compliance/ui/use-regulatory-mapping.ts index f91acb3..afeb091 100644 --- a/src/compliance/ui/use-regulatory-mapping.ts +++ b/src/compliance/ui/use-regulatory-mapping.ts @@ -10,6 +10,7 @@ import { acceptedControlIds, type RegulatoryMappingCandidate, type RegulatoryMappingDecisionValue, + type RegulatoryFramework, type RegulatoryMappingResponse } from '../regulatory-mapping'; @@ -26,7 +27,9 @@ export function useRegulatoryMapping( projectId: string, /** Result DB row of the record: a scanner finding occurrence or a GPO baseline comparison. */ resultProjectionId: string, - backendStatus: BackendStatus + backendStatus: BackendStatus, + /** ISO controls by default; `bsi-kernel` for the direct BSI mapping of the same finding (Phase 13F, stage 2). */ + framework: RegulatoryFramework = 'iso27001' ) { // A switched-off Compliance Engine shows no control, opens nothing and sends no request. const available = useEnginePreferences().compliance; @@ -41,7 +44,7 @@ export function useRegulatoryMapping( const [storedResponse, setStoredResponse] = useState<{ scope: { key: string }; value: RegulatoryMappingResponse }>(); const [loading, setLoading] = useState(false); const [error, setError] = useState(''); - const scopeKey = `${projectId}\u0000${resultProjectionId}\u0000${backendStatus}`; + const scopeKey = `${projectId}\u0000${resultProjectionId}\u0000${backendStatus}\u0000${framework}`; const activeScope = useRef({ key: scopeKey }); if (activeScope.current.key !== scopeKey) activeScope.current = { key: scopeKey }; const requestScope = activeScope.current; @@ -58,7 +61,7 @@ export function useRegulatoryMapping( if (!projectId || !resultProjectionId || backendStatus !== 'connected' || activeScope.current !== requestScope) return; setLoading(true); try { - const next = await assessorApi.getRegulatoryMappings(projectId, resultProjectionId); + const next = await assessorApi.getRegulatoryMappings(projectId, resultProjectionId, framework); trackRegulatoryMappingRun(projectId, resultProjectionId, next); if (activeScope.current !== requestScope) return; setStoredResponse({ scope: requestScope, value: next }); @@ -69,20 +72,21 @@ export function useRegulatoryMapping( } finally { if (activeScope.current === requestScope) setLoading(false); } - }, [backendStatus, projectId, requestScope, resultProjectionId]); + }, [backendStatus, framework, projectId, requestScope, resultProjectionId]); // Loaded with the finding, so the reader can name its accepted controls. A running mapping is polled // once by the run tracker, which keeps its activity entry until the backend ends it. useEffect(() => { if (available && !response && !loading && !error) void load(); }, [available, error, load, loading, response]); useEffect(() => subscribeRegulatoryMappingRuns((key, next) => { - if (key === mappingRunKey(projectId, resultProjectionId) && activeScope.current === requestScope) setStoredResponse({ scope: requestScope, value: next }); - }), [projectId, requestScope, resultProjectionId]); + if (key === mappingRunKey(projectId, resultProjectionId, framework) && activeScope.current === requestScope) setStoredResponse({ scope: requestScope, value: next }); + }), [framework, projectId, requestScope, resultProjectionId]); const start = useCallback(async () => { if (!projectId || !resultProjectionId || activeScope.current !== requestScope) return; setLoading(true); try { - const next = await assessorApi.generateRegulatoryMappings(projectId, { resultProjectionId, origin: origin.current }); + const next = await assessorApi.generateRegulatoryMappings(projectId, { resultProjectionId, origin: origin.current, + ...(framework === 'iso27001' ? {} : { framework }) }); trackRegulatoryMappingRun(projectId, resultProjectionId, next); if (activeScope.current !== requestScope) return; setStoredResponse({ scope: requestScope, value: next }); @@ -93,7 +97,7 @@ export function useRegulatoryMapping( } finally { if (activeScope.current === requestScope) setLoading(false); } - }, [projectId, requestScope, resultProjectionId]); + }, [framework, projectId, requestScope, resultProjectionId]); const decide = useCallback(async (candidate: Pick, decision: RegulatoryMappingDecisionValue, comment?: string) => { if (!projectId || !resultProjectionId || !response?.run || activeScope.current !== requestScope) return; @@ -102,6 +106,7 @@ export function useRegulatoryMapping( try { const next = await assessorApi.saveRegulatoryMappingDecision(projectId, { resultProjectionId, + ...(framework === 'iso27001' ? {} : { framework }), controlId: candidate.controlId, decision, ...(comment?.trim() ? { comment: comment.trim() } : {}), @@ -117,16 +122,17 @@ export function useRegulatoryMapping( } finally { if (activeScope.current === requestScope) setLoading(false); } - }, [projectId, requestScope, response, resultProjectionId]); + }, [framework, projectId, requestScope, response, resultProjectionId]); const acceptedControls = useMemo(() => available && response ? acceptedControlIds(response.decisions).map(controlId => { // A control decided in an earlier run keeps its title after the current run stopped retrieving it. const candidate = response.run?.candidates.find(item => item.controlId === controlId) ?? response.earlierControls?.find(item => item.controlId === controlId); - return { controlId, annexAControlId: candidate?.annexAControlId ?? `A.${controlId}`, ...(candidate?.title ? { title: candidate.title } : {}) }; + return { controlId, annexAControlId: candidate?.annexAControlId ?? (framework === 'iso27001' ? `A.${controlId}` : controlId), + ...(candidate?.title ? { title: candidate.title } : {}) }; }).sort((left, right) => left.controlId.localeCompare(right.controlId, 'en', { numeric: true })) - : [], [available, response]); + : [], [available, framework, response]); const localDisabledReason = !resultProjectionId ? 'This record has no stored finding or current GPO baseline comparison.' : backendStatus !== 'connected' ? 'The backend is not connected.' diff --git a/src/features/finding-reader/FindingReader.tsx b/src/features/finding-reader/FindingReader.tsx index 4e54f57..e59b448 100644 --- a/src/features/finding-reader/FindingReader.tsx +++ b/src/features/finding-reader/FindingReader.tsx @@ -579,8 +579,8 @@ export function FindingReader({ record, title, review: storedReview, previousPol : }
)} {/* Compliance Engine: controls accepted for this finding, read like Description. */} - {/* Compliance Engine: BSI requirements derived from those controls through the BSI mapping. */} - + {/* Compliance Engine: BSI requirements derived from those controls through the BSI mapping, and the direct BSI mapping. */} + {optionalFields.map(field =>
{highlightText(field.label, search)} {resolvedToolId === 'maester' && normalizeLabel(field.label) === 'test details' diff --git a/src/features/results-browser/ResultsBrowser.tsx b/src/features/results-browser/ResultsBrowser.tsx index 1f0ae52..91358a8 100644 --- a/src/features/results-browser/ResultsBrowser.tsx +++ b/src/features/results-browser/ResultsBrowser.tsx @@ -182,8 +182,8 @@ export function ResultsBrowser({ backendStatus, projectId, scanRunIds, assessmen {regulatoryBatch.available && } + title={regulatoryBatch.error ?? regulatoryBatch.response?.disabledReason ?? 'Generate ISO and BSI proposals for every result matching the current filters'} + onClick={() => setBatchDialogOpen(true)}>Compliance mapping}
StartedTriggerScopeStatusFindingsSent to modelReusedSucceededFailed
StartedFrameworkTriggerScopeStatusFindingsSent to modelReusedSucceededFailed
{dateTime(entry.createdAt)}{FRAMEWORK_LABEL[entry.framework ?? 'iso27001']} {triggerLabel(entry)} {SCOPE_LABEL[entry.scope]} {STATUS_LABEL[entry.status]}