From f83edc83acaff9477e9fee7e35dab56c93d05d7d Mon Sep 17 00:00:00 2001 From: Binh Date: Mon, 12 Oct 2026 13:30:53 +0200 Subject: [PATCH] feat(data-flow): show the Remediation Engine and its engine-owned schema - Engines & relationships: Remediation Engine, its tables and UI in a new right column; port, published Context, own data and UI relationships. - DB Structure reads the declared engine schema assessor_remediation and lists its tables schema-qualified. --- docs/IMPLEMENTATION_PLAN.md | 1 + .../postgres/postgres-assessor-repository.ts | 71 ++++++++++++------- src/features/db-structure/EngineDataFlow.tsx | 11 ++- src/styles/datahub.css | 9 ++- tests/features/db-structure-flow.test.tsx | 44 ++++++++---- 5 files changed, 94 insertions(+), 42 deletions(-) diff --git a/docs/IMPLEMENTATION_PLAN.md b/docs/IMPLEMENTATION_PLAN.md index ed16bd2..db25adf 100644 --- a/docs/IMPLEMENTATION_PLAN.md +++ b/docs/IMPLEMENTATION_PLAN.md @@ -4346,6 +4346,7 @@ Verifiziert 2026-09-09 (Phase 7): Bereichswechsel beschleunigt und Performance-V Verifiziert 2026-09-08 (Phase 7/13B): GPO-Quellenkorrektur fuer Work und Client Info. Aktuelle Evaluationen aus dem vorhandenen autoritativen Readmodell statt ergebnisabhaengiger Backupgruppierung; Import- und Auswertungszeit getrennt, Source-Scan-Provenienz auch auf GPO-only-Blaettern. 1.014 Tests bestanden, acht optionale PostgreSQL-Tests uebersprungen; Local-only-/Designpruefung und beide Builds erfolgreich. Dummy lesend bestaetigt: eine statt drei GPO-Quellenoptionen, 3.805 aktuelle Zeilen, elf historische Evaluationen unveraendert. Keine Persistenzmutation oder Neuanalyse. | Datum | Änderung | +| 2026-10-12 | Phase 7 Data Flow (Core-Freigabe Presentation, Core Tests, Store): Die Remediation Engine erscheint im Diagramm `Engines & relationships` als fuenfte Engine in einer eigenen rechten Spalte mit `Remediation tables` (Schema `assessor_remediation`) und `Work packages in the UI`; Beziehungen: Core-Read-only-Port -> Remediation (eigener Read-only-Pool), Context -> Remediation (`Published`, vom Core-Backend zusammengesetzt), Remediation <-> eigene Tabellen, Remediation -> UI. Kein Pfeil kreuzt einen anderen; die Portbeschreibung nennt den Read-only-Zugriff der Remediation Engine. Die Tabellenuebersicht liest zusaetzlich das deklarierte Engine-Schema `assessor_remediation`; dessen Tabellen erscheinen schemaqualifiziert (`assessor_remediation.package_snapshots`, Schicht Result) unter `Other tables` und im Woerterbuch, Spalten, Fremdschluessel und Zeilenzahlen werden je Schema aufgeloest. Am kanonischen Datastore lesend geprueft: 58 Public-Tabellen plus `assessor_remediation.package_snapshots` mit drei Spalten, keine Namensueberschneidung. Diagrammtests und neuer Sichtbarkeitstest gruen; einziger offener Test bleibt der DB-Structure-Radiustest (#6). Keine Versionserhoehung, weil der Bereich Platform nicht freigegeben war. | | 2026-10-12 | Importvertraege (Nutzerzustimmung nach Diff-Pruefung, Core-Freigabe Core Tests): Die geschuetzten Snapshots `pingcastle-html` und `pingcastle-xml` sind neu aufgenommen. Grund: die freigegebenen PingCastle-Aenderungen aus 2.19.1 (Profil 1.5.0 -> 1.7.0 im Verarbeitungsschluessel; XML-`DomainFQDN` als `domain.fqdn`/`Domain` mit Subjekt `domain-information`; daraus Vergleichsschluessel des Scores). Findings, Ergebnisse, Schweregrad, Evidenz und Scorewerte unveraendert; Vertragskennungen, Fixture-Hashes und Assertions unveraendert; Reimport-Empfehlung wie in 2.19.1 dokumentiert. Importgate gruen. | | 2026-10-12 | Integration (Core-Freigabe alle, Governance): Michaels Commit `614ac1b` (Remediation Engine, geschuetzte Importvertraege, Client-Info-Tab `Security`, Technical Report) ist die Basis; die 39 Commits seit `250c213` (AI/ISO/BSI-Compliance-Engine, Defender for IoT) sind darauf rebased (Branch `rebase/michael-backup`, `compliance-mapping` unveraendert). Beide Engines stehen nebeneinander in Engines-Popup (Core, Context, Report, Remediation, Compliance), Hook-Tabelle, Grenztest, Server und Work-Facetten (Work Package an der Stelle von Result Message, ISO/BSI-Facetten weiter direkt vor `Source: Scan`). Client Info nutzt Michaels `ClientInfoTabs` in der vereinbarten Reihenfolge Overview, Security, Microsoft Defender for IoT, Compliance, AI Overview; die drei Engine-/Defender-Tabs sind `lazy` und mounten nur ausgewaehlt (neuer Verhaltenstest), die alte Tab-Navigation und ihre CSS-Regeln entfallen. Finding Reader: `Map ISO controls` steht direkt vor Michaels fester Aktionsgruppe `AI search`/Bookmark/`Results`. Abgeleiteter Forest-Hinweis nach `client-info-facts.ts` portiert. Importgate: zwei additive synthetische Defender-Vertraege (Schwachstellen- und Alert-CSV, je 3 Findings); die Snapshots `pingcastle-html`/`pingcastle-xml` weichen durch die freigegebenen PingCastle-Aenderungen aus 2.19.1 ab (Profil 1.5.0 -> 1.7.0, XML-Domaenenfakten, Vergleichsschluessel) und bleiben bis zur Nutzerpruefung unveraendert, das Gate ist deshalb rot. Sechs weitere Testfehler (Context-Cluster-Reihenfolge, Pathfinder-Mock, Data-Hub-Donut- und DB-Structure-Layouttest) bestehen bereits auf `614ac1b` allein. Produktversion 2.25.0 (Minor: beide Funktionslinien vereint); Versionstests, beide Produktionsbuilds, Local-only- und Designpruefung bestanden. Vor dem ersten Serverstart legt die Remediation Engine das Schema `assessor_remediation` an; vorher Backup des kanonischen Datastores. | | 2026-10-06 | Phase 7: Data Hub Deviation/Passed now use the same assessed-percentage helper as Client Info; all six donuts state their population basis. 29 focused tests and synthetic Chrome Dark/Light checks passed; absolute counts, other population shares and assessment semantics retained. Client build, design and UI/API version checks passed. Nine out-of-scope report test failures, a report Local-only failure and report server type errors prevent full release verification; attempted 2.16.9 reverted to 2.16.8 pending successful production builds. | diff --git a/server/postgres/postgres-assessor-repository.ts b/server/postgres/postgres-assessor-repository.ts index d587594..d7ac856 100644 --- a/server/postgres/postgres-assessor-repository.ts +++ b/server/postgres/postgres-assessor-repository.ts @@ -395,10 +395,18 @@ const DB_STRUCTURE_LAYER_BY_TABLE: Record = { assessor_comparisons: 'governance', assessor_projects: 'core', assessor_assessments: 'core', - assessor_repository_state: 'core' + assessor_repository_state: 'core', + 'assessor_remediation.package_snapshots': 'result' }; const DB_STRUCTURE_EXCLUDED_TABLES = new Set(['assessor_schema_migrations']); +/** Engine-owned schemas shown beside the Core tables; their tables are named `schema.table`. */ +const DB_STRUCTURE_ENGINE_SCHEMAS = ['assessor_remediation']; +const DB_STRUCTURE_SCHEMAS = ['public', ...DB_STRUCTURE_ENGINE_SCHEMAS]; + +function dbStructureTableName(schema: string, table: string): string { + return schema === 'public' ? table : `${schema}.${table}`; +} function dbStructureLayer(tableName: string): DbStructureLayerId { return DB_STRUCTURE_LAYER_BY_TABLE[tableName] ?? 'other'; @@ -2823,72 +2831,83 @@ export class PostgresAssessorRepository implements AssessorRepository { } async getDatabaseStructure(): Promise { - const tablesResult = await this.pool.query<{ table_name: string }>(` - SELECT table_name FROM information_schema.tables - WHERE table_schema = 'public' AND table_type = 'BASE TABLE' - ORDER BY table_name - `); + const tablesResult = await this.pool.query<{ table_schema: string; table_name: string }>(` + SELECT table_schema, table_name FROM information_schema.tables + WHERE table_schema = ANY($1::text[]) AND table_type = 'BASE TABLE' + ORDER BY table_schema <> 'public', table_schema, table_name + `, [DB_STRUCTURE_SCHEMAS]); const tableNames = tablesResult.rows - .map(row => row.table_name) + .map(row => dbStructureTableName(row.table_schema, row.table_name)) .filter(name => !DB_STRUCTURE_EXCLUDED_TABLES.has(name)); if (tableNames.length === 0) { return { generatedAt: new Date().toISOString(), persistence: 'postgresql', tables: [] }; } - const columnsResult = await this.pool.query<{ table_name: string; column_name: string; data_type: string; is_nullable: string }>(` - SELECT table_name, column_name, data_type, is_nullable FROM information_schema.columns - WHERE table_schema = 'public' AND table_name = ANY($1::text[]) - ORDER BY table_name, ordinal_position - `, [tableNames]); + const columnsResult = await this.pool.query<{ table_schema: string; table_name: string; column_name: string; data_type: string; is_nullable: string }>(` + SELECT table_schema, table_name, column_name, data_type, is_nullable FROM information_schema.columns + WHERE table_schema = ANY($1::text[]) + ORDER BY table_schema, table_name, ordinal_position + `, [DB_STRUCTURE_SCHEMAS]); const foreignKeysResult = await this.pool.query<{ + table_schema: string; table_name: string; column_name: string; + referenced_table_schema: string; referenced_table_name: string; referenced_column_name: string; }>(` SELECT + source_namespace.nspname AS table_schema, source_table.relname AS table_name, source_column.attname AS column_name, + target_namespace.nspname AS referenced_table_schema, target_table.relname AS referenced_table_name, target_column.attname AS referenced_column_name FROM pg_constraint constraint_info JOIN pg_class source_table ON source_table.oid = constraint_info.conrelid + JOIN pg_namespace source_namespace ON source_namespace.oid = source_table.relnamespace JOIN pg_class target_table ON target_table.oid = constraint_info.confrelid + JOIN pg_namespace target_namespace ON target_namespace.oid = target_table.relnamespace JOIN LATERAL unnest(constraint_info.conkey, constraint_info.confkey) WITH ORDINALITY AS cols(source_attnum, target_attnum, ordinal) ON true JOIN pg_attribute source_column ON source_column.attrelid = source_table.oid AND source_column.attnum = cols.source_attnum JOIN pg_attribute target_column ON target_column.attrelid = target_table.oid AND target_column.attnum = cols.target_attnum WHERE constraint_info.contype = 'f' - AND source_table.relnamespace = 'public'::regnamespace - AND source_table.relname = ANY($1::text[]) - ORDER BY source_table.relname, cols.ordinal - `, [tableNames]); + AND source_namespace.nspname = ANY($1::text[]) + ORDER BY source_namespace.nspname, source_table.relname, cols.ordinal + `, [DB_STRUCTURE_SCHEMAS]); - const rowCountResult = await this.pool.query<{ relname: string; reltuples: string }>(` - SELECT relname, reltuples FROM pg_class - WHERE relkind = 'r' AND relname = ANY($1::text[]) - `, [tableNames]); + const rowCountResult = await this.pool.query<{ nspname: string; relname: string; reltuples: string }>(` + SELECT namespace.nspname, class.relname, class.reltuples FROM pg_class class + JOIN pg_namespace namespace ON namespace.oid = class.relnamespace + WHERE class.relkind = 'r' AND namespace.nspname = ANY($1::text[]) + `, [DB_STRUCTURE_SCHEMAS]); + const included = new Set(tableNames); const columnsByTable = new Map(); for (const row of columnsResult.rows) { - const columns = columnsByTable.get(row.table_name) ?? []; + const table = dbStructureTableName(row.table_schema, row.table_name); + if (!included.has(table)) continue; + const columns = columnsByTable.get(table) ?? []; columns.push({ name: row.column_name, dataType: row.data_type, nullable: row.is_nullable === 'YES' }); - columnsByTable.set(row.table_name, columns); + columnsByTable.set(table, columns); } const foreignKeysByTable = new Map(); for (const row of foreignKeysResult.rows) { - const foreignKeys = foreignKeysByTable.get(row.table_name) ?? []; + const table = dbStructureTableName(row.table_schema, row.table_name); + if (!included.has(table)) continue; + const foreignKeys = foreignKeysByTable.get(table) ?? []; foreignKeys.push({ column: row.column_name, - referencedTable: row.referenced_table_name, + referencedTable: dbStructureTableName(row.referenced_table_schema, row.referenced_table_name), referencedColumn: row.referenced_column_name }); - foreignKeysByTable.set(row.table_name, foreignKeys); + foreignKeysByTable.set(table, foreignKeys); } - const rowCountByTable = new Map(rowCountResult.rows.map(row => [row.relname, Math.max(0, Math.round(Number(row.reltuples)))])); + const rowCountByTable = new Map(rowCountResult.rows.map(row => [dbStructureTableName(row.nspname, row.relname), Math.max(0, Math.round(Number(row.reltuples)))])); const tables: DbStructureTable[] = tableNames.map(name => ({ name, diff --git a/src/features/db-structure/EngineDataFlow.tsx b/src/features/db-structure/EngineDataFlow.tsx index 20e3cac..afb1bc5 100644 --- a/src/features/db-structure/EngineDataFlow.tsx +++ b/src/features/db-structure/EngineDataFlow.tsx @@ -8,12 +8,15 @@ const nodes = [ { id: 'input', name: 'Source inputs', caption: 'Scanner reports · GPO backups · catalogs', description: 'Original scanner reports, GPO backups and reference catalogs enter the Core import and evaluation pipeline.' }, { id: 'core', engine: true, name: 'Core Engine', caption: 'Authoritative results · always active', description: 'All existing ASSESSOR functions: import, normalization, evaluation, scoring, client records and disclosure. Owns canonical PostgreSQL data. Works independently of optional engines.' }, { id: 'database', name: 'Canonical PostgreSQL', caption: 'Core-owned data · disclosed Result DB', description: 'Canonical local datastore. The Core owns persistence, including the stored and verified Result DB projection.' }, - { id: 'port', name: 'Core read-only port', caption: 'Scoped results · provenance', description: 'Declared read-only Core port. Optional engines read authoritative results without changing stored data. Context and Report Engine read through read-only database access; the Compliance Engine still reads through the running Core repository.' }, + { id: 'port', name: 'Core read-only port', caption: 'Scoped results · provenance', description: 'Declared read-only Core port. Optional engines read authoritative results without changing stored data. Context, Report and Remediation Engine read through read-only database access; the Compliance Engine still reads through the running Core repository.' }, { id: 'context', engine: true, name: 'Context Engine', caption: 'Derived context · optional', description: 'Reads findings and GPO verdicts through the read-only Core port and read-only database access. Produces versioned clusters, confidence and evidence. In-memory results; no own persistence or Core writes.' }, { id: 'report', engine: true, name: 'Report Engine', caption: 'Read-only reporting · optional', description: 'Reads Core results through the read-only Core port and a separate read-only PostgreSQL pool. Produces report projections, management and technical reports. Own file archive; no Core writes.' }, { id: 'ui', name: 'Core UI & exports', caption: 'Findings · scores · Result DB', description: 'Application views show authoritative Core findings, verdicts and scores. Result DB exports obey the Core disclosure policy.' }, { id: 'context-ui', name: 'Context in the UI', caption: 'Work · Client Info · Pathfinder', description: 'Published Context results appear through existing UI extension points. They never override technical results or client records.' }, { id: 'archive', name: 'Report outputs', caption: 'File archive · downloads', description: 'Generated reports are stored in the separate Report Engine file archive and offered for download; no report storage in Core tables.' }, + { id: 'remediation', engine: true, name: 'Remediation Engine', caption: 'Work packages · optional', description: 'Groups the deviating Work records of a timeslot into work packages: one Context cluster and one platform per package, at most eight tasks (fifteen for Windows host/GPO). Reads through the read-only Core port with its own read-only PostgreSQL pool and uses the published Context of the same scope. Package numbers are identifiers, never priorities; verdicts, scores, reviews and Result DB stay unchanged.' }, + { id: 'remediation-tables', name: 'Remediation tables', caption: 'Package snapshots', description: 'Derived package snapshots in the engine-owned schema assessor_remediation (table package_snapshots), keyed by client, timeslot, Core revision and engine, catalog and Context versions. Written through the engine pool; no columns in Core tables.' }, + { id: 'remediation-ui', name: 'Work packages in the UI', caption: 'Work · Finding Reader · Packages window', description: 'The Work Package filter, the Work Packages window, related-finding rings and the Finding Reader link Context Cluster → AP-xx appear through existing UI extension points. Packages carry no owners, deadlines or effort.' }, { id: 'compliance', engine: true, name: 'Compliance Engine', caption: 'Regulatory control mapping · optional', description: 'Maps findings and GPO baseline comparisons to regulatory controls; ISO/IEC 27001/27002 is the first framework. Reads Core results through the read-only Core port and writes only its own tables. Proposals never change verdicts, scores or client records.' }, { id: 'compliance-tables', name: 'Compliance tables', caption: 'Mappings · decisions · batches', description: 'Mapping runs, accept/reject decisions and batches in three engine-owned tables of the canonical PostgreSQL. Written through a separate small pool; engine writes never advance the Core revision.' }, { id: 'retrieval', name: 'Retrieval & model', caption: 'Qdrant index · LiteLLM', description: 'The rebuildable Qdrant index of the uploaded control corpus and the configured LiteLLM model. They receive privacy-safe finding text and corpus evidence, never assets or customer values.' }, @@ -29,6 +32,10 @@ const relationships = [ { from: 'context', to: 'context-ui', label: 'Context', detail: 'Published context results through existing UI extension points', path: 'M710 290 V160' }, { from: 'report', to: 'archive', label: 'Reports', detail: 'Generated reports; separate from Core database storage', path: 'M1000 520 V390' }, { from: 'context', to: 'report', label: 'Published', detail: 'Versioned Context publication for the Management Report, composed by the Core backend; neither engine imports the other', path: 'M820 340 H855 V490 H940 V520', planned: false }, + { from: 'port', to: 'remediation', label: 'Read only', detail: 'Deviating Work records of the selected client and timeslot through a separate read-only PostgreSQL pool', path: 'M790 620 V690 H1290 V390' }, + { from: 'context', to: 'remediation', label: 'Published', detail: 'Published Context clusters of the same scope, composed by the Core backend; without them no packages are built, and neither engine imports the other', path: 'M800 290 V250 H1150 V340 H1180' }, + { from: 'remediation', to: 'remediation-tables', label: 'Own data', detail: 'Package snapshots per scope and version in the engine schema assessor_remediation; never Core tables', path: 'M1290 290 V160', bidirectional: true }, + { from: 'remediation', to: 'remediation-ui', label: 'Packages', detail: 'Work packages and their tasks through existing UI extension points', path: 'M1400 340 H1430 V800 H1400' }, { from: 'port', to: 'compliance', label: 'Read only', detail: 'Single Result DB row by client, filtered Result DB pages, client, assessment and scan-run lists', path: 'M710 620 V750' }, { from: 'compliance', to: 'compliance-tables', label: 'Own data', detail: 'Mapping runs, decisions and batches; never Core tables', path: 'M600 800 H530', bidirectional: true }, { from: 'compliance', to: 'retrieval', label: 'Retrieval', detail: 'Privacy-safe query, retrieved control evidence and model proposals', path: 'M820 800 H890', bidirectional: true }, @@ -46,7 +53,7 @@ export function EngineDataFlow() {

Engines & relationships

Solid: implemented · Dashed: planned
-