feat(comparison): compare scanners, GPO settings and score basis across timeslots

Separate new from regressed scanner deviations and accept a timeslot without
scanner runs on one side. Pair rolled-up GPO Work verdicts by scope and
technical identity with a newer backup set and the same baselines; a changed
value of a still deviating setting stays neutral. compareTimeslots yields the
change categories for the UI and a score delta over comparable sources only.
This commit is contained in:
Binh 2026-10-13 11:52:56 +02:00
parent 1fc4d0aa14
commit 24c0f959aa
8 changed files with 719 additions and 16 deletions

View File

@ -2,7 +2,7 @@
## Comparison Engine: decision and staging (planned 2026-10-13, Phase 8 timeslot comparison as an engine) ## Comparison Engine: decision and staging (planned 2026-10-13, Phase 8 timeslot comparison as an engine)
Status: **steps 1 and 2 done and verified (2026-10-13), steps 3-4 open**. Explicit user decision 2026-10-13: the timeslot comparison of [TIMESLOT_COMPARISON_PLAN.md](TIMESLOT_COMPARISON_PLAN.md) (P1-P6) becomes a separately switchable **Comparison Engine** that owns its comparison logic. Its folders follow the other engines without an `-engine` suffix: `src/comparison/`, `server/comparison/`, `tests/comparison/`. It reads the Core only through a declared read-only port and never writes, overrides or re-derives a Core verdict, score, review, risk acceptance or Result DB row. All contracts, the acceptance matrix and the conservative rules of the timeslot plan stay binding unchanged (object/scope identity, no `absent = resolved` without a declared profile negative-evidence contract, a missing source keeps its open deviations as unverified, score delta only over the shared population, UI is a consumer, never its own matcher). Status: **steps 1, 2 and 3a done and verified (2026-10-13), steps 3b and 4 open**. Explicit user decision 2026-10-13: the timeslot comparison of [TIMESLOT_COMPARISON_PLAN.md](TIMESLOT_COMPARISON_PLAN.md) (P1-P6) becomes a separately switchable **Comparison Engine** that owns its comparison logic. Its folders follow the other engines without an `-engine` suffix: `src/comparison/`, `server/comparison/`, `tests/comparison/`. It reads the Core only through a declared read-only port and never writes, overrides or re-derives a Core verdict, score, review, risk acceptance or Result DB row. All contracts, the acceptance matrix and the conservative rules of the timeslot plan stay binding unchanged (object/scope identity, no `absent = resolved` without a declared profile negative-evidence contract, a missing source keeps its open deviations as unverified, score delta only over the shared population, UI is a consumer, never its own matcher).
UI decision 2026-10-13 (mockups A and B in the local, git-ignored `artifacts/compare-engine/`; synthetic Timeslot A): entry is a Client Info tab `Compare` with A/B timeslot selection, change counts, Security Domain and source deltas, each count opening Work; Work receives a `Change` facet and change badges, and the Finding Reader a `Compare` section with A and B values side by side. A retained value change of a still deviating GPO setting is shown neutral (`Value changed`), neither better nor worse. Mockup C (better/worse bridge) is not selected. UI decision 2026-10-13 (mockups A and B in the local, git-ignored `artifacts/compare-engine/`; synthetic Timeslot A): entry is a Client Info tab `Compare` with A/B timeslot selection, change counts, Security Domain and source deltas, each count opening Work; Work receives a `Change` facet and change badges, and the Finding Reader a `Compare` section with A and B values side by side. A retained value change of a still deviating GPO setting is shown neutral (`Value changed`), neither better nor worse. Mockup C (better/worse bridge) is not selected.
@ -14,7 +14,9 @@ Staging and required Core releases:
Steps 1-2 (Core release Governance, Core Tests, Evaluation, Store, Presentation): the Phase 8 files moved unchanged to `src/evaluation/scan-run-comparison/` (test to `tests/evaluation/scan-run-comparison/`); only import paths changed in the three Core importers and two tests. `assessor_comparisons`, its routes and data are untouched (0 rows). `.claude/hooks/core-areas.mjs` registers `Comparison Engine` and drops `src/comparison/` from Evaluation; `AGENTS.md`, the boundary test and `docs/ONBOARDING.md` name the engine; new permanent invariance test `tests/comparison/comparison-engine-invariance.test.ts` (synthetic, real local repository: a comparison leaves export, revision and models unchanged; one verified fail-to-pass, one confirmed open). Verified: import gate 528/528 (twice); main suite 2,451 passed, 18 optional skipped, 1 failed: the pre-existing `db-structure-layout.test.ts` tile-radius check, unrelated (no CSS or DB Structure file changed). The first gate run hit the documented 5-second timeout of the combined Node import test once; it passed alone (757 ms) and in the rerun, limit unchanged. Both production builds pass (existing chunk-size warning only). No product version change: no visible behaviour or contract changed. Steps 1-2 (Core release Governance, Core Tests, Evaluation, Store, Presentation): the Phase 8 files moved unchanged to `src/evaluation/scan-run-comparison/` (test to `tests/evaluation/scan-run-comparison/`); only import paths changed in the three Core importers and two tests. `assessor_comparisons`, its routes and data are untouched (0 rows). `.claude/hooks/core-areas.mjs` registers `Comparison Engine` and drops `src/comparison/` from Evaluation; `AGENTS.md`, the boundary test and `docs/ONBOARDING.md` name the engine; new permanent invariance test `tests/comparison/comparison-engine-invariance.test.ts` (synthetic, real local repository: a comparison leaves export, revision and models unchanged; one verified fail-to-pass, one confirmed open). Verified: import gate 528/528 (twice); main suite 2,451 passed, 18 optional skipped, 1 failed: the pre-existing `db-structure-layout.test.ts` tile-radius check, unrelated (no CSS or DB Structure file changed). The first gate run hit the documented 5-second timeout of the combined Node import test once; it passed alone (757 ms) and in the rerun, limit unchanged. Both production builds pass (existing chunk-size warning only). No product version change: no visible behaviour or contract changed.
Change log 2026-10-13: recorded the Comparison Engine decision, naming and staging; relocated the Core Phase 8 scan-run comparison and registered the Comparison Engine with its invariance test. Step 3a (engine paths only, no Core release): `src/comparison/timeslot-comparison-engine.ts` (result 1.1.0) separates new from regressed scanner deviations and accepts a timeslot without scanner runs on one side; new `gpo-timeslot-comparison.ts` pairs rolled-up GPO Work verdicts across timeslots; new `timeslot-comparison-report.ts` (`compareTimeslots`, 1.0.0) yields the UI change categories (resolved, unchanged, value-changed, unverified, not-comparable, not-applicable, new, regressed; target-side deviations of a one-sided source are not comparable) and the A/B score basis over comparable sources only. Contract details in TIMESLOT_COMPARISON_PLAN.md P2/P4. Open for 3b: a read port that builds both timeslots' models and GPO Work results (a timeslot can hold several assessments; the GPO backup time comes from `backupCreatedAt`), Security Domain deltas, the backend route and the UI. 15 synthetic engine tests. Verified: import gate 528/528; main suite 2,462 passed, 18 optional skipped, 1 failed (the same pre-existing, unrelated `db-structure-layout.test.ts` tile-radius check); local-only and design-language checks pass; both production builds pass. No product version change: no visible behaviour.
Change log 2026-10-13: recorded the Comparison Engine decision, naming and staging; relocated the Core Phase 8 scan-run comparison and registered the Comparison Engine with its invariance test; step 3a engine comparison of scanners, GPO settings and score basis.
## Results cluster foreground correction (2026-10-13, Phase 7 Presentation, 2.25.6) ## Results cluster foreground correction (2026-10-13, Phase 7 Presentation, 2.25.6)

View File

@ -89,9 +89,9 @@ Die Paketstände beziehen sich ausschließlich auf nachgewiesene Ergebnisse dies
|---|---|---| |---|---|---|
| P0 · abgeschlossen | Synthetisches Bestandsaudit je Scanner/Format/GPO: Regelgranularität, bereits erhaltene Objekt-/Scopefelder, Quellenpfade, Vollständigkeitssignale und fehlende Informationen dokumentieren; notwendige Vertragsänderungen und Reimportbetroffenheit bestimmen | [TIMESLOT_COMPARISON_AUDIT.md](TIMESLOT_COMPARISON_AUDIT.md); keine echten Kundendaten; Fähigkeiten und Grenzen je unterstütztem Pfad dokumentiert | | P0 · abgeschlossen | Synthetisches Bestandsaudit je Scanner/Format/GPO: Regelgranularität, bereits erhaltene Objekt-/Scopefelder, Quellenpfade, Vollständigkeitssignale und fehlende Informationen dokumentieren; notwendige Vertragsänderungen und Reimportbetroffenheit bestimmen | [TIMESLOT_COMPARISON_AUDIT.md](TIMESLOT_COMPARISON_AUDIT.md); keine echten Kundendaten; Fähigkeiten und Grenzen je unterstütztem Pfad dokumentiert |
| P1 · in Arbeit | Versionierte Vergleichsidentität, Gegenprüfungs-/Diagnosevertrag und getrennte Ergebnis-/Verpflichtungszustände in der Domäne spezifizieren; Persistenz-/API-Entwurf und Übergangsregeln festlegen | Domänen- und Profilverträge sowie synthetische Scope-/Zeitfälle umgesetzt; Persistenz-/API- und Übergangsentwurf bleiben offen | | P1 · in Arbeit | Versionierte Vergleichsidentität, Gegenprüfungs-/Diagnosevertrag und getrennte Ergebnis-/Verpflichtungszustände in der Domäne spezifizieren; Persistenz-/API-Entwurf und Übergangsregeln festlegen | Domänen- und Profilverträge sowie synthetische Scope-/Zeitfälle umgesetzt; Persistenz-/API- und Übergangsentwurf bleiben offen |
| P2 · in Arbeit | Scanner-/GPO-Paarung anhand echter Quellreferenzen; vollständige Kompatibilitätsprüfungen; objektgenauer Lebenszyklus, Fail-zu-Pass, konservative Abwesenheit und einheitliche aktuelle Abweichungsmetriken | Sicherer Scannerpfad mit echter Tool-/Run-/File-Paarung, Processing-, Scope-, Reportzeit- und Diagnosegrenzen umgesetzt; GPO, Historie, Produktintegration und vollständige Matrix bleiben offen | | P2 · in Arbeit | Scanner-/GPO-Paarung anhand echter Quellreferenzen; vollständige Kompatibilitätsprüfungen; objektgenauer Lebenszyklus, Fail-zu-Pass, konservative Abwesenheit und einheitliche aktuelle Abweichungsmetriken | Sicherer Scannerpfad mit echter Tool-/Run-/File-Paarung, Processing-, Scope-, Reportzeit- und Diagnosegrenzen umgesetzt. 2026-10-13 in der Comparison Engine: Scanner trennen `neu` und `wieder aufgetreten` (genau ein belegtes Pass derselben Identität in A; mehrdeutige Historie behauptet nichts); einseitige Quellen sind nicht vergleichbar; GPO-Paarung über die unverändert gelesenen, wie in Work hochgerollten Verdikte je Scope und technischer Identität, nur mit neuerer Backupzeit und gleicher Baselinemenge; fehlende oder nicht entschiedene Einstellung bleibt ungeklärt; geänderter Istwert einer weiter abweichenden Einstellung ist neutral (`value-changed`). Historie, Produktintegration und vollständige Matrix bleiben offen |
| P3 | Verpflichtungsbezogene Fortschreibung über alle Zwischenstände, getrennt von der Score-Baseline; atomare/idempotente Ablage und API gemäß P1, Wiederladen und auditierter Scope-Ausstieg | P1/P2; nichts verschwindet bei Teilimport, Reload, Reimport oder mehrdeutiger Zuordnung | | P3 | Verpflichtungsbezogene Fortschreibung über alle Zwischenstände, getrennt von der Score-Baseline; atomare/idempotente Ablage und API gemäß P1, Wiederladen und auditierter Scope-Ausstieg | P1/P2; nichts verschwindet bei Teilimport, Reload, Reimport oder mehrdeutiger Zuordnung |
| P4 | Explizite A/B-Scorebasis, objekt-/scopebewusste Population und getrennte Freigaben für Delta/Gegenprüfung; native Scoregrenzen und Coverageausgabe | P1/P2/P3; keine Verbesserung durch Quellenauslassung, Objektwechsel oder historische Baselinevermischung | | P4 · begonnen | Explizite A/B-Scorebasis, objekt-/scopebewusste Population und getrennte Freigaben für Delta/Gegenprüfung; native Scoregrenzen und Coverageausgabe | P1/P2/P3; keine Verbesserung durch Quellenauslassung, Objektwechsel oder historische Baselinevermischung. 2026-10-13: Delta nur über Quellen mit vergleichbarem Paar in A und B (Scanner über `compareSecurityScores`, GPO als Pass-Rate über in beiden Timeslots entschiedene Einstellungen derselben Baselines); jede ausgeschlossene Quelle mit Grund. Security-Domain-Deltas, Coverageausgabe und P3-Abhängigkeit offen |
| P5 | A/B-Auswahl und dreiteiliger Bericht in der vorhandenen Oberfläche; Gründe, Messbasis, Quellen/Versionen und Belege pro Aussage; aktueller Workspace bleibt unverändert nutzbar | P2-P4 fachlich verifiziert; UI ist Konsument der Domänenlogik, kein eigener Matcher | | P5 | A/B-Auswahl und dreiteiliger Bericht in der vorhandenen Oberfläche; Gründe, Messbasis, Quellen/Versionen und Belege pro Aussage; aktueller Workspace bleibt unverändert nutzbar | P2-P4 fachlich verifiziert; UI ist Konsument der Domänenlogik, kein eigener Matcher |
| P6 | Gesamtabnahme, Dokumentationsabgleich und kontrollierte Aktivierung | Alle unten genannten Fälle; keine Freigabe allein wegen vorhandener Funktion oder grüner Buildausgabe | | P6 | Gesamtabnahme, Dokumentationsabgleich und kontrollierte Aktivierung | Alle unten genannten Fälle; keine Freigabe allein wegen vorhandener Funktion oder grüner Buildausgabe |

View File

@ -0,0 +1,164 @@
import type { RemediationEvidenceState } from '../domain/timeslot-comparison';
import type { PolicyWorkResult, PolicyWorkStatus } from '../evaluation/policy-work-results';
import { policyWorkVerdictPriority } from '../evaluation/policy-work-results';
import { canonicalJson } from '../shared/canonical-json';
export const GPO_TIMESLOT_COMPARISON_VERSION = '1.0.0' as const;
/** One timeslot's rolled-up GPO settings exactly as Work reads them, plus its backup observation time. */
export interface GpoTimeslotInput {
results: PolicyWorkResult[];
/** Latest backup creation time of the timeslot's GPO backups; never the import time. */
observedAt?: string;
}
type SettingVerdict = Exclude<PolicyWorkStatus, 'Conflicts' | 'Duplicates'>;
interface TimeslotSetting {
key: string;
scope: string;
identity: string;
verdict?: SettingVerdict;
baselineIds: string;
actualValues: string;
resultIds: string[];
ambiguous: boolean;
}
export interface GpoSettingDecision {
/** Cross-timeslot setting identity: scope and technical identity only, never a run or assessment. */
settingKey: string;
scope: string;
identity: string;
baselineVerdict: SettingVerdict;
targetVerdict?: SettingVerdict;
state: RemediationEvidenceState;
/** Retained configured value differs while the setting still deviates. Neutral, never better or worse. */
valueChanged: boolean;
reasons: string[];
baselineResultIds: string[];
targetResultIds: string[];
}
export interface GpoTargetChange {
settingKey: string;
baselineVerdict?: SettingVerdict;
targetResultIds: string[];
}
export interface GpoTimeslotComparison {
version: typeof GPO_TIMESLOT_COMPARISON_VERSION;
state: 'compared' | 'not-comparable';
reasons: string[];
decisions: GpoSettingDecision[];
newDeviations: GpoTargetChange[];
regressedDeviations: GpoTargetChange[];
/** Pass rate over settings decided (Pass or Fail) in both timeslots against the same baselines. */
sharedPassRate?: { settings: number; baseline: number; target: number };
}
const settingKey = (scope: string, identity: string) => JSON.stringify([scope, identity]);
/**
* Rolls each timeslot up like Work: one verdict per technical setting, chosen by the shared
* verdict priority. Conflict and duplicate entries are diagnostics and never a verdict. Several
* aggregation runs holding the same setting in one timeslot make it ambiguous.
*/
function rollUp(results: PolicyWorkResult[]): Map<string, TimeslotSetting> {
const byIdentity = new Map<string, PolicyWorkResult[]>();
for (const result of results) {
const key = settingKey(result.scope, result.identity);
byIdentity.set(key, [...(byIdentity.get(key) ?? []), result]);
}
const settings = new Map<string, TimeslotSetting>();
for (const [key, group] of byIdentity) {
const verdicts = group.filter((item): item is PolicyWorkResult & { status: SettingVerdict } =>
item.status !== 'Conflicts' && item.status !== 'Duplicates');
const verdict = verdicts.reduce<SettingVerdict | undefined>((best, item) =>
!best || policyWorkVerdictPriority[item.status] > policyWorkVerdictPriority[best] ? item.status : best, undefined);
const decisive = verdicts.filter(item => item.status === verdict);
settings.set(key, {
key, scope: group[0].scope, identity: group[0].identity, verdict,
baselineIds: canonicalJson([...new Set(verdicts.flatMap(item => item.comparison?.baselineId ?? []))].sort()),
actualValues: canonicalJson([...new Set(decisive.flatMap(item =>
item.comparison?.comparator.actual === undefined ? [] : [canonicalJson(item.comparison.comparator.actual)]))].sort()),
resultIds: group.map(item => item.id).sort(),
ambiguous: new Set(group.flatMap(item => item.settingKey ?? [])).size > 1
});
}
return settings;
}
const newer = (baseline?: string, target?: string) => Boolean(baseline && target
&& Number.isFinite(Date.parse(baseline)) && Number.isFinite(Date.parse(target)) && Date.parse(target) > Date.parse(baseline));
/**
* Conservative A/B comparison of rolled-up GPO settings. The authoritative verdicts are read as
* they are and never re-derived. Remediation needs the same setting with an explicit Pass against the
* same baselines in a newer backup set; an absent or undecided setting stays unverified, because a
* GPO backup set is no proof of a removed configuration.
*/
export function compareGpoTimeslots(baseline: GpoTimeslotInput, target: GpoTimeslotInput): GpoTimeslotComparison {
const version = GPO_TIMESLOT_COMPARISON_VERSION;
const baselineSettings = rollUp(baseline.results);
const targetSettings = rollUp(target.results);
const baselineFails = [...baselineSettings.values()].filter(item => item.verdict === 'Fail')
.sort((left, right) => left.key.localeCompare(right.key));
const sourceReasons = [
...(baselineSettings.size ? [] : ['baseline-source-missing']),
...(targetSettings.size ? [] : ['target-source-missing'])
];
const decision = (item: TimeslotSetting, state: RemediationEvidenceState, reasons: string[], other?: TimeslotSetting,
valueChanged = false): GpoSettingDecision => ({
settingKey: item.key, scope: item.scope, identity: item.identity, baselineVerdict: 'Fail',
...(other?.verdict ? { targetVerdict: other.verdict } : {}), state, valueChanged, reasons,
baselineResultIds: item.resultIds, targetResultIds: other?.resultIds ?? []
});
if (sourceReasons.length) return {
version, state: 'not-comparable', reasons: sourceReasons,
decisions: baselineFails.map(item => decision(item, 'not-comparable', sourceReasons)),
newDeviations: [], regressedDeviations: []
};
const timeVerified = newer(baseline.observedAt, target.observedAt);
const decisions = baselineFails.map(item => {
if (item.ambiguous) return decision(item, 'not-comparable', ['gpo-setting-ambiguous']);
const other = targetSettings.get(item.key);
if (!timeVerified) return decision(item, 'unverified-absent', ['newer-observation-not-verified'], other);
if (!other) return decision(item, 'unverified-absent', ['target-setting-absent']);
if (other.ambiguous) return decision(item, 'unverified-absent', ['gpo-setting-ambiguous'], other);
if (other.baselineIds !== item.baselineIds) return decision(item, 'not-comparable', ['baseline-set-changed'], other);
if (other.verdict === 'Fail') return decision(item, 'confirmed-open', ['same-setting-current-deviation'], other,
other.actualValues !== item.actualValues);
if (other.verdict === 'Pass') return decision(item, 'verified-remediated', ['same-setting-explicit-pass'], other);
return decision(item, 'unverified-absent', [`target-verdict-not-decisive:${other.verdict ?? 'none'}`], other);
});
const newDeviations: GpoTargetChange[] = [];
const regressedDeviations: GpoTargetChange[] = [];
if (timeVerified) {
for (const item of [...targetSettings.values()].sort((left, right) => left.key.localeCompare(right.key))) {
if (item.verdict !== 'Fail' || item.ambiguous) continue;
const before = baselineSettings.get(item.key);
if (before?.ambiguous || before?.verdict === 'Fail') continue;
const change = { settingKey: item.key, ...(before?.verdict ? { baselineVerdict: before.verdict } : {}), targetResultIds: item.resultIds };
// A regression needs an explicit Pass against the same baselines; undecided history claims nothing.
if (before?.verdict === 'Pass') { if (before.baselineIds === item.baselineIds) regressedDeviations.push(change); }
else if (!before?.verdict || before.verdict === 'Not configured' || before.verdict === 'No baseline rule') newDeviations.push(change);
}
}
const shared = [...baselineSettings.values()].flatMap(item => {
const other = targetSettings.get(item.key);
return other && !item.ambiguous && !other.ambiguous && item.baselineIds === other.baselineIds
&& (item.verdict === 'Pass' || item.verdict === 'Fail') && (other.verdict === 'Pass' || other.verdict === 'Fail')
? [{ before: item.verdict, after: other.verdict }] : [];
});
const rate = (values: string[]) => values.filter(value => value === 'Pass').length / values.length * 100;
return {
version, state: 'compared', reasons: timeVerified ? [] : ['newer-observation-not-verified'],
decisions, newDeviations, regressedDeviations,
...(timeVerified && shared.length ? { sharedPassRate: {
settings: shared.length, baseline: rate(shared.map(item => item.before)), target: rate(shared.map(item => item.after))
} } : {})
};
}

View File

@ -27,11 +27,14 @@ export interface TimeslotSourceComparison {
state: 'compared' | 'not-comparable'; state: 'compared' | 'not-comparable';
reasons: string[]; reasons: string[];
decisions: TimeslotOccurrenceDecision[]; decisions: TimeslotOccurrenceDecision[];
/** Current deviations whose identity was not a deviation and not a verified pass in A. */
newDeviationOccurrenceIds: string[]; newDeviationOccurrenceIds: string[];
/** Current deviations whose identity was a single verified pass in A. */
regressedDeviationOccurrenceIds: string[];
} }
export interface ProjectTimeslotEvidenceComparison { export interface ProjectTimeslotEvidenceComparison {
version: '1.0.0'; version: '1.1.0';
projectId: string; projectId: string;
baselineRunIds: string[]; baselineRunIds: string[];
targetRunIds: string[]; targetRunIds: string[];
@ -44,6 +47,7 @@ export interface ProjectTimeslotEvidenceComparison {
notComparable: number; notComparable: number;
notApplicable: number; notApplicable: number;
newDeviations: number; newDeviations: number;
regressedDeviations: number;
}; };
} }
@ -108,7 +112,8 @@ export function compareProjectTimeslotEvidence(
if (!models.length) throw new Error('Timeslot comparison requires project data.'); if (!models.length) throw new Error('Timeslot comparison requires project data.');
const projectId = models[0].project.id; const projectId = models[0].project.id;
if (models.some(model => model.project.id !== projectId)) throw new Error('Timeslots must belong to the same project.'); if (models.some(model => model.project.id !== projectId)) throw new Error('Timeslots must belong to the same project.');
if (!baselineRunIds.length || !targetRunIds.length) throw new Error('Both timeslots require scanner runs.'); // One side may lack scanner runs; its sources are then reported as not comparable, never resolved.
if (!baselineRunIds.length && !targetRunIds.length) throw new Error('At least one timeslot requires scanner runs.');
if (baselineRunIds.some(id => targetRunIds.includes(id))) throw new Error('Baseline and target timeslots must use different runs.'); if (baselineRunIds.some(id => targetRunIds.includes(id))) throw new Error('Baseline and target timeslots must use different runs.');
const runs = models.flatMap(model => model.scanRuns); const runs = models.flatMap(model => model.scanRuns);
@ -148,7 +153,8 @@ export function compareProjectTimeslotEvidence(
if (!baselineRun || !targetRun || !profile || profile.timeslotComparison.kind === 'unavailable') return { if (!baselineRun || !targetRun || !profile || profile.timeslotComparison.kind === 'unavailable') return {
toolId, baselineRunId: baselineRun?.id, targetRunId: targetRun?.id, toolId, baselineRunId: baselineRun?.id, targetRunId: targetRun?.id,
state: 'not-comparable', reasons: pairingReasons, state: 'not-comparable', reasons: pairingReasons,
decisions: unmatchedDecisions(baselineOccurrences, pairingReasons), newDeviationOccurrenceIds: [] decisions: unmatchedDecisions(baselineOccurrences, pairingReasons), newDeviationOccurrenceIds: [],
regressedDeviationOccurrenceIds: []
}; };
const definition = profile.timeslotComparison; const definition = profile.timeslotComparison;
@ -180,7 +186,8 @@ export function compareProjectTimeslotEvidence(
|| baselineTime.state !== 'verified' || targetTime.state !== 'verified') return { || baselineTime.state !== 'verified' || targetTime.state !== 'verified') return {
toolId, baselineRunId: baselineRun.id, targetRunId: targetRun.id, toolId, baselineRunId: baselineRun.id, targetRunId: targetRun.id,
state: 'not-comparable', reasons: [...new Set(sourceReasons)], state: 'not-comparable', reasons: [...new Set(sourceReasons)],
decisions: unmatchedDecisions(baselineOccurrences, [...new Set(sourceReasons)]), newDeviationOccurrenceIds: [] decisions: unmatchedDecisions(baselineOccurrences, [...new Set(sourceReasons)]), newDeviationOccurrenceIds: [],
regressedDeviationOccurrenceIds: []
}; };
const targetOccurrences = occurrences.filter(item => item.scanRunId === targetRun.id); const targetOccurrences = occurrences.filter(item => item.scanRunId === targetRun.id);
@ -225,20 +232,30 @@ export function compareProjectTimeslotEvidence(
}; };
}); });
const baselineKeys = new Set(baselineIdentities.flatMap(entry => entry.identity.state === 'verified' ? [entry.identity.identity.key] : [])); const baselineKeys = new Set(baselineIdentities.flatMap(entry => entry.identity.state === 'verified' ? [entry.identity.identity.key] : []));
const newDeviationOccurrenceIds = targetIdentities.flatMap(entry => { // A regression needs exactly one explicit pass of the same identity in A.
if (entry.identity.state !== 'verified' || !isDeviation(Boolean(entry.item.isIoe), occurrenceOutcome(entry.item))) return []; const baselinePassCounts = new Map<string, number>();
if ((targetByIdentity.get(entry.identity.identity.key) ?? []).length !== 1) return []; baselineOccurrences.filter(item => occurrenceOutcome(item) === 'pass').forEach(item => {
return baselineKeys.has(entry.identity.identity.key) ? [] : [entry.item.id]; const identity = identityFor(item, definition, baselineScope.scope);
if (identity.state === 'verified') baselinePassCounts.set(identity.identity.key, (baselinePassCounts.get(identity.identity.key) ?? 0) + 1);
}); });
const currentDeviations = targetIdentities.filter(entry => entry.identity.state === 'verified'
&& isDeviation(Boolean(entry.item.isIoe), occurrenceOutcome(entry.item))
&& (targetByIdentity.get(entry.identity.identity.key) ?? []).length === 1
&& !baselineKeys.has(entry.identity.identity.key));
const keyOf = (entry: typeof targetIdentities[number]) => entry.identity.state === 'verified' ? entry.identity.identity.key : '';
const regressed = currentDeviations.filter(entry => baselinePassCounts.get(keyOf(entry)) === 1);
// Several passes of one identity in A are ambiguous history: neither new nor regressed.
const newDeviationOccurrenceIds = currentDeviations.filter(entry => !baselinePassCounts.has(keyOf(entry))).map(entry => entry.item.id);
return { return {
toolId, baselineRunId: baselineRun.id, targetRunId: targetRun.id, toolId, baselineRunId: baselineRun.id, targetRunId: targetRun.id,
state: 'compared', reasons: [...new Set(blockingDiagnostics)], decisions, newDeviationOccurrenceIds state: 'compared', reasons: [...new Set(blockingDiagnostics)], decisions, newDeviationOccurrenceIds,
regressedDeviationOccurrenceIds: regressed.map(entry => entry.item.id)
}; };
}); });
const decisions = sources.flatMap(source => source.decisions); const decisions = sources.flatMap(source => source.decisions);
return { return {
version: '1.0.0', projectId, baselineRunIds: [...baselineRunIds], targetRunIds: [...targetRunIds], sources, version: '1.1.0', projectId, baselineRunIds: [...baselineRunIds], targetRunIds: [...targetRunIds], sources,
summary: { summary: {
baselineDeviations: decisions.length, baselineDeviations: decisions.length,
verifiedRemediated: decisions.filter(item => item.state === 'verified-remediated').length, verifiedRemediated: decisions.filter(item => item.state === 'verified-remediated').length,
@ -246,7 +263,8 @@ export function compareProjectTimeslotEvidence(
unverifiedAbsent: decisions.filter(item => item.state === 'unverified-absent').length, unverifiedAbsent: decisions.filter(item => item.state === 'unverified-absent').length,
notComparable: decisions.filter(item => item.state === 'not-comparable').length, notComparable: decisions.filter(item => item.state === 'not-comparable').length,
notApplicable: decisions.filter(item => item.state === 'not-applicable').length, notApplicable: decisions.filter(item => item.state === 'not-applicable').length,
newDeviations: sources.reduce((sum, source) => sum + source.newDeviationOccurrenceIds.length, 0) newDeviations: sources.reduce((sum, source) => sum + source.newDeviationOccurrenceIds.length, 0),
regressedDeviations: sources.reduce((sum, source) => sum + source.regressedDeviationOccurrenceIds.length, 0)
} }
}; };
} }

View File

@ -0,0 +1,227 @@
import type { CanonicalAssessmentModel, FindingOccurrence } from '../domain/canonical-model';
import type { RemediationEvidenceState } from '../domain/timeslot-comparison';
import { isDeviation } from '../domain/deviation-predicate';
import { outcomeFromLegacyStatus } from '../domain/unified-result';
import type { SecurityScanScore } from '../domain/security-scoring';
import { compareSecurityScores, defaultSecurityScoringPolicy, weightedSecurityScore } from '../domain/security-scoring';
import { canonicalSecurityScores } from '../scoring/canonical-security-score';
import { compareProjectTimeslotEvidence, type ProjectTimeslotEvidenceComparison } from './timeslot-comparison-engine';
import { compareGpoTimeslots, type GpoTimeslotComparison, type GpoTimeslotInput } from './gpo-timeslot-comparison';
export const TIMESLOT_COMPARISON_REPORT_VERSION = '1.0.0' as const;
export const GPO_SOURCE_ID = 'group-policy';
/**
* Change categories shown by the Comparison Engine. `value-changed` is a still deviating setting
* with a different retained value and is neutral (user decision 2026-10-13). Only `resolved` is better;
* only `new` and `regressed` are worse.
*/
export type TimeslotChange = 'resolved' | 'unchanged' | 'value-changed' | 'unverified' | 'not-comparable'
| 'not-applicable' | 'new' | 'regressed';
export interface TimeslotChangeEntry {
sourceId: string;
change: TimeslotChange;
/** Which timeslot the referenced records belong to. */
side: 'baseline' | 'target';
/** Scanner occurrence IDs or rolled-up GPO Work result IDs. */
recordIds: string[];
/** Counterpart records in the other timeslot, when paired. */
counterpartIds: string[];
identityKey?: string;
reasons: string[];
}
export interface TimeslotSourceScore {
sourceId: string;
baselineScore?: number;
targetScore?: number;
change?: number;
weight: number;
method: 'paired-scans' | 'shared-gpo-pass-rate';
reasons: string[];
}
export interface TimeslotComparisonReport {
version: typeof TIMESLOT_COMPARISON_REPORT_VERSION;
projectId: string;
baselineRunIds: string[];
targetRunIds: string[];
sources: Array<{ sourceId: string; state: 'compared' | 'not-comparable'; reasons: string[] }>;
changes: TimeslotChangeEntry[];
summary: Record<TimeslotChange, number> & { targetNotComparable: number };
score: {
/** Weighted only over sources with a comparable score in both timeslots. */
baselineScore?: number;
targetScore?: number;
change?: number;
sources: TimeslotSourceScore[];
excludedSources: Array<{ sourceId: string; reasons: string[] }>;
};
scanner: ProjectTimeslotEvidenceComparison;
gpo?: GpoTimeslotComparison;
}
const changeOf: Record<RemediationEvidenceState, TimeslotChange> = {
'verified-remediated': 'resolved',
'confirmed-open': 'unchanged',
'unverified-absent': 'unverified',
'not-comparable': 'not-comparable',
'not-applicable': 'not-applicable'
};
const outcomeOf = (item: FindingOccurrence) => item.outcome?.normalizedValue
?? outcomeFromLegacyStatus(item.status.normalizedValue ?? 'unknown');
function scannerChanges(models: CanonicalAssessmentModel[], scanner: ProjectTimeslotEvidenceComparison): TimeslotChangeEntry[] {
const occurrences = models.flatMap(model => model.findingOccurrences);
return scanner.sources.flatMap(source => [
...source.decisions.map((item): TimeslotChangeEntry => ({
sourceId: source.toolId, change: changeOf[item.state], side: 'baseline', recordIds: [item.baselineOccurrenceId],
counterpartIds: item.targetOccurrenceId ? [item.targetOccurrenceId] : [],
...(item.identityKey ? { identityKey: item.identityKey } : {}), reasons: item.reasons
})),
...source.newDeviationOccurrenceIds.map((id): TimeslotChangeEntry => ({
sourceId: source.toolId, change: 'new', side: 'target', recordIds: [id], counterpartIds: [], reasons: []
})),
...source.regressedDeviationOccurrenceIds.map((id): TimeslotChangeEntry => ({
sourceId: source.toolId, change: 'regressed', side: 'target', recordIds: [id], counterpartIds: [], reasons: []
})),
// Deviations of a source that cannot be compared are neither new nor worse.
...(source.state === 'not-comparable' && source.targetRunId ? occurrences
.filter(item => item.scanRunId === source.targetRunId && isDeviation(Boolean(item.isIoe), outcomeOf(item)))
.map((item): TimeslotChangeEntry => ({
sourceId: source.toolId, change: 'not-comparable', side: 'target', recordIds: [item.id], counterpartIds: [],
reasons: source.reasons
})) : [])
]);
}
function gpoChanges(gpo: GpoTimeslotComparison, target?: GpoTimeslotInput): TimeslotChangeEntry[] {
const changes = [
...gpo.decisions.map((item): TimeslotChangeEntry => ({
sourceId: GPO_SOURCE_ID,
change: item.state === 'confirmed-open' && item.valueChanged ? 'value-changed' : changeOf[item.state],
side: 'baseline', recordIds: item.baselineResultIds, counterpartIds: item.targetResultIds,
identityKey: item.settingKey, reasons: item.reasons
})),
...gpo.newDeviations.map((item): TimeslotChangeEntry => ({
sourceId: GPO_SOURCE_ID, change: 'new', side: 'target', recordIds: item.targetResultIds, counterpartIds: [],
identityKey: item.settingKey, reasons: []
})),
...gpo.regressedDeviations.map((item): TimeslotChangeEntry => ({
sourceId: GPO_SOURCE_ID, change: 'regressed', side: 'target', recordIds: item.targetResultIds, counterpartIds: [],
identityKey: item.settingKey, reasons: []
}))
];
if (gpo.state === 'not-comparable' && target) {
const keys = new Set(target.results.filter(item => item.status === 'Fail').map(item => JSON.stringify([item.scope, item.identity])));
for (const key of [...keys].sort()) changes.push({
sourceId: GPO_SOURCE_ID, change: 'not-comparable', side: 'target', identityKey: key, counterpartIds: [],
recordIds: target.results.filter(item => JSON.stringify([item.scope, item.identity]) === key).map(item => item.id).sort(),
reasons: gpo.reasons
});
}
return changes;
}
function scannerScores(models: CanonicalAssessmentModel[], baselineRunIds: string[], targetRunIds: string[]) {
const scans = canonicalSecurityScores(models);
const pick = (ids: string[]) => scans.filter(scan => scan.runIds.some(id => ids.includes(id)));
const baseline = pick(baselineRunIds);
const target = pick(targetRunIds);
const byTool = (items: SecurityScanScore[]) => {
const map = new Map<string, SecurityScanScore[]>();
items.forEach(item => map.set(item.toolId, [...(map.get(item.toolId) ?? []), item]));
return map;
};
const baselineByTool = byTool(baseline);
const targetByTool = byTool(target);
return [...new Set([...baselineByTool.keys(), ...targetByTool.keys()])].sort().map((toolId): TimeslotSourceScore => {
const left = baselineByTool.get(toolId) ?? [];
const right = targetByTool.get(toolId) ?? [];
const policy = (left[0] ?? right[0]).policy;
const weight = policy.weights[toolId] ?? policy.defaultWeight;
if (!left.length || !right.length) return {
sourceId: toolId, weight, method: 'paired-scans', reasons: [left.length ? 'target-source-missing' : 'baseline-source-missing']
};
const compared = compareSecurityScores(left, right);
const reasons = [
...(compared.comparable === 0 ? ['no-comparable-scan-pair'] : []),
...(compared.scopeChanged ? ['scan-scope-changed'] : [])
];
return {
sourceId: toolId, weight, method: 'paired-scans', reasons,
...(compared.change !== undefined && !compared.scopeChanged ? {
baselineScore: compared.baselineScore, targetScore: compared.currentScore, change: compared.change
} : {})
};
});
}
/**
* Explicit A/B comparison of two timeslots of one client. Scanner sources use the conservative
* evidence comparison, GPO settings their rolled-up Work verdicts. Score deltas cover only sources
* with a comparable score in both timeslots; every other source is listed as excluded with its reason.
*/
export function compareTimeslots(input: {
projectId: string;
models: CanonicalAssessmentModel[];
baselineRunIds: string[];
targetRunIds: string[];
gpo?: { baseline: GpoTimeslotInput; target: GpoTimeslotInput };
}): TimeslotComparisonReport {
if (input.models.some(model => model.project.id !== input.projectId)) throw new Error('Timeslots must belong to the same project.');
const scanner: ProjectTimeslotEvidenceComparison = input.baselineRunIds.length || input.targetRunIds.length
? compareProjectTimeslotEvidence(input.models, input.baselineRunIds, input.targetRunIds)
: { version: '1.1.0', projectId: input.projectId, baselineRunIds: [], targetRunIds: [], sources: [], summary: {
baselineDeviations: 0, verifiedRemediated: 0, confirmedOpen: 0, unverifiedAbsent: 0, notComparable: 0,
notApplicable: 0, newDeviations: 0, regressedDeviations: 0
} };
const gpo = input.gpo && (input.gpo.baseline.results.length || input.gpo.target.results.length)
? compareGpoTimeslots(input.gpo.baseline, input.gpo.target) : undefined;
const changes = [...scannerChanges(input.models, scanner), ...(gpo ? gpoChanges(gpo, input.gpo?.target) : [])];
const sourceScores = scannerScores(input.models, input.baselineRunIds, input.targetRunIds);
if (gpo) {
const policy = input.models[0]?.assessment.scoringPolicy ?? defaultSecurityScoringPolicy();
const rate = gpo.sharedPassRate;
sourceScores.push({
sourceId: GPO_SOURCE_ID, weight: policy.weights[GPO_SOURCE_ID] ?? policy.defaultWeight, method: 'shared-gpo-pass-rate',
reasons: rate ? [] : gpo.state === 'not-comparable' ? gpo.reasons : ['no-shared-decided-settings'],
...(rate ? { baselineScore: rate.baseline, targetScore: rate.target, change: rate.target - rate.baseline } : {})
});
}
const scored = sourceScores.filter(item => item.change !== undefined);
const baselineScore = weightedSecurityScore(scored.map(item => ({ score: item.baselineScore, weight: item.weight })));
const targetScore = weightedSecurityScore(scored.map(item => ({ score: item.targetScore, weight: item.weight })));
const count = (change: TimeslotChange, side?: 'baseline' | 'target') =>
changes.filter(item => item.change === change && (!side || item.side === side)).length;
return {
version: TIMESLOT_COMPARISON_REPORT_VERSION,
projectId: input.projectId,
baselineRunIds: [...input.baselineRunIds],
targetRunIds: [...input.targetRunIds],
sources: [
...scanner.sources.map(source => ({ sourceId: source.toolId, state: source.state, reasons: source.reasons })),
...(gpo ? [{ sourceId: GPO_SOURCE_ID, state: gpo.state, reasons: gpo.reasons }] : [])
],
changes,
summary: {
resolved: count('resolved'), unchanged: count('unchanged'), 'value-changed': count('value-changed'),
unverified: count('unverified'), 'not-comparable': count('not-comparable', 'baseline'),
'not-applicable': count('not-applicable'), new: count('new'), regressed: count('regressed'),
targetNotComparable: count('not-comparable', 'target')
},
score: {
...(baselineScore !== undefined && targetScore !== undefined
? { baselineScore, targetScore, change: targetScore - baselineScore } : {}),
sources: sourceScores,
excludedSources: sourceScores.filter(item => item.change === undefined)
.map(item => ({ sourceId: item.sourceId, reasons: item.reasons }))
},
scanner,
...(gpo ? { gpo } : {})
};
}

View File

@ -0,0 +1,122 @@
import { describe, expect, it } from 'vitest';
import type { PolicyWorkResult, PolicyWorkStatus } from '../../src/evaluation/policy-work-results';
import { compareGpoTimeslots } from '../../src/comparison/gpo-timeslot-comparison';
// Synthetic rolled-up GPO Work results. Only the fields the comparison reads are populated.
const observedA = '2026-01-01T10:00:00.000Z';
const observedB = '2026-02-01T10:00:00.000Z';
function setting(id: string, identity: string, status: PolicyWorkStatus, options: {
actual?: unknown; baselineId?: string; aggregationRun?: string; scope?: string;
} = {}): PolicyWorkResult {
const scope = options.scope ?? 'computer';
return {
id, status, title: identity, scope, identity, importedAt: observedA, sources: [],
settingKey: JSON.stringify(['dummy-project', options.aggregationRun ?? 'aggregation-1', scope, identity]),
comparison: {
baselineId: options.baselineId ?? 'baseline-server',
comparator: { actual: options.actual === undefined ? undefined : { kind: 'number', value: options.actual } }
} as unknown as PolicyWorkResult['comparison']
};
}
describe('GPO timeslot comparison', () => {
it('pairs rolled-up settings by scope and identity and keeps a value change neutral', () => {
const result = compareGpoTimeslots(
{ observedAt: observedA, results: [
setting('a-1', 'MinimumPasswordLength', 'Fail', { actual: 7 }),
setting('a-2', 'LockoutBadCount', 'Fail', { actual: 0 }),
setting('a-3', 'NoLMHash', 'Fail', { actual: 0 }),
setting('a-4', 'PasswordComplexity', 'Fail', { actual: 0 })
] },
{ observedAt: observedB, results: [
setting('b-1', 'MinimumPasswordLength', 'Fail', { actual: 10 }),
setting('b-2', 'LockoutBadCount', 'Fail', { actual: 0 }),
setting('b-3', 'NoLMHash', 'Pass', { actual: 1 }),
setting('b-4', 'PasswordComplexity', 'Not configured')
] }
);
const byIdentity = Object.fromEntries(result.decisions.map(item => [item.identity, item]));
expect(byIdentity.MinimumPasswordLength).toMatchObject({ state: 'confirmed-open', valueChanged: true, targetResultIds: ['b-1'] });
expect(byIdentity.LockoutBadCount).toMatchObject({ state: 'confirmed-open', valueChanged: false });
expect(byIdentity.NoLMHash).toMatchObject({ state: 'verified-remediated', reasons: ['same-setting-explicit-pass'] });
// A setting missing from the newer backup set is not proof of a removed configuration.
expect(byIdentity.PasswordComplexity).toMatchObject({ state: 'unverified-absent', reasons: ['target-verdict-not-decisive:Not configured'] });
});
it('rolls several baseline releases of one setting up with the Work verdict order', () => {
const result = compareGpoTimeslots(
{ observedAt: observedA, results: [
setting('a-1', 'SeBackupPrivilege', 'Pass', { baselineId: 'baseline-dc' }),
setting('a-2', 'SeBackupPrivilege', 'Fail', { baselineId: 'baseline-server' })
] },
{ observedAt: observedB, results: [
setting('b-1', 'SeBackupPrivilege', 'Pass', { baselineId: 'baseline-dc' }),
setting('b-2', 'SeBackupPrivilege', 'Pass', { baselineId: 'baseline-server' })
] }
);
expect(result.decisions).toHaveLength(1);
expect(result.decisions[0]).toMatchObject({ state: 'verified-remediated', baselineResultIds: ['a-1', 'a-2'] });
});
it('separates new and regressed settings and claims nothing from undecided history', () => {
const result = compareGpoTimeslots(
{ observedAt: observedA, results: [
setting('a-1', 'SeEnableDelegationPrivilege', 'Not configured'),
setting('a-2', 'SeRestorePrivilege', 'Pass'),
setting('a-3', 'SeAuditPrivilege', 'Review required'),
setting('a-4', 'SeLoadDriverPrivilege', 'Pass', { baselineId: 'baseline-old' })
] },
{ observedAt: observedB, results: [
setting('b-1', 'SeEnableDelegationPrivilege', 'Fail'),
setting('b-2', 'SeRestorePrivilege', 'Fail'),
setting('b-3', 'SeAuditPrivilege', 'Fail'),
setting('b-4', 'SeLoadDriverPrivilege', 'Fail'),
setting('b-5', 'DnsPolicyConfig', 'Fail')
] }
);
expect(result.newDeviations.map(item => item.targetResultIds[0]).sort()).toEqual(['b-1', 'b-5']);
expect(result.regressedDeviations.map(item => item.targetResultIds[0])).toEqual(['b-2']);
});
it('requires a newer backup set, the same baselines and a GPO source in both timeslots', () => {
const fail = { observedAt: observedA, results: [setting('a-1', 'NoLMHash', 'Fail')] };
const sameTime = compareGpoTimeslots(fail, { observedAt: observedA, results: [setting('b-1', 'NoLMHash', 'Pass')] });
expect(sameTime.decisions[0]).toMatchObject({ state: 'unverified-absent', reasons: ['newer-observation-not-verified'] });
expect(sameTime.sharedPassRate).toBeUndefined();
const changedBaseline = compareGpoTimeslots(fail, { observedAt: observedB, results: [setting('b-1', 'NoLMHash', 'Pass', { baselineId: 'baseline-new' })] });
expect(changedBaseline.decisions[0]).toMatchObject({ state: 'not-comparable', reasons: ['baseline-set-changed'] });
const missing = compareGpoTimeslots(fail, { observedAt: observedB, results: [] });
expect(missing).toMatchObject({ state: 'not-comparable', reasons: ['target-source-missing'] });
expect(missing.decisions[0].state).toBe('not-comparable');
});
it('marks a setting held by several aggregation runs of one timeslot as ambiguous', () => {
const result = compareGpoTimeslots(
{ observedAt: observedA, results: [
setting('a-1', 'NoLMHash', 'Fail', { aggregationRun: 'aggregation-1' }),
setting('a-2', 'NoLMHash', 'Fail', { aggregationRun: 'aggregation-2' })
] },
{ observedAt: observedB, results: [setting('b-1', 'NoLMHash', 'Pass')] }
);
expect(result.decisions[0]).toMatchObject({ state: 'not-comparable', reasons: ['gpo-setting-ambiguous'] });
});
it('computes the shared pass rate only over settings decided in both timeslots', () => {
const result = compareGpoTimeslots(
{ observedAt: observedA, results: [
setting('a-1', 'One', 'Fail'), setting('a-2', 'Two', 'Fail'), setting('a-3', 'Three', 'Pass'),
setting('a-4', 'OnlyInA', 'Fail')
] },
{ observedAt: observedB, results: [
setting('b-1', 'One', 'Pass'), setting('b-2', 'Two', 'Fail'), setting('b-3', 'Three', 'Pass'),
setting('b-5', 'OnlyInB', 'Pass')
] }
);
expect(result.sharedPassRate?.settings).toBe(3);
expect(result.sharedPassRate?.baseline).toBeCloseTo(100 / 3);
expect(result.sharedPassRate?.target).toBeCloseTo(200 / 3);
});
});

View File

@ -89,12 +89,37 @@ describe('conservative project timeslot comparison', () => {
const result = compareProjectTimeslotEvidence([baseline, target], ['run-a'], ['run-b']); const result = compareProjectTimeslotEvidence([baseline, target], ['run-a'], ['run-b']);
expect(result.summary).toEqual({ expect(result.summary).toEqual({
baselineDeviations: 2, verifiedRemediated: 1, confirmedOpen: 0, baselineDeviations: 2, verifiedRemediated: 1, confirmedOpen: 0,
unverifiedAbsent: 1, notComparable: 0, notApplicable: 0, newDeviations: 1 unverifiedAbsent: 1, notComparable: 0, notApplicable: 0, newDeviations: 1, regressedDeviations: 0
}); });
expect(result.sources[0].decisions.map(item => item.state).sort()) expect(result.sources[0].decisions.map(item => item.state).sort())
.toEqual(['unverified-absent', 'verified-remediated']); .toEqual(['unverified-absent', 'verified-remediated']);
}); });
it('separates a regression after a single verified pass from a new deviation and ambiguous history', () => {
const baseline = model('run-a', observedA, [
occurrence('a-1', 'run-a', 'RULE-1', 'passed'),
occurrence('a-2', 'run-a', 'RULE-2', 'passed'),
occurrence('a-2b', 'run-a', 'RULE-2', 'passed')
]);
const target = model('run-b', observedB, [
occurrence('b-1', 'run-b', 'RULE-1', 'failed'),
occurrence('b-2', 'run-b', 'RULE-2', 'failed'),
occurrence('b-3', 'run-b', 'RULE-3', 'failed')
]);
const result = compareProjectTimeslotEvidence([baseline, target], ['run-a'], ['run-b']);
expect(result.version).toBe('1.1.0');
expect(result.sources[0].regressedDeviationOccurrenceIds).toEqual(['b-1']);
expect(result.sources[0].newDeviationOccurrenceIds).toEqual(['b-3']);
expect(result.summary).toMatchObject({ newDeviations: 1, regressedDeviations: 1, baselineDeviations: 0 });
});
it('reports a source of only one timeslot as not comparable instead of failing', () => {
const target = model('run-b', observedB, [occurrence('b-1', 'run-b', 'RULE-1', 'failed')]);
const result = compareProjectTimeslotEvidence([target], [], ['run-b']);
expect(result.sources).toMatchObject([{ state: 'not-comparable', reasons: ['baseline-source-missing'] }]);
expect(result.summary).toMatchObject({ newDeviations: 0, regressedDeviations: 0 });
});
it('does not treat a missing scanner source or a same-time reimport as remediation', () => { it('does not treat a missing scanner source or a same-time reimport as remediation', () => {
const baseline = model('run-a', observedA, [occurrence('a-1', 'run-a', 'RULE-1', 'failed')]); const baseline = model('run-a', observedA, [occurrence('a-1', 'run-a', 'RULE-1', 'failed')]);
const otherBaseline = model('run-c', observedA, []); const otherBaseline = model('run-c', observedA, []);

View File

@ -0,0 +1,145 @@
import { describe, expect, it } from 'vitest';
import type {
CanonicalAssessmentModel,
CanonicalFindingStatus,
FindingOccurrence,
ImportFileRecord
} from '../../src/domain/canonical-model';
import type { EnvironmentObservation } from '../../src/domain/environment-context';
import type { PolicyWorkResult, PolicyWorkStatus } from '../../src/evaluation/policy-work-results';
import { compareTimeslots } from '../../src/comparison/timeslot-comparison-report';
// Synthetic two-timeslot client: CIS-CAT in both, Maester only in B, GPO settings in both.
const observedA = '2026-01-01T10:00:00.000Z';
const observedB = '2026-02-01T10:00:00.000Z';
const provenance = {
decisionSource: 'source-native' as const,
confidence: { level: 'high' as const, score: 1, reasons: ['synthetic'] },
mappingVersion: '1'
};
const tools = { 'cis-cat': { id: 'cis-cat', name: 'CIS-CAT Pro' }, maester: { id: 'maester', name: 'Maester' } };
function occurrence(id: string, runId: string, toolId: keyof typeof tools, ruleId: string, status: CanonicalFindingStatus): FindingOccurrence {
const outcome = status === 'passed' ? 'pass' : 'fail';
return {
id, definitionId: `definition-${ruleId}`, scanRunId: runId, importFileId: `file-${runId}`,
sourceIdentity: { tool: tools[toolId], canonicalRuleId: ruleId }, sourcePosition: { rowIndex: 1 },
originalFields: [], fieldMappings: [],
severity: { originalValue: 'high', normalizedValue: 'high', provenance },
status: { originalValue: status, normalizedValue: status, provenance },
outcome: { originalValue: outcome, normalizedValue: outcome, provenance }
};
}
function importFile(runId: string, toolId: keyof typeof tools): ImportFileRecord {
return {
id: `file-${runId}`, scanRunId: runId, fileName: `${runId}.html`, format: 'HTML', sizeBytes: 10, importedAt: observedA,
detectedTool: tools[toolId], detection: { level: 'high', score: 1, reasons: ['synthetic'] },
processedWith: {
parserId: 'html', parserVersion: '1', profileId: toolId, profileVersion: '1.4.0', mappingVersion: '1.3.0',
normalizationVersion: '1.3.0', presentationVersion: '1.3.0', compatibilityKey: `${toolId}-canonical-v1`
},
sourceArtifact: { kind: 'not-retained', reason: 'synthetic' }
};
}
function observation(runId: string, toolId: keyof typeof tools, property: string, value: string): EnvironmentObservation {
return {
version: 1, id: `${runId}-${property}`, scanRunId: runId, importFileId: `file-${runId}`, tool: tools[toolId],
subject: { type: 'host' }, property, label: property, originalValue: value, normalizedValue: value,
valueType: property === 'report.generated_at' ? 'datetime' : 'string', state: 'observed',
source: {}, mappingVersion: '1', attributes: {}
};
}
function model(slot: 'a' | 'b', runs: Array<{ runId: string; toolId: keyof typeof tools; findings: FindingOccurrence[] }>): CanonicalAssessmentModel {
const observedAt = slot === 'a' ? observedA : observedB;
const assessmentId = `assessment-${slot}`;
const findings = runs.flatMap(run => run.findings);
return {
schemaVersion: 1,
project: { id: 'dummy-project', name: 'Dummy Customer', createdAt: observedA, updatedAt: observedB, attributes: {} },
assessment: { id: assessmentId, projectId: 'dummy-project', name: slot, createdAt: observedAt, updatedAt: observedAt },
scanRuns: runs.map((run, index) => ({ id: run.runId, assessmentId, sequence: index + 1, label: run.runId, tool: tools[run.toolId], importedAt: observedAt })),
importFiles: runs.map(run => importFile(run.runId, run.toolId)), sourceTables: [],
environmentObservations: runs.flatMap(run => [
observation(run.runId, run.toolId, 'host.name', 'server-a'),
observation(run.runId, run.toolId, 'benchmark.name', 'benchmark-a'),
observation(run.runId, run.toolId, 'report.generated_at', observedAt),
observation(run.runId, run.toolId, 'report.engine_version', '4.0.0')
]),
findingDefinitions: [...new Set(findings.map(item => item.definitionId))].map(id => ({
id, canonicalKey: id, title: id, frameworkReferences: [], createdAt: observedA, updatedAt: observedAt
})),
findingOccurrences: findings, evidence: [], reviews: [], diagnostics: []
};
}
function setting(id: string, identity: string, status: PolicyWorkStatus, actual?: number): PolicyWorkResult {
return {
id, status, title: identity, scope: 'computer', identity, importedAt: observedA, sources: [],
settingKey: JSON.stringify(['dummy-project', 'aggregation', 'computer', identity]),
comparison: { baselineId: 'baseline-server', comparator: { actual: actual === undefined ? undefined : { kind: 'number', value: actual } } } as unknown as PolicyWorkResult['comparison']
};
}
function report(reversed = false) {
const order = <T,>(items: T[]) => reversed ? [...items].reverse() : items;
const a = model('a', [{ runId: 'cis-a', toolId: 'cis-cat', findings: [
occurrence('a-1', 'cis-a', 'cis-cat', '1.1.1', 'failed'),
occurrence('a-2', 'cis-a', 'cis-cat', '1.1.2', 'failed'),
occurrence('a-3', 'cis-a', 'cis-cat', '5.1', 'passed')
] }]);
const b = model('b', [
{ runId: 'cis-b', toolId: 'cis-cat', findings: [
occurrence('b-1', 'cis-b', 'cis-cat', '1.1.1', 'passed'),
occurrence('b-2', 'cis-b', 'cis-cat', '1.1.2', 'failed'),
occurrence('b-3', 'cis-b', 'cis-cat', '5.1', 'failed')
] },
{ runId: 'maester-b', toolId: 'maester', findings: [occurrence('m-1', 'maester-b', 'maester', 'MT.1001', 'failed')] }
]);
return compareTimeslots({
projectId: 'dummy-project', models: order([a, b]), baselineRunIds: ['cis-a'], targetRunIds: order(['cis-b', 'maester-b']),
gpo: {
baseline: { observedAt: observedA, results: order([setting('ga-1', 'MinimumPasswordLength', 'Fail', 7), setting('ga-2', 'NoLMHash', 'Fail', 0)]) },
target: { observedAt: observedB, results: order([setting('gb-1', 'MinimumPasswordLength', 'Fail', 10), setting('gb-2', 'NoLMHash', 'Pass', 1)]) }
}
});
}
describe('Timeslot comparison report', () => {
it('assigns every change category from scanner evidence and rolled-up GPO verdicts', () => {
const result = report();
expect(result.summary).toEqual({
resolved: 2, unchanged: 1, 'value-changed': 1, unverified: 0, 'not-comparable': 0, 'not-applicable': 0,
new: 0, regressed: 1, targetNotComparable: 1
});
const find = (change: string) => result.changes.filter(item => item.change === change).map(item => item.recordIds).flat().sort();
expect(find('resolved')).toEqual(['a-1', 'ga-2']);
expect(find('value-changed')).toEqual(['ga-1']);
expect(find('regressed')).toEqual(['b-3']);
// A source only in B is neither new nor worse.
expect(result.changes.find(item => item.change === 'not-comparable')).toMatchObject({ sourceId: 'maester', side: 'target', recordIds: ['m-1'] });
});
it('limits the score delta to sources comparable in both timeslots and lists every excluded source', () => {
const result = report();
const gpo = result.score.sources.find(item => item.sourceId === 'group-policy');
expect(gpo).toMatchObject({ method: 'shared-gpo-pass-rate', baselineScore: 0, targetScore: 50, change: 50 });
expect(result.score.excludedSources).toContainEqual({ sourceId: 'maester', reasons: ['baseline-source-missing'] });
const scored = result.score.sources.filter(item => item.change !== undefined);
expect(scored.map(item => item.sourceId)).toEqual(['cis-cat', 'group-policy']);
// Same three evaluated rules in A and B: one pass each, so the paired CIS-CAT pass rate is unchanged.
expect(result.score.sources.find(item => item.sourceId === 'cis-cat')).toMatchObject({ method: 'paired-scans', change: 0 });
const weight = scored.reduce((sum, item) => sum + item.weight, 0);
expect(result.score.targetScore).toBeCloseTo(scored.reduce((sum, item) => sum + item.targetScore! * item.weight, 0) / weight);
});
it('is deterministic and independent of the input order', () => {
const first = report();
const reversed = report(true);
expect(reversed.summary).toEqual(first.summary);
expect(reversed.changes).toEqual(first.changes);
expect(reversed.score).toEqual(first.score);
});
});