Assessor_2.0/docs/SCORING_MODEL.md

ASSESSOR scoring rule 1.0.0

Decision: 2026-09-10, explicitly approved by the user. Implementation belongs to Phase 7/13C (Client Info and scanner context), with score-comparison safeguards under Phase 8. This decision supersedes the display-only native-score policy.

Comparison qualification, 2026-09-11

The numeric contribution policy below remains unchanged. Its comparison safeguards are not a completed acceptance of object-level remediation or an explicitly selected A/B report. Code review identified an evaluation-population key without asset identity, weaker pending-revalidation checks than score pairing, and last-known-source replacement that can omit unresolved obligations after partial intermediate scans. Original stored findings are not deleted. The current historical baseline is not necessarily the user's selected A.

TIMESLOT_COMPARISON_PLAN.md defines the open Phase-8 follow-up: verified object/scope populations, an explicit A/B basis, obligation-level history and separate eligibility rules for score deltas and remediation evidence. Matching assessed and eligible counts alone does not prove coverage of absent objects. Native totals must not be recomputed for an arbitrary smaller scope without separately retained contributions. Missing historical risks stay visible separately; no historical score is relabelled as a current measurement and no new penalty formula is introduced.

This is a planning clarification, not a runtime fix, processing-manifest change or completed verification. The earlier implementation/test notes below remain historical evidence for their stated scope; they do not satisfy the new follow-up's acceptance matrix.

GPO scoring scope, decision 2026-09-16

The GPO contribution measures conformance of customer settings that are actually configured, independently comparable and inside an explicitly applicable Microsoft-baseline scope. It is not a measurement of complete Microsoft-baseline adoption. Recommended settings that are absent from the customer GPO set remain visible as Not configured, but they do not create operational findings, remediation obligations or score penalties in the normal assessment. Full baseline deployment is a separate explicitly commissioned hardening scope.

Clarification 2026-09-16: absence is reported within the analyzed backup scope. An applicable recommendation with no observed configured value in that imported set remains filterable as Not configured, including for a selected subset. This does not assert domain-wide absence or effective device state; the limited scope remains diagnostic evidence. The verdict remains excluded from the score.

Clarification later on 2026-09-16: Not configured is a score-neutral policy verdict. Duplicates and Conflicts are distinct diagnostic categories on a technical policy setting, not alternative verdicts. Roll up member comparisons once per technical setting: one independently comparable, in-scope configured member value with a valid Fail makes the complete policy-setting verdict Fail; retain the conflict/duplicate flags beside it. All applicable members passing yields Passed. Unresolved applicability, precedence or comparison yields Review required. Duplicate source rows, alternative presentations and diagnostic flags never add score weight.

This clarification supersedes only the earlier same-day interpretation that a conflict/duplicate container could remain score-neutral despite a proven member fail. It is binding for new GPO scoring work. Current runtime categories remain disjoint, so the rolled-up verdict plus diagnostic-flag model requires a dedicated result-population identity, synthetic conflict/duplicate fixtures and explicit verification before it may be claimed as implemented. If implementing it changes persisted evaluation semantics or score inputs, version the affected evaluator/scoring contract and follow the reimport/re-evaluation policy; do not reinterpret historical runs silently.

Scan contribution

Use a valid native score from the exact import file and scan run. Otherwise use 100 * passed / (passed + deviations) when the result population supports a pass rate. Never combine the native score and the pass rate of the same scan.

Source	Native contribution	Fallback
Purple Knight for Active Directory	Native AD posture, 0–100	Pass rate
Purple Knight for Entra ID	Native Entra posture, 0–100	Pass rate
CIS-CAT	Native benchmark score, 0–100	Pass rate
PingCastle Healthcheck	100 - native risk, bounded 0–100	No pass rate from findings-only reports
Maester	No native numeric score assumed	Pass rate
GPO Analysis	None	Pass rate of configured, independently comparable in-scope settings
Additional registered scanners	Native overall score declared by their profile	Declared result-population fallback
Unrecognized tabular sources	No native score inferred	Provisional pass rate only with evidence of passed results

Purple Knight AD and Purple Knight Entra are separate scan scopes. Their native posture values are bound respectively to the imported forest or Entra tenant and must never be substituted for one another. A mixed report with conflicting native scopes remains unavailable rather than being collapsed into one score. A missing native score on an old import requires the relevant summary context to be reimported; no score or tenant is fabricated.

Null, blank, boolean, nonnumeric, out-of-range and conflicting native values remain unavailable. A valid zero is a real score. Invalid native data and source integrity diagnostics are visible in the calculation details.

Passed and deviations use normalized outcomes. Not applicable is excluded. Execution errors, not run and unknown results do not pass and do not add a security failure. They affect known-check coverage. GPO context states retain their existing authoritative disposition. Not configured does not enter the GPO denominator. Conflict/duplicate flags do not enter independently; the rolled-up policy verdict enters once and is Fail when any independently comparable, in-scope configured member fails. No severity or missing-setting penalty is invented. Findings-only input cannot establish a pass denominator.

Future scanner support

Every new or extended scanner profile must first examine the supported report formats for a usable native overall score. ToolProfileDefinition.securityScoring is mandatory: declare exact mapped observation properties, overall scope, 0–100 scale, direction, incompatible scopes and fallback population, or record an explicit reason why native scoring is unavailable. A new profile registered in the profile catalog automatically uses the shared native-first calculation; there is no separate hardcoded scanner list in that calculation.

The extractor must retain the original score and its exact source path as file/run-bound report context. Different native scales or ambiguous scope need an explicitly documented and tested mapping before they are usable. Arbitrary fields named score, category scores and finding weights do not qualify. Until an unfamiliar format has such a profile, its native semantics remain unknown.

A valid native score, including zero, takes priority even for findings-only reports. Missing, invalid, conflicting or unrelated native observations lead to the declared fallback; without a defensible result population the contribution remains unavailable. Scoring-definition versions participate in comparison compatibility. Extend synthetic format fixtures and verify native priority, provenance, invalid values and fallback whenever adding scanner support.

This extension moves existing declarations into their profiles without changing their numeric methods or import mapping versions. No historic imports are rewritten. New extraction or semantic changes still require explicit versioning and a reimport recommendation for older data lacking the required context.

Source weighting and provenance

Each source tool has one fixed weight, initially 1. Distinct scan scopes are averaged equally inside that weight; the overall value is the weighted mean of scoreable source contributions. File count and check count do not increase a tool's overall weight. Contributions with no score remain explicitly visible. Incomplete contributions make the displayed index provisional.

Same-run representations are combined only with a verified tool, assessed target, benchmark scope, exact reported timestamp and processing identity, or an identical SHA-256 and processing identity. Preserve the contributing file, run and occurrence references. Conflicting native scores or result populations make the combined contribution unscoreable. Source rule/outcome signatures also detect different results with coincidentally equal totals.

When a scanner exports one native score-only summary beside its finding file, the shared scan-run and processing identity may join those complementary files. The native score remains the contribution and the finding file supplies the passed/deviation population. This association requires exactly one native carrier and one unambiguous result population; ambiguous result files remain separate instead of borrowing the native value.

Within a known target and benchmark, use the latest reported scan time in the assessment. Never use import time or a filename to infer scan identity. Unknown identity/time remains disclosed. Original values and timestamps are unchanged.

The Assessment.scoringPolicy stores rule version, tool weights and default weight through the existing JSON/JSONB assessment contract. New scanner import assessments and new Data Hub timeslots initialize this field. Missing policies on historical assessments permanently mean the version 1.0.0 default, not an alias to a future current policy. Existing records are not migrated or rewritten. Unsupported policy versions do not produce a score. Round only at display time.

Presentation and comparisons

Overall Score is a mixed-method assessment index, not a percentage of security. Native risk models and pass rates are not methodologically equivalent or empirically calibrated. Security Domain values are explicitly labelled pass rates and are not presented as a numeric decomposition of this index.

Source Overview shows the actual contribution and its method. The Scoring methodology dialog shows per-scan contribution, native input, source weight, evaluated-check basis, reported scan time and any qualifications. Critical findings and check coverage remain separate from the index.

Measured score progress uses only the one-to-one intersection of identical assessed targets/benchmarks with unchanged rule/evaluation population, processing, engine, method and weight, a newer source observation, and unqualified evidence. Both values are recomputed on this shared basis; omitting a weak source cannot become measured improvement. Changed scope/method is visibly labelled on the current index.

The workspace projects the latest known prior evidence per source scope from persisted historical assessments. Missing-source open obligations retain their original occurrence, run, assessment and observation-time references across successive absent follow-ups. Historical scores are never copied into current scan measurements. Unavailable history blocks a verified-progress statement and is visible without preventing access to the current workspace.

This is a read-only score/provenance projection. The broader Phase-8 workflow for persisted carry-forward lifecycle states, operational backlog integration and audited source retirement remains a separate unfinished work package. No new source retirement is inferred from a selection/filter operation.

Manufacturer references

• Semperis scoring methodology: native posture and category scores have independent calculations.
• PingCastle Healthcheck: overall risk uses the maximum of four capped category values. Inversion is an explicit ASSESSOR index convention.
• CIS-CAT report documentation: scores are recommendation-based; scored recommendations currently have equal weight.

Remaining limitations

• No new environment-area weighting or automatic mapping of native totals to the eight regex-assigned Security Domains.
• Unknown source identity or engine/processing changes limit time comparisons.
• Missing-result diagnostics qualify native scores rather than silently replacing source data. A source's scoring conventions can differ from check coverage conventions, especially for execution errors.
• GPO-only history remains under the existing policy-comparison workflow.
• This change does not reimport, migrate or inspect real assessment contents.

Verification, 2026-09-10

npm test: 1,188 passed, eight optional PostgreSQL integration tests skipped; local-only security and design checks passed. Final focused run: 55 passed, including unchanged rule-population comparisons and persisted GPO weights. Final npm run build: client and server successful; existing bundle-size advisory remains. All content-level checks used synthetic fixtures. No real database changes, live content inspection, reimport or backend restart.

Follow-up for future scanner profiles: ten additional synthetic regressions cover the mandatory profile contract, native priority for a new scanner through dataset projection, score direction and zero, invalid/conflicting/foreign observations, unknown semantics, findings-only fallback and scoring versions. npm test: 1,198 passed, eight optional PostgreSQL tests skipped; local-only and design checks passed. npm run build: client and server successful.

Purple Knight companion-file follow-up: the score-only assessment summary is now associated with one unambiguous findings population from the exact same scan run and processing contract. The regression with 65 native posture, 96 passed and 19 deviations yields 65 as the dashboard contribution while retaining all 19 deviations. npm test: 1,199 passed, eight optional PostgreSQL tests skipped; security/design checks and both builds passed.

Purple Knight Entra follow-up 2026-09-22: profile 1.4.0 recognizes the native Entra ID assessment summary, persists its tenant, posture, counters and category scores, and labels the scan independently from Purple Knight for Active Directory. The same native-first rule applies within the Entra scope; the tenant name is its target identity. Historical 1.3.0 imports do not gain these facts at read time and require reimport of the original files.