200 lines
15 KiB
Markdown
200 lines
15 KiB
Markdown
# ASSESSOR scoring rule 1.0.0
|
||
|
||
Decision: 2026-09-10, explicitly approved by the user. Implementation belongs to
|
||
Phase 7/13C (Client Info and scanner context), with score-comparison safeguards
|
||
under Phase 8. This decision supersedes the display-only native-score policy.
|
||
|
||
## Comparison qualification, 2026-09-11
|
||
|
||
The numeric contribution policy below remains unchanged. Its comparison safeguards are not a completed acceptance of object-level remediation or an explicitly selected A/B report. Code review identified an evaluation-population key without asset identity, weaker pending-revalidation checks than score pairing, and last-known-source replacement that can omit unresolved obligations after partial intermediate scans. Original stored findings are not deleted. The current historical baseline is not necessarily the user's selected A.
|
||
|
||
[TIMESLOT_COMPARISON_PLAN.md](TIMESLOT_COMPARISON_PLAN.md) defines the open Phase-8 follow-up: verified object/scope populations, an explicit A/B basis, obligation-level history and separate eligibility rules for score deltas and remediation evidence. Matching `assessed` and `eligible` counts alone does not prove coverage of absent objects. Native totals must not be recomputed for an arbitrary smaller scope without separately retained contributions. Missing historical risks stay visible separately; no historical score is relabelled as a current measurement and no new penalty formula is introduced.
|
||
|
||
This is a planning clarification, not a runtime fix, processing-manifest change or completed verification. The earlier implementation/test notes below remain historical evidence for their stated scope; they do not satisfy the new follow-up's acceptance matrix.
|
||
|
||
## GPO scoring scope, decision 2026-09-16
|
||
|
||
The GPO contribution measures conformance of customer settings that are actually configured, independently comparable and inside an explicitly applicable Microsoft-baseline scope. It is not a measurement of complete Microsoft-baseline adoption. Recommended settings that are absent from the customer GPO set remain visible as `Not configured`, but they do not create operational findings, remediation obligations or score penalties in the normal assessment. Full baseline deployment is a separate explicitly commissioned hardening scope.
|
||
|
||
Clarification 2026-09-16: absence is reported within the analyzed backup scope. An applicable recommendation with no observed configured value in that imported set remains filterable as `Not configured`, including for a selected subset. This does not assert domain-wide absence or effective device state; the limited scope remains diagnostic evidence. The verdict remains excluded from the score.
|
||
|
||
Clarification later on 2026-09-16: `Not configured` is a score-neutral policy verdict. `Duplicates` and `Conflicts` are distinct diagnostic categories on a technical policy setting, not alternative verdicts. Roll up member comparisons once per technical setting: one independently comparable, in-scope configured member value with a valid `Fail` makes the complete policy-setting verdict `Fail`; retain the conflict/duplicate flags beside it. All applicable members passing yields `Passed`. Unresolved applicability, precedence or comparison yields `Review required`. Duplicate source rows, alternative presentations and diagnostic flags never add score weight.
|
||
|
||
This clarification supersedes only the earlier same-day interpretation that a conflict/duplicate container could remain score-neutral despite a proven member fail. It is binding for new GPO scoring work. Current runtime categories remain disjoint, so the rolled-up verdict plus diagnostic-flag model requires a dedicated result-population identity, synthetic conflict/duplicate fixtures and explicit verification before it may be claimed as implemented. If implementing it changes persisted evaluation semantics or score inputs, version the affected evaluator/scoring contract and follow the reimport/re-evaluation policy; do not reinterpret historical runs silently.
|
||
|
||
## Scan contribution
|
||
|
||
Use a valid native score from the exact import file and scan run. Otherwise use
|
||
`100 * passed / (passed + deviations)` when the result population supports a
|
||
pass rate. Never combine the native score and the pass rate of the same scan.
|
||
|
||
| Source | Native contribution | Fallback |
|
||
| --- | --- | --- |
|
||
| Purple Knight for Active Directory | Native AD posture, 0–100 | Pass rate |
|
||
| Purple Knight for Entra ID | Native Entra posture, 0–100 | Pass rate |
|
||
| CIS-CAT | Native benchmark score, 0–100 | Pass rate |
|
||
| PingCastle Healthcheck | `100 - native risk`, bounded 0–100 | No pass rate from findings-only reports |
|
||
| Maester | No native numeric score assumed | Pass rate |
|
||
| GPO Analysis | None | Pass rate of configured, independently comparable in-scope settings |
|
||
| Additional registered scanners | Native overall score declared by their profile | Declared result-population fallback |
|
||
| Unrecognized tabular sources | No native score inferred | Provisional pass rate only with evidence of passed results |
|
||
|
||
Purple Knight AD and Purple Knight Entra are separate scan scopes. Their native
|
||
posture values are bound respectively to the imported forest or Entra tenant
|
||
and must never be substituted for one another. A mixed report with conflicting
|
||
native scopes remains unavailable rather than being collapsed into one score.
|
||
A missing native score on an old import requires the relevant summary context
|
||
to be reimported; no score or tenant is fabricated.
|
||
|
||
Null, blank, boolean, nonnumeric, out-of-range and conflicting native values
|
||
remain unavailable. A valid zero is a real score. Invalid native data and source
|
||
integrity diagnostics are visible in the calculation details.
|
||
|
||
Passed and deviations use normalized outcomes. Not applicable is excluded.
|
||
Execution errors, not run and unknown results do not pass and do not add a
|
||
security failure. They affect known-check coverage. GPO context states retain
|
||
their existing authoritative disposition. `Not configured` does not enter the
|
||
GPO denominator. Conflict/duplicate flags do not enter independently; the
|
||
rolled-up policy verdict enters once and is `Fail` when any independently
|
||
comparable, in-scope configured member fails. No
|
||
severity or missing-setting penalty is invented. Findings-only input cannot
|
||
establish a pass denominator.
|
||
|
||
## Future scanner support
|
||
|
||
Every new or extended scanner profile must first examine the supported report
|
||
formats for a usable native overall score. `ToolProfileDefinition.securityScoring`
|
||
is mandatory: declare exact mapped observation properties, overall scope,
|
||
0–100 scale, direction, incompatible scopes and fallback population, or record
|
||
an explicit reason why native scoring is unavailable. A new profile registered
|
||
in the profile catalog automatically uses the shared native-first calculation;
|
||
there is no separate hardcoded scanner list in that calculation.
|
||
|
||
The extractor must retain the original score and its exact source path as
|
||
file/run-bound report context. Different native scales or ambiguous scope need
|
||
an explicitly documented and tested mapping before they are usable. Arbitrary
|
||
fields named score, category scores and finding weights do not qualify. Until
|
||
an unfamiliar format has such a profile, its native semantics remain unknown.
|
||
|
||
A valid native score, including zero, takes priority even for findings-only
|
||
reports. Missing, invalid, conflicting or unrelated native observations lead
|
||
to the declared fallback; without a defensible result population the contribution
|
||
remains unavailable. Scoring-definition versions participate in comparison
|
||
compatibility. Extend synthetic format fixtures and verify native priority,
|
||
provenance, invalid values and fallback whenever adding scanner support.
|
||
|
||
This extension moves existing declarations into their profiles without changing
|
||
their numeric methods or import mapping versions. No historic imports are
|
||
rewritten. New extraction or semantic changes still require explicit versioning
|
||
and a reimport recommendation for older data lacking the required context.
|
||
|
||
## Source weighting and provenance
|
||
|
||
Each source tool has one fixed weight, initially 1. Distinct scan scopes are
|
||
averaged equally inside that weight; the overall value is the weighted mean of
|
||
scoreable source contributions. File count and check count do not increase a
|
||
tool's overall weight. Contributions with no score remain explicitly visible.
|
||
Incomplete contributions make the displayed index provisional.
|
||
|
||
Same-run representations are combined only with a verified tool, assessed
|
||
target, benchmark scope, exact reported timestamp and processing identity, or
|
||
an identical SHA-256 and processing identity. Preserve the contributing file,
|
||
run and occurrence references. Conflicting native scores or result populations
|
||
make the combined contribution unscoreable. Source rule/outcome signatures
|
||
also detect different results with coincidentally equal totals.
|
||
|
||
When a scanner exports one native score-only summary beside its finding file,
|
||
the shared scan-run and processing identity may join those complementary files.
|
||
The native score remains the contribution and the finding file supplies the
|
||
passed/deviation population. This association requires exactly one native
|
||
carrier and one unambiguous result population; ambiguous result files remain
|
||
separate instead of borrowing the native value.
|
||
|
||
Within a known target and benchmark, use the latest reported scan time in the
|
||
assessment. Never use import time or a filename to infer scan identity. Unknown
|
||
identity/time remains disclosed. Original values and timestamps are unchanged.
|
||
|
||
The `Assessment.scoringPolicy` stores rule version, tool weights and default
|
||
weight through the existing JSON/JSONB assessment contract. New scanner import
|
||
assessments and new Data Hub timeslots initialize this field. Missing policies
|
||
on historical assessments permanently mean the version 1.0.0 default, not an
|
||
alias to a future current policy. Existing records are not migrated or rewritten.
|
||
Unsupported policy versions do not produce a score. Round only at display time.
|
||
|
||
## Presentation and comparisons
|
||
|
||
Overall Score is a mixed-method assessment index, not a percentage of security.
|
||
Native risk models and pass rates are not methodologically equivalent or
|
||
empirically calibrated. Security Domain values are explicitly labelled pass
|
||
rates and are not presented as a numeric decomposition of this index.
|
||
|
||
Source Overview shows the actual contribution and its method. The Scoring
|
||
methodology dialog shows per-scan contribution, native input, source weight,
|
||
evaluated-check basis, reported scan time and any qualifications. Critical
|
||
findings and check coverage remain separate from the index.
|
||
|
||
Measured score progress uses only the one-to-one intersection of identical
|
||
assessed targets/benchmarks with unchanged rule/evaluation population, processing, engine, method and
|
||
weight, a newer source observation, and unqualified evidence. Both values are
|
||
recomputed on this shared basis; omitting a weak source cannot become measured
|
||
improvement. Changed scope/method is visibly labelled on the current index.
|
||
|
||
The workspace projects the latest known prior evidence per source scope from
|
||
persisted historical assessments. Missing-source open obligations retain their
|
||
original occurrence, run, assessment and observation-time references across
|
||
successive absent follow-ups. Historical scores are never copied into current
|
||
scan measurements. Unavailable history blocks a verified-progress statement
|
||
and is visible without preventing access to the current workspace.
|
||
|
||
This is a read-only score/provenance projection. The broader Phase-8 workflow
|
||
for persisted carry-forward lifecycle states, operational backlog integration
|
||
and audited source retirement remains a separate unfinished work package.
|
||
No new source retirement is inferred from a selection/filter operation.
|
||
|
||
## Manufacturer references
|
||
|
||
- [Semperis scoring methodology](https://www.semperis.com/blog/pk-scoring-improves-understanding-of-identity-system-security-vulnerabilities/): native posture and category scores have independent calculations.
|
||
- [PingCastle Healthcheck](https://www.pingcastle.com/documentation/healthcheck/): overall risk uses the maximum of four capped category values. Inversion is an explicit ASSESSOR index convention.
|
||
- [CIS-CAT report documentation](https://ciscat-assessor.docs.cisecurity.org/en/latest/ReportHTML/): scores are recommendation-based; scored recommendations currently have equal weight.
|
||
|
||
## Remaining limitations
|
||
|
||
- No new environment-area weighting or automatic mapping of native totals to
|
||
the eight regex-assigned Security Domains.
|
||
- Unknown source identity or engine/processing changes limit time comparisons.
|
||
- Missing-result diagnostics qualify native scores rather than silently
|
||
replacing source data. A source's scoring conventions can differ from check
|
||
coverage conventions, especially for execution errors.
|
||
- GPO-only history remains under the existing policy-comparison workflow.
|
||
- This change does not reimport, migrate or inspect real assessment contents.
|
||
|
||
## Verification, 2026-09-10
|
||
|
||
`npm test`: 1,188 passed, eight optional PostgreSQL integration tests skipped;
|
||
local-only security and design checks passed. Final focused run: 55 passed,
|
||
including unchanged rule-population comparisons and persisted GPO weights.
|
||
Final `npm run build`: client and server successful; existing bundle-size
|
||
advisory remains. All content-level checks used synthetic fixtures. No real
|
||
database changes, live content inspection, reimport or backend restart.
|
||
|
||
Follow-up for future scanner profiles: ten additional synthetic regressions
|
||
cover the mandatory profile contract, native priority for a new scanner through
|
||
dataset projection, score direction and zero, invalid/conflicting/foreign
|
||
observations, unknown semantics, findings-only fallback and scoring versions.
|
||
`npm test`: 1,198 passed, eight optional PostgreSQL tests skipped; local-only
|
||
and design checks passed. `npm run build`: client and server successful.
|
||
|
||
Purple Knight companion-file follow-up: the score-only assessment summary is
|
||
now associated with one unambiguous findings population from the exact same
|
||
scan run and processing contract. The regression with 65 native posture,
|
||
96 passed and 19 deviations yields 65 as the dashboard contribution while
|
||
retaining all 19 deviations. `npm test`: 1,199 passed, eight optional
|
||
PostgreSQL tests skipped; security/design checks and both builds passed.
|
||
|
||
Purple Knight Entra follow-up 2026-09-22: profile 1.4.0 recognizes the native
|
||
`Entra ID` assessment summary, persists its tenant, posture, counters and
|
||
category scores, and labels the scan independently from Purple Knight for
|
||
Active Directory. The same native-first rule applies within the Entra scope;
|
||
the tenant name is its target identity. Historical 1.3.0 imports do not gain
|
||
these facts at read time and require reimport of the original files.
|