168 lines
10 KiB
Markdown
168 lines
10 KiB
Markdown
# ASSESSOR scoring rule 1.0.0
|
||
|
||
Decision: 2026-09-10, explicitly approved by the user. Implementation belongs to
|
||
Phase 7/13C (Client Info and scanner context), with score-comparison safeguards
|
||
under Phase 8. This decision supersedes the display-only native-score policy.
|
||
|
||
## Scan contribution
|
||
|
||
Use a valid native score from the exact import file and scan run. Otherwise use
|
||
`100 * passed / (passed + deviations)` when the result population supports a
|
||
pass rate. Never combine the native score and the pass rate of the same scan.
|
||
|
||
| Source | Native contribution | Fallback |
|
||
| --- | --- | --- |
|
||
| Purple Knight AD | Native AD posture, 0–100 | Pass rate |
|
||
| CIS-CAT | Native benchmark score, 0–100 | Pass rate |
|
||
| PingCastle Healthcheck | `100 - native risk`, bounded 0–100 | No pass rate from findings-only reports |
|
||
| Maester | No native numeric score assumed | Pass rate |
|
||
| GPO Analysis | None | Existing context-aware pass rate |
|
||
| Additional registered scanners | Native overall score declared by their profile | Declared result-population fallback |
|
||
| Unrecognized tabular sources | No native score inferred | Provisional pass rate only with evidence of passed results |
|
||
|
||
The Purple Knight AD score must not stand for a hybrid report containing other
|
||
native posture scopes. Such reports use the disclosed fallback until supported
|
||
scope-specific contributions are available. A missing native score on an old
|
||
import requires the relevant context to be imported; no score is fabricated.
|
||
|
||
Null, blank, boolean, nonnumeric, out-of-range and conflicting native values
|
||
remain unavailable. A valid zero is a real score. Invalid native data and source
|
||
integrity diagnostics are visible in the calculation details.
|
||
|
||
Passed and deviations use normalized outcomes. Not applicable is excluded.
|
||
Execution errors, not run and unknown results do not pass and do not add a
|
||
security failure. They affect known-check coverage. GPO context states retain
|
||
their existing authoritative disposition; no severity or missing-setting
|
||
penalty is invented. Findings-only input cannot establish a pass denominator.
|
||
|
||
## Future scanner support
|
||
|
||
Every new or extended scanner profile must first examine the supported report
|
||
formats for a usable native overall score. `ToolProfileDefinition.securityScoring`
|
||
is mandatory: declare exact mapped observation properties, overall scope,
|
||
0–100 scale, direction, incompatible scopes and fallback population, or record
|
||
an explicit reason why native scoring is unavailable. A new profile registered
|
||
in the profile catalog automatically uses the shared native-first calculation;
|
||
there is no separate hardcoded scanner list in that calculation.
|
||
|
||
The extractor must retain the original score and its exact source path as
|
||
file/run-bound report context. Different native scales or ambiguous scope need
|
||
an explicitly documented and tested mapping before they are usable. Arbitrary
|
||
fields named score, category scores and finding weights do not qualify. Until
|
||
an unfamiliar format has such a profile, its native semantics remain unknown.
|
||
|
||
A valid native score, including zero, takes priority even for findings-only
|
||
reports. Missing, invalid, conflicting or unrelated native observations lead
|
||
to the declared fallback; without a defensible result population the contribution
|
||
remains unavailable. Scoring-definition versions participate in comparison
|
||
compatibility. Extend synthetic format fixtures and verify native priority,
|
||
provenance, invalid values and fallback whenever adding scanner support.
|
||
|
||
This extension moves existing declarations into their profiles without changing
|
||
their numeric methods or import mapping versions. No historic imports are
|
||
rewritten. New extraction or semantic changes still require explicit versioning
|
||
and a reimport recommendation for older data lacking the required context.
|
||
|
||
## Source weighting and provenance
|
||
|
||
Each source tool has one fixed weight, initially 1. Distinct scan scopes are
|
||
averaged equally inside that weight; the overall value is the weighted mean of
|
||
scoreable source contributions. File count and check count do not increase a
|
||
tool's overall weight. Contributions with no score remain explicitly visible.
|
||
Incomplete contributions make the displayed index provisional.
|
||
|
||
Same-run representations are combined only with a verified tool, assessed
|
||
target, benchmark scope, exact reported timestamp and processing identity, or
|
||
an identical SHA-256 and processing identity. Preserve the contributing file,
|
||
run and occurrence references. Conflicting native scores or result populations
|
||
make the combined contribution unscoreable. Source rule/outcome signatures
|
||
also detect different results with coincidentally equal totals.
|
||
|
||
When a scanner exports one native score-only summary beside its finding file,
|
||
the shared scan-run and processing identity may join those complementary files.
|
||
The native score remains the contribution and the finding file supplies the
|
||
passed/deviation population. This association requires exactly one native
|
||
carrier and one unambiguous result population; ambiguous result files remain
|
||
separate instead of borrowing the native value.
|
||
|
||
Within a known target and benchmark, use the latest reported scan time in the
|
||
assessment. Never use import time or a filename to infer scan identity. Unknown
|
||
identity/time remains disclosed. Original values and timestamps are unchanged.
|
||
|
||
The `Assessment.scoringPolicy` stores rule version, tool weights and default
|
||
weight through the existing JSON/JSONB assessment contract. New scanner import
|
||
assessments and new Data Hub timeslots initialize this field. Missing policies
|
||
on historical assessments permanently mean the version 1.0.0 default, not an
|
||
alias to a future current policy. Existing records are not migrated or rewritten.
|
||
Unsupported policy versions do not produce a score. Round only at display time.
|
||
|
||
## Presentation and comparisons
|
||
|
||
Overall Score is a mixed-method assessment index, not a percentage of security.
|
||
Native risk models and pass rates are not methodologically equivalent or
|
||
empirically calibrated. Security Domain values are explicitly labelled pass
|
||
rates and are not presented as a numeric decomposition of this index.
|
||
|
||
Source Overview shows the actual contribution and its method. The Scoring
|
||
methodology dialog shows per-scan contribution, native input, source weight,
|
||
evaluated-check basis, reported scan time and any qualifications. Critical
|
||
findings and check coverage remain separate from the index.
|
||
|
||
Measured score progress uses only the one-to-one intersection of identical
|
||
assessed targets/benchmarks with unchanged rule/evaluation population, processing, engine, method and
|
||
weight, a newer source observation, and unqualified evidence. Both values are
|
||
recomputed on this shared basis; omitting a weak source cannot become measured
|
||
improvement. Changed scope/method is visibly labelled on the current index.
|
||
|
||
The workspace projects the latest known prior evidence per source scope from
|
||
persisted historical assessments. Missing-source open obligations retain their
|
||
original occurrence, run, assessment and observation-time references across
|
||
successive absent follow-ups. Historical scores are never copied into current
|
||
scan measurements. Unavailable history blocks a verified-progress statement
|
||
and is visible without preventing access to the current workspace.
|
||
|
||
This is a read-only score/provenance projection. The broader Phase-8 workflow
|
||
for persisted carry-forward lifecycle states, operational backlog integration
|
||
and audited source retirement remains a separate unfinished work package.
|
||
No new source retirement is inferred from a selection/filter operation.
|
||
|
||
## Manufacturer references
|
||
|
||
- [Semperis scoring methodology](https://www.semperis.com/blog/pk-scoring-improves-understanding-of-identity-system-security-vulnerabilities/): native posture and category scores have independent calculations.
|
||
- [PingCastle Healthcheck](https://www.pingcastle.com/documentation/healthcheck/): overall risk uses the maximum of four capped category values. Inversion is an explicit ASSESSOR index convention.
|
||
- [CIS-CAT report documentation](https://ciscat-assessor.docs.cisecurity.org/en/latest/ReportHTML/): scores are recommendation-based; scored recommendations currently have equal weight.
|
||
|
||
## Remaining limitations
|
||
|
||
- No new environment-area weighting or automatic mapping of native totals to
|
||
the eight regex-assigned Security Domains.
|
||
- Unknown source identity or engine/processing changes limit time comparisons.
|
||
- Missing-result diagnostics qualify native scores rather than silently
|
||
replacing source data. A source's scoring conventions can differ from check
|
||
coverage conventions, especially for execution errors.
|
||
- GPO-only history remains under the existing policy-comparison workflow.
|
||
- This change does not reimport, migrate or inspect real assessment contents.
|
||
|
||
## Verification, 2026-09-10
|
||
|
||
`npm test`: 1,188 passed, eight optional PostgreSQL integration tests skipped;
|
||
local-only security and design checks passed. Final focused run: 55 passed,
|
||
including unchanged rule-population comparisons and persisted GPO weights.
|
||
Final `npm run build`: client and server successful; existing bundle-size
|
||
advisory remains. All content-level checks used synthetic fixtures. No real
|
||
database changes, live content inspection, reimport or backend restart.
|
||
|
||
Follow-up for future scanner profiles: ten additional synthetic regressions
|
||
cover the mandatory profile contract, native priority for a new scanner through
|
||
dataset projection, score direction and zero, invalid/conflicting/foreign
|
||
observations, unknown semantics, findings-only fallback and scoring versions.
|
||
`npm test`: 1,198 passed, eight optional PostgreSQL tests skipped; local-only
|
||
and design checks passed. `npm run build`: client and server successful.
|
||
|
||
Purple Knight companion-file follow-up: the score-only assessment summary is
|
||
now associated with one unambiguous findings population from the exact same
|
||
scan run and processing contract. The regression with 65 native posture,
|
||
96 passed and 19 deviations yields 65 as the dashboard contribution while
|
||
retaining all 19 deviations. `npm test`: 1,199 passed, eight optional
|
||
PostgreSQL tests skipped; security/design checks and both builds passed.
|