134 lines
9.4 KiB
Markdown
134 lines
9.4 KiB
Markdown
# Phase 13D: ATT&CK and D3FEND references
|
|
|
|
## Scope and semantics
|
|
|
|
The initial implementation recognizes Enterprise ATT&CK technique/sub-technique IDs,
|
|
tactic IDs and unambiguous exact catalog titles, plus D3FEND defensive-technique IDs
|
|
and unambiguous exact titles. It processes retained `OriginalField` values using one
|
|
framework-specific engine, independently of the scanner. A framework-named header
|
|
declares its context; IDs in other fields are merely **mentioned**. Titles are only
|
|
matched in declared context. Whitespace and case normalization is allowed; fuzzy
|
|
matching, translations and LLM classification are not.
|
|
|
|
`active`, `deprecated`, `revoked` and `not-in-catalog` describe the lookup result.
|
|
An unknown ID is not claimed to be invalid. Placeholder text creates no reference.
|
|
An ambiguous title (for example PowerShell in current and historical ATT&CK entries)
|
|
does not select either ID. Unmatched original content remains available in the source
|
|
fields. No ATT&CK-to-D3FEND defensive inference or automatic correlation is performed.
|
|
|
|
The derived references never enter `FindingDefinition.frameworkReferences`, the
|
|
comparison identity engine, source guidance, severity, verdict, score, Security Domain,
|
|
review state or the external Result DB disclosure. Existing report-correlation semantics
|
|
remain separate. A shared reference describes a shared topic, not a proven common cause. Explicit user decision 2026-09-30: the context layer (Phase 13E) groups findings whose scanner declares the same active ATT&CK ID and shows exactly this shared declared topic, never a common cause or attack path; mentioned references stay out. Explicit user decision 2026-09-30: declared D3FEND countermeasures are grouped the same way, separately from ATT&CK, as the same countermeasure according to the scanners.
|
|
|
|
## Catalogs and reproducibility
|
|
|
|
- `attack-reference-v19.2.json`: 873 technique/tactic entries, including historical
|
|
deprecated and revoked entries. Source: MITRE's version-pinned Enterprise STIX 19.2.
|
|
- `d3fend-reference-v1.6.0.json`: 272 defensive technique entries (`D3-…`).
|
|
Source: <https://d3fend.mitre.org/ontologies/d3fend/1.6.0/d3fend.json>.
|
|
D3A/D3E entities and inferred cross-framework relationships are outside this slice.
|
|
- Each asset contains its original download URL, source SHA-256 and MITRE copyright/
|
|
license notice. Snapshot SHA-256 covers the complete canonical JSON catalog used by
|
|
recognition, including entries, version, provenance and notice.
|
|
- `scripts/update-reference-assets.mts` regenerates the compact assets from official
|
|
source files. Recognition and reprocessing require no external network.
|
|
- The existing **Update from MITRE** downloads ATT&CK and atomically stores both its
|
|
graph and complete recognition catalog. An equal-version refresh can fill in missing
|
|
recognition metadata from an older ASSESSOR installation. A central update makes a
|
|
new catalog available; it does not silently replace previous reference runs.
|
|
- D3FEND is bundled at 1.6.0 in this initial release; no D3FEND runtime update UI has
|
|
been added. **Download from MITRE** and manual **Import JSON file** remain ATT&CK
|
|
matrix actions. The manual browser override affects the displayed matrix and transient
|
|
previews; stored recognition always uses the backend's versioned catalogs.
|
|
|
|
## Storage and execution
|
|
|
|
Migration `023_reference_recognition` adds immutable public `assessor_reference_catalogs`,
|
|
append-only client/assessment/scan-bound `assessor_reference_runs` and their catalog
|
|
membership table. 021/022 are already reserved in the canonical local datastore by
|
|
timeslot-comparison and obligation work; those migrations are not modified here.
|
|
No separate client databases are created in this phase.
|
|
|
|
The normal browser batch-import transaction includes its reference run and catalog
|
|
snapshots. A recognition failure therefore rolls back the batch instead of reporting a
|
|
partially completed import. The low-level staging API and pre-existing scans can be
|
|
indexed with the explicit client action below; the UI never silently fabricates a stored
|
|
run for them.
|
|
|
|
**Data Hub → client → Catalog references → Update references** processes all current
|
|
scanner runs in that client across its timeslots, from their retained fields. The button
|
|
shows available catalog versions and pending scan count. Its guarded endpoint is
|
|
`POST /api/v1/projects/:projectId/references/recognize`; status is available through
|
|
`GET /api/v1/projects/:projectId/references`. Concurrent requests for one client share
|
|
one job; unchanged input, catalogs and recognizer version are idempotent and do not
|
|
bump repository revision. Individual scans are committed atomically; after interruption,
|
|
retry skips scans already completed. This initial synchronous route is not a background
|
|
job scheduler; very large clients can require a later queued implementation.
|
|
|
|
Each result retains occurrence and file identity, source file hash if available, source
|
|
position, field key/label/path, JSON value pointer, exact matched substring and offsets.
|
|
Offsets are zero-based UTF-16 indices into the retained `rawValue`, when present, or the
|
|
individual string leaf of `value`; the end is exclusive. Each run retains recognizer
|
|
version, timestamp, source-input hash and exact catalog hashes. Older runs remain
|
|
exportable. Project export includes every referenced compact catalog snapshot, allowing
|
|
reference history to be read without the central catalog. Original full STIX/JSON-LD
|
|
downloads are not needed to replay recognition against these exact compact inputs.
|
|
|
|
## Presentation and boundaries
|
|
|
|
The Data Hub client overview includes reference readiness across every current timeslot.
|
|
Pending recognition or an outdated recognizer/catalog produces the existing yellow
|
|
update point and highlighted **Manage**, including before selecting the client. An
|
|
unavailable status remains a review notice; checking is neutral and never claims readiness.
|
|
An actual recognition failure is red. Existing scanner/GPO failures retain priority.
|
|
Status summaries load separately from overview metadata with at most three concurrent
|
|
requests, refresh on import/data changes and successful catalog updates, and reject stale
|
|
responses after a newer panel result. There is no added polling or persistent job state.
|
|
|
|
**Catalog references** is in the left client navigation immediately above **Timeslots**,
|
|
including while editing client master data. By explicit user decision on 2026-09-26,
|
|
**Update references** is red when work or retry is needed and yellow otherwise.
|
|
The generic explanation is omitted. The first line says **Please update**, **Up to date**
|
|
with **No action needed.**, or **Update successful** after execution and a fresh status
|
|
check confirms no remaining work. ATT&CK and D3FEND versions stay visible. Empty clients,
|
|
loading and failures use explicit separate messages and never claim success.
|
|
The section starts collapsed for a current client and opens automatically for pending
|
|
work or a failed status/update. Small green dots precede the heading and confirmed
|
|
current/success state; required action uses orange and unknown/empty/processing states
|
|
stay neutral. Catalog versions remain outside the disclosure and visible while collapsed.
|
|
After updating, the success message remains open; manual and keyboard toggling is retained.
|
|
Successful recognition updates the client card and its loaded workspace. A failed status
|
|
request offers **Retry status**. The navigation scrolls as one area below the client header
|
|
so the added controls do not make timeslots inaccessible on short screens.
|
|
|
|
Pathfinder consumes persisted active references and frozen tactic membership, with
|
|
declared ATT&CK mapped as native and mentioned ATT&CK mapped as derived. Unknown,
|
|
revoked and deprecated matches are visible in the Finding Reader but do not form active
|
|
graph relations. Unindexed stored occurrences remain unmapped until explicitly indexed.
|
|
Transient, unsaved previews use the same pure recognizer with their selected matrix.
|
|
Original raw references remain readable. The Finding Reader adds a provenance disclosure
|
|
after explanatory content; its Problem/Next step, Results and work tracking are unchanged.
|
|
|
|
Broader 13D work (other frameworks, profile-declared ambiguous contexts, standalone
|
|
reference search/filtering, technical identity resolution, correlation UI, catalog-version
|
|
selection and per-client database migration) remains planned separately.
|
|
|
|
## Verification
|
|
|
|
Synthetic domain, full batch HTTP import, client-button/Reader interaction and Pathfinder
|
|
tests cover declared/mentioned evidence, unknown IDs, exact offsets, nested values,
|
|
historical/ambiguous entries, client isolation, idempotence, source immutability and
|
|
catalog updates without retrospective changes. The opt-in PostgreSQL test uses only
|
|
generated synthetic IDs in the canonical datastore and deletes that exact test project.
|
|
No tenant report contents are inspected.
|
|
|
|
Verified release: **2.8.0**. Both production builds, UI/API version contract, local-only
|
|
and design checks pass. Synthetic Light/Dark checks include a 480px Reader. The full
|
|
suite has 1,569 passing and 14 skipped tests, with six pre-existing failures across
|
|
GPO evaluation visibility (2), Security Domain navigation (1), AppLocker markup (1),
|
|
Result DB scope source-text expectations (1) and dashboard cache source-text expectations
|
|
(1). Their root causes are not all resolved by this separate reference implementation.
|
|
The PostgreSQL integration check additionally verifies source-complete history export
|
|
after a synthetic scan replacement. No existing customer scans were reprocessed.
|