108 lines
4.5 KiB
Markdown
108 lines
4.5 KiB
Markdown
# Regulation mapping — quick command reference
|
|
|
|
The regulation retrieval feature maps a finding definition to the best-matching
|
|
regulation units (e.g. ISO 27002 controls) using a Qdrant vector database and an
|
|
OpenAI-compatible embedding endpoint. It is **disabled unless configured** and is
|
|
backend-only.
|
|
|
|
> This is the "ingestion + retrieval" slice (Phase 13D). LLM selection, review
|
|
> workflow and UI are not part of it yet. See `docs/IMPLEMENTATION_PLAN.md`
|
|
> (Phase 13D) and `SECURITY.md` (Regulierungssuche-Grenze).
|
|
|
|
## Prerequisites
|
|
|
|
- Docker (Qdrant runs in a container).
|
|
- An OpenAI-compatible embedding endpoint serving **bge-m3** (multilingual,
|
|
1024 dimensions, no prefixes). Either:
|
|
- **llama.cpp**: download `bge-m3-FP16.gguf` (or `bge-m3-Q8_0.gguf`) from
|
|
`gpustack/bge-m3-GGUF`, then
|
|
`llama-server -m bge-m3-FP16.gguf --embedding --pooling cls -c 8192 -ub 8192 --port 8080`
|
|
→ base URL `http://127.0.0.1:8080/v1`.
|
|
- **Ollama**: `ollama pull bge-m3` → base URL `http://127.0.0.1:11434/v1`.
|
|
- The catalog file `iso27002-controls-embed.jsonl` — it contains **licensed**
|
|
text and must stay **outside** the repository (it lives under `Mapping-test/`,
|
|
which is git-ignored).
|
|
|
|
## One-time setup
|
|
|
|
```bash
|
|
# 1. Start Qdrant (reachable at http://127.0.0.1:6333)
|
|
npm run qdrant:up
|
|
|
|
# 2. Start the embedding model (separate terminal / service)
|
|
# llama.cpp:
|
|
llama-server -m bge-m3-FP16.gguf --embedding --pooling cls -c 8192 -ub 8192 --port 8080
|
|
# or Ollama: ollama pull bge-m3 && ollama serve (port 11434)
|
|
```
|
|
|
|
Add to `.env.postgres` (create it from `.env.postgres.example` if needed):
|
|
|
|
```ini
|
|
ASSESSOR_QDRANT_URL=http://127.0.0.1:6333
|
|
ASSESSOR_QDRANT_COLLECTION=regulation_catalog_v1
|
|
ASSESSOR_EMBEDDING_BASE_URL=http://127.0.0.1:8080/v1
|
|
# ^ llama.cpp on :8080; for Ollama use http://127.0.0.1:11434/v1
|
|
ASSESSOR_EMBEDDING_MODEL=bge-m3
|
|
ASSESSOR_EMBEDDING_DIMENSIONS=1024
|
|
ASSESSOR_REGMAP_CATALOG_VERSION=DIN EN ISO/IEC 27002:2024-01
|
|
# For multilingual-e5-* instead of bge-m3, the prefixes must end with a space:
|
|
# ASSESSOR_EMBEDDING_QUERY_PREFIX=query:
|
|
# ASSESSOR_EMBEDDING_PASSAGE_PREFIX=passage:
|
|
```
|
|
|
|
```bash
|
|
# 3. Embed the catalog into Qdrant (idempotent — safe to re-run)
|
|
npm run regmap:ingest "Mapping-test/test/iso27002-controls-embed.jsonl"
|
|
```
|
|
|
|
## Run the app
|
|
|
|
```bash
|
|
# Development (client on :5173, backend API on :4173, reads .env.postgres)
|
|
npm run dev
|
|
|
|
# Production-like (build, then serve everything on :4173)
|
|
npm run serve:local
|
|
```
|
|
|
|
## Verify
|
|
|
|
```bash
|
|
# Capability should report "ready" once Qdrant + embeddings + collection are set
|
|
curl http://127.0.0.1:4173/api/v1/regulation-mapping/capability
|
|
|
|
# Health also carries the capability
|
|
curl http://127.0.0.1:4173/api/v1/health
|
|
|
|
# Retrieve candidates for a finding (POST needs the same-origin mutation header)
|
|
curl -X POST http://127.0.0.1:4173/api/v1/regulation-mapping/retrieve -H "Content-Type: application/json" -H "X-Assessor-Request: 1" -d '{"definition":{"title":"RC4 encryption enabled on Domain Controller","description":"Weak cipher permitted on DC"}}'
|
|
```
|
|
|
|
Capability states: `disabled` (no embedding URL) · `misconfigured` (collection
|
|
missing, dimension/model mismatch, or endpoint unreachable) · `ready`.
|
|
|
|
## Command reference
|
|
|
|
| Command | What it does |
|
|
|---|---|
|
|
| `npm run qdrant:up` | Start the Qdrant container (detached). |
|
|
| `npm run qdrant:down` | Stop and remove the Qdrant container. |
|
|
| `npm run qdrant:logs` | Follow Qdrant logs. |
|
|
| `npm run regmap:ingest <file.jsonl>` | Embed the catalog and upsert into Qdrant. |
|
|
| `npm run dev` | Dev stack: client `:5173`, backend `:4173` (loads `.env.postgres`). |
|
|
| `npm run serve:local` | Build and serve the whole app on `:4173`. |
|
|
| `npm run start:local` | Serve an already-built app on `:4173`. |
|
|
| `npm run db:up` / `db:down` | Start / stop the local PostgreSQL container. |
|
|
| `npm test` / `npm run build` | Test suite / production build. |
|
|
|
|
## Notes
|
|
|
|
- Qdrant's data persists in the `assessor-qdrant-data` Docker volume across
|
|
`qdrant:down` / `qdrant:up`. To wipe it: `docker volume rm assessor_assessor-qdrant-data`.
|
|
- `ASSESSOR_EMBEDDING_DIMENSIONS` **must** match the embedding model's output
|
|
size, or the collection validation fails as `misconfigured`. Changing models
|
|
means re-ingesting (delete the collection or use a new `ASSESSOR_QDRANT_COLLECTION`).
|
|
- On this machine, the dev Qdrant compose publishes port `6333` on the WSL
|
|
distro interface so Windows `127.0.0.1:6333` reaches it (WSL2 does not forward a
|
|
`127.0.0.1`-bound container port). It stays isolated from the LAN in WSL2 NAT mode.
|