log-guardian/project-docs/design/phase-1-plan.md

# Phase 1 Plan

Goal: make log monitoring useful in production without enabling dangerous automation.

Phase 1 targets a production Portainer deployment of Log Guardian. Loki will likely run on a separate server and use an object-storage backend, probably MinIO backed by NAS storage. Log Guardian must remain safe-by-default: authenticated UI, real alert delivery, incident cooldowns, Loki validation tooling, dry-run remediation evidence, and local-only AI analysis.

## Decisions made

- Deployment target: production Portainer.
- Loki location: separate server from Log Guardian.
- Loki storage direction: object storage via MinIO, with MinIO data on NAS-backed storage.
- UI auth: single admin user using basic auth.
- Health endpoint: `/healthz` remains unauthenticated.
- First alert channel: self-hosted ntfy.
- Additional alert channel types should be supported/configurable from UI, not ntfy-only.
- Incident cooldown: per rule, default `1h`.
- AI: local-only Ollama is approved; no external/cloud AI providers.
- Remediation: Phase 1 should implement permission modeling and dry-run evidence, not dangerous automatic actions.
- Phase 1 completion requires Docker build and real alert delivery validation.

## 1. Production deployment architecture

### Log Guardian

Run Log Guardian as a Portainer-managed service/container.

Required mounts/config:

- Persistent local config/data file mount for `LOG_GUARDIAN_DATA_PATH`.
- Environment configuration for Loki endpoint, auth, alert defaults, and optional Ollama endpoint.
- Network path to the remote Loki server.
- Optional network path to self-hosted ntfy, Gotify, SMTP relay, webhook targets, and Ollama.

### Loki

Loki will run on a different server. Recommended production direction:

- Loki single-binary or simple scalable deployment depending on expected volume.
- Object storage backend: S3-compatible MinIO.
- MinIO data stored on NAS-backed storage.
- Retention configured explicitly.
- Promtail or Grafana Alloy deployed on every Docker/Portainer host that needs logs collected.

Recommended initial storage sizing:

- Start with 50 GB usable object storage for modest private infrastructure.
- Use 14-day retention initially.
- Monitor ingestion and object growth for 7 days before increasing retention.
- Increase to 100-250 GB+ if services are noisy or retention must be longer.

Planning assumptions to validate:

- NAS latency is acceptable for MinIO/Loki workload.
- MinIO has backups/snapshots appropriate for desired retention.
- Loki retention deletes old chunks/index data correctly.
- Log Guardian can reach Loki over a private network/VPN only.

## 2. Loki ingestion, label discovery, and validation

Because actual labels are not known yet, Phase 1 must include label discovery and query testing.

### Collection strategy

Use Promtail or Grafana Alloy on Docker/Portainer hosts. Target labels should include stable, low-cardinality dimensions such as:

- `host`
- `container`
- `service`
- `stack`
- `stream`
- `compose_project` or equivalent
- `compose_service` or equivalent

Avoid high-cardinality labels, including request IDs, user IDs, IP addresses, paths, error messages, or raw log content.

### Log Guardian Loki diagnostics

Add authenticated UI/API support for:

- Loki readiness/status check.
- List labels from `/loki/api/v1/labels`.
- List values for a selected label from `/loki/api/v1/label/{name}/values`.
- Test a LogQL query before saving a rule.
- Show count and sanitized sample lines for a test query.

Implementation notes:

- Limit query time range and result count.
- Do not log raw sample payloads by default.
- Redact obvious secrets from displayed query samples where practical.
- Handle Loki tenant ID and basic auth from existing Loki config.

## 3. UI authentication

Add single-admin basic auth.

Configuration:

- `LOG_GUARDIAN_AUTH_USERNAME`
- `LOG_GUARDIAN_AUTH_PASSWORD`

Rules:

- `/healthz` is unauthenticated.
- All other UI/API routes require auth.
- Use constant-time comparison for credentials.
- Do not log usernames/passwords or auth headers.
- If credentials are unset, fail closed in production or clearly mark auth as disabled only for local development. Prefer fail closed for Portainer deployment.

## 4. Alert channels

Alert channels must be configurable from the UI and stored in local configuration. Secrets need careful handling.

### Required first channel: self-hosted ntfy

Support fields:

- Name
- Enabled
- Server URL
- Topic
- Optional bearer token or token file reference
- Priority
- Tags

Validation:

- Send a test notification from the UI.
- Record delivery result without logging token values.

### Additional Phase 1 channel types

Implement the data model and UI so multiple channel types are possible. Preferred Phase 1 support order:

1. `ntfy` - implement real delivery first.
2. `gotify` - real delivery if time allows; token-based API is straightforward.
3. `generic_webhook` - useful but must avoid leaking secrets; support explicit headers carefully.
4. `smtp` - useful, but can be deferred if it complicates TLS/auth validation.

Minimum acceptable Phase 1 scope:

- UI can create/edit/delete alert channels.
- UI can attach channels to rules.
- ntfy real delivery works.
- Other channel types may be present as disabled/scaffolded if not implemented, but the UI should clearly show unsupported status.

Secret handling:

- Do not print secrets in logs.
- Do not render stored secret values back into forms; show only `configured`/`not configured`.
- Prefer environment variable or mounted file references for sensitive channel tokens when possible.
- If secrets are stored in local JSON, the file must be written with `0600` permissions and documented as sensitive.

## 5. Rule evaluation and incident cooldown

Add per-rule cooldown.

Rule field:

```json
"cooldown": "1h"
```

Default behavior:

- If unset, cooldown is `1h`.
- If a rule fires outside cooldown:
  - create incident;
  - send configured alert channels;
  - record dry-run remediation evidence if actions are configured.
- If a rule fires during cooldown:
  - update rule last checked/matched/count fields;
  - do not create a duplicate incident;
  - do not send another alert;
  - increment or record suppressed count if implemented.

## 6. Remediation/action model

Phase 1 should not execute destructive remediation. Implement guardrails and dry-run evidence.

Recommended Phase 1 action types:

- `generic_webhook`: dry-run by default; real execution can be deferred or restricted.
- `portainer_restart`: dry-run only in Phase 1.
- `allowlisted_command`: dry-run only in Phase 1; command IDs only, never arbitrary shell from UI/AI.
- `record_recommendation`: safe action that stores an advisory recommendation on the incident.

Guardrails:

- Global dry-run defaults true.
- Per-rule action must be enabled.
- Per-action dry-run should be explicit or inherited from global dry-run.
- AI may recommend actions but cannot execute them.
- No arbitrary command execution.
- No Portainer write API calls in Phase 1 unless separately approved.

Validation evidence:

- At least one rule with a configured dry-run action.
- Incident shows what would have happened.
- Logs do not expose secrets or raw full log payloads.

## 7. Local-only AI analysis

Local-only Ollama is approved for Phase 1. External/cloud AI providers are not approved.

Configuration:

- `LOG_GUARDIAN_ANALYSIS_PROVIDER=none|ollama`
- `LOG_GUARDIAN_OLLAMA_URL=http://ollama:11434`
- `LOG_GUARDIAN_OLLAMA_MODEL=<model>`

Default:

- `LOG_GUARDIAN_ANALYSIS_PROVIDER=none`

Rules:

- AI analysis remains opt-in per rule.
- AI findings are advisory only.
- Limit number of log lines sent to Ollama.
- Redact obvious secrets before prompt construction.
- Do not log prompts or raw log payloads by default.
- Store only concise summaries/recommendations in incidents.

Prompt/redaction design must be documented before enabling Ollama in production.

## 8. UI scope

Add UI support for:

- Login-protected dashboard.
- Rule create/edit with cooldown and alert channel selection.
- Alert channel create/edit/test.
- Loki status/label discovery/query test.
- Incident list showing alert delivery state and remediation dry-run evidence where available.

Keep the UI served by Go. Do not add a server-side JavaScript runtime.

## 9. Tests and validation

Required implementation tests:

- Auth middleware unit tests.
- Store tests for alert channels, cooldown fields, and secret-preserving updates.
- Loki client tests using `httptest` fixtures for labels, label values, and query range.
- Alert dispatcher tests for ntfy using `httptest`.
- Rule engine tests for cooldown suppression.
- Remediation dry-run tests.
- Optional Ollama provider tests using `httptest` only; no live AI dependency in unit tests.

Required validation before Phase 1 completion:

```sh
gofmt -w cmd internal
go test ./...
make validate-structure
make docker-build
```

Additional required evidence:

- Loki connectivity/query validation against the remote Loki server.
- Label discovery output documented without secrets.
- Self-hosted ntfy test alert received.
- A rule-generated ntfy alert received.
- Remediation dry-run evidence captured.
- Security/privacy review covering logs, prompts, secrets, alert tokens, and action permissions.

## 10. Required Phase 1 artifacts

Create/update these before declaring Phase 1 complete:

- `project-docs/status/phase-1-summary.md`
- `project-docs/status/phase-1-test-results.md`
- `project-docs/status/phase-1-open-issues.md`

## 11. Open planning items

- Final Loki topology: single-binary vs simple scalable.
- Final MinIO/NAS capacity and retention target after observing ingestion.
- Whether Gotify, generic webhook, or SMTP are implemented fully in Phase 1 or scaffolded after ntfy.
- Whether alert channel secrets are stored directly in local JSON or referenced via env/file paths.
- Exact Ollama model to use once Ollama is stable.
- Whether Log Guardian reaches Loki/ntfy/Ollama over LAN, VPN, or Docker overlay network.