log-guardian/project-docs/design/phase-1-plan.md

Phase 1 Plan

Goal: make log monitoring useful in production without enabling dangerous automation.

Phase 1 targets a production Portainer deployment of Log Guardian. Loki will likely run on a separate server and use an object-storage backend, probably MinIO backed by NAS storage. Log Guardian must remain safe-by-default: authenticated UI, real alert delivery, incident cooldowns, Loki validation tooling, dry-run remediation evidence, and local-only AI analysis.

Decisions made

• Deployment target: production Portainer.
• Loki location: separate server from Log Guardian.
• Loki storage direction: object storage via MinIO, with MinIO data on NAS-backed storage.
• UI auth: single admin user using basic auth.
• Health endpoint: /healthz remains unauthenticated.
• First alert channel: self-hosted ntfy.
• Additional alert channel types should be supported/configurable from UI, not ntfy-only.
• Incident cooldown: per rule, default 1h.
• AI: local-only Ollama is approved; no external/cloud AI providers.
• Remediation: Phase 1 should implement permission modeling and dry-run evidence, not dangerous automatic actions.
• Phase 1 completion requires Docker build and real alert delivery validation.

1. Production deployment architecture

Log Guardian

Run Log Guardian as a Portainer-managed service/container.

Required mounts/config:

• Persistent local config/data file mount for LOG_GUARDIAN_DATA_PATH.
• Environment configuration for Loki endpoint, auth, alert defaults, and optional Ollama endpoint.
• Network path to the remote Loki server.
• Optional network path to self-hosted ntfy, Gotify, SMTP relay, webhook targets, and Ollama.

Loki

Loki will run on a different server. Recommended production direction:

• Loki single-binary or simple scalable deployment depending on expected volume.
• Object storage backend: S3-compatible MinIO.
• MinIO data stored on NAS-backed storage.
• Retention configured explicitly.
• Promtail or Grafana Alloy deployed on every Docker/Portainer host that needs logs collected.

Recommended initial storage sizing:

• Start with 50 GB usable object storage for modest private infrastructure.
• Use 14-day retention initially.
• Monitor ingestion and object growth for 7 days before increasing retention.
• Increase to 100-250 GB+ if services are noisy or retention must be longer.

Planning assumptions to validate:

• NAS latency is acceptable for MinIO/Loki workload.
• MinIO has backups/snapshots appropriate for desired retention.
• Loki retention deletes old chunks/index data correctly.
• Log Guardian can reach Loki over a private network/VPN only.

2. Loki ingestion, label discovery, and validation

Because actual labels are not known yet, Phase 1 must include label discovery and query testing.

Collection strategy

Use Promtail or Grafana Alloy on Docker/Portainer hosts. Target labels should include stable, low-cardinality dimensions such as:

• host
• container
• service
• stack
• stream
• compose_project or equivalent
• compose_service or equivalent

Avoid high-cardinality labels, including request IDs, user IDs, IP addresses, paths, error messages, or raw log content.

Log Guardian Loki diagnostics

Add authenticated UI/API support for:

• Loki readiness/status check.
• List labels from /loki/api/v1/labels.
• List values for a selected label from /loki/api/v1/label/{name}/values.
• Test a LogQL query before saving a rule.
• Show count and sanitized sample lines for a test query.

Implementation notes:

• Limit query time range and result count.
• Do not log raw sample payloads by default.
• Redact obvious secrets from displayed query samples where practical.
• Handle Loki tenant ID and basic auth from existing Loki config.

3. UI authentication

Add single-admin basic auth.

Configuration:

• LOG_GUARDIAN_AUTH_USERNAME
• LOG_GUARDIAN_AUTH_PASSWORD

Rules:

• /healthz is unauthenticated.
• All other UI/API routes require auth.
• Use constant-time comparison for credentials.
• Do not log usernames/passwords or auth headers.
• If credentials are unset, fail closed in production or clearly mark auth as disabled only for local development. Prefer fail closed for Portainer deployment.

4. Alert channels

Alert channels must be configurable from the UI and stored in local configuration. Secrets need careful handling.

Required first channel: self-hosted ntfy

Support fields:

• Name
• Enabled
• Server URL
• Topic
• Optional bearer token or token file reference
• Priority
• Tags

Validation:

• Send a test notification from the UI.
• Record delivery result without logging token values.

Additional Phase 1 channel types

Implement the data model and UI so multiple channel types are possible. Preferred Phase 1 support order:

1. ntfy - implement real delivery first.
2. gotify - real delivery if time allows; token-based API is straightforward.
3. generic_webhook - useful but must avoid leaking secrets; support explicit headers carefully.
4. smtp - useful, but can be deferred if it complicates TLS/auth validation.

Minimum acceptable Phase 1 scope:

• UI can create/edit/delete alert channels.
• UI can attach channels to rules.
• ntfy real delivery works.
• Other channel types may be present as disabled/scaffolded if not implemented, but the UI should clearly show unsupported status.

Secret handling:

• Do not print secrets in logs.
• Do not render stored secret values back into forms; show only configured/not configured.
• Prefer environment variable or mounted file references for sensitive channel tokens when possible.
• If secrets are stored in local JSON, the file must be written with 0600 permissions and documented as sensitive.

5. Rule evaluation and incident cooldown

Add per-rule cooldown.

Rule field:

"cooldown": "1h"

Default behavior:

• If unset, cooldown is 1h.
• If a rule fires outside cooldown:
  • create incident;
  • send configured alert channels;
  • record dry-run remediation evidence if actions are configured.
• If a rule fires during cooldown:
  • update rule last checked/matched/count fields;
  • do not create a duplicate incident;
  • do not send another alert;
  • increment or record suppressed count if implemented.

6. Remediation/action model

Phase 1 should not execute destructive remediation. Implement guardrails and dry-run evidence.

Recommended Phase 1 action types:

• generic_webhook: dry-run by default; real execution can be deferred or restricted.
• portainer_restart: dry-run only in Phase 1.
• allowlisted_command: dry-run only in Phase 1; command IDs only, never arbitrary shell from UI/AI.
• record_recommendation: safe action that stores an advisory recommendation on the incident.

Guardrails:

• Global dry-run defaults true.
• Per-rule action must be enabled.
• Per-action dry-run should be explicit or inherited from global dry-run.
• AI may recommend actions but cannot execute them.
• No arbitrary command execution.
• No Portainer write API calls in Phase 1 unless separately approved.

Validation evidence:

• At least one rule with a configured dry-run action.
• Incident shows what would have happened.
• Logs do not expose secrets or raw full log payloads.

7. Local-only AI analysis

Local-only Ollama is approved for Phase 1. External/cloud AI providers are not approved.

Configuration:

• LOG_GUARDIAN_ANALYSIS_PROVIDER=none|ollama
• LOG_GUARDIAN_OLLAMA_URL=http://ollama:11434
• LOG_GUARDIAN_OLLAMA_MODEL=<model>

Default:

• LOG_GUARDIAN_ANALYSIS_PROVIDER=none

Rules:

• AI analysis remains opt-in per rule.
• AI findings are advisory only.
• Limit number of log lines sent to Ollama.
• Redact obvious secrets before prompt construction.
• Do not log prompts or raw log payloads by default.
• Store only concise summaries/recommendations in incidents.

Prompt/redaction design must be documented before enabling Ollama in production.

8. UI scope

Add UI support for:

• Login-protected dashboard.
• Rule create/edit with cooldown and alert channel selection.
• Alert channel create/edit/test.
• Loki status/label discovery/query test.
• Incident list showing alert delivery state and remediation dry-run evidence where available.

Keep the UI served by Go. Do not add a server-side JavaScript runtime.

9. Tests and validation

Required implementation tests:

• Auth middleware unit tests.
• Store tests for alert channels, cooldown fields, and secret-preserving updates.
• Loki client tests using httptest fixtures for labels, label values, and query range.
• Alert dispatcher tests for ntfy using httptest.
• Rule engine tests for cooldown suppression.
• Remediation dry-run tests.
• Optional Ollama provider tests using httptest only; no live AI dependency in unit tests.

Required validation before Phase 1 completion:

gofmt -w cmd internal
go test ./...
make validate-structure
make docker-build

Additional required evidence:

• Loki connectivity/query validation against the remote Loki server.
• Label discovery output documented without secrets.
• Self-hosted ntfy test alert received.
• A rule-generated ntfy alert received.
• Remediation dry-run evidence captured.
• Security/privacy review covering logs, prompts, secrets, alert tokens, and action permissions.

10. Required Phase 1 artifacts

Create/update these before declaring Phase 1 complete:

• project-docs/status/phase-1-summary.md
• project-docs/status/phase-1-test-results.md
• project-docs/status/phase-1-open-issues.md

11. Open planning items

• Final Loki topology: single-binary vs simple scalable.
• Final MinIO/NAS capacity and retention target after observing ingestion.
• Whether Gotify, generic webhook, or SMTP are implemented fully in Phase 1 or scaffolded after ntfy.
• Whether alert channel secrets are stored directly in local JSON or referenced via env/file paths.
• Exact Ollama model to use once Ollama is stable.
• Whether Log Guardian reaches Loki/ntfy/Ollama over LAN, VPN, or Docker overlay network.