commit 6f7549d439bac6bfc047decd748bcb810d1f9d54 Author: WayfinderAK Date: Fri Jul 3 20:14:06 2026 -0800 Implement phase 1 foundations diff --git a/.dockerignore b/.dockerignore new file mode 100644 index 0000000..9faded9 --- /dev/null +++ b/.dockerignore @@ -0,0 +1,5 @@ +.git +.env +.env.* +data +.pi-subagents diff --git a/.env.example b/.env.example new file mode 100644 index 0000000..59f4005 --- /dev/null +++ b/.env.example @@ -0,0 +1,15 @@ +LOG_GUARDIAN_ADDR=:8080 +LOG_GUARDIAN_DATA_PATH=./data/config.json +LOG_GUARDIAN_POLL_INTERVAL=1m +LOG_GUARDIAN_DRY_RUN_ACTIONS=true +LOG_GUARDIAN_AUTH_USERNAME=admin +LOG_GUARDIAN_AUTH_PASSWORD=change-me +LOKI_URL=http://localhost:3100 +LOKI_TENANT_ID= +LOKI_USERNAME= +LOKI_PASSWORD= +LOG_GUARDIAN_ANALYSIS_PROVIDER=none +LOG_GUARDIAN_ANALYSIS_ENDPOINT= +LOG_GUARDIAN_ANALYSIS_TOKEN= +LOG_GUARDIAN_OLLAMA_URL=http://ollama:11434 +LOG_GUARDIAN_OLLAMA_MODEL=llama3.1 diff --git a/.gitea/workflows/build-image.yml b/.gitea/workflows/build-image.yml new file mode 100644 index 0000000..7215d24 --- /dev/null +++ b/.gitea/workflows/build-image.yml @@ -0,0 +1,13 @@ +name: build-image + +on: + push: + branches: [main] + +jobs: + docker: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Build image + run: make docker-build IMAGE_TAG=${{ gitea.sha }} diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 0000000..fb12327 --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,17 @@ +name: ci + +on: + push: + pull_request: + +jobs: + test: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-go@v5 + with: + go-version: '1.23' + - run: make validate-structure + - run: make go-fmt + - run: make go-test diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..80ebb01 --- /dev/null +++ b/.gitignore @@ -0,0 +1,9 @@ +.env +.env.* +!.env.example +data/ +dist/ +build/ +coverage/ +*.log +.DS_Store diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..7957d97 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,33 @@ +# AGENTS.md + +This repository is for a private log monitoring, alerting, and controlled remediation service built around Loki. + +## Hard rules + +- Server-side code must be Go. +- The web UI is served by Go; do not add a server-side JavaScript runtime. +- Never log alert destination secrets, API keys, tokens, AI prompts containing private logs, or full raw log payloads unless explicitly configured for a local-only debug session. +- Do not add external cloud/AI/data integrations without explicit user approval. +- Automatic remediation must be opt-in per rule and should default to dry-run. +- AI analysis is advisory by default; AI-generated actions must not execute unless a rule explicitly allows that action and passes guardrails. +- Preserve local data ownership: runtime configuration is stored in mounted local files unless a later phase explicitly changes storage. + +## Required artifacts per phase + +- `project-docs/status/phase-N-summary.md` +- `project-docs/status/phase-N-test-results.md` +- `project-docs/status/phase-N-open-issues.md` + +## Required validation before advancing + +Run the checks required by the phase, including as applicable: + +- formatting/lint +- unit tests +- Docker build +- Loki query validation +- alert delivery validation +- remediation dry-run evidence +- security/privacy review for logs, prompts, secrets, and action permissions + +If any required check fails, the phase is not complete. diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..98c2b53 --- /dev/null +++ b/Makefile @@ -0,0 +1,31 @@ +IMAGE_TAG ?= local +REGISTRY ?= gitea.wayfinderak.com +IMAGE_NAMESPACE ?= wayfinderak +IMAGE_PREFIX ?= log-guardian +IMAGE_REF ?= $(REGISTRY)/$(IMAGE_NAMESPACE)/$(IMAGE_PREFIX):$(IMAGE_TAG) +DOCKER_BUILD_FLAGS ?= + +.PHONY: validate-structure go-fmt go-test docker-build ci-fast + +validate-structure: + @test -f AGENTS.md + @test -f README.md + @test -f project-docs/status/phase-0-summary.md + @test -f project-docs/status/phase-0-test-results.md + @test -f project-docs/status/phase-0-open-issues.md + @test -f project-docs/status/phase-1-summary.md + @test -f project-docs/status/phase-1-test-results.md + @test -f project-docs/status/phase-1-open-issues.md + @test -f deploy/loki/docker-compose.yml + @echo "Structure validation passed" + +go-fmt: + @test -z "$$(gofmt -l cmd internal)" + +go-test: + go test ./... + +docker-build: + docker build $(DOCKER_BUILD_FLAGS) -f docker/server.Dockerfile -t $(IMAGE_REF) . + +ci-fast: validate-structure go-fmt go-test docker-build diff --git a/README.md b/README.md new file mode 100644 index 0000000..7d6bae4 --- /dev/null +++ b/README.md @@ -0,0 +1,92 @@ +# Log Guardian + +Private Loki-backed service for log monitoring, alerting, and guarded remediation. + +Current implementation includes the Phase 1 foundations: + +- watches configured LogQL rules; +- records incidents when thresholds are met; +- enforces per-rule cooldowns; +- protects the UI with basic auth while leaving `/healthz` open; +- provides Loki label/query diagnostics; +- supports UI-configurable alert channels with real ntfy delivery; +- keeps remediation in dry-run by default; +- includes local-only Ollama analysis support, disabled by default. + +## Run locally + +```sh +cp .env.example .env +# edit env values, then export them or use your shell dotenv helper +go run ./cmd/server +``` + +Open . + +## Configuration + +| Variable | Default | Purpose | +| --- | --- | --- | +| `LOG_GUARDIAN_ADDR` | `:8080` | HTTP listen address. | +| `LOG_GUARDIAN_DATA_PATH` | `/data/config.json` | Local JSON config. | +| `LOG_GUARDIAN_POLL_INTERVAL` | `1m` | Rule evaluation interval. | +| `LOG_GUARDIAN_DRY_RUN_ACTIONS` | `true` | Prevents real remediation actions. | +| `LOG_GUARDIAN_AUTH_USERNAME` / `LOG_GUARDIAN_AUTH_PASSWORD` | empty | Required basic auth credentials for UI/API routes except `/healthz`. | +| `LOKI_URL` | `http://loki:3100` | Loki base URL. | +| `LOKI_TENANT_ID` | empty | Optional Loki tenant header. | +| `LOKI_USERNAME` / `LOKI_PASSWORD` | empty | Optional basic auth. | +| `LOG_GUARDIAN_ANALYSIS_PROVIDER` | `none` | `none` or local-only `ollama`. | +| `LOG_GUARDIAN_OLLAMA_URL` / `LOG_GUARDIAN_OLLAMA_MODEL` | `http://ollama:11434` / `llama3.1` | Local Ollama analysis config. | + +## Initial Loki setup + +A starter Loki + Promtail stack is in `deploy/loki/`: + +```sh +cd deploy/loki +docker compose up -d +``` + +Promtail is configured to discover Docker containers and attach useful labels: + +- `container` +- `service` +- `stack` +- `stream` + +Example LogQL rules: + +```logql +{service="api"} |= "error" +{stack="second-brain"} |~ "(?i)(panic|fatal|exception)" +{container=~".*postgres.*"} |= "database system is ready" +``` + +## Deploy in Portainer + +Use `deploy/portainer-stack.yml` with values from `deploy/portainer.env.example`. + +Keep this set while testing: + +```text +LOG_GUARDIAN_DRY_RUN_ACTIONS=true +LOG_GUARDIAN_ANALYSIS_PROVIDER=none +``` + +## Build and validate + +```sh +make ci-fast +``` + +## Safety model + +Automatic remediation is intentionally constrained: + +1. A rule must match. +2. The configured threshold must be exceeded. +3. The action must be explicitly enabled on that rule. +4. Global dry-run must be disabled. +5. Future AI recommendations are advisory unless a rule explicitly allows a matching action type. + +Do not enable production remediation until a later phase explicitly approves concrete action types and guardrails. diff --git a/cmd/server/main.go b/cmd/server/main.go new file mode 100644 index 0000000..b7831c9 --- /dev/null +++ b/cmd/server/main.go @@ -0,0 +1,49 @@ +package main + +import ( + "context" + "log" + "net/http" + "os/signal" + "syscall" + "time" + + "gitea.wayfinderak.com/wayfinderak/log-guardian/internal/actions" + "gitea.wayfinderak.com/wayfinderak/log-guardian/internal/alerts" + "gitea.wayfinderak.com/wayfinderak/log-guardian/internal/analysis" + "gitea.wayfinderak.com/wayfinderak/log-guardian/internal/config" + "gitea.wayfinderak.com/wayfinderak/log-guardian/internal/loki" + "gitea.wayfinderak.com/wayfinderak/log-guardian/internal/rules" + "gitea.wayfinderak.com/wayfinderak/log-guardian/internal/server" + "gitea.wayfinderak.com/wayfinderak/log-guardian/internal/store" +) + +func main() { + cfg := config.Load() + ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM) + defer stop() + + fileStore := store.NewFileStore(cfg.DataPath) + lokiClient := loki.New(cfg.LokiURL, cfg.LokiTenantID, cfg.LokiUsername, cfg.LokiPassword) + var analyzer analysis.Analyzer = analysis.NoopAnalyzer{} + if cfg.AnalysisProvider == "ollama" { + analyzer = analysis.NewOllamaAnalyzer(cfg.OllamaURL, cfg.OllamaModel) + } + actionRunner := actions.NewRunner(cfg.DryRunActions) + dispatcher := alerts.NewDispatcher() + engine := rules.NewEngine(fileStore, lokiClient, analyzer, actionRunner, dispatcher) + go engine.Start(ctx, cfg.PollInterval) + + httpServer := &http.Server{Addr: cfg.Addr, Handler: server.New(fileStore, engine, lokiClient, dispatcher, cfg.AuthUsername, cfg.AuthPassword).Routes()} + go func() { + <-ctx.Done() + shutdownCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + _ = httpServer.Shutdown(shutdownCtx) + }() + + log.Printf("log-guardian listening on %s", cfg.Addr) + if err := httpServer.ListenAndServe(); err != nil && err != http.ErrServerClosed { + log.Fatal(err) + } +} diff --git a/deploy/loki/docker-compose.yml b/deploy/loki/docker-compose.yml new file mode 100644 index 0000000..1532834 --- /dev/null +++ b/deploy/loki/docker-compose.yml @@ -0,0 +1,24 @@ +services: + loki: + image: grafana/loki:3.2.1 + command: -config.file=/etc/loki/local-config.yaml + ports: + - "${LOKI_PORT:-3100}:3100" + volumes: + - loki_data:/loki + restart: unless-stopped + + promtail: + image: grafana/promtail:3.2.1 + command: -config.file=/etc/promtail/config.yml + volumes: + - /var/log:/var/log:ro + - /var/lib/docker/containers:/var/lib/docker/containers:ro + - /var/run/docker.sock:/var/run/docker.sock:ro + - ./promtail-config.yml:/etc/promtail/config.yml:ro + depends_on: + - loki + restart: unless-stopped + +volumes: + loki_data: diff --git a/deploy/loki/promtail-config.yml b/deploy/loki/promtail-config.yml new file mode 100644 index 0000000..ec59cf7 --- /dev/null +++ b/deploy/loki/promtail-config.yml @@ -0,0 +1,25 @@ +server: + http_listen_port: 9080 + grpc_listen_port: 0 + +positions: + filename: /tmp/positions.yaml + +clients: + - url: http://loki:3100/loki/api/v1/push + +scrape_configs: + - job_name: docker + docker_sd_configs: + - host: unix:///var/run/docker.sock + refresh_interval: 10s + relabel_configs: + - source_labels: ['__meta_docker_container_name'] + regex: '/(.*)' + target_label: 'container' + - source_labels: ['__meta_docker_container_label_com_docker_compose_service'] + target_label: 'service' + - source_labels: ['__meta_docker_container_label_com_docker_compose_project'] + target_label: 'stack' + - source_labels: ['__meta_docker_container_log_stream'] + target_label: 'stream' diff --git a/deploy/portainer-stack.yml b/deploy/portainer-stack.yml new file mode 100644 index 0000000..1e72d5f --- /dev/null +++ b/deploy/portainer-stack.yml @@ -0,0 +1,28 @@ +services: + log-guardian: + image: gitea.wayfinderak.com/wayfinderak/log-guardian:${IMAGE_TAG:-latest} + environment: + LOG_GUARDIAN_ADDR: :8080 + LOG_GUARDIAN_DATA_PATH: /data/config.json + LOG_GUARDIAN_POLL_INTERVAL: ${LOG_GUARDIAN_POLL_INTERVAL:-1m} + LOG_GUARDIAN_DRY_RUN_ACTIONS: ${LOG_GUARDIAN_DRY_RUN_ACTIONS:-true} + LOKI_URL: ${LOKI_URL:-http://loki:3100} + LOKI_TENANT_ID: ${LOKI_TENANT_ID:-} + LOKI_USERNAME: ${LOKI_USERNAME:-} + LOKI_PASSWORD: ${LOKI_PASSWORD:-} + LOG_GUARDIAN_ANALYSIS_PROVIDER: ${LOG_GUARDIAN_ANALYSIS_PROVIDER:-none} + LOG_GUARDIAN_ANALYSIS_ENDPOINT: ${LOG_GUARDIAN_ANALYSIS_ENDPOINT:-} + LOG_GUARDIAN_ANALYSIS_TOKEN: ${LOG_GUARDIAN_ANALYSIS_TOKEN:-} + volumes: + - log_guardian_data:/data + ports: + - "${LOG_GUARDIAN_PORT:-8081}:8080" + healthcheck: + test: ["CMD", "wget", "-qO-", "http://127.0.0.1:8080/healthz"] + interval: 30s + timeout: 5s + retries: 5 + restart: unless-stopped + +volumes: + log_guardian_data: diff --git a/deploy/portainer.env.example b/deploy/portainer.env.example new file mode 100644 index 0000000..80ae93d --- /dev/null +++ b/deploy/portainer.env.example @@ -0,0 +1,11 @@ +IMAGE_TAG=latest +LOG_GUARDIAN_PORT=8081 +LOG_GUARDIAN_POLL_INTERVAL=1m +LOG_GUARDIAN_DRY_RUN_ACTIONS=true +LOKI_URL=http://loki:3100 +LOKI_TENANT_ID= +LOKI_USERNAME= +LOKI_PASSWORD= +LOG_GUARDIAN_ANALYSIS_PROVIDER=none +LOG_GUARDIAN_ANALYSIS_ENDPOINT= +LOG_GUARDIAN_ANALYSIS_TOKEN= diff --git a/docker/server.Dockerfile b/docker/server.Dockerfile new file mode 100644 index 0000000..5f250c4 --- /dev/null +++ b/docker/server.Dockerfile @@ -0,0 +1,19 @@ +FROM golang:1.23-alpine AS build +WORKDIR /src +COPY go.mod ./ +COPY cmd ./cmd +COPY internal ./internal +COPY web ./web +RUN go build -trimpath -ldflags="-s -w" -o /out/log-guardian ./cmd/server + +FROM alpine:3.20 +LABEL org.opencontainers.image.title="log-guardian" +LABEL org.opencontainers.image.description="Loki-backed log monitoring, alerting, and guarded remediation service." +WORKDIR /app +RUN adduser -D -H guardian && mkdir -p /data && chown guardian:guardian /data +COPY --from=build /out/log-guardian /usr/local/bin/log-guardian +COPY --from=build /src/web ./web +USER guardian +EXPOSE 8080 +VOLUME ["/data"] +CMD ["/usr/local/bin/log-guardian"] diff --git a/go.mod b/go.mod new file mode 100644 index 0000000..c972f0e --- /dev/null +++ b/go.mod @@ -0,0 +1,3 @@ +module gitea.wayfinderak.com/wayfinderak/log-guardian + +go 1.23 diff --git a/internal/actions/actions.go b/internal/actions/actions.go new file mode 100644 index 0000000..19c64d1 --- /dev/null +++ b/internal/actions/actions.go @@ -0,0 +1,65 @@ +package actions + +import ( + "context" + "fmt" + + "gitea.wayfinderak.com/wayfinderak/log-guardian/internal/store" +) + +type Result struct { + Action string `json:"action"` + DryRun bool `json:"dry_run"` + Detail string `json:"detail"` +} + +type Runner struct { + dryRun bool +} + +func NewRunner(dryRun bool) *Runner { return &Runner{dryRun: dryRun} } + +func (r *Runner) Run(ctx context.Context, rule store.Rule, action store.Action) (Result, error) { + _ = ctx + if !action.Enabled { + return Result{Action: action.Type, DryRun: r.effectiveDryRun(action), Detail: "disabled"}, nil + } + switch action.Type { + case "generic_webhook", "webhook": + url := action.Params["url"] + if url == "" { + return Result{}, fmt.Errorf("webhook action requires url") + } + return Result{Action: action.Type, DryRun: true, Detail: "would POST webhook for rule " + rule.Name}, nil + case "record_recommendation": + return Result{Action: action.Type, DryRun: false, Detail: "recommendation recorded for operator review"}, nil + case "allowlisted_command", "command": + id := action.Params["command_id"] + if id == "" { + id = action.Params["id"] + } + if id == "" { + return Result{}, fmt.Errorf("allowlisted command requires command_id") + } + return Result{Action: action.Type, DryRun: true, Detail: "would run allowlisted command " + id}, nil + case "portainer_restart": + service := action.Params["service"] + if service == "" { + return Result{}, fmt.Errorf("portainer_restart requires service") + } + return Result{Action: action.Type, DryRun: true, Detail: "would restart Portainer service " + service}, nil + default: + return Result{}, fmt.Errorf("unsupported action type %q", action.Type) + } + return Result{}, fmt.Errorf("unsupported action type %q", action.Type) +} + +func (r *Runner) effectiveDryRun(action store.Action) bool { + if r.dryRun { + return true + } + if action.DryRun == nil { + return true + } + return *action.DryRun +} diff --git a/internal/alerts/alerts.go b/internal/alerts/alerts.go new file mode 100644 index 0000000..1ef0917 --- /dev/null +++ b/internal/alerts/alerts.go @@ -0,0 +1,108 @@ +package alerts + +import ( + "bytes" + "context" + "fmt" + "log" + "net/http" + "net/url" + "strings" + "time" + + "gitea.wayfinderak.com/wayfinderak/log-guardian/internal/analysis" + "gitea.wayfinderak.com/wayfinderak/log-guardian/internal/store" +) + +type Result struct { + ChannelID string + Detail string +} + +type Dispatcher struct { + http *http.Client +} + +func NewDispatcher() *Dispatcher { return &Dispatcher{http: &http.Client{Timeout: 15 * time.Second}} } + +func (d *Dispatcher) Send(ctx context.Context, rule store.Rule, finding analysis.Finding, channels []store.AlertChannel) ([]Result, error) { + if len(channels) == 0 { + log.Printf("alert for rule %q: %s", rule.Name, finding.Summary) + return []Result{{Detail: "no channels configured; logged alert intent"}}, nil + } + var results []Result + var firstErr error + for _, channel := range channels { + if !channel.Enabled { + results = append(results, Result{ChannelID: channel.ID, Detail: "disabled"}) + continue + } + var err error + switch channel.Type { + case "ntfy": + err = d.sendNtfy(ctx, rule, finding, channel) + case "gotify", "generic_webhook", "smtp": + err = fmt.Errorf("%s delivery is scaffolded but not implemented in phase 1", channel.Type) + default: + err = fmt.Errorf("unsupported channel type %q", channel.Type) + } + if err != nil { + if firstErr == nil { + firstErr = err + } + results = append(results, Result{ChannelID: channel.ID, Detail: err.Error()}) + continue + } + results = append(results, Result{ChannelID: channel.ID, Detail: "sent"}) + } + return results, firstErr +} + +func (d *Dispatcher) Test(ctx context.Context, channel store.AlertChannel) error { + finding := analysis.Finding{Summary: "Log Guardian test notification", Confidence: "test"} + rule := store.Rule{Name: "Test alert", Severity: "info"} + switch channel.Type { + case "ntfy": + return d.sendNtfy(ctx, rule, finding, channel) + case "gotify", "generic_webhook", "smtp": + return fmt.Errorf("%s delivery is scaffolded but not implemented in phase 1", channel.Type) + default: + return fmt.Errorf("unsupported channel type %q", channel.Type) + } +} + +func (d *Dispatcher) sendNtfy(ctx context.Context, rule store.Rule, finding analysis.Finding, channel store.AlertChannel) error { + server := strings.TrimRight(channel.Params["server_url"], "/") + topic := strings.Trim(channel.Params["topic"], "/") + if server == "" || topic == "" { + return fmt.Errorf("ntfy requires server_url and topic") + } + target, err := url.JoinPath(server, topic) + if err != nil { + return err + } + body := rule.Name + ": " + finding.Summary + req, err := http.NewRequestWithContext(ctx, http.MethodPost, target, bytes.NewBufferString(body)) + if err != nil { + return err + } + req.Header.Set("Title", "Log Guardian: "+rule.Severity) + if priority := channel.Params["priority"]; priority != "" { + req.Header.Set("Priority", priority) + } + if tags := channel.Params["tags"]; tags != "" { + req.Header.Set("Tags", tags) + } + if token := channel.Params["token"]; token != "" { + req.Header.Set("Authorization", "Bearer "+token) + } + resp, err := d.http.Do(req) + if err != nil { + return err + } + defer resp.Body.Close() + if resp.StatusCode < 200 || resp.StatusCode > 299 { + return fmt.Errorf("ntfy delivery failed: status %d", resp.StatusCode) + } + return nil +} diff --git a/internal/alerts/alerts_test.go b/internal/alerts/alerts_test.go new file mode 100644 index 0000000..fbf144d --- /dev/null +++ b/internal/alerts/alerts_test.go @@ -0,0 +1,31 @@ +package alerts + +import ( + "net/http" + "net/http/httptest" + "testing" + + "gitea.wayfinderak.com/wayfinderak/log-guardian/internal/analysis" + "gitea.wayfinderak.com/wayfinderak/log-guardian/internal/store" +) + +func TestNtfyDelivery(t *testing.T) { + var gotAuth string + ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/alerts" { + t.Fatalf("unexpected path %s", r.URL.Path) + } + gotAuth = r.Header.Get("Authorization") + w.WriteHeader(http.StatusNoContent) + })) + defer ts.Close() + + d := NewDispatcher() + _, err := d.Send(t.Context(), store.Rule{Name: "API", Severity: "warning"}, analysis.Finding{Summary: "failed"}, []store.AlertChannel{{ID: "c1", Name: "ntfy", Type: "ntfy", Enabled: true, Params: map[string]string{"server_url": ts.URL, "topic": "alerts", "token": "secret"}}}) + if err != nil { + t.Fatal(err) + } + if gotAuth != "Bearer secret" { + t.Fatalf("missing auth header: %q", gotAuth) + } +} diff --git a/internal/analysis/analysis.go b/internal/analysis/analysis.go new file mode 100644 index 0000000..470b663 --- /dev/null +++ b/internal/analysis/analysis.go @@ -0,0 +1,160 @@ +package analysis + +import ( + "bytes" + "context" + "encoding/json" + "fmt" + "net/http" + "strings" + "time" + "unicode" + + "gitea.wayfinderak.com/wayfinderak/log-guardian/internal/loki" + "gitea.wayfinderak.com/wayfinderak/log-guardian/internal/store" +) + +type Finding struct { + Summary string `json:"summary"` + RecommendedActions []string `json:"recommended_actions,omitempty"` + Confidence string `json:"confidence"` +} + +type Analyzer interface { + Analyze(ctx context.Context, rule store.Rule, matches []loki.Match) (Finding, error) +} + +type NoopAnalyzer struct{} + +func (NoopAnalyzer) Analyze(ctx context.Context, rule store.Rule, matches []loki.Match) (Finding, error) { + _ = ctx + return Finding{Summary: summarize(rule, matches), Confidence: "rule-only"}, nil +} + +type OllamaAnalyzer struct { + URL string + Model string + http *http.Client +} + +func NewOllamaAnalyzer(url, model string) *OllamaAnalyzer { + return &OllamaAnalyzer{URL: strings.TrimRight(url, "/"), Model: model, http: &http.Client{Timeout: 60 * time.Second}} +} + +func (a *OllamaAnalyzer) Analyze(ctx context.Context, rule store.Rule, matches []loki.Match) (Finding, error) { + if a.URL == "" || a.Model == "" { + return NoopAnalyzer{}.Analyze(ctx, rule, matches) + } + prompt := buildPrompt(rule, matches) + body, _ := json.Marshal(map[string]any{"model": a.Model, "prompt": prompt, "stream": false}) + req, err := http.NewRequestWithContext(ctx, http.MethodPost, a.URL+"/api/generate", bytes.NewReader(body)) + if err != nil { + return Finding{}, err + } + req.Header.Set("Content-Type", "application/json") + resp, err := a.http.Do(req) + if err != nil { + return Finding{}, err + } + defer resp.Body.Close() + if resp.StatusCode < 200 || resp.StatusCode > 299 { + return Finding{}, fmt.Errorf("ollama analysis failed: status %d", resp.StatusCode) + } + var raw struct { + Response string `json:"response"` + } + if err := json.NewDecoder(resp.Body).Decode(&raw); err != nil { + return Finding{}, err + } + summary := strings.TrimSpace(raw.Response) + if summary == "" { + return NoopAnalyzer{}.Analyze(ctx, rule, matches) + } + if len(summary) > 500 { + summary = summary[:500] + "..." + } + return Finding{Summary: summary, Confidence: "ollama-local"}, nil +} + +func buildPrompt(rule store.Rule, matches []loki.Match) string { + var b strings.Builder + b.WriteString("You are analyzing private application logs locally. Summarize likely cause and operator next steps in 4 concise sentences or fewer. Do not suggest destructive actions.\n") + b.WriteString("Rule: ") + b.WriteString(rule.Name) + b.WriteString("\nSeverity: ") + b.WriteString(rule.Severity) + b.WriteString("\nSanitized samples:\n") + limit := len(matches) + if limit > 5 { + limit = 5 + } + for i := 0; i < limit; i++ { + line := sanitizeLine(matches[i].Line) + line = redactSecrets(line) + if len(line) > 300 { + line = line[:300] + "..." + } + b.WriteString("- ") + b.WriteString(line) + b.WriteByte('\n') + } + return b.String() +} + +func summarize(rule store.Rule, matches []loki.Match) string { + if len(matches) == 0 { + return "No matching log entries." + } + line := redactSecrets(sanitizeLine(matches[0].Line)) + if len(line) > 180 { + line = line[:180] + "..." + } + return rule.Name + " matched " + plural(len(matches), "log entry", "log entries") + "; first match: " + line +} + +func sanitizeLine(line string) string { + line = strings.ReplaceAll(line, "\n", " ") + line = strings.ReplaceAll(line, "\r", " ") + return strings.TrimSpace(line) +} + +func redactSecrets(line string) string { + fields := strings.Fields(line) + for i, field := range fields { + lower := strings.ToLower(field) + if strings.Contains(lower, "token=") || strings.Contains(lower, "password=") || strings.Contains(lower, "secret=") || strings.Contains(lower, "apikey=") || strings.Contains(lower, "api_key=") || strings.HasPrefix(lower, "bearer") { + fields[i] = redactField(field) + } + } + return strings.Join(fields, " ") +} + +func redactField(field string) string { + idx := strings.IndexAny(field, "=:") + if idx < 0 { + return "[REDACTED]" + } + prefix := strings.TrimRightFunc(field[:idx], func(r rune) bool { return unicode.IsSpace(r) }) + return prefix + "=[REDACTED]" +} + +func plural(count int, singular, plural string) string { + if count == 1 { + return "1 " + singular + } + return strconvItoa(count) + " " + plural +} + +func strconvItoa(i int) string { + if i == 0 { + return "0" + } + var b [20]byte + bp := len(b) + for i > 0 { + bp-- + b[bp] = byte('0' + i%10) + i /= 10 + } + return string(b[bp:]) +} diff --git a/internal/analysis/analysis_test.go b/internal/analysis/analysis_test.go new file mode 100644 index 0000000..03ea04d --- /dev/null +++ b/internal/analysis/analysis_test.go @@ -0,0 +1,10 @@ +package analysis + +import "testing" + +func TestSanitizeLine(t *testing.T) { + got := sanitizeLine(" error\nwith\rspaces ") + if got != "error with spaces" { + t.Fatalf("got %q", got) + } +} diff --git a/internal/config/config.go b/internal/config/config.go new file mode 100644 index 0000000..4e8b59a --- /dev/null +++ b/internal/config/config.go @@ -0,0 +1,83 @@ +package config + +import ( + "os" + "strconv" + "time" +) + +type Config struct { + Addr string + DataPath string + PollInterval time.Duration + LokiURL string + LokiTenantID string + LokiUsername string + LokiPassword string + DryRunActions bool + AnalysisProvider string + AnalysisEndpoint string + AnalysisToken string + OllamaURL string + OllamaModel string + AuthUsername string + AuthPassword string +} + +func Load() Config { + return Config{ + Addr: env("LOG_GUARDIAN_ADDR", ":8080"), + DataPath: env("LOG_GUARDIAN_DATA_PATH", "/data/config.json"), + PollInterval: durationEnv("LOG_GUARDIAN_POLL_INTERVAL", 1*time.Minute), + LokiURL: trimRightSlash(env("LOKI_URL", "http://loki:3100")), + LokiTenantID: env("LOKI_TENANT_ID", ""), + LokiUsername: env("LOKI_USERNAME", ""), + LokiPassword: env("LOKI_PASSWORD", ""), + DryRunActions: boolEnv("LOG_GUARDIAN_DRY_RUN_ACTIONS", true), + AnalysisProvider: env("LOG_GUARDIAN_ANALYSIS_PROVIDER", "none"), + AnalysisEndpoint: env("LOG_GUARDIAN_ANALYSIS_ENDPOINT", ""), + AnalysisToken: env("LOG_GUARDIAN_ANALYSIS_TOKEN", ""), + OllamaURL: trimRightSlash(env("LOG_GUARDIAN_OLLAMA_URL", "http://ollama:11434")), + OllamaModel: env("LOG_GUARDIAN_OLLAMA_MODEL", "llama3.1"), + AuthUsername: env("LOG_GUARDIAN_AUTH_USERNAME", ""), + AuthPassword: env("LOG_GUARDIAN_AUTH_PASSWORD", ""), + } +} + +func env(key, fallback string) string { + if value := os.Getenv(key); value != "" { + return value + } + return fallback +} + +func boolEnv(key string, fallback bool) bool { + value := os.Getenv(key) + if value == "" { + return fallback + } + parsed, err := strconv.ParseBool(value) + if err != nil { + return fallback + } + return parsed +} + +func durationEnv(key string, fallback time.Duration) time.Duration { + value := os.Getenv(key) + if value == "" { + return fallback + } + parsed, err := time.ParseDuration(value) + if err != nil { + return fallback + } + return parsed +} + +func trimRightSlash(value string) string { + for len(value) > 1 && value[len(value)-1] == '/' { + value = value[:len(value)-1] + } + return value +} diff --git a/internal/loki/client.go b/internal/loki/client.go new file mode 100644 index 0000000..c5b2927 --- /dev/null +++ b/internal/loki/client.go @@ -0,0 +1,148 @@ +package loki + +import ( + "context" + "encoding/json" + "fmt" + "net/http" + "net/url" + "strconv" + "strings" + "time" +) + +type Client struct { + baseURL string + tenantID string + username string + password string + http *http.Client +} + +func New(baseURL, tenantID, username, password string) *Client { + return &Client{baseURL: strings.TrimRight(baseURL, "/"), tenantID: tenantID, username: username, password: password, http: &http.Client{Timeout: 20 * time.Second}} +} + +type Match struct { + Timestamp time.Time `json:"timestamp"` + Line string `json:"line"` +} + +type QueryResult struct { + Count int `json:"count"` + Matches []Match `json:"matches"` +} + +func (c *Client) Ready(ctx context.Context) error { + req, err := c.request(ctx, http.MethodGet, "/ready", nil) + if err != nil { + return err + } + resp, err := c.http.Do(req) + if err != nil { + return err + } + defer resp.Body.Close() + if resp.StatusCode < 200 || resp.StatusCode > 299 { + return fmt.Errorf("loki ready failed: status %d", resp.StatusCode) + } + return nil +} + +func (c *Client) Labels(ctx context.Context) ([]string, error) { + var raw labelsResponse + if err := c.getJSON(ctx, "/loki/api/v1/labels", nil, &raw); err != nil { + return nil, err + } + return raw.Data, nil +} + +func (c *Client) LabelValues(ctx context.Context, name string) ([]string, error) { + if name == "" || strings.Contains(name, "/") { + return nil, fmt.Errorf("invalid label name") + } + var raw labelsResponse + if err := c.getJSON(ctx, "/loki/api/v1/label/"+url.PathEscape(name)+"/values", nil, &raw); err != nil { + return nil, err + } + return raw.Data, nil +} + +func (c *Client) QueryRange(ctx context.Context, logql string, since time.Duration, limit int) (QueryResult, error) { + if limit <= 0 { + limit = 100 + } + values := url.Values{} + values.Set("query", logql) + values.Set("start", strconv.FormatInt(time.Now().Add(-since).UnixNano(), 10)) + values.Set("end", strconv.FormatInt(time.Now().UnixNano(), 10)) + values.Set("limit", strconv.Itoa(limit)) + var raw response + if err := c.getJSON(ctx, "/loki/api/v1/query_range", values, &raw); err != nil { + return QueryResult{}, err + } + var result QueryResult + for _, stream := range raw.Data.Result { + for _, pair := range stream.Values { + if len(pair) != 2 { + continue + } + ns, _ := strconv.ParseInt(pair[0], 10, 64) + result.Matches = append(result.Matches, Match{Timestamp: time.Unix(0, ns), Line: pair[1]}) + } + } + result.Count = len(result.Matches) + return result, nil +} + +func (c *Client) getJSON(ctx context.Context, path string, values url.Values, out any) error { + req, err := c.request(ctx, http.MethodGet, path, values) + if err != nil { + return err + } + resp, err := c.http.Do(req) + if err != nil { + return err + } + defer resp.Body.Close() + if resp.StatusCode < 200 || resp.StatusCode > 299 { + return fmt.Errorf("loki request failed: status %d", resp.StatusCode) + } + return json.NewDecoder(resp.Body).Decode(out) +} + +func (c *Client) request(ctx context.Context, method, path string, values url.Values) (*http.Request, error) { + if c.baseURL == "" { + return nil, fmt.Errorf("LOKI_URL is not configured") + } + target := c.baseURL + path + if len(values) > 0 { + target += "?" + values.Encode() + } + req, err := http.NewRequestWithContext(ctx, method, target, nil) + if err != nil { + return nil, err + } + if c.tenantID != "" { + req.Header.Set("X-Scope-OrgID", c.tenantID) + } + if c.username != "" || c.password != "" { + req.SetBasicAuth(c.username, c.password) + } + return req, nil +} + +type labelsResponse struct { + Status string `json:"status"` + Data []string `json:"data"` +} + +type response struct { + Status string `json:"status"` + Data struct { + Result []struct { + Stream map[string]string `json:"stream"` + Values [][]string `json:"values"` + } `json:"result"` + } `json:"data"` +} diff --git a/internal/loki/client_test.go b/internal/loki/client_test.go new file mode 100644 index 0000000..9a90e2c --- /dev/null +++ b/internal/loki/client_test.go @@ -0,0 +1,42 @@ +package loki + +import ( + "net/http" + "net/http/httptest" + "strings" + "testing" + "time" +) + +func TestLabelsValuesAndQueryRange(t *testing.T) { + ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch r.URL.Path { + case "/loki/api/v1/labels": + _, _ = w.Write([]byte(`{"status":"success","data":["service","container"]}`)) + case "/loki/api/v1/label/service/values": + _, _ = w.Write([]byte(`{"status":"success","data":["api"]}`)) + case "/loki/api/v1/query_range": + if !strings.Contains(r.URL.Query().Get("query"), "api") { + t.Fatalf("query not passed through: %s", r.URL.RawQuery) + } + _, _ = w.Write([]byte(`{"status":"success","data":{"result":[{"stream":{"service":"api"},"values":[["1700000000000000000","error token=secret"]]}]}}`)) + default: + http.NotFound(w, r) + } + })) + defer ts.Close() + + client := New(ts.URL, "", "", "") + labels, err := client.Labels(t.Context()) + if err != nil || len(labels) != 2 { + t.Fatalf("labels=%v err=%v", labels, err) + } + values, err := client.LabelValues(t.Context(), "service") + if err != nil || values[0] != "api" { + t.Fatalf("values=%v err=%v", values, err) + } + result, err := client.QueryRange(t.Context(), `{service="api"}`, time.Minute, 10) + if err != nil || result.Count != 1 { + t.Fatalf("result=%#v err=%v", result, err) + } +} diff --git a/internal/rules/engine.go b/internal/rules/engine.go new file mode 100644 index 0000000..3f2f0eb --- /dev/null +++ b/internal/rules/engine.go @@ -0,0 +1,152 @@ +package rules + +import ( + "context" + "log" + "time" + + "gitea.wayfinderak.com/wayfinderak/log-guardian/internal/actions" + "gitea.wayfinderak.com/wayfinderak/log-guardian/internal/alerts" + "gitea.wayfinderak.com/wayfinderak/log-guardian/internal/analysis" + "gitea.wayfinderak.com/wayfinderak/log-guardian/internal/loki" + "gitea.wayfinderak.com/wayfinderak/log-guardian/internal/store" +) + +type Store interface { + Load() (store.Data, error) + UpdateRule(id string, fn func(*store.Rule)) error + AddIncident(incident store.Incident) error +} + +type Engine struct { + store Store + loki *loki.Client + analyzer analysis.Analyzer + actions *actions.Runner + dispatcher *alerts.Dispatcher +} + +func NewEngine(s Store, lokiClient *loki.Client, analyzer analysis.Analyzer, actionRunner *actions.Runner, dispatcher *alerts.Dispatcher) *Engine { + return &Engine{store: s, loki: lokiClient, analyzer: analyzer, actions: actionRunner, dispatcher: dispatcher} +} + +func (e *Engine) Start(ctx context.Context, interval time.Duration) { + if interval <= 0 { + interval = time.Minute + } + ticker := time.NewTicker(interval) + defer ticker.Stop() + for { + e.CheckAll(ctx) + select { + case <-ctx.Done(): + return + case <-ticker.C: + } + } +} + +func (e *Engine) CheckAll(ctx context.Context) { + data, err := e.store.Load() + if err != nil { + log.Printf("load rules failed: %v", err) + return + } + channels := map[string]store.AlertChannel{} + for _, channel := range data.AlertChannels { + channels[channel.ID] = channel + } + for _, rule := range data.Rules { + if !rule.Enabled { + continue + } + if err := e.checkRule(ctx, rule, channels); err != nil { + log.Printf("rule %q check failed: %v", rule.Name, err) + } + } +} + +func (e *Engine) checkRule(ctx context.Context, rule store.Rule, channels map[string]store.AlertChannel) error { + window, err := time.ParseDuration(rule.Window) + if err != nil { + window = 5 * time.Minute + } + result, err := e.loki.QueryRange(ctx, rule.LogQL, window, 200) + if err != nil { + e.record(rule.ID, 0, err) + return err + } + if result.Count < rule.Threshold { + return e.record(rule.ID, result.Count, nil) + } + now := time.Now().UTC() + cooldown, err := time.ParseDuration(rule.Cooldown) + if err != nil || cooldown <= 0 { + cooldown = time.Hour + } + if !rule.LastAlertedAt.IsZero() && now.Sub(rule.LastAlertedAt) < cooldown { + return e.store.UpdateRule(rule.ID, func(r *store.Rule) { + r.LastCheckedAt = now + r.LastMatchedAt = now + r.LastMatchCount = result.Count + r.SuppressedCount++ + r.LastError = "" + }) + } + finding, err := e.analyzer.Analyze(ctx, rule, result.Matches) + if err != nil { + e.record(rule.ID, result.Count, err) + return err + } + selected := make([]store.AlertChannel, 0, len(rule.AlertChannels)) + for _, id := range rule.AlertChannels { + if channel, ok := channels[id]; ok { + selected = append(selected, channel) + } + } + alertResults, alertErr := e.dispatcher.Send(ctx, rule, finding, selected) + var alertEvidence []string + for _, result := range alertResults { + if result.ChannelID == "" { + alertEvidence = append(alertEvidence, result.Detail) + } else { + alertEvidence = append(alertEvidence, result.ChannelID+": "+result.Detail) + } + } + var actionEvidence []string + for _, action := range rule.Actions { + result, err := e.actions.Run(ctx, rule, action) + if err != nil { + log.Printf("action %q for rule %q failed: %v", action.Type, rule.Name, err) + actionEvidence = append(actionEvidence, action.Type+": "+err.Error()) + continue + } + actionEvidence = append(actionEvidence, result.Action+": "+result.Detail) + } + incident := store.Incident{RuleID: rule.ID, RuleName: rule.Name, Severity: rule.Severity, Count: result.Count, Summary: finding.Summary, AlertResults: alertEvidence, RemediationEvidence: actionEvidence, CreatedAt: now} + if err := e.store.AddIncident(incident); err != nil { + return err + } + if err := e.store.UpdateRule(rule.ID, func(r *store.Rule) { + r.LastCheckedAt = now + r.LastMatchedAt = now + r.LastAlertedAt = now + r.LastMatchCount = result.Count + r.LastError = "" + }); err != nil { + return err + } + return alertErr +} + +func (e *Engine) record(id string, count int, err error) error { + return e.store.UpdateRule(id, func(r *store.Rule) { + r.LastCheckedAt = time.Now().UTC() + r.LastMatchCount = count + if err != nil { + r.LastError = err.Error() + } else { + r.LastError = "" + } + }) +} diff --git a/internal/rules/engine_test.go b/internal/rules/engine_test.go new file mode 100644 index 0000000..9889d9e --- /dev/null +++ b/internal/rules/engine_test.go @@ -0,0 +1,42 @@ +package rules + +import ( + "net/http" + "net/http/httptest" + "testing" + "time" + + "gitea.wayfinderak.com/wayfinderak/log-guardian/internal/actions" + "gitea.wayfinderak.com/wayfinderak/log-guardian/internal/alerts" + "gitea.wayfinderak.com/wayfinderak/log-guardian/internal/analysis" + "gitea.wayfinderak.com/wayfinderak/log-guardian/internal/loki" + "gitea.wayfinderak.com/wayfinderak/log-guardian/internal/store" +) + +func TestCooldownSuppressesDuplicateIncident(t *testing.T) { + ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + _, _ = w.Write([]byte(`{"status":"success","data":{"result":[{"stream":{},"values":[["1700000000000000000","error"]]}]}}`)) + })) + defer ts.Close() + + s := store.NewFileStore(t.TempDir() + "/config.json") + if err := s.UpsertRule(store.Rule{Name: "Errors", Enabled: true, LogQL: `{service="api"}`, Threshold: 1, Window: "5m", Cooldown: "1h", Actions: []store.Action{{Type: "record_recommendation", Enabled: true}}}); err != nil { + t.Fatal(err) + } + engine := NewEngine(s, loki.New(ts.URL, "", "", ""), analysis.NoopAnalyzer{}, actions.NewRunner(true), alerts.NewDispatcher()) + engine.CheckAll(t.Context()) + engine.CheckAll(t.Context()) + data, err := s.Load() + if err != nil { + t.Fatal(err) + } + if len(data.Incidents) != 1 { + t.Fatalf("expected one incident, got %d", len(data.Incidents)) + } + if data.Rules[0].SuppressedCount != 1 { + t.Fatalf("expected suppressed count 1, got %d", data.Rules[0].SuppressedCount) + } + if time.Since(data.Rules[0].LastAlertedAt) > time.Minute { + t.Fatalf("last alerted not set: %s", data.Rules[0].LastAlertedAt) + } +} diff --git a/internal/server/middleware.go b/internal/server/middleware.go new file mode 100644 index 0000000..f9a8219 --- /dev/null +++ b/internal/server/middleware.go @@ -0,0 +1,51 @@ +package server + +import ( + "crypto/subtle" + "encoding/json" + "log" + "net/http" + "time" +) + +func writeJSON(w http.ResponseWriter, v any) { + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(v) +} + +func requestLog(next http.Handler) http.Handler { + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + start := time.Now() + next.ServeHTTP(w, r) + log.Printf("%s %s %s", r.Method, r.URL.Path, time.Since(start).Round(time.Millisecond)) + }) +} + +func securityHeaders(next http.Handler) http.Handler { + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("X-Content-Type-Options", "nosniff") + w.Header().Set("Referrer-Policy", "no-referrer") + w.Header().Set("Content-Security-Policy", "default-src 'self'; style-src 'self' 'unsafe-inline'; form-action 'self'") + next.ServeHTTP(w, r) + }) +} + +func basicAuth(next http.Handler, username, password string) http.Handler { + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/healthz" { + next.ServeHTTP(w, r) + return + } + if username == "" || password == "" { + http.Error(w, "basic auth is not configured", http.StatusServiceUnavailable) + return + } + gotUser, gotPass, ok := r.BasicAuth() + if !ok || subtle.ConstantTimeCompare([]byte(gotUser), []byte(username)) != 1 || subtle.ConstantTimeCompare([]byte(gotPass), []byte(password)) != 1 { + w.Header().Set("WWW-Authenticate", `Basic realm="log-guardian"`) + http.Error(w, "unauthorized", http.StatusUnauthorized) + return + } + next.ServeHTTP(w, r) + }) +} diff --git a/internal/server/middleware_test.go b/internal/server/middleware_test.go new file mode 100644 index 0000000..b711cec --- /dev/null +++ b/internal/server/middleware_test.go @@ -0,0 +1,34 @@ +package server + +import ( + "net/http" + "net/http/httptest" + "testing" +) + +func TestBasicAuthAllowsHealthzWithoutCredentials(t *testing.T) { + h := basicAuth(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(http.StatusNoContent) }), "admin", "secret") + req := httptest.NewRequest(http.MethodGet, "/healthz", nil) + rr := httptest.NewRecorder() + h.ServeHTTP(rr, req) + if rr.Code != http.StatusNoContent { + t.Fatalf("got %d", rr.Code) + } +} + +func TestBasicAuthProtectsUI(t *testing.T) { + h := basicAuth(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(http.StatusNoContent) }), "admin", "secret") + req := httptest.NewRequest(http.MethodGet, "/", nil) + rr := httptest.NewRecorder() + h.ServeHTTP(rr, req) + if rr.Code != http.StatusUnauthorized { + t.Fatalf("got %d", rr.Code) + } + req = httptest.NewRequest(http.MethodGet, "/", nil) + req.SetBasicAuth("admin", "secret") + rr = httptest.NewRecorder() + h.ServeHTTP(rr, req) + if rr.Code != http.StatusNoContent { + t.Fatalf("got %d", rr.Code) + } +} diff --git a/internal/server/server.go b/internal/server/server.go new file mode 100644 index 0000000..94bd8fd --- /dev/null +++ b/internal/server/server.go @@ -0,0 +1,202 @@ +package server + +import ( + "context" + "html/template" + "net/http" + "strconv" + "strings" + "time" + + "gitea.wayfinderak.com/wayfinderak/log-guardian/internal/alerts" + "gitea.wayfinderak.com/wayfinderak/log-guardian/internal/loki" + "gitea.wayfinderak.com/wayfinderak/log-guardian/internal/rules" + "gitea.wayfinderak.com/wayfinderak/log-guardian/internal/store" +) + +type Server struct { + store *store.FileStore + engine *rules.Engine + loki *loki.Client + dispatcher *alerts.Dispatcher + authUsername string + authPassword string +} + +func New(s *store.FileStore, engine *rules.Engine, lokiClient *loki.Client, dispatcher *alerts.Dispatcher, authUsername, authPassword string) *Server { + return &Server{store: s, engine: engine, loki: lokiClient, dispatcher: dispatcher, authUsername: authUsername, authPassword: authPassword} +} + +func (s *Server) Routes() http.Handler { + mux := http.NewServeMux() + mux.HandleFunc("GET /healthz", s.health) + mux.HandleFunc("GET /", s.index) + mux.HandleFunc("POST /rules", s.saveRule) + mux.HandleFunc("POST /check", s.checkNow) + mux.HandleFunc("POST /channels", s.saveChannel) + mux.HandleFunc("POST /channels/delete", s.deleteChannel) + mux.HandleFunc("POST /channels/test", s.testChannel) + mux.HandleFunc("GET /loki/labels", s.lokiLabels) + mux.HandleFunc("GET /loki/label-values", s.lokiLabelValues) + mux.HandleFunc("POST /loki/query-test", s.lokiQueryTest) + mux.Handle("GET /static/", http.StripPrefix("/static/", http.FileServer(http.Dir("web/static")))) + return securityHeaders(requestLog(basicAuth(mux, s.authUsername, s.authPassword))) +} + +func (s *Server) health(w http.ResponseWriter, r *http.Request) { + writeJSON(w, map[string]string{"status": "ok", "service": "log-guardian"}) +} + +func (s *Server) index(w http.ResponseWriter, r *http.Request) { + data, err := s.store.Load() + if err != nil { + http.Error(w, err.Error(), http.StatusInternalServerError) + return + } + render(w, data) +} + +func (s *Server) saveRule(w http.ResponseWriter, r *http.Request) { + if err := r.ParseForm(); err != nil { + http.Error(w, err.Error(), http.StatusBadRequest) + return + } + threshold, _ := strconv.Atoi(r.FormValue("threshold")) + rule := store.Rule{ID: strings.TrimSpace(r.FormValue("id")), Name: strings.TrimSpace(r.FormValue("name")), Enabled: r.FormValue("enabled") == "on", LogQL: strings.TrimSpace(r.FormValue("logql")), Severity: strings.TrimSpace(r.FormValue("severity")), Threshold: threshold, Window: strings.TrimSpace(r.FormValue("window")), Cooldown: strings.TrimSpace(r.FormValue("cooldown")), AnalysisEnabled: r.FormValue("analysis_enabled") == "on", AlertChannels: r.Form["alert_channels"]} + if rule.Name == "" || rule.LogQL == "" { + http.Error(w, "name and LogQL are required", http.StatusBadRequest) + return + } + if rule.Severity == "" { + rule.Severity = "warning" + } + if err := s.store.UpsertRule(rule); err != nil { + http.Error(w, err.Error(), http.StatusInternalServerError) + return + } + http.Redirect(w, r, "/", http.StatusSeeOther) +} + +func (s *Server) saveChannel(w http.ResponseWriter, r *http.Request) { + if err := r.ParseForm(); err != nil { + http.Error(w, err.Error(), http.StatusBadRequest) + return + } + channel := store.AlertChannel{ID: strings.TrimSpace(r.FormValue("id")), Name: strings.TrimSpace(r.FormValue("name")), Type: strings.TrimSpace(r.FormValue("type")), Enabled: r.FormValue("enabled") == "on", Params: map[string]string{}} + for _, key := range []string{"server_url", "topic", "token", "priority", "tags", "url", "api_url", "email"} { + if value := strings.TrimSpace(r.FormValue(key)); value != "" { + channel.Params[key] = value + } + } + if channel.Name == "" || channel.Type == "" { + http.Error(w, "name and type are required", http.StatusBadRequest) + return + } + if err := s.store.UpsertAlertChannel(channel); err != nil { + http.Error(w, err.Error(), http.StatusInternalServerError) + return + } + http.Redirect(w, r, "/", http.StatusSeeOther) +} + +func (s *Server) deleteChannel(w http.ResponseWriter, r *http.Request) { + if err := r.ParseForm(); err != nil { + http.Error(w, err.Error(), http.StatusBadRequest) + return + } + if err := s.store.DeleteAlertChannel(r.FormValue("id")); err != nil { + http.Error(w, err.Error(), http.StatusInternalServerError) + return + } + http.Redirect(w, r, "/", http.StatusSeeOther) +} + +func (s *Server) testChannel(w http.ResponseWriter, r *http.Request) { + if err := r.ParseForm(); err != nil { + http.Error(w, err.Error(), http.StatusBadRequest) + return + } + data, err := s.store.Load() + if err != nil { + http.Error(w, err.Error(), http.StatusInternalServerError) + return + } + for _, channel := range data.AlertChannels { + if channel.ID == r.FormValue("id") { + ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second) + defer cancel() + if err := s.dispatcher.Test(ctx, channel); err != nil { + http.Error(w, err.Error(), http.StatusBadGateway) + return + } + writeJSON(w, map[string]string{"status": "sent"}) + return + } + } + http.Error(w, "channel not found", http.StatusNotFound) +} + +func (s *Server) checkNow(w http.ResponseWriter, r *http.Request) { + s.engine.CheckAll(context.Background()) + http.Redirect(w, r, "/", http.StatusSeeOther) +} + +func (s *Server) lokiLabels(w http.ResponseWriter, r *http.Request) { + labels, err := s.loki.Labels(r.Context()) + if err != nil { + http.Error(w, err.Error(), http.StatusBadGateway) + return + } + writeJSON(w, map[string]any{"labels": labels}) +} + +func (s *Server) lokiLabelValues(w http.ResponseWriter, r *http.Request) { + values, err := s.loki.LabelValues(r.Context(), r.URL.Query().Get("name")) + if err != nil { + http.Error(w, err.Error(), http.StatusBadGateway) + return + } + writeJSON(w, map[string]any{"values": values}) +} + +func (s *Server) lokiQueryTest(w http.ResponseWriter, r *http.Request) { + if err := r.ParseForm(); err != nil { + http.Error(w, err.Error(), http.StatusBadRequest) + return + } + window, err := time.ParseDuration(r.FormValue("window")) + if err != nil { + window = 5 * time.Minute + } + result, err := s.loki.QueryRange(r.Context(), strings.TrimSpace(r.FormValue("logql")), window, 20) + if err != nil { + http.Error(w, err.Error(), http.StatusBadGateway) + return + } + if len(result.Matches) > 5 { + result.Matches = result.Matches[:5] + } + writeJSON(w, result) +} + +var tmpl = template.Must(template.New("index").Funcs(template.FuncMap{"secretState": secretState}).Parse(` +Log Guardian +

Log Guardian

Loki-backed log monitoring, alerting, and guarded remediation.

+

Loki diagnostics

Use authenticated endpoints: /loki/labels, /loki/label-values?name=service.

+

Add or replace rule

Alert channels{{range .AlertChannels}}{{else}}No channels configured.{{end}}
+

Alert channels

{{range .AlertChannels}}{{else}}{{end}}
NameTypeStatusSecretActions
{{.Name}}{{.Type}}{{if .Enabled}}enabled{{else}}disabled{{end}}{{secretState .Params}}
No alert channels configured.
+

Rules

{{range .Rules}}{{else}}{{end}}
NameLogQLCooldownLast countSuppressedStatus
{{.Name}}{{if not .Enabled}} disabled{{end}}{{.LogQL}}{{.Cooldown}}{{.LastMatchCount}}{{.SuppressedCount}}{{if .LastError}}{{.LastError}}{{else}}checked {{.LastCheckedAt}}{{end}}
No rules configured yet.
+

Recent incidents

{{range .Incidents}}{{else}}{{end}}
TimeRuleSeveritySummaryEvidence
{{.CreatedAt}}{{.RuleName}}{{.Severity}}{{.Summary}}{{range .AlertResults}}
{{.}}
{{end}}{{range .RemediationEvidence}}
{{.}}
{{end}}
No incidents recorded.
+
`)) + +func secretState(params map[string]string) string { + if params["token"] != "" || params["password"] != "" || params["secret"] != "" { + return "configured" + } + return "not configured" +} + +func render(w http.ResponseWriter, data store.Data) { + w.Header().Set("Content-Type", "text/html; charset=utf-8") + _ = tmpl.Execute(w, data) +} diff --git a/internal/store/store.go b/internal/store/store.go new file mode 100644 index 0000000..97f4e6b --- /dev/null +++ b/internal/store/store.go @@ -0,0 +1,260 @@ +package store + +import ( + "encoding/json" + "errors" + "os" + "path/filepath" + "strings" + "sync" + "time" +) + +type Rule struct { + ID string `json:"id"` + Name string `json:"name"` + Enabled bool `json:"enabled"` + LogQL string `json:"logql"` + Severity string `json:"severity"` + Threshold int `json:"threshold"` + Window string `json:"window"` + Cooldown string `json:"cooldown,omitempty"` + AlertChannels []string `json:"alert_channels,omitempty"` + AnalysisEnabled bool `json:"analysis_enabled"` + Actions []Action `json:"actions,omitempty"` + LastCheckedAt time.Time `json:"last_checked_at,omitempty"` + LastMatchedAt time.Time `json:"last_matched_at,omitempty"` + LastAlertedAt time.Time `json:"last_alerted_at,omitempty"` + LastMatchCount int `json:"last_match_count,omitempty"` + SuppressedCount int `json:"suppressed_count,omitempty"` + LastError string `json:"last_error,omitempty"` +} + +type Action struct { + Type string `json:"type"` + Enabled bool `json:"enabled"` + DryRun *bool `json:"dry_run,omitempty"` + Params map[string]string `json:"params,omitempty"` +} + +type AlertChannel struct { + ID string `json:"id"` + Name string `json:"name"` + Type string `json:"type"` + Enabled bool `json:"enabled"` + Params map[string]string `json:"params,omitempty"` +} + +type Incident struct { + ID string `json:"id"` + RuleID string `json:"rule_id"` + RuleName string `json:"rule_name"` + Severity string `json:"severity"` + Count int `json:"count"` + Summary string `json:"summary"` + AlertResults []string `json:"alert_results,omitempty"` + RemediationEvidence []string `json:"remediation_evidence,omitempty"` + CreatedAt time.Time `json:"created_at"` +} + +type Data struct { + Rules []Rule `json:"rules"` + AlertChannels []AlertChannel `json:"alert_channels"` + Incidents []Incident `json:"incidents"` +} + +type FileStore struct { + path string + mu sync.Mutex +} + +func NewFileStore(path string) *FileStore { return &FileStore{path: path} } + +func (s *FileStore) Load() (Data, error) { + s.mu.Lock() + defer s.mu.Unlock() + return s.loadLocked() +} + +func (s *FileStore) Save(data Data) error { + s.mu.Lock() + defer s.mu.Unlock() + return s.saveLocked(data) +} + +func (s *FileStore) UpsertRule(rule Rule) error { + s.mu.Lock() + defer s.mu.Unlock() + data, err := s.loadLocked() + if err != nil { + return err + } + if rule.ID == "" { + rule.ID = slug(rule.Name) + } + if rule.ID == "" { + return errors.New("rule id or name is required") + } + if rule.Threshold <= 0 { + rule.Threshold = 1 + } + if rule.Window == "" { + rule.Window = "5m" + } + if rule.Cooldown == "" { + rule.Cooldown = "1h" + } + for i := range data.Rules { + if data.Rules[i].ID == rule.ID { + rule.LastCheckedAt = data.Rules[i].LastCheckedAt + rule.LastMatchedAt = data.Rules[i].LastMatchedAt + rule.LastAlertedAt = data.Rules[i].LastAlertedAt + rule.LastMatchCount = data.Rules[i].LastMatchCount + rule.SuppressedCount = data.Rules[i].SuppressedCount + rule.LastError = data.Rules[i].LastError + data.Rules[i] = rule + return s.saveLocked(data) + } + } + data.Rules = append(data.Rules, rule) + return s.saveLocked(data) +} + +func (s *FileStore) UpsertAlertChannel(channel AlertChannel) error { + s.mu.Lock() + defer s.mu.Unlock() + data, err := s.loadLocked() + if err != nil { + return err + } + if channel.ID == "" { + channel.ID = slug(channel.Name) + } + if channel.ID == "" { + return errors.New("channel id or name is required") + } + if channel.Params == nil { + channel.Params = map[string]string{} + } + for i := range data.AlertChannels { + if data.AlertChannels[i].ID == channel.ID { + preserveSecrets(data.AlertChannels[i].Params, channel.Params) + data.AlertChannels[i] = channel + return s.saveLocked(data) + } + } + data.AlertChannels = append(data.AlertChannels, channel) + return s.saveLocked(data) +} + +func preserveSecrets(oldParams, newParams map[string]string) { + for _, key := range []string{"token", "password", "secret", "authorization"} { + if newParams[key] == "" && oldParams[key] != "" { + newParams[key] = oldParams[key] + } + } +} + +func (s *FileStore) DeleteAlertChannel(id string) error { + s.mu.Lock() + defer s.mu.Unlock() + data, err := s.loadLocked() + if err != nil { + return err + } + channels := data.AlertChannels[:0] + for _, channel := range data.AlertChannels { + if channel.ID != id { + channels = append(channels, channel) + } + } + data.AlertChannels = channels + for i := range data.Rules { + ids := data.Rules[i].AlertChannels[:0] + for _, channelID := range data.Rules[i].AlertChannels { + if channelID != id { + ids = append(ids, channelID) + } + } + data.Rules[i].AlertChannels = ids + } + return s.saveLocked(data) +} + +func (s *FileStore) UpdateRule(id string, fn func(*Rule)) error { + s.mu.Lock() + defer s.mu.Unlock() + data, err := s.loadLocked() + if err != nil { + return err + } + for i := range data.Rules { + if data.Rules[i].ID == id { + fn(&data.Rules[i]) + return s.saveLocked(data) + } + } + return errors.New("rule not found") +} + +func (s *FileStore) AddIncident(incident Incident) error { + s.mu.Lock() + defer s.mu.Unlock() + data, err := s.loadLocked() + if err != nil { + return err + } + if incident.ID == "" { + incident.ID = slug(incident.RuleName) + "-" + incident.CreatedAt.Format("20060102150405") + } + data.Incidents = append([]Incident{incident}, data.Incidents...) + if len(data.Incidents) > 200 { + data.Incidents = data.Incidents[:200] + } + return s.saveLocked(data) +} + +func (s *FileStore) loadLocked() (Data, error) { + var data Data + contents, err := os.ReadFile(s.path) + if errors.Is(err, os.ErrNotExist) { + return data, nil + } + if err != nil { + return data, err + } + if len(contents) == 0 { + return data, nil + } + return data, json.Unmarshal(contents, &data) +} + +func (s *FileStore) saveLocked(data Data) error { + if err := os.MkdirAll(filepath.Dir(s.path), 0o700); err != nil { + return err + } + contents, err := json.MarshalIndent(data, "", " ") + if err != nil { + return err + } + return os.WriteFile(s.path, append(contents, '\n'), 0o600) +} + +func slug(value string) string { + value = strings.ToLower(strings.TrimSpace(value)) + var b strings.Builder + lastDash := false + for _, r := range value { + ok := r >= 'a' && r <= 'z' || r >= '0' && r <= '9' + if ok { + b.WriteRune(r) + lastDash = false + continue + } + if !lastDash { + b.WriteByte('-') + lastDash = true + } + } + return strings.Trim(b.String(), "-") +} diff --git a/internal/store/store_test.go b/internal/store/store_test.go new file mode 100644 index 0000000..c604227 --- /dev/null +++ b/internal/store/store_test.go @@ -0,0 +1,38 @@ +package store + +import "testing" + +func TestUpsertRuleDefaults(t *testing.T) { + s := NewFileStore(t.TempDir() + "/config.json") + if err := s.UpsertRule(Rule{Name: "API Errors", LogQL: `{service="api"} |= "error"`}); err != nil { + t.Fatal(err) + } + data, err := s.Load() + if err != nil { + t.Fatal(err) + } + if len(data.Rules) != 1 { + t.Fatalf("expected one rule, got %d", len(data.Rules)) + } + rule := data.Rules[0] + if rule.ID != "api-errors" || rule.Threshold != 1 || rule.Window != "5m" || rule.Cooldown != "1h" { + t.Fatalf("unexpected defaults: %#v", rule) + } +} + +func TestUpsertAlertChannelPreservesSecret(t *testing.T) { + s := NewFileStore(t.TempDir() + "/config.json") + if err := s.UpsertAlertChannel(AlertChannel{ID: "ntfy", Name: "ntfy", Type: "ntfy", Enabled: true, Params: map[string]string{"token": "secret", "topic": "alerts"}}); err != nil { + t.Fatal(err) + } + if err := s.UpsertAlertChannel(AlertChannel{ID: "ntfy", Name: "ntfy", Type: "ntfy", Enabled: true, Params: map[string]string{"topic": "alerts2"}}); err != nil { + t.Fatal(err) + } + data, err := s.Load() + if err != nil { + t.Fatal(err) + } + if got := data.AlertChannels[0].Params["token"]; got != "secret" { + t.Fatalf("secret was not preserved: %q", got) + } +} diff --git a/project-docs/design/phase-1-plan.md b/project-docs/design/phase-1-plan.md new file mode 100644 index 0000000..e6fa601 --- /dev/null +++ b/project-docs/design/phase-1-plan.md @@ -0,0 +1,285 @@ +# Phase 1 Plan + +Goal: make log monitoring useful in production without enabling dangerous automation. + +Phase 1 targets a production Portainer deployment of Log Guardian. Loki will likely run on a separate server and use an object-storage backend, probably MinIO backed by NAS storage. Log Guardian must remain safe-by-default: authenticated UI, real alert delivery, incident cooldowns, Loki validation tooling, dry-run remediation evidence, and local-only AI analysis. + +## Decisions made + +- Deployment target: production Portainer. +- Loki location: separate server from Log Guardian. +- Loki storage direction: object storage via MinIO, with MinIO data on NAS-backed storage. +- UI auth: single admin user using basic auth. +- Health endpoint: `/healthz` remains unauthenticated. +- First alert channel: self-hosted ntfy. +- Additional alert channel types should be supported/configurable from UI, not ntfy-only. +- Incident cooldown: per rule, default `1h`. +- AI: local-only Ollama is approved; no external/cloud AI providers. +- Remediation: Phase 1 should implement permission modeling and dry-run evidence, not dangerous automatic actions. +- Phase 1 completion requires Docker build and real alert delivery validation. + +## 1. Production deployment architecture + +### Log Guardian + +Run Log Guardian as a Portainer-managed service/container. + +Required mounts/config: + +- Persistent local config/data file mount for `LOG_GUARDIAN_DATA_PATH`. +- Environment configuration for Loki endpoint, auth, alert defaults, and optional Ollama endpoint. +- Network path to the remote Loki server. +- Optional network path to self-hosted ntfy, Gotify, SMTP relay, webhook targets, and Ollama. + +### Loki + +Loki will run on a different server. Recommended production direction: + +- Loki single-binary or simple scalable deployment depending on expected volume. +- Object storage backend: S3-compatible MinIO. +- MinIO data stored on NAS-backed storage. +- Retention configured explicitly. +- Promtail or Grafana Alloy deployed on every Docker/Portainer host that needs logs collected. + +Recommended initial storage sizing: + +- Start with 50 GB usable object storage for modest private infrastructure. +- Use 14-day retention initially. +- Monitor ingestion and object growth for 7 days before increasing retention. +- Increase to 100-250 GB+ if services are noisy or retention must be longer. + +Planning assumptions to validate: + +- NAS latency is acceptable for MinIO/Loki workload. +- MinIO has backups/snapshots appropriate for desired retention. +- Loki retention deletes old chunks/index data correctly. +- Log Guardian can reach Loki over a private network/VPN only. + +## 2. Loki ingestion, label discovery, and validation + +Because actual labels are not known yet, Phase 1 must include label discovery and query testing. + +### Collection strategy + +Use Promtail or Grafana Alloy on Docker/Portainer hosts. Target labels should include stable, low-cardinality dimensions such as: + +- `host` +- `container` +- `service` +- `stack` +- `stream` +- `compose_project` or equivalent +- `compose_service` or equivalent + +Avoid high-cardinality labels, including request IDs, user IDs, IP addresses, paths, error messages, or raw log content. + +### Log Guardian Loki diagnostics + +Add authenticated UI/API support for: + +- Loki readiness/status check. +- List labels from `/loki/api/v1/labels`. +- List values for a selected label from `/loki/api/v1/label/{name}/values`. +- Test a LogQL query before saving a rule. +- Show count and sanitized sample lines for a test query. + +Implementation notes: + +- Limit query time range and result count. +- Do not log raw sample payloads by default. +- Redact obvious secrets from displayed query samples where practical. +- Handle Loki tenant ID and basic auth from existing Loki config. + +## 3. UI authentication + +Add single-admin basic auth. + +Configuration: + +- `LOG_GUARDIAN_AUTH_USERNAME` +- `LOG_GUARDIAN_AUTH_PASSWORD` + +Rules: + +- `/healthz` is unauthenticated. +- All other UI/API routes require auth. +- Use constant-time comparison for credentials. +- Do not log usernames/passwords or auth headers. +- If credentials are unset, fail closed in production or clearly mark auth as disabled only for local development. Prefer fail closed for Portainer deployment. + +## 4. Alert channels + +Alert channels must be configurable from the UI and stored in local configuration. Secrets need careful handling. + +### Required first channel: self-hosted ntfy + +Support fields: + +- Name +- Enabled +- Server URL +- Topic +- Optional bearer token or token file reference +- Priority +- Tags + +Validation: + +- Send a test notification from the UI. +- Record delivery result without logging token values. + +### Additional Phase 1 channel types + +Implement the data model and UI so multiple channel types are possible. Preferred Phase 1 support order: + +1. `ntfy` - implement real delivery first. +2. `gotify` - real delivery if time allows; token-based API is straightforward. +3. `generic_webhook` - useful but must avoid leaking secrets; support explicit headers carefully. +4. `smtp` - useful, but can be deferred if it complicates TLS/auth validation. + +Minimum acceptable Phase 1 scope: + +- UI can create/edit/delete alert channels. +- UI can attach channels to rules. +- ntfy real delivery works. +- Other channel types may be present as disabled/scaffolded if not implemented, but the UI should clearly show unsupported status. + +Secret handling: + +- Do not print secrets in logs. +- Do not render stored secret values back into forms; show only `configured`/`not configured`. +- Prefer environment variable or mounted file references for sensitive channel tokens when possible. +- If secrets are stored in local JSON, the file must be written with `0600` permissions and documented as sensitive. + +## 5. Rule evaluation and incident cooldown + +Add per-rule cooldown. + +Rule field: + +```json +"cooldown": "1h" +``` + +Default behavior: + +- If unset, cooldown is `1h`. +- If a rule fires outside cooldown: + - create incident; + - send configured alert channels; + - record dry-run remediation evidence if actions are configured. +- If a rule fires during cooldown: + - update rule last checked/matched/count fields; + - do not create a duplicate incident; + - do not send another alert; + - increment or record suppressed count if implemented. + +## 6. Remediation/action model + +Phase 1 should not execute destructive remediation. Implement guardrails and dry-run evidence. + +Recommended Phase 1 action types: + +- `generic_webhook`: dry-run by default; real execution can be deferred or restricted. +- `portainer_restart`: dry-run only in Phase 1. +- `allowlisted_command`: dry-run only in Phase 1; command IDs only, never arbitrary shell from UI/AI. +- `record_recommendation`: safe action that stores an advisory recommendation on the incident. + +Guardrails: + +- Global dry-run defaults true. +- Per-rule action must be enabled. +- Per-action dry-run should be explicit or inherited from global dry-run. +- AI may recommend actions but cannot execute them. +- No arbitrary command execution. +- No Portainer write API calls in Phase 1 unless separately approved. + +Validation evidence: + +- At least one rule with a configured dry-run action. +- Incident shows what would have happened. +- Logs do not expose secrets or raw full log payloads. + +## 7. Local-only AI analysis + +Local-only Ollama is approved for Phase 1. External/cloud AI providers are not approved. + +Configuration: + +- `LOG_GUARDIAN_ANALYSIS_PROVIDER=none|ollama` +- `LOG_GUARDIAN_OLLAMA_URL=http://ollama:11434` +- `LOG_GUARDIAN_OLLAMA_MODEL=` + +Default: + +- `LOG_GUARDIAN_ANALYSIS_PROVIDER=none` + +Rules: + +- AI analysis remains opt-in per rule. +- AI findings are advisory only. +- Limit number of log lines sent to Ollama. +- Redact obvious secrets before prompt construction. +- Do not log prompts or raw log payloads by default. +- Store only concise summaries/recommendations in incidents. + +Prompt/redaction design must be documented before enabling Ollama in production. + +## 8. UI scope + +Add UI support for: + +- Login-protected dashboard. +- Rule create/edit with cooldown and alert channel selection. +- Alert channel create/edit/test. +- Loki status/label discovery/query test. +- Incident list showing alert delivery state and remediation dry-run evidence where available. + +Keep the UI served by Go. Do not add a server-side JavaScript runtime. + +## 9. Tests and validation + +Required implementation tests: + +- Auth middleware unit tests. +- Store tests for alert channels, cooldown fields, and secret-preserving updates. +- Loki client tests using `httptest` fixtures for labels, label values, and query range. +- Alert dispatcher tests for ntfy using `httptest`. +- Rule engine tests for cooldown suppression. +- Remediation dry-run tests. +- Optional Ollama provider tests using `httptest` only; no live AI dependency in unit tests. + +Required validation before Phase 1 completion: + +```sh +gofmt -w cmd internal +go test ./... +make validate-structure +make docker-build +``` + +Additional required evidence: + +- Loki connectivity/query validation against the remote Loki server. +- Label discovery output documented without secrets. +- Self-hosted ntfy test alert received. +- A rule-generated ntfy alert received. +- Remediation dry-run evidence captured. +- Security/privacy review covering logs, prompts, secrets, alert tokens, and action permissions. + +## 10. Required Phase 1 artifacts + +Create/update these before declaring Phase 1 complete: + +- `project-docs/status/phase-1-summary.md` +- `project-docs/status/phase-1-test-results.md` +- `project-docs/status/phase-1-open-issues.md` + +## 11. Open planning items + +- Final Loki topology: single-binary vs simple scalable. +- Final MinIO/NAS capacity and retention target after observing ingestion. +- Whether Gotify, generic webhook, or SMTP are implemented fully in Phase 1 or scaffolded after ntfy. +- Whether alert channel secrets are stored directly in local JSON or referenced via env/file paths. +- Exact Ollama model to use once Ollama is stable. +- Whether Log Guardian reaches Loki/ntfy/Ollama over LAN, VPN, or Docker overlay network. diff --git a/project-docs/runbooks/loki-setup.md b/project-docs/runbooks/loki-setup.md new file mode 100644 index 0000000..ac62d35 --- /dev/null +++ b/project-docs/runbooks/loki-setup.md @@ -0,0 +1,29 @@ +# Loki Setup Runbook + +## Starter local/Portainer stack + +Use `deploy/loki/docker-compose.yml` and `deploy/loki/promtail-config.yml`. + +```sh +cd deploy/loki +docker compose up -d +``` + +## Verify Loki + +```sh +curl -fsS http://localhost:3100/ready +curl -G http://localhost:3100/loki/api/v1/labels +``` + +## Useful LogQL examples + +```logql +{service="api"} |= "error" +{stack="second-brain"} |~ "(?i)(panic|fatal|exception)" +{stream="stderr"} |~ "(?i)(error|failed|timeout)" +``` + +## Portainer notes + +For Portainer deployment, mount the Docker socket and Docker container log directory read-only into Promtail. The included compose does this for a single Docker host. Multi-node Swarm setups need Promtail on each node or another log shipping strategy. diff --git a/project-docs/status/phase-0-open-issues.md b/project-docs/status/phase-0-open-issues.md new file mode 100644 index 0000000..af989eb --- /dev/null +++ b/project-docs/status/phase-0-open-issues.md @@ -0,0 +1,8 @@ +# Phase 0 Open Issues + +- Add UI authentication before exposing beyond a trusted private network. +- Validate the Loki + Promtail setup against actual Portainer/Docker hosts. +- Decide alert channels: email, ntfy, Gotify, Discord, Matrix, SMS, or another destination. +- Decide which remediation actions are allowed. Candidates: send webhook, restart Portainer service, scale service, pause updater, open issue, or run a tightly allowlisted command. +- AI analysis provider is intentionally unset. Any external AI/cloud provider requires explicit approval and prompt/log redaction design. +- Add real alert delivery, retry behavior, deduplication, silences, and escalation policies. diff --git a/project-docs/status/phase-0-summary.md b/project-docs/status/phase-0-summary.md new file mode 100644 index 0000000..0630bd9 --- /dev/null +++ b/project-docs/status/phase-0-summary.md @@ -0,0 +1,16 @@ +# Phase 0 Summary + +Created initial `log-guardian` scaffold. + +Included: + +- Go HTTP service with health endpoint and basic web UI. +- Local JSON-backed rule, alert-channel, and incident storage. +- Loki query client for LogQL range queries. +- Rule engine with thresholds and incident creation. +- No-op analyzer interface for future AI agent analysis. +- Dry-run remediation action runner scaffold. +- Alert dispatcher scaffold that logs alert intent only. +- Dockerfile, Portainer stack, Loki + Promtail starter compose, example environment, Makefile, CI workflow, and phase artifacts. + +Phase 0 does not certify production alerting or remediation. diff --git a/project-docs/status/phase-0-test-results.md b/project-docs/status/phase-0-test-results.md new file mode 100644 index 0000000..7a6c9ff --- /dev/null +++ b/project-docs/status/phase-0-test-results.md @@ -0,0 +1,19 @@ +# Phase 0 Test Results + +Validation run from `/data/code/log-guardian` after scaffold creation. + +Passed: + +```sh +gofmt -w cmd internal +go test ./... +make validate-structure +make go-fmt +make docker-build +``` + +Evidence: + +- `go test ./...` passed for all packages, including `internal/analysis` and `internal/store` tests. +- `make validate-structure` printed `Structure validation passed`. +- Docker image built successfully as `gitea.wayfinderak.com/wayfinderak/log-guardian:local`. diff --git a/project-docs/status/phase-1-open-issues.md b/project-docs/status/phase-1-open-issues.md new file mode 100644 index 0000000..5a650a2 --- /dev/null +++ b/project-docs/status/phase-1-open-issues.md @@ -0,0 +1,10 @@ +# Phase 1 Open Issues + +- Live deployment validation is still required against production Portainer, remote Loki, and self-hosted ntfy. +- Loki topology is not finalized: single-binary vs scalable deployment, exact MinIO/NAS capacity, and retention settings need confirmation. +- Promtail/Grafana Alloy labels must be discovered on the real Docker/Portainer hosts. +- Gotify, generic webhook, and SMTP channels are scaffolded but not implemented for real delivery. +- Alert channel secrets are currently stored in the local JSON config if entered through the UI; mounted secret-file/env references should be considered for stronger secret handling. +- Ollama provider is implemented as local-only and opt-in, but the final model and runtime deployment still need validation. +- No destructive remediation is implemented; Portainer restart and allowlisted command actions are dry-run evidence only. +- The UI supports create/replace workflows but does not yet provide polished edit forms for existing rules/channels. diff --git a/project-docs/status/phase-1-summary.md b/project-docs/status/phase-1-summary.md new file mode 100644 index 0000000..189fab4 --- /dev/null +++ b/project-docs/status/phase-1-summary.md @@ -0,0 +1,16 @@ +# Phase 1 Summary + +Implemented Phase 1 foundations for production use: + +- Basic auth protects UI/API routes; `/healthz` remains unauthenticated. +- Loki diagnostics endpoints were added for labels, label values, and limited query testing. +- Rules now support per-rule cooldowns with default `1h` and duplicate incident suppression. +- Alert channels are configurable in the UI. +- Real self-hosted ntfy delivery is implemented with test-send support. +- Gotify, generic webhook, and SMTP channel types are represented as scaffolded options. +- Incidents can record alert delivery results and remediation dry-run evidence. +- Remediation action runner supports guarded dry-run evidence for generic webhook, Portainer restart, allowlisted command, and recommendation actions. +- Local-only Ollama analysis support was added and remains disabled by default. +- Secret handling was improved for alert channel token preservation and display masking. + +Phase 1 production rollout still requires live validation against the user's remote Loki and self-hosted ntfy deployments. diff --git a/project-docs/status/phase-1-test-results.md b/project-docs/status/phase-1-test-results.md new file mode 100644 index 0000000..97e9312 --- /dev/null +++ b/project-docs/status/phase-1-test-results.md @@ -0,0 +1,37 @@ +# Phase 1 Test Results + +Validation run from `/data/code/log-guardian` after Phase 1 implementation. + +Passed: + +```sh +gofmt -w cmd internal +go test ./... +make validate-structure +make go-fmt +make go-test +make docker-build +``` + +Docker image built successfully as: + +```text +gitea.wayfinderak.com/wayfinderak/log-guardian:local +``` + +Unit/integration-style tests added: + +- Basic auth middleware protects UI routes and leaves `/healthz` unauthenticated. +- Store defaults include rule cooldown and alert channel secret-preserving updates. +- Loki client label, label-value, and query-range behavior uses `httptest` fixtures. +- ntfy dispatcher delivery uses an `httptest` fixture. +- Rule engine suppresses duplicate incidents during cooldown. + +Pending live production validation before Phase 1 can be called operationally complete: + +- Remote Loki connectivity and label discovery. +- Remote Loki LogQL query validation. +- Self-hosted ntfy test notification received. +- Rule-generated ntfy alert received. +- Remediation dry-run evidence captured from a real rule. +- Security/privacy review of logs, prompts, secrets, alert tokens, and action permissions. diff --git a/web/static/app.css b/web/static/app.css new file mode 100644 index 0000000..65ec92f --- /dev/null +++ b/web/static/app.css @@ -0,0 +1 @@ +:root{color-scheme:dark;--bg:#0d1117;--panel:#161b22;--text:#f0f6fc;--muted:#8b949e;--accent:#58a6ff;--danger:#ff7b72}*{box-sizing:border-box}body{margin:0;background:var(--bg);color:var(--text);font-family:system-ui,-apple-system,Segoe UI,sans-serif}.shell{max-width:1200px;margin:0 auto;padding:2rem}.card{background:var(--panel);border:1px solid #30363d;border-radius:16px;padding:1rem;margin:1rem 0}.grid{display:grid;grid-template-columns:repeat(auto-fit,minmax(220px,1fr));gap:1rem;align-items:end}.wide{grid-column:1/-1}label{display:grid;gap:.35rem;color:var(--muted)}.check{display:flex;gap:.5rem;align-items:center}input{width:100%;padding:.65rem;border-radius:10px;border:1px solid #30363d;background:#010409;color:var(--text)}button{padding:.65rem .9rem;border:0;border-radius:10px;background:var(--accent);color:#06111f;font-weight:700;cursor:pointer}.row{display:flex;justify-content:space-between;gap:1rem;align-items:center}table{width:100%;border-collapse:collapse}th,td{text-align:left;padding:.7rem;border-top:1px solid #30363d;vertical-align:top}code{color:#a5d6ff}.muted{color:var(--muted)}.error{color:var(--danger)}@media(max-width:760px){.shell{padding:1rem}table{display:block;overflow-x:auto}} \ No newline at end of file