Implement phase 1 foundations
All checks were successful
build-image / docker (push) Successful in 53s
All checks were successful
build-image / docker (push) Successful in 53s
This commit is contained in:
commit
6f7549d439
5
.dockerignore
Normal file
5
.dockerignore
Normal file
@ -0,0 +1,5 @@
|
||||
.git
|
||||
.env
|
||||
.env.*
|
||||
data
|
||||
.pi-subagents
|
||||
15
.env.example
Normal file
15
.env.example
Normal file
@ -0,0 +1,15 @@
|
||||
LOG_GUARDIAN_ADDR=:8080
|
||||
LOG_GUARDIAN_DATA_PATH=./data/config.json
|
||||
LOG_GUARDIAN_POLL_INTERVAL=1m
|
||||
LOG_GUARDIAN_DRY_RUN_ACTIONS=true
|
||||
LOG_GUARDIAN_AUTH_USERNAME=admin
|
||||
LOG_GUARDIAN_AUTH_PASSWORD=change-me
|
||||
LOKI_URL=http://localhost:3100
|
||||
LOKI_TENANT_ID=
|
||||
LOKI_USERNAME=
|
||||
LOKI_PASSWORD=
|
||||
LOG_GUARDIAN_ANALYSIS_PROVIDER=none
|
||||
LOG_GUARDIAN_ANALYSIS_ENDPOINT=
|
||||
LOG_GUARDIAN_ANALYSIS_TOKEN=
|
||||
LOG_GUARDIAN_OLLAMA_URL=http://ollama:11434
|
||||
LOG_GUARDIAN_OLLAMA_MODEL=llama3.1
|
||||
13
.gitea/workflows/build-image.yml
Normal file
13
.gitea/workflows/build-image.yml
Normal file
@ -0,0 +1,13 @@
|
||||
name: build-image
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
|
||||
jobs:
|
||||
docker:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Build image
|
||||
run: make docker-build IMAGE_TAG=${{ gitea.sha }}
|
||||
17
.github/workflows/ci.yml
vendored
Normal file
17
.github/workflows/ci.yml
vendored
Normal file
@ -0,0 +1,17 @@
|
||||
name: ci
|
||||
|
||||
on:
|
||||
push:
|
||||
pull_request:
|
||||
|
||||
jobs:
|
||||
test:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-go@v5
|
||||
with:
|
||||
go-version: '1.23'
|
||||
- run: make validate-structure
|
||||
- run: make go-fmt
|
||||
- run: make go-test
|
||||
9
.gitignore
vendored
Normal file
9
.gitignore
vendored
Normal file
@ -0,0 +1,9 @@
|
||||
.env
|
||||
.env.*
|
||||
!.env.example
|
||||
data/
|
||||
dist/
|
||||
build/
|
||||
coverage/
|
||||
*.log
|
||||
.DS_Store
|
||||
33
AGENTS.md
Normal file
33
AGENTS.md
Normal file
@ -0,0 +1,33 @@
|
||||
# AGENTS.md
|
||||
|
||||
This repository is for a private log monitoring, alerting, and controlled remediation service built around Loki.
|
||||
|
||||
## Hard rules
|
||||
|
||||
- Server-side code must be Go.
|
||||
- The web UI is served by Go; do not add a server-side JavaScript runtime.
|
||||
- Never log alert destination secrets, API keys, tokens, AI prompts containing private logs, or full raw log payloads unless explicitly configured for a local-only debug session.
|
||||
- Do not add external cloud/AI/data integrations without explicit user approval.
|
||||
- Automatic remediation must be opt-in per rule and should default to dry-run.
|
||||
- AI analysis is advisory by default; AI-generated actions must not execute unless a rule explicitly allows that action and passes guardrails.
|
||||
- Preserve local data ownership: runtime configuration is stored in mounted local files unless a later phase explicitly changes storage.
|
||||
|
||||
## Required artifacts per phase
|
||||
|
||||
- `project-docs/status/phase-N-summary.md`
|
||||
- `project-docs/status/phase-N-test-results.md`
|
||||
- `project-docs/status/phase-N-open-issues.md`
|
||||
|
||||
## Required validation before advancing
|
||||
|
||||
Run the checks required by the phase, including as applicable:
|
||||
|
||||
- formatting/lint
|
||||
- unit tests
|
||||
- Docker build
|
||||
- Loki query validation
|
||||
- alert delivery validation
|
||||
- remediation dry-run evidence
|
||||
- security/privacy review for logs, prompts, secrets, and action permissions
|
||||
|
||||
If any required check fails, the phase is not complete.
|
||||
31
Makefile
Normal file
31
Makefile
Normal file
@ -0,0 +1,31 @@
|
||||
IMAGE_TAG ?= local
|
||||
REGISTRY ?= gitea.wayfinderak.com
|
||||
IMAGE_NAMESPACE ?= wayfinderak
|
||||
IMAGE_PREFIX ?= log-guardian
|
||||
IMAGE_REF ?= $(REGISTRY)/$(IMAGE_NAMESPACE)/$(IMAGE_PREFIX):$(IMAGE_TAG)
|
||||
DOCKER_BUILD_FLAGS ?=
|
||||
|
||||
.PHONY: validate-structure go-fmt go-test docker-build ci-fast
|
||||
|
||||
validate-structure:
|
||||
@test -f AGENTS.md
|
||||
@test -f README.md
|
||||
@test -f project-docs/status/phase-0-summary.md
|
||||
@test -f project-docs/status/phase-0-test-results.md
|
||||
@test -f project-docs/status/phase-0-open-issues.md
|
||||
@test -f project-docs/status/phase-1-summary.md
|
||||
@test -f project-docs/status/phase-1-test-results.md
|
||||
@test -f project-docs/status/phase-1-open-issues.md
|
||||
@test -f deploy/loki/docker-compose.yml
|
||||
@echo "Structure validation passed"
|
||||
|
||||
go-fmt:
|
||||
@test -z "$$(gofmt -l cmd internal)"
|
||||
|
||||
go-test:
|
||||
go test ./...
|
||||
|
||||
docker-build:
|
||||
docker build $(DOCKER_BUILD_FLAGS) -f docker/server.Dockerfile -t $(IMAGE_REF) .
|
||||
|
||||
ci-fast: validate-structure go-fmt go-test docker-build
|
||||
92
README.md
Normal file
92
README.md
Normal file
@ -0,0 +1,92 @@
|
||||
# Log Guardian
|
||||
|
||||
Private Loki-backed service for log monitoring, alerting, and guarded remediation.
|
||||
|
||||
Current implementation includes the Phase 1 foundations:
|
||||
|
||||
- watches configured LogQL rules;
|
||||
- records incidents when thresholds are met;
|
||||
- enforces per-rule cooldowns;
|
||||
- protects the UI with basic auth while leaving `/healthz` open;
|
||||
- provides Loki label/query diagnostics;
|
||||
- supports UI-configurable alert channels with real ntfy delivery;
|
||||
- keeps remediation in dry-run by default;
|
||||
- includes local-only Ollama analysis support, disabled by default.
|
||||
|
||||
## Run locally
|
||||
|
||||
```sh
|
||||
cp .env.example .env
|
||||
# edit env values, then export them or use your shell dotenv helper
|
||||
go run ./cmd/server
|
||||
```
|
||||
|
||||
Open <http://localhost:8080>.
|
||||
|
||||
## Configuration
|
||||
|
||||
| Variable | Default | Purpose |
|
||||
| --- | --- | --- |
|
||||
| `LOG_GUARDIAN_ADDR` | `:8080` | HTTP listen address. |
|
||||
| `LOG_GUARDIAN_DATA_PATH` | `/data/config.json` | Local JSON config. |
|
||||
| `LOG_GUARDIAN_POLL_INTERVAL` | `1m` | Rule evaluation interval. |
|
||||
| `LOG_GUARDIAN_DRY_RUN_ACTIONS` | `true` | Prevents real remediation actions. |
|
||||
| `LOG_GUARDIAN_AUTH_USERNAME` / `LOG_GUARDIAN_AUTH_PASSWORD` | empty | Required basic auth credentials for UI/API routes except `/healthz`. |
|
||||
| `LOKI_URL` | `http://loki:3100` | Loki base URL. |
|
||||
| `LOKI_TENANT_ID` | empty | Optional Loki tenant header. |
|
||||
| `LOKI_USERNAME` / `LOKI_PASSWORD` | empty | Optional basic auth. |
|
||||
| `LOG_GUARDIAN_ANALYSIS_PROVIDER` | `none` | `none` or local-only `ollama`. |
|
||||
| `LOG_GUARDIAN_OLLAMA_URL` / `LOG_GUARDIAN_OLLAMA_MODEL` | `http://ollama:11434` / `llama3.1` | Local Ollama analysis config. |
|
||||
|
||||
## Initial Loki setup
|
||||
|
||||
A starter Loki + Promtail stack is in `deploy/loki/`:
|
||||
|
||||
```sh
|
||||
cd deploy/loki
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
Promtail is configured to discover Docker containers and attach useful labels:
|
||||
|
||||
- `container`
|
||||
- `service`
|
||||
- `stack`
|
||||
- `stream`
|
||||
|
||||
Example LogQL rules:
|
||||
|
||||
```logql
|
||||
{service="api"} |= "error"
|
||||
{stack="second-brain"} |~ "(?i)(panic|fatal|exception)"
|
||||
{container=~".*postgres.*"} |= "database system is ready"
|
||||
```
|
||||
|
||||
## Deploy in Portainer
|
||||
|
||||
Use `deploy/portainer-stack.yml` with values from `deploy/portainer.env.example`.
|
||||
|
||||
Keep this set while testing:
|
||||
|
||||
```text
|
||||
LOG_GUARDIAN_DRY_RUN_ACTIONS=true
|
||||
LOG_GUARDIAN_ANALYSIS_PROVIDER=none
|
||||
```
|
||||
|
||||
## Build and validate
|
||||
|
||||
```sh
|
||||
make ci-fast
|
||||
```
|
||||
|
||||
## Safety model
|
||||
|
||||
Automatic remediation is intentionally constrained:
|
||||
|
||||
1. A rule must match.
|
||||
2. The configured threshold must be exceeded.
|
||||
3. The action must be explicitly enabled on that rule.
|
||||
4. Global dry-run must be disabled.
|
||||
5. Future AI recommendations are advisory unless a rule explicitly allows a matching action type.
|
||||
|
||||
Do not enable production remediation until a later phase explicitly approves concrete action types and guardrails.
|
||||
49
cmd/server/main.go
Normal file
49
cmd/server/main.go
Normal file
@ -0,0 +1,49 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log"
|
||||
"net/http"
|
||||
"os/signal"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"gitea.wayfinderak.com/wayfinderak/log-guardian/internal/actions"
|
||||
"gitea.wayfinderak.com/wayfinderak/log-guardian/internal/alerts"
|
||||
"gitea.wayfinderak.com/wayfinderak/log-guardian/internal/analysis"
|
||||
"gitea.wayfinderak.com/wayfinderak/log-guardian/internal/config"
|
||||
"gitea.wayfinderak.com/wayfinderak/log-guardian/internal/loki"
|
||||
"gitea.wayfinderak.com/wayfinderak/log-guardian/internal/rules"
|
||||
"gitea.wayfinderak.com/wayfinderak/log-guardian/internal/server"
|
||||
"gitea.wayfinderak.com/wayfinderak/log-guardian/internal/store"
|
||||
)
|
||||
|
||||
func main() {
|
||||
cfg := config.Load()
|
||||
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
|
||||
defer stop()
|
||||
|
||||
fileStore := store.NewFileStore(cfg.DataPath)
|
||||
lokiClient := loki.New(cfg.LokiURL, cfg.LokiTenantID, cfg.LokiUsername, cfg.LokiPassword)
|
||||
var analyzer analysis.Analyzer = analysis.NoopAnalyzer{}
|
||||
if cfg.AnalysisProvider == "ollama" {
|
||||
analyzer = analysis.NewOllamaAnalyzer(cfg.OllamaURL, cfg.OllamaModel)
|
||||
}
|
||||
actionRunner := actions.NewRunner(cfg.DryRunActions)
|
||||
dispatcher := alerts.NewDispatcher()
|
||||
engine := rules.NewEngine(fileStore, lokiClient, analyzer, actionRunner, dispatcher)
|
||||
go engine.Start(ctx, cfg.PollInterval)
|
||||
|
||||
httpServer := &http.Server{Addr: cfg.Addr, Handler: server.New(fileStore, engine, lokiClient, dispatcher, cfg.AuthUsername, cfg.AuthPassword).Routes()}
|
||||
go func() {
|
||||
<-ctx.Done()
|
||||
shutdownCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
defer cancel()
|
||||
_ = httpServer.Shutdown(shutdownCtx)
|
||||
}()
|
||||
|
||||
log.Printf("log-guardian listening on %s", cfg.Addr)
|
||||
if err := httpServer.ListenAndServe(); err != nil && err != http.ErrServerClosed {
|
||||
log.Fatal(err)
|
||||
}
|
||||
}
|
||||
24
deploy/loki/docker-compose.yml
Normal file
24
deploy/loki/docker-compose.yml
Normal file
@ -0,0 +1,24 @@
|
||||
services:
|
||||
loki:
|
||||
image: grafana/loki:3.2.1
|
||||
command: -config.file=/etc/loki/local-config.yaml
|
||||
ports:
|
||||
- "${LOKI_PORT:-3100}:3100"
|
||||
volumes:
|
||||
- loki_data:/loki
|
||||
restart: unless-stopped
|
||||
|
||||
promtail:
|
||||
image: grafana/promtail:3.2.1
|
||||
command: -config.file=/etc/promtail/config.yml
|
||||
volumes:
|
||||
- /var/log:/var/log:ro
|
||||
- /var/lib/docker/containers:/var/lib/docker/containers:ro
|
||||
- /var/run/docker.sock:/var/run/docker.sock:ro
|
||||
- ./promtail-config.yml:/etc/promtail/config.yml:ro
|
||||
depends_on:
|
||||
- loki
|
||||
restart: unless-stopped
|
||||
|
||||
volumes:
|
||||
loki_data:
|
||||
25
deploy/loki/promtail-config.yml
Normal file
25
deploy/loki/promtail-config.yml
Normal file
@ -0,0 +1,25 @@
|
||||
server:
|
||||
http_listen_port: 9080
|
||||
grpc_listen_port: 0
|
||||
|
||||
positions:
|
||||
filename: /tmp/positions.yaml
|
||||
|
||||
clients:
|
||||
- url: http://loki:3100/loki/api/v1/push
|
||||
|
||||
scrape_configs:
|
||||
- job_name: docker
|
||||
docker_sd_configs:
|
||||
- host: unix:///var/run/docker.sock
|
||||
refresh_interval: 10s
|
||||
relabel_configs:
|
||||
- source_labels: ['__meta_docker_container_name']
|
||||
regex: '/(.*)'
|
||||
target_label: 'container'
|
||||
- source_labels: ['__meta_docker_container_label_com_docker_compose_service']
|
||||
target_label: 'service'
|
||||
- source_labels: ['__meta_docker_container_label_com_docker_compose_project']
|
||||
target_label: 'stack'
|
||||
- source_labels: ['__meta_docker_container_log_stream']
|
||||
target_label: 'stream'
|
||||
28
deploy/portainer-stack.yml
Normal file
28
deploy/portainer-stack.yml
Normal file
@ -0,0 +1,28 @@
|
||||
services:
|
||||
log-guardian:
|
||||
image: gitea.wayfinderak.com/wayfinderak/log-guardian:${IMAGE_TAG:-latest}
|
||||
environment:
|
||||
LOG_GUARDIAN_ADDR: :8080
|
||||
LOG_GUARDIAN_DATA_PATH: /data/config.json
|
||||
LOG_GUARDIAN_POLL_INTERVAL: ${LOG_GUARDIAN_POLL_INTERVAL:-1m}
|
||||
LOG_GUARDIAN_DRY_RUN_ACTIONS: ${LOG_GUARDIAN_DRY_RUN_ACTIONS:-true}
|
||||
LOKI_URL: ${LOKI_URL:-http://loki:3100}
|
||||
LOKI_TENANT_ID: ${LOKI_TENANT_ID:-}
|
||||
LOKI_USERNAME: ${LOKI_USERNAME:-}
|
||||
LOKI_PASSWORD: ${LOKI_PASSWORD:-}
|
||||
LOG_GUARDIAN_ANALYSIS_PROVIDER: ${LOG_GUARDIAN_ANALYSIS_PROVIDER:-none}
|
||||
LOG_GUARDIAN_ANALYSIS_ENDPOINT: ${LOG_GUARDIAN_ANALYSIS_ENDPOINT:-}
|
||||
LOG_GUARDIAN_ANALYSIS_TOKEN: ${LOG_GUARDIAN_ANALYSIS_TOKEN:-}
|
||||
volumes:
|
||||
- log_guardian_data:/data
|
||||
ports:
|
||||
- "${LOG_GUARDIAN_PORT:-8081}:8080"
|
||||
healthcheck:
|
||||
test: ["CMD", "wget", "-qO-", "http://127.0.0.1:8080/healthz"]
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
restart: unless-stopped
|
||||
|
||||
volumes:
|
||||
log_guardian_data:
|
||||
11
deploy/portainer.env.example
Normal file
11
deploy/portainer.env.example
Normal file
@ -0,0 +1,11 @@
|
||||
IMAGE_TAG=latest
|
||||
LOG_GUARDIAN_PORT=8081
|
||||
LOG_GUARDIAN_POLL_INTERVAL=1m
|
||||
LOG_GUARDIAN_DRY_RUN_ACTIONS=true
|
||||
LOKI_URL=http://loki:3100
|
||||
LOKI_TENANT_ID=
|
||||
LOKI_USERNAME=
|
||||
LOKI_PASSWORD=
|
||||
LOG_GUARDIAN_ANALYSIS_PROVIDER=none
|
||||
LOG_GUARDIAN_ANALYSIS_ENDPOINT=
|
||||
LOG_GUARDIAN_ANALYSIS_TOKEN=
|
||||
19
docker/server.Dockerfile
Normal file
19
docker/server.Dockerfile
Normal file
@ -0,0 +1,19 @@
|
||||
FROM golang:1.23-alpine AS build
|
||||
WORKDIR /src
|
||||
COPY go.mod ./
|
||||
COPY cmd ./cmd
|
||||
COPY internal ./internal
|
||||
COPY web ./web
|
||||
RUN go build -trimpath -ldflags="-s -w" -o /out/log-guardian ./cmd/server
|
||||
|
||||
FROM alpine:3.20
|
||||
LABEL org.opencontainers.image.title="log-guardian"
|
||||
LABEL org.opencontainers.image.description="Loki-backed log monitoring, alerting, and guarded remediation service."
|
||||
WORKDIR /app
|
||||
RUN adduser -D -H guardian && mkdir -p /data && chown guardian:guardian /data
|
||||
COPY --from=build /out/log-guardian /usr/local/bin/log-guardian
|
||||
COPY --from=build /src/web ./web
|
||||
USER guardian
|
||||
EXPOSE 8080
|
||||
VOLUME ["/data"]
|
||||
CMD ["/usr/local/bin/log-guardian"]
|
||||
3
go.mod
Normal file
3
go.mod
Normal file
@ -0,0 +1,3 @@
|
||||
module gitea.wayfinderak.com/wayfinderak/log-guardian
|
||||
|
||||
go 1.23
|
||||
65
internal/actions/actions.go
Normal file
65
internal/actions/actions.go
Normal file
@ -0,0 +1,65 @@
|
||||
package actions
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
|
||||
"gitea.wayfinderak.com/wayfinderak/log-guardian/internal/store"
|
||||
)
|
||||
|
||||
type Result struct {
|
||||
Action string `json:"action"`
|
||||
DryRun bool `json:"dry_run"`
|
||||
Detail string `json:"detail"`
|
||||
}
|
||||
|
||||
type Runner struct {
|
||||
dryRun bool
|
||||
}
|
||||
|
||||
func NewRunner(dryRun bool) *Runner { return &Runner{dryRun: dryRun} }
|
||||
|
||||
func (r *Runner) Run(ctx context.Context, rule store.Rule, action store.Action) (Result, error) {
|
||||
_ = ctx
|
||||
if !action.Enabled {
|
||||
return Result{Action: action.Type, DryRun: r.effectiveDryRun(action), Detail: "disabled"}, nil
|
||||
}
|
||||
switch action.Type {
|
||||
case "generic_webhook", "webhook":
|
||||
url := action.Params["url"]
|
||||
if url == "" {
|
||||
return Result{}, fmt.Errorf("webhook action requires url")
|
||||
}
|
||||
return Result{Action: action.Type, DryRun: true, Detail: "would POST webhook for rule " + rule.Name}, nil
|
||||
case "record_recommendation":
|
||||
return Result{Action: action.Type, DryRun: false, Detail: "recommendation recorded for operator review"}, nil
|
||||
case "allowlisted_command", "command":
|
||||
id := action.Params["command_id"]
|
||||
if id == "" {
|
||||
id = action.Params["id"]
|
||||
}
|
||||
if id == "" {
|
||||
return Result{}, fmt.Errorf("allowlisted command requires command_id")
|
||||
}
|
||||
return Result{Action: action.Type, DryRun: true, Detail: "would run allowlisted command " + id}, nil
|
||||
case "portainer_restart":
|
||||
service := action.Params["service"]
|
||||
if service == "" {
|
||||
return Result{}, fmt.Errorf("portainer_restart requires service")
|
||||
}
|
||||
return Result{Action: action.Type, DryRun: true, Detail: "would restart Portainer service " + service}, nil
|
||||
default:
|
||||
return Result{}, fmt.Errorf("unsupported action type %q", action.Type)
|
||||
}
|
||||
return Result{}, fmt.Errorf("unsupported action type %q", action.Type)
|
||||
}
|
||||
|
||||
func (r *Runner) effectiveDryRun(action store.Action) bool {
|
||||
if r.dryRun {
|
||||
return true
|
||||
}
|
||||
if action.DryRun == nil {
|
||||
return true
|
||||
}
|
||||
return *action.DryRun
|
||||
}
|
||||
108
internal/alerts/alerts.go
Normal file
108
internal/alerts/alerts.go
Normal file
@ -0,0 +1,108 @@
|
||||
package alerts
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"fmt"
|
||||
"log"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.wayfinderak.com/wayfinderak/log-guardian/internal/analysis"
|
||||
"gitea.wayfinderak.com/wayfinderak/log-guardian/internal/store"
|
||||
)
|
||||
|
||||
type Result struct {
|
||||
ChannelID string
|
||||
Detail string
|
||||
}
|
||||
|
||||
type Dispatcher struct {
|
||||
http *http.Client
|
||||
}
|
||||
|
||||
func NewDispatcher() *Dispatcher { return &Dispatcher{http: &http.Client{Timeout: 15 * time.Second}} }
|
||||
|
||||
func (d *Dispatcher) Send(ctx context.Context, rule store.Rule, finding analysis.Finding, channels []store.AlertChannel) ([]Result, error) {
|
||||
if len(channels) == 0 {
|
||||
log.Printf("alert for rule %q: %s", rule.Name, finding.Summary)
|
||||
return []Result{{Detail: "no channels configured; logged alert intent"}}, nil
|
||||
}
|
||||
var results []Result
|
||||
var firstErr error
|
||||
for _, channel := range channels {
|
||||
if !channel.Enabled {
|
||||
results = append(results, Result{ChannelID: channel.ID, Detail: "disabled"})
|
||||
continue
|
||||
}
|
||||
var err error
|
||||
switch channel.Type {
|
||||
case "ntfy":
|
||||
err = d.sendNtfy(ctx, rule, finding, channel)
|
||||
case "gotify", "generic_webhook", "smtp":
|
||||
err = fmt.Errorf("%s delivery is scaffolded but not implemented in phase 1", channel.Type)
|
||||
default:
|
||||
err = fmt.Errorf("unsupported channel type %q", channel.Type)
|
||||
}
|
||||
if err != nil {
|
||||
if firstErr == nil {
|
||||
firstErr = err
|
||||
}
|
||||
results = append(results, Result{ChannelID: channel.ID, Detail: err.Error()})
|
||||
continue
|
||||
}
|
||||
results = append(results, Result{ChannelID: channel.ID, Detail: "sent"})
|
||||
}
|
||||
return results, firstErr
|
||||
}
|
||||
|
||||
func (d *Dispatcher) Test(ctx context.Context, channel store.AlertChannel) error {
|
||||
finding := analysis.Finding{Summary: "Log Guardian test notification", Confidence: "test"}
|
||||
rule := store.Rule{Name: "Test alert", Severity: "info"}
|
||||
switch channel.Type {
|
||||
case "ntfy":
|
||||
return d.sendNtfy(ctx, rule, finding, channel)
|
||||
case "gotify", "generic_webhook", "smtp":
|
||||
return fmt.Errorf("%s delivery is scaffolded but not implemented in phase 1", channel.Type)
|
||||
default:
|
||||
return fmt.Errorf("unsupported channel type %q", channel.Type)
|
||||
}
|
||||
}
|
||||
|
||||
func (d *Dispatcher) sendNtfy(ctx context.Context, rule store.Rule, finding analysis.Finding, channel store.AlertChannel) error {
|
||||
server := strings.TrimRight(channel.Params["server_url"], "/")
|
||||
topic := strings.Trim(channel.Params["topic"], "/")
|
||||
if server == "" || topic == "" {
|
||||
return fmt.Errorf("ntfy requires server_url and topic")
|
||||
}
|
||||
target, err := url.JoinPath(server, topic)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
body := rule.Name + ": " + finding.Summary
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodPost, target, bytes.NewBufferString(body))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
req.Header.Set("Title", "Log Guardian: "+rule.Severity)
|
||||
if priority := channel.Params["priority"]; priority != "" {
|
||||
req.Header.Set("Priority", priority)
|
||||
}
|
||||
if tags := channel.Params["tags"]; tags != "" {
|
||||
req.Header.Set("Tags", tags)
|
||||
}
|
||||
if token := channel.Params["token"]; token != "" {
|
||||
req.Header.Set("Authorization", "Bearer "+token)
|
||||
}
|
||||
resp, err := d.http.Do(req)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode < 200 || resp.StatusCode > 299 {
|
||||
return fmt.Errorf("ntfy delivery failed: status %d", resp.StatusCode)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
31
internal/alerts/alerts_test.go
Normal file
31
internal/alerts/alerts_test.go
Normal file
@ -0,0 +1,31 @@
|
||||
package alerts
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
|
||||
"gitea.wayfinderak.com/wayfinderak/log-guardian/internal/analysis"
|
||||
"gitea.wayfinderak.com/wayfinderak/log-guardian/internal/store"
|
||||
)
|
||||
|
||||
func TestNtfyDelivery(t *testing.T) {
|
||||
var gotAuth string
|
||||
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path != "/alerts" {
|
||||
t.Fatalf("unexpected path %s", r.URL.Path)
|
||||
}
|
||||
gotAuth = r.Header.Get("Authorization")
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}))
|
||||
defer ts.Close()
|
||||
|
||||
d := NewDispatcher()
|
||||
_, err := d.Send(t.Context(), store.Rule{Name: "API", Severity: "warning"}, analysis.Finding{Summary: "failed"}, []store.AlertChannel{{ID: "c1", Name: "ntfy", Type: "ntfy", Enabled: true, Params: map[string]string{"server_url": ts.URL, "topic": "alerts", "token": "secret"}}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if gotAuth != "Bearer secret" {
|
||||
t.Fatalf("missing auth header: %q", gotAuth)
|
||||
}
|
||||
}
|
||||
160
internal/analysis/analysis.go
Normal file
160
internal/analysis/analysis.go
Normal file
@ -0,0 +1,160 @@
|
||||
package analysis
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
"unicode"
|
||||
|
||||
"gitea.wayfinderak.com/wayfinderak/log-guardian/internal/loki"
|
||||
"gitea.wayfinderak.com/wayfinderak/log-guardian/internal/store"
|
||||
)
|
||||
|
||||
type Finding struct {
|
||||
Summary string `json:"summary"`
|
||||
RecommendedActions []string `json:"recommended_actions,omitempty"`
|
||||
Confidence string `json:"confidence"`
|
||||
}
|
||||
|
||||
type Analyzer interface {
|
||||
Analyze(ctx context.Context, rule store.Rule, matches []loki.Match) (Finding, error)
|
||||
}
|
||||
|
||||
type NoopAnalyzer struct{}
|
||||
|
||||
func (NoopAnalyzer) Analyze(ctx context.Context, rule store.Rule, matches []loki.Match) (Finding, error) {
|
||||
_ = ctx
|
||||
return Finding{Summary: summarize(rule, matches), Confidence: "rule-only"}, nil
|
||||
}
|
||||
|
||||
type OllamaAnalyzer struct {
|
||||
URL string
|
||||
Model string
|
||||
http *http.Client
|
||||
}
|
||||
|
||||
func NewOllamaAnalyzer(url, model string) *OllamaAnalyzer {
|
||||
return &OllamaAnalyzer{URL: strings.TrimRight(url, "/"), Model: model, http: &http.Client{Timeout: 60 * time.Second}}
|
||||
}
|
||||
|
||||
func (a *OllamaAnalyzer) Analyze(ctx context.Context, rule store.Rule, matches []loki.Match) (Finding, error) {
|
||||
if a.URL == "" || a.Model == "" {
|
||||
return NoopAnalyzer{}.Analyze(ctx, rule, matches)
|
||||
}
|
||||
prompt := buildPrompt(rule, matches)
|
||||
body, _ := json.Marshal(map[string]any{"model": a.Model, "prompt": prompt, "stream": false})
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodPost, a.URL+"/api/generate", bytes.NewReader(body))
|
||||
if err != nil {
|
||||
return Finding{}, err
|
||||
}
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
resp, err := a.http.Do(req)
|
||||
if err != nil {
|
||||
return Finding{}, err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode < 200 || resp.StatusCode > 299 {
|
||||
return Finding{}, fmt.Errorf("ollama analysis failed: status %d", resp.StatusCode)
|
||||
}
|
||||
var raw struct {
|
||||
Response string `json:"response"`
|
||||
}
|
||||
if err := json.NewDecoder(resp.Body).Decode(&raw); err != nil {
|
||||
return Finding{}, err
|
||||
}
|
||||
summary := strings.TrimSpace(raw.Response)
|
||||
if summary == "" {
|
||||
return NoopAnalyzer{}.Analyze(ctx, rule, matches)
|
||||
}
|
||||
if len(summary) > 500 {
|
||||
summary = summary[:500] + "..."
|
||||
}
|
||||
return Finding{Summary: summary, Confidence: "ollama-local"}, nil
|
||||
}
|
||||
|
||||
func buildPrompt(rule store.Rule, matches []loki.Match) string {
|
||||
var b strings.Builder
|
||||
b.WriteString("You are analyzing private application logs locally. Summarize likely cause and operator next steps in 4 concise sentences or fewer. Do not suggest destructive actions.\n")
|
||||
b.WriteString("Rule: ")
|
||||
b.WriteString(rule.Name)
|
||||
b.WriteString("\nSeverity: ")
|
||||
b.WriteString(rule.Severity)
|
||||
b.WriteString("\nSanitized samples:\n")
|
||||
limit := len(matches)
|
||||
if limit > 5 {
|
||||
limit = 5
|
||||
}
|
||||
for i := 0; i < limit; i++ {
|
||||
line := sanitizeLine(matches[i].Line)
|
||||
line = redactSecrets(line)
|
||||
if len(line) > 300 {
|
||||
line = line[:300] + "..."
|
||||
}
|
||||
b.WriteString("- ")
|
||||
b.WriteString(line)
|
||||
b.WriteByte('\n')
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
func summarize(rule store.Rule, matches []loki.Match) string {
|
||||
if len(matches) == 0 {
|
||||
return "No matching log entries."
|
||||
}
|
||||
line := redactSecrets(sanitizeLine(matches[0].Line))
|
||||
if len(line) > 180 {
|
||||
line = line[:180] + "..."
|
||||
}
|
||||
return rule.Name + " matched " + plural(len(matches), "log entry", "log entries") + "; first match: " + line
|
||||
}
|
||||
|
||||
func sanitizeLine(line string) string {
|
||||
line = strings.ReplaceAll(line, "\n", " ")
|
||||
line = strings.ReplaceAll(line, "\r", " ")
|
||||
return strings.TrimSpace(line)
|
||||
}
|
||||
|
||||
func redactSecrets(line string) string {
|
||||
fields := strings.Fields(line)
|
||||
for i, field := range fields {
|
||||
lower := strings.ToLower(field)
|
||||
if strings.Contains(lower, "token=") || strings.Contains(lower, "password=") || strings.Contains(lower, "secret=") || strings.Contains(lower, "apikey=") || strings.Contains(lower, "api_key=") || strings.HasPrefix(lower, "bearer") {
|
||||
fields[i] = redactField(field)
|
||||
}
|
||||
}
|
||||
return strings.Join(fields, " ")
|
||||
}
|
||||
|
||||
func redactField(field string) string {
|
||||
idx := strings.IndexAny(field, "=:")
|
||||
if idx < 0 {
|
||||
return "[REDACTED]"
|
||||
}
|
||||
prefix := strings.TrimRightFunc(field[:idx], func(r rune) bool { return unicode.IsSpace(r) })
|
||||
return prefix + "=[REDACTED]"
|
||||
}
|
||||
|
||||
func plural(count int, singular, plural string) string {
|
||||
if count == 1 {
|
||||
return "1 " + singular
|
||||
}
|
||||
return strconvItoa(count) + " " + plural
|
||||
}
|
||||
|
||||
func strconvItoa(i int) string {
|
||||
if i == 0 {
|
||||
return "0"
|
||||
}
|
||||
var b [20]byte
|
||||
bp := len(b)
|
||||
for i > 0 {
|
||||
bp--
|
||||
b[bp] = byte('0' + i%10)
|
||||
i /= 10
|
||||
}
|
||||
return string(b[bp:])
|
||||
}
|
||||
10
internal/analysis/analysis_test.go
Normal file
10
internal/analysis/analysis_test.go
Normal file
@ -0,0 +1,10 @@
|
||||
package analysis
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestSanitizeLine(t *testing.T) {
|
||||
got := sanitizeLine(" error\nwith\rspaces ")
|
||||
if got != "error with spaces" {
|
||||
t.Fatalf("got %q", got)
|
||||
}
|
||||
}
|
||||
83
internal/config/config.go
Normal file
83
internal/config/config.go
Normal file
@ -0,0 +1,83 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"os"
|
||||
"strconv"
|
||||
"time"
|
||||
)
|
||||
|
||||
type Config struct {
|
||||
Addr string
|
||||
DataPath string
|
||||
PollInterval time.Duration
|
||||
LokiURL string
|
||||
LokiTenantID string
|
||||
LokiUsername string
|
||||
LokiPassword string
|
||||
DryRunActions bool
|
||||
AnalysisProvider string
|
||||
AnalysisEndpoint string
|
||||
AnalysisToken string
|
||||
OllamaURL string
|
||||
OllamaModel string
|
||||
AuthUsername string
|
||||
AuthPassword string
|
||||
}
|
||||
|
||||
func Load() Config {
|
||||
return Config{
|
||||
Addr: env("LOG_GUARDIAN_ADDR", ":8080"),
|
||||
DataPath: env("LOG_GUARDIAN_DATA_PATH", "/data/config.json"),
|
||||
PollInterval: durationEnv("LOG_GUARDIAN_POLL_INTERVAL", 1*time.Minute),
|
||||
LokiURL: trimRightSlash(env("LOKI_URL", "http://loki:3100")),
|
||||
LokiTenantID: env("LOKI_TENANT_ID", ""),
|
||||
LokiUsername: env("LOKI_USERNAME", ""),
|
||||
LokiPassword: env("LOKI_PASSWORD", ""),
|
||||
DryRunActions: boolEnv("LOG_GUARDIAN_DRY_RUN_ACTIONS", true),
|
||||
AnalysisProvider: env("LOG_GUARDIAN_ANALYSIS_PROVIDER", "none"),
|
||||
AnalysisEndpoint: env("LOG_GUARDIAN_ANALYSIS_ENDPOINT", ""),
|
||||
AnalysisToken: env("LOG_GUARDIAN_ANALYSIS_TOKEN", ""),
|
||||
OllamaURL: trimRightSlash(env("LOG_GUARDIAN_OLLAMA_URL", "http://ollama:11434")),
|
||||
OllamaModel: env("LOG_GUARDIAN_OLLAMA_MODEL", "llama3.1"),
|
||||
AuthUsername: env("LOG_GUARDIAN_AUTH_USERNAME", ""),
|
||||
AuthPassword: env("LOG_GUARDIAN_AUTH_PASSWORD", ""),
|
||||
}
|
||||
}
|
||||
|
||||
func env(key, fallback string) string {
|
||||
if value := os.Getenv(key); value != "" {
|
||||
return value
|
||||
}
|
||||
return fallback
|
||||
}
|
||||
|
||||
func boolEnv(key string, fallback bool) bool {
|
||||
value := os.Getenv(key)
|
||||
if value == "" {
|
||||
return fallback
|
||||
}
|
||||
parsed, err := strconv.ParseBool(value)
|
||||
if err != nil {
|
||||
return fallback
|
||||
}
|
||||
return parsed
|
||||
}
|
||||
|
||||
func durationEnv(key string, fallback time.Duration) time.Duration {
|
||||
value := os.Getenv(key)
|
||||
if value == "" {
|
||||
return fallback
|
||||
}
|
||||
parsed, err := time.ParseDuration(value)
|
||||
if err != nil {
|
||||
return fallback
|
||||
}
|
||||
return parsed
|
||||
}
|
||||
|
||||
func trimRightSlash(value string) string {
|
||||
for len(value) > 1 && value[len(value)-1] == '/' {
|
||||
value = value[:len(value)-1]
|
||||
}
|
||||
return value
|
||||
}
|
||||
148
internal/loki/client.go
Normal file
148
internal/loki/client.go
Normal file
@ -0,0 +1,148 @@
|
||||
package loki
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
type Client struct {
|
||||
baseURL string
|
||||
tenantID string
|
||||
username string
|
||||
password string
|
||||
http *http.Client
|
||||
}
|
||||
|
||||
func New(baseURL, tenantID, username, password string) *Client {
|
||||
return &Client{baseURL: strings.TrimRight(baseURL, "/"), tenantID: tenantID, username: username, password: password, http: &http.Client{Timeout: 20 * time.Second}}
|
||||
}
|
||||
|
||||
type Match struct {
|
||||
Timestamp time.Time `json:"timestamp"`
|
||||
Line string `json:"line"`
|
||||
}
|
||||
|
||||
type QueryResult struct {
|
||||
Count int `json:"count"`
|
||||
Matches []Match `json:"matches"`
|
||||
}
|
||||
|
||||
func (c *Client) Ready(ctx context.Context) error {
|
||||
req, err := c.request(ctx, http.MethodGet, "/ready", nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
resp, err := c.http.Do(req)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode < 200 || resp.StatusCode > 299 {
|
||||
return fmt.Errorf("loki ready failed: status %d", resp.StatusCode)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (c *Client) Labels(ctx context.Context) ([]string, error) {
|
||||
var raw labelsResponse
|
||||
if err := c.getJSON(ctx, "/loki/api/v1/labels", nil, &raw); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return raw.Data, nil
|
||||
}
|
||||
|
||||
func (c *Client) LabelValues(ctx context.Context, name string) ([]string, error) {
|
||||
if name == "" || strings.Contains(name, "/") {
|
||||
return nil, fmt.Errorf("invalid label name")
|
||||
}
|
||||
var raw labelsResponse
|
||||
if err := c.getJSON(ctx, "/loki/api/v1/label/"+url.PathEscape(name)+"/values", nil, &raw); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return raw.Data, nil
|
||||
}
|
||||
|
||||
func (c *Client) QueryRange(ctx context.Context, logql string, since time.Duration, limit int) (QueryResult, error) {
|
||||
if limit <= 0 {
|
||||
limit = 100
|
||||
}
|
||||
values := url.Values{}
|
||||
values.Set("query", logql)
|
||||
values.Set("start", strconv.FormatInt(time.Now().Add(-since).UnixNano(), 10))
|
||||
values.Set("end", strconv.FormatInt(time.Now().UnixNano(), 10))
|
||||
values.Set("limit", strconv.Itoa(limit))
|
||||
var raw response
|
||||
if err := c.getJSON(ctx, "/loki/api/v1/query_range", values, &raw); err != nil {
|
||||
return QueryResult{}, err
|
||||
}
|
||||
var result QueryResult
|
||||
for _, stream := range raw.Data.Result {
|
||||
for _, pair := range stream.Values {
|
||||
if len(pair) != 2 {
|
||||
continue
|
||||
}
|
||||
ns, _ := strconv.ParseInt(pair[0], 10, 64)
|
||||
result.Matches = append(result.Matches, Match{Timestamp: time.Unix(0, ns), Line: pair[1]})
|
||||
}
|
||||
}
|
||||
result.Count = len(result.Matches)
|
||||
return result, nil
|
||||
}
|
||||
|
||||
func (c *Client) getJSON(ctx context.Context, path string, values url.Values, out any) error {
|
||||
req, err := c.request(ctx, http.MethodGet, path, values)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
resp, err := c.http.Do(req)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode < 200 || resp.StatusCode > 299 {
|
||||
return fmt.Errorf("loki request failed: status %d", resp.StatusCode)
|
||||
}
|
||||
return json.NewDecoder(resp.Body).Decode(out)
|
||||
}
|
||||
|
||||
func (c *Client) request(ctx context.Context, method, path string, values url.Values) (*http.Request, error) {
|
||||
if c.baseURL == "" {
|
||||
return nil, fmt.Errorf("LOKI_URL is not configured")
|
||||
}
|
||||
target := c.baseURL + path
|
||||
if len(values) > 0 {
|
||||
target += "?" + values.Encode()
|
||||
}
|
||||
req, err := http.NewRequestWithContext(ctx, method, target, nil)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if c.tenantID != "" {
|
||||
req.Header.Set("X-Scope-OrgID", c.tenantID)
|
||||
}
|
||||
if c.username != "" || c.password != "" {
|
||||
req.SetBasicAuth(c.username, c.password)
|
||||
}
|
||||
return req, nil
|
||||
}
|
||||
|
||||
type labelsResponse struct {
|
||||
Status string `json:"status"`
|
||||
Data []string `json:"data"`
|
||||
}
|
||||
|
||||
type response struct {
|
||||
Status string `json:"status"`
|
||||
Data struct {
|
||||
Result []struct {
|
||||
Stream map[string]string `json:"stream"`
|
||||
Values [][]string `json:"values"`
|
||||
} `json:"result"`
|
||||
} `json:"data"`
|
||||
}
|
||||
42
internal/loki/client_test.go
Normal file
42
internal/loki/client_test.go
Normal file
@ -0,0 +1,42 @@
|
||||
package loki
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestLabelsValuesAndQueryRange(t *testing.T) {
|
||||
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
switch r.URL.Path {
|
||||
case "/loki/api/v1/labels":
|
||||
_, _ = w.Write([]byte(`{"status":"success","data":["service","container"]}`))
|
||||
case "/loki/api/v1/label/service/values":
|
||||
_, _ = w.Write([]byte(`{"status":"success","data":["api"]}`))
|
||||
case "/loki/api/v1/query_range":
|
||||
if !strings.Contains(r.URL.Query().Get("query"), "api") {
|
||||
t.Fatalf("query not passed through: %s", r.URL.RawQuery)
|
||||
}
|
||||
_, _ = w.Write([]byte(`{"status":"success","data":{"result":[{"stream":{"service":"api"},"values":[["1700000000000000000","error token=secret"]]}]}}`))
|
||||
default:
|
||||
http.NotFound(w, r)
|
||||
}
|
||||
}))
|
||||
defer ts.Close()
|
||||
|
||||
client := New(ts.URL, "", "", "")
|
||||
labels, err := client.Labels(t.Context())
|
||||
if err != nil || len(labels) != 2 {
|
||||
t.Fatalf("labels=%v err=%v", labels, err)
|
||||
}
|
||||
values, err := client.LabelValues(t.Context(), "service")
|
||||
if err != nil || values[0] != "api" {
|
||||
t.Fatalf("values=%v err=%v", values, err)
|
||||
}
|
||||
result, err := client.QueryRange(t.Context(), `{service="api"}`, time.Minute, 10)
|
||||
if err != nil || result.Count != 1 {
|
||||
t.Fatalf("result=%#v err=%v", result, err)
|
||||
}
|
||||
}
|
||||
152
internal/rules/engine.go
Normal file
152
internal/rules/engine.go
Normal file
@ -0,0 +1,152 @@
|
||||
package rules
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log"
|
||||
"time"
|
||||
|
||||
"gitea.wayfinderak.com/wayfinderak/log-guardian/internal/actions"
|
||||
"gitea.wayfinderak.com/wayfinderak/log-guardian/internal/alerts"
|
||||
"gitea.wayfinderak.com/wayfinderak/log-guardian/internal/analysis"
|
||||
"gitea.wayfinderak.com/wayfinderak/log-guardian/internal/loki"
|
||||
"gitea.wayfinderak.com/wayfinderak/log-guardian/internal/store"
|
||||
)
|
||||
|
||||
type Store interface {
|
||||
Load() (store.Data, error)
|
||||
UpdateRule(id string, fn func(*store.Rule)) error
|
||||
AddIncident(incident store.Incident) error
|
||||
}
|
||||
|
||||
type Engine struct {
|
||||
store Store
|
||||
loki *loki.Client
|
||||
analyzer analysis.Analyzer
|
||||
actions *actions.Runner
|
||||
dispatcher *alerts.Dispatcher
|
||||
}
|
||||
|
||||
func NewEngine(s Store, lokiClient *loki.Client, analyzer analysis.Analyzer, actionRunner *actions.Runner, dispatcher *alerts.Dispatcher) *Engine {
|
||||
return &Engine{store: s, loki: lokiClient, analyzer: analyzer, actions: actionRunner, dispatcher: dispatcher}
|
||||
}
|
||||
|
||||
func (e *Engine) Start(ctx context.Context, interval time.Duration) {
|
||||
if interval <= 0 {
|
||||
interval = time.Minute
|
||||
}
|
||||
ticker := time.NewTicker(interval)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
e.CheckAll(ctx)
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (e *Engine) CheckAll(ctx context.Context) {
|
||||
data, err := e.store.Load()
|
||||
if err != nil {
|
||||
log.Printf("load rules failed: %v", err)
|
||||
return
|
||||
}
|
||||
channels := map[string]store.AlertChannel{}
|
||||
for _, channel := range data.AlertChannels {
|
||||
channels[channel.ID] = channel
|
||||
}
|
||||
for _, rule := range data.Rules {
|
||||
if !rule.Enabled {
|
||||
continue
|
||||
}
|
||||
if err := e.checkRule(ctx, rule, channels); err != nil {
|
||||
log.Printf("rule %q check failed: %v", rule.Name, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (e *Engine) checkRule(ctx context.Context, rule store.Rule, channels map[string]store.AlertChannel) error {
|
||||
window, err := time.ParseDuration(rule.Window)
|
||||
if err != nil {
|
||||
window = 5 * time.Minute
|
||||
}
|
||||
result, err := e.loki.QueryRange(ctx, rule.LogQL, window, 200)
|
||||
if err != nil {
|
||||
e.record(rule.ID, 0, err)
|
||||
return err
|
||||
}
|
||||
if result.Count < rule.Threshold {
|
||||
return e.record(rule.ID, result.Count, nil)
|
||||
}
|
||||
now := time.Now().UTC()
|
||||
cooldown, err := time.ParseDuration(rule.Cooldown)
|
||||
if err != nil || cooldown <= 0 {
|
||||
cooldown = time.Hour
|
||||
}
|
||||
if !rule.LastAlertedAt.IsZero() && now.Sub(rule.LastAlertedAt) < cooldown {
|
||||
return e.store.UpdateRule(rule.ID, func(r *store.Rule) {
|
||||
r.LastCheckedAt = now
|
||||
r.LastMatchedAt = now
|
||||
r.LastMatchCount = result.Count
|
||||
r.SuppressedCount++
|
||||
r.LastError = ""
|
||||
})
|
||||
}
|
||||
finding, err := e.analyzer.Analyze(ctx, rule, result.Matches)
|
||||
if err != nil {
|
||||
e.record(rule.ID, result.Count, err)
|
||||
return err
|
||||
}
|
||||
selected := make([]store.AlertChannel, 0, len(rule.AlertChannels))
|
||||
for _, id := range rule.AlertChannels {
|
||||
if channel, ok := channels[id]; ok {
|
||||
selected = append(selected, channel)
|
||||
}
|
||||
}
|
||||
alertResults, alertErr := e.dispatcher.Send(ctx, rule, finding, selected)
|
||||
var alertEvidence []string
|
||||
for _, result := range alertResults {
|
||||
if result.ChannelID == "" {
|
||||
alertEvidence = append(alertEvidence, result.Detail)
|
||||
} else {
|
||||
alertEvidence = append(alertEvidence, result.ChannelID+": "+result.Detail)
|
||||
}
|
||||
}
|
||||
var actionEvidence []string
|
||||
for _, action := range rule.Actions {
|
||||
result, err := e.actions.Run(ctx, rule, action)
|
||||
if err != nil {
|
||||
log.Printf("action %q for rule %q failed: %v", action.Type, rule.Name, err)
|
||||
actionEvidence = append(actionEvidence, action.Type+": "+err.Error())
|
||||
continue
|
||||
}
|
||||
actionEvidence = append(actionEvidence, result.Action+": "+result.Detail)
|
||||
}
|
||||
incident := store.Incident{RuleID: rule.ID, RuleName: rule.Name, Severity: rule.Severity, Count: result.Count, Summary: finding.Summary, AlertResults: alertEvidence, RemediationEvidence: actionEvidence, CreatedAt: now}
|
||||
if err := e.store.AddIncident(incident); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := e.store.UpdateRule(rule.ID, func(r *store.Rule) {
|
||||
r.LastCheckedAt = now
|
||||
r.LastMatchedAt = now
|
||||
r.LastAlertedAt = now
|
||||
r.LastMatchCount = result.Count
|
||||
r.LastError = ""
|
||||
}); err != nil {
|
||||
return err
|
||||
}
|
||||
return alertErr
|
||||
}
|
||||
|
||||
func (e *Engine) record(id string, count int, err error) error {
|
||||
return e.store.UpdateRule(id, func(r *store.Rule) {
|
||||
r.LastCheckedAt = time.Now().UTC()
|
||||
r.LastMatchCount = count
|
||||
if err != nil {
|
||||
r.LastError = err.Error()
|
||||
} else {
|
||||
r.LastError = ""
|
||||
}
|
||||
})
|
||||
}
|
||||
42
internal/rules/engine_test.go
Normal file
42
internal/rules/engine_test.go
Normal file
@ -0,0 +1,42 @@
|
||||
package rules
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.wayfinderak.com/wayfinderak/log-guardian/internal/actions"
|
||||
"gitea.wayfinderak.com/wayfinderak/log-guardian/internal/alerts"
|
||||
"gitea.wayfinderak.com/wayfinderak/log-guardian/internal/analysis"
|
||||
"gitea.wayfinderak.com/wayfinderak/log-guardian/internal/loki"
|
||||
"gitea.wayfinderak.com/wayfinderak/log-guardian/internal/store"
|
||||
)
|
||||
|
||||
func TestCooldownSuppressesDuplicateIncident(t *testing.T) {
|
||||
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
_, _ = w.Write([]byte(`{"status":"success","data":{"result":[{"stream":{},"values":[["1700000000000000000","error"]]}]}}`))
|
||||
}))
|
||||
defer ts.Close()
|
||||
|
||||
s := store.NewFileStore(t.TempDir() + "/config.json")
|
||||
if err := s.UpsertRule(store.Rule{Name: "Errors", Enabled: true, LogQL: `{service="api"}`, Threshold: 1, Window: "5m", Cooldown: "1h", Actions: []store.Action{{Type: "record_recommendation", Enabled: true}}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
engine := NewEngine(s, loki.New(ts.URL, "", "", ""), analysis.NoopAnalyzer{}, actions.NewRunner(true), alerts.NewDispatcher())
|
||||
engine.CheckAll(t.Context())
|
||||
engine.CheckAll(t.Context())
|
||||
data, err := s.Load()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(data.Incidents) != 1 {
|
||||
t.Fatalf("expected one incident, got %d", len(data.Incidents))
|
||||
}
|
||||
if data.Rules[0].SuppressedCount != 1 {
|
||||
t.Fatalf("expected suppressed count 1, got %d", data.Rules[0].SuppressedCount)
|
||||
}
|
||||
if time.Since(data.Rules[0].LastAlertedAt) > time.Minute {
|
||||
t.Fatalf("last alerted not set: %s", data.Rules[0].LastAlertedAt)
|
||||
}
|
||||
}
|
||||
51
internal/server/middleware.go
Normal file
51
internal/server/middleware.go
Normal file
@ -0,0 +1,51 @@
|
||||
package server
|
||||
|
||||
import (
|
||||
"crypto/subtle"
|
||||
"encoding/json"
|
||||
"log"
|
||||
"net/http"
|
||||
"time"
|
||||
)
|
||||
|
||||
func writeJSON(w http.ResponseWriter, v any) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_ = json.NewEncoder(w).Encode(v)
|
||||
}
|
||||
|
||||
func requestLog(next http.Handler) http.Handler {
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
start := time.Now()
|
||||
next.ServeHTTP(w, r)
|
||||
log.Printf("%s %s %s", r.Method, r.URL.Path, time.Since(start).Round(time.Millisecond))
|
||||
})
|
||||
}
|
||||
|
||||
func securityHeaders(next http.Handler) http.Handler {
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("X-Content-Type-Options", "nosniff")
|
||||
w.Header().Set("Referrer-Policy", "no-referrer")
|
||||
w.Header().Set("Content-Security-Policy", "default-src 'self'; style-src 'self' 'unsafe-inline'; form-action 'self'")
|
||||
next.ServeHTTP(w, r)
|
||||
})
|
||||
}
|
||||
|
||||
func basicAuth(next http.Handler, username, password string) http.Handler {
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path == "/healthz" {
|
||||
next.ServeHTTP(w, r)
|
||||
return
|
||||
}
|
||||
if username == "" || password == "" {
|
||||
http.Error(w, "basic auth is not configured", http.StatusServiceUnavailable)
|
||||
return
|
||||
}
|
||||
gotUser, gotPass, ok := r.BasicAuth()
|
||||
if !ok || subtle.ConstantTimeCompare([]byte(gotUser), []byte(username)) != 1 || subtle.ConstantTimeCompare([]byte(gotPass), []byte(password)) != 1 {
|
||||
w.Header().Set("WWW-Authenticate", `Basic realm="log-guardian"`)
|
||||
http.Error(w, "unauthorized", http.StatusUnauthorized)
|
||||
return
|
||||
}
|
||||
next.ServeHTTP(w, r)
|
||||
})
|
||||
}
|
||||
34
internal/server/middleware_test.go
Normal file
34
internal/server/middleware_test.go
Normal file
@ -0,0 +1,34 @@
|
||||
package server
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestBasicAuthAllowsHealthzWithoutCredentials(t *testing.T) {
|
||||
h := basicAuth(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(http.StatusNoContent) }), "admin", "secret")
|
||||
req := httptest.NewRequest(http.MethodGet, "/healthz", nil)
|
||||
rr := httptest.NewRecorder()
|
||||
h.ServeHTTP(rr, req)
|
||||
if rr.Code != http.StatusNoContent {
|
||||
t.Fatalf("got %d", rr.Code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBasicAuthProtectsUI(t *testing.T) {
|
||||
h := basicAuth(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(http.StatusNoContent) }), "admin", "secret")
|
||||
req := httptest.NewRequest(http.MethodGet, "/", nil)
|
||||
rr := httptest.NewRecorder()
|
||||
h.ServeHTTP(rr, req)
|
||||
if rr.Code != http.StatusUnauthorized {
|
||||
t.Fatalf("got %d", rr.Code)
|
||||
}
|
||||
req = httptest.NewRequest(http.MethodGet, "/", nil)
|
||||
req.SetBasicAuth("admin", "secret")
|
||||
rr = httptest.NewRecorder()
|
||||
h.ServeHTTP(rr, req)
|
||||
if rr.Code != http.StatusNoContent {
|
||||
t.Fatalf("got %d", rr.Code)
|
||||
}
|
||||
}
|
||||
202
internal/server/server.go
Normal file
202
internal/server/server.go
Normal file
@ -0,0 +1,202 @@
|
||||
package server
|
||||
|
||||
import (
|
||||
"context"
|
||||
"html/template"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.wayfinderak.com/wayfinderak/log-guardian/internal/alerts"
|
||||
"gitea.wayfinderak.com/wayfinderak/log-guardian/internal/loki"
|
||||
"gitea.wayfinderak.com/wayfinderak/log-guardian/internal/rules"
|
||||
"gitea.wayfinderak.com/wayfinderak/log-guardian/internal/store"
|
||||
)
|
||||
|
||||
type Server struct {
|
||||
store *store.FileStore
|
||||
engine *rules.Engine
|
||||
loki *loki.Client
|
||||
dispatcher *alerts.Dispatcher
|
||||
authUsername string
|
||||
authPassword string
|
||||
}
|
||||
|
||||
func New(s *store.FileStore, engine *rules.Engine, lokiClient *loki.Client, dispatcher *alerts.Dispatcher, authUsername, authPassword string) *Server {
|
||||
return &Server{store: s, engine: engine, loki: lokiClient, dispatcher: dispatcher, authUsername: authUsername, authPassword: authPassword}
|
||||
}
|
||||
|
||||
func (s *Server) Routes() http.Handler {
|
||||
mux := http.NewServeMux()
|
||||
mux.HandleFunc("GET /healthz", s.health)
|
||||
mux.HandleFunc("GET /", s.index)
|
||||
mux.HandleFunc("POST /rules", s.saveRule)
|
||||
mux.HandleFunc("POST /check", s.checkNow)
|
||||
mux.HandleFunc("POST /channels", s.saveChannel)
|
||||
mux.HandleFunc("POST /channels/delete", s.deleteChannel)
|
||||
mux.HandleFunc("POST /channels/test", s.testChannel)
|
||||
mux.HandleFunc("GET /loki/labels", s.lokiLabels)
|
||||
mux.HandleFunc("GET /loki/label-values", s.lokiLabelValues)
|
||||
mux.HandleFunc("POST /loki/query-test", s.lokiQueryTest)
|
||||
mux.Handle("GET /static/", http.StripPrefix("/static/", http.FileServer(http.Dir("web/static"))))
|
||||
return securityHeaders(requestLog(basicAuth(mux, s.authUsername, s.authPassword)))
|
||||
}
|
||||
|
||||
func (s *Server) health(w http.ResponseWriter, r *http.Request) {
|
||||
writeJSON(w, map[string]string{"status": "ok", "service": "log-guardian"})
|
||||
}
|
||||
|
||||
func (s *Server) index(w http.ResponseWriter, r *http.Request) {
|
||||
data, err := s.store.Load()
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
render(w, data)
|
||||
}
|
||||
|
||||
func (s *Server) saveRule(w http.ResponseWriter, r *http.Request) {
|
||||
if err := r.ParseForm(); err != nil {
|
||||
http.Error(w, err.Error(), http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
threshold, _ := strconv.Atoi(r.FormValue("threshold"))
|
||||
rule := store.Rule{ID: strings.TrimSpace(r.FormValue("id")), Name: strings.TrimSpace(r.FormValue("name")), Enabled: r.FormValue("enabled") == "on", LogQL: strings.TrimSpace(r.FormValue("logql")), Severity: strings.TrimSpace(r.FormValue("severity")), Threshold: threshold, Window: strings.TrimSpace(r.FormValue("window")), Cooldown: strings.TrimSpace(r.FormValue("cooldown")), AnalysisEnabled: r.FormValue("analysis_enabled") == "on", AlertChannels: r.Form["alert_channels"]}
|
||||
if rule.Name == "" || rule.LogQL == "" {
|
||||
http.Error(w, "name and LogQL are required", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
if rule.Severity == "" {
|
||||
rule.Severity = "warning"
|
||||
}
|
||||
if err := s.store.UpsertRule(rule); err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
http.Redirect(w, r, "/", http.StatusSeeOther)
|
||||
}
|
||||
|
||||
func (s *Server) saveChannel(w http.ResponseWriter, r *http.Request) {
|
||||
if err := r.ParseForm(); err != nil {
|
||||
http.Error(w, err.Error(), http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
channel := store.AlertChannel{ID: strings.TrimSpace(r.FormValue("id")), Name: strings.TrimSpace(r.FormValue("name")), Type: strings.TrimSpace(r.FormValue("type")), Enabled: r.FormValue("enabled") == "on", Params: map[string]string{}}
|
||||
for _, key := range []string{"server_url", "topic", "token", "priority", "tags", "url", "api_url", "email"} {
|
||||
if value := strings.TrimSpace(r.FormValue(key)); value != "" {
|
||||
channel.Params[key] = value
|
||||
}
|
||||
}
|
||||
if channel.Name == "" || channel.Type == "" {
|
||||
http.Error(w, "name and type are required", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
if err := s.store.UpsertAlertChannel(channel); err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
http.Redirect(w, r, "/", http.StatusSeeOther)
|
||||
}
|
||||
|
||||
func (s *Server) deleteChannel(w http.ResponseWriter, r *http.Request) {
|
||||
if err := r.ParseForm(); err != nil {
|
||||
http.Error(w, err.Error(), http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
if err := s.store.DeleteAlertChannel(r.FormValue("id")); err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
http.Redirect(w, r, "/", http.StatusSeeOther)
|
||||
}
|
||||
|
||||
func (s *Server) testChannel(w http.ResponseWriter, r *http.Request) {
|
||||
if err := r.ParseForm(); err != nil {
|
||||
http.Error(w, err.Error(), http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
data, err := s.store.Load()
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
for _, channel := range data.AlertChannels {
|
||||
if channel.ID == r.FormValue("id") {
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second)
|
||||
defer cancel()
|
||||
if err := s.dispatcher.Test(ctx, channel); err != nil {
|
||||
http.Error(w, err.Error(), http.StatusBadGateway)
|
||||
return
|
||||
}
|
||||
writeJSON(w, map[string]string{"status": "sent"})
|
||||
return
|
||||
}
|
||||
}
|
||||
http.Error(w, "channel not found", http.StatusNotFound)
|
||||
}
|
||||
|
||||
func (s *Server) checkNow(w http.ResponseWriter, r *http.Request) {
|
||||
s.engine.CheckAll(context.Background())
|
||||
http.Redirect(w, r, "/", http.StatusSeeOther)
|
||||
}
|
||||
|
||||
func (s *Server) lokiLabels(w http.ResponseWriter, r *http.Request) {
|
||||
labels, err := s.loki.Labels(r.Context())
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusBadGateway)
|
||||
return
|
||||
}
|
||||
writeJSON(w, map[string]any{"labels": labels})
|
||||
}
|
||||
|
||||
func (s *Server) lokiLabelValues(w http.ResponseWriter, r *http.Request) {
|
||||
values, err := s.loki.LabelValues(r.Context(), r.URL.Query().Get("name"))
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusBadGateway)
|
||||
return
|
||||
}
|
||||
writeJSON(w, map[string]any{"values": values})
|
||||
}
|
||||
|
||||
func (s *Server) lokiQueryTest(w http.ResponseWriter, r *http.Request) {
|
||||
if err := r.ParseForm(); err != nil {
|
||||
http.Error(w, err.Error(), http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
window, err := time.ParseDuration(r.FormValue("window"))
|
||||
if err != nil {
|
||||
window = 5 * time.Minute
|
||||
}
|
||||
result, err := s.loki.QueryRange(r.Context(), strings.TrimSpace(r.FormValue("logql")), window, 20)
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusBadGateway)
|
||||
return
|
||||
}
|
||||
if len(result.Matches) > 5 {
|
||||
result.Matches = result.Matches[:5]
|
||||
}
|
||||
writeJSON(w, result)
|
||||
}
|
||||
|
||||
var tmpl = template.Must(template.New("index").Funcs(template.FuncMap{"secretState": secretState}).Parse(`<!doctype html>
|
||||
<html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><title>Log Guardian</title><link rel="stylesheet" href="/static/app.css"></head>
|
||||
<body><main class="shell"><header><h1>Log Guardian</h1><p>Loki-backed log monitoring, alerting, and guarded remediation.</p></header>
|
||||
<section class="card"><h2>Loki diagnostics</h2><p>Use authenticated endpoints: <code>/loki/labels</code>, <code>/loki/label-values?name=service</code>.</p><form method="post" action="/loki/query-test" class="grid"><label class="wide">Test LogQL <input name="logql" placeholder='{service="api"} |= "error"'></label><label>Window <input name="window" value="5m"></label><button type="submit">Return JSON sample</button></form></section>
|
||||
<section class="card"><h2>Add or replace rule</h2><form method="post" action="/rules" class="grid"><label>Name <input name="name" required placeholder="API errors"></label><label>ID <input name="id" placeholder="optional stable id"></label><label>Severity <input name="severity" value="warning"></label><label>Window <input name="window" value="5m"></label><label>Cooldown <input name="cooldown" value="1h"></label><label>Threshold <input name="threshold" type="number" min="1" value="1"></label><label class="wide">LogQL <input name="logql" required placeholder='{service="api"} |= "error"'></label><fieldset class="wide"><legend>Alert channels</legend>{{range .AlertChannels}}<label class="check"><input type="checkbox" name="alert_channels" value="{{.ID}}"> {{.Name}} ({{.Type}})</label>{{else}}<span class="muted">No channels configured.</span>{{end}}</fieldset><label class="check"><input type="checkbox" name="enabled" checked> Enabled</label><label class="check"><input type="checkbox" name="analysis_enabled"> AI analysis</label><button type="submit">Save rule</button></form></section>
|
||||
<section class="card"><h2>Alert channels</h2><form method="post" action="/channels" class="grid"><label>Name <input name="name" required></label><label>ID <input name="id" placeholder="optional stable id"></label><label>Type <select name="type"><option value="ntfy">ntfy</option><option value="gotify">gotify (scaffold)</option><option value="generic_webhook">generic webhook (scaffold)</option><option value="smtp">smtp (scaffold)</option></select></label><label class="check"><input type="checkbox" name="enabled" checked> Enabled</label><label>Server URL <input name="server_url" placeholder="https://ntfy.example.com"></label><label>Topic <input name="topic"></label><label>Token <input name="token" type="password" placeholder="leave blank to preserve"></label><label>Priority <input name="priority" placeholder="default, high, urgent"></label><label>Tags <input name="tags" placeholder="warning,rotating_light"></label><button type="submit">Save channel</button></form><table><thead><tr><th>Name</th><th>Type</th><th>Status</th><th>Secret</th><th>Actions</th></tr></thead><tbody>{{range .AlertChannels}}<tr><td>{{.Name}}</td><td>{{.Type}}</td><td>{{if .Enabled}}enabled{{else}}disabled{{end}}</td><td>{{secretState .Params}}</td><td><form method="post" action="/channels/test" class="inline"><input type="hidden" name="id" value="{{.ID}}"><button>Test</button></form><form method="post" action="/channels/delete" class="inline"><input type="hidden" name="id" value="{{.ID}}"><button>Delete</button></form></td></tr>{{else}}<tr><td colspan="5" class="muted">No alert channels configured.</td></tr>{{end}}</tbody></table></section>
|
||||
<section class="card"><div class="row"><h2>Rules</h2><form method="post" action="/check"><button type="submit">Check now</button></form></div><table><thead><tr><th>Name</th><th>LogQL</th><th>Cooldown</th><th>Last count</th><th>Suppressed</th><th>Status</th></tr></thead><tbody>{{range .Rules}}<tr><td>{{.Name}}{{if not .Enabled}} <span class="muted">disabled</span>{{end}}</td><td><code>{{.LogQL}}</code></td><td>{{.Cooldown}}</td><td>{{.LastMatchCount}}</td><td>{{.SuppressedCount}}</td><td>{{if .LastError}}<span class="error">{{.LastError}}</span>{{else}}checked {{.LastCheckedAt}}{{end}}</td></tr>{{else}}<tr><td colspan="6" class="muted">No rules configured yet.</td></tr>{{end}}</tbody></table></section>
|
||||
<section class="card"><h2>Recent incidents</h2><table><thead><tr><th>Time</th><th>Rule</th><th>Severity</th><th>Summary</th><th>Evidence</th></tr></thead><tbody>{{range .Incidents}}<tr><td>{{.CreatedAt}}</td><td>{{.RuleName}}</td><td>{{.Severity}}</td><td>{{.Summary}}</td><td>{{range .AlertResults}}<div>{{.}}</div>{{end}}{{range .RemediationEvidence}}<div>{{.}}</div>{{end}}</td></tr>{{else}}<tr><td colspan="5" class="muted">No incidents recorded.</td></tr>{{end}}</tbody></table></section>
|
||||
</main></body></html>`))
|
||||
|
||||
func secretState(params map[string]string) string {
|
||||
if params["token"] != "" || params["password"] != "" || params["secret"] != "" {
|
||||
return "configured"
|
||||
}
|
||||
return "not configured"
|
||||
}
|
||||
|
||||
func render(w http.ResponseWriter, data store.Data) {
|
||||
w.Header().Set("Content-Type", "text/html; charset=utf-8")
|
||||
_ = tmpl.Execute(w, data)
|
||||
}
|
||||
260
internal/store/store.go
Normal file
260
internal/store/store.go
Normal file
@ -0,0 +1,260 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
type Rule struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
Enabled bool `json:"enabled"`
|
||||
LogQL string `json:"logql"`
|
||||
Severity string `json:"severity"`
|
||||
Threshold int `json:"threshold"`
|
||||
Window string `json:"window"`
|
||||
Cooldown string `json:"cooldown,omitempty"`
|
||||
AlertChannels []string `json:"alert_channels,omitempty"`
|
||||
AnalysisEnabled bool `json:"analysis_enabled"`
|
||||
Actions []Action `json:"actions,omitempty"`
|
||||
LastCheckedAt time.Time `json:"last_checked_at,omitempty"`
|
||||
LastMatchedAt time.Time `json:"last_matched_at,omitempty"`
|
||||
LastAlertedAt time.Time `json:"last_alerted_at,omitempty"`
|
||||
LastMatchCount int `json:"last_match_count,omitempty"`
|
||||
SuppressedCount int `json:"suppressed_count,omitempty"`
|
||||
LastError string `json:"last_error,omitempty"`
|
||||
}
|
||||
|
||||
type Action struct {
|
||||
Type string `json:"type"`
|
||||
Enabled bool `json:"enabled"`
|
||||
DryRun *bool `json:"dry_run,omitempty"`
|
||||
Params map[string]string `json:"params,omitempty"`
|
||||
}
|
||||
|
||||
type AlertChannel struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
Type string `json:"type"`
|
||||
Enabled bool `json:"enabled"`
|
||||
Params map[string]string `json:"params,omitempty"`
|
||||
}
|
||||
|
||||
type Incident struct {
|
||||
ID string `json:"id"`
|
||||
RuleID string `json:"rule_id"`
|
||||
RuleName string `json:"rule_name"`
|
||||
Severity string `json:"severity"`
|
||||
Count int `json:"count"`
|
||||
Summary string `json:"summary"`
|
||||
AlertResults []string `json:"alert_results,omitempty"`
|
||||
RemediationEvidence []string `json:"remediation_evidence,omitempty"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
}
|
||||
|
||||
type Data struct {
|
||||
Rules []Rule `json:"rules"`
|
||||
AlertChannels []AlertChannel `json:"alert_channels"`
|
||||
Incidents []Incident `json:"incidents"`
|
||||
}
|
||||
|
||||
type FileStore struct {
|
||||
path string
|
||||
mu sync.Mutex
|
||||
}
|
||||
|
||||
func NewFileStore(path string) *FileStore { return &FileStore{path: path} }
|
||||
|
||||
func (s *FileStore) Load() (Data, error) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
return s.loadLocked()
|
||||
}
|
||||
|
||||
func (s *FileStore) Save(data Data) error {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
return s.saveLocked(data)
|
||||
}
|
||||
|
||||
func (s *FileStore) UpsertRule(rule Rule) error {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
data, err := s.loadLocked()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if rule.ID == "" {
|
||||
rule.ID = slug(rule.Name)
|
||||
}
|
||||
if rule.ID == "" {
|
||||
return errors.New("rule id or name is required")
|
||||
}
|
||||
if rule.Threshold <= 0 {
|
||||
rule.Threshold = 1
|
||||
}
|
||||
if rule.Window == "" {
|
||||
rule.Window = "5m"
|
||||
}
|
||||
if rule.Cooldown == "" {
|
||||
rule.Cooldown = "1h"
|
||||
}
|
||||
for i := range data.Rules {
|
||||
if data.Rules[i].ID == rule.ID {
|
||||
rule.LastCheckedAt = data.Rules[i].LastCheckedAt
|
||||
rule.LastMatchedAt = data.Rules[i].LastMatchedAt
|
||||
rule.LastAlertedAt = data.Rules[i].LastAlertedAt
|
||||
rule.LastMatchCount = data.Rules[i].LastMatchCount
|
||||
rule.SuppressedCount = data.Rules[i].SuppressedCount
|
||||
rule.LastError = data.Rules[i].LastError
|
||||
data.Rules[i] = rule
|
||||
return s.saveLocked(data)
|
||||
}
|
||||
}
|
||||
data.Rules = append(data.Rules, rule)
|
||||
return s.saveLocked(data)
|
||||
}
|
||||
|
||||
func (s *FileStore) UpsertAlertChannel(channel AlertChannel) error {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
data, err := s.loadLocked()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if channel.ID == "" {
|
||||
channel.ID = slug(channel.Name)
|
||||
}
|
||||
if channel.ID == "" {
|
||||
return errors.New("channel id or name is required")
|
||||
}
|
||||
if channel.Params == nil {
|
||||
channel.Params = map[string]string{}
|
||||
}
|
||||
for i := range data.AlertChannels {
|
||||
if data.AlertChannels[i].ID == channel.ID {
|
||||
preserveSecrets(data.AlertChannels[i].Params, channel.Params)
|
||||
data.AlertChannels[i] = channel
|
||||
return s.saveLocked(data)
|
||||
}
|
||||
}
|
||||
data.AlertChannels = append(data.AlertChannels, channel)
|
||||
return s.saveLocked(data)
|
||||
}
|
||||
|
||||
func preserveSecrets(oldParams, newParams map[string]string) {
|
||||
for _, key := range []string{"token", "password", "secret", "authorization"} {
|
||||
if newParams[key] == "" && oldParams[key] != "" {
|
||||
newParams[key] = oldParams[key]
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (s *FileStore) DeleteAlertChannel(id string) error {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
data, err := s.loadLocked()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
channels := data.AlertChannels[:0]
|
||||
for _, channel := range data.AlertChannels {
|
||||
if channel.ID != id {
|
||||
channels = append(channels, channel)
|
||||
}
|
||||
}
|
||||
data.AlertChannels = channels
|
||||
for i := range data.Rules {
|
||||
ids := data.Rules[i].AlertChannels[:0]
|
||||
for _, channelID := range data.Rules[i].AlertChannels {
|
||||
if channelID != id {
|
||||
ids = append(ids, channelID)
|
||||
}
|
||||
}
|
||||
data.Rules[i].AlertChannels = ids
|
||||
}
|
||||
return s.saveLocked(data)
|
||||
}
|
||||
|
||||
func (s *FileStore) UpdateRule(id string, fn func(*Rule)) error {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
data, err := s.loadLocked()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for i := range data.Rules {
|
||||
if data.Rules[i].ID == id {
|
||||
fn(&data.Rules[i])
|
||||
return s.saveLocked(data)
|
||||
}
|
||||
}
|
||||
return errors.New("rule not found")
|
||||
}
|
||||
|
||||
func (s *FileStore) AddIncident(incident Incident) error {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
data, err := s.loadLocked()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if incident.ID == "" {
|
||||
incident.ID = slug(incident.RuleName) + "-" + incident.CreatedAt.Format("20060102150405")
|
||||
}
|
||||
data.Incidents = append([]Incident{incident}, data.Incidents...)
|
||||
if len(data.Incidents) > 200 {
|
||||
data.Incidents = data.Incidents[:200]
|
||||
}
|
||||
return s.saveLocked(data)
|
||||
}
|
||||
|
||||
func (s *FileStore) loadLocked() (Data, error) {
|
||||
var data Data
|
||||
contents, err := os.ReadFile(s.path)
|
||||
if errors.Is(err, os.ErrNotExist) {
|
||||
return data, nil
|
||||
}
|
||||
if err != nil {
|
||||
return data, err
|
||||
}
|
||||
if len(contents) == 0 {
|
||||
return data, nil
|
||||
}
|
||||
return data, json.Unmarshal(contents, &data)
|
||||
}
|
||||
|
||||
func (s *FileStore) saveLocked(data Data) error {
|
||||
if err := os.MkdirAll(filepath.Dir(s.path), 0o700); err != nil {
|
||||
return err
|
||||
}
|
||||
contents, err := json.MarshalIndent(data, "", " ")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return os.WriteFile(s.path, append(contents, '\n'), 0o600)
|
||||
}
|
||||
|
||||
func slug(value string) string {
|
||||
value = strings.ToLower(strings.TrimSpace(value))
|
||||
var b strings.Builder
|
||||
lastDash := false
|
||||
for _, r := range value {
|
||||
ok := r >= 'a' && r <= 'z' || r >= '0' && r <= '9'
|
||||
if ok {
|
||||
b.WriteRune(r)
|
||||
lastDash = false
|
||||
continue
|
||||
}
|
||||
if !lastDash {
|
||||
b.WriteByte('-')
|
||||
lastDash = true
|
||||
}
|
||||
}
|
||||
return strings.Trim(b.String(), "-")
|
||||
}
|
||||
38
internal/store/store_test.go
Normal file
38
internal/store/store_test.go
Normal file
@ -0,0 +1,38 @@
|
||||
package store
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestUpsertRuleDefaults(t *testing.T) {
|
||||
s := NewFileStore(t.TempDir() + "/config.json")
|
||||
if err := s.UpsertRule(Rule{Name: "API Errors", LogQL: `{service="api"} |= "error"`}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
data, err := s.Load()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(data.Rules) != 1 {
|
||||
t.Fatalf("expected one rule, got %d", len(data.Rules))
|
||||
}
|
||||
rule := data.Rules[0]
|
||||
if rule.ID != "api-errors" || rule.Threshold != 1 || rule.Window != "5m" || rule.Cooldown != "1h" {
|
||||
t.Fatalf("unexpected defaults: %#v", rule)
|
||||
}
|
||||
}
|
||||
|
||||
func TestUpsertAlertChannelPreservesSecret(t *testing.T) {
|
||||
s := NewFileStore(t.TempDir() + "/config.json")
|
||||
if err := s.UpsertAlertChannel(AlertChannel{ID: "ntfy", Name: "ntfy", Type: "ntfy", Enabled: true, Params: map[string]string{"token": "secret", "topic": "alerts"}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := s.UpsertAlertChannel(AlertChannel{ID: "ntfy", Name: "ntfy", Type: "ntfy", Enabled: true, Params: map[string]string{"topic": "alerts2"}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
data, err := s.Load()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := data.AlertChannels[0].Params["token"]; got != "secret" {
|
||||
t.Fatalf("secret was not preserved: %q", got)
|
||||
}
|
||||
}
|
||||
285
project-docs/design/phase-1-plan.md
Normal file
285
project-docs/design/phase-1-plan.md
Normal file
@ -0,0 +1,285 @@
|
||||
# Phase 1 Plan
|
||||
|
||||
Goal: make log monitoring useful in production without enabling dangerous automation.
|
||||
|
||||
Phase 1 targets a production Portainer deployment of Log Guardian. Loki will likely run on a separate server and use an object-storage backend, probably MinIO backed by NAS storage. Log Guardian must remain safe-by-default: authenticated UI, real alert delivery, incident cooldowns, Loki validation tooling, dry-run remediation evidence, and local-only AI analysis.
|
||||
|
||||
## Decisions made
|
||||
|
||||
- Deployment target: production Portainer.
|
||||
- Loki location: separate server from Log Guardian.
|
||||
- Loki storage direction: object storage via MinIO, with MinIO data on NAS-backed storage.
|
||||
- UI auth: single admin user using basic auth.
|
||||
- Health endpoint: `/healthz` remains unauthenticated.
|
||||
- First alert channel: self-hosted ntfy.
|
||||
- Additional alert channel types should be supported/configurable from UI, not ntfy-only.
|
||||
- Incident cooldown: per rule, default `1h`.
|
||||
- AI: local-only Ollama is approved; no external/cloud AI providers.
|
||||
- Remediation: Phase 1 should implement permission modeling and dry-run evidence, not dangerous automatic actions.
|
||||
- Phase 1 completion requires Docker build and real alert delivery validation.
|
||||
|
||||
## 1. Production deployment architecture
|
||||
|
||||
### Log Guardian
|
||||
|
||||
Run Log Guardian as a Portainer-managed service/container.
|
||||
|
||||
Required mounts/config:
|
||||
|
||||
- Persistent local config/data file mount for `LOG_GUARDIAN_DATA_PATH`.
|
||||
- Environment configuration for Loki endpoint, auth, alert defaults, and optional Ollama endpoint.
|
||||
- Network path to the remote Loki server.
|
||||
- Optional network path to self-hosted ntfy, Gotify, SMTP relay, webhook targets, and Ollama.
|
||||
|
||||
### Loki
|
||||
|
||||
Loki will run on a different server. Recommended production direction:
|
||||
|
||||
- Loki single-binary or simple scalable deployment depending on expected volume.
|
||||
- Object storage backend: S3-compatible MinIO.
|
||||
- MinIO data stored on NAS-backed storage.
|
||||
- Retention configured explicitly.
|
||||
- Promtail or Grafana Alloy deployed on every Docker/Portainer host that needs logs collected.
|
||||
|
||||
Recommended initial storage sizing:
|
||||
|
||||
- Start with 50 GB usable object storage for modest private infrastructure.
|
||||
- Use 14-day retention initially.
|
||||
- Monitor ingestion and object growth for 7 days before increasing retention.
|
||||
- Increase to 100-250 GB+ if services are noisy or retention must be longer.
|
||||
|
||||
Planning assumptions to validate:
|
||||
|
||||
- NAS latency is acceptable for MinIO/Loki workload.
|
||||
- MinIO has backups/snapshots appropriate for desired retention.
|
||||
- Loki retention deletes old chunks/index data correctly.
|
||||
- Log Guardian can reach Loki over a private network/VPN only.
|
||||
|
||||
## 2. Loki ingestion, label discovery, and validation
|
||||
|
||||
Because actual labels are not known yet, Phase 1 must include label discovery and query testing.
|
||||
|
||||
### Collection strategy
|
||||
|
||||
Use Promtail or Grafana Alloy on Docker/Portainer hosts. Target labels should include stable, low-cardinality dimensions such as:
|
||||
|
||||
- `host`
|
||||
- `container`
|
||||
- `service`
|
||||
- `stack`
|
||||
- `stream`
|
||||
- `compose_project` or equivalent
|
||||
- `compose_service` or equivalent
|
||||
|
||||
Avoid high-cardinality labels, including request IDs, user IDs, IP addresses, paths, error messages, or raw log content.
|
||||
|
||||
### Log Guardian Loki diagnostics
|
||||
|
||||
Add authenticated UI/API support for:
|
||||
|
||||
- Loki readiness/status check.
|
||||
- List labels from `/loki/api/v1/labels`.
|
||||
- List values for a selected label from `/loki/api/v1/label/{name}/values`.
|
||||
- Test a LogQL query before saving a rule.
|
||||
- Show count and sanitized sample lines for a test query.
|
||||
|
||||
Implementation notes:
|
||||
|
||||
- Limit query time range and result count.
|
||||
- Do not log raw sample payloads by default.
|
||||
- Redact obvious secrets from displayed query samples where practical.
|
||||
- Handle Loki tenant ID and basic auth from existing Loki config.
|
||||
|
||||
## 3. UI authentication
|
||||
|
||||
Add single-admin basic auth.
|
||||
|
||||
Configuration:
|
||||
|
||||
- `LOG_GUARDIAN_AUTH_USERNAME`
|
||||
- `LOG_GUARDIAN_AUTH_PASSWORD`
|
||||
|
||||
Rules:
|
||||
|
||||
- `/healthz` is unauthenticated.
|
||||
- All other UI/API routes require auth.
|
||||
- Use constant-time comparison for credentials.
|
||||
- Do not log usernames/passwords or auth headers.
|
||||
- If credentials are unset, fail closed in production or clearly mark auth as disabled only for local development. Prefer fail closed for Portainer deployment.
|
||||
|
||||
## 4. Alert channels
|
||||
|
||||
Alert channels must be configurable from the UI and stored in local configuration. Secrets need careful handling.
|
||||
|
||||
### Required first channel: self-hosted ntfy
|
||||
|
||||
Support fields:
|
||||
|
||||
- Name
|
||||
- Enabled
|
||||
- Server URL
|
||||
- Topic
|
||||
- Optional bearer token or token file reference
|
||||
- Priority
|
||||
- Tags
|
||||
|
||||
Validation:
|
||||
|
||||
- Send a test notification from the UI.
|
||||
- Record delivery result without logging token values.
|
||||
|
||||
### Additional Phase 1 channel types
|
||||
|
||||
Implement the data model and UI so multiple channel types are possible. Preferred Phase 1 support order:
|
||||
|
||||
1. `ntfy` - implement real delivery first.
|
||||
2. `gotify` - real delivery if time allows; token-based API is straightforward.
|
||||
3. `generic_webhook` - useful but must avoid leaking secrets; support explicit headers carefully.
|
||||
4. `smtp` - useful, but can be deferred if it complicates TLS/auth validation.
|
||||
|
||||
Minimum acceptable Phase 1 scope:
|
||||
|
||||
- UI can create/edit/delete alert channels.
|
||||
- UI can attach channels to rules.
|
||||
- ntfy real delivery works.
|
||||
- Other channel types may be present as disabled/scaffolded if not implemented, but the UI should clearly show unsupported status.
|
||||
|
||||
Secret handling:
|
||||
|
||||
- Do not print secrets in logs.
|
||||
- Do not render stored secret values back into forms; show only `configured`/`not configured`.
|
||||
- Prefer environment variable or mounted file references for sensitive channel tokens when possible.
|
||||
- If secrets are stored in local JSON, the file must be written with `0600` permissions and documented as sensitive.
|
||||
|
||||
## 5. Rule evaluation and incident cooldown
|
||||
|
||||
Add per-rule cooldown.
|
||||
|
||||
Rule field:
|
||||
|
||||
```json
|
||||
"cooldown": "1h"
|
||||
```
|
||||
|
||||
Default behavior:
|
||||
|
||||
- If unset, cooldown is `1h`.
|
||||
- If a rule fires outside cooldown:
|
||||
- create incident;
|
||||
- send configured alert channels;
|
||||
- record dry-run remediation evidence if actions are configured.
|
||||
- If a rule fires during cooldown:
|
||||
- update rule last checked/matched/count fields;
|
||||
- do not create a duplicate incident;
|
||||
- do not send another alert;
|
||||
- increment or record suppressed count if implemented.
|
||||
|
||||
## 6. Remediation/action model
|
||||
|
||||
Phase 1 should not execute destructive remediation. Implement guardrails and dry-run evidence.
|
||||
|
||||
Recommended Phase 1 action types:
|
||||
|
||||
- `generic_webhook`: dry-run by default; real execution can be deferred or restricted.
|
||||
- `portainer_restart`: dry-run only in Phase 1.
|
||||
- `allowlisted_command`: dry-run only in Phase 1; command IDs only, never arbitrary shell from UI/AI.
|
||||
- `record_recommendation`: safe action that stores an advisory recommendation on the incident.
|
||||
|
||||
Guardrails:
|
||||
|
||||
- Global dry-run defaults true.
|
||||
- Per-rule action must be enabled.
|
||||
- Per-action dry-run should be explicit or inherited from global dry-run.
|
||||
- AI may recommend actions but cannot execute them.
|
||||
- No arbitrary command execution.
|
||||
- No Portainer write API calls in Phase 1 unless separately approved.
|
||||
|
||||
Validation evidence:
|
||||
|
||||
- At least one rule with a configured dry-run action.
|
||||
- Incident shows what would have happened.
|
||||
- Logs do not expose secrets or raw full log payloads.
|
||||
|
||||
## 7. Local-only AI analysis
|
||||
|
||||
Local-only Ollama is approved for Phase 1. External/cloud AI providers are not approved.
|
||||
|
||||
Configuration:
|
||||
|
||||
- `LOG_GUARDIAN_ANALYSIS_PROVIDER=none|ollama`
|
||||
- `LOG_GUARDIAN_OLLAMA_URL=http://ollama:11434`
|
||||
- `LOG_GUARDIAN_OLLAMA_MODEL=<model>`
|
||||
|
||||
Default:
|
||||
|
||||
- `LOG_GUARDIAN_ANALYSIS_PROVIDER=none`
|
||||
|
||||
Rules:
|
||||
|
||||
- AI analysis remains opt-in per rule.
|
||||
- AI findings are advisory only.
|
||||
- Limit number of log lines sent to Ollama.
|
||||
- Redact obvious secrets before prompt construction.
|
||||
- Do not log prompts or raw log payloads by default.
|
||||
- Store only concise summaries/recommendations in incidents.
|
||||
|
||||
Prompt/redaction design must be documented before enabling Ollama in production.
|
||||
|
||||
## 8. UI scope
|
||||
|
||||
Add UI support for:
|
||||
|
||||
- Login-protected dashboard.
|
||||
- Rule create/edit with cooldown and alert channel selection.
|
||||
- Alert channel create/edit/test.
|
||||
- Loki status/label discovery/query test.
|
||||
- Incident list showing alert delivery state and remediation dry-run evidence where available.
|
||||
|
||||
Keep the UI served by Go. Do not add a server-side JavaScript runtime.
|
||||
|
||||
## 9. Tests and validation
|
||||
|
||||
Required implementation tests:
|
||||
|
||||
- Auth middleware unit tests.
|
||||
- Store tests for alert channels, cooldown fields, and secret-preserving updates.
|
||||
- Loki client tests using `httptest` fixtures for labels, label values, and query range.
|
||||
- Alert dispatcher tests for ntfy using `httptest`.
|
||||
- Rule engine tests for cooldown suppression.
|
||||
- Remediation dry-run tests.
|
||||
- Optional Ollama provider tests using `httptest` only; no live AI dependency in unit tests.
|
||||
|
||||
Required validation before Phase 1 completion:
|
||||
|
||||
```sh
|
||||
gofmt -w cmd internal
|
||||
go test ./...
|
||||
make validate-structure
|
||||
make docker-build
|
||||
```
|
||||
|
||||
Additional required evidence:
|
||||
|
||||
- Loki connectivity/query validation against the remote Loki server.
|
||||
- Label discovery output documented without secrets.
|
||||
- Self-hosted ntfy test alert received.
|
||||
- A rule-generated ntfy alert received.
|
||||
- Remediation dry-run evidence captured.
|
||||
- Security/privacy review covering logs, prompts, secrets, alert tokens, and action permissions.
|
||||
|
||||
## 10. Required Phase 1 artifacts
|
||||
|
||||
Create/update these before declaring Phase 1 complete:
|
||||
|
||||
- `project-docs/status/phase-1-summary.md`
|
||||
- `project-docs/status/phase-1-test-results.md`
|
||||
- `project-docs/status/phase-1-open-issues.md`
|
||||
|
||||
## 11. Open planning items
|
||||
|
||||
- Final Loki topology: single-binary vs simple scalable.
|
||||
- Final MinIO/NAS capacity and retention target after observing ingestion.
|
||||
- Whether Gotify, generic webhook, or SMTP are implemented fully in Phase 1 or scaffolded after ntfy.
|
||||
- Whether alert channel secrets are stored directly in local JSON or referenced via env/file paths.
|
||||
- Exact Ollama model to use once Ollama is stable.
|
||||
- Whether Log Guardian reaches Loki/ntfy/Ollama over LAN, VPN, or Docker overlay network.
|
||||
29
project-docs/runbooks/loki-setup.md
Normal file
29
project-docs/runbooks/loki-setup.md
Normal file
@ -0,0 +1,29 @@
|
||||
# Loki Setup Runbook
|
||||
|
||||
## Starter local/Portainer stack
|
||||
|
||||
Use `deploy/loki/docker-compose.yml` and `deploy/loki/promtail-config.yml`.
|
||||
|
||||
```sh
|
||||
cd deploy/loki
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
## Verify Loki
|
||||
|
||||
```sh
|
||||
curl -fsS http://localhost:3100/ready
|
||||
curl -G http://localhost:3100/loki/api/v1/labels
|
||||
```
|
||||
|
||||
## Useful LogQL examples
|
||||
|
||||
```logql
|
||||
{service="api"} |= "error"
|
||||
{stack="second-brain"} |~ "(?i)(panic|fatal|exception)"
|
||||
{stream="stderr"} |~ "(?i)(error|failed|timeout)"
|
||||
```
|
||||
|
||||
## Portainer notes
|
||||
|
||||
For Portainer deployment, mount the Docker socket and Docker container log directory read-only into Promtail. The included compose does this for a single Docker host. Multi-node Swarm setups need Promtail on each node or another log shipping strategy.
|
||||
8
project-docs/status/phase-0-open-issues.md
Normal file
8
project-docs/status/phase-0-open-issues.md
Normal file
@ -0,0 +1,8 @@
|
||||
# Phase 0 Open Issues
|
||||
|
||||
- Add UI authentication before exposing beyond a trusted private network.
|
||||
- Validate the Loki + Promtail setup against actual Portainer/Docker hosts.
|
||||
- Decide alert channels: email, ntfy, Gotify, Discord, Matrix, SMS, or another destination.
|
||||
- Decide which remediation actions are allowed. Candidates: send webhook, restart Portainer service, scale service, pause updater, open issue, or run a tightly allowlisted command.
|
||||
- AI analysis provider is intentionally unset. Any external AI/cloud provider requires explicit approval and prompt/log redaction design.
|
||||
- Add real alert delivery, retry behavior, deduplication, silences, and escalation policies.
|
||||
16
project-docs/status/phase-0-summary.md
Normal file
16
project-docs/status/phase-0-summary.md
Normal file
@ -0,0 +1,16 @@
|
||||
# Phase 0 Summary
|
||||
|
||||
Created initial `log-guardian` scaffold.
|
||||
|
||||
Included:
|
||||
|
||||
- Go HTTP service with health endpoint and basic web UI.
|
||||
- Local JSON-backed rule, alert-channel, and incident storage.
|
||||
- Loki query client for LogQL range queries.
|
||||
- Rule engine with thresholds and incident creation.
|
||||
- No-op analyzer interface for future AI agent analysis.
|
||||
- Dry-run remediation action runner scaffold.
|
||||
- Alert dispatcher scaffold that logs alert intent only.
|
||||
- Dockerfile, Portainer stack, Loki + Promtail starter compose, example environment, Makefile, CI workflow, and phase artifacts.
|
||||
|
||||
Phase 0 does not certify production alerting or remediation.
|
||||
19
project-docs/status/phase-0-test-results.md
Normal file
19
project-docs/status/phase-0-test-results.md
Normal file
@ -0,0 +1,19 @@
|
||||
# Phase 0 Test Results
|
||||
|
||||
Validation run from `/data/code/log-guardian` after scaffold creation.
|
||||
|
||||
Passed:
|
||||
|
||||
```sh
|
||||
gofmt -w cmd internal
|
||||
go test ./...
|
||||
make validate-structure
|
||||
make go-fmt
|
||||
make docker-build
|
||||
```
|
||||
|
||||
Evidence:
|
||||
|
||||
- `go test ./...` passed for all packages, including `internal/analysis` and `internal/store` tests.
|
||||
- `make validate-structure` printed `Structure validation passed`.
|
||||
- Docker image built successfully as `gitea.wayfinderak.com/wayfinderak/log-guardian:local`.
|
||||
10
project-docs/status/phase-1-open-issues.md
Normal file
10
project-docs/status/phase-1-open-issues.md
Normal file
@ -0,0 +1,10 @@
|
||||
# Phase 1 Open Issues
|
||||
|
||||
- Live deployment validation is still required against production Portainer, remote Loki, and self-hosted ntfy.
|
||||
- Loki topology is not finalized: single-binary vs scalable deployment, exact MinIO/NAS capacity, and retention settings need confirmation.
|
||||
- Promtail/Grafana Alloy labels must be discovered on the real Docker/Portainer hosts.
|
||||
- Gotify, generic webhook, and SMTP channels are scaffolded but not implemented for real delivery.
|
||||
- Alert channel secrets are currently stored in the local JSON config if entered through the UI; mounted secret-file/env references should be considered for stronger secret handling.
|
||||
- Ollama provider is implemented as local-only and opt-in, but the final model and runtime deployment still need validation.
|
||||
- No destructive remediation is implemented; Portainer restart and allowlisted command actions are dry-run evidence only.
|
||||
- The UI supports create/replace workflows but does not yet provide polished edit forms for existing rules/channels.
|
||||
16
project-docs/status/phase-1-summary.md
Normal file
16
project-docs/status/phase-1-summary.md
Normal file
@ -0,0 +1,16 @@
|
||||
# Phase 1 Summary
|
||||
|
||||
Implemented Phase 1 foundations for production use:
|
||||
|
||||
- Basic auth protects UI/API routes; `/healthz` remains unauthenticated.
|
||||
- Loki diagnostics endpoints were added for labels, label values, and limited query testing.
|
||||
- Rules now support per-rule cooldowns with default `1h` and duplicate incident suppression.
|
||||
- Alert channels are configurable in the UI.
|
||||
- Real self-hosted ntfy delivery is implemented with test-send support.
|
||||
- Gotify, generic webhook, and SMTP channel types are represented as scaffolded options.
|
||||
- Incidents can record alert delivery results and remediation dry-run evidence.
|
||||
- Remediation action runner supports guarded dry-run evidence for generic webhook, Portainer restart, allowlisted command, and recommendation actions.
|
||||
- Local-only Ollama analysis support was added and remains disabled by default.
|
||||
- Secret handling was improved for alert channel token preservation and display masking.
|
||||
|
||||
Phase 1 production rollout still requires live validation against the user's remote Loki and self-hosted ntfy deployments.
|
||||
37
project-docs/status/phase-1-test-results.md
Normal file
37
project-docs/status/phase-1-test-results.md
Normal file
@ -0,0 +1,37 @@
|
||||
# Phase 1 Test Results
|
||||
|
||||
Validation run from `/data/code/log-guardian` after Phase 1 implementation.
|
||||
|
||||
Passed:
|
||||
|
||||
```sh
|
||||
gofmt -w cmd internal
|
||||
go test ./...
|
||||
make validate-structure
|
||||
make go-fmt
|
||||
make go-test
|
||||
make docker-build
|
||||
```
|
||||
|
||||
Docker image built successfully as:
|
||||
|
||||
```text
|
||||
gitea.wayfinderak.com/wayfinderak/log-guardian:local
|
||||
```
|
||||
|
||||
Unit/integration-style tests added:
|
||||
|
||||
- Basic auth middleware protects UI routes and leaves `/healthz` unauthenticated.
|
||||
- Store defaults include rule cooldown and alert channel secret-preserving updates.
|
||||
- Loki client label, label-value, and query-range behavior uses `httptest` fixtures.
|
||||
- ntfy dispatcher delivery uses an `httptest` fixture.
|
||||
- Rule engine suppresses duplicate incidents during cooldown.
|
||||
|
||||
Pending live production validation before Phase 1 can be called operationally complete:
|
||||
|
||||
- Remote Loki connectivity and label discovery.
|
||||
- Remote Loki LogQL query validation.
|
||||
- Self-hosted ntfy test notification received.
|
||||
- Rule-generated ntfy alert received.
|
||||
- Remediation dry-run evidence captured from a real rule.
|
||||
- Security/privacy review of logs, prompts, secrets, alert tokens, and action permissions.
|
||||
1
web/static/app.css
Normal file
1
web/static/app.css
Normal file
@ -0,0 +1 @@
|
||||
:root{color-scheme:dark;--bg:#0d1117;--panel:#161b22;--text:#f0f6fc;--muted:#8b949e;--accent:#58a6ff;--danger:#ff7b72}*{box-sizing:border-box}body{margin:0;background:var(--bg);color:var(--text);font-family:system-ui,-apple-system,Segoe UI,sans-serif}.shell{max-width:1200px;margin:0 auto;padding:2rem}.card{background:var(--panel);border:1px solid #30363d;border-radius:16px;padding:1rem;margin:1rem 0}.grid{display:grid;grid-template-columns:repeat(auto-fit,minmax(220px,1fr));gap:1rem;align-items:end}.wide{grid-column:1/-1}label{display:grid;gap:.35rem;color:var(--muted)}.check{display:flex;gap:.5rem;align-items:center}input{width:100%;padding:.65rem;border-radius:10px;border:1px solid #30363d;background:#010409;color:var(--text)}button{padding:.65rem .9rem;border:0;border-radius:10px;background:var(--accent);color:#06111f;font-weight:700;cursor:pointer}.row{display:flex;justify-content:space-between;gap:1rem;align-items:center}table{width:100%;border-collapse:collapse}th,td{text-align:left;padding:.7rem;border-top:1px solid #30363d;vertical-align:top}code{color:#a5d6ff}.muted{color:var(--muted)}.error{color:var(--danger)}@media(max-width:760px){.shell{padding:1rem}table{display:block;overflow-x:auto}}
|
||||
Loading…
x
Reference in New Issue
Block a user