Alloy/Loki/Prometheus/Grafana observability demo

A production-style observability demo that demonstrates how to instrument a Node.js application end-to-end: log aggregation (Loki), metrics (Prometheus), and dashboards (Grafana) wired through an Alloy pipeline.

This will be a ready-to-run sandbox for testing real-world workflows — deploy locally or to the cloud, run load tests and fault injection, and iterate on alerting and dashboards to validate observability practices you can apply to production systems.

No configuration needed - just docker compose up.

🚀 Quickstart

git clone <repo> && cd <dir>
docker compose up -d  # Builds demo app, starts everything

See what's running and which ports are open:

docker compose ps

📊 Access

Service	URL	Notes
Grafana	http://localhost:3000	Metrics dashboards and visualizations (admin / admin)
Prometheus	http://localhost:9090	Metrics collection and query system
Loki	http://localhost:3100	Logs backend: /ready, /metrics
Alloy	http://localhost:12345	Unified observability pipeline
Demo App	http://localhost:8081	GET: /health, /metrics; POST: /login, /action/:type
Demo App (dev)	http://localhost:8082	Live-reload container for code changes, started with the dev profile

🧪 Test Flow

# Login (generates "LOGIN" log + metric)
SESSION=$(curl -s -X POST http://localhost:8081/login \
  -H "Content-Type: application/json" \
  -d '{"username":"demo"}' | jq -r .sessionId)

# Action (generates "ACTION" log + metric)
curl -X POST http://localhost:8081/action/click \
  -H "X-Session-ID: $SESSION" \
  -H "Content-Type: application/json" \
  -d '{}'

Query Prometheus

• View request rate: rate(http_requests_total{job="demo-app"}[5m])
• View total requests: sum(http_requests_total{job="demo-app"})

Explore Grafana

• Logs (Loki):
  • {service_name="demo-app"} → App logs
  • {service_name="demo-app"} |~ "LOGIN|ACTION" → User actions labeled by user/session
  • {service_name="grafana"} → Grafana logs
  • {service_name="loki"} → Loki logs
  • {job!=""} → All logs in the default Grafana Explore filter
  • {service_name!=""} → Same logs, using the Compose service label
• Metrics (Prometheus): rate(http_requests_total{job="demo-app"}[5m]) → Rate metrics

🔧 Commands

docker compose ps                # Status/ports
docker compose logs -f demo-app  # App logs
docker compose down -v           # Stop + wipe volumes
docker compose up --build -d     # Force rebuild

🛠 Development

Use the dev profile when you want live reload inside the container:

docker compose --profile dev up --build demo-app-dev

The dev container runs alongside the regular demo app if both are up. It listens on http://localhost:8082, while the production-style app stays on http://localhost:8081.

The dev service mounts ./services/demo-app into /app, so edits to server.js are picked up by nodemon automatically. The named demo-app-node_modules volume is mounted at /app/node_modules so the bind mount does not hide installed dependencies.

Typical edit loop:

1. Start the stack with docker compose up -d and the dev app with docker compose --profile dev up --build demo-app-dev.
2. Edit services/demo-app/server.js.
3. Refresh http://localhost:8082 or rerun your curl commands.

If you add or change dependencies, run docker compose --profile dev exec demo-app-dev pnpm install or rebuild the dev image.

prom-client stays as an explicit dependency on purpose: @matteodisabatino/express-prometheus-middleware still imports it and declares it as a peer dependency.

Structure

.
├── compose.yaml
├── config/
│ ├── alloy/config.alloy                        # Docker discovery and log forwarding to Loki
│ ├── grafana/datasources.yaml                  # Grafana datasource provisioning
│ ├── grafana/dashboards/provisioning.yaml      # Grafana dashboard provider config
│ ├── grafana/dashboards/demo-dashboard.json    # Grafana dashboard for monitoring the demo app in real-time
│ ├── loki/loki-config.yaml                     # Loki config using the single-store tsdb index
│ └── prometheus/prometheus.yaml                # Prometheus scrape config for the demo app and Alloy
├── services/demo-app/                          # Node.js app + prod and dev dockerfiles
└── README.md

🎉 Production-Style Features

• ✅ Alloy + Docker Compose labels → service_name="demo-app" and job="demo-app"
• ✅ Structured logs → LOGIN user=hacker session=abc
• ✅ Live tail → {service_name="demo-app"} |~ "LOGIN|ACTION"
• ✅ Pre-provisioned Grafana → instant dashboards
• ✅ Metrics + Logs → http_requests_total{job="demo-app"}

License

ISC