Logging Infrastructure

Overview

The API uses a structured logging pipeline:

API Container → stdout (JSON) → Promtail → Loki → Grafana

Service	Image	Port	Binding	Purpose
Loki	grafana/loki:3.4.2	3100	`127.0.0.1` (localhost only)	Log storage & query engine
Promtail	grafana/promtail:3.4.2	9080	Internal only	Scrapes Docker container logs
Grafana	grafana/grafana:11.5.2	3000	`0.0.0.0` (all interfaces)	Dashboard & alerting UI

Accessing Logs

Remote Access via Grafana

Grafana is bound to all interfaces and accessible remotely:

http://<server-ip>:3000

Default credentials: admin / admin

Important: Change the default password immediately when exposing Grafana to the network. Set GF_SECURITY_ADMIN_PASSWORD in docker-compose.yml or change it on first login.

Pre-built Dashboard

An API Overview dashboard is auto-provisioned with:

Request Rate by Endpoint
Error Rate Over Time
Request Latency (p50/p95/p99)
Active Users
Total Requests (1h)
Error Count (1h)
Top Endpoints
Recent Errors (detailed log view)

The dashboard auto-refreshes every 30 seconds and shows the last hour of data.

LogQL Queries

In Grafana, go to Explore → select Loki datasource:

# All API logs
{service="org-chart-api"}

# Errors and fatals only
{service="org-chart-api", level=~"error|fatal"}

# Logs for a specific endpoint
{service="org-chart-api"} | json | path="/api/orgcharts"

# Request rate over time
count_over_time({service="org-chart-api"} [1m])

# Filter by request ID
{service="org-chart-api"} | json | requestId="abc-123"

# Filter by user
{service="org-chart-api"} | json | userId="user-id-here"

Raw Docker Logs

For quick access without Grafana:

docker compose logs -f api              # Follow API logs
docker compose logs -f --tail=100 api   # Last 100 lines
docker compose logs -f loki             # Loki health
docker compose logs -f promtail         # Promtail scraping status

Security: Port Bindings

Internal services are bound to localhost only to prevent unauthorized network access:

Service	Port	Binding	Remote Access
API	8789	`0.0.0.0`	Yes (public API)
Grafana	3000	`0.0.0.0`	Yes — dashboard UI
Redis	6379	`127.0.0.1`	No (localhost only)
Loki	3100	`127.0.0.1`	No (localhost only)

Loki and Redis do not need external access. Promtail and Grafana communicate with them over the internal Docker network. The 127.0.0.1 binding keeps them off the network while still allowing local debugging via SSH.

Recommendations for Internet-Facing Servers

Change the default Grafana password — set GF_SECURITY_ADMIN_PASSWORD in docker-compose.yml
Firewall — restrict port 3000 to your IP or trusted networks
Reverse proxy — put Grafana behind Nginx/Caddy with HTTPS

Alerting

Three alert rules are pre-configured in Grafana:

Alert	Condition	Severity
High Error Rate	>10 errors/min	Warning
Fatal Error Detected	Any fatal-level log	Critical
API Heartbeat Missing	No logs for 5 minutes	Critical

Email Alerts

Set these environment variables in .env to enable email alerts on error/fatal logs:

[email protected]
RESEND_API_KEY=your_resend_api_key

Configuration

Environment Variables

Variable	Default	Description
`LOG_LEVEL`	`info` (prod) / `debug` (dev)	Minimum log level
`LOKI_URL`	`http://loki:3100`	Loki endpoint (internal)
`ALERT_EMAIL`	—	Email for error/fatal alerts
`RESEND_API_KEY`	—	Resend API key for email delivery

Log Levels

debug (0) → info (1) → warn (2) → error (3) → fatal (4)

Data Retention

Retention period: 30 days (configured in config/loki.yml)
Storage: Local filesystem via Docker volume loki_data
Compaction: Runs every 10 minutes

Config Files

File	Purpose
`config/loki.yml`	Loki server and storage configuration
`config/promtail.yml`	Log scraping rules and label extraction
`config/grafana-provisioning/`	Grafana datasources and alert rules
`config/dashboards/api-overview.json`	Pre-built Grafana dashboard