---
name: linux-server-ops
description: Production-grade Linux server operations for systemd services, Caddy reverse proxying, and homelab container networking.
---

# Linux Server Operations (systemd, Caddy, Homelab Infrastructure)

Best practices and patterns for deploying, securing, and maintaining production services on Debian/Ubuntu-based Linux servers (such as homelabs, VPS, or dedicated hosts).

---

## 1. Caddy Reverse Proxy & Web Server

Caddy provides automatic TLS, modern HTTP/2 & HTTP/3 support, and simple declarative configuration.

### Core Caddyfile Patterns

#### Standard Reverse Proxy to Local Service / Docker
```caddy
# Expose an internal service (e.g., FastAPI, Node, or Docker container)
subdomain.example.com {
    reverse_proxy 127.0.0.1:8085
}
```

#### Homelab Internal TLS (`tls internal`)
For local-only IP addresses or internal domain names without public DNS:
```caddy
http://192.168.1.183:8085 {
    reverse_proxy 127.0.0.1:18511
}

# Or with self-signed trusted homelab TLS:
my-server.local {
    tls internal
    reverse_proxy 127.0.0.1:8080
}
```

#### WebSockets & Streaming (SSE)
Caddy automatically handles WebSockets and Server-Sent Events (SSE). To ensure real-time streaming without buffering:
```caddy
api.example.com {
    reverse_proxy 127.0.0.1:8085 {
        # Flush response buffers immediately for SSE and real-time feeds
        flush_interval -1
    }
}
```

#### Reverse Proxy with Security Headers
```caddy
app.example.com {
    header {
        Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
        X-Content-Type-Options "nosniff"
        X-Frame-Options "SAMEORIGIN"
        Referrer-Policy "strict-origin-when-cross-origin"
    }
    reverse_proxy 127.0.0.1:3000
}
```

### Operational Commands for Caddy
```bash
# Validate Caddyfile without applying
caddy validate --config /etc/caddy/Caddyfile

# Zero-downtime configuration reload
systemctl reload caddy
# or directly via Caddy CLI:
caddy reload --config /etc/caddy/Caddyfile

# View live Caddy logs
journalctl -u caddy -f --no-pager
```

---

## 2. systemd Service Unit Management

systemd is the standard init and service manager for Linux. Every standalone Python, Node, Go, or Rust service should run under a hardened systemd unit.

### Production Unit Template: `/etc/systemd/system/<service-name>.service`

```ini
[Unit]
Description=My Application Service
After=network.target network-online.target
Wants=network-online.target

[Service]
Type=simple
# Run as non-privileged service user
User=max
Group=max
WorkingDirectory=/home/max/projects/my-app

# Command line execution (use absolute binary paths)
ExecStart=/home/max/projects/my-app/venv/bin/python main.py

# Auto-recovery and restart behavior
Restart=always
RestartSec=5s
KillMode=mixed
TimeoutStopSec=15s

# Environment configuration
Environment=PYTHONUNBUFFERED=1
Environment=PORT=8085
# Or load secrets from an external protected file:
# EnvironmentFile=/etc/default/my-app

# Hardening & Security Sandboxing
NoNewPrivileges=true
ProtectSystem=full
ProtectHome=read-only
PrivateTmp=true

[Install]
WantedBy=multi-user.target
```

### Standard systemd Operations Workflow
```bash
# 1. Reload daemon after modifying any .service file
sudo systemctl daemon-reload

# 2. Enable service on boot and start immediately
sudo systemctl enable --now <service-name>

# 3. Check status, process ID, memory, and recent logs
systemctl status <service-name>

# 4. Graceful restart
sudo systemctl restart <service-name>

# 5. Live log tailing
journalctl -u <service-name> -f

# 6. View recent error lines only
journalctl -u <service-name> -p err -n 50 --no-pager
```

---

## 3. Homelab Coexistence: Docker + Local Services + Caddy

When running both Docker containers and native host services on the same server:

1. **Loopback Binding for Containers:**
   Never bind containers to `0.0.0.0:PORT` if they are intended to go through Caddy. Always bind to `127.0.0.1:PORT:PORT` in `docker-compose.yml`:
   ```yaml
   services:
     open-webui:
       image: ghcr.io/open-webui/open-webui:main
       ports:
         - "127.0.0.1:8080:8080"
       restart: unless-stopped
   ```
2. **Reverse Proxy Routing:**
   Point Caddy to `127.0.0.1:8080`. This keeps internal ports closed to the outside network while allowing Caddy to handle authentication, TLS, and access logging.

3. **Port Collisions:**
   Always check active listeners before assigning a port:
   ```bash
   ss -tulpn | grep LISTEN
   ```