Systemd Services and Journalctl

Systemd is the init system on most modern Linux distros. It manages services (start, stop, restart, dependencies), captures logs (journalctl), and runs timers (13_cron_and_systemd_timers.md). For Python developers: how to wrap your app as a service and read its logs.

A minimal service unit

/etc/systemd/system/myapp.service:

[Unit]
Description=My Python web app
After=network.target postgresql.service
Requires=postgresql.service

[Service]
Type=simple
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
EnvironmentFile=/etc/myapp/env
ExecStart=/opt/myapp/.venv/bin/gunicorn myapp.wsgi:application --bind 0.0.0.0:8000 --workers 4
Restart=always
RestartSec=5
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

Three sections:

Enabling and managing

sudo systemctl daemon-reload                # re-read unit files after editing
sudo systemctl enable myapp.service         # start at boot
sudo systemctl start myapp.service          # start now
sudo systemctl enable --now myapp.service   # both at once

sudo systemctl status myapp                 # detailed status + last 10 lines of logs
sudo systemctl stop myapp
sudo systemctl restart myapp
sudo systemctl reload myapp                 # SIGHUP — only if app supports reload
sudo systemctl disable myapp                # don't start at boot (still running for now)

systemctl list-units --type=service         # list active services
systemctl list-units --type=service --state=failed   # failed services
systemctl list-unit-files                   # all known units

The .service extension is implied; systemctl status myapp works.

journalctl — reading logs

Systemd captures stdout/stderr from services and stores them in the journal:

journalctl -u myapp                         # all logs for myapp
journalctl -u myapp -f                      # follow (like tail -f)
journalctl -u myapp -n 100                  # last 100 lines
journalctl -u myapp --since "10 min ago"
journalctl -u myapp --since today
journalctl -u myapp --since "2024-01-15 10:00" --until "2024-01-15 12:00"

journalctl -u myapp -p err                  # only errors and worse
journalctl -u myapp --grep "ERROR"          # filter (regex)

journalctl -k                               # kernel messages (= dmesg)
journalctl -b                               # current boot
journalctl -b -1                            # previous boot

journalctl --disk-usage                     # how much disk the journal uses
sudo journalctl --vacuum-time=7d            # keep only last 7 days

Journal is binary; journalctl is the only sane way to read it. For long-term log shipping, configure rsyslog/syslog-ng/fluentd to forward.

Type — how systemd tracks the process

For Python services, simple is almost always right (Gunicorn, uvicorn, your own scripts).

For “I want to know when the service is actually ready, not just when it’s started”:

Type=notify
NotifyAccess=all
ExecStart=/opt/myapp/.venv/bin/python myapp.py

The app calls systemd.daemon.notify("READY=1") when ready. Useful for services that take a while to initialize.

Restart policies

Restart=always         # restart on any exit (clean or crash)
Restart=on-failure     # restart only if exit code is non-zero
Restart=on-abort       # restart only on signal
Restart=no             # never restart (default)

RestartSec=5           # wait 5s between restarts
StartLimitBurst=5      # allow N restarts in StartLimitInterval
StartLimitInterval=60  # window for the burst limit (seconds)

Restart=on-failure + RestartSec=5 is a sane default. If the app crash-loops 5 times in 60s, systemd gives up — examine logs.

Environment

Environment="DEBUG=true"
Environment="DATABASE_URL=postgres://..."
EnvironmentFile=/etc/myapp/env             # one VAR=value per line
EnvironmentFile=-/etc/myapp/env-optional   # leading - = ok if missing

Don’t put secrets in the unit file (world-readable in /etc/systemd/system/). Use EnvironmentFile with restricted perms (chmod 600).

Security hardening

Systemd has knobs for sandboxing services:

[Service]
User=myapp                          # don't run as root
Group=myapp
NoNewPrivileges=true                # prevent privilege escalation
PrivateTmp=true                     # private /tmp
ProtectSystem=strict                # /usr, /boot, /etc are read-only
ProtectHome=true                    # /home is invisible
ReadWritePaths=/var/lib/myapp      # explicit write access
CapabilityBoundingSet=              # drop all capabilities
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX

systemd-analyze security myapp.service scores the service’s exposure (lower = more locked down).

For a Python web app, the basics (User, NoNewPrivileges, ProtectSystem, ProtectHome) cost nothing and contain damage if the app is compromised.

ExecStartPre / ExecStartPost / ExecStop

[Service]
ExecStartPre=/opt/myapp/.venv/bin/python manage.py migrate --check
ExecStart=/opt/myapp/.venv/bin/gunicorn myapp.wsgi
ExecStop=/bin/kill -TERM $MAINPID         # default — usually fine
ExecStartPost=/usr/bin/curl -X POST https://...   # notify on start

ExecStartPre is for “verify or prepare before the main process.” Failure here aborts the service start.

Resource limits

[Service]
MemoryMax=512M                # OOM kill if exceeded
MemoryHigh=400M               # throttle approaching limit
CPUQuota=200%                 # max 2 cores
TasksMax=100                  # max threads/processes
LimitNOFILE=65535             # file descriptor limit

Useful for services that occasionally leak memory or fork-bomb — fail safely instead of taking the host down.

Reload vs restart

reload sends SIGHUP (or whatever ExecReload= says). The service must support graceful reload — if not, it’s a no-op or crash.

restart stops then starts — drops connections, fast.

For nginx, reload is true zero-downtime. For Gunicorn, kill -HUP reloads workers gracefully:

[Service]
ExecReload=/bin/kill -HUP $MAINPID

Then sudo systemctl reload myapp does graceful reload.

User services

Per-user systemd (services run as the user, no root needed):

mkdir -p ~/.config/systemd/user
# write ~/.config/systemd/user/myapp.service
systemctl --user daemon-reload
systemctl --user enable --now myapp.service
journalctl --user -u myapp

Useful for personal automation, dev tools. By default, user services stop when the user logs out — loginctl enable-linger USERNAME keeps them running.

When systemd isn’t right

• Kubernetes / containers — k8s manages process lifecycle; running systemd inside containers is awkward. Use the container’s PID 1 process directly.
• Heroku / PaaS — the platform handles process management.
• Quick experiments — tmux or nohup is faster than writing a unit file.
• Cron-style “run at this time” — use systemd timers (13_cron_and_systemd_timers.md) or cron.

Common pitfalls

• Edited the unit file but didn’t daemon-reload — systemd uses the cached version. Always sudo systemctl daemon-reload after editing.
• Type=simple for a forking daemon — systemd thinks the parent is the service, the parent exits, systemd marks it failed. Use Type=forking with PIDFile=.
• Secrets in unit files — readable by anyone with read on /etc/systemd/system/. Use EnvironmentFile= with chmod 600.
• Forgetting WorkingDirectory — relative paths in your script break.
• Forgetting daemon-reload after editing a .service file — the change isn’t picked up.
• Running as root — convenience, security hole. Always create a dedicated user (useradd -r -s /bin/false myapp).

Common interview confusions

• “Systemd is bloated.” — community opinion, not technically meaningful for “should I use it.” It’s universal on Linux distros and the standard way to manage services.
• “reload and restart are the same.” — reload sends SIGHUP (or the configured ExecReload); restart stops then starts. Reload is meant to be graceful; restart drops connections.
• “enable starts the service.” — enable only configures the service to start at boot. enable --now does both.
• “systemd writes logs to /var/log/myapp.log.” — by default, to the journal (binary, accessed via journalctl). Configure StandardOutput= to send elsewhere.

Interview angle

• “What does a systemd unit file look like for a Python web app?” — three sections: [Unit] (description, deps), [Service] (User, WorkingDirectory, ExecStart, Restart=always), [Install] (WantedBy=multi-user.target). Drop in /etc/systemd/system/myapp.service, daemon-reload, enable --now.
• “How do you read a service’s logs?” — journalctl -u service-name, with -f for tail-style follow, -n 100 for last 100 lines, --since "10 min ago" for time-bounded.
• “What’s Type=simple vs Type=forking?” — simple if your process stays in the foreground (most modern apps). forking if it self-daemonizes (parent forks and exits) — old-school daemons.
• “How do you make a service auto-restart on crash?” — Restart=on-failure and RestartSec=5. Optionally cap with StartLimitBurst / StartLimitInterval to avoid crash-looping forever.
• “How do you reload a service without dropping connections?” — ExecReload=/bin/kill -HUP $MAINPID in the unit, then systemctl reload service. The app must support SIGHUP-based graceful reload (gunicorn, nginx do).
• “What’s the difference between enable and start?” — enable means “start at boot”; start means “start now.” enable --now does both.
• “How do you prevent a service from running as root?” — User= and Group= in [Service]. Combine with NoNewPrivileges=true, ProtectSystem=strict, ProtectHome=true to sandbox further.