Lesson 11 · The system

Logs

A log is a diary a program keeps: one line per event, each with a time. On a Linux system that runs systemd, one service, systemd-journald, collects the lines that services print and the kernel's own messages, stamps each with the time, the unit and the process ID, files it at a priority, and stores them in the journal. Follow a three-line Python service's two lines into the journal, then read them back with journalctl, filtered by unit and by priority.

10 min9 steps5 questionsIntermediate

Fig 11 · Interactive

Loading figure…

or ← →

Check yourself

5 quick questions

Pick an answer to see why. Your score saves with your progress.

01A Python program run as a systemd service calls print("started", flush=True). Where does that line go by default?

systemd connects a service's stdout and stderr to systemd-journald by default (StandardOutput=journal). journald stamps the line and stores it in the journal.

02Which journal fields are added by journald itself, so the program cannot fake them?

Fields beginning with "_" are trusted fields that journald adds, such as _PID, _SYSTEMD_UNIT and _TRANSPORT. MESSAGE and PRIORITY come from the program.

03A service writes "<3>no config" to stderr. How does it appear in the journal?

With SyslogLevelPrefix= on (the default), a line starting with a level such as <3> is filed at that priority, 3 = err, and the prefix is removed.

04What does journalctl -p err show?

With a single level, journalctl shows that level and every lower number, which means more important: emerg, alert, crit and err.

05By default, how much disk may the persistent journal use?

journald.conf's SystemMaxUse= defaults to 10% of the filesystem, capped at 4G. Past the limit, journald deletes the oldest archived journal files first.

Key terms

Glossary

Log: A record a program keeps while it runs: one line per event, each with a time.
systemd-journald: The service that collects log lines from services, the kernel and syslog, stamps them and stores them in the journal.
Journal: journald's binary log files, in /var/log/journal (kept across reboots) or /run/log/journal (in memory).
journalctl: The command that reads and filters the journal, for example journalctl -u hello or journalctl -p err.
Priority: How serious an entry is, from 0 (emerg) to 7 (debug); 3 is err, 6 is info.
stdout / stderr: A program's two output streams: standard output and standard error. A service's are both sent to journald.
Ring buffer: The kernel's own log in memory; dmesg prints it and journald reads it through /dev/kmsg.
Unit: Something systemd manages, such as the service hello.service.

Fine print

Notes and sources

The journalctl lines on the card are cut: each really starts with the date, time and host name, for example "Oct 25 09:14:02 lab hello[812]: started". On a real machine -p err also lists errors from other services, and -u hello also shows systemd's own lines about the unit, such as "systemd[1]: Started hello.service."
The unit file sets SyslogIdentifier=hello. Without it, journald would tag the lines with the process name, python3.
Python buffers stdout when it is not a terminal, so a service's prints can arrive late or all at once. flush=True (or the PYTHONUNBUFFERED=1 setting) sends each line at once. stderr is line-buffered already.
journald deletes whole archived journal files, not single lines; the figure fades out the oldest entry to stand for that. journalctl --vacuum-time and --vacuum-size delete old logs for good: use them with care.
Many systems, Ubuntu among them, also run rsyslog, which copies the same messages into plain text files such as /var/log/syslog.

Quiz Unlocks after the last step.