What a log is
A log is a diary a program keeps while it runs: one line per event, each with a time. When something breaks, the logs tell you what happened and when. On most Linux systems, one service collects them all.
Our example is a three-line Python program, run by systemd as hello.service. To log, it simply prints: line 2 writes "started" to standard output (stdout). A service has no screen, so where does that line go?
Into the journal
systemd connects every service's stdout and stderr to systemd-journald, the logging service. journald stamps it with facts the program cannot fake: the time, the unit hello.service and the process ID 812. It also gives it a priority, here 6: info, the default for stdout.
journald appends the entry to the journal: binary files in /var/log/journal, kept across reboots. If that folder does not exist, it keeps them in /run/log/journal, in memory, and they are lost at reboot.
Every entry has a priority from 0 (emerg) to 7 (debug); 3 is err, an error. Line 3 writes to stderr and starts with <3>, so journald files "no config" as an error and strips the prefix.
The kernel keeps its own log in memory, the ring buffer: drivers, disks, USB. journald reads it from /dev/kmsg, so a USB stick being plugged in lands in the same journal. The dmesg command prints the ring buffer directly.
Reading the journal
To read logs, run journalctl. It opens the journal files itself and prints entries as text, oldest first. journalctl -u hello keeps only lines from our unit: both of its lines, and not the kernel's.
journalctl -p err shows priority err and anything more urgent (0 to 3), from every unit and the kernel. Here that is one line: "no config". Add -f to follow new lines as they arrive, or -b for this boot only.
Logs keep growing, so journald limits them. By default the journal may use at most 10% of its filesystem, capped at 4 GiB; past that, the oldest archived files are deleted first. journalctl --disk-usage shows how much it uses.
What happens
;; step 1/9 · what log one line per event, each with a time who which program wrote it what the message how bad a priority, from emergency to debug collector systemd-journald (on systems that run systemd) reader journalctl
The journalctl lines on the card are cut: each really starts with the date, time and host name, for example "Oct 25 09:14:02 lab hello[812]: started". On a real machine -p err also lists errors from other services, and -u hello also shows systemd's own lines about the unit, such as "systemd[1]: Started hello.service." The unit file sets SyslogIdentifier=hello. Without it, journald would tag the lines with the process name, python3. Python buffers stdout when it is not a terminal, so a service's prints can arrive late or all at once. flush=True (or the PYTHONUNBUFFERED=1 setting) sends each line at once. stderr is line-buffered already.