Lesson 10 · The system

Services with systemd

A service is a program that runs in the background with no window: a web server, a database, a backup job. On most Linux distributions the program that runs them is systemd, process 1. Write a unit file for a small Python script, then follow it through systemctl daemon-reload, start and status, see its output land in the journal, crash it with SIGKILL and watch systemd restart it, enable it for every boot, and stop it cleanly.

11 min10 steps5 questionsIntermediate

Fig 10 · Interactive

Loading figure…

or ← →

Check yourself

5 quick questions

Pick an answer to see why. Your score saves with your progress.

01You have just saved a new unit file in /etc/systemd/system. What should you run before systemctl start can use it?

systemd does not notice new or changed unit files by itself. daemon-reload makes it read every unit file again; then start can find the new service. No reboot is needed.

02What is the difference between systemctl start hello and systemctl enable hello?

enable reads the [Install] section and creates a link in multi-user.target.wants, so systemd starts it at boot. It does not start it now; enable --now does both.

03hello.service has Restart=on-failure. Its process is ended with kill -9 (SIGKILL). What does systemd do?

Being killed by SIGKILL counts as a failure, so on-failure restarts it. A restart is always a new process, so it gets a new PID (here 1234 becomes 1240).

04A service prints to standard output. Where can you read those lines?

By default systemd connects a service's output to the journal. journalctl -u hello shows only the lines from that unit; systemctl status shows the last few as well.

05You run sudo systemctl stop hello. Why does Restart=on-failure not bring it back?

stop sends SIGTERM (signal 15), and SIGKILL after 90 seconds if needed. When the process ends because systemd stopped it, that is not a failure, so nothing restarts. The service stays enabled for the next boot.

Key terms

Glossary

Service: A program that runs in the background, with no window, often for as long as the computer is on: a web server, a database, a log collector.
systemd: The service manager on most Linux distributions. It runs as process 1 and starts, watches, restarts and stops services.
PID: Process ID: the number the kernel gives each running process. systemd is PID 1; a restarted service gets a new PID.
Unit file: A small text file that describes something systemd manages. A .service unit says what command to run (ExecStart) and when to restart it.
systemctl: The command for talking to systemd: daemon-reload, start, stop, status, enable and more.
Journal: The system log kept by systemd-journald. Services' output goes there; journalctl reads it.
Signal: A short message the kernel delivers to a process. SIGTERM (15) asks it to end; SIGKILL (9) ends it at once.
Target: A group of units reached together. multi-user.target is the normal running system; WantedBy=multi-user.target means: start me at boot.

Fine print

Notes and sources

The PIDs 1234 and 1240 are examples. Your machine picks its own numbers, and the journal lines start with the date and the computer's name, cut to "…" here.
ExecStart may name a program without its folder, like python3: systemd looks it up in a fixed list of folders such as /usr/bin. Writing the full path, /usr/bin/python3, works everywhere too.
print(..., flush=True) sends each line out at once. Without it Python keeps short lines in a buffer when its output is not a terminal, and the journal would show them late.
journalctl may need sudo: members of the adm or systemd-journal groups can read the system journal without it. systemctl status also prints the last few journal lines.
kill -9 is a last resort: the process cannot save its work or tidy up. To restart a service on purpose, use sudo systemctl restart hello instead.
Restart=on-failure does not restart after SIGHUP, SIGINT, SIGTERM or SIGPIPE, the signals a clean shutdown uses. After SIGKILL, as here, it does.

Quiz Unlocks after the last step.