A new service manager for Linux distributions

The last piece of a full alternative to systemd

(Update: the project has found a sponsor! Expect more news in this space in the months to come.)

Service managers are a fundamental part of Linux distributions — and, more generally, of Unix systems. Unfortunately, for such a critical piece of infrastructure, the offer is lacking. There is no service manager that performs all the duties users can expect from it while being flexible enough to address multiple use cases, including dynamic network management, usability in containers, and usability in musl-based distributions.

I am working on a service manager that aims to solve the issues in that space. In particular, it aims to be a suitable replacement for OpenRC, which currently powers the Alpine Linux distribution, but which does not provide all the features that Alpine needs. Due to its requirements for small and clean software, and its increasing prevalence in container images, Alpine is a typical case where this new service manager would be useful; but I expect the benefits of the project to extend well beyond Alpine.

This is a long-term endeavour that requires a precise understanding of the specs, a certain level of skill and experience in software design, and copious amounts of coding time; I don't think it would necessarily benefit from a larger team, at least on the coding side, because it's still at a level of complexity that can be fully understood by one person — although, as always, I value external inputs such as technical discussions and user experience returns immensely. But it cannot be completed in any reasonable time frame if I am only working on it in my free time. For these reasons, I am looking for funding, in order to be able to work on it full-time; I believe the benefits to corporations, especially ones that use Alpine Linux or similar container-suitable distributions, will far outweigh the expense. This document aims to explain the details of the project.

Table of contents

Definitions

Not much formalization has been done around terminology such as service manager and init system, so these expressions are often used without a clear understanding of their semantics. Let's define these terms first.

An init system is the set of software components that is launched by the kernel at boot time, and whose job is to bring the machine to its fully operational state, with all its configuration done and all its services running.

As explained in this video, an init system is made of 4 parts:

  1. the /sbin/init program
  2. the pid 1 program
  3. a process supervisor system
  4. a service manager.

These parts are conceptually distinct, but should be able to work together. For instance, without going into details, in order to offer strong supervision, the process supervisor system needs to be tied to the pid 1 program.

Most of the time, several of these parts are implemented in the same software suite, which is why we generally talk about init systems. sysvinit, GNU/Linux's traditional init system, is /sbin/init and pid 1 at the same time, and also includes a very rudimentary process supervisor. (But it does not include a service manager, which is why it is often paired with sysv-rc or OpenRC.) systemd, Upstart and MacOS's launchd are "complete" init systems, implementing all 4 parts.

/sbin/init and pid 1 are focused on the early boot of machine and are out of scope of this document. Process supervisor systems are in charge of managing long-lived processess (daemons): making sure they are always started with the exact same environment, restarting them when they die under certain conditions, redirecting their log output to the proper places, providing an interface to send them signals without needing to know their pid, etc. They are also, strictly speaking, out of scope for this document, although we will have to mention them because they need to closely interact with the service manager.

The service manager is the component that brings the machine from the state where nothing is running yet, and nothing is configured, to the state where all the filesystems are mounted, the network interfaces are up, everything that needs to be initialized has been initialized, and all the services configured by the administrator are up and running. Conversely, it must also handle state changes when required by the admin: stop a service, start another, reconfigure another. In particular, it must be able to reliably stop all the services in an orderly fashion when the administrator requires a shutdown.

The ideal service manager

A good service manager must follow a certain number of constraints that apply to its design, its implementation, or the features it provides.

  • The service manager is system infrastructure software, meaning that it is run at a very low level, and the whole machine operation relies on it. This means that:
    • It must be extremely reliable. This is achieved by having clear and complete semantics, doing the minimum amount of operations, having as few run-time dependencies as possible, and making sure the code remains fully understandable by the dev team at all times and never grows out of control.
    • It must be as lightweight as possible. Resources are meant to be used by applications, not system software; heavyweight system software is unusable on old computers and embedded devices, and wasteful in VMs and containers.
    • It should be easy to bootstrap. Since it is a critical piece of a distribution's software offer, and one of the first packages to be built when porting a distribution to another architecture or another platform, the service manager should have as few build-time dependencies as possible. Building and installing it should not be an hours-long chore.
  • The service manager must always start and stop services in the same environment, no matter whether it's invoked at boot time or by an administrator command.
  • The service manager must tie-in with a process supervision system to make daemon management and monitoring easier.
  • The service manager should guarantee system bootability. It should provide a mechanism that vets changes to a boot-time set of services before the reboot; a mechanism that says either "change accepted; if you reboot now, the machine will not get stuck in an unrecoverable state, you will get to the state that you want" or "change invalid, rejected".
  • The service manager must be parallel: it must be able to start independent services at the same time. That means it needs to understand and maintain a set of dependencies between all the services it knows about. Traditionally, this has not received much attention, because proper dependency management and parallelization is difficult; and historical service managers have just started services sequentially, following a topologically sorted list. However, there are significant gains to be had with a really parallel service manager, especially in the presence of heavy services that take a long time to start and needlessly monopolize the CPU or needlessly idle.
  • The service manager must support readiness notification from daemons it starts. It ensures that services are only started once their dependencies not only have been spawned, but are also ready (e.g. make sure that a daemon has actually created, and is listening to, a socket, before running a client that connects to it).
  • The service manager must interface with external events, to allow users to define services that dynamically start and stop when certain conditions are met, without needing to encode every possible condition in the service manager configuration. A typical use of this is network daemons listening on a specific interface: they must be started when the interface comes up, and stopped when the interface goes down. Managing the state of network interfaces in a generic way is too complex a problem to be handled in a service manager, and is best delegated to a network manager written for this explicit purpose; and when the network manager detects interface state changes, it sends events to the service manager, which reacts accordingly.
  • The service manager should support declarative service files, which experience has shown are more intuitive and less error-prone for users. Writing script snippets is unavoidable, but those snippets should be as small as possible, and most common service functionality (setting the user/group a daemon will run as, setting its environment, etc.) should be made available via a declarative syntax that will minimize errors.
  • The service manager should be usable in containers as well as in virtual machines and real iron. Increasingly, entire systems with entire service hierarchies are run in containers, and having tools to start container contents in an orderly fashion would be an improvement to the quality of numerous container images. This is especially true for the Alpine distribution, which often serves as a base for container images.

The state of the art

The dichotomy

Over the years, I have done a lot of research on init systems and service managers; they usually fall into one of two categories.

  • Traditional service starters. They're more starters than managers, because they don't have a fully fleshed out dependency model — they're only used to start services at boot time and stop them at shutdown time, with the occasional extra start. All of them are sequential and favor simplicity; when parallelism is added, it's often as an afterthought and it greatly increases code complexity. They are generally not designed to work with a supervision system: if a daemon gets killed, they will not react.
  • Integrated init systems. Those were written as attempts to cover the flaws of traditional service starters, and provide all the dependency management, parallelism, integration with supervision, and other features that are necessary to implement a complete init system. Unfortunately, they're tightly integrated (as opposed to modular), complex beasts, and embed a lot of policy within their mechanism, which manifests as being difficult to operate with other software and wanting to overencompass all the administration tasks on a machine rather than follow the "one job, one tool" philosophy.

Typical examples of traditional service starters on Linux include sysv-rc and OpenRC.

The prime example of an integrated init system, on Linux, is obviously systemd; there are others, less invasive and less prevalent, like Upstart (a Ubuntu precursor to systemd). On MacOS, launchd falls into that category.

None of these options are satisfactory. Distributions make do with what they have: some embrace systemd — and in doing so abandon all hope of ever getting rid of it, or of becoming a suitable option for systems with serious requirements of low resource usage or code auditability; others resist the pull, and scrape by, using OpenRC or similar software, and working around the lack of features. Some, like Void Linux, have even elected not to use a service manager at all. (They use a process supervision system, which is good, but then have to script their entire init procedure, which is one of their major pain points.)

s6-rc

I have been interested in the init system space for a long time, and have written my own process supervisor, s6, and a companion package, s6-linux-init, that makes it work as a /sbin/init and pid 1 as well. In order for s6 to be a complete init system, a service manager was needed; this was going to be a much more complex endeavour, so I wrote a small one, s6-rc, as a prototype — to see how far it would go, and to get an idea of the kind of problems that emerge when designing a service manager.

For what it's worth, I consider the attempt a success. s6-rc's current version is 0.5.2.1, and it's a service manager engine that works very well in a lot of cases. It is smaller than OpenRC while having more features, and faster than systemd while remaining strict with its launching order. It was even adopted by some distributions, namely Obarun and Artix.

However, it is only a prototype, and lacks crucial features before it can satisfactorily compete with established players: in particular, it is still lacking dynamic event management, which makes it unsuitable for integration with a network manager, and it lacks a friendly user interface based on declarative service files, which several distributions have clearly signalled they require before considering s6-rc for adoption as their chosen service manager.

So, as of today, there is no service manager that fulfills all the needs of a distribution. It is a "pick your poison" situation.

A comparison table

To visualize that situation, and the path forward, here is a comparison table, showing how OpenRC (as an example of traditional service launcher), systemd (as an example of integrated init system), and s6-rc (the current prototype version) measure up against each other. We list all the requirements of an ideal service manager, which we detailed above.

Requirement OpenRC systemd s6-rc 0.5.2.1
Extremely reliable? Not really. OpenRC's code paths are more convoluted than they need to be. No. systemd is known for its complexity and unpredictability. Yes. s6-rc is as simple and straightforward as a service manager can be.
Lightweight? Reasonably. OpenRC consumes a little more resources than would strictly be needed. No. systemd uses way too many resources for system infrastructure software. Yes. s6-rc was designed to be as lightweight as possible.
Easy to bootstrap? Yes. OpenRC is C and shell. Yes. systemd is C. However, it only supports the GNU libc, which may be a problem for musl-based distributions. Yes. s6-rc is C.
Reproducible service environment? No. OpenRC starts services as scions of the openrc or service command. Yes. systemd always spawns the services from pid 1. Yes. s6-rc always spawns the services from its supervision tree.
Working with a supervision system? Not really. OpenRC can spawn rudimentary supervisors for its daemons, but the supervisors are not supervised themselves and offer few benefits. Yes. systemd provides process supervision itself. Yes. s6-rc works with the s6 process supervision system.
Guarantees system bootability? No. A configuration change can make the system unbootable. No. A configuration change can make the system fail to boot — or even to shut down. Yes. If a configuration applies successfully, the system will boot.
Parallel? Not really. OpenRC has a parallel mode that has never worked properly and is actively discouraged by its developers. Yes, and too much: systemd cheats and ignores some dependencies in order to start services faster. It impacts its reliability and predictability. Yes. s6-rc starts services exactly when it's safe to do so.
Supports readiness notification? No. OpenRC's startup sequence is full of race conditions. Yes, via the sd_notify protocol. Yes, via the s6 notification protocol.
Interfaces with external events? No, which is a main reason why Alpine aims to replace it. systemd supports dynamic events internally, and provides its own network manager, but does not interface with other software. No. The 0.5.2.1 version of s6-rc uses a static service database and does not accommodate external events.
Declarative service files? Simple services can be configured purely declaratively. For complex cases, code snippets are still needed. The unit files are as declarative as can be. Service configuration files are scripts, no syntactic sugar is provided to the user.
Usable in containers? In theory yes, but OpenRC embeds a lot of policy that makes it difficult to adapt and more of an annoyance than the benefit is worth. As with everything systemd, the features are there, as long as you fully opt into the model; here, the host has to run systemd. Plus, systemd does not support musl, and is too big to be a good fit for containers. s6-rc is small and separates mechanism from policy, so it's possible to write container-specific service sets.

The plan

As illustrated, despite being a 3-month 1-person project, the s6-rc prototype fares better than both OpenRC and systemd on the requirements chart. It is the closest we have to our ideal. Unfortunately, it is still not suitable as a flexible, user-friendly service manager that could be used in any distribution; for that, those two red squares on the bottom right need to turn green. And that represents a large amount of work.

Putting in the work, though, would actually lead to a good piece of software, because it's just a question of implementation. It is still easier to achieve all the goals of an ideal service manager by using the s6-rc prototype as a basis for the final software than it is to tweak and modify OpenRC or systemd, because the flaws in these are design flaws, and cannot be fixed by adding more code or even refactoring.

So, my proposition is to work on a real, production-quality implementation of s6-rc, keeping the sound design but adding the features it needs:

  • A client-server model where the server, a long-lived process, maintains both the current state and the wanted state of the machine, performing the needed transitions as they become available.
  • Dynamic external event management by the server, with a specialized client program that can be invoked by external entities, such as a network manager, to notify the server with an event.
  • User-friendly administration commands that hide internal details and follow the model of openrc, service or systemctl. Users will get a command interface they are familiar with.
  • Declarative service files. The model of unit files, where users can just write key=value lines instead of scripts, has been a resounding success for systemd — and it is also good end-user design (provided the parser correctly validates the files). And it can be done as a separate user interface layer, running on top of the more minimalistic engine.

The goal is known, the path is clear, and the milestones are laid out. But it is a serious project, and I would like to see it completed before I die — which is not likely to happen if I can only spend on it the meager amounts of free time I can devote to writing free software while needing to sustain myself writing other software — likely much less pleasant, and much less free.

Which is why I am looking for funding, in order to be able to work on this full-time. My estimate is that it needs about one year of work, and should give really good returns on investment. Among the probable benefits:

  • Faster, more reliable Alpine boots, and faster and more reliable worldwide boots when other distributions start adopting s6-rc
  • Clean service ordering in containers
  • Easier administration of network equipment with services that depend on dynamic interface states
  • Easy integration with configuration managers such as Chef or Puppet
  • Ultimately, the emergence of an efficient, reliable and modular software ecosystem as a real alternative to systemd

But the real benefit, of course, is that you will be able to say that all of this happened thanks to you.

The developer

My name is Laurent Bercot, also known as Ska or skarnet in some circles. I have a computer science degree, and have been working with Linux, and participating in the Free Software movement, since 1999.

I have studied supervision systems and low-level Linux userspace for two decades. Init systems are my domain of expertise; I honestly think I am one of the best candidates, if not the best, to write a service manager that can address all the current needs for good and finally provide a credible alternative, feature-wise, to integrated init systems.

I have written C code for about as long as well. My users can vouch that I am in the habit of delivering quality code, with extensive documentation. Free software I have written is available on the skarnet.org site; the centerpiece is the s6 process supervision system, which manages a significant fraction of all Docker containers in use today.

I am involved in several Free Software communities that have formed around small, compact software, such as the musl libc, the Alpine Linux distribution, or the busybox project.

I have worked as a freelance C/Linux expert software designer and developer since 2015, and would love nothing more than doing my job on projects I care about most and am at my best with.