PC-BSD needs a better startup system
It's 2015 and we still haven't fixed init systems yet.
- Problem: Duplication of code in the scripts. For example, every script contains code that checks to make sure the script is not disabled. This is best handled by the caller.
- Problem: Startup scripts are disabled by looking for the name of a magic word in the startup script and manually adding it to a global config file. This is clunky and there has to be a better way.
- Problem: Startup waits 30s for wireless to not connect when booting my laptop away from home. I can ^C but shouldn't have to.
Stuff that I should have read before writing this:
- A history of modern init systems
In rc.d, the init(8) daemon executes /etc/rc, which runs the rcorder(8) program for calculating ordering dependencies on the initscripts in /etc/rc.d/.
- Look into systemd
- Look into depinit
- Look into launchd:
Revolves around pure lazy loading of services, with no formal dependency model, instead expecting services to synchronize themselves via IPC throughout the rest of the OS X stack.
- Look into runit:
Fast parallel init, very simple init scripts (it’s just bash, but they’re usually around 3 lines), and easy service management. Services are enabled by creating symlinks, like symlinking sites-available to site-enabled in nginx.
- [Edit March 30] Look into OpenRC and eudev
A set of startup scripts could be seen as a dependency tree in which some scripts run after (branch off) other scripts. In a theoretical startup system, a process could read the startup scripts and generate a tree of startup programs.
Startup system components
The system would include at least three components:
- A general-purpose tree library for the tree relationships.
- A component to map the items in the tree relationship to programs, kernel modules, or anything else that the init system initializes.
- A component to launch the items and track whether they succeed or fail.
- A component to communicate with the console, update the correct "launching foo..." line with "OK" or "Fail", and report status messages in groups.
Any sub-program could theoretically have its own launcher system using the same components. For example, the networking subsystem could run the same launcher against a different directory of scripts. A window manager can use the same launcher system. The startup system could be generalized to a launcher launching launchers.
A tree of parent -> children relationships is generated.
A table of parent, child relationships is generated.
After any program succeeds, the table is queried for any children of the parent.
A text file with a domain-specific language will define "x requires y" relationships between startup programs. This language should be general enough to apply to any such set of relationships.
At boot time, a program reads the DSL to generate the a tree of relationships.
Design of the domain-specific language
- Launchables - For the purpose of the init system, a Launchable is any program or kernel module that the system can run.
- List - Any number of Launchables separated by whitespace and/or commas.
- req - establish a backwards dependency.
- enables / ena - establish a forward dependency
foo req bar
bar will not be started until and unless foo starts.
foo ena bar baz quux
Result is same as:
bar req foo baz req foo quux req foo
Idea: it is proposed that the DSL manage only the matter of dependencies and not whether the programs are intended to run or not.
Rejected idea: Proposal for Keyword: launches
foo launches bar
Result: the caller will launch bar if foo runs successfully
Reason for rejection:
- This can be done in the startup script for foo
- This overlaps with "bar req foo"
Rejected idea: Proposal for Logical operators - () & |
web_log_watcher req httpd | nginx
Result: web_log_watch will wait until either httpd or nginx runs
Reason for rejection:
- It may not be desirable for the system to have the complexity that these operators would add.
- It is unclear what behavior should be expeted from logical operators in other contexts.
- web_log_watch could be added to the startup scripts for both web systems.
Idea: No DSL
Alternate proposal: Kill the DSL. We don't need it. Kill the whole tree-tracking idea and go in the opposite direction. Every subsystem is expected to know its own subsystems and run them from its own startup scripts. Every script runs its subsystems in GNU Parallel or something similar.
Problem: By going back to shell scripts, we lose the ability for management utilities to have an overview of the whole startup system.
We would need:
- A component for launching programs in parallel.
- A component for observing the parallelized programs and reporting success and failure.
- A component to manage communications between the the observer and the console.
- A console/syslog management component that can update the console, ensure the logs look nice, and report the boot status to userspace programs during run time. Call it bootlogd.
Let's start over.
Basic goals of a startup system:
- Procedure. Start programs in the correct order.
- Parallelization. Start programs that have no unmet dependencies in parallel.
- Observability. Communicate program status to an observer.
Design goals of the runner:
- Simplicity. Each aspect of the startup system should have as few details as possible.
- Speed. Minimize waiting times.
- Reuse. The same startup system should be usable as a launcher for any complicated piece of software such as a window manager.
Design goals of the configuration:
- Transparent. It should be trivial for the user to read the startup configurations and learn what processes spawn other processes.
- Customizable. It should be trivial to make changes to the startup routine.
- Extensible. If the user wants to make an optional custom startup routine (a new runlevel) while leaving the old routine in place, it should be trivial to do so.
- Organizable. It should be possible to group similar startup scripts together so a user who does not know the name of a startup script can more easily find it with similar scripts.
Basic goals of the observer:
- Collect status and output from running and failed startup items
- Report status to the console, a logger, a UI, etc
Design goals of the observer:
- Portability. It does not know whether it is reporting status to a console, a logger, a UI, etc. It may be an interface and little else.
Thoughts on procedure
If a startup program depends upon a service provided by another startup program, the question of whether the previous program has run or not can be determined by the availability of the needed service. While this is a simple concept, it presents multiple problems.
- An observer cannot map out these dependencies ahead of time. A hypothetical graphical configuration viewer would not be able to tell that the dependency exists.
- Assuming that the needed resource is a file, we cannot distinguish between an old data file that is not being updated and an old file from a previous run that we should wait for another startup script to update.
- If services are checked directly in this manner, the observer would not
know why a required service is not ready and would not know which script
runs the service.
- Should the observer/shell respond to failures by waiting 5s and trying again?
- Should this be limited to a certain class of failures?
It should be possible to state that a startup item depends upon multiple items.
It should be possible to state that a startup item would prefer to run after another item, but does not depend upon it and will be run if the other item fails or does not exist.
Where is the best place to put dependencies? In a separate tree file? Inside the file as a pragma? What if the file is not a shell script? See thought on configuration.
Thoughts on parallelization
Linux used to use numbered startup files like S45foo, S50bar. They would run sequentially in the order of the number in the file name. Startup files with the same number could be parallelized but this design would still lead to bottlenecks and slower-than-optimal startup compared to a dependency-based startup system.
- The system administrator must know which startup items are safe to run simultaneously to properly configure their startup. Most won't know their system that well.
- While an init system could theoretically run all items of the same number in parallel, it would not be able to proceed to a higher numbered item with no remaining dependencies until all items are completed.
Idea 1: On boot: carry information across a pipe fid
- Start the console display utility and collect its pipe fid
- Start the observer and give it the pipe fid of the display utility
- Start the startup tool and give it the pipe fid of the observer.
The observer runs the program, watches both of its pipes, prints the program's output to console and log, and reports OK/ERR status. Error status may based on exit status, whether stderr was used, whether the word "ERROR" appeared in stdout or stderr, or some other protocol.
What information does the observer need from the program?
- Program status EXIT_SUCCESS or not
- Program stdout
- Program stderr
This describes a shell in the most basic sense. There should already be this capacity for observation and control in the standard libraries.
PROBLEM: What if the program wants to say it is waiting for something or in an in-between state?
Output wrapper / logging structure
An internal logging system may be used to collect program output so that the program can post lines of output of the same program near each other on the console, wrap up the outputs of OK programs into one line on close, and still record full program output to the system log.
Information to log:
- process id (the observer keeps the name in a table)
- process path and arguments
- file descriptor (stderr, stdout, etc)
- Sequence counter (line number) per pid and fd
- Error level -- for lines that begin with ERROR or WARN in allcaps
- Data byte size if needed
The streams will be "tee"'d with one target sending them to disk immediately in an appropriate text format, and another sending them upstream to whatever manages the user interface.
It should be possible and easy to send the logs to a remote server for locations that do remote logging.
BUG: If the log is written to disk with the metadata, we may have reinvented systemd's unpopular binary logs. Workaround: send the outs to an immediate writing process that converts it all to text.
Idea: Boolean startup options
- Start up Q if any of (A, B, C) started successfully
- Start up P if all of (D, E, F) started successfully
Problem: If we are going to include && and || and () logic, there will be demands to add support for scripting features until we have reinvented shell scripting, so we can just use shell for this.
How is startup success measured?
- Error code == 0
- Stderr is empty
- Presence of a file or service
- Presence of an environment variable