Layers of abstraction
At the hardware layer we have native bytecode created from amd64 through a translator. We operate on registers, have to keep track of what data is in what register, and if the data has any meaning there is no way for the computer to know about it.
At the low level of abstraction we operate on bytes and sets of bytes. OS, POSIX, and standard library interfaces are written in C. One might consider high level assembler with strings and matrices to be at this level. There are multiple data definition formats to define how data is laid out in memory: C header files, SQL data definitions, a custom format for each language.
At the intermediate level of abstraction we define classes to associate the data structures with the methods that work on the data. We have templates, interfaces, abstracts, generics to allow code reuse.
At higher levels everything is abstract. We create anonymous objects on the fly and tell the computer to run operations on members that we hope will have been defined by runtime.
There are similar layers of abstraction for methods, functions, subroutines. At the hardware layer a method is just a series of instructions that you jump to. Low-level programming allows you to describe the inputs and outputs: this function takes a byte pointer and returns a byte pointer. At the intermediate level of abstraction you can define a template function that allows the same code to be reused on different data types. At higher levels of abstraction all functions do that by default.
Abstracting operating environments
We have had all of these advances in abstraction in the programming area but our shells are still stuck in the early 1990s. Programs are written to read bytes or lines at a time. Streams of characters are passed between programs with no information about what the characters mean. Each program in the sequence must re-parse the stream.
Consider this sequence.
sh$ ls -l | bar > baz
Presently this outputs a chunk of text, and the bar program needs to know what to do with the text. Imagine instead that ls -l generates a table or recordset of files and their attributes, and what it outputs is context-dependent.
- If the receiver is a text terminal, the shell uses a formatter provided with ls to generate a text table compatible with today's ls -l.
- If the receiver is a hypertext terminal running in a windowed environment, the shell uses a formatter to send the data and metadata to the terminal in a given standard format that the terminal will use to produce an interactive display of files.
- If the receiver is a .csv file, the shell automatically uses the standard table-to-CSV converter from the operating system's libcsv.
- If the receiver is a program that requests input of a given class, the shell attempts to present the input as that class.
In a high-level shell design,
- Programs are written to read more abstract sequences of objects. The implementation of the reader is chosen at runtime. Programs will have metadata exposing their expected input types.
- Programs may pass both data and metadata. There will likely be a standard for a data+metadata format, and all standard readers will automatically recognize it.
- Programs may pass pre-parsed chunks of data to skip the step of parsing by subsequent programs. This may be as simple as copying a chunk of memory from one program to another on the same computer. The high level program code will be the same whether the receiving program runs on the same computer (memory copy) or sends the data over the network.