Tang's DW

Optimizable Malleability

The life of a productive environment may look like:

1. High malleability - you can do almost anything, with few restrictions.
2. Exploration - certain workflows are found to be better than others.
3. Optimization - simplified interfaces increase productivity while leaving out lesser-used features.
4. Stagnation - The existence of a standard way of doing things discourages the development of alternative ideas.
5. Replacement - Somebody invests a new system that replaces the entire old system.

Examples of malleable systems and optimized systems:

• Assembly language lets you do anything, leading to goto spaghetti. C gives you formal looping and function call patterns along with a subset of assembly's features. Higher level languages have additional formal implementations of common programming patterns and lack some of C's features to prevent entire categories of bugs. Driver writers and OS programmers still need the lower-level features, but 90% of programming can be done without them.
• Text files allow you to store data in any format. Every Unix tool invented its own text file format. INI, CSV, JSON, and XML are types of text files that are optimized toward storing certain types of information.
• Tail call recursion is an optimizable subset of recursion.
• Excel lets you freely place data without structure. Access requires you to define the structure first.
• Static typing versus dynamic typing. A language could be dynamic by default with optional static typing and optional type constraints. In this case the optimized system (static typing) came first and the malleable system came later.

Stagnated systems and their replacements:

• Why aren't webforums based on Usenet/NNTP? The environment of HTML+SQL allowed for the development of alternatives at a lower cost.
• Why don't new languages use object files? The burden of compatibility is not worth it.
• Why aren't most programs made of shell scripts? Fewer features, security concerns, etc.

Reconsidering the notion of a "program"

Let us view the entire operating environment as a set of mechanisms for directing input to a function.

Consider the command line shell.

• The user provides the program with representations of the function's arguments. The arguments are not passed directly, but you provide a string that represents the argument. Code inside the program converts the string to a filename or URL, parses the data, and provides that as the argument to the function.
• The program provides standard input and output methods, using pipes
• The program may allow the user to set the initial state of the program, using switches

We could say that a program is a wrapper that sets state, has the standard input and output, and provides access to the desired function(s). Let us consider a future operating environment that separates these tasks.

• The future system could automate the generation of wrapper code.
• The future system may use dynamic type casting, late binding, just-in-time compilation, etc to rewrite part of the program based on the user's input.
• The future system might allow the caller to directly set global variables / class members by name, doing away with the need to parse switches.
• Inputs and outputs may be something other than raw byte streams. The pipes may have a class type. The system may provide a standard iterator for a given type and handle casting automatically. The standard Unix tools may be rewritten to work on interfaces.

Incomplete classes and comprehensive dynamic typing

Consider the evolution of classes.

• 1970s - C - A struct is, basically, a way to arrange data
• 1980s - C++ - A class is, basically, a way to arrange functions around the data they operate on
• 1990s - C#/Java - An interface is, basically, a definition for a set of functions that we will expect any given type to implement.

The struct, class, and interface are all key/value pairs.

Consider the function:

sub doStuff(x){ return x + 1 }

In the "message passing" concept we might consider x:Dynamic and expect the runtime to check whether the given x has a ._plus() method at call time. (A more complex system could check for this when the object to be passed as x is known.) In the "interface" concept we might check that x implements the INumeric interface which includes a ._plus() function. "INumeric" is a way of saying that a dynamic type will implement a standard set of methods, what C# calls a contract.

An interface might be seen as an incomplete data type. An incomplete data type might be seen as a type of filter. A data type filter could describe the partially resolved value of a dynamic type, or could be used to implement very strict typing.

interface IPercent:UInt8 {
  _value = 0..100 // Value is between 0 and 100 inclusive 
} 

interface asdf {
  foo:String and len = 0..8 // .foo will be a string whose length is > 8
  
  // let's propose something outrageous... 
  baz: fnord(_) between &("camel"), dronf("llama") and & > 3 
  // * we will have a .baz member
  // * the namespace will also include functions fnord() and dronf()
  // * the result of fnord(baz) has a comparator that puts its result
  // between the results of fnord("camel") and dronf("llama"),
  // * and the value of fnord(baz) will be greater than 3
  // * or else the program throws a type mismatch exception,
  // preferably at compile time 
} 

This would be of little use to a programmer, but might be useful to a compiler, runtime, or IDE.

// as written by the programmer
function doSomething(futz): 

// as interpreted internally 
function doSomething(futz:{
  foo:String = "foo" // string with a known value 
  bar():Int // an undefined function that returns int
  } 
)

The same sort of filter or incomplete class can also be used as a search object.

interface SalariesOver30000:Employee { 
  salary:Numeric >30000   
}

select SalariesOver30000 from Employees // pseudo-sql

Function constraints

A compiler could theoretically determine that it is impossible for a function to do certain things:

• a given function may not modify its inputs
• the function may not make any asynchronous calls
• the function may not open any additional inputs or outputs
• the function's inputs may have values known at compile time or runtime

A compiler could potentially use this information to optimize the function, possibly running it at compile time if enough information is known.

My TODO list for language design is to look into:

• intermediate languages
• continuation-passing-style (CPS) interpreters
• A-normal form (ANF) representations
• static single assignment (SSA) representations
• Haskell's universal machine and Higher Order Abstract Syntax
• read: static single assignment for functional programmers
• ANTLR

Has-A pattern:

class MyClass { 
 FooType foo;
 BarType bar;
}

• each MyClass allocates space for a FooType and a BarType.
• foo and bar establish their own namespaces.

Inheritance:

class MyClass extends FooType, BarType { }

• each MyClass allocates space for a FooType and a BarType.
• The namespaces for FooType and BarType are incorporated into the MyClass namespace.

So what's the problem? Something in FooType or BarType might overlap.

General outline of a solution:

• All namespace conflicts must be resolved by the programmer.

Considering specific examples:

1. Both FooType and BarType have a .x property?

• Solution 1: Forced upcasting. All attempts to access MyClass.x must be upcasted to the parent. All parent methods work on their own .x property.

2. Both FooType and BarType have a .fooMethod()?

• Solution 1: Forced upcasting. The compiler should warn if the programmer fails to resolve the conflict, and throw an error if the programmer tries to run MyClass.fooMethod() without overriding it. This can cause problems (see case 4).

3. MyClass has its own .x property?

• Solution 1: Override parent access methods. If the property is interface-compatible with the parent's .x property, the FooType/BarType methods will run on MyClass.x instead of their own .x properties. If the property is not interface-compatible with the parent's .x, the compiler should throw an error and refuse to compile. PROBLEM: FooType and BarType may intend for their .x property to mean different incompatible things or to hold separate pieces of information. Sharing the same piece of data will introduce bugs.
• Solution 2: Do not override parent access methods. Parents continue to operate on their own .x properties.

4. MyClass has its own .fooMethod()? Theoretically, this should override the FooType/BarType methods. However, it introduces a double-call problem.

• Solution 1: Override parent methods. This introduces a double-call problem. If both parents have a method called every second that calls their own fooMethod(), this causes the overridden fooMethod() to be called twice per second instead of once per second.
• Solution 2: Do not override parent methods. Parent calls to fooMethod() will call the parent version of fooMethod().

Conclusions

More thought needs to go into this.

• Multiple inheritance is fine until you have a namespace collision, and then you're in trouble.
• If you have the parent classes share data when both have a .x property, then you have the problems that each might treat .x as a different conceptual thing and that conflicting actions on the same .x may introduce race conditions that do not exist when each class uses its own data.
• If you do not have the parent classes share data, then you have multiple copies of what might be the same conceptual thing, updates to one do not affect the other, and the state of ".x" is unclear.
• If the child class .x overrides the property for both parent classes, then you risk the parent methods modifying the data in ways you did not intend.
• If the child's .foo() overrides the method for both parent classes, then you might get a double-call problem when methods in both parent classes call their overridden .foo() method.
• If the child's .foo() does not override the parent method, then you lose the ability to extend the parent class by having its .foo() method do something different.
• The programmer may want to resolve namespace collisions in a different way for different properties and methods in the same class.

Consider that a class in c++ or java defines both an implementation and an interface.

Consider the similarities in these three examples:

// Example 1 
interface IFoo {
 // interface
 doStuff();

 // implementation not defined
}

// Example 2
class Foo {

 // implementation
 int x;
 string y;

 //interface
 doStuff(){...}
}

// Example 3
// implementation 
struct foo {
 int x;
 string y;
}

void foo_doStuff(); // interface

An ordinary compiler might be expected to convert lower-level code to a low-level implementation immediately upon reading it. The programmer has defined exactly how the code is to be implemented, and there is no apparent need for higher level considerations.

A hypothetical high-level compiler might first convert as much low level code as possible into a higher-level intermediary language.

• class Foo is automatically built from struct foo and is automatically populated by the foo_() functions that operate on a struct foo.
• interface IFoo is automatically built from the internal representation of class Foo
• This high level representation is saved to an object file and can be accessed by any programming language that can load libraries.

Additional thoughts:

• An autodoc tool can produce documentation from the resulting high-level objects, especially if the high level object includes comments or links back to the original source code.
• An IPC mechanism can send the object definition to the receiving side if the receiving side does not have the structure defined.
• The compiler can detect classes that are binary-compatible with one another and allow them to be casted without harm.

Also, it should be possible for a sufficiently smart compiler to recognize that some for-loops are examples of an iteration. These can be interpreted upwards to a high-level description of an iteration like foreach x in myArray{...} for which the developer has provided a low-level implementation for (i=0; i<length; i++){...}

I cannot think of any benefit to doing this, but it could potentially be done.

pattern for resolution of an abstraction to a meaningful value

A = an abstract something

function f(A) {
  if(A == "FOO")
    return FOO
  if (A == "BAR")
    return BAR 
  if (A matches /[A-Za-z0-9]+/)
    return TOKEN_ALNUM
  if (A has method "next")
    return I_SEQUENCE
  if ( not (A contains meat) )
    return VEGETARIAN

  return DEFAULT 
}

This can be generalized to a function taking input:

   f(A, DEFAULT, array of [f_comparator -> f_resolver])

Possible error conditions:

• All comparators fail. Solution: force caller to provide a DEFAULT value, have the caller handle it.
• Multiple comparators match. Possible solutions:
  1. Use the first matching comparator.
  2. Give different values to comparators. Use the comparator with the highest matching score.
  3. (combination of 1 and 2) Sort the comparator->resolver pairs by comparator value before passing them in.
  4. Throw an exception (nooooooo!)
  5. [Edit Aug 2] Perform ALL of the matching operations. Assume the function is a decorator.
  6. [Edit Aug 2] Perform all of the matching operations and return a set of results. Assume the function returns results, plural, rather than one result.

Possible optimizations:

• Inline the function calls.

These are not particularly new ideas. All of this has to be well tread ground.

[Edit Aug 2] Thoughts on "inline the function calls"

The comparator functions may be very simple and easily optimizable in theory. My preference is to write code to pass them in as a set and then let the compiler somehow produce code that has the simple comparison functions inlined as in the original example. This might be possible in theory if the functions are defined at compile time. The optimizer would need to be smart enough to unroll the loop and examine its contents and understand them.

If the functions are defined at run time, the optimizer would need to be part of the run time environment. It seems that what I want is the ability to arbitrarily optimize away function calls at runtime.

PC-BSD needs a better startup system

It's 2015 and we still haven't fixed init systems yet.

• Problem: Duplication of code in the scripts. For example, every script contains code that checks to make sure the script is not disabled. This is best handled by the caller.
• Problem: Startup scripts are disabled by looking for the name of a magic word in the startup script and manually adding it to a global config file. This is clunky and there has to be a better way.
• Problem: Startup waits 30s for wireless to not connect when booting my laptop away from home. I can ^C but shouldn't have to.

Goal: allow a parallel system startup ( Read more... )

Go claims to have strings. That would be a nice feature except that most of the core library functions operate on byte arrays and you need to make an explicit cast every time you move from strings to byte arrays and back.

Do you want to interpret a boolean as an int? Fuck you.

It is difficult to define complex types such as sets of function references. The syntax that works in a parameter definition produces a syntax error when I try to cast an array. By the way, you need to cast your arrays when you define them because the compiler cannot determine that information from the array's contents.

Error handling is... different. For example, if you wanted to write to a file in any other language you would do something like this:

status = fopen("foo.txt", "w")

or:

try {
 fopen("foo.txt", "w")
} catch e {
...
}

In Golang you need to do something like this:

var writer *bufio.Writer 
f1, err1 := os.Open("foo.txt")
if(err1 != nil){
	return writer, err1
}
tmp2, err2 := io.Writer(f1)
if(err2 ! nil){
	return writer, err2
}

return bufio.Writer(tmp2)

... so you end up with long repeating branches in a function that would be a few lines in any other language.

Do you want to close the file, allow other subprocesses to work on it, and reopen it later for reading? Fuck you.

Have I mentioned that I don't like golang's error handling pattern? I don't like golang's error handling pattern.

Go gets one thing right and that is its goroutines. Parallelization of a function should be as easy as putting the word "go" in front of the function. Other languages should pick up on this.

Let us say that we have a data object and we want to manage metadata about the object. There are multiple ways of handling this. ( Read more... )

I was experimenting with Nim earlier this year.

Useful links:

• Nim manual
• Nim tutorial
• Num tutorial (part 2 - OOP, exceptions and macros)

I had intentions to write a blog post declaring 2011 as the Year of the Hacker but never got around to it. We had LulzSec, J3ster, and Web Ninjas. We had Stuxnet. We had Iran hacking certificate agencies. We had the iPhone "Towson" hack. We had noobs getting arrested for using LOIC from home. We had a virus hit control computers for the CIA's drones.

The benefit of modern Javascript+HTML is that you can do anything with it. The drawback of making Javascript+HTML a Turing-complete environment is that you can do anything with it.

I was studying Android programming a few months years ago, and they made this recommendation: "Don't call the UI-construction code directly! Use XML for your interfaces!"

The Android XML format is so painful to look at that I thought it worth my time to design my own alternate domain-specific language rather than use the one they gave me. (I never finished it)

The old practice of web development

<HTML>
<HEAD>
<TITLE>My Webpage</TITLE>

The new practice of web development

var node_html = document.createElement("HTML");
var node_html_head = document.createElement("HEAD");
var node_html_head_title = document.createElement("TITLE");
node_html_head_title.innerHTML ="My Webpage" 

Android and the Web seem to be going in opposite directions there.

I propose a new baseball statistic that weights and combines multiple different types of failure. Call it Derps Per Perp.

For pitchers:

• any appearance including:
  • an inning surrendering four or more runs, OR
  • any appearance of under one inning that is not the final appearance of that inning.
• Balks
• Hit batters

for batters:

• Hitting into double and triple plays
• Fielding errors

Divide batter stats by factors related to at-bats

Divide pitcher stats by factors related to innings pitched

Statistics for the AL might need to be adjusted due to pitchers not hitting and designated hitters not fielding.