Thursday, July 29, 2021

Glue Code is the Success Condition

My previous post titled Glue: the Dark Matter of Software may have given the impression that I see glue code as exclusively a problem. And I have to admit that my follow-up (and reaction to Github's copilot) called Don't Generate Glue...Exterminate may not have done much to dissuade anyone of that impression, but I just couldn't resist the Dalek reference.

However, I think it is important to remember that the fact that we have so much glue is a symptom of one of our biggest successes in software technology. Even as recently as the late 80s and early 90s, we just didn't have all that much to glue together, and software reuse was the holy grail, the unobtainium of computing, both in its desirability and unobtainability.

Now we have reuse. Boy do we have reuse! We have so much reuse that we need tool support to manage all the reuse. As far as I can tell, all new programming languages now come with such tooling, and are considered incomplete until they have it.

The price of success is having a new set of problems, problems you never dreamed of before.

So how will we solve these problems?

Data format adaptation, as suggested by the O'Reilly article? Yes. Model-driven approaches that allow us or our tools and languages to generate a lot of the more obvious adapter code? Sounds good, why not?

This one neat trick (click here!) that will automatically solve all these problems? No.

Simpler components, written with composability and minimization of dependencies in mind? Surely. Education, so developers get better at writing code that composes well without turning into architecture astronauts? Very much yes.

However, my contention is that developers have a hard time with this in large part because our languages only support implementing such glue, which is a start, but do not support expressing it directly, or abstracting over it, encapsulating it, playing with it. So new linguistic mechanisms like Objective-S are needed to help developers write better and thus less glue code so we can better enjoy the fruits of our reusability success.

Sunday, July 25, 2021

Deleting Code to Double the Performance of my Trivial Objective-S Tasks Backend

About two months ago, I showed a trivial tasks backend for a hypothetical ToDoMVC app. At the time, I noted that the performance was pretty insane for something written in a (slow) scripting language: 7K requests per second when fetching a single task.

That was using an encoder method (that writes key/value pairs to the JSON encoder) written in Objective-S, and I wondered how much faster it would go if that was no longer the case. Twice as fast, it turns out.

Yesterday, I wrote about tuning the Objective-S's SQLite insert performance to around 130M rows/minute, coincidentally also for a simple tasks schema. One part of that performance story was the fact that the encoder method (writing key/value pairs to the SQLite encoder) was generated by pasting together Objective-C blocks and installing the whole thing as an Objective-C method. No interpretation, except for calling a series of blocks stored in an NSArray. I had completely forgotten about the hand-written Objective-S encoder method in the back-end's Task class! Since generation is automatic, but won't override an already existing method, all I had to do in order to get the better performance was delete the old method.

> wrk -c 1 -t 1 http://localhost:8082/tasks 
Running 10s test @ http://localhost:8082/tasks
  1 threads and 1 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    66.60us    9.69us   1.08ms   96.72%
    Req/Sec    14.95k   405.55    15.18k    98.02%
  150275 requests in 10.10s, 30.67MB read
Requests/sec:  14879.22
Transfer/sec:      3.04MB
> curl  http://localhost:8082/tasks         
[{"id":1,"done":0,"title":"Clean Room"},{"id":2,"done":1,"title":"Check Twitter"}]%

More than twice the performance, and that while fetching two tasks instead of just one, so around 30K tasks/second! (And yes, I checked that I wasn't hitting a 404...).

So what's the performance if we actually fetch more than a minimal number of tasks? For 128 tasks, 64x more than before, it's still around 9K requests/s, so most of the time so far was per-request overhead. At this point we are serving a little over 1M tasks/s:

> wrk -c 1 -t 1 'http://localhost:8082/tasks/' 
Running 10s test @ http://localhost:8082/tasks/
  1 threads and 1 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   112.13us   76.17us   5.57ms   99.63%
    Req/Sec     9.05k   397.99     9.21k    97.03%
  90923 requests in 10.10s, 483.41MB read
Requests/sec:   9002.44
Transfer/sec:     47.86MB

If memory serves, that was around the rate we were seeing with the Wunderlist backend when we had a couple of million users, not that these are comparable in any meaningful way. For 1024 tasks there's a significant drop to slightly above 1.8K requests/s, with the task-rate almost doubling to 1.8M/s:

> wrk -c 1 -t 1 'http://localhost:8082/tasks/' 
Running 10s test @ http://localhost:8082/tasks/
  1 threads and 1 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   552.06us   62.77us   1.84ms   81.08%
    Req/Sec     1.82k    52.95     1.89k    90.10%
  18267 requests in 10.10s, 778.36MB read
Requests/sec:   1808.59
Transfer/sec:     77.06MB

Of course, those larger request sizes also see a much larger increase in performance than 2x. With the old code, the 128 task case clocks in at 147 requests/s and the 1024 task case at 18 requests/s, at which point it's a 100x improvement. So gives you an idea just how slow my Objective-S interpreter is.

Saturday, July 24, 2021

Inserting 130M SQLite Rows per Minute...from a Scripting Language

The other week, I stumbled on the post Inserting One Billion Rows in SQLite Under A Minute, which was a funny coincidence, as I was just in the process of giving my own SQLite/Objective-S adapter a bit of tune-up. (The post's title later had "Towards" prepended, because the author wasn't close to hitting that goal).

This SQLite adapater was a spin-off of my earlier article series on optimizing JSON performance, itself triggered by the ludicrously bad performance of Swift Coding at this rather simple and relevant task. To recap: Swift's JSON coder clocked in at about 10MB/s. By using a streaming approach and a bit of tuning, we got that to around 200MB/s.

Since then, I have worked on making Objective-S much more useful for UI work, with the object-literal syntax making defining UIs as convenient as the various "declarative" functional approaches such as React or SwiftUI. Except it is still using the same AppKit or UIKit objects we know and love, and doesn't force us to embrace the silly notion that the UI is a pure function of the model. Oh, and you get live previews that actually work. But more on that later.

So I am slowly inching towards doing a ToDoMVC, a benchmark that feels rather natural to me. While I am still very partial to just dumping JSON files, and the previous article series hopefully showed that this approach is plenty fast enough, I realize that a lot of people prefer a "real" database, especially on the back-end, and I wanted to build that as well. One of the many benchmarks I have for Objective-S is that it should be possible to build a nicer Rails with it. (At this point in time I am pretty sure I will hit that benchmark).

One of the ways to figure out if you have a good design is to stress-test it. One very useful stress-test is seeing how fast it can go, because that will tell you if the thing you built is lean, or if you put in unnecessary layers and indirections.

This is particularly interesting in a Scripted Components (pdf) system that combines a relatively slow but flexible interactive scripting language with fast, optimized components. The question is whether you can actually combine the flexibility of the scripting language while reaping the benefits of the fast components, rather than having to dive into adapting and optimizing the components for each use case, or just getting slow performance despite the fast components. My hunch was that the streaming approach I have been using for a while now and that worked really well for JSON and Objective-C would also do well in this more challenging setting.

Spoiler alert: it did!

The benchmark

The benchmark was a slightly modified version of the script that serves as a tasks backend. Like said sample script it also creates a tasks database and inserts some example rows. Instead of inserting two rows, it inserts 10 million. Or a hundred million.

#!env stsh

class Task {
	var  id.
	var  done.
	var  title.
	-description { "". }
	+sqlForCreate {
		'( [id] INTEGER PRIMARY KEY, [title] VARCHAR(220) NOT NULL, [done] INTEGER );'.

scheme todo : MPWAbstractStore {
	var db.
	var tasksTable.
	-initWithRef:ref {
		this:db := (MPWStreamQLite alloc initWithPath:ref path).
		this:tasksTable :=  #MPWSQLTable{ #db: this:db , #tableClass: Task, #name: 'tasks'  }.
		this:db open.
	-createTable {
		this:tasksTable create.
	    this:tasksTable := this:db tables at:'tasks'.
        this:tasksTable createEncoderMethodForClass: Task.
	-createTaskListToInsert:log10ofSize {
		baseList ← #( #Task{  #title: 'Clean Room', #done: false }, #Task{  #title: 'Check Twitter', #done: true } ).
		...replicate ...
	-insertTasks {
	    taskList := self createTaskListToInsert:6.
		1 to:10 do: {
			this:tasksTable insert:taskList.
todo := todo alloc initWithRef:dbref.
todo createTable.
todo insertTasks.

(I have removed the body of the method that replicates the 2 tasks into the list of millions of tasks we need to insert. It was bulky and not relevant.)

In this sample we define the Task class and use that to create the SQL Table. We could also have simply created the table and generated a Tasks class from that.

Anyway, running this script yields the following result.

> time ./ /tmp/tasks1.db 
./ /tmp/tasks1.db  4.07s user 0.20s system 98% cpu 4.328 total
> ls -al  /tmp/tasks1.db* 
-rw-r--r--  1 marcel  wheel   214M Jul 24 20:11 /tmp/tasks1.db
> sqlite3 /tmp/tasks1.db 'select count(id) from tasks;' 

So we inserted 10M rows in 4.328 seconds, yielding several hundred megabytes of SQLite data. This would be 138M rows had we let it run for a minute. Nice. For comparison, the original article's numbers were 11M rows/minute for CPython, 40M rows/minute for PyPy and 181M rows/minute for Rust, though on a slower Intel MacBook Pro whereas I was running this on an M1 Air. I compiled and ran the Rust version on my M1 Air and it did 100M rows in 21 seconds, so just a smidgen over twice as fast as my Objective-S script, though with a simpler schema (CHAR(6) instead of VARCHAR(220)) and less data (1.5GB vs. 2.1GB for 100M rows).

Getting SQLite fast

The initial version of the script was far, far slower, and at first it was, er, "sub-optimal" use of SQLite that was the main culprit, mostly inserting every row by itself without batching. When SQLite sees an INSERT (or an UPDATE for that matter) that is not contained in a transaction, it will automatically wrap that INSERT inside a generated transaction and commit that transaction after the INSERT is processed. Since SQLite is very fastidious about ensuring that transactions get to disk atomically, this is slow. Very slow.

The class handling SQLite inserts is a Polymorphic Write Stream, so it knows what an array is. When it encounters one, it sends itself the beginArray message, writes the contents of the array and finishes by sending itself the endArray message. Since writing an array sort of implies that you want to write all of it, this was a good place to insert the transactions:

-(void)beginArray {

-(void)endArray {

So now, if you want to write a bunch of objects as a single transaction, just write them as an array, as the benchmark code does. There were some other minor issues, but after that less than 10% of the total time were spent in SQLite, so it was time to optimize the caller, my code.

Column keys and Cocoa Strings

At this point, my guess was that the biggest remaining slowdown would be my, er, "majestic" Objective-S interpreter. I was wrong, it was Cocoa string handling. Not only was I creating the SQLite parameter placeholder keys dynamically, so allocating new NSString objects for each column of each row, it also happens that getting character data from an NSString object nowadays involves some very complex and slow internal machinery using encoding conversion streams. -UTF8String is not your friend, and other methods appear to fairly consistently use the same slow mechanism. I guess making NSString horribly slow is one way to make other string handling look good in comparison.

After a few transformations, the code would just look up the incoming NSString key in a dictionary that mapped it to the SQLite parameter index. String-processing and character accessing averted.

Jitting the encoder method. Without a JIT

One thing you might have noticed about the class definition in the benchmark code is that there is no encoder method, it just defines its instance variables and some other utilities. So how is the class data encoded for the SQLTable? KVC? No, that would be a bit slow, though it might make a good fallback.

The magic is the createEncoderMethodForClass: method. This method, as the name suggests, creates an encoder method by pasting together a number of blocks, turns the top-level into a method using imp_implementationWithBlock(), and then finally adds that method to the class in question using class_addMethod().

    NSArray *ivars=[theClass allIvarNames];
    if ( [[ivars lastObject] hasPrefix:@"_"]) {
        ivars=(NSArray*)[[ivars collect] substringFromIndex:1];
    NSMutableArray *copiers=[[NSMutableArray arrayWithCapacity:ivars.count] retain];
    for (NSString *ivar in ivars) {
        MPWPropertyBinding *accessor=[[MPWPropertyBinding valueForName:ivar] retain];
        [ivar retain];
        [accessor bindToClass:theClass];
        id objBlock=^(id object, MPWFlattenStream* stream){
            [stream writeObject:[accessor valueForTarget:object] forKey:ivar];
        id intBlock=^(id object, MPWFlattenStream* stream){
            [stream writeInteger:[accessor integerValueForTarget:object] forKey:ivar];
        int typeCode = [accessor typeCode];
        if ( typeCode == 'i' || typeCode == 'q' || typeCode == 'l' || typeCode == 'B' ) {
            [copiers addObject:Block_copy(intBlock)];
        } else {
            [copiers addObject:Block_copy(objBlock)];
    void (^encoder)( id object, MPWFlattenStream *writer) = Block_copy( ^void(id object, MPWFlattenStream *writer) {
        for  ( id block in copiers ) {
            void (^encodeIvar)(id object, MPWFlattenStream *writer)=block;
            encodeIvar(object, writer);
    void (^encoderMethod)( id blockself, MPWFlattenStream *writer) = ^void(id blockself, MPWFlattenStream *writer) {
        [writer writeDictionaryLikeObject:blockself withContentBlock:encoder];
    IMP encoderMethodImp = imp_implementationWithBlock(encoderMethod);
    class_addMethod(theClass, [self streamWriterMessage], encoderMethodImp, "v@:@" );

What's kind of neat is that I didn't actually write that method for this particular use-case: I had already created it for JSON-coding. Due to the fact that the JSON-encoder and the SQLite writer are both Polymorphic Write Streams (as are the targets of the corresponding decoders/parsers), the same method worked out of the box for both.

(It should be noted that this encoder-generator currently does not handle all variety of data types; this is intentional).

Getting the data out of Objective-S objects

The encoder method uses MPWPropertyBinding objects to efficiently access the instance variables via the object's accessors, caching IMPs and converting data as necessary, so they are both efficient and flexible. However, the actual accessors that Objective-S generated for its instance variables were rather baroque, because they used the same basic mechanism used for Objective-S methods, which can only deal with objects, not with primitive data types.

In order to interoperate seamlessly with Objective-C, which expected methods that can take data types other than objects, all non-object method arguments are converted to objects on the way in, and return values are converted from objects to primitive values on the way out.

So even the accessors for primitive types such as the integer "id" or the boolean "done" would have their values converted to and from objects by the interface machinery. As I noted above, I was a bit surprised that this inefficiency was overshadowed by the NSString-based key handling.

In fact, one of the reason for pursuing the SQLite insert benchmark was to have a reason for finally tackling this Rube-Goldberg mechanism. In the end, actually addressing it turned out to be far less complex than I had feared, with the technique being very similar to that used for the encoder-generator above, just simpler.

Depending on the type, we use a different block that gets parameterised with the offset to the instance variable. I show the setter-generator below, because there the code for the object-case is actually different due to retain-count handling:

#define pointerToVarInObject( type, anObject ,offset)  ((type*)(((char*)anObject) + offset))

#ifndef __clang_analyzer__
// This leaks because we are installing into the runtime, can't remove after

    SEL aSelector=NSSelectorFromString([self objcMessageName]);
    const char *typeCode=NULL;
    int ivarOffset = (int)[ivarDef offset];
    IMP getterImp=NULL;
    switch ( ivarDef.objcTypeCode ) {
        case 'd':
        case '@':
            typeCode = "v@:@";
            void (^objectSetterBlock)(id object,id arg) = ^void(id object,id arg) {
                id *p=pointerToVarInObject(id,object,ivarOffset);
                if ( *p != arg ) {
                    [*p release];
                    [arg retain];
        case 'i':
        case 'l':
        case 'B':
            typeCode = "v@:l";
            void (^intSetterBlock)(id object,long arg) = ^void(id object,long arg) {
            [NSException raise:@"invalidtype" format:@"Don't know how to generate set accessor for type '%c'",ivarDef.objcTypeCode];
    if ( getterImp && typeCode ) {
        class_addMethod(aClass, aSelector, getterImp, typeCode );

At this point, profiles were starting to approach around two thirds of the time being spent in sqlite_ functions, so the optimisation efforts were starting to get into a region of diminishing returns.

Linear scan beats dictionary

One final noticeable point of obvious overhead was the (string) key to parameter index mapping, which the optimizations above had left at a NSDictionary mapping from NSString to NSNumber. As you probably know, NSDictionary isn't exactly the fastest. One idea was to replace that lookup with a MPWFastrStringTable, but that means either needing to solve the problem of fast access to NSString character data or changing the protocol.

So instead I decided to brute-force it: I store the actual pointers to the NSString objects in a C-Array indexed by the SQLite parameter index. Before I do the other lookup, which I keep to be safe, I do a linear scan in that table using the incoming string pointer. This little trick largely removed the parameter index lookup from my profiles.


With those final tweaks, the code is probably quite close to as fast as it is going to get. Its slower performance compared to the Rust code can be attributed to the fact that it is dealing with more data and a more complex schema, as well as having to actually obtain data from materialized objects, whereas the Rust code just generates the SQlite calls on-the-fly.

All this is achieved from a slow, interpreted scripting language, with all the variable parts (data class, steering code) defined in said slow scripting language. So while I look forward to the native compiler for Objective-S, it is good to know that it isn't absolutely necessary for excellent performance, and that the basic design of these APIs is sound.

Wednesday, June 30, 2021

Don't Generate Glue...Exterminate!!

Today I saw the news of github's release of the "AI Autopilot". As far as I can tell, it's an impressive piece of engineering that shouldn't exist. I mean, "Paste Code from Stack Overflow as a Service" was supposed to be a joke, not a product spec.

As of this writing, the first example given on the product page is some code to call a REST service that does sentiment analysis, which the AI helpfully completes. For reference, this is the code:

#!/usr/bin/env ts-node

import { fetch } from "fetch-h2";

// Determine whether the sentiment of text is positive
// Use a web service
async function isPositive(text: string): Promise<boolean> {
  const response = await fetch(``, {
    method: "POST",
    body: `text=${text}`,
    headers: {
      "Content-Type": "application/x-www-form-urlencoded",
  const json = await response.json();
  return json.label === "pos";

Here is the same script in Objective-S:
#!env stsh
((ref: postForm:#{ #text: text }) at:'label') = 'pos'

And once you have it, reuse it. And keep those Daleks at bay.

Monday, June 28, 2021

Generating ARM Assembly: First Steps

Finally took the plunge to start generating ARM64 assembly. As expected, the actual coding was much easier than overcoming the barrier to just start doing it.

The following snippet generates a program that prints a message to stdout, so a classic "Hello World":

#!env stsh

messageLabel ← 'message'.
main ← '_main'.

framework:ObjSTNative load
arm := MPWARMAssemblyGenerator stream 
arm global: main;
    mov:0 value:1;
    adr:1 address:messageLabel;
    mov:2 value: msg length;
    mov:16 value:4;
    mov:0 value:0;

file:hello-main.s := arm target 

One little twist is that the message to print gets passed to the generator. I like how Smalltalk's keyword syntax keeps the code uncluttered, and often pretty close to the actual assembly that will be generated.

Of particular help here is message cascading using the semicolon. This means I don't have to repeat the receiver of the message, but can just keep sending it messages. Cascading works well together with streams, because there are no return values to contend with, we just keep appending to the stream.

When invoked using ./ 'Hi Marcel, finish that blog post and get on your bike!', the generated code is as follows:

.global	_main
.align	2
	mov	X0, #1
	adr	X1, message
	mov	X2, #54
	mov	X16, #4
	svc	#128
	mov	X0, #0
	.asciz	"Hi Marcel, finish that blog post and get on your bike!"

And now I am going to do what it says :-)

Tuesday, June 15, 2021

if let it be

One of funkier aspects of Swift syntax is the if let statement. As far as I can tell, it exists pretty much exclusively to check that an optional variable actually does contain a value and if it does, work with a no-longer-optional version of that variable.

Swift packages this functionality in a combination if statement and let declaration:

if let value = value {
   print("value is \(value)")

This has a bunch of problems that are explained nicely in a Swift Evolution thread (via Michael Tsai) together with some proposals to fix it. One of the issues is the idiomatic repitition of the variable name, because typically you do want the same variable, just with less optionality. Alas, code-completion apparently doesn't handle this well, so the temptation is to pick a non-descriptive variable name.

In my previous post (Asynchronous Sequences and Polymorphic Streams) I noted how the fact that iteration in Smalltalk and Objecive-S is done via messages and blocks means that there is no separate concept of a "loop-variable", that is just an argument to the block.

Conditionals are handled the same way, with blocks and messages, but normally don't pass arguments to their argument blocks, because in normal conditionals those arguments would always be just the constants true or false. Not very interesting.

When I added ifNotNil: some time ago, I used the same logic, but it turns out the object is now actually potentially interesting. So ifNotNil: now passes the now-known-to-be-non-nil value to the block and can be used as follows:

value ifNotNil:{ :value |
    stdout println:value.

This doesn't eliminate the duplication, but does avoid the issue of having the newly introduced variable name precede the original variable. Well, that and the whole weird if let in the first place.

With anonymous block arguments, we actually don't have to name the parameter at all:

value ifNotNil:{ stdout println:$0. }

Alternatively, we can just take advantage of some conveniensces and use a HOM instead:

value ifNotNil printOn:stdout.

Of course, Objective-S currently doesn't care about optionality, and with the current nil-eating behavior, the ifNotNil is not strictly necessary, you could just write it as follow:

value printOn:stdout.

I haven't really done much thinking about it, but the whole idea of optionality shouldn't really be handled in the space of values, but in the space of references. Which are first class objects in Objective-S.

So you don't ask a value if it is nil or not, you ask the variable if it contains a value:

ref:value ifBound:{ :value | ... }

To me that makes a lot more sense than having every type be accompanied by an optional type.

So if we were to care about optionality so in the future, we have the tools to create a sensible solution. And we can let if let just be.

Sunday, June 13, 2021

Asynchronous Sequences and Polymorphic Streams

Browsing the WWDC '21 session videos, I came across the session on Asynchronous Sequences. The preview image showcased some code for asynchronously fetching and massaging current earthquake data from the U.S. Geological Survey:
struct QuakesTool {
   static func main() async throws {
      let endpointURL = URL(string: "")!

      for try await event in endpointURL.lines.dropFirst() {
         let values = event.split(separator: ",")
         let time = values[0]
         let latitude = values[1]
         let longitude = values[2]
         let magnitude = values[4]
         print("Magnitude \(magnitude) on \(time) at \(latitude) \(longitude)")

This is nice, clean code, and it certainly looks like it serves as a good showcase for the benefits of asynchronous coding with async/await and asynchronous sequences built on top of async/await.

Or does it?

Here is the equivalent code in Objective-S:

#!env stsh
stream ← ref: linesAfter:1.

stream do: { :theLine |
   values ← theLine componentsSeparatedByString:','.
   time ← values at:0.
   latitude ← values at:1.
   longitude ← values at:2.
   magnitude ← values at:4.
   stdout println:"Quake: magnitude {magnitude} on {time} at {latitude} {longitude}".
stream awaitResultForSeconds:20.

Objective-S does not (and will not) have async/await, but it can nevertheless provide the equivalent functionality easily and elegantly. How? Two features:

  1. Polymorphic Write Streams
  2. Messaging
Let's see how these two conspire to make adding something equivalent to for try await trivial.

Polymorphic Write Streams

In the Objective-S implementation, is not a string, but an actual identifier, a Polymorphic Identifier, adding the ref: prefix turns it into a binding, a first class variable. You can ask a binding for its value, but for bindings that can also be regarded as collections of some kind, you can also ask them for a stream of their values, in this particular case a MPWURLStreamingStream. This stream is a Polymorphic Write Stream that can be easily composed with other filters to create pipelines. The linesAfter: method is a convenience method that does just that: it composes the URL fetcher with a filter that converts from bytes to lines of text and another filter that drops the first n items.

Objective-S actually has convenient syntax for creating these compositions without having to do it via convenience methods, but I wanted to keep differences in the surrounding scaffolding small for this example, which is about the for try away and do:.

When I encountered the example, Polymorphic Write Streams actually did not have a do: for iteration, but it was trivial to add:

    [self setFinalTarget:[MPWBlockTargetStream streamWithBlock:aBlock]];
    [self run];

(This code lives in MPWFoundation, so it is in Objective-C, not Objective-S).

Those 5 lines were all that was needed. I did not have to make substantive changes to the language or its implementation. One reason for this is that Polymorphic Write Streams are asynchrony-agnostic: although they are mostly implemented as straightforward synchronous code, they work just as well if parts of the pipeline they are in are asynchronous. It just doesn't make a difference, because the semantics are in the data flow, not in the control flow.


The other big reason an asynchronous do: was easy to add is messaging.
If you focus on just messaging -- and realize that a good metasystem can late bind the various 2nd level architectures used in objects -- then much of the language-, UI-, and OS based discussions on this thread are really quite moot.
One of the many really, really neat ideas in Smalltalk is how control structures, which in most other languages are special language features, are just plain old messages and implemented in the library, not in the language.

So the for ... in loop in Swift is just the do: message sent to a collection, and the keyword syntax makes this natural:

for event in lines {
lines do: { :event |

Note how making loops regular like this also makes the special concept of "loop variable" disappear. The "loop variable" is just the block argument. And I just realized the same would go for a not-nil result of a nil test.

Anyway, if "loops" are just messages, it's easy to add a method implementing iteration to some other entity, for example a stream, the way that I did. (Smalltalk streams also support the iteration messages).

And when you can easily make stream processing, which can handle asynchrony naturally and easily, just as convenient as imperative programming, you don't need async/await, which tries to make asynchronous programming look like imperative programming in order to make it convenient.

Wednesday, June 9, 2021

Glue: the Dark Matter of Software

"Software seems 'large' and 'complicated' for what it does". I keep coming back to this quote by Alan Kay.

The same feeling has been nagging me pretty me much ever since I started writing software. On the one hand, there is the magic, almost literally: we write some text (spells) and the machine does things in the real world. On the other hand, it seems just way too much work to make the machine do anything more complex than:

10 PRINT "Hello"
20 GOTO 10
Almost like threading a needle with boxing gloves. And that's even if we are careful, if we avoid unnecessary complexity.

And the numbers appear to back that up, Alan Kay mentions Microsoft office at several hundred million lines of code. From my personal experience, the Wunderlist iOS client was not quite 200 KLOC. For the latter, I can attest to the attention given by the team to not introduce unnecessary bloat, and even to actively reduce it. (For example, we cut our core code by around 30KLOC thanks to some of the architectural mechanisms such as Storage Combinators). I am fairly sure I am not the only one with this experience.

So why so much code? After all Wunderlist was just a To Do List, albeit a really nice one. I can't really say much about Office, I don't think anyone can, because 400 MLOC is just way too much code to comprehend. I think the answer is:

Glue Code.

It's the unglamorous, invisible code that connects two pieces of software, makes sure that data that's in location A reaches location B unscathed (from the datbase to the UI, from the UI to the model, from the model to the backend and so on...). And like Dark Matter, it is invisible and massive.

Why do I say it is "invisible"? After all, the code is right there, isn't it? As far as I can tell, there are several related reasons:

  1. Glue code is deemed not important. It's just a couple of lines here, and another couple of lines over there ... and soon enough you're talking real MLOCs!
  2. We cannot directly express glue code. Most of our languages are what I call "DSLs for Algorithms" (See ALGOL, the ALGOrithmic Language), so glue can not be expressed intentionally, but only by describing algorithms for implementing the glue.
That's why it is invisible, and also partly why it is massive: not being able to express it directly means we cannot abstract and encapsulate it, we keep repeating slight variations of that glue. There is another reason why it's massive:
  1. Glue is quadratic. If you have N features that interact with each other, you have O(N²) pieces of glue to get them to talk to each other.

This last point was illustrated quite nicely by Kevin Greer in a video comparing Multics and Unix development, with the crucial insight being that you need to "program the perimeter, not the area":

For him, the key difference is that Unix had the pipe, and I would agree. The pipe is one-character glue: "|". This is absolutely crucial.

If you have to write even a little custom code every time you connect two modules, you will be in quadratic complexity, meaning that as your features grow your glue code will overwhelm the core functionality. And you will only notice this when it's far too late to do anything about it, because the initial growth rate will be low.

So what can we do about it? I think we need to make glue first class so we can actually write down the glue itself, and not the algorithms that implement the glue. Once we have that, we can and hopefully will create better kinds of glue, ones like the Unix pipe in that they can connect components generically, without requiring custom glue per component pair.


There were some questions as to what to do about this. Well, I am working on it, with Objective-S, and I write fairly frequently on this blog (and occasionally submit my writing to scientific conferences), one post that would be immediately relevant is: Why Architecture Oriented Programming Matters.

I also don't see Unix Pipes and Filters as The Answer™, they just demonstrate the concept of minimized and constant glue. Expanding on this, and as I wrote in Why Architecture Oriented Programming Matters, I also don't see any one single connector as "the" solution. We need different kinds of connectors, and we need to write them down, to abstract over them and use them natively. Not simulate everything by calling procedures, methods or functions. See also Foxes vs. Hedgehogs.

Tuesday, June 1, 2021

Towards a ToDoMVC Backend in Objective-S

A couple of weeks ago, I showed a little http backend. Well, tiny is probably a more apt description, and also aptly describes its functionality, which is almost non-existent. All it does is define a simplistic Task class, create an array with two sample instances and then serves that array of tasks over http. And it serves the -description of those tasks rather than anything usefuk like a JSON encoding.

For reference, this is the original code, hacked up in maybe 15 minutes:

#!env stsh
framework:ObjectiveHTTPD load.

class Task {
   var <bool> done.
   var title.
   -description { "Task: {this:title} done: {this:done}". }

taskList ← #( #Task{ #title: 'Clean my room', #done: false }, #Task{ #title: 'Check twitter feed', #done: true } ).

scheme todo {
   var taskList.
   /tasks { 
      |= { 

todo := #todo{ #taskList: taskList }.
server := #MPWSchemeHttpServer{ #scheme: todo, #port: 8082 }.
server start.
shell runInteractiveLoop.

What would it take to make this borderline useful? First, we would probably need to encode the result as JSON, rather than serving a description. This is where Storage Combinators come in. We (now) have a MPWJSONConverterStore that's a mapping store, it passes its "REST" requests through while performing certain transformations on the data and/or the references. In this case the transformation is serializing or deserialzing objects from/to JSON, depending on which way the request is going and which way the converter is pointing.

In this case, the converter is pointing "up", that is it serializes objects read from its source to JSON and deserializes data written to its source from JSON to objects. We also tell it that it is dealing with Task objects. When we have the converter we connect it to our todo scheme and tell the HTTP server to talk to the json converter (which talks to our todo scheme):

todo := #todo{ #taskList: taskList, #store: persistence }.
json := #MPWJSONConverterStore{  #up: true, #class: class:Task }.
json → todo.
server := #MPWSchemeHttpServer{ #scheme: json, #port: 8082 }.

Second, we also want to be to interact with individual tasks. No problem, just add a /task/:id proprerty path to our store/scheme handler, along with GET ("|=") and PUT ("=|") handlers. I am not fully sold yet on the "|=" syntax for this, but I would like to avoid names for this sort of structural component. Maybe arrows?
	/task/:id {
		|= {
			this:taskDict at:id .
		=| {
			this:taskDict at:id put:newValue.

In order to facilitate this, the taskList was changed to a dictionary. Once we make changes to our data, we probably also want to persist it. One easy way to do this is to store the tasks as JSON on disk. This allows us to reuse the JSON converter from above, but this time pointing "down". We connect this converter to the filesystem at the directory /tmp/tasks and to the store:
json → todo → #MPWJSONConverterStore{  #class: class:Task } → ref:file:/tmp/tasks/ asScheme.

In addition, we need to trigger saving in the PUT handler:
		=| {
			this:taskDict at:id put:newValue.
			self persist.
	-persist {
		source:tasks := this:taskDict allValues.

This will (synchronously) write the entire task list on every PUT. The full code is here:
#!env stsh
framework:ObjectiveHTTPD load.

class Task {
	var id.
	var  done.
	var title.
	-description { "Task: {this:title} done: {this:done} id: {this:id}". }
	-writeOnJSONStream:aStream {
		aStream writeDictionaryLikeObject:self withContentBlock:{ :writer |
			writer writeInteger: this:id forKey:'id'.
			writer writeString: this:title forKey:'title'.
			writer writeInteger: this:done forKey:'done'.

taskList ← #( #Task{ #id: '1', #title: 'Clean Room', #done: false }, #Task{ #id: '2', #title: 'Check Twitter', #done: true } ).

scheme todo : MPWMappingStore {
	var taskDict.
	-setTaskList:aList {
		this:taskDict := NSMutableDictionary dictionaryWithObjects: aList forKeys: aList collect id.
	/tasks { 
		|= { 
			this:taskDict allValues.
	/task/:id {
		|= {
			this:taskDict at:id .
		=| {
			this:taskDict at:id put:newValue.
			self persist.
	-persist {
		source:tasks := this:taskDict allValues.

todo := #todo{ #taskList: taskList }.
json := #MPWJSONConverterStore{  #up: true, #class: class:Task }.
json → todo → #MPWJSONConverterStore{  #class: class:Task } → ref:file:/tmp/tasks/ asScheme.
server := #MPWSchemeHttpServer{ #scheme: json, #port: 8082 }.
server start.
shell runInteractiveLoop.

The writeOnJSONStream: method is currently still needed by the serializer to encode the task object as JSON. The parser doesn't need any support, it can figure things out by itself for simple mappings. Yes, this makes no sense, as serializing is easier than parsing, but I haven't gotten around to the automation for serializing yet.


So there you have it, an almost functional Todo backend, in refreshingly little code, and with refreshingly little magic. What I find particularly pleasing is that this conciseness can be achieved while keeping the architecture fully visible and maintaining a hexagonal/ports-and-adapters style.

What is the architecture of this app? It says so right at the end: the server is parametrized by its scheme, and that scheme is a JSON serializer hooked up to my todo scheme handler, hooked up to another JSON serializer hooked up to the directory /tmp/tasks.

Although a Rails app contains comparably little code, this code is scattered over different classes and is only comprehensible as a plugin to Rails. All the architecture is hidden inside Rails, it is not at all visible in the code and simply cannot be divined from looking at the code. Although there are many reasons for this, one fundamental one is that Ruby is a call/return language, and Rails does its best to translate from the REST architectural style to something that is more natural in the call/return style. And it does an admirable job at it.

I do think that this example gives us a little glimpse into what I believe to be the power of Architecture Oriented Programming: the power and succinctness of frameworks, but with the simplicity, straightforwardness and reusability of more library-oriented styles.


I obviously couldn't resist benchmarking this, and to my great joy found that wrk now works on the M1. Since the interpreter isn't thread safe, I had to restrict it to a single connection and thread. My expectations were that it requests/s would be in the double to low triple digits, my fear was that it would be single digits. (The reason for that fear is the writeOnJSONStream: method that is called for every object serialized and is in interpreted Objective-S, probably one of the slowest language implementations currently in existence). To say I was surprised is an understatement. Stunned is more like it:
wrk -c 1 -t 1 http://localhost:8082/task/1 
Running 10s test @ http://localhost:8082/task/1
  1 threads and 1 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   133.62us   14.45us   0.97ms   98.52%
    Req/Sec     7.50k   311.09     7.62k    99.01%
  75326 requests in 10.10s, 12.28MB read
Requests/sec:   7458.60
Transfer/sec:      1.22MBTransfer/sec:      1.97MB

More than 7K requests per second! Those M1 Macs really are fast. I wonder what it will be once I remove the need for the manually written writeOnJSONStream: method.

(NOTE: previous version said >12K requests/s, which is even more insane, but was with an incorrect URL that had the server returning 404s)

Thursday, May 20, 2021

Why are there no return statements in Objective-S?

My previous example raised a question: why no return statements? I am assuming this was about this part of the example:

   -description { "Task: {this:title} done: {this:done}". }

The answer is that I would like to do without return statements if and as much as I can. We will see how much that is. In general, I am in favor of expression-orientation in programming languages. A simple example is if-statements vs. conditional expressions. In most languages today, like C, Objective-C and Swift, if is a statement. That means I write something as follows:
if ( condition ) {
   do something if true
} else {
   do something different if false

This seems obvious and is general, but very often you don't want to do arbitrary stuff, you just want to have some variable have some value in one case and a different value in another case.
int foo;
if ( condition ) {
   foo = 1;
} else {
   foo = 42;

In that case, it is annoying that the if is defined to be a statement and not an expression, because you can't just write the following:
int foo;
foo = if ( condition ) { 1; } else { 42; }

In addition, as hinted to in the previous examples, you can't use a statement to initialize a variable, that definitely has to be an expression. Which is why C and many derived languages have the "ternary" operator (?:), which is really just an if/else in expression form.
int foo=condition ? 1 : 42;

That solves the problem, but now you have two conditionals. Why not have just one? LISP, most of the FP languages as well as Smalltalk and Objective-S have an if that returns a value.
a := condition ifTrue:{ 1. } ifFalse:{ 42. }.

So that's why expression-orientation is useful in general. What about methods? The same general idea applies. Whereas in Java, for example, a read accessor is called getX(), indicating an action that is performed ("get the value of x") in Objective-C Smalltalk and Objective-S, it is just called x, ("the value of x").

The same idea applies to dropping return statements where possible. It's not "get me the description of this object", it is "the description of this object is...". And inside the method, it's not "this statement now returns the following string as the description", but, again, "the description is...".

Describing things that are, rather than actions to perform, is at the heart of Objective-S, as discussed in Can Programmers Escape the Gentle Tyranny of Call/Return.

As Guy Steele put it:

Another weakness of procedural and functional programming is that their viewpoint assumes a process by which "inputs" are transformed into "outputs"; there is equal concern for correctness and for termination (and proofs thereof). But as we have connected millions of computers to form the Internet and the World Wide Web, as we have caused large independent sets of state to interact–I am speaking of databases, automated sensors, mobile devices, and (most of all) people–in this highly interactive, distributed setting, the procedural and functional models have failed, another reason why objects have become the dominant model. Ongoing behavior, not completion, is now of primary interest. Indeed, object-oriented programming had its origins in efforts to simulate the ongoing behavior of interacting real-world entities–thus the programming language SIMULA was born.
So wherever possible, Objective-S tries to push towards expressing things as statically as possible, pushing away from action-orientation. For example, hooking up a timed source to a pin:
#Blinker{ #seconds: 1, #active: true} → ref:gpio:17. 

instead of executing a loop:
while True: 
    GPIO.output(17, True) 
    GPIO.output(17, False) 

The same goes for many other relationships: instead of writing procedural code that initiates and/or maintains the relatinship, with the actual relationship remainig implicit, describe the actual relationship, make that explicit, and instead keep the procedural code that maintains it as a hidden implementation detail.

If the return statement comes back, and it very well might, I am hoping it will be in a slightly more general form. I recall Smalltalk's "^" being described as "send back". I've already taken that and generalised it to mean "send result", using it in filter definitions, where "^" means "send a result to the next filter in the pipeline". It is needed there because filters are not limited to sending a single result, they can send zero or many.

With those more general semantics, "^" might also be used to send back results to the sender of an asynchronous message, which is obviously quite different from a "return".

And of course it would be useful for early returns, which are currently not possible.

What about void methods?

Objective-S does have void methods, after all its procedural part is essentially identical to Objective-C, which also has them. However, I agree with the FP folk that functions (procedures, methods) should be as (side-)effect free as possible, and void methods by definition are effectful (or no-ops).

So where do the effects go? Two places:

  1. The left hand side of the "←".

    In most current programming languages, assignment is severely crippled, and therefore not really useful for generalised effects. With Polymorphic Identifiers and Storage Combinators, there is enough expressive power and ability to abstract that we should need far fewer void methods.

  2. Connecting via "→"

    Much of the need for effectful methods in OO is for constructing and connecting objects. In Objective-S, you don't need to call methods that result in a connection being established as a side effect of munging on some state, you define connections between objects directly using "→".

    Well, and you define objects using object literals such as  #Blinker{ #seconds: 1, #active: true}  instead of setting instance variables procedurally.

That's the plan, anyway. Although a lot of that plan is coming true at the moment. Exciting times! (And one of the reasons I haven't been blogging all that much).

A far too simple (hardcoded) tasks backend in Objective-S

Recently there was a question as to what one should use to create a backend for an iOS/macOS app these days. I couldn't resist mentioning Objective-S, and just to check for myself whether that's feasible, I quickly jotted down the following tiny backend that returns a hardcoded list of tasks via HTTP:
#!env stsh
framework:ObjectiveHTTPD load.

class Task {
   var <bool> done.
   var title.
   -description { "Task: {this:title} done: {this:done}". }

taskList ← #( #Task{ #title: 'Clean my room', #done: false }, #Task{ #title: 'Check twitter feed', #done: true } ).

scheme todo {
   var taskList.
   /tasks { 
      |= { 

todo := #todo{ #taskList: taskList }.
server := #MPWSchemeHttpServer{ #scheme: todo, #port: 8082 }.
server start.
shell runInteractiveLoop.

After loading the HTTP framework, we define a Task and a list of two example tasks. Then we define a scheme with a single path, just /tasks, which returns said tasks list. We then instantiate the scheme and serve it via HTTP on port 8082. Since this is a shell script and starting the server does not block, we finally start up the REPL.

Details such as coding the tasks as JSON, accessing a single task and modifying tasks are left as exercises for the reader.

Sunday, May 9, 2021

Talking to pins

The last few weeks, I spent a little time getting Objective-S working well on the Raspberry Pi, specifically my Pi400. It's a really wonderful little machine, and the form factor and price remind me very much of the early personal computers.

What's missing, IMHO, is an experience akin to the early BASICs. And I really mean "akin", not a nostalgia project, but recovering a real quality that has been lost: not really "simplicity", more "straightforwardness".

Of course, one of the really cool thing about the Pi is its GPIO interface that lets you do all sorts of electronics experiments, and I hear that the equivalent of "Hello World" for the Raspi is making an LED blink.

import RPi.GPIO as GPIO 
from time import sleep 
GPIO.setup(17, GPIO.OUT) 
while True: 
    GPIO.output(17, True) 
    GPIO.output(17, False) 

Hmm. That's a a lot of semantic noise for something so conceptually simple. All we want to is set the value of a pin. As soon as I saw this, I knew it would be ideal for Polymorphic Identifiers, because a pin is the ultimate state, and PIs and their stores are made for abstracting over state.

Of course, I first had to to get Objective-S running on the Pi, which meant getting GNUstep to run. While there is a wonderful set of install scripts, the one for the Raspi only worked with an ancient clang version and libobjc 1.9. Alas, that version has some bugs on the Raspi, for example with the imp_implentationWithBlock() runtime function that Objective-S uses to define methods.

Long story short, after learning about GNUstep installs and waiting for the wonderful David Chisnall to remove some obsolete 32 bit exception-version detection code from libobjc, we now have a script that installs current GNUstep with a reasonably current clang: With that in hand, a few bug fixes in MPWFoundation and Objective-S, I could add a really rudimentary Store that manages talking to the pins. And this allows me to write the following in an interactive shell to drive the customary GPIO pin 17 that I connected to the LED via resistor:

gpio:17 ← 1.

Now that's what I am talking about!

Of course, we're supposed to make it blink, not just turn it on. We could use the same looping approach as the Python script, or convenience methods like the ones provided, but the breadboard and pins make me think of wanting to connect components to do the job instead.

So let's connect some components, software architecture style! The following script creates an instance of a Blinker object (using an object literal), which emits alternating ones and zeros and connects it to the pin.

blinker ← #Blinker{ #seconds: 1 }. 
blinker → ref:gpio:17. 
blinker run.
gpio:17 ← 0.

Once connected it tells the blinker to start running, which creates an NSTimer adds it to the current runloop and then runs the run loop. That run is interruptible, so Ctrl-C breaks and runs the cleanup code.

What about setting up the pin for output? Happens automatically when you first output to it, but I will add code so you can do it manually.

Where does the Blinker come from? That's actually an object-template based on an MPWFixedValueSource.

object Blinker : #MPWFixedValueSource{ #values: #(0,1) }

You can, of course, hook up a fixed-value source to any kind of stream.

While getting here took a lot of work, and resulted in me (re-)learning a lot about GNUstep, the result, even this intermediate one, is completely worth it and makes me very happy. This stuff really works even better than I thought it would.