Sunday, June 13, 2021

Asynchronous Sequences and Polymorphic Streams

Browsing the WWDC '21 session videos, I came across the session on Asynchronous Sequences. The preview image showcased some code for asynchronously fetching and massaging current earthquake data from the U.S. Geological Survey:
struct QuakesTool {
   static func main() async throws {
      let endpointURL = URL(string: "")!

      for try await event in endpointURL.lines.dropFirst() {
         let values = event.split(separator: ",")
         let time = values[0]
         let latitude = values[1]
         let longitude = values[2]
         let magnitude = values[4]
         print("Magnitude \(magnitude) on \(time) at \(latitude) \(longitude)")

This is nice, clean code, and it certainly looks like it serves as a good showcase for the benefits of asynchronous coding with async/await and asynchronous sequences built on top of async/await.

Or does it?

Here is the equivalent code in Objective-S:

#!env stsh
stream ← ref: linesAfter:1.

stream do: { :theLine |
   values ← theLine componentsSeparatedByString:','.
   time ← values at:0.
   latitude ← values at:1.
   longitude ← values at:2.
   magnitude ← values at:4.
   stdout println:"Quake: magnitude {magnitude} on {time} at {latitude} {longitude}".
stream awaitResultForSeconds:20.

Objective-S does not (and will not) have async/await, but it can nevertheless provide the equivalent functionality easily and elegantly. How? Two features:

  1. Polymorphic Write Streams
  2. Messaging
Let's see how these two conspire to make adding something equivalent to for try await trivial.

Polymorphic Write Streams

In the Objective-S implementation, is not a string, but an actual identifier, a Polymorphic Identifier, adding the ref: prefix turns it into a binding, a first class variable. You can ask a binding for its value, but for bindings that can also be regarded as collections of some kind, you can also ask them for a stream of their values, in this particular case a MPWURLStreamingStream. This stream is a Polymorphic Write Stream that can be easily composed with other filters to create pipelines. The linesAfter: method is a convenience method that does just that: it composes the URL fetcher with a filter that converts from bytes to lines of text and another filter that drops the first n items.

Objective-S actually has convenient syntax for creating these compositions without having to do it via convenience methods, but I wanted to keep differences in the surrounding scaffolding small for this example, which is about the for try away and do:.

When I encountered the example, Polymorphic Write Streams actually did not have a do: for iteration, but it was trivial to add:

    [self setFinalTarget:[MPWBlockTargetStream streamWithBlock:aBlock]];
    [self run];

(This code lives in MPWFoundation, so it is in Objective-C, not Objective-S).

Those 5 lines were all that was needed. I did not have to make substantive changes to the language or its implementation. One reason for this is that Polymorphic Write Streams are asynchrony-agnostic: although they are mostly implemented as straightforward synchronous code, they work just as well if parts of the pipeline they are in are asynchronous. It just doesn't make a difference, because the semantics are in the data flow, not in the control flow.


The other big reason an asynchronous do: was easy to add is messaging.
If you focus on just messaging -- and realize that a good metasystem can late bind the various 2nd level architectures used in objects -- then much of the language-, UI-, and OS based discussions on this thread are really quite moot.
One of the many really, really neat ideas in Smalltalk is how control structures, which in most other languages are special language features, are just plain old messages and implemented in the library, not in the language.

So the for ... in loop in Swift is just the do: message sent to a collection, and the keyword syntax makes this natural:

for event in lines {
lines do: { :event |

Note how making loops regular like this also makes the special concept of "loop variable" disappear. The "loop variable" is just the block argument. And I just realized the same would go for a not-nil result of a nil test.

Anyway, if "loops" are just messages, it's easy to add a method implementing iteration to some other entity, for example a stream, the way that I did. (Smalltalk streams also support the iteration messages).

And when you can easily make stream processing, which can handle asynchrony naturally and easily, just as convenient as imperative programming, you don't need async/await, which tries to make asynchronous programming look like imperative programming in order to make it convenient.


  1. This comment has been removed by the author.

  2. swift has lots of whiz-bang features but one of the dissapointing aspects is defintely the lack of true metabject protocol where all these features could be elegantly implemented as you imply here....

    these kinds of questions pop out to me...

    1. could millions of lines of c++ be replaced with an elegant metaobject system?


    2. would such a system be as performant (at building and running) as the c++ that powers the swift compiler?

    i have heard it said the swift complier and language implementation is to protect invariants, be type-safe and memory-safe... would such a metaobject system be able to do that? or would it afford too much freedom to accomplish the goals of the swift project...?

  3. Yes, the meta system is a great frame for this, and deserving of another post.

    To first answer your questions:

    1. Could millions of lines of C++ be replaced?

    Yes, I am quite certain of this.

    2. Would that be as performant as Swift (building and running)?

    God, I hope not. ;-). Swift code is actually pretty slow to execute, and comically slow to compile. Well, sometimes also comically slow to execute. My goal is similar to Jonathan Blow's Jai, for medium-sized projects to compile in under a second, but additionally for such whole-project compiles to rarely be necessary.

    For compilation times, the idea is just to keep the compiler simple. Most of the things that Swift is doing don't actually help that much.

    For execution time: many of the techniques I describe in my book can be helped with language support, the idea is to provide that support, so something that actually does help to make programs fast.

    In terms of protecting invariants: that is a laudable goal, but the problem is that Swift protects the wrong invariants. The idea with Objective-S is that in order to do less metaprogramming, we need to support other architectural styles. And then
    "user" code can be more straightforward.

    That's the rough outline.