Embedding Objective-Smalltalk

Ilja just asked for embedded scripting-language suggestions, presumably for his GarageSale E-Bay listings manager, and so of course I suggested Objective-Smalltalk.

Unironically :-)

This is a bit scary. On the one hand, Objective-Smalltalk has been in use in my own applications for well over a decade and runs the http://objective.st site, both without a hitch and the latter shrugging of a Hacker News "Hug of Death" without even the hint of glitch. On the other hand, well, it's scary.

As for usability, you include two frameworks in your application bundle, and the code to start up and interact with the interpreter or interpreters is also fairly minimal, not least of all because I've been doing so in quite a number of applications now, so inconvenience gets whittled away over time.

In terms of suitability, I of course can't answer that except for saying it is absolutely the best ever. I can also add that another macOS embeddable Smalltalk, FScript, was used successfully in a number of products. Anyway, Ilja was kind enough to at least pretend to take my suggestion seriously, and responded with the following question as to how code would look in practice:

It is not obvious to me how such a custom script in Objective-SmallTalk would look like. Here is a JavaScript pseudo-code example that ends & restarts certain listings. How would this look like in Obj-ST? pic.twitter.com/UjbwUD0Q4b

— Ilja A. Iwas (@iljawascoding) May 13, 2020

I am only too happy to answer that question, but the answer is a bit beyond the scope of twitter, hence this blog post.

First, we can keep things very close to the original, just replacing the loop with a -select: and of course changing the syntax to Objective-Smalltalk.

---

runningListings := context getAllRunningListings.
listingsToRelist := runningListings select:{ :listing |
    listing daysRunning > 30 and: listing watchers < 3 .
}
ebay endListings:listingsToRelist ended:{ :ended | 
     ebay relistListings:ended relisted: { :relisted |
         ui alert:"Relisted: {relisted}".
     }
}

---

Note the use of "and:" instead of "&&" and the general reduction of sigils. Although I personally don't like the pyramid of doom, the keyword message syntax makes it significantly less odious.

So much in fact, that Swift recently adopted open keyword syntax for the special case of multiple trailing closures. Of course the mind boggles a bit, but that's a topic for a separate post.

So how else can we simplify? Well, the context seems a little unspecific, and getAllRunningListings a bit specialized, it probably has lots of friends that result from mapping a website with lots of resources onto a procedural interface.

Let's instead use URLs for this, so an ebay: scheme that encapsulates the resources that EBay lets us play with.

---

listingsToRelist := ebay:listings/running select:{ :listing |
    listing daysRunning > 30 and: listing watchers < 3 .
}
ebay endListings:listingsToRelist ended:{ :ended | 
     ebay relistListings:ended relisted: { :relisted |
         ui alert:"Relisted {relisted} listings".
     }
}

---

I have to admit I also don't really understand the use of callbacks in the relisting process, as we are waiting for everything to complete before moving to the next stage. So let's just implement this as plain sequential code:

---

listingsToRelist := ebay:listings/running select:{ :listing |
    listing daysRunning > 30 and: listing watchers < 3 .
}
ended := ebay endListings:listingsToRelist.
relisted := ebay relistListings:ended.
ui alert:"Relisted: {relisted}".

---

(In scripting contexts, Objective-Smalltalk currently allows defining variables by assigning to them. This can be turned off.)

However, it seems odd and a bit non-OO that the listings shouldn't know how to do stuff, so how about just having relist and end be methods on the listings themselves? That way the code simplifies to the following:

---

listingsToRelist := ebay:listings/running select:{ :listing |
    listing daysRunning > 30 and: listing watchers < 3 .
}
ended := listingsToRelist collect end.
relisted := ended collect relist.
ui alert:"Relisted: {relisted}".

---

If batch operations are typical, it probably makes sense to have a listings collection that understands about those operations:

---

listingsToRelist := ebay:listings/running select:{ :listing |
    listing daysRunning > 30 and: listing watchers < 3 .
}
ended := listingsToRelist end.
relisted := ended relist.
ui alert:"Relisted: {relisted}".

---

Here I am assuming that ending and relisting can fail and therefore these operations need to return the listings that succeeded.

Oh, and you might want to give that predicate a name, which then makes it possible to replace the last gobbledygook with a clean, "do what I mean" Higher Order Message. Oh, and since we've had Unicode for a while now, you can also use '←' for assignment, if you want.

---

extension EBayListing {
  -<bool>shouldRelist {
      self daysRunning > 30 and: self watchers < 3.
  }
}

listingsToRelist ← ebay:listings/running select shouldRelist.
ended ← listingsToRelist end.
relisted ← ended relist.
ui alert:"Relisted: {relisted}".

---

To my obviously completely unbiased eyes, this looks pretty close to a high-level, pseudocode specification of the actions to be taken, except that it is executable.

This is a nice step-by-step script, but with everything so compact now, we can get rid of the temporary variables (assuming the extension) and make it a one-liner (plus the alert):

---

relisted ← ebay:listings/running select shouldRelist end relist.
ui alert:"Relisted: {relisted}".

---

It should be noted that the one-liner came to be not as a result of sacrificing readability in order to maximally compress the code, but rather as an indirect result of improving readability by removing the cruft that's not really part of the problem being solved.

Although not needed in this case (the precedence rules of unary message sends make things unambiguous) some pipe separators may make things a bit more clear.

---

relisted ← ebay:listings/running select shouldRelist | end | relist.
ui alert:"Relisted: {relisted}".

---

Whether you prefer the one-liner or the step-by-step is probably a matter of taste.

Presenting (in) Objective-Smalltalk

2019 has been the year that I have started really talking about Objective-Smalltalk in earnest, because enough of the original vision is now in place.

My first talk was at the European Smalltalk User Group's (ESUG) annual conference in my old hometown of Cologne: (pdf)

This year's ESUG was was my first since Essen in 2001, and it almost seemed like a bit of a timewarp. Although more than half the talks were about Pharo, the subjects seemed mostly the same as back when: a bit of TDD, a bit of trying to deal with native threads (exactly the same issues I struggled with when I was doing the CocoaSqueak VM), a bit of 3D graphics that weren't any better than 3D graphics in other environments, but in Smalltalk.

One big topic was getting large (and very profitable) Smalltalk code-bases running on mobile devices such as iPhones. The top method was transpiling to JavaScript, another translating the VM code to JavaScript and then having that run off-the-shelf images. Objective-Smalltalk can also be put in this class, with a mix of interpretation and native compilation.

My second talk, I was at Germany's oldest Mac conference, Macoun in Frankfurt. The videos from there usually take a while, but here was a reaction:

danke @mpweiher für den super vortrag zu https://t.co/bllQH4qANs gestern auf der #macoun — wer einen blick auf die zukunft erhaschen wollte, der hätte deinen vortrag sehen sollen. pic.twitter.com/Dw4M4QHLNl

— oɯʎxou (@noxymo) October 5, 2019

"Anyone who wants a glimpse at the future should have watched @mpweiher's talk"

Aww, shucks, thanks, but I'll take it. :-)

I also had two papers accepted at SPLASH '19, one was Standard Object Out: Streaming Objects with Polymorphic Write Streams at the Dynamic Languages Symposium, the other was Storage Combinators at Onward!.

Anyway, one aspect of those talks that I didn't dwell on is that the presentations themselves were implemented in Objective-Smalltalk, in fact the definitions were Objective-Smalltalk expressions, complex object literals to be precise.

What follows is an abridged version of the ESUG presentation:

---

controller := #ASCPresentationViewController{
    #Name : 'ESUG Demo'.
    #Slides : #(

      #ASCChapterSlide { 
               #text : 'Objective-SmallTalk'.
               #subtitle : 'Marcel Weiher (@mpweiher)'
         }  ,

        #ASCBulletSlide{ 
             #title : 'Objective-SmallTalk'.
             #bullets : #( 
                'Embeddable SmallTalk language (Mac, iOS, Linux, Windows)',
                'Objective-C framework (peer/interop)',
                'Generalizes Objects+Messages to Components+Connectors',
                'Enable composition by solving Architectural Mismatch',
             )
        } ,
      #ASCBulletSlide{ 
             #title : 'The Gentle Tyranny of Call/Return'.
             #bullets : #( 
                'Feymnan: we name everything just a little wrong',
                'Multiparadigm: Procedural, OO and FP!',
                "Guy Steele: it's no longer about completion",
                "Oscar Nierstrasz: we were told we could just model the domain",
                "Andrew Black: good OO students antropmorphise the objects",
             )
        } ,

         #ProgramVsSystem { 
              #lightIntensities : #( 0.2 , 0.7 )
              
         }  ,


       #ASCSlideWithFigure{ 
             #delayInSeconds : 5.0.
             #title : 'Objects and Messages'.
             #bullets : #( 
                'Objective-C compatible semantics',
                'Interpreted and native-compiled',
                '"C" using type annotations',
                'Higher Order Messaging',
                'Framework-oriented development',
                'Full platform integration',
             )
        } ,
  

       #ASCBulletSlide{ 
             #title : 'Pipes and Filters'.
             #bullets : #( 
                'Polymorphic Write Streams (DLS ''19)',
                '#writeObject:anObject',
                'Triple Dispatch + Message chaining',
                'Asynchrony-agnostic',
                'Streaming / de-materialized objects',
                'Serialisation, PDF/PS (Squeak), Wunderlist, MS , To Do',
                'Outlook: filters generalise methods?',
            )
        } ,
 
       #ASCBulletSlide{ 
             #title : 'In-Process REST'.
             #bullets : #( 
                'What real large-scale networks use',
                'Polymorphic Identifiers',
                'Stores',
                'Storage Combinators',
                'Used in a number of applications',
             )
        } ,


       #ASCBulletSlide{ 
             #title : 'Polymorphic Identifiers'.
             #bullets : #( 
                'All identifiers are URIs',
                "var:hello := 'World!",
                'file:{env:HOME}/Downloads/site := http://objective.st',
                'slider setValueHolder: ref:var:celsius',
             )
        } ,

       #ASCBulletSlide{ 
             #title : 'Storage Combinators'.
             #bullets : #( 
                'Onward! ''19',
                'Combinator exposes + consumes REST interfaces',
                'Uniform interface (REST) enables pluggability',
                'Narrow, semantically tight interface enables intermediaries',
                '10x productivity/code improvments',
             )
        } ,


      #ImageSlide{ 
               #text : 'Simple Composed Store'.
               #imageURL : '/Users/marcel/Documents/Writing/Dissertation/Papers/StorageCombinators/disk-cache-json-aligned.png'.
               #xOffset : 2.0 .
               #imageScale : 0.8
         }  , 
      #ASCBulletSlide{ 
             #title : 'Outlook'.
             #bullets : #( 
                'Port Stores and Polymorphic Write Streams',
                'Documentation / Sample Code',
                'Improve native compiler',
                'Tooling (Debugger)',
                'You! (http://objective.st)',
             )
        }  ,


      #ASCChapterSlide { 
               #text : 'Q&A   http://objective.st'.
               #subtitle : 'Marcel Weiher (@mpweiher)'
         }  ,
      )
}. 

---

There are a number of things going on here:

• Complex object literals
• A 3D presentation framework
• Custom behavior via custom classes
• Framework-oriented programming

Let's look at these in turn.

Complex object literals

Objective-Smalltalk has literals for arrays (really: ordered collections) and dictionaries, like many other languages now. Array literals are taken from Smalltalk, with a hash and round braces: #(). Unlike other Smalltalks, entries are separated via commas, so #( 1,2,3) rather than #( 1 2 3 ). For dictionaries, I borrowed the curly braces from Objective-C, so #{}.

This gives us the ability to specify complex property lists directly in code. A common idiom in Mac/iOS development circles is to initialize objects from property lists, so something like the following:

---

presentation = [[MyPresentation alloc] initWithDictionary:aDictionary];

---

All complex object literals really do is add a little bit of syntactic support for this idiom, by noticing that the two respective character at the start of array and dictionay literals give us a space to put a name, a class name, between those two characters:

---

presentation := #MyPresentation{ ... };

---

This will parse the text between the curly brackets as a dictionary and then initialize a MyPresentation object with that dictionary using the exact -initWithDictionary: message given above. This may seem like a very minor convenience, and it is, but it actually makes it possible to simply write down objects, rather than having to write code that constructs objects. The difference is subtle but significant.

The benefit becomes more obvious once you have nested structures. A normal plist contains no specific class information, just arrays, dictionaries numbers and strings, and in the Objective-C example, that class information is provided externally, by passing the generic plist to a specific class instance.

(JSON has a similar problem, which is why I still prefer XML for object encoding.)

So either that knowledge must also be provided externally, for example by the implicit knowledge that all substructure is uniform, or custom mechanisms must be devised to encode that information inside the dictionaries or arrays. Ad hoc. Every single time.

Complex object identifiers create a common mechanism for this: each subdictionary or sub-array can be tagged with the class of the object to create, and there is a convenient and distinct syntax to do it.

A 3D presentation framework

One of the really cool wow! effects of Alan Kay's Squeak demos is always when he breaks through the expected boundaries of a presentation with slides and starts live programming and interactive sketching on the slide. The effect is verey similar to when characters break the "fourth wall", and tends to be strongest on the very jaded, who were previously dismissive of the whole presentation.

Alas, a drawback is that those presentations in Squeak tend to look a bit amateurish and cartoonish, not at all polished.

Along came the Apple SceneKit Team's presentations, which were done as Cocoa/SceneKit applications. Which is totally amazing, as it allows arbitrary programmability and integration with custom code, just like Alan's demos, but with a lot more polish.

Of course, an application like that isn't reusable, the effort is pretty high and interactivity low.

I wonder what we could do about that?

First: turn the presentation application into a framework (Slides3D). Second, drive that framework interactively with Objective-Smalltalk from my Workspace-like "Smalltalk" application: presentation.txt. After a bit of setup such as loading the framework (framework:Slides3D load.) and defining a few custom slide classes, it goes on to define the presentation using the literal shown above and then starts the presentation by telling the presentation controller to display itself in a window.

---

framework:Slides3D load.     
class ProgramVsSystem : ASCSlide {
   var code.
   var system.
   ...
}.
class ImageSlide : ASCSlide { 
     var text.
     var image.


      #ASCChapterSlide { 
               #text : 'Q&A   http://objective.st'.
               #subtitle : 'Marcel Weiher (@mpweiher)'
         }  ,
      )
}. 

controller := #ASCPresentationViewController{
    #Name : 'ESUG Demo'.
    #Slides : #(

      #ASCChapterSlide { 
               #text : 'Objective-SmallTalk'.
               #subtitle : 'Marcel Weiher (@mpweiher)'
         }  ,

       ...
      )
}. 
     
controller view openInWindow:'Objective-SmallTalk (ESUG 2019)'. 

---

Voilà: highly polished, programmatically driven presentations that I can edit interactively and with a somewhat convenient format. Of course, this is not a one-off for presentations: the same mechanism can be used to define other object hierarchise, including but not limited to interactive GUIs.

Framework-oriented programming

Which brings us to the method behind all this madness: the concept I call framework-oriented programming.

The concept is worth at least another article or two, but at its most basic boils down to: for goodness sake, put the bulk of your code in frameworks, not in an application. Even if all you are building is an application. One app that does this right is Xcode. On my machine, the entire app bundle is close to 10GB. But the actual Xcode binary in /Applications/Xcode.app/Contents/MacOS? 41KB. Yes, Kilobytes. And most of that is bookkeeping and boilerplate, it really just contains a C main() function, which I presume largely matches the one that Xcode generates.

Why?

Simple: an Apple framework (i.e.: a .framework bundle) is at least superficially composable, but a .app bundle is not. You can compose frameworks into bigger frameworks, and you can take a framework and use it in a different app. This is difficult to impossible with apps (and no, kludged-together AppleScript concoctions don't count).

And doing it is completely trivial: after you create an app project, just create a framework target alongside the app target, add that framework to the app and then add all code and resources to the framework target instead of to the app target. Except for the main() function. If you already have an app, just move the code to the framework target, making adjustments to bundle loading code (the relevant bundle is now the framework and no longer the app/main bundle). This is what I did to derive Slides3D from the WWDC 2013 SceneKit App.

What I've described so fa is just code packaging. If you also organize the actual code as an object-oriented framework, you will notice that with time it will evolve into a black-box framework, with objects that are created, configured and composed. This is somewhat tedious to do in the base language (see: creating Views programmatically), so the final evolutionary step is considered a DSL (Hello, SwiftUI!). However, most of this DSL tends to be just creating, configuring and connecting objects. In other words: complex object literals.

Protocol-Oriented Programming is Object-Oriented Programming

Crusty here, I just saw that my young friend Dave Abrahams gave a talk that was based on a little keyboard session we had just a short while ago. Really sharp fellow, you know, I am sure he'll go far someday, but that's the problem with young folk these days: they go rushing out telling everyone what they've learned when the lesson is only one third of the way through.

You see, I was trying to impart some wisdom on the fellow using the old Hegelian dialectic: thesis, antithesis, synthesis. And yes, I admit I wasn't completely honest with him, but I swear it was just a little white lie for a good educational cause. You see, I presented ADT (Abstract Data Type) programming to him and called it OOP. It's a little ruse I use from time to time, and decades of Java, C++ and C# have gone a long way to making it an easy one.

Thesis

So the thesis was simple: we don't need all that fancy shmancy OOP stuff, we can just use old fashioned structs 90% of the time. In fact, I was going to show him how easy things look in MC68K assembly language, with a few macros for dispatch, but then thought better of it, because he might have seen through my little educational ploy.

Of course, a lot of what I told him was nonsense, for example OOP isn't at all about subclassing, for example the guy who coined the term, Alan I think, wrote: "So I decided to leave out inheritance as a built-in feature until I understood it better." So not only isn't inheritance not the defining feature of OOP as I let on, it actually wasn't even in the original conception of the thing that was first called "object-oriented programming".

Absolute reliance on inheritance and therefore structural relationships is, in fact, a defining feature of ADT-oriented programming, particularly when strong type systems are involved. But more on that later. In fact, OOP best practices have always (since the late 80ies and early 90ies) called for composition to be used for known axes of customization, with inheritance used for refinement, when a component needs to be adapted in a more ad-hoc fashion. If that knowledge had filtered down to young turks writing their master's thesis back in what, 1997, you can rest assured that the distinction was well known and not exactly rocket science.

Anyway, I kept all that from Dave in order to really get him excited about the idea I was peddling to him, and it looks like I succeeded. Well, a bit too well, maybe.

Antithesis

Because the idea was really to first get him all excited about not needing OOP, and then turn around and show him that all the things I had just shown him in fact were OOP. And still are, as a matter of fact. Always have been. It's that sort of confusion of conflicting truth seeming ideas that gets the gray matter going. You know, "sound of one hand clapping" kind of stuff.

The reason I worked with him on a little graphics context example was, of course, that I had written a graphics context wrapper on top of CoreGraphics a good three years ago. In Objective-C. With a protocol defining the, er, protocol. It's called MPWDrawingContext and live on github, but I also wrote about it, showed how protocols combine with blocks to make CoreGraphics patterns easy and intuitive to use and how to combine this type of drawing context with a more advanced OO language to make live coding/drawing possible. And of course this is real live programming, not the "not-quite-instant replay" programming that is all that Swift playgrounds can provide.

The simple fact is that actual Object Oriented Programming is Protocol Oriented Programming, where Protocol means a set of messages that an object understands. In a true and pure object oriented language like Smalltalk, it is all that can be, because the only way to interact with an object is to send messages. Even if you do simple metaprogramming like checking the class, you are still sending a message. Checking for object identity? Sending a message. Doing more intrusive metaprogramming like "directly" accessing instance variables? Message. Control structures like if and while? Message. Creating ranges? Message. Iterating? Message. Comparing object hierarchies? I think you get the drift.

So all interacting is via messages, and the set of messages is a protocol. What does that make OO? Say it together: Protocol Oriented Programming.

Synthesis

So we don't need objects when we have POP, but at the same time POP is OOP. Confused? Well, that's kind of the point of a good dialectic argument.

One possible solution to the conflict could be that we don't need any of this stuff. C, FORTRAN and assembly were good enough for me, they should be good enough for you. And that's true to a large extent. Excellent software was written using these tools (and ones that are much, much worse!), and tooling is not the biggest factor determining success or failure of software projects.

On the other hand, if you want to look beyond what OOP has to offer, statically typed ADT programming is not the answer. It is the question that OOP answered. And statically typed ADT programming is not Protocol Oriented Programming, OOP is POP. Repeat after me: OOP is POP, POP is OOP.

To go beyond OOP, we actually need to go beyond it, not step back in time to the early 90ies, forget all we learned in the meantime and declare victory. My personal take is that our biggest challenges are in "the big", meaning programming in the large. How to connect components together in a meaningful, tractable and understandable fashion. Programming the components is, by and large, a solved problem, making it a tiny bit better may make us feel better, but it won't move the needle on productivity.

Making architecture malleable, user-definable and thus a first class citizen of our programming notation, now that is a worthwhile goal and challenge.

Crusty out.

As always, comments welcome here and on HN.

Looking at a scripting language...and imagining the consequences

After thinking about the id subset and being pointed to WebScript, Brent Simmons imagines a scripting language. I have to admit I have been imagining pretty much the same language...and at some time decided to stop imagining and start building Objective-Smalltalk:

• Peer of Objective-C: objects are Objective-C objects, methods are Objective-C methods, added to the runtime and indistinguishable from the outside. "You can subclass UIViewController, or write a category on it."

  <void>alertView:alertView clickedButtonAtIndex:<int>buttonIndex
     self newGame.
     self view setNeedsDisplay.
  

  The example is from the site, it was copied from an actual program. As you can see, interoperability with the C parts of Objective-C is still necessary, but not bothersome.
• It has blocks:

  <void>deleteFile:filename
     thumbs := self thumbsView subviews.
     viewsToRemove := thumbs selectWhereValueForKey:'filename' isEqual:filename.
     aView := viewsToRemove firstObject.
  
     UIView animateWithDuration:0.4
            animations: [ aView setAlpha: 0.0. ]
            completion: [ aView removeFromSuperview. 
                          UIView animateWithDuration: 0.2
                                 animations: [ self thumbsView layoutSubviews. ]
                                 completion: [ 3  ]. 
                        ].
     url := self urlForFile:aFilename.
     NSFileManager defaultManager removeItemAtURL:url  error:nil.
     (self thumbsView afterDelay:0.4) setNeedsLayout.
  

  This example was also copied from an actual small educational game that was ported over from Flash.

  You also get Higher Order Messaging, Polymorpic Identifiers etc.

• Works with the toolchain: this is a a little more tricky, but I've made some progress...part of that is an llvm based native compiler, part is tooling that enables some level of integration with Xcode, part is a separate toolset that has comparable or better capabilities.

While Objective-Smalltalk would not require shipping source code with your applications, due to the native compiler, it would certainly allow it, and in fact my own BookLightning imposition program has been shipping with part of its Objective-Smalltalk source hidden its Resources folder for about a decade or so. Go ahead, download it, crack it open and have a look! I'll wait here while you do.

Did you have a look? The part that is in Smalltalk is the distilled (but very simple) imposition algorithm shown here.

---

	drawOutputPage:<int>pageNo
		isBackPage := (( pageNo / 2 ) intValue ) isEqual: (pageNo / 2 ).

		pages:=self pageMap objectAtIndex:pageNo.
		page1:=pages integerAtIndex:0.
		page2:=pages integerAtIndex:1.

		self drawPage:page1 andPage:page2 flipped:(self shouldFlipPage:pageNo).

	drawPage:<int>page1 andPage:<int>page2 flipped:<int>flipped
		drawingStream := self drawingStream.
		base := MPWPSMatrix matrixRotate:-90.

		drawingStream saveGraphicsState.
		flipped ifTrue: [ drawingStream concat:self flipMatrix ].
		width := self inRect x.

		self drawPage:page1 transformedBy:(base matrixTranslatedBy: (width * -2) y:0). 
		self drawPage:page2 transformedBy:(base matrixTranslatedBy: (width * -1) y:0). 
		
		drawingStream restoreGraphicsState.

Page imposition code

---

What this means is that any user of BookLightning could adapt it to suit their needs, though I am pretty sure that none have done so to this date. This is partly due to the fact that this imposition algorithm is too limited to allow for much variation, and partly due to the fact that the feature is well hidden and completely unexpected.

There are two ideas behind this:

1. Open Source should be more about being able to tinker with well-made apps in useful ways, rather than downloading and compiling gargantuan and incomprehensible tarballs of C/C++ code.
2. There is no hard distinction between programming and scripting. A higher level scripting/programming language would not just make developer's jobs easier, it could also enable the sort of tinkering and adaptation that Open Source should be about.

I don't think the code samples shown above are quite at the level needed to really enable tinkering, but maybe they can be a useful contribution to the discussion.