My previous post titled Glue: the Dark Matter of Software may have given the impression that I see glue code as exclusively a problem. And I have to admit
that my follow-up (and reaction to Github's copilot) called Don't Generate Glue...Exterminate may not have done much to dissuade anyone of that
impression, but I just couldn't resist the Dalek reference.
However, I think it is important to remember that the fact that we have so much glue is a symptom of one of our
biggest successes in software technology. Even as recently as the late 80s and early 90s, we just
didn't have all that much to glue together, and software reuse was the holy grail, the unobtainium of computing, both in its
desirability and unobtainability.
Now we have reuse. Boy do we have reuse! We have so much reuse that we need tool support to manage all
the reuse. As far as I can tell, all new programming languages now come with such tooling, and are considered
incomplete until they have it.
The price of success is having a new set of problems, problems you never dreamed of before.
So how will we solve these problems?
Data format adaptation, as suggested by the O'Reilly article? Yes. Model-driven approaches that allow us or our tools
and languages to generate a lot of the more obvious adapter code? Sounds good, why not?
This one neat trick (click here!) that will automatically
solve all these problems? No.
Simpler components, written with composability and minimization of dependencies in mind? Surely.
Education, so developers get better at writing code that composes well without turning into
architecture astronauts? Very much yes.
However, my contention is that developers have a hard time with this in large part because our languages only
support implementing such glue, which is a start, but do not support expressing it directly, or
abstracting over it, encapsulating it, playing with it. So new linguistic
mechanisms like Objective-S are needed to help developers write better and
thus less glue code so we can better enjoy the fruits of our reusability success.
Whenever discussing problems with programming today and potential solutions, invariably someone will pop up and declare that the problem is obviously the fact that programs are linear text and if only programming were visual, all problems would immediately disappear in some unspecified way.
I understand the attraction of visual programming, particularly for visual thinkers. However, it's not as if this hasn't been
tried, with so far very limited success. Brad Myers, in his 1989 paper Taxonomies of Visual Programming gave, along with the titular taxonomy,
a non-exhaustive summary of the problems, starting with visual languages in general:
Difficulty with large programs or large data. Almost all visual representations are physically larger than the text they replace, so there is often a problem that too little will fit on the screen. This problem is alleviated to some extent by scrolling and various abstraction mechanisms.
Need for automatic layout. When the program or data gets to be large, it can be very tedious for the user to have to place each component, so the system should lay out the picture automatically. Unfortunately, for many graphical representations, generating an attractive layout can be difficult, and generating a perfect layout may be intractable. For example, generating an optimal layout of graphs and trees is NP-Complete [95]. More research is needed, therefore, on fast layout algorithms for graphs that have good user interface characteristics, such as avoiding large scale changes to the display after a small edit.
Lack of formal specification. Currently, there is no formal way to describe a Visual Language. Something equivalent to the BNFs used for textual languages is needed. This would provide the field with a ‘‘hard science’’ foundation, and may allow tools to be created that will make the construction of editors and compilers for Visual Languages easier. Chang [49] [96], Glinert [97] and Selker [98] have made attempts in this direction, but much more work is needed.
Tremendous difficulty in building editors and environments. Most Visual Languages require a specialized editor, compiler, and debugger to be created to allow the user to use the language. With textual languages, conventional, existing text editors can be used and only a compiler and possibly a debugger needs to be written. Currently, each graphical language requires its own editor and environment, since there are no general purpose Visual Language editors. These editors are hard to create because there are no ‘‘editor-compilers’’ or other similar tools to help. The ‘‘compiler-compiler’’ tools used to build compilers for textual languages are also rarely useful for building compilers and interpreters for Visual Languages. In addition, the language designer must create a system to display the pictures from the language, which usually requires low-level graphics programming. Other tools that traditionally exist for textual languages must also be created, including pretty-printers, hard-copy facilities, program checkers, indexers, cross- referencers, pattern matching and searching (e.g., ‘‘grep’’ in Unix), etc. These problems are made worse by the historical lack of portability of most graphics programs.
Lack of evidence of their worth. There are not many Visual Languages that would be generally agreed are ‘‘successful,’’ and there is little in the way of formal experiments or informal experience that shows that Visual Languages are good. It would be interesting to see experimental results that demonstrated that visual programming techniques or iconic languages were better than good textual methods for performing the same tasks. Metrics might include learning time, execution speed, retention, etc. Fortunately, preliminary results are appearing for the advantages of using graphics for teaching students how to program [36].
Poor representations. Many visual representations are simply not very good. Programs are hard to understand once created and difficult to debug and edit. This is especially true once the programs get to be a non-trivial size.
Lack of Portability of Programs. A program written in a textual language can be sent through electronic mail, and used, read and edited by anybody. Graphical languages require special software to view and edit; otherwise they can only be viewed on hard- copy.
In addition, most visual programming languages are "unstructured" in the software engineering sense. They
use gotos and explicit transfer of control (often through wires),
only have global variables,
have no procedural abstraction,
if they have procedural abstraction, they may not have parameters for the procedures,
have no place for comments.
Furthermore, he notes that most visual languages don't interoperate with programs created in other languages, with some
exceptions.
I am not saying that visual programming languages will not and cannot work, in fact, I am quite a fan myself. As these
are specific problems, they probably can be solved, and I have a few ideas for solving some of them. However, before
making claims that visual programming by itself is obviously and singularly the solution to our computing
woes, please mention at least in passing how you've addressed the problems identified by Myers.
Swift recently achieved ABI stability, meaning that we can now ship Swift binaries without having to ship the corresponding
Swift libraries. While it's been a long time coming, it's also great to have finally reached this point. However, it
turns out that this does not mean you can reasonably ship binary Swift frameworks, for reasons described very
well by Peter Steinberger of PSPDFKit and the good folks at instabug.
To reach this not-quite-there-yet state took almost 5 years, which is pretty much the total time NeXT shipped their hardware,
and it mirrors the state with C++, which is still not generally suitable for binary distribution of libraries. Objective-C
didn't have these problems, and as it turns out this is not a coincidence.
Software ICs
Objective-C was created specifically to implement the concept of Software-ICs. I briefly referenced the concept
in a previous article, and also mentioned its relationship to the scripted components pattern, but the comments indicated
that this is no longer a concept people are familiar with.
As the name suggests the intention was to bring the benefits the hardware world had
reaped from the introduction of the Integrated Circuits to
the software world.
It is probably hard to overstate the importance of ICs to the development of the computer industry. Instead of assembling
computers from discrete components, you could now put entire subsystem onto one component, and then compose these
subsystems to form systems. The interfaces are standardised pins, and the relationship between the outside interface
and the complexity hidden inside can be staggering. Although the socket of the CPU I am writing is a beast, with
1151 pins, the chip inside has a staggering 2.1 billion transistors. With
a ratio of one million to one, that's a very deep interface, even if you disregard the fact that the bulk of those pins are
actually voltage supply and ground pins.
The important point is that you do not have to, and in fact cannot, look inside the IC. You get the pins, very much a binary
interface, and the documentation, a specification sheet. With Software-ICs, the idea was the same: you get a binary,
the interface and a specification sheet. Here are two BYTE articles that describe the concepts:
A lot of what they write seems quaint now, for example a MailFolder that inherits from Array(!),
but the concepts are very relevant, particularly with a couple of decades worth of perspective and the new circumstances
we find ourselves in.
Although the authors pretty much equate Software-ICs with objects and object-oriented programming, it is a slightly
different form of object-oriented programming than the one we mostly use today. They do write about object/message
programming, similar to Alan Kay's note that 'The big idea is "messaging"'.
With messaging as the interconnect, similar to Unix pipes, our interfaces are sufficiently well-defined and dynamic
that we really can deliver our Software-ICs in binary form and be compatible, something our more static languages
like C++ and Swift struggle with.
ObjC is pretty awesome in how it manages to embed a “COM”. Swift doesn’t (currently) provide anything like it (and IMO it lost a chance not just using the ObjC runtime for that)
Virtually all systems based on static languages eventually grow an additional, separate and more dynamic component mechanism.
Windows has COM, IBM has SOM, Qt has signals and slots and the meta-object system, Be had BMessages etc.
In fact, the problem of binary compatibility of C++ objects was one of the reasons for creating COM:
Unlike C++, COM provides a stable application binary interface (ABI) that does not change between compiler releases.
COM has been incredibly successful, it enables(-ed?) much of the Windows and Office ecosystems. In fact, there is even
a COM implementation on macOS: CFPlugin, part of CoreFoundation.
CFPlugIn provides a standard architecture for application extensions. With CFPlugIn, you can design your application as a host framework that uses a set of executable code modules called plug-ins to provide certain well-defined areas of functionality. This approach allows third-party developers to add features to your application without requiring access to your source code. You can also bundle together plug-ins for multiple platforms and let CFPlugIn transparently load the appropriate plug-in at runtime. You can use CFPlugIn to add plug-in capability to, or write a plug-in for, your application.
That COM implementation is still in use, for example for writing Spotlight importers. However, there are, er, issues:
Creating a new Spotlight importer is tricky because they are based on CFPlugIn, and CFPlugIn is… well, how to say this diplomatically?… super ugly )-: One option here is to use Xcode 9 to create your plug-in based on the old template. Honestly though, I don’t recommend that because the old template… again, diplomatically… well, let’s just say that the old template lets the true nature of CFPlugIn shine through! (-:
Having written both Spotlight importers and even some COM component on Windows (I think it was just for testing), I can confirm
that COM's success is not due to the elegance or ease-of-use of the implementation, but due to the fact that having an interoperable,
stable binary interface is incredibly enabling for a platform.
That said, all this talk of COM is a bit confusing, because we already have NSBundle.
Apple uses bundles to represent apps, frameworks, plug-ins, and many other specific types of content.
So NSBundle already does everything a CFPlugin does and a lot more, but is really just a tiny
wrapper around a directory that may contain a dynamic shared library. All the interfacing, introspection and binary
compatibility features come automagically with Objective-C. In fact, NeXT had a Windows product called d'OLE that pretty
automagically turned Objective-C libraries into COM-comptible OLE servers (.NET has similar capabilities). Again, this is not a coincidence, the
Software-IC concept that Objective-C is based on is predicated on exactly this sort of interoperation scenario.
Objective-C is middleware with language features.
Frameworks and Microservices
To me, the idea of a Software-IC is actually somewhat higher level than a single object, I tend to see it at the
level of a framework, which just so happens to provide all the trappings of a self-contained Software-IC: a
binary, headers to define the interface and hopefully some documentation, which could even be provided in
the Resources directory of the bundle. In addition, frameworks are instances of NSBundle, so
they aren't limited to being linked into an application, they can also be loaded dynamically.
I use that capability in Objective-Smalltalk, particularly together with
the stsh the Smalltalk Scripting Shell. By loading
frameworks, this shell can easily be transformed into an application-specific scripting language. An
example of this is pdfsh, a shell for examining an manipulating PDF files using EGOS, the
Extensible Graphical Object System.
The same binary framework is also used in in PdfCompress, PostView and BookLightning.
With this framework, my record for creating a drag-and-drop applicaton to do something useful with a PDF file was 5 minutes,
and the only reason I was so slow was that I thought I had remembered the PDF dictionary entry...and had not.
Framework-oriented programming is awesome, alas it was very much deprecated by Apple for quite some time, in fact
even impossible on iOS until dynamic libraries were allowed. Even now, though, the idea is that you create an app, which
consists of all the source-code needed to create it (exception: Apple code!), even if some of that code may be organised
into framework units that otherwise don't have much meaning to the build.
Apps, however are not Software-ICs, they aren't the right packaging technology for reuse (AppleScript notwithstanding). And so iOS and macOS development shops routinely
get themselves into big messes, also known as the Big Ball of Mud architectural pattern.
Of course, there are reasons that things didn't quite work out the way we would have liked. Certainly
Apple's initial Mac OS X System Architecture book showed a much more flexible arrangement, with groups
of applications able to share a set of frameworks, for example. However, DLL hell is a thing, and so we got
a much more restricted approach where every app is a little fortress and frameworks in general and binary
frameworks in particular are really something for Apple to provide and for the rest to use. However, the
fact that we didn't manage to get this right doesn't mean that the need went away.
Swift has been making this worse, by strongly "suggesting" that everything be compiled together and leading to such wonderful
oxymorons as "whole module optimisation in debug mode", meaning without optimisation. That and not having a binary
modularity story for going on half a decade. The reason for compiling whole modules together is that the modularity
mechanism is, practically speaking,
very much source-code based, with generics and specialisation etc. (Ironically, Swift also does some pretty crazy things to
enable separate compilation, but that hasn't really panned out so far).
On the other hand, Swift's compiler is so slow that teams are rediscovering forms of framework-oriented programming as
a self-defense mechanism. In order to get feedback cycles down from ludicrously bad to just plain awful, they split
up their projects into independent frameworks that they then compile and run independently during development. So in
a somewhat roundabout way, Swift is encouraging good development practices.
I find it somewhat interesting that the industry is rediscovering variants of the Software-IC, in this case
on the backend in the form of Microservices.
Why do I say that Microservices are a form of Software-IC? Well, they are a binary unit of deployability,
fairly loosely coupled and dynamically typed. In fact, Fred George, one of the people who came up
with the idea refers to them as Smalltalk objects:
Of course, there are issues with this approach, one being that reliable method calls are replaced with unreliable
network calls. Stepping back for a second should make it clear that the purported benefits of Microservices also
largely apply to Software-ICs. At least real Software-ICs. Objective-C made the mistake of equating Software-ICs
with objects, and while the concepts are similar with quite a bit of overlap, they are not quite the same.
You certainly can use Objective-C to build and connect Software-ICs if you want to do that. It will also help you
in this endeavour, but of course you have to know that this is something you want. It doesn't do
this automatically and over time the usage of Objective-C has shifted to just a regular old object-oriented language,
something it is OK but not that brilliant at.
Interoperability
One of the interesting aspects of Microservices is that they are language-agnostic, it doesn't matter what language
a particular services is written in, as long as they can somehow communicate via HTTP(S).
This is another similarity to Software-ICs (and other middleware such as
COM, SOM, etc.): there is a fairly narrowly defined, simple interface, and as long as you can somehow service
that interface, you can play.
Microservices are pretty good at this, Unix filters probably the most extreme example and
just about every language and every kind of application on Windows can talk to and via COM. NeXT only ever sold
50000 computers, but in a short number of years the NeXT community had bridges to just about every language imaginable. There were
a number of Objective- languages, including Objective-Fortran. Apple
alone has around 140K employees (though probably a large number of those in retail), and there are over 2.8 million
iOS developers, yet the only language integration for Swift I know of is the Python support, and that took
significant effort, compiler changes and the original Swift creator, Chris Lattner.
This is not a coincidence. Swift is designed as a programming language, not as middleware with language features.
Therefore its modularity features are an add-on to the language, and try to transport the full richness of that
programming model. And Swift's programming model is very rich.
Objective-Swift
The middlewares I've talked about use the opposite approach, from the outside in. For SOM, it is described as such:
SOM allows classes of objects to be defined in one programming language and used in another, and it allows libraries of such classes to be updated without requiring client code to be recompiled.
So you define interfaces separately from their implementations. I am guessing this is part of the reason we have
@interface in Objective-C. Having to write things down twice can be a pain (and I've worked on projects
that auto-generated Objective-C headers from implementation files), but having a concrete
manifestation of the interface that precedes the implementation is also very valuable. (One of the reasons TDD is
so useful is that it also forces you to think about the interface to your object before you implement it).
In Swift, a class is a single implementation-focused entity, with its interface at best a second-class and
second-order effect. This makes writing components more convenient (no need to auto-generate headers...),
but connecting components is more complicated.
Which brings us back to that other complication, the lack of stable binary compatibility for libraries and frameworks.
One consequence of this is to write frameworks exclusively in Objective-C, which was particularly necessary before
ABI stability had been reached. The other workaround, if you have Swift code, is to have an
Objective-C wrapper as an interface to your framework. The fact that Swift interoperates transparently with the Objective-C runtime
makes this fairly straightforward.
Did I mention that Objective-C is middleware with language features?
So maybe this supposed "workaround" is actually the solution? Have Objective-C or Objective- as our message-oriented
middleware, the way it was always intended? Maybe with a bit of a tweak so that
it loses most of the C legacy and gains support for pipes and filters, REST/Microservices and other architectural
patterns?
It's been a long time coming. NeXTStep in 1989 featured DisplayPostscript, and therefore
a device independent imaging model that meant you did not specify graphics in pixels, but
rather in physical units. The default was a variant of the printer's point at 1/72nd of
an inch, which happened to be close the the typical pixel resolution of displays at the
time. However, 1 point never meant 1 pixel, it meant 1/72nd of an inch,
and the combination of floating point coordinates and transformation matrices
meant you could use pretty much any unit you wanted. When NeXT bought Apple, it
brought this imaging model with it, although with some modifications due to Adobe
intransigence about licensing and the addition of anti-aliasing.
However, despite the device-independent APIs, we still have pixel-based content,
and "pixel-accurate" graphics. This has made less and less sense over time,
with retina displays making pixel-accuracy moot (no more screen fonts!) scaled
modes making it impossible and both iOS 7 and OS X 10.10 going for a more
geometric look. Still, the design community has resisted, talking about
@3 pixel art etc.
No more.
The iPhone 6 Plus has a 1920x1080 panel, but the simulator renders at 3x. These
two resolutions don't match and so the pixels will need to be downsampled to
the display resolution. Whether that is accomplished by downsampling pixel art
(which happens automagically with Quartz and the proper device transform set)
or as a separate step that downsamples the entire rendered framebuffer doesn't
matter (much). Either way, there are no more "pixel perfect" pre-rendered
designs.
Device-independent graphics, here we come at last.
We're only a quarter century late.
Update: "Its 401 PPI display is the first display I’ve ever used on which, no matter how close I hold it to my eyes, I can’t perceive the pixels. " - John Gruber (emphasis mine)
Traviscautions against lazy initialization. Spooky coincidence: I just
managed to fix an extremely mysterious memory smasher in an Objective-C
program's exception handling code by moving the lazily initialized localization
code to app startup. Not sure wether localizing exceptions is really such a good idea
in the first place, but having the localization code run inside the exception handling
code does seem pushing it a bit.
