Maybe Visual Programming is The Answer. Maybe Not

Whenever discussing problems with programming today and potential solutions, invariably someone will pop up and declare that the problem is obviously the fact that programs are linear text and if only programming were visual, all problems would immediately disappear in some unspecified way.

I understand the attraction of visual programming, particularly for visual thinkers. However, it's not as if this hasn't been tried, with so far very limited success. Brad Myers, in his 1989 paper Taxonomies of Visual Programming gave, along with the titular taxonomy, a non-exhaustive summary of the problems, starting with visual languages in general:

• Difficulty with large programs or large data. Almost all visual representations are physically larger than the text they replace, so there is often a problem that too little will fit on the screen. This problem is alleviated to some extent by scrolling and various abstraction mechanisms.
• Need for automatic layout. When the program or data gets to be large, it can be very tedious for the user to have to place each component, so the system should lay out the picture automatically. Unfortunately, for many graphical representations, generating an attractive layout can be difficult, and generating a perfect layout may be intractable. For example, generating an optimal layout of graphs and trees is NP-Complete [95]. More research is needed, therefore, on fast layout algorithms for graphs that have good user interface characteristics, such as avoiding large scale changes to the display after a small edit.
• Lack of formal specification. Currently, there is no formal way to describe a Visual Language. Something equivalent to the BNFs used for textual languages is needed. This would provide the field with a ‘‘hard science’’ foundation, and may allow tools to be created that will make the construction of editors and compilers for Visual Languages easier. Chang [49] [96], Glinert [97] and Selker [98] have made attempts in this direction, but much more work is needed.
• Tremendous difficulty in building editors and environments. Most Visual Languages require a specialized editor, compiler, and debugger to be created to allow the user to use the language. With textual languages, conventional, existing text editors can be used and only a compiler and possibly a debugger needs to be written. Currently, each graphical language requires its own editor and environment, since there are no general purpose Visual Language editors. These editors are hard to create because there are no ‘‘editor-compilers’’ or other similar tools to help. The ‘‘compiler-compiler’’ tools used to build compilers for textual languages are also rarely useful for building compilers and interpreters for Visual Languages. In addition, the language designer must create a system to display the pictures from the language, which usually requires low-level graphics programming. Other tools that traditionally exist for textual languages must also be created, including pretty-printers, hard-copy facilities, program checkers, indexers, cross- referencers, pattern matching and searching (e.g., ‘‘grep’’ in Unix), etc. These problems are made worse by the historical lack of portability of most graphics programs.
• Lack of evidence of their worth. There are not many Visual Languages that would be generally agreed are ‘‘successful,’’ and there is little in the way of formal experiments or informal experience that shows that Visual Languages are good. It would be interesting to see experimental results that demonstrated that visual programming techniques or iconic languages were better than good textual methods for performing the same tasks. Metrics might include learning time, execution speed, retention, etc. Fortunately, preliminary results are appearing for the advantages of using graphics for teaching students how to program [36].
• Poor representations. Many visual representations are simply not very good. Programs are hard to understand once created and difficult to debug and edit. This is especially true once the programs get to be a non-trivial size.
• Lack of Portability of Programs. A program written in a textual language can be sent through electronic mail, and used, read and edited by anybody. Graphical languages require special software to view and edit; otherwise they can only be viewed on hard- copy.

In addition, most visual programming languages are "unstructured" in the software engineering sense. They

• use gotos and explicit transfer of control (often through wires),
• only have global variables,
• have no procedural abstraction,
• if they have procedural abstraction, they may not have parameters for the procedures,
• have no place for comments.

Furthermore, he notes that most visual languages don't interoperate with programs created in other languages, with some exceptions.

I am not saying that visual programming languages will not and cannot work, in fact, I am quite a fan myself. As these are specific problems, they probably can be solved, and I have a few ideas for solving some of them. However, before making claims that visual programming by itself is obviously and singularly the solution to our computing woes, please mention at least in passing how you've addressed the problems identified by Myers.

Thanks!

Protocol-Oriented Programming is Object-Oriented Programming

Crusty here, I just saw that my young friend Dave Abrahams gave a talk that was based on a little keyboard session we had just a short while ago. Really sharp fellow, you know, I am sure he'll go far someday, but that's the problem with young folk these days: they go rushing out telling everyone what they've learned when the lesson is only one third of the way through.

You see, I was trying to impart some wisdom on the fellow using the old Hegelian dialectic: thesis, antithesis, synthesis. And yes, I admit I wasn't completely honest with him, but I swear it was just a little white lie for a good educational cause. You see, I presented ADT (Abstract Data Type) programming to him and called it OOP. It's a little ruse I use from time to time, and decades of Java, C++ and C# have gone a long way to making it an easy one.

Thesis

So the thesis was simple: we don't need all that fancy shmancy OOP stuff, we can just use old fashioned structs 90% of the time. In fact, I was going to show him how easy things look in MC68K assembly language, with a few macros for dispatch, but then thought better of it, because he might have seen through my little educational ploy.

Of course, a lot of what I told him was nonsense, for example OOP isn't at all about subclassing, for example the guy who coined the term, Alan I think, wrote: "So I decided to leave out inheritance as a built-in feature until I understood it better." So not only isn't inheritance not the defining feature of OOP as I let on, it actually wasn't even in the original conception of the thing that was first called "object-oriented programming".

Absolute reliance on inheritance and therefore structural relationships is, in fact, a defining feature of ADT-oriented programming, particularly when strong type systems are involved. But more on that later. In fact, OOP best practices have always (since the late 80ies and early 90ies) called for composition to be used for known axes of customization, with inheritance used for refinement, when a component needs to be adapted in a more ad-hoc fashion. If that knowledge had filtered down to young turks writing their master's thesis back in what, 1997, you can rest assured that the distinction was well known and not exactly rocket science.

Anyway, I kept all that from Dave in order to really get him excited about the idea I was peddling to him, and it looks like I succeeded. Well, a bit too well, maybe.

Antithesis

Because the idea was really to first get him all excited about not needing OOP, and then turn around and show him that all the things I had just shown him in fact were OOP. And still are, as a matter of fact. Always have been. It's that sort of confusion of conflicting truth seeming ideas that gets the gray matter going. You know, "sound of one hand clapping" kind of stuff.

The reason I worked with him on a little graphics context example was, of course, that I had written a graphics context wrapper on top of CoreGraphics a good three years ago. In Objective-C. With a protocol defining the, er, protocol. It's called MPWDrawingContext and live on github, but I also wrote about it, showed how protocols combine with blocks to make CoreGraphics patterns easy and intuitive to use and how to combine this type of drawing context with a more advanced OO language to make live coding/drawing possible. And of course this is real live programming, not the "not-quite-instant replay" programming that is all that Swift playgrounds can provide.

The simple fact is that actual Object Oriented Programming is Protocol Oriented Programming, where Protocol means a set of messages that an object understands. In a true and pure object oriented language like Smalltalk, it is all that can be, because the only way to interact with an object is to send messages. Even if you do simple metaprogramming like checking the class, you are still sending a message. Checking for object identity? Sending a message. Doing more intrusive metaprogramming like "directly" accessing instance variables? Message. Control structures like if and while? Message. Creating ranges? Message. Iterating? Message. Comparing object hierarchies? I think you get the drift.

So all interacting is via messages, and the set of messages is a protocol. What does that make OO? Say it together: Protocol Oriented Programming.

Synthesis

So we don't need objects when we have POP, but at the same time POP is OOP. Confused? Well, that's kind of the point of a good dialectic argument.

One possible solution to the conflict could be that we don't need any of this stuff. C, FORTRAN and assembly were good enough for me, they should be good enough for you. And that's true to a large extent. Excellent software was written using these tools (and ones that are much, much worse!), and tooling is not the biggest factor determining success or failure of software projects.

On the other hand, if you want to look beyond what OOP has to offer, statically typed ADT programming is not the answer. It is the question that OOP answered. And statically typed ADT programming is not Protocol Oriented Programming, OOP is POP. Repeat after me: OOP is POP, POP is OOP.

To go beyond OOP, we actually need to go beyond it, not step back in time to the early 90ies, forget all we learned in the meantime and declare victory. My personal take is that our biggest challenges are in "the big", meaning programming in the large. How to connect components together in a meaningful, tractable and understandable fashion. Programming the components is, by and large, a solved problem, making it a tiny bit better may make us feel better, but it won't move the needle on productivity.

Making architecture malleable, user-definable and thus a first class citizen of our programming notation, now that is a worthwhile goal and challenge.

Crusty out.

As always, comments welcome here and on HN.

React.native isn't

While we're on the subject of terminological disasters, Facebook's react.native seems to be doing a good job of muddling the waters.

While some parts make use of native infrastructure, a lot do not:

1. uses views as drawing results, rather than as drawing sources, has a
2. parallel component hierarchy,
3. ListView isn't UITableView (and from what I read, can't be),
4. even buttons aren't UIButton instances,
5. doesn't use responder chain, but implements something "similar", and finally,
6. oh yes, JavaScript

None of this is necessarily bad, but whatever it is, it sure ain't "native".

What's more, the rationale given for React and the Components framework that was also just released echoes the misunderstandings Apple shows about the MVC pattern:

Just as a reminder: what's shown here with controllers pushing data to view at any time is not MVC, unless you use that to mean "Massive View Controller".

In Components and react.native, this "pushing of mutable state to the UI" is supposed to be replaced by "a (pure) function of the model". That's what a View (UIView or NSView) is, and what drawRect:: does. So next time you are annoyed by pushing data to views, instead of creating a whole new framework, just drag a Custom View from the palette into your UI and then implement the drawRect::. Creating views as a result of drawing (and/or turning components into view state mutations) is more stateful than drawRect::, not less.

Again, that doesn't mean it's bad or useless, it just means it isn't what it says on the tin. And that's a problem. From what I've heard so far, the most enthusiastic response to react.native has come from web developers who can finally code "native" apps without learning Objective-C/Swift or Java. That may or may not be useful (past experience suggests not), but it's something completely different from what the claims are.

Oh and finally, the "react" part seems to refer to "one-way reactive data flow", an even bigger terminological disaster that I will examine in a future post.

As always, comments welcome here or at HN

iPhone 6 Plus and The End of Pixels

It's been a long time coming. NeXTStep in 1989 featured DisplayPostscript, and therefore a device independent imaging model that meant you did not specify graphics in pixels, but rather in physical units. The default was a variant of the printer's point at 1/72nd of an inch, which happened to be close the the typical pixel resolution of displays at the time. However, 1 point never meant 1 pixel, it meant 1/72nd of an inch, and the combination of floating point coordinates and transformation matrices meant you could use pretty much any unit you wanted. When NeXT bought Apple, it brought this imaging model with it, although with some modifications due to Adobe intransigence about licensing and the addition of anti-aliasing.

However, despite the device-independent APIs, we still have pixel-based content, and "pixel-accurate" graphics. This has made less and less sense over time, with retina displays making pixel-accuracy moot (no more screen fonts!) scaled modes making it impossible and both iOS 7 and OS X 10.10 going for a more geometric look. Still, the design community has resisted, talking about @3 pixel art etc.

No more.

The iPhone 6 Plus has a 1920x1080 panel, but the simulator renders at 3x. These two resolutions don't match and so the pixels will need to be downsampled to the display resolution. Whether that is accomplished by downsampling pixel art (which happens automagically with Quartz and the proper device transform set) or as a separate step that downsamples the entire rendered framebuffer doesn't matter (much). Either way, there are no more "pixel perfect" pre-rendered designs.

Device-independent graphics, here we come at last. We're only a quarter century late.

Update: "Its 401 PPI display is the first display I’ve ever used on which, no matter how close I hold it to my eyes, I can’t perceive the pixels. " - John Gruber (emphasis mine)

More Objective-C Drawing Context Pleasantries

It's been a little over half a year since I first made my pleasant Objective-C drawing contextpublic, and I haven't been idle. In the process of retrofitting my own code to use MPWDrawingContext and adding more and more graphics (for example, I now do my icons in code), I've discovered a lot about making drawing with code a more pleasant experience.

Blocks

Blocks seem to be a really wonderful match for a graphics context, and most of the changes involve blocks in some way.

Bracketing operations such as gsave/grestore now have block versions, so the Objective-C block structure reflects the nesting:

    [context ingsave:^(Drawable c ){
        [c translate:@[ @130 ,@140]];
        [c setFont:[context fontWithName:@"ArialMT" size:345]];
        [c setTextPosition:NSMakePoint(0, 0)];
        [c show:@"\u2766"];
    }];

This is somewhat more compact than the plain code, which for correctness should also have a @try/@finally block wrapped around the basic drawing so exceptions don't mess up the graphics state stack.

    [context gsave];
    [context translate:@[ @130 ,@140]];
    [context setFont:[context fontWithName:@"ArialMT" size:345]];
    [context setTextPosition:NSMakePoint(0, 0)];
    [context show:@"\u2766"];
    [context grestore];

Similar for drawing shadows:

    [context withShadowOffset:NSMakeSize(0, -8 * scale) blur:12 * scale  color:[context  colorGray:0 alpha: 0.75] draw:^(Drawable c ){
        [[[c setFillColorGray:0.9 alpha:1.0] ellipseInRect:ellipseRect] fill];
    }];

Again, this seems a little clearer than having to explicitly set and unset, makes it harder to miss the end of the bracket when moving code around and remains exception-safe.

    [context sethadowOffset:NSMakeSize(0, -8 * scale) blur:12 * scale  color:[context  colorGray:0 alpha: 0.75]];
    [[[context setFillColorGray:0.9 alpha:1.0] ellipseInRect:ellipseRect] fill];
    [context clearShadow];

Stored, delayed and repeated drawing

You can create an object for later drawing by sending the -laterWithSize:(NSSize)size content:(DrawingBlock)commands message. For example, here is a simple diamond shape:

    NSSize diamondSize=NSMakeSize(16,16);
        id diamond = [context laterWithSize:diamondSize
                              content:^(id  context){
            id red = [context colorRed:1.0 green:0.0 blue:0.0 alpha:1.0];
            [context setFillColor:red];
            [[context moveto:diamondSize.width/2 :2] 
				lineto:diamondSize.width-2 :diamondSize.height/2];
            [[context lineto:diamondSize.width/2 :diamondSize.height-2]
				lineto:2 :diamondSize.height/2];
            [[context closepath] fill];
        }];

We can now draw this anywhere we want, and at any scale or orientation, using the -drawImage: message.

    [context drawImage:diamond];

You also have layerWitSize:content: and bitmapWithSize:content: messages if you want to specifically use CGLayer or CGImage instead, but using laterWithSize:content: preserves maximum quality, and it will automatically switch to a CGLayer when rendering to a PDF context in order to minimize PDF file size.

Patterns

I talked about patterns earlier. What I didn't mention then was that this is just the ability to use a stored set of drawing commands (see previous section) as a color:

    [context setColor:diamond];

I am not going to post the comparison to plain CG here, you can read it in the original Apple documentation.

I should note that this currently works for colored patterns, not for uncolored patterns, due to the fact that I haven't yet exposed color spaces. The basic process will be very similar.

Polymorphic object arguments

Path construction and graphics state messages with point arguments are now available in a version that takes a single object argument, in addition to the format with anonymous float arguments (moveto:(float)x :(float)y).

• moveto:
• lineto:
• translate:
• scale:

The single argument can be an Objective-C array:

    [context moveto:@[ @10, @20]];

Alternatively, any custom object that responds to count and either objectAtIndex:, (float)realAtIndex: or getReals:(float*)buffer length:(int)maxLen can be used. The scale: message can also take a single NSNumber and will treat that as uniform x and y scale.

Linecap parameters

Linecap parameters can now be set using distinct message:

• setlinecapRound
• setlinecapButt
• setlinecapSquare

Having multiple messages rather than a single message with a parameter probably seems odd, but it actually reduces the number of names involved, and these names are nicely scoped to this context. The constant strings or enums that are typically used have global scope and therefore tend to need ugly and hard-to-remember prefixes:

    [context setlinecapRound];

vs.

    [context setLinecap: kCGContextLinecapRound];

Future

Another reason to be as purely message-based as possible is that it makes bridging to other languages easier, for example for interactive drawing environments: Creating a badge (youtube).

I've also started experimenting with other outputs, for example creating a version of the same badge composed of CALayer objects using the same drawing commands. Other output should follow, for example web with SVG or HTML 5 Canvas or direct OpenGL textures.

I also want to finally add image processing operations both stand-alone and as chained drawing contexts, as well as getting more complex text layout options in there.

p.s.: now on hacker news

CGLayer performance and quality

The CoreGraphics CGLayer object seems to keep causing confusion. I hope I can clear that up a bit.

The intent of CGLayer was to have some sort of object for optimized repeated drawing, similar to how you would call a drawing procedure multiple times, have multiple instances of a view or simply draw the same PDF or bitmap image multiple times. The difference was supposed to be that the drawing layer could do secret magic "stuff" to optimize the repeated instance drawings. Not giving away the specific representation or optimizations performed means that whatever is done can be adapted both to different contexts and with changing technology.

Currently, that primarily means a texture stored on the graphics card, and that is and was one of the possible optimizations. However, the reasoning for such an opaque optimized object goes back further, at least to NeXTStep/Rhapsody: DisplayPostscript was implemented as a server process, and despite Mach's fast memory-remapping messages, shipping images back and forth between the client and server was not an optimization, and so you couldn't use (client side) bitmaps for optimized drawing. Well you could, but it wouldn't be optimized.

With Quartz becoming a client-side drawing library in Mac OS X, some of the reasons for these distinctions disappeared, but the distinctions (such as NSBitmapImageRep vs. NSCachedImageRep) remained. I think there were plans for CGLayer to become a kind of NSCachedImageRep on steroids, for example maintaining OpenGL display lists, but as far as I could tell, those plans never really panned out: in my tests drawing a CGLayer always looked the same as drawing a cached bitmap, and also performed similarly.

In 10.5, most of the accumulated cruft was cleaned up, NSBitmapImageRep for examples is now only a fairly thin wrapper around its underlying CGImage, and NSCachedImageRep was deprecated completely in 10.6, as it no longer has any advantages. With these changes, drawing a CGImage or a NSBitmapImageRep became as fast as possible, with CGImages stored on the graphics card. At this point CGLayer was essentially abandoned, and for drawing to the screen, you can just as easily use a NSBitmapImageRep, CGImage, UIImage etc.

So is CGLayer now completely useless? Not quite! It turns out that when drawing to a PDF context, CGLayer will generate a PDF Form object, which stores the original drawing commands (vectors, text, images). This is obviously much better than drawing a bitmap, both in terms of quality and in terms of file size and performance.

So if your plans include printing or generating a PDF, then I would recommend taking a look at CGLayer for repeated drawing of (vector) content.

CoreGraphics, patterns and resolution independence (not just) for retina displays

In a recent post with followup, Mark Granoff demonstrates how to intelligently deal with the need for higher resolution backgrounds by using CoreGraphics pattern images, particularly using the [UIColor colorWithPatternImage:] method. However, he does wonder why he still has to deal with retina resolution issues at some points in the code, when "…the docs say that CoreGraphics handles scaling issues automatically."

That's a good question, and the answer lies in the fact that the example uses pattern images and mask images, rather than CoreGraphics patterns and geometric primitives. Once you explicitly ask for bitmap representations, you will be dealing with pixels and different resolution. The clue is to avoid going to pixels as much and as long as possible. The doughnut shape, for example, can easily be achieved using basic geometry and a little knowledge of the Postscript/PDF fill rules.

Using the standard "nonzero-winding-number" rule, a doughnut effect can be achieved by having the two arcs that are nested inside each other drawn in opposite directions. That's one of the reasons the extra "clockwise" parameter exists.

---

  NSPoint centerPoint = NSMakePoint([view frame].size.width/2, 150);
  [context arcWithCenter:centerPoint
           radius:50 
           startDegrees:0
           endDegrees:360  
           clockwise:YES];
  [context arcWithCenter:centerPoint
           radius:100
           startDegrees:0
           endDegrees:360  
           clockwise:NO];
  [context fill];

---

(The code examples here use MPWDrawingContext for convenience, pure CoreGraphics code tends to be two to three times more verbose). The second way to achieve the doughnut would be to just use the even/odd fill rule, in which case the direction doesn't matter. matter.

Patterns can also be specified geometrically, or rather with callbacks to draw the pattern shape. Objective-C Blocks are really a perfect fit for specifying these sorts of callbacks, but were only introduced much later than the CoreGraphics pattern callback API. The following code shows how to specify the diamond pattern via an Objective-C block, courtesy of some glue API provided by MPWDrawingContext.

---

        NSSize patternSize=NSMakeSize(16,16);
        id diamond = [context laterWithSize:patternSize
                              content:^(id  context){
            id red = [context colorRed:1.0 green:0.0 blue:0.0 alpha:1.0];
            [context setFillColor:red];
            [[context moveto:patternSize.width/2 :2] 
				lineto:patternSize.width-2 :patternSize.height/2];
            [[context lineto:patternSize.width/2 :patternSize.height-2]
				lineto:2 :patternSize.height/2];
            [[context closepath] fill];
        }];
        [context setFillColor:diamond];
        [[context nsrect:[[self view] frame]] fill];

---

The "laterWithSize:content:" message creates a callback object that not only encapsulates the block, but also implements a -CGColor method so the callback can be used directly as a color in -setFillColor:.

With all the graphics specified using pure geometry, CoreGraphics can now do its thing and automatically handle varying device resolutions, wether it's a retina display or a zoomable interface or even print, all without ever having to deal with the different resolutions in code. Although I haven't tested it, the code should also use less memory, because it doesn't create potentially large temporary bitmaps, and for the cherry on top it's also a fraction of the code. CoreGraphics rules!

Forked project on github.

A Pleasant Objective-C Drawing Context

In a recent post, Peter Hosey muses On the API design of CGBitmapContextCreate and apparently finds it somewhat lacking. Apart from agreeing violently, I'd extend that not just to the rest of CoreGraphics, but really to the state of drawing APIs in OSX and iOS in general.

On OSX, we have the Cocoa APIs, which are at least somewhat OO, but on the other hand don't have a real graphics context, only the stunted NSGraphcisContext which doesn't actually allow you to do, you know, graphics. Instead we have a whole bunch of "use-once" objects that are typically instantiated, told to draw themselves to an implicit global drawing context and then discarded. This seems exactly backwards, I would expect an explicit graphics context that I can use to draw using a consolidated API.

That consolidated API is only available in the form of the CGContextRef and its associated functions, but alas that's a C API. So extremely long names, but without named parameters or useful polymorphism. Despite the fact that a graphics context is a quintessential example of OO, only Apple can create subclasses of a CGContext, and even then in only a sorta-kinda sort of way: CGBitmapContextCreate returns a CGContextRef that silently knows how to do things that other CGContextRefs do not. Same for CGPDFContextCreate.

On iOS, CoreGraphics is really your only choice, and double so if you want code that works on both iOS and OSX, but that means having to put up with the API, constantly converting between UI/NS objects and CoreGraphics and then there's the whole text mess.

Having been a great fan of algorithmic drawing every since my exposure to DisplayPostscript on the NeXT Cube, I found all of this sufficiently unsatisfactory that I decided to work on a solution, the first step of which is a lightweight drawing context that provides a reasonable Objective-C drawing API on top of CoreGraphics and works the same on OSX and iOS.

The result is on github: MPWDrawingContext, embedded in a version of Mark Gallagher's excellent IconApp AppKit drawing example. The same code is also used in the iOS target (a variant of Marcus Crafter's version of IconApp). I've also started using the context in a bunch of my own projects (that was the purpose after all) and so far it's made my graphics coding much more pleasant.

Another advantage, at least for me, is that bridging to scripting languages is automatic due to Objective-C's runtime information, whereas C functions have to have to be bridged separately and maintained, which is burdensome and doesn't necessarily get done.

At least one of Peter's problem with CGBitmapContextCreate is also solved: creating a bitmap context is as easy as

    [MPWCGDrawingContext rgbBitmapContext:NSMakeSize(595,842)]

Last not least: although it wasn't an explicit goal, there is also a pleasant reduction in code bulk.

	Lines	Characters	Lines %	Characters %
AppKit	146	6593	67.12%	76.69%
CGContext	134	7367	73.13%	68.63%
MPWDrawingContext	98	5056	-	-