metablog

Objective-C language of the year 2011!

2012-01-11T04:05:00.001-08:00

After just barely missing out 2 years in a row (first to Go and then to Python), Objective-C finally managed to snatch Tiobe's programming language of the year award for 2011. Yay!

Objective-C managed a jump of 3.91% to 6.92% in 2011, and is now only a percent or so shy of C++ (8.06%), which incidentally just got edged out by C# (8.78%). It also zoomed ahead of Python (3.22%), which squeaked by Objective-C just barely last year to take top honors, but has now fallen below PHP and Visual BASIC.

Ruby, Perl and JavaScript are in the 1-3% range.

As TIOBE doesn't appear to keep old posts around the above link is likely to go stale in about a month. Here is the table:

Position Jan 2012	Position Jan 2011	Delta in Position	Programming Language	Ratings Jan 2012	Delta Jan 2011	Status
1	1		Java	17.479%	-0.29%	A
2	2		C	16.976%	+1.15%	A
3	6		C#	8.781%	+2.55%	A
4	3		C++	8.063%	-0.72%	A
5	8		Objective-C	6.919%	+3.91%	A
6	4		PHP	5.710%	-2.13%	A
7	7		(Visual) Basic	4.531%	-1.34%	A
8	5		Python	3.218%	-3.05%	A
9	9		Perl	2.773%	-0.08%	A
10	11		JavaScript	2.322%	+0.73%	A
11	12		Delphi/Object Pascal	1.576%	+0.29%	A
12	10		Ruby	1.441%	-0.34%	A
13	13		Lisp	1.111%	+0.00%	A
14	14		Pascal	0.798%	-0.12%	A
15	17		Transact-SQL	0.772%	+0.01%	A
16	24		PL/SQL	0.709%	+0.15%	A
17	20		Ada	0.634%	-0.05%	B
18	39		Logo	0.632%	+0.29%	B
19	25		R	0.609%	+0.07%	B
20	21		Lua	0.559%	-0.08%	B

And the graph:

Convention over configuration is subclassing?

2012-01-11T01:30:00.001-08:00

Having now played a little bit with Ruby on Rails, I am starting to wonder wether the much touted "convention over configuration" is really just a different implementation of subclassing. That is certainly what it felt like, and the Wikipedia article referenced states that "a developer only needs to specify unconventional aspects of the application."

Or perhaps slightly more accurately, both subclassing and convention over configuration would appear to be a form of programming by difference or "refinement".

Ruby (and Rails) scalability?

2011-12-09T10:37:00.001-08:00

Recently I wrote about Node.js performance, comparing it to my (still non-public, sorry!) Objective-C based libµhttp wrapper and Apache on my main development machine of the time, a MacBook Pro.

Node.js did really well on tasks that have lots of concurrent requests that are mostly waiting, did OK on basic static serving tasks and not so well on compute-intensive tasks.

Having developed an interest in minimal web-servers, I wondered how Sinatra and, by association, Ruby on Rails would do.

For Sinatra I used the scanty blog engine and the basic "Hello World" example:

require 'sinatra'

get '/hi' do
  "Hello World!"
end

For Ruby on Rails, I used the blog tutorial "out of the box", invoking it with "rails s" to start the server. In addition I also had RoR just serving a static file instead of the database-backed blog. All this on my new dev machine, a 2011 MacBook Air with 1.8 GHz Intel Core i7 and 4 GB of DRAM. I also discovered that httperf is a much better benchmark program for my needs than ab. I used it with 100 requests per connection, a burst length of 100 and a sufficient number of connections to get stable results without taking all day.

Platform	# requests/sec
Apache	5884
Sinatra Hello World	357
Ruby on Rails static page	312
Sinatra scanty blog	101
Ruby on Rails blog	17

This seems really slow, even when doing virtually nothing, and absolutely abysmal when using the typical RoR setup, serving records from an SQL database via templates. I distinctly remember my G5 serving in the thousands of requests/s using WebObjects 6-7 years ago.

Is this considered normal or am I doing something wrong?

The surprising thing about Objective-C...

2011-07-21T02:19:00.001-07:00

Alex Payne, when asked "Are you surprised at the popularity of any current languages?" during the preview of emerging languages for OSCON 2011's emerging languages track:

Alex Payne: I'm constantly surprised at the popularity and success of Objective-C. Almost everyone I know tells the same story about Objective-C: they started learning it and they hated it. They thought it was the worst of C and the worst of dynamic languages. And then eventually, they learned to love it. Most of the time, that software works pretty darn well, so who am I to judge? I'm pleasantly surprised by the continued success of Objective-C, and I think it should be taken as a lesson for the language designers out there.

This is echoed by the the first (and as of this writing only) comment to the post:

Alasdair Allan [18 July 2011 10:09 AM] I certainly agree with Alex about Objective-C, when I was initially learning the language I deeply despised it. Now I love it, and think it's one of the more elegant and powerful of the (many) languages I know. Definitely a lesson to language designers, do what you think is right and ignore the crowds. If you are right people will grow to love your language, just as soon as they figure it out.

I actually liked Objective-C pretty much from the start, but then again at that time (1986) there simply wasn't anything close that I had access to, and writing an Objective-C pre-processor and runtime on my Amiga was simply more feasible than a C++ frontend or a complete Smalltalk VM.

Modifying the sentiment expressed slightly, I'd say that from a theoretical point of view, I hate Objective-C and think it's a bad joke, a trainwreck. However, from practical experience, I love it and find it's one of the most productive languages out there for actually building stuff. And no, it's not just about the frameworks, as I've used Objective-C in non-NeXT, non-Apple environments where we had to build most of our own frameworks.

So while I support Alasdair's comment, my lesson for language designers is that our theory appears to not be particularly good at predicting reality. In other words: our theory ~~sucks~~ has many research opportunities.

Lazy Initialization

2011-04-27T11:22:00.001-07:00

Travis cautions against lazy initialization. Spooky coincidence: I just managed to fix an extremely mysterious memory smasher in an Objective-C program's exception handling code by moving the lazily initialized localization code to app startup. Not sure wether localizing exceptions is really such a good idea in the first place, but having the localization code run inside the exception handling code does seem pushing it a bit.

So couldn't agree more.

Speed matters

2011-03-12T03:25:00.001-08:00

Greg Linden recounts Marissa Mayer's talk at Web 2.0 showing how even very small decreases in performance have highly measurable impacts on users, and for web businesses on the bottom line.

The change was an increase from 10 to 30 search results, which was expected to produce an increase in user satisfaction, because users had asked for more search results. Instead, there was a completely unexpected and at first inexplicable 20% drop in traffic after the change was implemented. Only after some time did the team discover that the new results page took half a second longer to display, and in further testing they found that every 100 ms delay caused a measurable drop in clicks.

While I am not aware of similar research on desktop apps, I am sure that the same principle applies: speed matters, a lot; and it matters pre-consciously, that is, long before users will mention speed as an issue.

The experienced craftsman plans less

2011-02-17T09:15:00.001-08:00

Christopher Alexander via 37 signals:

The essence of this process is very fundamental indeed. We may understand it best by comparing the work of a fifty-year-old carpenter with the work of a novice. The experienced carpenter keeps going. He doesn’t have to keep stopping, because every action he performs, is calculated in such a way that some later action can put it right to the extent that it is imperfect now. What is critical here, is the sequence of events. The carpenter never takes a step which he cannot correct later; so he can keep working, confidently, steadily.
The novice by comparison, spends a great deal of his time trying to figure out what to do. He does this essentially because he knows that an action he takes now may cause unretractable problems a little further down the line; and if he is not careful, he will find himself with a joint that requires the shortening of some crucial member – at a stage when it is too late to shorten that member. The fear of these kinds of mistakes forces him to spend hours trying to figure ahead: and it forces him to work as far as possible to exact drawings because they will guarantee that he avoids these kinds of mistakes.
The difference between the novice and the master is simply that the novice has not learnt, yet, how to do things in such a way that he can afford to make small mistakes. The master knows that the sequence of his actions will always allow him to cover his mistakes a little further down the line. It is this simple but essential knowledge which gives the work of a master carpenter its wonderful, smooth, relaxed, and almost unconcerned simplicity.

Mac App Store won't let me buy apps: solution

2011-02-16T22:05:00.001-08:00

Just tried to buy an app via the Mac App Store and it was absolutely refusing to take my money. Various suggestions I've seen on the web such as clearing caches,resetting them via iTunes advanced preferences, rebooting, retrying, using slight variations of my account name all made no difference whatsoever.

The solution turned out to be manually signing in using the Store menu (manually sign out if you are already signed in). At that point I was allowed to update/verify my billing information and subsequent purchase attempts worked.

In previous attempts, I had not signed in manually, but rather had the App Store do the sign-in after I attempted to purchase.

So needs a little more work...

Train wreck management

2011-02-16T20:34:00.001-08:00

I wish the article Train Wreck Management by Poppendieck didn't strike such a chord.

Only 1 GHz

2011-02-14T21:15:00.001-08:00

Tim Bray wonders (well, wondered a while ago) about the excellent perceived performance of the iPad at 'only' 1Ghz.

Hmm

1 GHz actually seems like quite a lot to me. 1000 times an Apple ][, 40 times a NeXT, and the latter was driving a megapixel display. I guess we have gotten used to wasted cycles.

PhoneGeometry.h

2011-02-14T18:02:00.001-08:00

One things that's been tripping me up a bit when writing code that's supposed to be portable between iOS and Cocoa is the removal of NSPoint, NSSize, NSRect and their associated functions from Foundation in iOS. This is a real shame, because otherwise the Foundations are highly compatible.

One way to rectify this situation would be to start using CG* structs and functions on the desktop as well. However, this introduces a dependency on CoreGraphics that shouldn't be there for Foundation-based code.

My alternative is to standardize on NSPoint and friends, and map those to their CG alternatives on iOS. That way, I have minimized my dependencies, with only a small header file to pay for it: PhoneGeomtry.h

This is now part of MPWFoundation (on github).

//
//  PhoneGeometry.h
//  MPWFoundation
//
//  Created by Marcel Weiher on 11/11/10.
//  Copyright 2010-2011 Marcel Weiher. All rights reserved.
//


#if TARGET_OS_IPHONE
#ifndef PHONE_GEOMETRY
#define PHONE_GEOMETRY
#import <CoreGraphics/CoreGraphics.h>
typedef CGRect NSRect;
typedef CGPoint NSPoint;
typedef CGSize NSSize;
#define NSMakeRect  CGRectMake
#define NSMakePoint CGPointMake
#define NSMakeSize  CGSizeMake
#define NSEqualPoints  CGPointEqualToPoint
#define NSEqualRects   CGRectEqualToRect
#define NSIntersectsRect  CGRectIntersectsRect
static inline NSString *NSStringFromRect( CGRect r ) { return [NSString stringWithFormat:@"(%g,%g - %g,%g)",r.origin.x,r.origin.y,r.size.width,r.size.height]; }
static inline NSString *NSStringFromPoint( CGPoint p ) { return [NSString stringWithFormat:@"(%g,%g)",p.x,p.y]; }
static inline NSString *NSStringFromSize( CGSize s ) { return [NSString stringWithFormat:@"(%g,%g)",s.width,s.height]; }



#endif
#endif

Objective-XML and MPWFoundation now available on github

2011-01-31T17:26:00.001-08:00

By popular demand, both Objective-XML and MPWFoundation are now available on github. Which means I am finally learning git...so far it looks very nice and I am impressed with the performance.

Thanks to Todd Blanchard for providing the necessary impetus to learn git.

On switching away from CoreData

2011-01-18T08:38:00.001-08:00

Like Brent Simmons, I have a project where I am currently in the process of switching away from CoreData. Unlike Brent, and somewhat surprisingly given my proclivities, the reason is not performance.

Rather, the issues we have had with CoreData were additional complexity and more importantly gratuitous dependencies that, at least for our application, were not offset by noticeable benefits.

One of the most significant structural dependencies is that CoreData requires all your model classes to be subclasses of NSManagedObject, a class provided by CoreData. This may not seem like a big problem at first, but it gets in the way of defining a proper DomainModel, which should always be independent. The Java community actually figured this out a while ago, which is why there was a recent move to persistence frameworks supporting POJOs. (Of course, POOO doesn't have quite the same ring to it, and also the Java frameworks were a lot more heavy-handed than CoreData). The model is where your value is, it should be unencumbered. For example, when we started looking at the iPhone, there was no CoreData there, so we faced the prospect of duplicating all our model code.

In addition to initially not having CoreData, the iPhone app also used (and still uses) a completely different persistence mechanism (more feed oriented), and there were other applications where yet a third persistence mechanism was used (more document centric than DB-centric, with an externally defined file format). A proper class hierarchy would have had an abstract superclass without any reference to a specific persistence mechanism, but capturing the domain knowledge of our model. With CoreData, this hierarchy was impossible.

Since we had externally defined file formats in every case, we had to write an Atomic Store adapter and thus also couldn't really benefit from CoreData's change management. When we did the move, it turned out that the Atomic Store adapter we had written was significantly more code than just serializing and de-serializing the XML ourselves.

Another benefit of CoreData is its integration with Bindings, but that also turned out to be of little use to us. The code we managed to save with Bindings was small and trivial, whereas the time and effort to debug bindings when they went wrong or to customize them for slightly specialized needs was very, very large. So we actually ditched Bindings a long time before we got rid of CoreData.

So why was CoreData chosen in the first place? Since I wasn't around for that decision, I don't know 100%, but as far as I can tell it was mostly "Shiny Object Syndrome". CoreData and Bindings were new Apple technologies at the time, therefore they had to be used.

So are there any lessons here? The first would be to avoid Shiny Object Syndrome. By all means have fun and play around, but not in production code. Second and related is to really examine your needs. CoreData is probably highly appropriate in many contexts, it just wasn't in ours. Finally, it would be a huge improvement if CoreData were to support Plain Old Objective-C Objects. In fact, if that were the case we probably would not have to ditch it.

Little Message Dispatch

2011-01-10T12:55:00.001-08:00

Brent Simmons's recent notes on threading show a great, limited approach to threading that appears to work well in practice. If you haven't read it and are at all interested in threading on OS X or iOS, I suggest you head over there right now.

I feel much the same way, that is although I think Grand Central Dispatch is awesome, I simply haven't been able to justify spending much time with it, because it usually turns out that my own threading needs so far have been far more modest than what GCD provides. In fact, I find that an approach that's even more constrained than the one based on NSOperationQueue that Brent describes has been working really well in a number of projects.

Instead of queueing up operations and letting them unwind however, I just spawn a single I/O thread (at most a few) and then have that perform the I/O deterministically. This is paired with a downloader that uses the NSURL loading system to download any number of requests in parallel.

- (void)downloadNewsContent
{       
        id pool=[NSAutoreleasePool new];
        
        [[self downloader] downloadRequests:[self thumbnailRequests]];
        [[self downloader] downloadRequests:[self contentRequests]];
        [[self downloader] downloadOnlyRequests:[self imageRequests]];
        [pool release];
}

This loads 3 types of objects: first the thumbnails, then article content, then images associated with the articles. The sequencing is both deliberate (thumbs first, article images cannot be loaded before the article content is present) and simply expressed in the code by the well-known means of just writing the actions one after the other, rather than having those dependencies expressed in call-backs, completion blocks or NSOperation subclasses.

So work is done semi-sequentially in the background, while coordination is done on the main thread, with liberal use of performSelectorOnMainThread. Of course, I make that a little simpler with a couple of HOMs that dispatch messages to threads:

async runs the message on a new thread, I use it for long-running, intrinsically self contained work. It is equivalent to performSelectorInBackground: except for being able to take an arbitrary message.
asyncOnMainThread and syncOnMainThread are the equivalents of performSelectorOnMainThread, with the waitUntilDone flag set to YES or NO
afterDelay: sends he message after the specified delay

Here is a bit of code that shows how to have a dispatch a long-running thread and have it communicate status to the main thread.

-(void)loadSections {
	[[self asyncOnMainThread] showSyncing];
	[[[self sections] do] downloadNewsContent];
	[[self asyncOnMainThread] showDoneSyncing];
}
 ...
 -(IBAction)syncButtonClicked {
	[[self async] loadSections];
}

Brent sums it up quite well in his post:

Here’s the thing about code: the better it is, the more it looks and reads like a children’s book.

Yep.

Node.js performance? µhttpd performance!

2011-01-03T17:12:00.001-08:00

There's been a lot of hoopla recently about node.js. Being an object-head, I've always liked the idea of reactive (event-driven) web servers, after all, that means it's just like a typical object, sitting there waiting for something to happen and then reacting to it.

Of course, there is also a significant body of research on this topic, showing for example that user-level thread implementations tend to get very similar performance to event-based servers. There is also the issue that the purity of "no blocking APIs" is somewhat naive on a modern Unix, because blocking on I/O can happen in lots of different non-obvious places. At the very least, you may encounter a page-fault, and this may even be desirable in order to use memory mapped files.

In those cases, the fact that you have purified all your APIs makes no difference, you are still blocked on I/O, and if you've completely foregone kernel threads like node.js appears to do, then your entire server is now blocked!

Anyway, baving seen some interesting node.js benchmarking, I was obviously curious to see how my little embedded Objective-C http-server based on the awesome GNU microhttp stacked up.

The baseline is a typical static serving test, where Apache (out-of-the box configuration on Mac OS X client) serves a small static file and the two app servers serve a small static string.

Platform	# requests/sec
Static (via Apache)	6651.58
Node.js	5793.44
MPWHttp	8557.83

The sleep(2) example showed node.js at it's best. Here, each requests sleeps for 2 seconds before returning a small result.

Platform	# requests/sec
Static (via Apache)	-
Node.js	88.48
MPWHttp	47.04

The compute example is where MPWHTTP shines. The task is trivial, just counting up from 1 to 10000000 (ten million).

Platform	# requests/sec
Static (via Apache)	-
Node.js	9.62
MPWHttp	7698.65

So counting up, libµhttp with MPWHTTP is almost a thousand times faster? The reason is of course that such a simple task is taken care of by strength reduction in the optimizer, which replaces the loop 10 million increments with a single addition of 10 million. Cheating? On a benchmark, probably, but on the other hand that's the sort of benefit you get from a good optimizing compiler.

To make the comparison a little bit more fair, I added an xor with a randomly initialized value so that the optimizer could not remove the loop (verified by varying the loop count).

Platform	# requests/sec
Static (via Apache)	-
Node.js	9.62
MPWHttp	222.9

So still around 20 times faster. It was also using both cores of my Mac Book Pro, whereas node.js was limited to 1 core (so 10x single core speed difference).

Cross-checking on my 8 core Mac Pro gave the following results:

Platform	# requests/sec
Static (via Apache)	-
Node.js	10.72
MPWHttp	1011.86

Due to utilzing the available cores, MPWHTTP/µhttp is now 100 times faster than node.js on the compute-bound task.

In conclusion, I think it is fair to say that node.js succeeds admirably in a certain category of tasks: lots of concurrency, lots of blocked I/O, very little computation, very little memory use so we don't page fault. In more typical mixes with some concurrency, some computation some I/O and a bit of memory use (so chances of paging), a more balanced approach may be better.

This is not LISP

2010-12-02T09:50:00.001-08:00

Patrick Logan counters complaints about LISP's syntax with the following example of LISP's power:

(apply + (take 1000 (iterate inc 1)))

Hmm..I find the following preferable:

(1 to: 1000 ) reduce + 0

iPhone XML: from DOM to incremental SAX

2010-05-09T14:47:00.001-07:00

In my last post, I extended Ray Wenderlich's XML parser comparison to MAX, and performance seemed to be about 2x better than the nearest competitor, TBXML and around 7x faster than NSXMLParser, Cocoa Touch's built-in XML parser.

While the resulting code has been posted here, I haven't yet explained how I got there. First, I downloaded both Ray's project and Objective-XML 5.3. iPhone OS requires a bit of trickiness to get a reusable library, partly due to the fact that frameworks or dynamic libraries are not supported, partly due to the fact that simulator and device are not just different Xcode architectures of a single SDK, and not even just different SDKs, but actually different platforms. If anyone can tell me how to create a normal target that can compile/links for both the simulator and the device, I'd love to hear about it!

So, in order to compile for iPhoneOS, you'll need to select the iXmlKit target:

You'll need to build twice, changing Active SDK settings once to the the device and once to the simulator. You will then have two copies of the libiXmlKit.a library, one in the directory Release-iphoneos, the other in Release-iphonsimulator (both relative to your base build directory):

marcel@mpwls[LD]ls -la ~/programming/Build/Release-iphoneos/

-rw-r--r--   1 marcel  staff   521640 May  9 19:52 libiXmlKit.a

These two copies then need to be joined together using the lipo command to create a single library that can be used both with the simulator and the device.

lipo -create Release-iphoneos/libiXmlKit.a Release-iphonesimulator/libiXmlKit.a -output  Release/libiXmlKit.a

(Newer Objective-XML versions will have a shell-script target that automates this process). Once I had the fat libiXmlKit.a library, I created a new MAX group in the XMLPerformance project, and copied both the library and the MAX header file into that group:

I then created a new MAX Song parser class:

#import "iTunesRSSParser.h"
 
@interface MAXSongParser : iTunesRSSParser {
   id parser;
   NSDateFormatter *parseFormatter;
}
@end

The implementation is also fairly straightforward, with your basic init method:

 
#import "MAXSongParser.h"
#import "MPWMAXParser.h"
#import "Song.h"
 
@implementation MAXSongParser
 
-init
{
    self=[super init];
    parseFormatter = [[NSDateFormatter alloc] init];
    [parseFormatter setDateStyle:NSDateFormatterLongStyle];
    [parseFormatter setTimeStyle:NSDateFormatterNoStyle];
    MPWMAXParser *newParser=[MPWMAXParser parser];
    [newParser setUndefinedTagAction:MAX_ACTION_NONE];
    [newParser setHandler:self forElements:[NSArray arrayWithObjects:                 @"item",@"album",@"title",@"channel",@"rss",@"category",nil]
               inNamespace:nil prefix:@"" map:nil];
    [newParser setHandler:self forElements:[NSArray arrayWithObjects:
                 @"releasedate",@"artist",@"album",nil]
             inNamespace:@"http://phobos.apple.com/rss/1.0/modules/itms/" 
             prefix:@"" map:nil];
    parser=[newParser retain];
   return self;
}

The MAX parser is initialized in this init method. We define the elements we care about using the two "setHandler:forElements:inNamespace:prefix:map:" messages, one for each namespace we will be dealing with. In the default (RSS) namespace, we are interested in the "item", "album", "title", "channel", "rss" and "category" elements. In Apple's special "itms" namespace, we will handle "releasdate", "artist" and "album". Setting MAX_ACTION_NONE as the undefined tag action means that the parser will ignore elements not listed as interesting and all their sub-elements.

Songs are created in the -itemElement:... method, which turns the relevant child-elements of the item element into Song attributes:

-itemElement:children attributes:attributes parser:parser

   Song *song=[[Songalloc] init];

   [song setAlbum:[children objectForUniqueKey:@"album"]];

   [song setTitle:[children objectForUniqueKey:@"title"]];

   [song setArtist:[children objectForUniqueKey:@"artist"]];

   [song setAlbum:[children objectForUniqueKey:@"album"]];

   [song setCategory:[children objectForUniqueKey:@"category"]];

   [song setReleaseDate:[parseFormatter dateFromString:[children objectForUniqueKey:@"releasedate"]] ];

   return song;

Two more methods make the actual parse process complete: <channel> elements have one or more <item> elements, so we want to return all of them, using "objectsForKey:":

-channelElement:children attributes:attributes parser:parser

  return [[children objectsForKey:@"item"] retain];

Finally, there are a bunch of elements that we have defined interest in but treat identically...these can be handled using the "default" element handler:

-defaultElement:children attributes:attributes parser:parser

  return [[children lastObject] retain];

That concludes the routines that actually parse the XML into objects, now for kicking off the parser. With the timing code removed, the method is fairly straightforward:

- (void)downloadAndParse:(NSURL *)url {

   id pool=[NSAutoreleasePool new];

   [parserparse: [NSData dataWithContentsOfURL:url]];

   for ( id song in [parserparseResult] ) {

       [self performSelectorOnMainThread:@selector(parsedSong:)

             withObject:song waitUntilDone:NO];

   [pool release];

With the timing code, it all gets a bit messier:

- (void)downloadAndParse:(NSURL *)url {

   id pool=[NSAutoreleasePool new];

   [self performSelectorOnMainThread:@selector(downloadStarted) withObject:nil waitUntilDone:NO];

   NSData *data=[NSData dataWithContentsOfURL:url];

   [self performSelectorOnMainThread:@selector(downloadEnded) withObject:nil waitUntilDone:NO];

   NSTimeInterval start = [NSDate timeIntervalSinceReferenceDate];

   [parserparse:data];

   for ( id song in [parserparseResult] ) {

      [selfperformSelectorOnMainThread:@selector(parsedSong:) withObject:song waitUntilDone:NO];

   NSTimeInterval duration = [NSDatetimeIntervalSinceReferenceDate] - start;

   [selfperformSelectorOnMainThread:@selector(addToParseDuration:) withObject:[NSNumbernumberWithDouble:duration] waitUntilDone:NO];

   [selfperformSelectorOnMainThread:@selector(parseEnded) withObject:nilwaitUntilDone:NO];

[[NSURLCachesharedURLCache] removeAllCachedResponses];

[pool release];

This produces a non-incremental DOM-style parser, so we first download the data, then process it into a DOM and finally transfer the processed objects to the main thread for display. It differs from other DOM-syle XML parsers in that it actually produces domain objects (as a Domain Object Model parser arguably should), rather than a generic XML DOM that must then be converted to objects.

Turning the DOM-style parser into a SAX-stye parser is almost completely trivial. Instead of returning the Song objects at the end of itemElement:..

   [song setReleaseDate:[parseFormatterdateFromString:[children objectForUniqueKey:@"releasedate"]] ];

  return song;

we instead pass them to the delegate there and return nil so no tree is constructed:

   [song setReleaseDate:[parseFormatterdateFromString:[children objectForUniqueKey:@"releasedate"]] ];

[self performSelectorOnMainThread:@selector(parsedSong:)

withObject:song waitUntilDone:NO];

[song release];

return nil;

This means we can also remove the "channelElement" method and the for loop in downloadAndParse: that passed the Song objects to the main thread. This is a SAX-style parser (though it doesn't use the SAX methods and does produce domain objects), but it is still non-incremental because it first downloads all the data and then parses it. If we want to turn the SAX parser into an incremental parser that overlaps processing with downloading, there is one final tweak that fortunately further simplifies the downloadAndParse method (again with timing code removed):

- (void)downloadAndParse:(NSURL *)url {

   id pool=[NSAutoreleasePoolnew];

  [parserparseDataFromURL:url];

   [[NSURLCachesharedURLCache] removeAllCachedResponses];

   [pool release];

While this is probably best not only in terms of performance and responsiveness, but also in terms of code size, it doesn't play well with the XMLPerformance example, because there are no external measurement hooks that allow us to separate downloading from parsing for performance measurement purposes.

In addition, the XMLPerformance example is odd in that is both multi-threaded and measure real time rather than CPU time when measuring parse performance. The reason this is odd is that when both parsing and display are active, the scheduler can take the CPU away from the XML parsing thread at any time and switch to the display thread, but this time will be counted as XML parsing time by the measurement routines. This is obviously incorrect and penalizes all the incremental parsers, which is why Ray's comparison showed all the DOM parsers as performing better than the SAX parsers.

I hope these explanations show how to create different styles of parsers using MAX.

iPhone XML Performance Revisited

2010-05-02T12:12:00.001-07:00

Ray Wenderlich has done a great comparison of iPhone XML parsers, using the same sample I had looked at earlier in the context of responsiveness.

As Ray was comparing performance, my hobby-horse, I was obviously curious as to how MAX stacked up against all the upstart competition. Quite well, as it turns out (average of 5 runs on an iPad with 3.2):

Figure 1: total times (seconds)

MAX was about 50% faster than the closest competition, TBXML, at 0.43s vs. 0.61s.

However, the XMLPerformance sample is a bit weird in that it measures elapsed time, not CPU time, and is multi-threaded, updating the display as results come in.

In order to account for this overhead that has nothing to do with the XML parsers, I added a "Dummy" parser that doesn't actually parse anything, but rather just generates dummy Song entries as quickly as possible. This takes around 0.2 seconds all by itself. Subtracting this non-XML overhead from the total times yields the following results:

Figure 2: XML parse times (seconds)

When measuring just the XML parsers themselves, MAX is around twice as fast as the closest competition and seven times as fast as the built in NSXMLParser.

Sweet.

[Update] I forgot the link to where I had uploaded the source: XMLPerformance.tgz at http://www.metaobject.com/downloads/Objective-C/

This blog has moved

2010-04-22T19:31:00.001-07:00

This blog is now located at http://metaoblog.blogspot.com/. You will be automatically redirected in 30 seconds, or you may click here. For feed subscribers, please update your feed subscriptions to http://metaoblog.blogspot.com/feeds/posts/default.

Objective-XML 5.3

2010-01-24T15:54:00.001-08:00

New in this release:

Cocotron targets for Windows support.
XMLRPC support.
No longer uses 'private' API that was causing AppStore rejections for some iPhone apps using Objective-XML.
Support for numeric entitites.

http://www.metaobject.com/downloads/Objective-C/Objective-XML-5.3.tgz.

The Responder Chain is a Collection

2009-12-13T13:05:00.001-08:00

Timothy Wood voices some great ideas on modernizing the Cocoa responder chain. I'd like to venture that if we treat the Responder Chain as a simple collection, a singly-linked list, then such alternatives become easier to model and reason about.

NSEnumerator *responderEnumerator = [[firstResponder mapToNextObjectFromMessage] nextResponder];

I am currently abstracting from the intricate delegate mapping and other ops, these could be handled in an analog fashion. With the enumerator in place, we can obviously snapshot it to get the current state of the responder chain, and also log that.

NSArray *responders = [responderEnumerator allObjects];
NSLog(@"full responder chain:  %@",responders);

Now we can express both current features and possible variations of the Responder Chain architecture compactly as common collection operations. The current dispatch mechanism simply sends the message to the first object that is capable of responding. This corresponds to using the first object of a -select, which is expressed in the -selectFirst convenience method.

Current dispatch

[[[responders selectFirst] respondsToSelector:action] performSelector:action withObject:sender];

If I understood him correctly, Tim wants the objects in the responder chain to return an object that they would like to respond to the message. This turns the -select into a -collect (without a -collectFirst), but is otherwise very similar.

Tim's dispatch

possibleResponders = [[responders collect] responsibleTargetForAction:theAction sender:sender]];
[[possibleResponders objectAtIndex:0] performSelector:action withObject:sender];

I hope this does Tim's ideas justice, but I think the succinct formulation should make it easy to tell wether it does or not.

In terms of combining validation with target/action, I'd be somewhat wary of accidentally triggering actions when validation was meant, though I do appreciate the advantages of combining the two operations. I am not sure what value the block is adding over just having an additional BOOL parameter in the target/action method.

Combined action and validation

typedef BOOL IBAction;
-(IBAction)delete:sender  :(BOOL)onlyValidate
{
    NSArray *selection = [self selectedItems];

   if ( onlyValidate || [selection count] == 0 ) {
        return NO;
   }
   // perform the action
}
// or if you're worried about the naming issues
-(IBAction)delete:sender
{
}

Simple Objective-XML example

2009-12-09T08:38:00.001-08:00

Many times now, I've been asked about more Objective-XML examples. Here's a very simple one. It is adapted from Marcus Zarra's very helpful libxml and xmlreader tutorial. That tutorial shows how to parse a very simple XML format using libxml2.

The XML file parsed is the following:

<?xml version="1.0" encoding="UTF-8"?>

<root>

<person>

<name>John Doe</name>

<age>14</age>

</person>

<person>

<name>Mary Doe</name>

<age>14</age>

</person>

<person>

<name>John Smith</name>

<age>15</age>

</person>

</root>

It is parsed using at application startup using the following code:

- (void)applicationDidFinishLaunching:(NSNotification*)notification

{

NSString *path = [[NSBundle mainBundle] pathForResource:@"xmlExample" ofType:@"xml"];

NSData *xmlData = [NSData dataWithContentsOfFile:path];

xmlTextReaderPtr reader = xmlReaderForMemory([xmlData bytes],

[xmlData length],

[path UTF8String], nil,

(XML_PARSE_NOBLANKS | XML_PARSE_NOCDATA | XML_PARSE_NOERROR | XML_PARSE_NOWARNING));

if (!reader) {

NSLog(@"Failed to load xmlreader");

return;

}

NSString *currentTagName = nil;

NSDictionary *currentPerson = nil;

NSString *currentTagValue = nil;

NSMutableArray *people = [NSMutableArray array];

char* temp;

while (true) {

if (!xmlTextReaderRead(reader)) break;

switch (xmlTextReaderNodeType(reader)) {

case XML_READER_TYPE_ELEMENT:

//We are starting an element

temp = (char*)xmlTextReaderConstName(reader);

currentTagName = [NSString stringWithCString:temp

encoding:NSUTF8StringEncoding];

if ([currentTagName isEqualToString:@"person"]) {

currentPerson = [NSMutableDictionary dictionary];

[people addObject:currentPerson];

}

continue;

case XML_READER_TYPE_TEXT:

//The current tag has a text value, stick it into the current person

temp = (char*)xmlTextReaderConstValue(reader);

currentTagValue = [NSString stringWithCString:temp

encoding:NSUTF8StringEncoding];

if (!currentPerson) return;

[currentPerson setValue:currentTagValue forKey:currentTagName];

currentTagValue = nil;

currentTagName = nil;

default: continue;

}

}

NSLog(@"%@:%s Final data: %@", [self class], _cmd, people);

[self setRecords:people];

}

To parse it using MAX you need to add MPWXmlKit and MPWFoundation to your project, and then replace the code above with the following:

- (void)applicationDidFinishLaunching:(NSNotification*)notification

{

NSString *path = [[NSBundle mainBundle] pathForResource:@"xmlExample" ofType:@"xml"];

NSArray *people=[[MPWMAXParser parser] parsedDataFromURL:[NSURL fileURLWithPath:path]];

[self setRecords:people];

}

Some test-driven-development notes

2009-12-09T02:54:00.001-08:00

A couple of random points that might be of interest:

Code coverage tools

  if ( rare-condition ) {
      -is this code tested?-
  }

If you actually followed test-first, then the code in the rare if is definitely tested, because if there isn't failing test case for the rare condition, then there is no reason for the code or the test to exist.

Another objection could be that people won't follow the techniques. I haven't found this to be a big or recurring practical problem so far, and agile techniqes tend to be empirically driven. If you suspect that this is a problem you are seeing in your environment, running a code-coverage tool to put some data behind your suspicion may be a good idea.

Test before or test after?

Note that the solution to the code-coverage question above does not work if tests are written after the fact: in this case, the rare-case is likely not to be covered because it was written without being forced by a failing unit test.

Many if not most of the benefits of TDD are related to the way they shape the design of the code, all of these benefits obviously don't accrue if you've already designed or even written the code. In fact, if you ask the XP folks about it, they will tell you that TDD is not for ensuring quality, it is exclusively for helping with coding and design.

For example, figuring out how to test something will force you to come to a clarity about what the code is supposed to do that just writing the code usually does not.

Knowing that your tests cover your code (see above) allows you to do extremely radical refactorings at any point in the development process. The ability to refactor at any time in turn allows you to keep your initial designs simple without coding for anticipated changes. Not coding for anticipated changes that may not occur or may occur differently than you expect in turns allows you to move more quickly, which more than pays for the expense of the tests.

Furthermore, the tests force you to think how you can call the functionality you are about to implement, which means it shapes architecture towards simplicity, high cohesion and low-coupling.

Generating tests

Auto-generating tests for existing methods is a means of subverting the test-driven approach: there will be the appearance of testing, but with virtually none of the benefits. It is probably worse than not having tests, because in the latter case you at least know that you're not covered.

Is it a good way of starting with unit test coverage for legacy code? No. See the C2 wiki entry for a good explanation of how to approach this case. In short, start refactoring and adding unit tests when you actually need to touch the code, be it for new features or to fix defects that are scheduled to be fixed.

Blocked-C II

2009-11-10T00:25:00.001-08:00

Damien Pollet thinks my comparison between Objective-C blocks and HOM is not completely fair:

… from my (Smalltalk) experience, the block passed to #collect: is often not a single message send, but rather a small adhoc expression, for which it does not really make sense to define a named method. Or you might need both the element and its key/index… how does HOM deal with that?

These are certainly valid observations, and were some of the reasons that I didn't really think that much of HOM for the first couple of years after coming up with it back in 1997 or so. Since then, I've become less and less convinced that the problems raised are a big concern, for a number of reasons.

Inline vs. Named

One reason is that I actually looked at usage of blocks in the Squeak image, and found that the majority of blocks with at least one argument (so not ifTrue:, whileTrue: and other control structures) actually did contain just a single message send, and so could be immediately expressed as HOMs. Second, I noticed that there were a lot of fairly large (3+ LOC) blocks that should have been separate methods but weren't. That's when I discovered that the presence of blocks actually encourages bad code, and the 'limitation' of HOMs actually was encouraging better(-factored) code.

Of course, I wasn't particularly convinced by that line of reasoning, because it smelled too much like "that's not a bug, that's a feature". Until that is, I saw others with less vested interest reporting the same observation:

But are these really limitations? After using higher order messages for a while I've come to think that they are not. The first limitation encourages you move logic that belongs to an object into that object's implementation instead of in the implementation of methods of other objects. The second limitation encourages you to represent application concepts as objects rather than procedural code. Both limitations have the surprising effect of guiding the code away from a procedural style towards better object-oriented design.

My experience has been that Nat is right, having a mechanism that pushes you towards factoring and naming is better for your code that one that pushes you towards inlining and anonymizing.

Objective-C I

In fact, the Cocoa example that Apple gives for blocks illustrates this idea very well. They implement a "Finder like" sorting mechanism using blocks:

static NSStringCompareOptions comparisonOptions = NSCaseInsensitiveSearch | NSNumericSearch |
        NSWidthInsensitiveSearch | NSForcedOrderingSearch;
NSLocale *currentLocale = [NSLocale currentLocale];
 
NSComparator finderSort = ^(id string1, id string2) {
    NSRange string1Range = NSMakeRange(0, [string1 length]);
    return [string1 compare:string2 options:comparisonOptions range:string1Range locale:currentLocale];
};
 
NSLog(@"finderSort: %@", [stringsArray sortedArrayUsingComparator:finderSort]);

The block syntax is so verbose that there is no hope of actually defining the block inline, the supposed raison d'etre for blocks. So we actually need to take the block out-of-line and name it. So it looks suspiciously like an equivalent implementation using functions:

static NSStringCompareOptions comparisonOptions = NSCaseInsensitiveSearch | NSNumericSearch |
        NSWidthInsensitiveSearch | NSForcedOrderingSearch;
NSLocale *currentLocale = [NSLocale currentLocale];
 
static NSComparisonResult finderSort(id string1, id string2) {
    NSRange string1Range = NSMakeRange(0, [string1 length]);
    return [string1 compare:string2 options:comparisonOptions range:string1Range locale:currentLocale];
};
 
NSLog(@"finderSort: %@", [stringsArray sortedArrayUsingFunction:finderSort context:nil hint:nil]);

Of course, something as useful as a Finder-like comparison sort really deserves to be exposed and made available for reuse, rather than hidden inside one specific sort. Objective-C categories are just the mechanism for this sort of thing:

@implementation NSString(finderCompare)
-(NSSComparisonResult)finderCompare:(NSString*)string2) {
    NSRange myRange = NSMakeRange(0, [self length]);
    return [self compare:string2 options: NSCaseInsensitiveSearch | NSNumericSearch |
        NSWidthInsensitiveSearch | NSForcedOrderingSearch range:string1Range locale:[NSLocale currentLocale]];
}
@end
NSLog(@"finderSort: %@", [stringsArray sortedArrayUsingSelector:@selector(finderCompare:)]);

Note that some of these criticisms are specific to Apple's implementation of blocks, they do not apply in the same way to Smalltalk blocks, which are a lot less noisy.

Objective-C II

Objective-C has at least one other pertinent difference from Smalltalk, which is that it already contains control structures in the basic language, without blocks. (Of course, those control structures can also take blocks as arguments, but these are the different types of blocks that are delimited by curly braces and cannot be passed around as first class objects).

This means that in Objective-C, we already have the ability to do all the iterating we need, mechanisms such as blocks and HOM are mostly conveniences, not required building blocks. If we need indices, use a for loop. If we require keys, use a key-enumerator and iterate over that.

In fact, I remember when my then colleagues started working with a enum-filters, a HOM-precursor that's strikingly similar to the Google Toolbox's GTMSEnumerator+Filter.m. They really took to the elegance, but then also wanted to use it for various special cases. They laughed when they realized that those special-cases were actually already handled better by existing C control structures such as for-loops.

FP, HANDs and Aggregate Operations

While my dislike of blocks is easy to discount by the usual inventor's pride (your child must be ugly for mine to be pretty), that interpretation actually reverses the causation: I came up with HOM because I was never very fond of blocks. In fact, when I first encountered Smalltalk during my university years I was enthralled until I saw the iteration methods.

That's not to say that do:, collect: and friends were not light-years ahead of Algol-type control structures, they most definitely were and still are. Having some sort of higher-order mechanism is vastly superior than not having a higher-order mechanism. I do wish that "higher order mechanism" and "blocks" weren't used as synonyms quite as much, because they are not, in fact, synonymous.

When I first encountered Smalltalk blocks, I had just previously been exposed to Backus's FP, and that was just so much prettier! In FP functions are composed using functionals without ever talking about actual data, and certainly without talking about individual elements. I have always been on the lookout for higher levels of expression, and this was such a higher level. Now taking things down to "here's another element, what do you want to do with that" was definitely a step back, and quite frankly a bit of a let-down.

The fundamental difference I see is that in Smalltalk there is still an iteration, even if it is encapsulated: we iterate over some collection and then execute some code for each element. In FP, and in HOM, there is instead an aggregate operation: we take an existing operation and lift it up as applying to an entire collection.

This difference might seem contrived, but the research done with the HANDS system demonstrates that it is very real:

After creating HANDS, I conducted another user study to examine the effectiveness of three features of HANDS: queries, aggregate operations, and data visibility. HANDS was compared with a limited version that lacked these features. In the limited version, programmers were able to achieve the desired results but had to use more traditional programming techniques. Children using the full-featured HANDS system performed significantly better than their peers who used the limited version.

I also find this difference to be very real.

The difference between iterating with blocks and lifting operations to be aggregate operations also shows up in the fact that the lifting can be done on any combination of the involved parameters, whereas you tend to only iterate over one collection at a time, because the collection and the iteration are in focus.

Symmetry

Finally, the comparison to functional languages shows a couple of interesting asymmetries: in a functional language, higher order functions can be applied both to named functions and to anonymous functions. In essence, the higher order mechanism just takes functions and doesn't care wether they are named or not. Also the higher order mechanism uses the same mechanisms (functions) as the base system,

With block-based higher order mechanisms, on the other hand, we must make the argument an anonymous function (that's what a block is), and we cannot use a named function, bringing us back to the conundrum mentioned at the start that this mechanisms encourages bad code. Not only that, it also turns out that the base mechanism (messages and methods) is different from the higher order mechanism, which requires anonymous functions, rather than methods.

HOM currently solves only the latter part of this asymmetry, making the higher order mechanism the same as the base mechanism, that mechanism being messaging in both cases. However, it currently cannot solve the other asymmetry: where blocks support unnamed, inline code and not named code, HOM supports named but not unnamed code. While I think that this is the better choice in the larger number of cases, it would be nice to actually suport both.

One solution to this problem might be to simply support both blocks and Higher Order Messaging, but it seems to me that the more elegant solution would be to support inline definition of more-or-less anonymous methods that could then be integrated into the Higher Order Messaging framework.

Exploring the Weather Underground with Objective-XML and Objective-Smalltalk

2009-11-07T22:55:00.001-08:00

Having taken up various forms of flying last year, I have developed a strong interest in the weather, particularly wind information. While there are various web-sites with relevant information, for example Jeff Greenbaum's excellent Wind Conditions Page for Pacifica page, they don't really present the information quite the way I need, and also don't really work well on small mobile devices...

Fixing that should hopefully just be ASMOP. The Weather Underground fortunately has some reasonably well-documented XML APIs, let's see what they have to offer and wether we can get to the data we want.

First, let's fire up the interactive Smalltalk Shell (stsh) and load the Objective-XML framework.

marcel@spock[Projects]stsh
> context loadFramework:'MPWXmlKit'

Next, let's have a look at the raw XML returned by the Weather Underground APIs.

> urlstr:='http://api.wunderground.com/weatherstation/WXCurrentObXML.asp?ID=KCADALYC1'
> url:=NSURL URLWithString: urlstr
> NSString stringWithContentsOfURL:url.
>

Hmm...no result. (It turns out that Weather Underground checks the user agent and errors if it doesn't find one. The various convenience methods do not send a User Agent). Maybe curl can help?

>context addExternalCommand:'curl'.
>curl run:'http://api.wunderground.com/weatherstation/WXCurrentObXML.asp?ID=KCADALYC1'
<?xml version="1.0"?>
	<current_observation>
		<credit>Weather Underground Personal Weather Station</credit>
		<credit_URL>http://wunderground.com/weatherstation/</credit_URL>
		<image>
		<url>http://icons.wunderground.com/graphics/bh-wui_logo.gif</url>
		<title>Weather Underground</title>
		<link>http://wunderground.com/weatherstation/</link>
		</image>
		<location>
		<full>Mussel Rock, Daly City, CA</full>
		<neighborhood>Mussel Rock</neighborhood>
		<city>Daly City</city>
		<state>CA</state>
		<zip></zip>
		<latitude>37.667347</latitude>
		<longitude>-122.489342</longitude>
		<elevation>514 ft</elevation>
		</location>
		<station_id>KCADALYC1</station_id>
		<station_type>Fan-aspirated Davis Vantage Pro 2 Plus</station_type>
		<observation_time>Last Updated on November 7, 1:55 PM PST</observation_time>
		<observation_time_rfc822>Sat, 07 November 2009 21:55:21 GMT</observation_time_rfc822>
		<weather></weather>
		<temperature_string>56.9 F (13.8 C)</temperature_string>
		<temp_f>56.9</temp_f>
		<temp_c>13.8</temp_c>
		<relative_humidity>83</relative_humidity>
		<wind_string>From the NW at 15.0 MPH Gusting to 16.0 MPH</wind_string>
		<wind_dir>NW</wind_dir>
		<wind_degrees>313</wind_degrees>
		<wind_mph>15.0</wind_mph>
		<wind_gust_mph>16.0</wind_gust_mph>
		<pressure_string>30.07" (1018.2 mb)</pressure_string>
		<pressure_mb>1018.2</pressure_mb>
		<pressure_in>30.07</pressure_in>
		<dewpoint_string>51.8 F (11.0 C)</dewpoint_string>
		<dewpoint_f>51.8</dewpoint_f>
		<dewpoint_c>11.0</dewpoint_c>
		
		<heat_index_string></heat_index_string>
		<heat_index_f></heat_index_f>
		<heat_index_c></heat_index_c>
		
		<windchill_string></windchill_string>
		<windchill_f></windchill_f>
		<windchill_c></windchill_c>
		
		<solar_radiation>483.00</solar_radiation>
		<UV>2.5</UV>
		<precip_1hr_string>0.00 in (0.0 mm)</precip_1hr_string>
		<precip_1hr_in>0.00</precip_1hr_in>
		<precip_1hr_metric>0.0</precip_1hr_metric>
		<precip_today_string>0.01 in (0.0 mm)</precip_today_string>
		<precip_today_in>0.01</precip_today_in>
		<precip_today_metric>0.0</precip_today_metric>
		<history_url>http://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=KCADALYC1</history_url>
		<ob_url>http://www.wunderground.com/cgi-bin/findweather/getForecast?query=37.667347,-122.489342</ob_url>
	</current_observation>
<!-- 0.029:0 -->

Much better. Now let's see if we can parse that XML data into a Cocoa Property List.

> parser := MPWMAXParser parser.
> parser parsedDataFromURL: 'http://api.wunderground.com/weatherstation/WXCurrentObXML.asp?ID=KCADALYC1'
{
    UV = "2.5";
    credit = "Weather Underground Personal Weather Station";
    "credit_URL" = "http://wunderground.com/weatherstation/";
    "dewpoint_c" = "11.1";
    "dewpoint_f" = "51.9";
    "dewpoint_string" = "51.9 F (11.1 C)";
    "heat_index_c" =     {
    };
    "heat_index_f" =     {
    };
    "heat_index_string" =     {
    };
    "history_url" = "http://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=KCADALYC1";
    image =     {
        link = "http://wunderground.com/weatherstation/";
        title = "Weather Underground";
        url = "http://icons.wunderground.com/graphics/bh-wui_logo.gif";
    };
    location =     {
        city = "Daly City";
        elevation = "514 ft";
        full = "Mussel Rock, Daly City, CA";
        latitude = "37.667347";
        longitude = "-122.489342";
        neighborhood = "Mussel Rock";
        state = CA;
        zip =         {
        };
    };
    "ob_url" = "http://www.wunderground.com/cgi-bin/findweather/getForecast?query=37.667347,-122.489342";
    "observation_time" = "Last Updated on November 7, 1:55 PM PST";
    "observation_time_rfc822" = "Sat, 07 November 2009 21:55:51 GMT";
    "precip_1hr_in" = "0.00";
    "precip_1hr_metric" = "0.0";
    "precip_1hr_string" = "0.00 in (0.0 mm)";
    "precip_today_in" = "0.01";
    "precip_today_metric" = "0.0";
    "precip_today_string" = "0.01 in (0.0 mm)";
    "pressure_in" = "30.07";
    "pressure_mb" = "1018.2";
    "pressure_string" = "30.07\" (1018.2 mb)";
    "relative_humidity" = 83;
    "solar_radiation" = "482.00";
    "station_id" = KCADALYC1;
    "station_type" = "Fan-aspirated Davis Vantage Pro 2 Plus";
    "temp_c" = "13.9";
    "temp_f" = "57.0";
    "temperature_string" = "57.0 F (13.9 C)";
    weather =     {
    };
    "wind_degrees" = 342;
    "wind_dir" = NNW;
    "wind_gust_mph" = "24.0";
    "wind_mph" = "18.0";
    "wind_string" = "From the NNW at 18.0 MPH Gusting to 24.0 MPH";
    "windchill_c" =     {
    };
    "windchill_f" =     {
    };
    "windchill_string" =     {
    };
}

That looks good, we can see the wind information near the bottom of the output, with keys "wind_degrees" and "wind_mph". So let's grab the values for those keys using the collect Higher Order Message and -objectForKey:.

> (parser parsedDataFromURL:'http://api.wunderground.com/weatherstation/WXCurrentObXML.asp?ID=KCADALYC1' ) collect objectForKey: #( wind_mph wind_dir wind_string ) each. 
21.0
NW
From the NW at 21.0 MPH Gusting to 24.0 MPH

Almost what we wanted, except that we grabbed the wind direction as a string instead of the exact numeric direction. Easy fix:

> (parser parsedDataFromURL: 'http://api.wunderground.com/weatherstation/WXCurrentObXML.asp?ID=KCADALYC1' ) collect objectForKey: #( wind_mph wind_degrees wind_string ) each.
12.0
318
From the NW at 12.0 MPH Gusting to 24.0 MPH
>

Perfect. We have the wind speed, the direction and an informative text in case we want to display that.

Blocked-C

2009-11-06T00:24:00.001-08:00

Update: It appears that the original article has been removed, and has been superseded by material at: http://developer.apple.com/mac/articles/cocoa/introblocksgcd.html. The original article had more on the Cocoa block APIs and gave a refreshingly honest assessment of the for-loop vs. Block-iteration comparison.

While the news that Apple is adding blocks to C and Objective-C in the SnowLeopard time frame has been around for some time, a recent article shed some light on the actual API.

While there probably are some places where Objective-C blocks can be useful, I am not really impressed. In the following samples, red is used to show noise, meaning code that is just there to make the compiler happy.


NSMutableArray *filteredItems= [NSMutableArray array];
[items enumerateObjectsWithOptions:0 withBlock:
    ^(id item, NSUInteger index, BOOL *stop) {
        [filteredItems addObject:[item stringByAppendingString:@"suffix"]];
    }
];

As you can see, the version using blocks is very, very noisy, both syntactically and semantically, especially compared with the HOM version:

[[items collect] stringByAppendingString:@"suffix"];

No prizes for guessing which I'd prefer. To put some numbers on my preference: 234 characters vs. 52, 19 tokens vs. 3, 5 lines vs. 1. In fact, even a plain old C for-loop is more compact and less noisy than our "modern" blocked version:

NSMutableArray *filteredItems= [NSMutableArray array];
for (int i=0; i < [items count]; i++ ) {
     [filteredItems addObject:[items objectAtIndex:i] stringByAppendingString:@"suffix"];
    }
];

Why not Objective-C?

2009-11-05T23:08:00.001-08:00

Patrick Logan can't understand why projects use C++ rather than Ojective-C. Neither can I.

For the 95% (or more) of code that isn't performance sensitive, it gives you expressiveness very close to Smalltalk, and for the 5% or less that need high performance, it gets you the performance and predictability of C.

Cocoa(touch) memory management is as easy as 1-2-3

2009-09-20T00:13:00.001-07:00

There is a common misconception that Cocoa memory management is hard. It's not.

Use auto-generated accessors religiously
Release your instance variables in dealloc
Always use convenience methods to create objects

Wow, that wasn't too hard!

Objective-XML-5.0.1

2009-02-08T14:27:00.001-08:00

Just pushed out a minor bugfix release to Objective-XML-5.0:

Re-enabled character-set conversion code that had gotten disabled
Fixed a compile-error for some targets
Other minor improvements

Download here: http://www.metaobject.com/downloads/Objective-C/Objective-XML-5.0.1.tgz

Objective-XML 5.0

2009-01-25T14:21:00.001-08:00

I've just pushed out a new release of Objective-XML, with some pretty significant new features.

Incremental parsing

This feature, which was already discussed a little in an earlier post, is now available in an official release. In short, Objective-XML will now stream data from network data sources (specified by URL) and produce results incrementally, rather than reading all of the data first and then parsing it. This can make a huge difference in responsiveness and perceived performance for slow networks. CPU and memory consumption will be slightly higher because of extra buffering and buffer stitching required, so this should only be used when necessary.

Static iPhone library

Although Objective-XML has always been compatible with the iPhone, previous releases required copying the pre-requisite files into your project. This burden has now been eased by the inclusion of a static library target. You still need to copy the headers, either MPWMAXParser.h or MPWXmlParser.h (or both).

Unique keys

Previous releases of Objective-XML had an -objectForTag:(int)tag method for quickly retrieving attribute or element values.

enum songtags {
  item_tag=10, title_tag, category_tag	
};
...
  [parser setHandler:self forElements:[NSArray arrayWithObjects:@"item",@"title",@"category",nil]
          inNamespace:nil prefix:@"" map:nil tagBase:item_tag];
...
-itemElement:(MPWXMLAttributes*)children attributes:(MPWXMLAttributes*)attributes parser:(MPWMAXParser*)p
{
   ...
   [song setTitle:[children objectForTag:title_tag]];
   ...

Objective-XML adds an -objectForUniqueKey:aKey method that removes the need for these additional integer tags.

...
  [parser setHandler:self forElements:[NSArray arrayWithObjects:@"item",@"title",@"category",nil]
          inNamespace:nil prefix:@"" map:nil];
...
-itemElement:(MPWXMLAttributes*)children attributes:(MPWXMLAttributes*)attributes parser:(MPWMAXParser*)p
{
   ...
   [song setTitle:[children objectForUniqueKey:@"title"]];
   ...

In addition to providing faster access, the integer tags also served to disambiguate tag names that might occur in multiple namespaces. To handle these conflicts, there now is a -objectForUniqueKey:aKey namespace:aNamespace method. The namespace objects required for this disambiguation process are now returned by the -setHandler:... and -declareAttributes:... methods, which were previously void.

Default methods

One of the attractive features of DOM parsers is that they do something useful "out of the box": point a DOM parser at some XML and you get back a generic in-memory representation of that XML that you can then start taking apart. However, once you go down that road, you are stuck with the substantial CPU and memory overheads of that generic representation.

Streaming parser like SAX or MAX can be a lot more efficient, but it takes a lot more time and effort until achieving a first useful result. Default methods overcome this hurdle by also delivering an immediately useful generic representation without any extra work. Unlike a DOM, however, this generic representation can be incrementally replaced by more specialized and efficient processing later on.

Cocoa HTML parsing with Objective-XML

2009-01-20T10:39:00.001-08:00

Although Objective-XML's MPWSAXParser mostly provides NSXMLParser compatibility it also provides a number of useful additional features. Among these features is the ability to parse HTML files via the settings of two flags: enforceTagNesting and ignoreCase. By default, these are on and off, respectively, which gives you strict XML behavior. However, by setting enforceTagNesting to NO and ignoreCase to YES, you get a SAX parser that will happily and speedily process HTML.

Semantic Noise

2009-01-17T23:57:00.001-08:00

Martin Fowler and Gilad Bracha write about Syntactic Noise, making similar points and using similar typographical techniques as I did in my HOM paper.

By Syntactic Noise, what people mean is extraneous characters that aren't part of what we really need to say, but are there to satisfy the language definition. Noise characters are bad because they obscure the meaning of our program, forcing us to puzzle out what it's doing.

Couldn't have said it better myself, so I'll just quote Martin Fowler. Syntactic noise is one of the reasons I think neither the for(each) statement nor the blocks added to Objective-C are particularly good replacements for Higher Order Messaging.

newArray = [existingArray map:^(id obj){ return [obj  stringByAppendingString:@"suffix"]; }];
newArray = [[existingArray map] stringByAppendingString:@"suffix"]];

To me, that extra syntax is quite noisy, though the noise isn't, in fact, just syntactic. We also have to introduce, name and even correctly type a completely redundant stand-in (obj) that we don't really care about. Introducing extra entities is semantic noise. Apart from having to puzzle out what that extra entity is (and that it is, in fact, redundant) every time we read the code, it also brings us back to "element at a time" programming and thinking.

Simple HOM

2009-01-15T11:26:00.001-08:00

While it is good to see that Higher Order Messaging is still inspiring new work, I feel a bit guilty that part of that inspiration are sentiments such as the following:

"Still I have yet to find a simple implementation that I like and that does not use private methods. The last thing I want is a relying on classes which can break at any time."

Mea culpa.

While I did explain a bit why the current HOM implementation is a bit gnarly, code probably speaks more loudly than repeated mea-culpas.

So, without further ado, a really simple HOM implementation. An NSArray category provides the interface and does the actual processing:

@interface NSArray(hom)

-collect;

@end

@implementation NSArray(hom)

-(NSArray* )collect:(NSInvocation*)anInvocation
{
  NSMutableArray *resultArray=[NSMutableArray array];
  for (id obj in self ) {
    id resultObject;
    [anInvocation invokeWithTarget:obj];
    [anInvocation getReturnValue:&resultObject];
    [resultArray addObject:resultObject];
  }
  return resultArray;
}

-collect {
  return [HOM homWithTarget:self selector:@selector(collect:)];
}

@end

The fact that NSInvocation deals with pointers to values rather than values makes this a bit longer than it needs to be, but the gist is simple enough: iterate over the array, invoke the invocation, return the result.

That leaves the actual trampoline, which is really just an implementation detail for conveniently creating NSInvocation objects.


@interface HOM : NSProxy {
  id xxTarget;
  SEL xxSelector;
}

@end

@implementation HOM

-(void)forwardInvocation:(NSInvocation*)anInvocation
{
  [xxTarget performSelector:xxSelector withObject:anInvocation];
}

-methodSignatureForSelector:(SEL)aSelector
{
  return [[xxTarget objectAtIndex:0] methodSignatureForSelector:aSelector];
}

-xxinitWithTarget:aTarget selector:(SEL)newSelector
{
  xxTarget=aTarget;
  xxSelector=newSelector;
  return self;
}

+homWithTarget:aTarget selector:(SEL)newSelector
{
  return [[[self alloc] xxinitWithTarget:aTarget selector:newSelector] autorelease];
}

@end

This code compiles without warnings, does not use any private API, and runs on both Leopard and the iPhone. The Xcode project can be downloaded here.

iPhone XML performance

2009-01-11T16:01:00.001-08:00

Shortly after becoming an iPhone developer, I found a clever little piece of example code called XML Performance (login required). Having done some high performance XML processing code that works on the iPhone, I was naturally intrigued.

The example pits Cocoa's NSXMLParser against a custom parser based on libxml2, the benchmark is downloading a top 300 list of songs from iTunes.

More responsiveness using libxml2 instead of NSXMLParser

Based on my previous experience, I was expecting libxml2 to be noticeably faster, but with the advantage in processing speed being less and less important with lower and lower I/O data rates (WiFi to 3G to Edge), as I/O would start to completely overwhelm processing. Was I ever wrong!

While my expectations were technically correct for overall performance, I had completely failed to take responsiveness into account. Depending on the network selected, the NSXMLParser sample would appear to hang for 3 to 50 seconds before starting to show results. Needless to say, that is an awful user experience. The libxml example, on the other hand, would start displaying some results almost immediately. While it also was a bit faster in the total time taken, this effect seemed pretty insignificant compared to the fact that results were arriving continually pretty much during the entire time.

The difference, of course, is incremental processing. Whereas NSXMLParser's -initWithContentsOfURL: method apparently downloads the entire document first and then begins processing, the libxml2-based code in the sample downloads the XML in small chunks and processes those chunks immediately.

Alas, going with libxml2 has clear and significant disadvantages, with the code that uses libxml2 being around twice the size of the NSXMLParser-based code, at around 150 lines (non-comment, non-whitespace). If you have worked with NSXMLParser before, you will know that that is already pretty painful, so just imagine that particular brand of joy doubled, with the 150 lines of code giving you the simplest of parsers, with just 5 tags processed. Fortunately, there is a simpler way.

A simpler way: Objective-XML's SAX

Assuming you have already written a Cocoa-(Touch-)based parser using NSXMLParser, all you need to do is include Objective-XML in your projects and replace the reference to NSXMLParser with a reference to MPWSAXParser, everything else will work just as before. Well, the same except for being significantly faster (even faster than libxml2) and now also more responsive on slow connections due to incremental processing.

I have to admit that not having incremental processing was a "feature" Objective-XML shared with NSXMLParser until very recently, due to my not taking into account the fact that latency lags bandwidth. This silly oversight has now been fixed, with both MPWMAXParser and MPWSAXParser sporting URL-based parsing methods that do incremental processing.

So that's all there is to it, Objective-XML provides a drop-in replacement for NSXMLParser that has all the performance and responsiveness-benefits of a libxml2-based solution without the coding horror.

Even simpler: Messaging API for XML (MAX)

However, even a Cocoa version of the SAX API represents a pretty low-bar in terms of ease of coding. With MAX, Objective-XML provides an API that can do the same job much more simply. MAX naturally integrates XML processing with Objective-C messaging using the following two main features:

Clients get sent element-specific messages for processing
The parser handles nesting, controlled by the client

The following code for building Song objects out of iTunes <item> elements illustrates these two features:

-itemElement:(MPWXMLAttributes*)children attributes:(MPWXMLAttributes*)attributes parser:(MPWMAXParser*)p
{
  Song *song=[[Song alloc] init];
  [song setArtist:[children objectForTag:artist_tag]];
  [song setAlbum:[children objectForTag:album_tag]];
  [song setTitle:[children objectForTag:title_tag]];
  [song setCategory:[children objectForTag:category_tag]];
  [song setReleaseDate:[parseFormatter dateFromString:[children objectForTag:releasedate_tag]]];
  [self parsedSong:song];
  [song release];
  return nil;
}

MAX sends the -itemElement:attributes:parser: message to its client whenever it has encountered a complete <item> element, so there is no need for the client to perform string processing on tag names or manage partial state as in a SAX parser. The method constructs a song object using data from the <item> element's child elements which it then passes directly to the rest of the app via the parsedSong: message. It does not return an value, so MAX will not build a tree at this level.

Artist, album, title and category are the values of nested child elements of the <item> element. The (common) code shared by all these child-elements gets the character content of the respective elements and is shown below:

-defaultElement:children attributes:atrs parser:parser
{
	return [[children combinedText] retain];
}

Unlike the <item> processing code, which did not return a value, this method does return a value. MAX uses this return value to build a DOM-like structure which is then consumed by the next higher-level, in this case the -itemElement:attributes:parser: method shown above. Unlike a traditional DOM, the MAX tree structure is built out of domain-specific objects returned incrementally by the client.

These two pieces of sample code demonstrate how MAX can act like both a DOM parser or a SAX parser, controlled simply by wether the processing methods return objects (DOM) or not (SAX). They also demonstrated both element-specific and generic processing.

In the iTunes Song parsing example, I was able to build a MAX parser using about half the code required for the NSXMLParser-based example, a ratio that I have also encountered in larger projects. What about performance? It is slightly better than MPWSAXParser, so also somewhat better than libxml2 and significantly better than NSXMLParser.

Summary and Conclusion

The slightly misnamed XML Performance sample code for the iPhone demonstrates how important managing latency is for perceived end user performance, while showing only very little in terms of actual XML processing performance.

While ably demonstrating the performance problems of NSXMLParser, the sample code's solution of using libxml2 is really not a solution, due to the significant increase in code complexity. Objective-XML provides both a drop-in replacement for NSXMLParser with all the performance and latency benefits of the libxml2 solution, as well as a new API that is not just faster, but also much more straightforward than either NSXMLParser or libxml2.

Best of Show, MacWorld 2009

2009-01-11T14:22:00.001-08:00

Since I recently became the Mac tech lead for Livescribe, responsible for delivering the Mac desktop software, I am happy to report that not only did we meet all of our target dates, we also won Best of Show at MacWorld 2009.

Spending 3 days at the booth was both exhausting and rewarding, the enthusiasm exhibited by customers was absolutely mind-blowing.

More on one test class per class

2008-12-24T18:35:00.001-08:00

I knew there was some other place this recommendation had recently popped up: "Emergent Design" by Scott Bain, via Ralph Johnson.

Unit test the class

2008-12-20T15:45:00.001-08:00

Travis Griggs comes to the conclusion that unit test objects should map 1:1 to classes under test.

I agree.

In fact, I would go a bit further: tests should be an integral part of a class. While this helps avoid negative outcomes such as parallel class hierarchies or having code and tests diverge, it more importantly simplifies the test/code relationship and drives home the point that code is incomplete without its tests.

While I was working with JUnit on a reasonably large Java system, both finding a good place for a particular test and finding the tests for a specific class became quite burdensome after a while.

For this reason MPWTest simply asks classes to test themselves. Furthermore, only frameworks are tested, so the test tool simply loads each framework to test, enumerates the classes within that particular framework and then runs the tests it finds. TestCases and TestSuites are implicitly created from this structure, removing most of the administrative burdens of unit testing, and also any explicit dependence of the tests on the testing framework.

Having no dependencies on the testing framework makes it easier to ship tests in production code without having to also ship the testing framework. While this may sound odd at first, it avoids potential issues with code compiled for testing being different than code destined to be shipped, and further reinforces the idea that tests are an integral part of each class, rather than an optional add-on.

Binary XML

2008-10-11T18:44:00.001-07:00

Jimmy Zhang hits the nail on the head when he notes that parsing ASCII text is not the primary problem in XML performance, object allocation is. I was surprised by the same finding when I started working on Objective-XML around a decade ago.

Sean McGrath claims that Binary XML solves the wrong problem.

Yes and no: it doesn't help much with existing structures and parsing methods, but with the right methods, it can be extremely helpful!

Also: "...how weird is it that we have not moved on from the DOM and SAX in terms of "standard" APIs for XML processing?"

Code is not an asset

2008-08-09T21:39:00.001-07:00

Michael Feathers wonders how to go beyond technical debt. I have been wondering about this for some time, and I think the answer is to account for code as a liability, not an asset.

The functionality that the code has, the value it delivers is an asset, but the code itself is a liability. This easily explains how refactoring and removing code add value, as long as functionality is preserved.

Higher Order Messaging backgrounded

2008-04-20T13:57:00.001-07:00

Piers Cawley talks about RSpec's use of HOM:

It is, however, encouraging to see initiatives like Rspec which, through judicious use of higher order messages enables a much more fluent environment for writing tests:

I think that's the first time I've seen HOM used to explain something else, rather than being the object of attention itself. So HOM is starting to be seen as simply a part of the computing landscape, at least by some. Cool. HOM was never conceived of as an interesting thing by itself but rather as a (meta-)building block for building more expressive computational forms. RSpec looks exactly like one of those cool things HOM enables that I would never have come up with myself. I look forward to seeing more.

Not just high performance Objective-C

2008-04-19T21:42:00.001-07:00

Addendum to my article on implementing a high performance Postscript interpreter in Objective-C: not just is performance better, so is accuracy. Despite the fact that we are optimizing the heck out of the Objective-C objects we are using, they still give us encapsulation and polymorphism, allowing us to choose arbitrary representations. For example, most Postscript interpreters use a fixed-size value object (polymorphic in a C-union type of way) that constraints floating point precision to 32 bits. With Objective-C, we have no such constraints, so EGOS floats are actually 64 bit doubles, so running the modified benchmark below in PostView doesn't just yield the result 75% faster than Preview, it also produces it with 7 orders of magnitude less error. Not that that is necessarily important in Postscript, but it is a pleasant side effect and shows the power of combining performance with abstraction.

%!
  usertime
  1000 0 1 10000000 { pop 0.0001 sub  } bind for
  exch usertime exch sub dup ==
  20 20 moveto /Times-Roman 24 selectfont
  100 string cvs show ( ms) show ( error:  ) show 
  1000.0 div 100.0 mul abs  100 string cvs show ( %) show
  showpage

Better HOM selects

2007-11-28T18:27:00.001-08:00

Wincent Colaiuta discusses a variant of the select HOM that takes an arbitrary number of argument messages. I like it!

My initial implementation actually had arbitrary nesting, but as he discusses with collect, that requires a trigger message to start, as there is no way of knowing at runtime when the expression terminates. It never occurred to me that this limitation did not apply to select, which can look at the return type of the messages sent and stop when it reaches a BOOL (char).

Nice.

p.s.: the arbitrary nesting is still in the implementation, with each of the collection processing HOMs actually running an enumerator and those enumerators stackable, and this is two of the reasons the implementation is so gnarly: (1) there is extra generality that is not needed and (2) making that more general mechanism run fast was really, really tricky.

p.p.s: He actually discussed it almost a year ago, but I just saw it now.

Reading Helps

2007-10-03T11:41:00.001-07:00

James Robertson slams the proposal by the EU free market institute to unbundle the OS from PCs sold in the EU.

I've seen a lot of stupid ideas float past, but this one from the EU's Globalization Institute makes it into the top 5 - only the existence of the RIAA and the MPAA prevent a complete victory for these morons:
[..] The think tank recommended to the EU that all computers be sold without an operating system and sees no reason "why computer operating systems could not follow the same model as computer hard drives and processors."
Yes, installing an OS from scratch is exactly what most buyers long to do - it's such a productive use of their time.

Hmmm, what could they mean with the "same model as computer hard drives and processors"?

Well, of course! We all buy processors and hard drives separately, mount the CPU on our separately purchased motherboard, hook up hard-drive and power-supply and stick it all in a chassis. That *must* be what they meant with that phrase.

Or maybe, they meant that you can configure your computer with different CPUs and hard drives, and have the vendor ship you a machine configured to your specifications, whereas you cannot actually get a computer and not pay the Microsoft tax? Nah, that's just *crazy*:

IT professionals are being forced to adopt Microsoft's operating systems — even if they tell their PC supplier they want a system free of Microsoft software, ZDNet UK's research has revealed.

Oh.

'Thousands' or transistor²

2007-09-30T17:28:00.001-07:00

The Alan Kay quote in my previous post made me think of Montecito, the new Itanic version with 1.72 billion transistors. Compare that to the ARM6, which had a measly 35K transistors, including its 4K cache.

Dividing the two numbers gets you almost 50K. That's how many ARM6 CPUs you could get on the same chip with the same transistor budget as the Montecito. A processor for every object. Or viewed another way, more ARM6-equivalents than the ARM6 has transistors. Which begs the question: is the Montecito proportionally as much an improvement in computational capacity as an ARM6 is over a single transistor?

Objective-C future(s)

2007-09-25T17:57:00.001-07:00

Via LtU, I got alerted to the fact that theEtoile project now has an implementation of futures. Cool.

However, their implementation has specific objects reacting asynchronously to messages, making it more similar to the actor model,which as they mention is also very much Alan Kay's original conceptual model for Smalltalk:

Bob Barton, the main designer of the B5000 and a professor at Utah had said in one of his talks a few days earlier: "The basic principle of recursive design is to make the parts have the same power as the whole." For the first time I thought of the whole as the entire computer and wondered why anyone would want to divide it up into weaker things called data structures and procedures. Why not divide it up into little computers, as time sharing was starting to? But not in dozens. Why not thousands of them, each simulating a useful structure? [Emphasis mine]

Actors are inherently asynchronous, each actor runs in a separate process/thread and messages arealso asynchronous, with the sender not waiting for the message to be delivered or ever gettinga return value. Of course the actor model also makes all objects active, so the Etoile model, whichonly makes objects of specific classes active, is somewhere inbetween.

Futures, on the other hand, as introduced in MULTLSIP (pdf), tryto integrate asynchronous execution into a traditional call/return control- and data-flow. So messages(or functions in MULTILSIP) appear to have normal synchronous semantics and immediately yielda return value, but when annotated with the future keyword execution of that return valueis done in a background thread and the immediate return value is just a proxy for the value that is still being computed.

In the HOM paper (pdf) presented at OOPSLA 2005, I also describe a Future implementationbased on Higher Order Messaging that comes very close to the way it was done in MULTILSIP. A -futureHOM is all that is needed to indicate that you would like a result computed in a background thread:

  result = [anObject lengthyOperation:parameter];           //  synchronous
  result = [[anObject future] lengthyOperation:parameter];  //  asynchronous with future

I am probably biased, but this seems about as easy-to-use as possible,with all the nasty machinery (worker-queues, lockless FIFOs, etc.)hidden behind a single -future message.

Message-oriented persistence

2007-09-22T15:33:00.001-07:00

The good folks at Omni posted an interesting discussion of their persistence strategy for OmniFocus. In short they found that using a database, specifically a CoreData data store, was not exactly ideal for their primary public data format.

Instead, they appear to be using a pattern that Martin Fowler calls EventPoster. After reading David Reed's thesis, I think I prefer to call it message-oriented persistence.

I first stumbled on this pattern when designing a replacement for a feed processor at the BBC. The basic task was to process a feed of information snippets encoded as XML and generate/update web and interactive TV (Ceefax) pages.

Like a good little enterprise architect, and similar to the existing system, I had originally planned to use a central SQL database for storage, though designing a data model for that was proving difficult due to the highly irregular nature of the feed data. As an auditing/logging measure, I also wanted to keep a copy of the incoming feed data, so when the time came to do the first implementation spikes, I decided we would implemented the display, update and XML feed processing logic, but not the datastore. Instead, we would just re-play the feed data from the log we had kept.

This worked so well that we never got around to implementing the central database.

Leaving out the database vastly simplified both our code-base and the deployed system, which I could run in its entirety on my 12" AlBook whereas the system we were replacing ran around a dozen networked machines. Apart from making us popular with our sysadmin team both in terms of reliability and deployment/maintenance complexity (essentially a jar and a working directory was all it needed), a fringe benefit was being able to work on the system on said AlBook while disconnected from the network, working from home or from a sunny patch of grass outside the office.

In addition to personal happiness, systen performance was also positively affected: since we kept our working state completely in memory, the AlBook mentioned outperformed the original cluster by 2-3 orders of magnitude, producing hundreds of pages per second versus taking from several seconds to several minutes to produce a single page.

Performance and simplicity are exactly the benefits claimed for prevlayer, a persistence layer for Java based on the same principles.

TeaTime, the theoretical foundation and actual engine working underneath Croquet, takes this idea to a whole different level: objects do not have state, but are simply names for message histories. This is truly "objects [and messages] all the way down". Turtles need not apply.

More on MPWObjectCache

2007-09-01T11:53:00.000-07:00

Now that I've motivated why an MPWObjectCache might be useful, let's go into some more detail as to how it actually works. To follow along, or if you'd rather just read the source code than my ramblings, MPWObjectCache is part of MPWFoundation, which can be downloaded here: http://www.metaobject.com/downloads/Objective-C/.

As I mentioned before, the algorithm for MPWObjectCache is quite simple: there is a circular buffer of object slots. We try to get pre-allocated objects from this circular buffer if possible. If we find an object in the cache and it is available for reuse, we just return it and have just saved the cost of allocation. Two things can prevent this happy state of affairs: (1) we don't have an object yet or (2) we cannot reuse the object because it is still in use. In both cases we will need to allocate a new object, but in the second case we also remove the old object from the cache position.

#if  SLOW_SAMPLE_IMPLEMENTATION_WANTED
-getObject
{
    id obj;
    objIndex++;
    if ( objIndex >= cacheSize ) {
        objIndex=0;
    }
    obj=objs[objIndex];
    if ( obj==nil ||  [obj retainCount] > 1 ) {
        if ( obj!=nil ) {
            [obj release];
        }
        obj = [[objClass alloc] init];
        objs[objIndex]=obj;
    }
    return [[obj retain] autorelease];
}
#else

This is what a naive implementation looks like. A couple of notes on the code:

objects must be reinitialized by the client (and reinitializable in the first place)
only one attempt is made to find an object
the retain/autorelease will prevent the cache from working unless a fairly tight autorelease pool regime is maintained
there are quite a few message sends
it's not what is used in production

The effectiveness of the cache obviously depends on your allocation patterns and the size of the object-cache. Larger caches take longer to be filled up before they start wrapping around with the potential for reuse, but smaller sizes can mean that the object will still be in use when we do wrap around. The actual implementation is very similar to the one presented above, except that it does a little more probing and uses IMP-caching for all the messages sent on the critical path. These optimizations ensure that object-caches are no slower than normal allocations even in worst-case situations such as every allocated object being retained. In addition the cache can also be set to not do the retain/autorelease, which is safe when you are pushing objects and have control over the cache:

-doSomething:target
{
 // cache is an ivar
 id obj=GETOBJECT(cache);
 // target does not have access to cache
 [target doSomethingWithObject:obj];
 // obj now either has an extra retain or can be reused
}

This pleasant property is a side effect of the decision to turn the object-cache into an object that can be instantiated and placed in an instance variable, rather than the typical object pools that are implemented as class methods. The class method that maintains such a pool usually has no information about the lifetime of objects, so to be safe such an implementation always has to protect the objects it returns, negating much of the advantage of caching. Similar caveats apply to multi-threading and locking. Those caveats notwithstanding, MPWObjectCache also provides the CACHING_ALLOC macro for creating class-side allocation methods backed by an object cache, which is used in the HOM implementation to reduce the cost of allocating trampolines:

 CACHING_ALLOC( quickTrampoline, 5, YES )

This creates a +quickTramplone method backed by an object cache with 5 entries. The YES flag allows objects to be returned from the cache without the retain/autorelease despite the fact that it isn't one of the safe "push" patterns described above. However, this use is also safe because the trampoline is used only temporarily to catch and forward the message, all of which is code controlled by the implementation. It is no longer needed once any client code implementing the actual HOM is run. So, this is how and why object-caches can make your (temporary) object allocations much, much faster.

High performance Objective-C I: a Postscript interpreter

2007-08-27T13:17:00.000-07:00

A key component of the metaobject product suite is EGOS, which includes as a central ingredient a custom Postscript Level 3 compatible interpreter. The project was started in part as a hedge against the chance of Apple dropping DisplayPostscript, in part because our Postscript virtualization technique was hitting limits, and in part because it would make getting Objective-C objects out of the interpreter much easier.

At its core, Postscript is a stack-oriented, dynamically typed and highly polymorphic interpreted programming language. So implementing Postscript with Objective-C objects is actually not just convenient when you want to get Objective-C objects out, it is also a good match for the semantics of the language.

So all is good, right? Well, we also need to make sure that performance is competitive, otherwise there really isn't much of a point. How do we find out if performance is competitive? Fortunately, we have the gold standard handily available: Adobe's interpreter was not just used in NeXT's DisplayPostscript, but is also available as the PS Normalizer on Mac OS X . So let's test performance with a little Postscript program:

  %!
  usertime
  0 1 1000000 { 4 mul pop } bind for
  usertime exch sub dup ==
  20 20 moveto /Times-Roman 24 selectfont
  100 string cvs show ( ms) show
  showpage

The program times a loop that multiplies some numbers one million times. It exercises a good deal of the basic execution machinery in the Postscript language: stack manipulation, procedure invocation, array access (a procedure is just an array with the executable bit set), looping and arithmetic. The loop is timed with the usertime command, which returns CPU time used in milliseconds.

This test clocks in at 513 ms (513 ns per iteration) in Preview, which isn't too shabby.

1. The problem

As proof of concept, let's code up some Objective-C equivalent of what the Postscript interpreter has to do in this loop. That should give us a good lower bound for the time taken (lower bound because there will be additional interpretation overhead, and Postscript semantics are slightly more complicated). We need a stack, some number objects and a bit of arithmetic. Easy:

 id startcounter=[NSNumber numberWithInt:0];
 id endcounter=[NSNumber numberWithInt:1000000];
 id counter=startcounter;
 id four=[NSNumber numberWithInt:4];
 while ( [counter intValue] < [endcounter intValue] ) {
  int intResult;
  id result;
  [stack addObject:counter];
  [stack addObject:four];
  intResult = [[stack lastObject] intValue] * [[stack objectAtIndex:[stack count]-2] intValue];
  result=[NSNumber numberWithInt:intResult];
  [stack removeLastObject];
  [stack removeLastObject];
  [stack addObject:result];
  [stack removeLastObject];
  counter=[NSNumber numberWithInt:[counter intValue]+1];
 }

Sadly, this takes 4.8 µs per iteration, so our 'lower' bound is almost 10 times slower than our target, and that's without accounting for interpretation. Clearly not good enough. What if we get rid of all that silly stack manipulation code and use a plain C loop?

  id b=[NSNumber numberWithInt:4];
  for (i=0;i < 10000000;i++) {
  id a=[NSNumber numberWithInt:i];
  id c=[NSNumber numberWithInt:[a intValue] * [b intValue]];
 }

2. Mutable State

Objective-C is an imperative object oriented language, meaning objects can change state. However, we have treated numbers as immutable value objects, requiring them to be recreated from scratch. Allocating objects tends to be around 25x more costly than an Objective-C message send, so what if we don't allocate new integer objects, but instead reuse an existing one and just change its value? It turns out we can't use NSNumber for this as it doesn't allow its value to be set, so we need a (trivial) wrapper class for a single integer.

 
   id b=[MPWInteger numberWithInt:4];
   id a=[MPWInteger numberWithInt:0];
   id c=[MPWInteger numberWithInt:0];
   for (i=0;i <10000000;i++) {
  [a setIntValue:i];
  [c setIntValue:[a intValue] * [b intValue]];
  }

That's more like it: 50ns per iteration is 100x better than our first attempt and also 10x better than the target we're aiming for. So taking advantage of mutable state makes our basic plan possible, at least in principle. Of course, we now have to reintroduce the stack and add interpretation.

3. Save the planet

Alas, it turns out that the interpreter really does need fresh instances. While it will discard them quickly in most cases, it sometimes stores them away meaning we can't statically reuse objects the way we did above.

Instead, we need to figure out a way to recycle temporary objects so we can reuse them without spending a lot of time. The common way to do this is to keep a pool of objects from which requests for new MPWInteger instances are satisfied. However, due to the unpredictable nature of the interpreted code, we cannot use the explicit checkin/checkout policy such pools usually require.

Instead we make the pool a circular buffer and use the retain count to verify that an object can be reused. When we get to a position in the pool that has an object, we can reuse that object if the retain count is one, meaning that only the pool has a valid reference. If the retain count of the object is greater than one, someone other than the pool is holding on to the object and it cannot be reused (yet), so we need to get another instance.

This logic is encapsulated in the class MPWObjectCache, which can be used very similarly to a class (factory object) in creating new instances.

 MPWObjectCache* intCache=[[MPWObjectCache alloc] initWithCapacity:20 
        class:[MPWInteger class]];
 id b=[MPWInteger integer:5];
 for (i=0;i < 1000000;i++) {
  id a=GETOBJECT(intCache);
  id d=GETOBJECT(intCache);
  [a setIntValue:i];
  [d setIntValue:[a intValue] * [b intValue]];

This code runs in 100ns per iteration, so we now have a solution that gives us new or safely recycled objects quickly enough to build on with the confidence the end result will perform acceptably.

4. Results

Running the Postscript test program from the start of this post in PostView yields a result of 260ns per iteration, meaning that our Objective-C Postscript interpreter is almost twice as fast as Adobe's, at least on this particular workload. While I wouldn't generalize this isolated result to say that EGOS is a faster interpreter, it clearly shows that it is at least competitive, which was the goal of the exercise.

The fact that it took a measly 20 KLOC illustrates the leverage Objective-C provides: Ghostscript weighs in at around 250+ KLOC (without drivers).

5. Conclusion

One of the things I've always liked about Objective-C is that it lets you have your cake and eat it, too: great expressiveness to solve your problem effectively is always coupled with the ability to get down and dirty and get really great performance, without losing the structure of the original solution.

The most important factor to watch out for in terms of performance tends to be object allocation. Controlling this factor with a transparent object-cache allowed us to get an overall performance improvement of around 10-20x in the case of a Postscript interpreter, taking performance from unacceptably slow up to and beyond the industry standard.

Of course, this isn't the only factor and Postscript interpretation not the only application. Stay tuned!

Onward the Blogosphere

2007-05-12T14:12:00.000-07:00

Somewhat late to the party, metaobject now has a blog. Expect to see more details about our products, as well as more general ramblings about programming in general, dynamic languages, Objective-C, Cocoa and (Higher Order) Messaging in particular, and anything else that strikes my fancy.