Saturday, April 25, 2020

Maybe Visual Programming is The Answer. Maybe Not

Whenever discussing problems with programming today and potential solutions, invariably someone will pop up and declare that the problem is obviously the fact that programs are linear text and if only programming were visual, all problems would immediately disappear in some unspecified way.

I understand the attraction of visual programming, particularly for visual thinkers. However, it's not as if this hasn't been tried, with so far very limited success. Brad Myers, in his 1989 paper Taxonomies of Visual Programming gave, along with the titular taxonomy, a non-exhaustive summary of the problems, starting with visual languages in general:

  • Difficulty with large programs or large data. Almost all visual representations are physically larger than the text they replace, so there is often a problem that too little will fit on the screen. This problem is alleviated to some extent by scrolling and various abstraction mechanisms.
  • Need for automatic layout. When the program or data gets to be large, it can be very tedious for the user to have to place each component, so the system should lay out the picture automatically. Unfortunately, for many graphical representations, generating an attractive layout can be difficult, and generating a perfect layout may be intractable. For example, generating an optimal layout of graphs and trees is NP-Complete [95]. More research is needed, therefore, on fast layout algorithms for graphs that have good user interface characteristics, such as avoiding large scale changes to the display after a small edit.
  • Lack of formal specification. Currently, there is no formal way to describe a Visual Language. Something equivalent to the BNFs used for textual languages is needed. This would provide the field with a ‘‘hard science’’ foundation, and may allow tools to be created that will make the construction of editors and compilers for Visual Languages easier. Chang [49] [96], Glinert [97] and Selker [98] have made attempts in this direction, but much more work is needed.
  • Tremendous difficulty in building editors and environments. Most Visual Languages require a specialized editor, compiler, and debugger to be created to allow the user to use the language. With textual languages, conventional, existing text editors can be used and only a compiler and possibly a debugger needs to be written. Currently, each graphical language requires its own editor and environment, since there are no general purpose Visual Language editors. These editors are hard to create because there are no ‘‘editor-compilers’’ or other similar tools to help. The ‘‘compiler-compiler’’ tools used to build compilers for textual languages are also rarely useful for building compilers and interpreters for Visual Languages. In addition, the language designer must create a system to display the pictures from the language, which usually requires low-level graphics programming. Other tools that traditionally exist for textual languages must also be created, including pretty-printers, hard-copy facilities, program checkers, indexers, cross- referencers, pattern matching and searching (e.g., ‘‘grep’’ in Unix), etc. These problems are made worse by the historical lack of portability of most graphics programs.
  • Lack of evidence of their worth. There are not many Visual Languages that would be generally agreed are ‘‘successful,’’ and there is little in the way of formal experiments or informal experience that shows that Visual Languages are good. It would be interesting to see experimental results that demonstrated that visual programming techniques or iconic languages were better than good textual methods for performing the same tasks. Metrics might include learning time, execution speed, retention, etc. Fortunately, preliminary results are appearing for the advantages of using graphics for teaching students how to program [36].
  • Poor representations. Many visual representations are simply not very good. Programs are hard to understand once created and difficult to debug and edit. This is especially true once the programs get to be a non-trivial size.
  • Lack of Portability of Programs. A program written in a textual language can be sent through electronic mail, and used, read and edited by anybody. Graphical languages require special software to view and edit; otherwise they can only be viewed on hard- copy.

In addition, most visual programming languages are "unstructured" in the software engineering sense. They

  • use gotos and explicit transfer of control (often through wires),
  • only have global variables,
  • have no procedural abstraction,
  • if they have procedural abstraction, they may not have parameters for the procedures,
  • have no place for comments.

Furthermore, he notes that most visual languages don't interoperate with programs created in other languages, with some exceptions.

I am not saying that visual programming languages will not and cannot work, in fact, I am quite a fan myself. As these are specific problems, they probably can be solved, and I have a few ideas for solving some of them. However, before making claims that visual programming by itself is obviously and singularly the solution to our computing woes, please mention at least in passing how you've addressed the problems identified by Myers.



  1. I am fairly convinced that the problem with past efforts has been the lower information density combined with the small displays computers had in the past.

    But now I use a 60” TV as my primary monitor (HIGHLY recommended) and we have affordable and practical VR.

    I would offer better than even odds that it would be possible to develop a graphical, functional programming environment in VR that is compelling.

  2. While your second list of problems is true in general, none of it applies to Boxer (an old Mac-only visual Logo). It is a pity that it is so hard to find information about it on the web.

  3. We will solve this problem soon. Probably in 3 years or maybe 30 but it will be solved. Right now we are just testing things out.

  4. There was Rational Rose with UML diagrams. Also Sparx Enterprise Architect could import large code bases as UML diagrams. Yet they all lacked essential diagram types, e.g. activity or sequence diagram. Also code wasn't represented as graphical elements only. Most of the control flow was simply put into a formatted document/ comment-section without any fancy editing beyond colors and font styles. This never replaced my IDE but was only for plotting out parts of the system for documentation.

    Have you ever read about Software Through Pictures…go ahead, all dead. Maybe you join formal methods as bricks+mortar with a sort of template based renderer (Stackoverflow/ Stack Sort?) to add something meaningful to the world of software development styles.

  5. Interesting article. I've been programming in a visual language (LabVIEW) for 10 years now and have worked in a number of text based languages as well so I've given some thought to some of your points.

    I think many of the points are spot on. Layout is an additional pain (literally an extra dimension over text layout) but in LabVIEW auto layout has been tried and sucks.

    The portability and effort in building IDEs is a big point for me. LabVIEW is proprietary and can borrow little from the wider world in terms of tooling (since it is designed around text) which seems to make it a larger effort to develop (I'm not involved in it's development so this is conjecture)

    I would argue the structure is not a problem though, rather they can't just be visual representations of imperative programs. In LabVIEW variables are available but heavily discouraged (and can be scoped). Instead it works on a dataflow principle which determines execution order and you create functions (subVIs) driven by this.

    Thanks for the write up. It's given me a few more points to think over.

  6. The two great unsolved problems in computer programming today, are 1) the lack of interchangeable parts, and 2) the terrific waste of time entailed in debugging caused by the inability to run backwards (i.e. a car with no reverse gear). Visual environments do nothing to solve either #1 or #2, so this problem of making programming more visual is not particularly important. But given that a whole generation of kids is learning to program, and they grew up with video games, they find it anachronistic for programming to be so clumsy and visually barren. So there is a large pent-up demand for a more visual process.

  7. I've used 3 different visual programming languages. The first was LabView, though I much preferred LabWindows, which was the textual equivalent. I could see, though, how LabView might suit a practical scientist who was very familiar connecting networks of measurement instruments. I have taught MIT's Scratch to youngsters and I think it very good as an introduction to programming, though older ones are directed to a textual language (Python). I currently use SAS DI-Studio (as well as text based SAS). There is extra effort involved in maintaining the layout - there is an auto-layout feature but it tends to run out of steam just about at the point where manual layout starts to become difficult.

    I think that visual programming would allow some people who are not comfortable with large chunks of code to be involved in programming. It could also be a useful adjunct to mainstream text based programming if it could auto-generate (MUCH better than I've seen up to now) sensibly abstracted diagrams (editable?) of text code.