Low-level Communication:
Restore, Build, Format, Paint, Events, Destroy Protocols

In general, behaviors do not directly invoke one another's methods as this would lock out changes by other behaviors. That is, closed communication patterns would lock out behaviors with new ideas about how the system should operate. Instead, the general template for all communication is the following:
  1. system or behavior initiates or requests some action,
  2. all behaviors potentially interested in this action have a chance to modify or even cancel (or just record or ignore) the action, and
  3. the modified action is executed.

All digital document systems share a fundamental document lifecycle: the application is loaded, the document is read in and internal data structures are built for it, the document is formatted, then it is painted on the screen, at which point the system waits for the user do something. To open this to arbitrary new behaviors, the process is reified into sharply segregated protocols: restore behaviors, build document tree, format document tree, paint document tree, low-level events (such as from the mouse and keyboard), semantic events (described next section). During the build protocol, say, all interested behaviors have the opportunity to build on the document tree data structure, modifying the work of other behaviors.

The system is a framework where the system is in charge of the overall flow of control. Frameworks contrast with systems in which the developer's program defines the flow of control, sometimes passing through libraries. For every protocol in the framework, the system will pass through relevant behaviors, briefly handing off the flow of control to a behavior to perform all of its work for that protocol only. Behaviors may not have any work to perform in a particular protocol, but while in one protocol it is illegal to work on another protocol.

Most protocols have before and after phases: the before phase of the protocol is executed in its entirety, then the after phase. With arbitrary behaviors active at any point in the document lifecycle and with potential dependencies on other behaviors, the division of protocols into phases helps sequence behaviors. The rule of thumb is that behaviors that build structures do so in before, so that all such activity is known to be completed by after, which can modify it or cancel it. For example, during build before, one behavior can load in the main body of a document, so that it will be available for annotations to hook into during build after. A behavior can short-circuit from before to after phases thus bypassing lower priority behaviors, or from after to end the protocol.

As appropriate to their intrinsic nature, protocols are either round robin or tree based. Round robin protocols flow through the before phase of all active behaviors from highest priority to lowest, then the after phase in reverse order. Thus the highest priority behavior is called first and last, getting the first word and the last say, as it were. The round robin protocols are Restore, Build (at the start of which the tree does not exist), Semantic Events (such as openDocument and newBrowserInstance), and Save.

Tree-based protocols proceed through a depth-first tree walk, during which tree nodes can also affect control flow. Behaviors interested in a structural portion of the tree register interest (programmatically, add themselves as observers) to the node at the head of the subtree, and during the tree walk the before methods of the observers are called before the node and its children are traversed, and the after methods of the observers are called after the node is done. Behaviors can short-circuit from before to after, thus bypassing that subtree; and after can short-circuit to cancel the remainder of the tree walk. The tree-based protocols are Format, Paint, and Low-level Events.

Below are described the low-level protocols: those performance-intensive protocols concerning construction and display of the document, and system input.

Restore Protocol

When a behavior instance is created, it is restored, during which it can perform initialization. Behaviors take their attributes from their hubs, which provides an easy means of end user customization. For example, the menubar behavior takes the list and order of its menus from an attribute.

Build Protocol

The build protocol iterates through the build before methods of all behaviors, then the build after of each in reverse order. Media adaptors execute most of their work during this protocol, reading a concrete format and building its runtime manifestation as a document tree. With media adaptors working in the build before phase, annotations, which are stored separately from documents and can annotate any document format, attach themselves in the after phase, at which time the basic document tree is known to have been constructed.

Format Protocol

Formatting is largely left to document format-specific nodes to carry out. Generally, nodes determine how large they would like to be (width and height dimensions), and parents set the locations (x,y) of their children. Formatting occurs during a walk of the document tree, top down propagating maximum dimension constraints and property settings from style sheets, then bottom up propagating requested dimensions to be positioned.
Core Format and Paint Properties
foreground and background colors
font family, style, size
underline, overstrike
horizontal and vertical alignment
float side
margins, padding, border


All nodes, whether heavily medium-dependent leaves or largely medium-independent internal nodes, must respect a core set of properties during Format and Paint protocols, the full set of which is given in the table at right. Behaviors rely on a this guaranteed level of functionality across media types. For instance, the selection and the highlighting annotation rely on setting the background of a word to a given color. Most media types can simply draw the word over the colored background, but scanned paper must identify the white pixel in its image and convert it to transparent before drawing the image of that word.

Example: formatting scanned paper images.

Two core properties that must be respected are space above and below the baseline, which is used to open up space between lines for short text annotations. For flowed document formats such as HTML and manual pages, this is simply another variable to consider during formatting. For scanned paper, repecting this property requires reformatting a fixed image, pushing areas that would be overlapped farther down or to the right.

Paint Protocol

The Paint protocol renders a formatted data structure to the screen, printer, or other device. It is very similar to Format in the management of core properties. The before phase can be used to draw backgrounds or set graphics transformations, and after can draw on top of the corresponding subtree.

All painting is performed under a clipping region, which at largest is the size of the visible screen. During the tree walk, only nodes that lie within the clipping region are pursued; thus, when an internal node lies outside of the clipping region, its entire subtree is ignored, and the system is able to quickly paint, with just a small amount of wasted effort, that part of a potentially long document that is visible on the screen. Lenses operate by setting the clip to just their bounds, or their intersected bounds, and repainting the entire document with certain properties set. This process is so rapid that the entire screen can be repainted in full continuously during scrolling. This is as opposed to a "bitblt" which copies that part of the document that remains onscreen but at a different location; bitblt-based scrolling is problematic in Multivalent because notes and lenses can be fixed to a point on the screen and so don't scroll with the rest of the document.

Example: Move-To annotatation.

As shown in the demonstration of the Multivalent Browser, the Move-To annotation type draws an arrow from its source span to its destination point. This arrow must be drawn the document's coordinate space, which may be scrolled; it should be fast to redraw, since the destination point is interactively chosen; and it should be efficient so that we can have 1000s of annotations that draw on a document 100s of pages long, and so only those arrows that will appear on the screen should be drawn.

The Move-To behavior accomplishes this by using a tree node function to determine the lowest node in the tree common to both start and end points, and registers interest on that node. At this time it computes the coordinates of the start and end points of the arrow relative to the common node, to be used in any number of future paintings. Now for all protocols the behavior receives notification of activity on the node; in this case, after the content of the subtree has been painted, which is to say in the after phase, the behavior draws the arrow using the precomputed coordinates. Since the lowest common node frequently extends beyond the visible screen, parts of the arrow are at times painted uselessly, but the total amount of wasted work is contained within a relatively small portion of the document. During interactive setting of the arrow, the behavior quickly unregisters from the last lowest common node, computes the new node, registers on that, computes coordinates, and repaints the screen.

Low-level Events Protocol

Low-level events, such as keystrokes, mouse clicks, and OS window activity (close, iconify) are propagated through the tree, its path winnowed by the event's coordinates if any, where they can be seized by behaviors that have registered interest in that part of the tree.

Examples: spans, document popup, disabled menu items.

Span behaviors receive events by virtue of being anchored to leaves. Hyperlinks of course translate a mouse click into a request to load a new page. The document popup menu behavior seizes a button 3-down event, in the after phase in order to give any other behavior priority, and then creates the menu according to behaviors active at the cursor point. The disabling of menu items is not built into menus or menu items, but is added as a behavior. A structural behavior registers itself on each disabled menu item and short-circuits events on that subtree so mouse movement cannot select that menu item. (It also participates in Paint after, graying out the appearance of the menu item by drawing over the content spaced lines the same color as the background.) This same method could implement guards for widgets with dangerous effect: click once to remove the guard, click again to execute.

High-level Communication: Semantic Events

The low-level protocols open document manifestation and interaction with the user. They do not address higher-level logical or semantic actions, such as opening documents and sorting tables, but the spirit of the system demands that, no less than low-level actions, these high-level actions must be open to modification by any behavior.

As a high level action, semantic events are relatively infrequent and thus not performance critical. They are sent to all behaviors, with before and after phases. A semantic event consists of a message, such as openDocument, and three fields labeled argument, in, and out, whose exact content types depend on the message, though the rule of thumb is that argument corresponds to the most commonly needed auxiliary data, in holds the sender of the event, and out collects results from participating behaviors.

Semantic events are most often sent to request action and to announce state. Even when a behavior could directly invoke itself, it is often better to request the action and give other behaviors a chance to modify or cancel the request. Other semantic events announce potentially interesting state. Semantic events are most often acted upon by behaviors to implement a requested action, to modify the event, or to update state in response to an annoucement event.

For example, when a document has been loaded, surviving potential 404 File Not Found's and redirections, the system announces openedDocument (note past tense), and the forward/backward behavior adds it to history list. The search behavior announces its results in a searchHits semantic event, which is caught by the scrollbar visualization behavior. Media adaptors for HTML, directory listings, and Zip archive listings all send the table sort behavior, with sortTable as the message, and node and direction in fields. The table sorting behavior catches the event, inspects table to determine the data type of the requested column, sorts, and rearranges the document tree to reflect the sort order.

Example: TeX DVI.

TeX's DVI is a page description language like PostScript, but very simple: it has movement commands, registers, font changes, character drawing, and little else. Moreover, its author froze the format. Everything from hyperlinks to images to graphics drawing is implemented by means of "specials", which are embedded strings that are left to the viewer to interpret. There are probably hundreds of specials that have been defined, and different viewers implement different, usually small subsets. Supporting a new special involves hacking the source code, if the source code is available, and hoping the viewer's main author accepts the changes.

The Multivalent DVI media adaptor supports specials without modification of the parser-renderer with the same mechanism as used in the rest of the system, the behavior. The parser announces specials as semantic events, passing the special string, and geometric and logical (document tree) positions in fields. Behaviors implementing specials listen for the relevant messages, and have enough information in the fields to accomplish their work. The HyperTeX hypertext and PageSize specials are currently supported, and image and color will be.

Example: menu construction.

All behaviors have a chance to contribute to all menus, whether they drop down from the menu bar or pop up from document content, hyperlinks, or the selection. Menus are built on demand. At the moment the menubar behavior needs to build a menu, it sends a semantic event with message createWidget/menu-title, and seeds the out field with an empty menu. Arbitrary behaviors can add to the menu, and when the event returns to the menubar, the menubar formats and paints the resulting menu.

See Graphics User Interface.

Semantic Events in the Basic Browser

Some of the more important semantic events that are defined by built-in behaviors include the following. See Document.java, Browser.java, and others in API.


fields named MSG_xxx