Coding In Paradise

Douglas Engelbart's HyperScope, the project I've been working on the last half year as architect and implementor, is now out. I'm extremely excited about this system: it layers expanded, new hyperlinking abilities onto the existing web, working with existing technology, such as web browsers, Ajax, and OPML. HyperScope 1.0 is out the door and is real.

From the HyperScope web site:

"The HyperScope is a high-performance thought processor that enables you to navigate, view, and link to documents in sophisticated ways. It's the brainchild of Doug Engelbart, the inventor of hypertext and the mouse, and is the first step towards his larger vision for an Open Hyperdocument System.

The HyperScope is written in JavaScript using the Dojo toolkit and works in Firefox (recommended) and Internet Explorer. It uses OPML as its base file format. It is open source and available under the GPL."

A review of comments from the blogosphere on the project:

Read/Write Web has a great writeup:

Douglas Engelbart's HyperScope 1.0 Launched

See Slashdot's take on the system.

Dave Winer notes our OPML support:

ReadWriteWeb on Doug Engelbart's Hyperscope 1.0, a modern web app. Its native file format is OPML.

Here's an example of a Hyperscope document. View source reveals that it is an OPML document with a stylesheet.

See more coverage on the blogsphere with Technorati: http://technorati.com/search/hyperscope

In-depth technical writeups of how we work with OPML are here:

http://codinginparadise.org/weblog/2006/09/hyperscope-opml-and-html-hyperlinks-on.html
http://hyperscope.org/dev/index.html
http://codinginparadise.org/hyperscope/src/demos/finalarch.opml#06:wg (a little bit out of date)

Six Apart just bought the old company I used to work for, Rojo. I knew about this for awhile but couldn't talk about it. It's the end of an era... What a ride it was.

I'm the software architect and coder for HyperScope, and thought I'd provide some more info on the OPML capabilities of what you can do with it.

HyperScope is like OPML and HTML hyperlinks on steroids. There are so many different kinds of addressing your nodes that its not even funny. You can do direct addressing using hiearchical placement, even if there are no anchors:

mydoc.opml#2A

Where 2A is the second node (2), followed by it's first child (A). Node numbering always alternates between letters and numbers for readability; so you might have 2A3B. These are generated automatically and will work across all OPML documents, not just ones that have been marked up with the optional HyperScope namespaced attributes.

You can target node IDs, which are unique values that never change for a node even if it is moved around; these always start with a 0:

mydoc.opml#0626

Node IDs are optional, and placed on an OPML outline node using the hs: namespace:

<outline text="foobar" hs:nid="0626"/>

Node IDs are optional, and are created on document creation or editing, while the node hierachical numbers above work for any OPML file.

You can target based on label, using the optional hs:label attribute:

<outline text="foobar" hs:label="mylabel">

This would be the following address:

mydoc.opml#mylabel

Now, here's some cool stuff; once you've hit your target address using direct addressing, you can start to use relative addresses to move relative to the one you just targeted. For example, once I've gone to 'mylabel', I could jump down in the node hierarchy then to the next 2 nodes using the following:

mydoc.opml#mylabel.d2n

Which is .d for down and .2n for next two. You can jump through the hierarchy using many of these, including successor (.s) for going to the next sibling of the current node, independent of children; predecessor (.p) for the same, but for the previous sibling; back (.b), which will take you to the previous node that is right before you, independent of hierarchy; next (.n) for the same but forward; up (.u) to move up an ancestor; down (.d) to move down; tail (.t) to move to the last child sibling on my level; head (.h) to move to the first child sibling on my level; and more.

Once you've done these, you can start doing some other fun stuff. I can now do an indirect link, which is .l (that's a lower case L):

mydoc.opml#mylabel.d2nl

The 'l' is magical; it says: "once you've found the final node after dereferencing the other addresses, look inside the contents of this node, and take the first address you find then follow it."

For example, if the node that lies at mylabel.d2n has the following contents (that's an encoded HTML hyperlink in there):

<outline text='Make sure to see the document: &lt;a href=&quot;http://foobar.com/important.opml#2&quot;Important&lt;/&lt;/&gt;

then adding that .l will cause HyperScope to extract
http://foobar.com/important.opml#2, and follow it and dereference it, which will cause the browser to jump to important.opml, then jump to node #2.

So indirect links let you create a layer of indirection between you and a set of links. You could imagine combining this with the node labels to create a kind of link database; just create a file that you always use that has named nodes for important links that you want to go to, then use indirect links against these. This is exactly what Engelbart does; I've sat and watched him and studied how he uses the system. You can see modern screencasts of that system and his work practices at these blog posts of mine: http://codinginparadise.org/weblog/2006/03/new-screencast-of-douglas-engelbarts.html and http://codinginparadise.org/weblog/2006/03/recording-douglas-engelbart-skype.html

I'm just scratching the surface here. You have viewspecs, which allow you to control how a remote document displays itself within the address itself, such as saying "show me just your top level outline", "show me everything", or "show me just this branch and nothing else"; there's a whole bunch of these.

You have full transclusions/inclusions, and these work using all of this addressing; we use standard OPML 2.0 inclusion, but allow for even more types. So here would be an inclusion:

<outline type="include" url="http://foobar.com/important.opml#2"/>

This would cause HyperScope to fetch node #2 of important.opml and then include it into the parent document.

By default inclusion will just grab the node given; but, you can sprinkle on an optional HyperScope namespaced attribute to control this:

<outline type="include" hs:include-type="branch" url="http://foobar.com/important.opml#2"/>

This is hs:include-type, which can be "node", "branch", or "plex". Node is the default and is obvious; branch will take this node and all of it's children; and plex is less obvious but very powerful once you get use to it. Plex says "grab this node, and all siblings both forwards and backwards that have the same up parent". Using these three types you can do some very cool stuff. And inclusion can be done on ALL OPML documents out there, not just ones with HyperScope attributes; HyperScope fully works with normal OPML, it's just that we provide some optional magical namespaced HyperScope attributes you can sprinkle in to get even more abilities.

I haven't even talked about content filters yet, which are very cool; this is a way to specify in your address that only OPML nodes that pass this filter be displayed. Here's an example:

mydoc.opml#:i ;"IMPORTANT";

This says: for "mydoc.opml", only display nodes that have the word "IMPORTANT" in them. The :i is a viewspec which says "turn on content filtering" (it is off by default).

That line above will filter out and only show nodes that have that word; anything between the semicolons is a content filter. You can even put a full Perl 5 regular expression in there for greater control... (it's a JavaScript regexp):

mydoc.opml#:i ;/important|todo|warning/i;

delimited with a slash; this says "only show me nodes that have the words important, todo, or warning, and they can be case-insensitive (i)".

When you put these all together, they start creating some amazing possibilities. Imagine a single synthetic document that does transclusions of other documents, applys content filters across them, uses indirect links inside transclusions, uses viewspecs to break a document into pieces (viewspecs can be used to just show a given branch, for example, which means you can use a single document as a presentation container, for example). You can even have your transclusions do their own transclusions...

To get these abilities all you have to do is add a single XSLT stylesheet line to your OPML, which will cause the OPML to transform itself into the full HyperScope UI; you can also optionally add the hs namespace if you want to use our optional attributes. You must also upload the HyperScope distribution to your webserver, but since it is all done on the client side using Ajax this entails just copying a bunch of static JavaScript files to your web server; there is a single optional PHP 4 file that will gateway cross-host transclusion/inclusions. If it is not there the system will continue to work, and documents that do cross-host transclusions will just display a little red error message in that node's contents when it does it's transcludes.

The possibilities are endless, and really damn cool, especially when we get the transformers online, which will transparently turn other document formats like Microsoft Word and PDF into OPML on the fly, and allow you to apply all of these addressing types across your entire universe of documents.

Or imagine hooking this into your web based source code control system, so you can zoom around your source using these abilities; this is actually what the Augment guys did, using Augment itself to write all of their own code and study it.

And best of all: all of this exists now, delivered, in a 1.0 state that is open source, so it's not just empty promises or a paper specification. You can play with the real system now. It currently works across Firefox and Internet Explorer. Firefox is preferred.

One last thing: these are the original hyperlinks, fully brought to the web; this is the system that invented hypertext. Our goal with HyperScope was to fully translate these onto the web, changing them as little as possible for this iteration; our goal was to transfer, not innovate, for this round (future dev rounds will innovate and change things). We wanted to make them immediately accessible for a contemporary audience, using modern technology (browsers, Ajax, OPML etc.) Keep in mind that you're getting to play with and click on the original links; I just think thats so damn cool. Even if you argue with their present utility, which I think is still very useful for today's web, for direct historical value this project is important.

Come join us at http://hyperscope.org :)