Coding In Paradise

Dojo has been good to me. I remember in 2005 when I was a whee JavaScript programmer hacking around with weird stuff like AMASS to do client-side storage when Alex Russell contacted me and asked me to port the work over to Dojo. I was blown away; what an opportunity! I had heard of Dojo but never thought I would get the chance to contribute.

Being a part of the Dojo project has been an amazing opportunity for me. Dojo Storage helped legitimize the idea of doing client-side storage, and the experience working on it helped shape parts of the HTML5 Local Storage API when it was being developed. Creating Dojo Offline went from crazy prototype idea to real shipping code thanks to SitePen and Google, and led to my involvement with Gears and also helped shape aspects of the HTML5 Offline work.

And what an awesome community Dojo is! I got a chance to meet fellow JavaScript hackers who had a mad gleam in their eye on trying out some interesting new scheme to try out in the browser. I consider many Dojo-ites close friends and colleagues and continue enjoying meeting up and scheming strange browser ideas.

The last year and a half, though, I really haven't been able to be a part of the Dojo community, and I don't see that changing. Dojo Storage and Dojo Offline are being used by real users and real sites but I simply haven't had the bandwidth to fix important bugs or add new functionality. There comes a time when an open source programmer has to admit that they simply can't juggle so many balls in the air at once.

I've essentially left the Dojo community the last year and a half but consider this blog post more formal. Other things have swept me up and forced my time. I can no longer maintain Dojo Storage and Dojo Offline. This is a great chance for the users and developers who use both of these packages to step up to maintain them and continue developing them; I pass the baton to you. You won't be sorry being a part of Dojo; I know I haven't.

I wish I was good at everything; I'm obviously not ;) The particular thing I'm good at is coming up with some strange new idea, then doing several passes of engineering and work to make it real and bring it to a shippable state and get it past the 'giggle factor'. I've done this with things like Really Simple History, coworking, and more. I'm good at the 0.1, 1.0, and 2.0 phases. Past that, I'm not particularly good. I guess the wisdom of 'old age' is accepting your strengths and weaknesses ;)

So long, and thanks for all the fish!

Labels: dojo, dojo offline toolkit, dojo storage

Two new bits of news today.

First, Jean-Baptiste Boisseau has done some great work porting Dojo SQL to work not only on Gears, but in a Java and AIR environment (and HTML 5 when that appears). His work is in alpha and can be seen here, including a demo here. He has setup a blog as well to track his progress. Just in case you don't know Dojo SQL is an easy-to-use SQL layer for JavaScript. Under the covers it uses Gears to do the actual storage, and now Java if available thanks to Jean-Baptiste Boisseau.

Second, Russell Leggett was inspired by all the talk around the Open Web and Gears. Russell likes the Open Web, but wants to see JavaScript, HTML, CSS, etc. not just evolved but replaced by better layers that can help with web applications; he just wants to see these new layers remain open, just like HTML. I told him that Gears could potentially act as a way to deploy some of his ideas, and challenged him to write up his ideas and create a blog. Russell accepted the challenge and setup a new weblog, named www.fightfortheopenweb.com, as well as a Google Group here. Join the group and join the discussion!

Some excerpts from Russ's first few blog posts:

"I believe there is an alternative solution [to Flash/Silverlight/AIR/etc]. I believe it is possible to fight fire with fire. If Flex/Silverlight/JavaFX threaten the open web, is there a way to compete on the same playing field? If the w3c technologies can't compete, can we take a different route? I propose that one very real solution to the problem would be to create an open source plugin technology to compete. It would allow a few things. First of all, it could ignore backwards compatibility because there would be nothing to be compatible to. Secondly, the cross browser issue would be resolved by being a plugin instead of a single browser implementation."

"I recently had a conversation with Brad Neuberg about the concept of using a plugin to have an Open Web competitor. Brad suggested that this was precisely what Google Gears was trying to do (sort of). In a recent post of his (which has since sparked a conversation across the blogosphere), Brad discussed the definition of the term Open Web and its importance, but also how Gears can help to push the web forward. In our conversation, he asked, "If you were to add functionality to Gears that doesn't enhance the web's existing technologies, but rather creates new ones that live in the browser through Gears what would these look like?" The following was my response:"

Russ wants better data-binding primitives and a refactoring of CSS to include variables, better layout management, and hooking into some kind of new component model for the web.

Russ also isn't gung-ho about JavaScript 2:

"I'm in the group of people that isn't totally gung ho about JS2. I understand the motivation, but its looking like a bit of a kitchen sink language. I think JS needs some improvements, but I'm actually looking at Erlang for inspiration instead of Java and Python. In my vision of a future JavaScript, I see a few things. First of all, I think there are some functional language features that would be good to add considering JavaScript is already a very functional language. I would like to add overloaded functions that use matching and guards for differentiation. I would also like to steal some aspects of the big Erlang feature of concurrent processes. Here, I think, is a perfect convergence with Gears. The Gears worker pools are a lot like Erlang processes (which I'm guessing you knew). No shared state, separate process, and no access to the dom. As I'm sure you know, this can be extremely helpful when trying to stay secure doing mashups, offload intensive operations from the main thread, and communicate with the server. Additionally, I think that there needs to be a little more in the way of modularizing code, allowing private data members, and facilitating better code reuse. I think prototypal inheritance and mixins are definitely better than classes for the language, and I'd like to add some more syntax to encourage them."

As they say in open source, code rules, so I told Russ to grab the Gears source and build it. We can have lots of discussions about how the web should be, but nothing speaks like actually getting a Gears module out that rolls some of the ideas above, at least in prototype form.

Labels: dojo storage, gears, open web

[Updated with correct ZIP file download link]

The ZIP file I put up for the new release of Dojo Storage was broken. Here is a new one with the critical bug fixed (basically the test file did not point to the right location). Also, I'm now hosting the new Dojo Storage release on my web server so you can play with the testing file here and the Hello World demo here to get a taste of Dojo Storage without having to download things yourself. Note that this is based on Dojo 1.1 and is a pre-release, since Dojo 1.1 will go out Real Soon Now (tm) but is not out yet. Thanks to Andrew Woolridge for pointing out that things were broken.

Labels: dojo, dojo offline toolkit, dojo storage, release

I've put up a simple ZIP file that has just Dojo Storage. This is an optimized build of Dojo with just the small files necessary to use Dojo Storage. I've also created a simple Hello World example that folks can use to get started with Dojo Storage quickly. Note that this is a pre-release version of Dojo 1.1 which is coming out soon but has not been wrapped up yet. This version has a fully functioning Flash Storage Provider, HTML 5 storage, and Google Gears storage.

Labels: dojo, dojo storage, flash, flash storage provider, gears

I am proud to announce a new beta release of Dojo Offline. This release has a huge amount of exciting new functionality, including a full port to Google Gears, a port from Dojo 0.4 to 0.9, and more.

Dojo Offline is an open-source toolkit that makes it easy to create sophisticated, offline web applications. It sits on top of Google Gears, a plugin from Google that helps extend web browsers with new functionality. Dojo Offline makes working with Google Gears easier; extends it with important functionality; creates a higher-level API than Google Gears provides; and exposes developer productivity features. In particular, Dojo Offline provides the following functionality:

• An offline widget that you can easily embed in your web page with just a few lines of code, automatically providing the user with network feedback, sync messages, offline instructions, and more
  
• A sync framework to help you store actions done while offline and sync them with a server once back on the network
  
• Automatic network and application-availability detection to determine when your application is on- or off-line so that you can take appropriate action
  
• A slurp() method that automatically scans the page and figures out all the resources that you need offline, including images, stylesheets, scripts, etc.; this is much easier than having to manually maintain which resources should be available offline, especially during development.
  
• Dojo Storage, an easy to use hashtable abstraction for storing offline data for when you don't need the heaviness of Google Gear's SQL abstraction; under the covers Dojo Storage saves its data into Google Gears
  
• Dojo SQL, an easy to use SQL layer that executes SQL statements and returns them as ordinary JavaScript objects
  
• New ENCRYPT() and DECRYPT() SQL keywords that you can mix in when using Dojo SQL, to get transparent cryptography for columns of data. Cryptography is done on a Google Worker Pool thread, so that the browser UI is responsive.
  
• Integration with the rest of Dojo, such as the Dojo Event system
  

To get started: See the Dojo Offline home page; read the new tutorial titled "Creating Offline Web Applications With Dojo Offline"; download the new Dojo Offline 0.9 beta SDK; and play with the demos.

Labels: announcement, dojo, dojo offline toolkit, dojo storage, google gears, release

Lucas Birdeau, one of the TIBCO GI folks, pointed me to this awesome Ajax web site he created with Dojo Storage and the Flash storage provider. He emailed me recently:

I just wanted to let you know that I had been using dojo storage on several GI applications and have had a really good experience. With all of the marketing behind the new entrants to this space, I think I've got a pretty robust implementation of dojo storage that really showcases how mature your technology is. Feel free to check it out. It's just a personal page that I put together on the side. It's a reorganization of craigslist that uses dojo storage to save and annotate posts.

http://remixedby.us/

Labels: dojo, dojo storage, flash

Every so often I look back over the projects I'm doing and take stock. I have a little inner thermometer that I check in on from time to time, and if I'm feeling stressed out and not having fun anymore it gives me information and a clue that I need to revisit what I've been doing. When I feel this, I create a Not Do Do List and figure out stuff that I need to let go of. Lately, I've been feeling stressed out and not having as much fun, feeling like there is a monkey on my back, so I've been doing things like morning pages, meditation, visual journaling, and other stuff to see what I need to change.

On a deeper level, I've been looking over what I'm good at and what I'm not good at. It seems my natural role is as an inventor pragmatist. I initially create an idea that is different than what is out there, envisioning something wierd and different, then do the hard work of making it initially pragmatic and real to prove to the world that it should be taken seriously whereas before it was ignored. Then, about a year or two down the line, others show up who have other skills that I am not as good at but are very important at this stage, such as community building, event planning, and more. I'm really good at brainstorming an idea, prototyping it, and then taking it into the initial reality stage with working code that can be used in the real world, in real software. I'm good at release 1 and 1.5, but not as good at release 2 and release 3. I'm not good at organizing groups of people, creating events, or cranking on the same code base once the initial idea and implementation has been proven to the community. I used to beat myself up about this, but I've realized that I just have to focus on what I'm good at and create situations where others can help me at the things I'm not as good at.

I've realized there are a number of projects that I created that have moved beyond this initial phase and are now successful and standing on their own legs, and it's time for me to find new maintainers for them and let them go, put them on my personal Not To Do List so to speak. These are coworking, the Really Simple History library, and Flash-based client-side storage. I need to find new maintainers for some of these projects so I can focus more time and attention on Dojo Offline. Also remember that almost all of the time I spend on these projects is free-time that I don't get paid for, and I only have so many hours in the day to hack on cool new stuff.

Coworking

I created coworking about 2 years ago. I had been working for a startup, Rojo, and wanted to go into business for myself as a consultant. However, I somehow wanted the community and structure of a workplace with the independence and freedom of working for myself, so I created coworking. Coworking was a way for independent contractors and workers to have a common space to work from, rather than hacking from home, with a more community minded focus. I rented some funky space from a small non-profit called Spiral Muse and ran it for about 8 months. Through a nice coincidence the Spiral Muse space was where I met my sweetheart, Bekka Fink, at a party months before. We couldn't store permanent stuff at the space, so I had to setup folding card tables every morning when I got there then break them down at the end of the day. For the first month I had no one, and just went to the space and worked there by myself, going through the action of setting up the folding card tables and wondering if anyone would ever show up. Eventually I had a full house, with an eclectic group of folks: a screenplay writer, a computer researcher, an open source hacker rolling his own computer language, and others.

From time to time people would drop by who were interested in the idea of coworking but weren't able to join the Spiral Muse space. I would tell them "if you were to steal this idea and make it your own, what would you do?" They would then wander off, a seed planted.

I had to close the Spiral Muse space after about 8 months since it was hard to grow within it's small size and it was getting hard to staff it; during that time those folks who had dropped by to check out the space started banding together to create an even better space, one that was larger and open more days of the week. This was the Hat Factory space, in San Francisco.

Since then coworking has spread around the world, with spaces already in or forming in New York, Paris, and more. There is an active Google Group with hundreds of members and a very active wiki. Coworking has been covered by Business Week, the New York Post, and more. Chris Messina and Tara Hunt are now the most active leaders of the coworking movement and are instrumental in moving it forward, creating events, organizing, and more. I've stepped out of an organizing role in the coworking movement to focus on other projects and am no longer a member of a space.

Really Simple History

I created the Really Simple History (RSH) library about two years ago. It was one of the first libraries to effectively solve the history and bookmarking issues inherent with Ajax applications. I had two goals with RSH: one, to show that history and bookmarking were important in Ajax applications (most developers at the time thought you could just throw this feature out and ignore that you were working in a browser); and two, that you could achieve this in contemporary web browsers using various browser tricks.

Today, RSH is used by a huge number of websites to solve Ajax bookmarking and history issues; it is also integrated into alot of other open source projects. Members from the Yahoo YUI team and the Google GWT team have told me that the BSD-licensed Really Simple History code base and techniques were instrumental in their own history libraries. RSH has been written up in online articles and books, and is now used by developers in their daily practice. Respecting the back button and creating a bookmark mechanism are now accepted practices in the Ajax industry, so I don't need to beat that drum anymore.

Unfortunately, I have not been able to do any real work on RSH for about 1 1/2 years. Important work remains: doing better QA on Internet Explorer 7, using one of the various Safari history/bookmarking techniques that have been discovered, fixing some bugs that can occur in various edge cases, and more. My focus on Dojo Storage and Dojo Offline have not allowed me to do work on this library; I am constantly getting emails with bug fixes and inquiries, and I feel terrible that I haven't gotten a new release out the door for awhile.

I am looking for a new maintainer for the RSH library. I want to make sure I find someone who will treat it well because alot of folks depend on it, so if you are interested please email me (bkn3 AT AT columbia.edu) and give me your qualifications, open source experience, and where you plan on taking the library.

Flash-Based Client-Side Storage

I've been hacking on the offline web problem for a few years. I knew before I could work on offline access to an application's user-interface, though, I had to figure out how to store data permanently on the client-side, beyond the 4K cookie limit. About 2 1/2 years ago I started searching around for a way to do this, experimenting with giant bookmarklets and other wierd stuff to see if I could find a clever back door. I then realized I could use a hidden Flash applet to store megabytes of info.

I rolled a prototype of this Flash based storage called AMASS (Ajax Massive Storage System) to see if the idea was tenable, and it turned out to work well. I then rolled another version of it and refactored it into Dojo, and got it to work in Safari as well; I also did a bunch of (grotty/interesting) hacks to get it super-fast for storing large datasets.

Once Flash-based storage stabilized, I moved on to the other part of the offline problem, which is finding your user-interface when you are away from the network, and rolled Dojo Offline in partnership with SitePen. Of course, one month after we released Google released Google Gears, a similar offline solution, so we created a partnership with them to port the Dojo Offline APIs on top of Gears. I'm in the middle of coding that port now. The next release of Dojo Offline will include a Dojo SQL module to use SQL-based storage, and a new Dojo Storage provider that uses Gears to store its data. Dojo Offline will no longer be dependent on the Flash Dojo Storage provider or the Firefox 2 Dojo Storage provider, and I would like to find a new maintainer for those. My preference for client-side storage is now to use Dojo Storage in conjunction with Google Gears. There are a number of startup companies that are using the Flash storage provider, however, and many users have told me they want to keep it; there are a number of enterprises that are also using it. I will port the Flash Storage Provider to the new Dojo 0.9 release that is coming out soon, but moving forward I would only like to personally maintain the GearsStorageProvider and the public Dojo Storage API. There are a number of cool performance and bug fixes that folks would like to get into the Flash Storage Provider, and this would be a great way to get your feet wet with the Dojo project. Please email me (bkn3 AT AT columbia.edu) your qualifications, open source experience, and future plans for the Flash Storage Provider if you would like to take this on.

Labels: ajax, coworking, dojo offline toolkit, dojo storage, flash, flash storage provider, personal, projects, rsh, storage