Coding In Paradise

Via Niall Kennedy comes a PDF presentation on Flickr's architecture. Sweet.

Overview:

1. How Useful is Swarming and the Onion Networks SwarmStreaming Product?
2. What are the Potential Drawbacks of the SwarmStream Product?
3. Why BitTorrent is Important and SwarmStream is Doomed

How Useful is Swarming and the Onion Networks SwarmStreaming Product?

There are two seperate things to consider: the importance of the swarming concept and the SwarmStream product itself.

Swarming is an extremely powerful idea that is here to stay. In the short term it is redefining the speed and economy of sending large files around; it extends the benefits of an Akamai-like network to everyone, not just the Fortune 500. It is a robust solution for both flash crowds and serving content to many people over long periods of time.

In terms of the SwarmStream product, Justin Chapweske is to be credited with creating the swarming concept and introducing technologies and algorithms to achieve it. The SwarmStream product has an excellent Java API that makes it easy to integrate into existing applications as well as for use in enterprise settings. It also has extremely good stream control, offering progressive rendering of content as it is swarmed as well as pause and resume control. I believe it will have uptake by two primary crowds: forward-thinking enterprise developers who will use it as an efficient way to move large amounts of data either behind the firewall or over global WANs, as well as well-heeled large application companies like Real Networks that will integrate swarming into their client and server products using SwarmStream.

What are the Potential Drawbacks of the SwarmStream Product?

SwarmStream has two major drawbacks that I believe will spell its doom. The first major issue is its price and proprietary nature. The SwarmStream Development Kit is twenty-five thousand dollars to use in applications. While this is pennies for a large enterprise or application company, small startups, open source developers, and tinkerers can simply not use the SwarmStream API. This would not matter if BitTorrent did not exist as BitTorrent is the achilles heel of SwarmStream.

Why BitTorrent is Important and SwarmStream is Doomed

I predict that in several years BitTorrent will be a core Internet protocol. It is following exactly the same evolution as the RSS blogging protocols, which are themselves being integrated into the Internet's infrastructure and evolving beyond their humble roots. BitTorrent is experiencing the same kind of grassroots excitement and evolution that occured around the blogging standards, which produced RSS, OPML, XML-RPC, the blogging XML-RPC APIs, and more. It is the thin end of the wedge that will introduce P2P content distribution into mainstream Internet practices. Bram Cohen created an excellent protocol without biting off too much. Now, other like minded individuals are evolving this "good enough" protocol and extending its functionality and solving its issues incrementally. Some are attempting to distribute trackers, for example, using a variety of approaches such as distributed hash tables. Others are integrating BitTorrent with other existing protocols like RSS, audio streaming, and digital video. While the machinery of BitTorrent is only currently viable and realistic for large files above 3 megabytes, I do believe it will be slowly optimized to make it possible to swarm and retrieve content that is in the one hundred kilobytes to 1 meg file range, opening its use in distributing popular websites. Also in the long term I believe we will see BitTorrent transparently integrated with Mozilla's download manager, making BitTorrent connections through the browser as transparent as FTP ones are now, as well as innovations in making it extremely easy for anyone to host content that can be swarmed through BitTorrent. Even better, this evolved BitTorrent will be open, non-proprietary, and available in a variety of open source SDKs that SwarmStream will simply not be able to compete with. If SwarmStream were an order of magnitude better than BitTorrent than I might reconsider, but it is only incrementally better in some areas.

Update: Matthew Gertner posts a well-written, balanced response to this blog entry on his blog.