Wednesday, December 29, 2004
Prototype Feed Mesh/Weblog Ping Server and Client Now Available
The Feed Mesh mailing list is a working group to discuss Feed Mesh technology for weblog ping aggregation and peering. The exact details of what a Feed Mesh ping implementation might look like are still being worked out, but in a hazy outline the goal is to have a high-performance, stable and distributed weblog pinging service that has protections against spamming and DoS attacks.
If you are new to weblog pinging, it is a simple way for weblogs to tell a centralized server that they have been updated. The centralized server, known as a ping server, then pushes whatever weblogs have changed to a file known as changes.xml, which contains a rolling list of all recently changed weblogs. Aggregators that wish to track changes can then grab changes.xml at regular intervals rather than hitting individual weblogs to get faster updates. From the original XML-RPC ping spec on weblogs.com by Dave Winer:
Weblogs.Com is a Web application that tracks changes to news-oriented websites. It was launched in late 1999, when XML-RPC was quite young and there weren't many weblogs. Today XML-RPC is widely deployed, and the weblog community has grown from a few hundred sites to many thousands.
To meet the growth, we've designed and are deploying a new mechanism for tracking updates to weblogs. The transition is fully described here. This page explains the XML-RPC mechanism for updates.
weblogUpdates.ping (weblogname, weblogurl, changesurl=weblogurl, categoryname="none") returns struct
To tell Weblogs.Com that a weblog has changed, call weblogUpdates.ping on rpc.weblogs.com, port 80, path /RPC2........
If the call succeeds, the weblog will appear in changes.xml.
One issue with existing ping servers is they do not scale well and introduce focal points for DoS attacks. Feed Mesh is attempting to craft a new distributed ping standard that scales better.
In support of this goal, Bob Wyman recently posted on the list the existence of a sample C++ program for reading Feed Mesh output:
We've updated http://sandbox.pubsub.com/pinginfo/ to include a small sample program written in C++ for reading FeedMesh ping feeds. The output of the program is a changes.xml file but it can easily be modified to do just about anything you need. Please try it out and provide feedback.
The missing piece is the code to generate feeds. Clearly, given the recent comments from Jeff Barr and Ben Hyde, we need to do a little more work on what we've got before that will be ready to release...
Once these three pieces are done, you'll have all the code needed to:
1. Receive either basic or extended pings and push them onto the processing queue
2. Generate a changes.xml file by reading a FeedMesh ping feed
3. Generate FeedMesh ping feeds
At that point, the code that will be left "as an exercise for the reader" will be that code which actually processes and validates incoming pings. From what I can see, that code will be very service specific since I think most implementations will be inextricably tied into other service-specific operations.
Also on the site are the beginnings of a mod Apache module, mod_pingservice:
Below are some files for the pingservice module we're writing to implement a high-performance, stable and distributed weblog pinging service. See the comments in the mod_pingservice.c file for more information on the aims of this project.
The apache module itself is placed under the Apache license; see the source file for more information. The sample file for reading from messages queues can be used for any purpose; it's really just for illustration.
If you are new to weblog pinging, it is a simple way for weblogs to tell a centralized server that they have been updated. The centralized server, known as a ping server, then pushes whatever weblogs have changed to a file known as changes.xml, which contains a rolling list of all recently changed weblogs. Aggregators that wish to track changes can then grab changes.xml at regular intervals rather than hitting individual weblogs to get faster updates. From the original XML-RPC ping spec on weblogs.com by Dave Winer:
Weblogs.Com is a Web application that tracks changes to news-oriented websites. It was launched in late 1999, when XML-RPC was quite young and there weren't many weblogs. Today XML-RPC is widely deployed, and the weblog community has grown from a few hundred sites to many thousands.
To meet the growth, we've designed and are deploying a new mechanism for tracking updates to weblogs. The transition is fully described here. This page explains the XML-RPC mechanism for updates.
weblogUpdates.ping (weblogname, weblogurl, changesurl=weblogurl, categoryname="none") returns struct
To tell Weblogs.Com that a weblog has changed, call weblogUpdates.ping on rpc.weblogs.com, port 80, path /RPC2........
If the call succeeds, the weblog will appear in changes.xml.
One issue with existing ping servers is they do not scale well and introduce focal points for DoS attacks. Feed Mesh is attempting to craft a new distributed ping standard that scales better.
In support of this goal, Bob Wyman recently posted on the list the existence of a sample C++ program for reading Feed Mesh output:
We've updated http://sandbox.pubsub.com/pinginfo/ to include a small sample program written in C++ for reading FeedMesh ping feeds. The output of the program is a changes.xml file but it can easily be modified to do just about anything you need. Please try it out and provide feedback.
The missing piece is the code to generate feeds. Clearly, given the recent comments from Jeff Barr and Ben Hyde, we need to do a little more work on what we've got before that will be ready to release...
Once these three pieces are done, you'll have all the code needed to:
1. Receive either basic or extended pings and push them onto the processing queue
2. Generate a changes.xml file by reading a FeedMesh ping feed
3. Generate FeedMesh ping feeds
At that point, the code that will be left "as an exercise for the reader" will be that code which actually processes and validates incoming pings. From what I can see, that code will be very service specific since I think most implementations will be inextricably tied into other service-specific operations.
Also on the site are the beginnings of a mod Apache module, mod_pingservice:
Below are some files for the pingservice module we're writing to implement a high-performance, stable and distributed weblog pinging service. See the comments in the mod_pingservice.c file for more information on the aims of this project.
The apache module itself is placed under the Apache license; see the source file for more information. The sample file for reading from messages queues can be used for any purpose; it's really just for illustration.
Subscribe to Posts [Atom]
