Would love to know the progress of this project.

Greg Elin http://twitter.com/gregelin

As for progress on greptheweb, I’d hoped to get a Trac site up to begin making inroads visible. Having a wiki would be handy for jotting down ideas for getting on with, and making code web-browsable is easier on anyone interested in looking at it than forcing them to check it out over svn. If I still can’t get Trac running on Dreamhost on the next attempt, I think I’ll either fork out for an account with WebFaction, VPSville or some such, or I’ll set up code & wiki hosting via Google code or similar.

I’ve parsed out a list of URLs from the Open Directory Project’s massive RDF index, and made some calculations about how large a minimally-greppable local copy of all the pages from those URLs would be: ~500GB. In other words, I’m not going to be able to fit the index on my Dreamhost account if I want to have any breathing room to speak of (and I do want breathing room, for photos and other bits and pieces). I picked up a cheap HP server recently, and put a couple of big SATA drives in it, so my plan is to use this for building the index on, assuming I can learn enough sysadmin skills to set it up with Debian or FreeBSD and indexing the web from behind my router. I’m a little concerned that my ISP will think I’m trying to overwhelm their DNS server if I’m sending out 3600 requests an hour, so this could be another hurdle. I might have to implement a round-robin system to query multiple DNS servers in sequence, or something.

Then there’s the question of how to store, and grep, a million or more web pages. So far, I’m thinking of using Nutch to build the index, because that’s exactly what it’s designed for. But Nutch doesn’t have the capability to query that index with regexps once it’s built. Also, regexp queries are expensive. So I think a hybrid approach might work:

• First, parse the submitted regexp to see if it contains any mandatory strings (in practice, many probably will). If it does, use Nutch’s built-in Lucene engine to search for entries with those strings (i.e. to filter out those that don’t).
• Then, run the regexp match on the remaining entries. How? Er, at this point it looks like I might have to write a Nutch plug-in…

Their back-end probably wouldn’t be very useful, but I immediately thought of the http://www.yubnub.org interface when I read this post.