A few users have expressed interest in being able to ban robots from indexing their journal. That is, they don't want their parents to be able to search for their name on google and find their journal. Sure, they could make it private, but what fun is that?
So now there's an option at edit info to block robots. The www.livejournal.com robots.txt file is regenerated every hour. The users.livejournal.com/robots.txt file is generated dynamically.