whitehouse.gov’s open door policy. Even for spiders.
While doing research on robot.txt files, which sit on a web site and tell visiting search engine spiders what pages and directories to skip when they index your pages, I found this article at codeulate.com regarding the new whitehouse.gov robot.txt file.
Two days ago, under the Bush administration, the file listed over 2400 directories that were not searchable on whitehouse.gov. Paranoid much?
Under Obama, one.
Before:
After:
User-agent: *
Disallow: /includes/
No Comments »
RSS feed for comments on this post. TrackBack URL
![[ sturtle.com ]](http://queerandloathing.com/wordpress/wp-content/uploads/blogroll/sturtle.png "sturtle.com")
