Google Webmaster Tools Releases Robots Testing Update

The robots.txt exclusion standard is a bit of an ambiguous convention, acting as sort of the opposite of a sitemap. But while one might see the value in telling a web crawler what pages are on your site, and how they’re structured, it’s more difficult to see the value of telling them where they can’t [...]