Robots.txt turns 20

Happy Birthday, Robots.txt! The file known primarily for its ability to tell search engine crawlers which pages to crawl or ignore is celebrating its 20th anniversary. The Robots.txt concept was conceived in 1994 by Martijn Koster, who wanted to tell search engines like Lycos and Alta Vista which pages to crawl, and which to ignore …

Robots.txt turns 20 Read More »