User-agent: * Disallow: /cgi-bin/quoter Disallow: /cgi-bin/debug # Name Protect's bot. They have a stupid bot crawling the # web looking for infringment. That's nice, I suppose, but # not on my nickle, thanks. User-agent: NPBot Disallow: / # No idea who these monkeys are, but they specifically crawled the entirety # quoter against the above request (even after reading robots.txt). Worse, # they crawled all of quoter in about two minutes, slamming me. Idiots. # Hopefully they'll respect this. Ought to check back later, see if I can # track them down and send them hate mail. User-agent: NaverRobot Disallow: / # SEO and/or marketing robots User-agent: SemrushBot Disallow: / User-agent: AhrefsBot Disallow: / User-agent: dotbot Disallow: / User-agent: rogerbot Disallow: / # Mysterious and incredibly rude User-agent: The Knowledge AI Disallow: /