Stephan Spencer's Scatterings

The Scattered Wisdom of a scientist turned web marketing virtuoso

November 2008
S M T W T F S
 << <   > >>
            1
2 3 4 5 6 7 8
9 10 11 12 13 14 15
16 17 18 19 20 21 22
23 24 25 26 27 28 29
30            

Ask Jeeves wants your Robots.txt!

David Naylor from Bronco, who was one of the speakers at the Organic Listings Forum session at the Search Engine Strategies conference, advised site owners to have a robots.txt file, even if it's just an empty file, because Ask Jeeves' spider seems to favor web sites that have one.

Anyone noticed an improvement with your presence in Ask Jeeves after creating a robots.txt file?

Of course there's also the side benefit that you'll eliminate all those "File Not Found" error messages for robots.txt in your server error log, which tend to overwhelm the error log, making it harder to spot more concerning error messages. That assumes of course that you actually examine your error log on occasion. ;-)

Posted by Stephan Spencer on 12/10/2005 | Permalink

Comments (2)| Comments RSS | Filed under: Search Engines , , , ,