Stephan Spencer's Scatterings

The Scattered Wisdom of a scientist turned web marketing virtuoso

March 2010
S M T W T F S
 << <   > >>
  1 2 3 4 5 6
7 8 9 10 11 12 13
14 15 16 17 18 19 20
21 22 23 24 25 26 27
28 29 30 31      

Ask Jeeves wants your Robots.txt!

David Naylor from Bronco, who was one of the speakers at the Organic Listings Forum session at the Search Engine Strategies conference, advised site owners to have a robots.txt file, even if it's just an empty file, because Ask Jeeves' spider seems to fa… more »
Posted by Stephan Spencer on 12/10/2005 | Permalink

Comments (2)| Comments RSS | Filed under: Search Engines ask jeeves, error_log, robots.txt, search_engine_strategies, spiders            

What's wrong with Google Sitemaps

Last Friday it seemed like the whole blogosphere was abuzz with the news that Google unveiled its new Google Sitemaps service, a free inclusion service where you publish an XML file of your site pages to Google so its spider can get a better sense of wha… more »
Posted by Stephan Spencer on 06/06/2005 | Permalink

Comments (0)| Comments RSS | Filed under: Search Engines google, google sitemaps, googlebot, gravitystream, pagerank, pagerank dilution, pagerank score, spiders            

Google's index hits 8 billion pages. Yes folks, size does matter.

On Wednesday, the day before Microsoft unveiled the beta of Microsoft Search, Google announced that their index was now over eight billion pages strong. Impeccable timing from the Googleplex. Just a couple days later, and Microsoft could have proudly tou… more »
Posted by Stephan Spencer on 11/14/2004 | Permalink

Comments (2)| Comments RSS | Filed under: Search Engines google, search_engines, spiders, spider_trap            

Free pass into password-protected content

Many sites that require registration or payment in order to access their premium content have realized that they can't keep the search engine spiders (such as Googlebot and Yahoo Slurp) out of their password protected areas or they take a serious hit on… more »
Posted by Stephan Spencer on 11/02/2004 | Permalink

Comments (2)| Comments RSS | Filed under: Search Engines search_engines, spiders            

Spiders like Googlebot choke on Session IDs

Many ecommerce sites have session IDs or user IDs in the URL of their pages. This tends to cause either the pages to not get indexed by search engines like Google, or to cause the pages to get included many times over and over, clogging up the index with… more »
Posted by Stephan Spencer on 06/25/2004 | Permalink

Comments (1)| Comments RSS | Filed under: General ecommerce sites, googlebot, pagerank, pagerank_dilution, seo, session id, spiders, spider_trap, urls