Stephan Spencer's Scatterings

The Scattered Wisdom of a scientist turned web marketing virtuoso

July 2010
S M T W T F S
 << <   > >>
        1 2 3
4 5 6 7 8 9 10
11 12 13 14 15 16 17
18 19 20 21 22 23 24
25 26 27 28 29 30 31

Free pass into password-protected content

Many sites that require registration or payment in order to access their premium content have realized that they can't keep the search engine spiders (such as Googlebot and Yahoo Slurp) out of their password protected areas or they take a serious hit on their search engine traffic and visibility. Therefore, they let their search engine spiders in, but keep humans out (at least those who don't have an account, of course). Smart humans can take advantage of the back doors the spiders get shown by simply going into Google or Yahoo and doing a search that is site-specific (using the site: query operator). Then, in the search results, click on the Cached link in the search listing of the page that you wish to read. No Cached link present? Then try clicking on the title of the search listing. You may get redirected to a password entry page, but in many cases you will get through to the content! This is because subscription sites often times let search engine users go just one page deep without requiring log-in. So, after reading that page, simply go back to the search results and click through again to read another page. This works on LATimes.com, ChicagoTribune.com, Webmasterworld.com, and many others. Try it out. Enjoy!

Posted by Stephan Spencer on 11/02/2004 | Permalink

Comments (2)| Comments RSS | Filed under: Search Engines search_engines, spiders            

2 comments

  1. wouldn't it be easier to change your user agent to Googlebot? (Oh, you'll need firefox.)

    Comment by sean [Visitor] — 11/02/04 @ 21:50


  2. That only works if you the website detects search engine bots by user agent. It won't work if they do IP-based bot detection. Brilliant suggestion though!

    Comment by Stephan Spencer [Visitor] Email · http://www.stephanspencer.com — 11/02/04 @ 23:45


Leave a comment


Your email address will not be revealed on this site.

Your URL will be displayed.
(Line breaks become <br />)
(Name, email & website)
(Allow users to contact you through a message form (your email will not be revealed.)