As I mentioned over on Searchlight, Yahoo Suggest is a quick-n-dirty, free keyword research tool that can serve as a nice alternative to Google Suggest. Google Suggest has Yahoo Suggest beat for convenience if you have the Google Toolbar installed, since you just start typing keystrokes into the toolbar's search field and it displays a dropdown list of suggestions (think: auto-complete). The suggestions are listed in order of popularity. This was how my daughter Chloe identified her top search term, "neopets cheats" for naming her site ("The Ultimate Neopets Cheats Site") - Google Suggest showed that "neopets cheats" was the second most popular search term after "neopets". But with Google Suggest, you can only compare search terms that start with the same keystrokes. You can't compare, say, "new cars" with "used cars". In that regard, Yahoo Suggest has a leg up on Google Suggest, since it returns search terms where your keystrokes may be in the middle or end. Try it out: go to Yahoo's home page or the Yahoo Search page and start typing in the search box.

I demoed this tool at my presentation at the Web 2.0 Expo yesterday, but the demo didn't quite work as expected. I typed in "cars", and as you can see below in the screenshot, the results were pretty bad. In particular, it was with suggestion #3 - "toyota car malaysia new car cars" - that I lost faith in this tool's data for keyword research.

I'm here at the Web 2.0 Expo in San Francisco. Today (actually yesterday now that it's after midnight) I sat in on the SEO Workshop being presented by Todd Friesen ("The Oilman") and Greg Boser ("WebGuerrilla").

This session turned out to be a lot of fun. It was reminiscent of their "SEO Rock Stars" radio show on Webmaster Radio. I cracked up when Todd plugged their show and explained the name of the show by saying: "Yes, we really are that arrogant."

Much more of the session than I was expecting turned to "black hat" or "gray hat" SEO tactics -- things that are outside the search engine guidelines. Both Todd and Greg believe in being pragmatic about SEO. Greg analogized SEO to speeding: nobody goes the speed limit; just don't drive recklessly swerving in and out of traffic so you get in a wreck. He "hates the guidelines" and longs for the good ol' days when the engines didn't have such idealistic guidelines and if you went too far, you were simply "torched forever and you're gone." Greg's guiding principle: "I don't want to upset my mother" -- i.e. he gauges whether he's gone too far based on whether she'd be unhappy with the search results because they're useless. They say the search engine guidelines are simply that: guidelines.

Hmm. I'm not sure I'd take that tack with an audience of webheads from Web 2.0 startups. An audience of SEOs at SES (Search Engine Strategies) is one thing, because they can be discerning about how far to take various bits of advice. But Web 2.0 geeks? That aren't savvy enough about SEO to know when they are playing with fire. Greg and Todd made compelling arguments for playing with conditional redirects (serving different destination URLs to Googlebot than to humans). But the 4 major engines had specifically warned against doing that in the session "Search Engine Q&A on Links" last Friday at Search Engine Strategies in NYC. So you'd better REALLY know what you're doing if you're going to play with that stuff! Todd and Greg also testified to the benefits of cloaking. One of their client's sites was cloaked for 3 years -- a new ecommerce platform was purchased and launched by the client, but the old HTML-based site was served up selectively to the bots because it ranked so well. I'm sure that was worth a pile of money to his client. But boy if you get that wrong (like not keeping up to date with all the ever-changing list of IP addresses associated with Googlebot), things could go very badly.

Todd made the point that if you are a big brand or company, you're not going to get kicked out of Google. Or at worst, maybe it'll be for a day. He cited eBay and the NY Times as examples of sites that won't get banned despite operating outside the Google Guidelines by serving millions of "related search" results and/or cloaking. Todd also cited Yahoo's new spam-reporting feature within Yahoo Site Explorer as evidence that the engines (and Yahoo in particular) aren't very good at webspam detection: "they're asking you to report the spam because they can't find it themselves." I was ROFLMAO when I heard that one. I doubt Yahoo would find that as funny though. ;-)

Speaking of Yahoo... Greg told the audience that he has consulted for Yahoo. Wow! That surprised me; impressed me too. Nice one, Greg!

Speaking of Yahoo again... Todd loves Yahoo's paid inclusion program (Search Submit Pro). That's because with it, Yahoo "gives you a complete pass on the off-page factors". Yahoo tells everyone publicly that Search Submit Pro doesn't improve your rankings, but it's not true according to Todd. Although you shouldn't expect a rankings lift with big money terms like "credit report" or "data recovery," it will lift your rankings on most non-ultracompetitive search terms. Nonetheless, I'd exercise caution with this one. That's because, as Todd admits, when you stop paying all your listings in the SERPs "will mysteriously vanish and it's really hard to get them back in for free." A bit of evil advice from Todd (I think he was only half-serious! ;-) ... "Submit your competitor and then turn it off". I was again rolling on the floor laughing at that one! I've heard of "Google bowling" your competitors into "Supplemental hell", but now I guess you might call this one "Yahoo bowling!"

Attendees got a little dose of conspiracy theory. Todd and Greg don't trust Google and their "don't be evil" mantra -- and they weren't afraid to spread a little hysteria into the audience. Google Analytics is one service they are very suspect of. They go far as to advise you don't use it: "giving your ROI data to the company you're buying your ads from -- that's just assinine." Personally, I trust Google. So much so that I'm on their waiting list for the Google Implant (beta). ;-)

Greg weighed in about the Open Directory, referring to it as "a dilapidated piece of crap." And the ODP editors? -- "they take bribes all the time." I'd love to know who the editors are that I can bribe and how much it'll cost!

Their recommended SEO tools were useful. For example: TouchGraph. They gave a very cool little demo of the tool in action. Todd said: "It's like Diggswarm, but actually useful. Is Kevin Rose in the room? He's going to kick my ass." LOL!

Some of the other free link research tools recommended:

• Yahoo's Site Explorer
• Netconcepts' Link Checker
• WeBuildPages' Neat-o Tool
• SEO for Firefox
• Tattler

Hmm. That's funny, this list of tools (except for the last one) looked mighty familiar. So did some of the previous slides. I couldn't resist... I got the microphone during Q&A, and I alluded to the slides, saying "Thanks for the plug for Netconcepts on that Tools slide." Well you won't believe what happened next! They admitted it to the audience that they lifted one of my decks from our website and then customized it! OMG!!

They praised me for making such excellent Powerpoints available online. Umm, thanks... I think! Geez!

Actually I'm not mad. Perhaps I should be, but they fessed up and poked fun at themselves for it -- in front of 300 people! -- so no harm done.

Just don't do it again, k? Or if you do, at least keep my Netconcepts logo on the slides!

What other nuggets did they share? Todd shared that Endeca can be a secret weapon for SEO when you know how to make it work for you. I concur with that statement, totally. For example, with our (Netconcepts') client Northern Tool, we were able to blow past previous search traffic records by deploying our GravityStream technology on top of Endeca Guided Navigation. Todd said that one of their (Range's) biggest SEO successes was with Endeca. Todd gave a word of caution though: "Out of the box, Endeca is just a piece of junk for SEO". The good news though is that, if you know what to ask for, "they will customize."

They also recommended Google's "Domain Park" program if you are a domainer with a bunch of domains you want to monetize.

They also advised folks to stay out of Google's contextual advertising space so that you don't get your AdWords campaigns onto the domain parking pages, because it is junk traffic that doesn't convert.

What was the most interesting revelation of the day? That Todd was a Viagra webspammer before working for Range. I knew he was a reformed black hat SEO, but I never knew what industry. Greg joked that in Todd's space, "PPC" stood for "Pills, Porn and Casinos" rather than "Pay Per Click." Todd regaled us with tales of "churn and burn" SEO going after Viagra rankings -- he could make $10k in a week, and all it cost him was a day of his time and $7 for a domain name -- then the site would get banned and he'd move on to using another domain name. "Viagra" would have to be the most competitive keyword in SEO bar none (well, with the probable exception of "sex"). Todd *must* have been a "rock star" at SEO (but on the Dark Side of the Force of course! ;-) to make such a good living from it. So the name of his Webmaster Radio show really IS fitting and merited. I highly recommend downloading some audio recordings of previous SEO Rockstars episodes; even Matt Cutts has been known to listen to their show!

Rock on!

I've never had much faith in the estimated number of search results returned by Google (for good reason). Every so often I run queries that I know will return the largest results set possible and I note the number of pages found. For example: searching for "the" and "http". And boy do the supposed number of results keep getting bigger and bigger! Like the length of the preverbial "fish that got away" with each telling of the tale...

Today when I searched for "http", Google returned 15,470,000,000 results. That doesn't seem too out of whack until you consider that only 8 months ago (September 2005) that same query on Google returned 2,380,000,000 pages. (Back in November 2004 that query returned 588 million pages.)

We could chalk that monumental increase up to changes in Google's algorithm in how they handle that unusual search term. So let's look at another large-result-set query...

Today a Google search for "the" returned 23,850,000,000 pages. In September 2005 that query returned 3,420,000,000 pages.

And today a search on Google for "www" returned 25,270,000,000 results. In September 2005 that number was 3,500,000,000.

Then, a quick dive over to Yahoo revealed that the estimated results numbers actually DROPPED over the past 8 months. Yahoo today returned 1,400,000,000 for "http" compared to 2,130,000,000 in September 2005. Today Yahoo returned 7,370,000,000 for "the" compared to 11,200,000,000 in September. And today 8,810,000,000 for "www" compared to 14,300,000,000 last September.

So Google's estimated numbers of results appear to be ballooning while Yahoo's are shrinking. Hmmm.

Moral of this story: Don't bet your life on the numbers of results returned by Google. They're most certainly overrepresented.

Of course this is based on anecdotal evidence. I'd love to hear if anyone has done a more thorough study of estimated numbers of search results over time.

Just when you thought Google was the Big Kahuna, a report comes out to challenge your assumptions. Like this one from BIGresearch which yields some new insights into which search engines have the most influence on buying decisions. Hint: It's not Google.

The study looked at the influence of Internet advertising on purchase decisions and the findings may surprise you. Yahoo was ranked overall the #1 search engine choice for influencing purchase decisions and Google #4. I'm not terribly surprised that Yahoo! leads the pack here. Even though you can't (and shouldn't) sterotype searchers, I nonetheless tend to imagine the prototypical Google searcher as more savvy and research-oriented and the prototypical Yahoo! searcher as a ready buyer easily parted from his/her money. What does it say about the average Yahooer that one of the most popular search terms entered into Yahoo! is the word "google"? ;-) To the tune of 34,960,495 searches last month alone! This according to their Keyword Selector Tool. Can't they figure out how to type google.com in the Location bar?

You also have to consider too that Yahoo puts the ads more in-your-face, so to speak, than Google.

Nonetheless, the rankings do move around once you start segmenting. Sometimes Google comes out in front — like in the Electronics sector. Here are some key sectors ranked in order of search engine preference:

Electronics
1. Google
2. Yahoo!
3. MSN
4. AOL
5. Ask Jeeves

Apparel/Clothing
1. Yahoo!
2. AOL
3. MSN
4. Google
5. Ask Jeeves

Car/Truck
1. MSN
2. Yahoo!
3. Google
4. Ask Jeeves
5. AOL

Grocery
1. AOL
2. Yahoo!
3. Ask Jeeves
4. MSN
5. Google

Hmm.... This might cause one to reassess his/her assumptions and priorities (and BUDGET) around paid search!

BIG hat tip to Shimon Sandler!

If you think SEO is hard now, just wait until search engines start varying the results for each individual, depending on their profile, search history, geographic location, and now... mindset. Yahoo Mindset is Yahoo Labs' foray into intent-driven search, where sorting of the results are partially dependent upon the searcher's intentions (whether they are in a shopping frame-of-mind or a research frame-of-mind). Yahoo is using machine learning to score web results as commercial or informational. Scores range from -2 (most commercial) to +2 (most informational).

It's a pretty cool idea but I don't think they've quite nailed it yet. (They do refer to it as a "demo" on the home page so I can't be too hard on them!) The big question is: does Yahoo decide what each page is, or does it decide on the whole site? Imagine being a commercial site and Yahoo decides you are researching only.

A trawl through some common search queries reveals that results on the research end are no less commercial than the commercial end, but the slider bar on the left of each result is different, so maybe the machine learning algo is scoring the pages differently.

Fold in personalized search (where search results vary depending on the searcher's profile and previous search behavior) and local search (where results vary by the searchers' geographic location) and then it starts getting really interesting for us SEOs. It'll make life a lot tougher for optimizers because we'll need three times the content if we want to be found at the commercial and research ends of the search, as well as in the middle.

Just as with personalized search, local search, and other innovations coming out of the search engines' labs, intent-driven search is yet another nail in the coffin of old-school SEO.

From an SEO standpoint, there is consensus among experts - Google, Yahoo and MSN are it. However, there’s a yappy little underdog called Teoma, which, despite its size, is a good contender in the search engine stakes. Teoma, which means "expert" in Gaelic, is owned by Ask Jeeves and powers the algorithmic search results on their properties (like Ask.com, Excite.com).

Yes, there's a big technology difference between Teoma and the other "big three" but Teoma does it differently with its localized approach. As Ammon Johns explained it in the MarketingProfs Thought Leaders Summit on SEO:

PageRank and link popularity is a bit like going out into the street and asking everyone who the best scientist is — you are going to get the obvious names: Albert Einstein, Stephen Hawking. They’re popular answers.

Teoma looks within the topic. It finds the authority sites within the topic related to "scientists" and then asks "Who is the best scientist?" Chances are, Teoma is going to come up with names you have never heard of before, but are actually much better answers. It gives you the specialist answer instead of the popular answer.

Another difference with Teoma is that it is keyword-dependent. So when you type "blue widgets" into that search box, it pulls the community together and conducts a local search which refines and finds the authoritative sites on that particular subject.

The model of organizing the Web into topical communities and pinpointing the authorities (pages that have garnered a lot of inbound links from reputable, topically-relevant pages) and the hubs (those pages that link to a lot of reputable, topically-relevant pages) is an important model to grasp (read more about it in Mike Grehan's paper on topic distillation), because I predict that all the major engines will become keyword-dependent over time. If you grasp this concept now and are picky about the sites you garner links from and link to, then you'll be doing a lot to futureproof your SEO.

Ever feel like you blinked and you missed some hugely important new trend online? I admit it; sometimes I feel that way. I take a break from my RSS aggregator for a day and I feel totally out of the loop. But I usually catch up pretty quickly.

For those of you who don't live and die by your RSS aggregator (i.e. you have a life outside of your computer), there's some exciting stuff happening:

• Flock is reinventing the web browser
• 37 Signals' Writeboard is reinventing collaborative authoring
• Rollyo and Eurekster's Swicki are reinventing vertical search
• Memeorandum is reinventing how we consume tech news and spot trends
• Ning is reinventing the way we build complex, socially networked web applications

I love it when disruptive technologies leave "old school" companies wondering "Wha' happen'd?". One behemoth that won't be left in the dust, though, is Yahoo. Despite Yahoo's size, they are poised to cash in big-time to the transition of the Web from passive browsing to a platform for collaboration and social networking (in other words, Web 2.0). Yahoo!'s made some really smart Web 2.0-ish purchases lately, including Flickr and Upcoming.org. Not to be outdone there's also Ebay who just bought Skype and AOL who just bought Weblogs, Inc.

I'm a bit behind on my conference session blogging. Waaay too many parties going on; doesn't leave much time for blogging. The Google Dance last night. Yahoo! party at Great America the night before. And tonight I've got another party to go to. Yesterday I spoke on RSS. I'll post a recap on that session later.

I just attended "Search Engine Q&A On Links", which was great. Lots of useful advice from Google and Yahoo! about linking (nobody seemed to want to ask poor Ask Jeeves any questions). It was funny how obviously diametrically opposed the engines were to the immediately prior session on "Buying and Selling Links". It's hard to reconcile the two different sets of advice. Matt in the hallway before this session was adamant: "Don't buy links!"

Anyways, without any further ado, here's the session recap:

Kaushal Kurapati from Ask Jeeves:
Be cautious of: reciprocal links and purchasing links
Avoid: link farms, cloaking pages, invisible or hidden links that trick the crawler
Become an authority on a subject
Focus on your busines and content. Rest will follow. [I say: "yeah, right..."]
Teoma uses subject specific popularity: garner respect in your industry, subject-specific text based links can be understood. (hubs and authorities model)

Tim Mayer from Yahoo!:
Here's some important news!! Yahoo! has just launched a brand new service: Site Explorer from Yahoo! Search. Stop scraping the Yahoo site for backlink results and use Site Explorer instead. Access via an API is offered too. And you can export as a CSV file.
Yahoo has 19.2 billion web objects in its index. Over 20 billion objects, when you include the audio and video.
Plans to use community to improve search quality. Social search = within a trusted network, where someone within your network vouches for a site.
Create natural linking strategies. when things start to look unnatural, is when you'll start getting into trouble. We look at intent (linking to plasma TVs, diamonds, and Viagra all on the same page) and extent (i.e. what looks normal. Having everything on the page as links or 200 links on the page is too much!)
Yahoo! offers a much more comprehensive sample of backlinks than Google, but not a complete set of backlinks. New system (Site Explorer) will be reasonably comprehensive, in his opinion the most comprehensive out there.
It's unnatural to link to sitemap-1 sitemap-2 sitemap-3 sitemap-4 sitemap-5. If you are doing this, you're headed in the wrong direction.

Matt Cutts from Google:
Good links are earned links, links that are based on editorial discretion.
Create services that really useful. e.g newsletters, an article a day, syndicate through RSS (attribute my article and give me a link). start a blog.
Matt launched his blog today: mattcutts.com
Think outside the box.
Only SEOs and librarians do backlink searches. Historically we decided to dedicate a subset of our servers to backlinks. Only a sampling of backlinks would be displayed but only for a threshold of PageRank 4 or higher pages. A suggestion was made to show backlinks for lower PageRank pages too. We liked that idea so we now show a random sampling of backlinks, including low PageRank scoring pages too. We show twice as many backlinks as shown before, but still it's only a sampling of the backlinks.
In graph theory, a clique in every node in the graph is very unnatural. So don't link to every single node in your network of sites; it'll get flagged.
For dynamic sites, you're very safe if you have fewer than 2 parameters; keep the values of those parameters to fewer than 5 digits, and don't name a parameter "id". Googlebot sometimes tries variations of URLs by dropping parameters, but we only do that deep level analysis on big, quality sites.
Another good approach that alltheweb came up with: spider would always go 1 dynamic page deep from a static page.
Search engines only grab 100k or 200k or 500k so be careful loading up a huge page with a lot of links.
PageRank isn't as important as SOME people make it out to be. BUT it's NOT like "PageRank? Oh yeah let's shuffle that one under the rug! That was sooo 4 years ago!"
"BO" = backlink obsession
We export PageRank only once every 3 months or so.

Technorati tag: Search Engine Strategies

I find it a bit unbelievable that the major search engines — Google, Yahoo!, MSN Search, and Ask Jeeves — still don't offer RSS feed searching combined with RSS search results feeds as part of their Web search. Specialized RSS feed search engines like Feedster, PubSub and Technorati have risen to the occasion, filling the void left by the major engines' inaction. Bloglines, the AskJeeves-owned company, has announced a blog/RSS search engine service that'll compete with Feedster, PubSub, and Technorati, but still that's a far cry from embedding RSS search right into the Web search box.

Here's how each of the majors handles RSS feeds:

Google:

• has URLs of valid RSS feeds in its index (due to links that point to those feeds)
• doesn't recognize the XML file format of RSS feeds (as you can read on the excerpted screenshots above)
• only rarely indexes the feed (I base that not just on the fact that nearly all RSS feeds are shown in Google results with no title or snippet as in the first screenshot above, but also because, out of 64,000 RSS feed files hosted by feeds.feedburner.com, only 19 are shown to contain the word cheese, the last 2 of which show up in the results only because cheese appears in links pointing to the feed; yet the same search on Yahoo! shows over 400. So clearly a lot of files that should have matched are missing from the Google search results.)
• only rarely caches the XML (see example) with most caches being blank (like this)
• associates words in links pointing to the page (as demonstrated with this search)
• doesn't allow refining of your query with the operators — filetype:rss, filetype:xml, or filetype:rdf

Yahoo:

• has URLs of valid RSS feeds in its index
• indexes the feed (Evidenced by above screenshot, which was a match for a search on text contained within the feed. Also, ResearchBuzz found this to be the case too.)
• caches the XML (see example)
• doesn't display the "Add to My Yahoo!" link for RSS feed listings (this is a disappointing omission, as Yahoo! displays this link on listings for HTML pages that have an associated RSS feed but not for the listing of the RSS feed itself)
• associates words in links pointing to the page
• doesn't allow refining of your query with the operators — filetype:rss, filetype:xml, or filetype:rdf

MSN Search:

• doesn't have URLs of valid RSS feeds in its index (Evidence of this: not a single feed out of 64,000 feeds at feeds.feedburner.com is displayed, even though there are links that point to those feeds. Note that the couple feeds that are displayed are not valid feeds but error pages outputted in HTML.)
• doesn’t recognize the XML file format of RSS feeds (file type is displayed in the search listing after Cached link when it's a recognized non-HTML file type)
• doesn't index the feed
• doesn't cache the XML
• doesn't allow refining of your query with the operators — filetype:rss, filetype:xml, or filetype:rdf

Teoma (Ask Jeeves):

• has URLs of valid RSS feeds in its index
• indexes the feed
• (View Cached feature not supported by Teoma)
• associates words in links pointing to the page
• (filetype: operator not supported by Teoma)

As you can see from my little comparison, MSN Search is the farthest behind when it comes to RSS feed indexing. Hopefully Scoble will read this and tell the MSN Search team to get on the ball. ;-)

Even though the major engines have been slow to make RSS an integral part of their indices, I predict that the engines will, within the next year or so, wake from their slumber and overtake and even acquire their specialized RSS feed search engine competitors.

What that will mean for web marketers is that search engine optimizing RSS feeds will become a science unto itself (currently it's limited mainly to optimizing the item titles for purposes of link text on syndicating sites) and that the feeds that are not optimized will get drowned out by those that are.

I'm in sunny Orlando at the Annual Catalog Conference... In yesterday's Q&A With the Search Engines session, Matt Lydon, Senior Sales Director of Search at MSN, stated quite categorically that they will not offer paid inclusion, the practice where advertisers can buy their way into the natural (organic) search results (but without any control over actual rankings).

Google frowns on the practice of paid inclusion and has previously vowed never to offer it, whereas Yahoo! offers it through their Site Match Exchange program.

I wonder if at some point the FTC will put a stop to paid inclusion because there is no disclosure to the searcher that it's a paid search result...