Search This Blog

Aug 22, 2011

Make Friend with Search Engine Robots

A robot is a mechanical intelligent agent which can perform tasks on its own, or with guidance. In practice a robot is usually an electro-mechanical machine which is guided by computer and electronic programming.

This mechanical agent is also used in indexing all websites available on the internet. It is primarily used by search engine such as Yahoo, Google, Bing and others when indexing website and do rankings.

Robots craws, evaluate, index and archive the pages of a specific site and send it on search engine server which then use for search results.

Important Search Engine Robots

The following are the most important search engine robots that you need to consider. The list include the robot name and link where you can try it.


  • Google Search Googlebot/2.1 ( http://www.google.com/bot.html)
  • Google Search Googlebot/2.1 ( http://www.googlebot.com/bot.html)
  • Google Image Search Googlebot-Image/1.0
  • Google Image Search Googlebot-Image/1.0 ( http://www.googlebot.com/bot.html)
  • MSN Search msnbot/x.xx ( http://search.msn.com/msnbot.htm)
  • MSN Search MSNBOT/0.xx (http://search.msn.com/msnbot.htm)
  • MSN Media Search Robot msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)
  • Windows Live Product Search msnbot-Products/1.0 (+http://search.msn.com/msnbot.htm)
  • Microsoft Search for Mobiles MSNBOT_Mobile MSMOBOT Mozilla/2.0 (compatible; MSIE 4.02; Windows CE; Default)
  • Alexa / The Internet Archive ia_archiver
  • Alexa / The Internet Archive ia_archiver-web.archive.org
  • Alexa / The Internet Archive ia_archiver/1.6
  • Yahoo Blog Search Yahoo-Blogs/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/us/ysearch/crawling/crawling-02.html )
  • Yahoo Multimedia Search Yahoo-MMAudVid/1.0 (mms dash mmaudvidcrawler dash support at yahoo dash inc dot com)
  • Yahoo Product Search YahooSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/us/shop/merchant/)
  • · Yahoo Product Search YahooSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5; http://search.yahoo.com/yahooseeker.html)
  • Yahoo Product Search YahooSeeker/1.1 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/us/shop/merchant/)
  • Ask/Teoma Search Mozilla/2.0 (compatible; Ask Jeeves)
  • Ask/Teoma Search Mozilla/2.0 (compatible; Ask Jeeves/Teoma)
  • Ask/Teoma Search Mozilla/2.0 (compatible; Ask Jeeves/Teoma; http://about.ask.com/en/docs/about/webmasters.shtml)


Robot Meta Tags


Like real robot, it also needs a machine language in order for you to communicate on it. Search Engine

Robot has several tags that you need to include on your site in order for robot to recognize your site.


<meta name="ROBOT NAME" content="ARGUMENTS" />

ROBOT NAME can be either “robots” for all robots
or the user-agent of a specific robot.
See robot user-agent list to the left.

Arguments

noindex
Google, Yahoo, Live,
Ask
Page Not Indexed
nofollow
Google, Yahoo, Live,
Ask

All Links on Page
Become No Followed

noarchive
Google, Yahoo, Live,
Ask

Page Not Cached

noodp
Google, Yahoo, Live

Stops Description and Title
Tag Overwrite by DMOZ
(Only for Homepage)

noydir
Yahoo

Stops Description and
Title Tag Overwrite by
Yahoo Directory

nosnippet
Google

Stops Google from
Generating Description
Based on On-page Text





Enough knowledge on SEO tags includes these robot tags really helps on choosing right decision on how to market and leverage your site among the others. It also avoidscreating errors and penalizes by the search engines.

You can download the seomoz.org seo cheat sheet for more information about search engine robots.

2 comments:

Paul said...

By using robots tag, You can tell the
search engines to index all of your web page, only the ones promoted, or none of your WebPages... nice article of robots tag for SEO purposes

sportsnoice said...

Wow.. First time I see about different types about robot names and how it works in different search engines.

web design and development