Make Friend with Search Engine Robots

A robot is a mechanical intelligent agent which can perform tasks on its own, or with guidance. In practice a robot is usually an electro-mechanical machine which is guided by computer and electronic programming.

This mechanical agent is also used in indexing all websites available on the internet. It is primarily used by search engine such as Yahoo, Google, Bing and others when indexing website and do rankings.

Robots craws, evaluate, index and archive the pages of a specific site and send it on search engine server which then use for search results.

Important Search Engine Robots

The following are the most important search engine robots that you need to consider. The list include the robot name and link where you can try it.

• Google Search Googlebot/2.1 ( http://www.google.com/bot.html)
• Google Search Googlebot/2.1 ( http://www.googlebot.com/bot.html)
• Google Image Search Googlebot-Image/1.0
• Google Image Search Googlebot-Image/1.0 ( http://www.googlebot.com/bot.html)
• MSN Search msnbot/x.xx ( http://search.msn.com/msnbot.htm)
• MSN Search MSNBOT/0.xx (http://search.msn.com/msnbot.htm)
• MSN Media Search Robot msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)
• Windows Live Product Search msnbot-Products/1.0 (+http://search.msn.com/msnbot.htm)
• Microsoft Search for Mobiles MSNBOT_Mobile MSMOBOT Mozilla/2.0 (compatible; MSIE 4.02; Windows CE; Default)
• Alexa / The Internet Archive ia_archiver
• Alexa / The Internet Archive ia_archiver-web.archive.org
• Alexa / The Internet Archive ia_archiver/1.6
• Yahoo Blog Search Yahoo-Blogs/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/us/ysearch/crawling/crawling-02.html )
• Yahoo Multimedia Search Yahoo-MMAudVid/1.0 (mms dash mmaudvidcrawler dash support at yahoo dash inc dot com)
• Yahoo Product Search YahooSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/us/shop/merchant/)
• · Yahoo Product Search YahooSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5; http://search.yahoo.com/yahooseeker.html)
• Yahoo Product Search YahooSeeker/1.1 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/us/shop/merchant/)
• Ask/Teoma Search Mozilla/2.0 (compatible; Ask Jeeves)
• Ask/Teoma Search Mozilla/2.0 (compatible; Ask Jeeves/Teoma)
• Ask/Teoma Search Mozilla/2.0 (compatible; Ask Jeeves/Teoma; http://about.ask.com/en/docs/about/webmasters.shtml)

Robot Meta Tags

Like real robot, it also needs a machine language in order for you to communicate on it. Search Engine

Robot has several tags that you need to include on your site in order for robot to recognize your site.

<meta name="ROBOT NAME" content="ARGUMENTS" /> ROBOT NAME can be either “robots” for all robots or the user-agent of a specific robot. See robot user-agent list to the left.
	Arguments
noindex	Google, Yahoo, Live, Ask	Page Not Indexed
nofollow	Google, Yahoo, Live, Ask	All Links on Page Become No Followed
noarchive	Google, Yahoo, Live, Ask	Page Not Cached
noodp	Google, Yahoo, Live	Stops Description and Title Tag Overwrite by DMOZ (Only for Homepage)
noydir	Yahoo	Stops Description and Title Tag Overwrite by Yahoo Directory
nosnippet	Google	Stops Google from Generating Description Based on On-page Text

Enough knowledge on SEO tags includes these robot tags really helps on choosing right decision on how to market and leverage your site among the others. It also avoidscreating errors and penalizes by the search engines.

You can download the seomoz.org seo cheat sheet for more information about search engine robots.