Make Googlebot crawling smoother by using Webmaster Tools

Major search engines, including Google, provide free tools for webmasters. Google's Webmaster Tools help webmasters better control how Google interacts with their websites and get useful information from Google about their site. Using Webmaster Tools won't help your site get preferential treatment; however, it can help you identify issues that, if addressed, can help your site perform better in search results.

With the service, webmasters can:

• See which parts of a site Googlebot had problems crawling 
• Notify search engine of an XML Sitemap file 
• Analyze and generate robots.txt files 
• Remove URLs already crawled by Googlebot 
• Specify your preferred domain 
• Identify issues with title and description meta tags 
• Understand the top searches used to reach a site 
• Get a glimpse at how Googlebot sees pages 
• Remove unwanted sitelinks that Google may use in results 
• Receive notification of quality guideline violations and request a site reconsideration

Yahoo! (Yahoo! Site Explorer) and Microsoft (Bing Webmaster Tools) also offer free tools for webmasters.