--> Why might GoogleBot get errors when trying to access my robots.txt file?. | Experience Lab - Online business creation and development guide for bloggers and startups

Why might GoogleBot get errors when trying to access my robots.txt file?.

A great thing about Google is, it gives webmasters all the help they need to get their websites into Google's index. There's a nice ...




GoogleBot




A great thing about Google is, it gives webmasters all the help they need to get their websites into Google's index. There's a nice tool available in Google Webmaster Tools called 'Fetch as GoogleBot'. This tool, as we discussed in our SEO Tips for start-ups, can be a great help diagnosing errors and getting a website in Google's index faster. A robots.txt file is used for crawling efficiency, and preventing certain pages from being crawled etc. Sometimes though, GoogleBot might have difficulty fetching your robots.txt file. Here's a solution from Google to this problem.









The original question asked o the GWT forum had to with crawling inefficiency. The GoogleBot was unable to crawl a robots.txt file 50% of the time, even though the file could be fetched from other hosts with a 100% success rate. It is worth noting that this was on a plain nginx server and a mit.edu host, so that should have a pretty good up-time. So the problem seems to be with Google, right?










Sometimes, people try cloaking on their websites. Cloaking means hiding content from crawlers, so that different content is served to crawlers and users. So what a user might see on their websites might be a lot different than what crawlers such as GoogleBot see. Not only is this a bad SEO practice, it can also have consequences.





During cloaking, people sometimes make a mistake, and end-up reverse-cloaking. So while browsers and user agents see the website fine, crawlers don't see any content. Making such a mistake is like axing your own foot. So this could be one of the reasons to the problem.





As we discussed about at the start, the Fetch as GoogleBot feature in Google Webmaster Tools is a pretty awesome tool. You can use it to fetch your robots.txt file. t will tell you when there's a problem. Many people might not know this, but sometimes, their web hosts might alternate between different systems and hosts. So a 50% success rate might be accounted for one of the hosts being improperly configured. You might want to contact your hosting company about this.





These two could be the most probable causes for robot.txt crawling errors. Did this help? Please do let us know. And stay tuned for more SEO questions and their answers :)

COMMENTS

Name

Affiliate Marketing,12,Announcement,34,Bing,9,Bitcoin,38,blog,7,Blogger Resources,42,Blogger Templates,4,blogger tricks,156,Blogging ethics,70,Blogging tips,198,Bugs and Errors,34,Business,9,Copyright Violation,9,CSS and HTMLTricks,95,Designs,8,drop down menu,7,eBook,12,Email Marketing,7,Events,30,Facebook,30,Facebook tricks,49,Google,157,Google AdSense,42,Google Analytics,7,Google Plus,51,Google Plus Tricks,38,Guest Posts,112,home,2,How To,77,Internet,1,JSON Feeds,25,Kitchen Recipes,2,Label Based Sitemap Themes,1,Make Money Online,108,Marketing,16,MBT Blogger Templates,7,Menus,1,News,146,Pages,1,Posts,10,presentations,15,Responsive,10,Reviews,7,SEO,307,Settings,6,Shortcode,15,Sitemap Themes,1,Social Media,155,Technology,7,Templates,1,Tips,2,Tools,1,Traffic Tips,80,Video,19,Web Designing,62,web hosting,18,Webmaster Tools,97,Widgets,199,wordpress,26,
ltr
item
Experience Lab - Online business creation and development guide for bloggers and startups: Why might GoogleBot get errors when trying to access my robots.txt file?.
Why might GoogleBot get errors when trying to access my robots.txt file?.
https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj0f6nYAoZQpzffRNsJqB1uqFvDGA4v2eB5seLN0PzkVCKW8a-ff9uiMsbZYWWxPNbQNCY-Ug_AiE7m2z79wPJ4-PAc5CaqnG6PhykDYVDuLzy_pFtAY_HfhgfFPr2o_9ydxWxZm9uwrdzI/s320/GoogleBot.png
https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj0f6nYAoZQpzffRNsJqB1uqFvDGA4v2eB5seLN0PzkVCKW8a-ff9uiMsbZYWWxPNbQNCY-Ug_AiE7m2z79wPJ4-PAc5CaqnG6PhykDYVDuLzy_pFtAY_HfhgfFPr2o_9ydxWxZm9uwrdzI/s72-c/GoogleBot.png
Experience Lab - Online business creation and development guide for bloggers and startups
https://www.experiencelab.info/2012/11/why-might-googlebot-get-errors-when.html
https://www.experiencelab.info/
https://www.experiencelab.info/
https://www.experiencelab.info/2012/11/why-might-googlebot-get-errors-when.html
true
2959477579779989044
UTF-8
Loaded All Posts Not found any posts VIEW ALL Readmore Reply Cancel reply Delete By Home PAGES POSTS View All RECOMMENDED FOR YOU LABEL ARCHIVE SEARCH ALL POSTS Not found any post match with your request Back Home Sunday Monday Tuesday Wednesday Thursday Friday Saturday Sun Mon Tue Wed Thu Fri Sat January February March April May June July August September October November December Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec just now 1 minute ago $$1$$ minutes ago 1 hour ago $$1$$ hours ago Yesterday $$1$$ days ago $$1$$ weeks ago more than 5 weeks ago Followers Follow THIS PREMIUM CONTENT IS LOCKED STEP 1: Share. STEP 2: Click the link you shared to unlock Copy All Code Select All Code All codes were copied to your clipboard Can not copy the codes / texts, please press [CTRL]+[C] (or CMD+C with Mac) to copy