--> How To Test Your Robots.txt File In Google Webmaster Tools? | Experience Lab - Online business creation and development guide for bloggers and startups

How To Test Your Robots.txt File In Google Webmaster Tools?

A robots.txt file can be important for a sufficiently large website that might want to keep some of its page away from search engines. M...



Test your robots.txt file

A robots.txt file can be important for a sufficiently large website that might want to keep some of its page away from search engines. Most smaller websites won't even need this file, but for others, it can be sometimes difficult to make and maintain the correct robots.txt files, as finding the directives within a large robots.txt that are blocking individual URLs can be tricky. For that, you need a debugging or testing tool to check out any problems with your robots file. And as always, Google is there to help.


The Google Webmaster Tools dashboard now has an updated robots.txt testing tool which would let webmasters find problems with their robots.txt files!


Robots.txt testing tool



You can see that this tool is available under the Crawl Section in Google Webmaster Tools.




robots.txt tester




In this sub-section, you can see the current robots.txt file associated with your website, and you can test new URLs to see whether they're allowed or disallowed for crawling. Under your file, you can see a text field where you enter the URL in question.








Finalize your robots.txt file in the editor, and then test a URL to see if you've got the directive right, and that it works properly. The changes take effect live, so you can add more directives on the spot to test them. Of course, you will have to copy and paste these new directives on your production site's hosting server separately, since Google Webmaster Tools has no way of updating that automatically.




If the URL actually gets blocked, the tool will guide you through complicated directives, and will highlight the specific one that lead to the decision to clock that URL. This way, you can really check whether a certain rule you set up works or not.




Additionally, you'll be able to review older versions of your robots.txt file, and see when access issues block Googlebot from crawling. For example, if Googlebot sees a 500 server error for the robots.txt file, Google will generally pause further crawling of the website.




You can also combine it with other parts of Webmaster Tools: for example, you might use the updated Fetch as Google tool to render important pages on your website. If any blocked URLs are reported, you can use this robots.txt tester to find the directive that's blocking them, and, of course, then improve that. A common problem comes from old robots.txt files that block CSS, JavaScript, or mobile content — fixing that is often trivial once you've seen it.



Do give this tool a go, and tell us what you think in the comments section below. Cheers :)

COMMENTS

Name

Affiliate Marketing,12,Announcement,34,Bing,9,Bitcoin,38,blog,7,Blogger Resources,42,Blogger Templates,4,blogger tricks,156,Blogging ethics,70,Blogging tips,198,Bugs and Errors,34,Business,4,Copyright Violation,9,CSS and HTMLTricks,95,Designs,8,drop down menu,7,eBook,12,Email Marketing,7,Events,30,Facebook,30,Facebook tricks,49,Google,157,Google AdSense,42,Google Analytics,7,Google Plus,51,Google Plus Tricks,38,Guest Posts,112,home,2,How To,77,Internet,1,JSON Feeds,25,Kitchen Recipes,2,Label Based Sitemap Themes,1,Make Money Online,108,Marketing,16,MBT Blogger Templates,7,Menus,1,News,146,Pages,1,Posts,10,presentations,15,Responsive,10,Reviews,7,SEO,307,Settings,6,Shortcode,15,Sitemap Themes,1,Social Media,155,Technology,3,Templates,1,Tips,2,Tools,1,Traffic Tips,80,Video,19,Web Designing,62,web hosting,18,Webmaster Tools,97,Widgets,199,wordpress,26,
ltr
item
Experience Lab - Online business creation and development guide for bloggers and startups: How To Test Your Robots.txt File In Google Webmaster Tools?
How To Test Your Robots.txt File In Google Webmaster Tools?
http://1.bp.blogspot.com/-NLYnwoT0G04/U8k8vovmF7I/AAAAAAAAGC4/HwJif2hsbcU/s1600/robots.txt+file.jpg
http://1.bp.blogspot.com/-NLYnwoT0G04/U8k8vovmF7I/AAAAAAAAGC4/HwJif2hsbcU/s72-c/robots.txt+file.jpg
Experience Lab - Online business creation and development guide for bloggers and startups
https://www.experiencelab.info/2014/07/how-to-test-your-robotstxt-file-in.html
https://www.experiencelab.info/
https://www.experiencelab.info/
https://www.experiencelab.info/2014/07/how-to-test-your-robotstxt-file-in.html
true
2959477579779989044
UTF-8
Loaded All Posts Not found any posts VIEW ALL Readmore Reply Cancel reply Delete By Home PAGES POSTS View All RECOMMENDED FOR YOU LABEL ARCHIVE SEARCH ALL POSTS Not found any post match with your request Back Home Sunday Monday Tuesday Wednesday Thursday Friday Saturday Sun Mon Tue Wed Thu Fri Sat January February March April May June July August September October November December Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec just now 1 minute ago $$1$$ minutes ago 1 hour ago $$1$$ hours ago Yesterday $$1$$ days ago $$1$$ weeks ago more than 5 weeks ago Followers Follow THIS PREMIUM CONTENT IS LOCKED STEP 1: Share. STEP 2: Click the link you shared to unlock Copy All Code Select All Code All codes were copied to your clipboard Can not copy the codes / texts, please press [CTRL]+[C] (or CMD+C with Mac) to copy