--> How To Ensure Optimal Crawling Of Your Website? | Experience Lab - Online business creation and development guide for bloggers and startups

How To Ensure Optimal Crawling Of Your Website?

It is one thing to create a website and put up some content, but quite another to get it noticed by Google . Often, the more content you hav...

Ensure Optimal Crawling Of Your Site

It is one thing to create a website and put up some content, but quite another to get it noticed by Google. Often, the more content you have, the higher your number of crawled and indexed pages in search engines. But that is not always the case. If the crawling process is not optimal, search engines might miss out on some of your content. Today, we have some guidelines for you from Google, explaining which fields in sitemaps are important, when to use XML Sitemaps and RSS/Atom feeds, and how to optimize then for Google.

XML Sitemaps or RSS feeds?

The first question that you could ask is, which to use; XML Sitemaps or RSS/Atom feeds? Should you use RSS/Atom feeds alongside XML Sitemaps? XML Sitemaps are an indispensable part of your site, and they describe a whole set of URLs within it. On the other hand, RSS/Atom feeds describe the most recent changes.


The problem with XML Sitemaps is, they contain complete site information, and hence are much larger than RSS feeds. Ergo they're also downloaded less frequently. So it's not a question of why, and rather why not use both these formats? Each has its own use, and complements the other.

XML sitemaps give Google information about all the pages on your site, while RSS/Atom feeds let Google know what has been most recently updated on your site. Google also adds that “submitting sitemaps or feeds does not guarantee the indexing of those URLs.”

Sitemap and RSS feeds best practices

In order to optimize the crawl process, you should use XML Sitemaps along with RSS/Atom feeds. Here are some best practices for them from Google.

  • The two most important pieces of information for Google are the URL itself and its last modification time.

  • Only include URLs that can be fetched by Googlebot (ie, don’t include URLs blocked by robots.txt).

  • Only include canonical URLs.

  • Specify a last modification time for each URL in an XML sitemap and RSS/Atom feed

  • For a single XML sitemap, update it at least once a day and ping Google each time.

  • For a set of XML sitemaps, maximize the number of URLs in each XML sitemap. The limit is 50,000 URLs or a maximum size of 10MB uncompressed. Ping Google when each XML sitemap is updated.

  • When a new page is added or an existing page meaningfully changed, add the URL and the modification time to the RSS/Atom feed.

  • In order for Google to not miss updates, the RSS/Atom feed should have all updates in it since at least the last time Google downloaded it. The best way to achieve this is by using PubSubHubbub.

Good luck getting your webpages crawled quickly :)



Affiliate Marketing,12,Announcement,34,Bing,9,Bitcoin,38,blog,7,Blogger Resources,42,Blogger Templates,4,blogger tricks,156,Blogging ethics,70,Blogging tips,198,Bugs and Errors,34,Business,7,Copyright Violation,9,CSS and HTMLTricks,95,Designs,8,drop down menu,7,eBook,12,Email Marketing,7,Events,30,Facebook,30,Facebook tricks,49,Google,157,Google AdSense,42,Google Analytics,7,Google Plus,51,Google Plus Tricks,38,Guest Posts,112,home,2,How To,77,Internet,1,JSON Feeds,25,Kitchen Recipes,2,Label Based Sitemap Themes,1,Make Money Online,108,Marketing,16,MBT Blogger Templates,7,Menus,1,News,146,Pages,1,Posts,10,presentations,15,Responsive,10,Reviews,7,SEO,307,Settings,6,Shortcode,15,Sitemap Themes,1,Social Media,155,Technology,6,Templates,1,Tips,2,Tools,1,Traffic Tips,80,Video,19,Web Designing,62,web hosting,18,Webmaster Tools,97,Widgets,199,wordpress,26,
Experience Lab - Online business creation and development guide for bloggers and startups: How To Ensure Optimal Crawling Of Your Website?
How To Ensure Optimal Crawling Of Your Website?
Experience Lab - Online business creation and development guide for bloggers and startups
Loaded All Posts Not found any posts VIEW ALL Readmore Reply Cancel reply Delete By Home PAGES POSTS View All RECOMMENDED FOR YOU LABEL ARCHIVE SEARCH ALL POSTS Not found any post match with your request Back Home Sunday Monday Tuesday Wednesday Thursday Friday Saturday Sun Mon Tue Wed Thu Fri Sat January February March April May June July August September October November December Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec just now 1 minute ago $$1$$ minutes ago 1 hour ago $$1$$ hours ago Yesterday $$1$$ days ago $$1$$ weeks ago more than 5 weeks ago Followers Follow THIS PREMIUM CONTENT IS LOCKED STEP 1: Share. STEP 2: Click the link you shared to unlock Copy All Code Select All Code All codes were copied to your clipboard Can not copy the codes / texts, please press [CTRL]+[C] (or CMD+C with Mac) to copy