XML sitemaps: How to help Google discover your pages | James Clark | Wix SEO Hub (2024)

Discovery is the very first step in SEO—if a search engine can’t discover your content, it will never crawl and index it, which means searchers won’t be able to access it. An XML sitemap is an optional, but powerful, tool to support the discovery process (and by extension, your technical SEO efforts).

But, what should your sitemap contain? How do you create it? And how do you tell search engines about it? Let’s go on a discovery process of our own—drawing from the official sitemaps protocol and Google’s documentation—to understand how to use sitemaps for better SEO.

Table of contents:

What is an XML sitemap?

Although sitemaps can come in different formats, the most common is XML (extensible markup language)—a language that uses tags to “mark up” and structure information (a bit like HTML). One of the benefits of XML is that both people and computer programs can easily read it.

XML sitemaps follow the sitemaps protocol, which Google, Microsoft, and Yahoo all support. This protocol defines what a sitemap can contain, how to format it, and even how to submit it to search engines.

How XML sitemaps help your SEO

The main way a search engine discovers web pages on a site is by following backlinks. These could be links from your own site (internal links), or links on another site (external links).

Some pages can be difficult for search engines to find. An “orphan page” is one that doesn’t have any inbound links pointing to it, meaning that search engines will never discover it by following links. Your website can even have small groups of orphan pages that only link to each other.

Another challenge that search engines face is knowing when pages were updated. Although search engines will periodically revisit crawled pages to see whether the content has changed, this isn’t particularly efficient either for search engines or website owners.

That isn’t to say that a sitemap can replace an effective internal linking policy. Remember, links don’t just help discovery, they also tell search engines about the relationship between pages—something that sitemaps can’t do.

Types of sitemap

You might think that “sitemap” and “XML sitemap” are synonymous, but the sitemaps protocoldefines three valid sitemap formats:

Google and other major search engines can work with any of these formats. In most cases, you’ll want an XML sitemap—but if your platform or CMS doesn’t provide you with this, consider the other two formats:

News sites often create feeds for individual categories (or “channels”)—here’s the start of The Guardian’s RSS feed for its culture category:

One big drawback of using a feed as a sitemap is that it usually only contains the most recent content. Nonetheless, it can still help search engines discover that content (and, through internal links, other content on your site).

The rest of this article focuses on XML sitemaps as these are the most common, and versatile, type of sitemap.

What an XML sitemap should (and shouldn’t) contain

Your sitemap should contain the URLs of all the pages you want search engines to crawl (and subsequently show in search results). Each page has its own pair of opening and closing <url> tags, containing a <loc> element that specifies the page’s location—like this:

<url> <loc>https://example.com/myurl1/</loc></url><url> <loc>https://example.com/myurl2/</loc></url>

There are plenty of pages you shouldn’t include in your sitemap, for example:

In other words, pages you wouldn’t want Google to attempt to crawl and index.

The <loc> element is mandatory, but there are other optional elements you can include with each URL to add more guidance for crawlers:

Google saysit ignores <priority> and <changefreq> values, while Bing saysit “largely disregards” them. The <lastmod> tag, then, is the most useful way of indicating to search engines that a previously discovered page needs to be recrawled.

Sitemap extensions

The sitemap protocol only specifies how to include URLs in a sitemap. However, one of the most powerful features of the protocol is that you can use it to include other types of content (the “X” in “XML” stands for “eXtensible”).

There are Google-supported extensions for the following content types:

You can create separate sitemaps for these content types or include them in your existing sitemap.

The extensions introduce many new mandatory and optional elements. For example, video content requires the <video:thumbnail_loc> tag pointing to the location of the video thumbnail.

Static and dynamic sitemaps

Dynamic sitemaps are generated each time they are requested from the server, so they will always be up to date. In other words, if you create a new page on your website, then load your dynamic XML sitemap in a browser tab, it should list your new page. Likewise, if you change an existing page, the sitemap should update the <lastmod> value for that page. (If your sitemap is supposed to be dynamic but isn’t updating, you might have a caching issue.)

Static sitemaps, on the other hand, aren’t generated on the fly and don’t automatically update. As the name suggests, they are just static files.

Size limits for XML sitemaps and sitemap indexes

The sitemaps protocol specifies size limitsfor XML sitemaps “to ensure that your web server does not get bogged down serving very large files” (but also to make the process more efficient for search engines).

Your XML sitemap should:

The size limit refers to the size of the uncompressedfile, so compressing the file won’t help you get around this requirement. Instead you should follow the advice given in the protocol:

A sitemap indexis an XML file that lists multiple XML sitemaps. You might have one sitemap for your posts, one for your pages, and another for your categories—all listed in your index.

Sitemap index files have size limits, too. Similar to individual sitemaps, they should:

The sitemaps protocol also has restrictions around content. Some characters must be “escaped”—an ampersand (“&”) is written as “&amp;”, for example.

If you’re using your web platform or CMS to generate your sitemap, it will likely follow the protocol, so you only need to worry about these restrictions if you’re creating your sitemap manually (which is rare).

How to generate an XML sitemap

How you generate your XML sitemap will depend on the CMS or platform you use for your website. Let’s look at how this works for Wix websites as well as how to do this with Screaming Frog, a popular SEO tool.

Sitemaps on Wix

Wix websites come with sitemapsautomatically. I say “sitemaps” because the platform provides different sitemaps for different types of pages. The sitemap index lives at https://yoursite.com/sitemap.xml, but this could link to sitemaps for events, forum posts, or more, depending on the functionality your site uses.

For example, this London barbershop has a sitemap specifically for the products it sells on its Wix website:

Also, when you complete your Wix SEO Setup Checklist, Wix automatically submits your XML sitemap to Google for you. You’ll need a Premium plan and your own domain to take advantage of this.

Generate sitemaps with Screaming Frog

Maybe your platform or CMS doesn’t generate an XML sitemap for you. Maybe you aren’t even using a platform or CMS and instead hand-coded your site from scratch! In these situations, you’ll have to get a little creative.

If your site is small, you could use the text sitemap format we looked at earlier, create the file manually, and host it on your server.

But, this isn’t very practical if you have more than a couple dozen pages. Instead, you could use an SEO tool called Screaming Frogto crawl your site and create an XML sitemap for you.

This is a powerful option as it will automatically exclude pages that are blocked by robots.txt, set as “noindex,” or have a canonical tag pointing to a different URL—in other words, all the pages you ordinarily wouldn’t want Google to attempt to crawl and index.

The free version of Screaming Frog will crawl up to 500 URLs, so if your site is bigger than this then you’ll need to pay for a license.

Just as with a text sitemap, the next step is to host your new XML sitemap on your server (preferably in the root directory) then submit it to Google. If your site changes often, you could even look at scheduling an automated crawl.

One downside of the Screaming Frog approach is that it gives you a static sitemap. If you create a new page or update an existing one, the sitemap won’t automatically change to reflect that.

Submitting your XML sitemap to Google (and Bing)

Once you’ve generated your sitemap, the next step is to inform the major search engines so they can use it. There are two ways to do this.

The first is to specify the path to your sitemap or sitemap index in your robots.txt file, like this example from the Manchester United website:

This small change will enable Google and other search engines to find your sitemap the next time they crawl your robots.txt file. The downside here is that you don’t get any feedback: you won’t know when those search engines last read your sitemap, how many pages they discovered, and so on.

For that, you’ll need special tools provided by the search engines themselves. These tools let you both submit your sitemap and see how it is being read:

You don’t have to choose one approach or the other. It’s definitely worth both specifying the path to your sitemap in your robots.txt file andsubmitting your sitemap to search engines individually.

Validating your XML sitemap

It may seem strange to talk about validating your XML sitemap after submitting it, but that’s because submitting your sitemap is actually the best way to validate it. When you submit your sitemap to Google using Search Console (or Bing using Bing Webmaster Tools), the tool will tell you whether your sitemap is valid.

In Search Console, you get a green “success” message if everything is OK:

But if you get a red message instead, something has gone wrong. Just click on the error to find out more:

Once you’ve fixed any errors, resubmit it to prompt Google to fetch it again.

There are also free third-party tools you can useto validate your XML sitemap, either by pasting in a link or uploading an XML file.

However, even if your XML is valid, there might be another reason why Google can’t fetch your sitemap: Perhaps your robots.txt file is blocking Googlebot from accessing it. Unlike Google Search Console, a third-party validation tool wouldn’t pick up on this kind of issue.

HTML vs. XML sitemaps

We’ve seen that XML sitemaps are intended for search engines, but there’s another type of sitemap aimed at human users: the HTML sitemap. This is a directory of the main pages or sections on a site, and can help users quickly understand the site’s structure and navigate around.

Your HTML sitemap may sit on a dedicated page, or perhaps in the footer—as with this example from Apple:

So called “mega menus” in the site header are, in effect, another kind of HTML sitemap:

HTML sitemaps do serve an SEO purpose, too: They are a collection of internal links, which Googlebot will happily use to discover new pages and understand the value of those pages. So in that sense they complement the work of your XML sitemaps.

That doesn’t mean you should use just an HTML sitemap. Thinking specifically about search engine discovery, they have some major drawbacks compared to XML sitemaps:

For those reasons, you should focus on providing an XML sitemap that is useful to search engines and an HTML sitemap that is useful to your users. If an HTML sitemap wouldn’t be useful to your users, simply don’t include one.

Take control over your discoverability with XML sitemaps

Now, you have an in-depth understanding of sitemaps. Put your knowledge into practice by working through the following questions:

The exercise will help you come up with a plan for your site, revealing any actions you need to take to improve your sitemap coverage and boost page discoverability. And, if you’re in doubt about anything, refer to the sitemaps protocoland Google’s documentation!

James Clark is a web analyst from London, with a background in the publishing sector. When he isn't helping businesses with their analytics, he's usually writing how-to guides over on his website Technically Product.Twitter| Linkedin