Complete Guide to XML sitemaps

By | July 16, 2012

XML Sitemaps Protocol Summary

The XML sitemaps protocol was created by Google to assist search engines in crawling websites. This file format uses XML for describing its content, a list of URLs in a website, and it usually referred to as XML Sitemaps. The original concept was created by Google, and was therefore in the beginning also simply called Google Sitemaps. If you do not know what the differences are between different sitemap file formats such as XML Sitemaps, HTML sitemaps and RSS feeds you should read the sitemaps explained guide.

Crawl Entire Website First

Before building the XML sitemap, A1 Sitemap Generator needs to scan your website. If you want certain areas restricted, you can either create custom crawler filters or tick options for obeying no-index, no-follow, robots.txt etc. With most websites, you only need to set root path of your website followed by clicking start scan.

XML sitemaps generator tutorial - scan website

Website Analysis and Links Check

After the website crawl has finished you view the website structure and detailed information for the individual pages. These tools can help you solve various website problems related to crawling. When you are done, click the Create sitemap tab.

XML sitemaps generator tutorial - website analysis

Creating the XML Sitemap

If all you need to do is create an XML sitemap, simply pick XML Sitemap in the drop-down and click build sitemap.

If your website is rich on multimedia, you can also create both image and video sitemaps for your website. If you intend to do so, continue by reading more about how to do so.

The XML Sitemaps Protocol has some limits to max number of URLs per XML sitemap file. However, if exceeded, A1 Sitemap Generator will automatically split all URLs out to multiple XML sitemaps and create an XML sitemap index file.

XML sitemaps tutorial - building sitemap

Priority Values in XML Sitemaps Explained

XML sitemaps protocol was originally created by Google and called Google Sitemaps. Later other search engines such as Yahoo, Ask and MSN adopted the Google Sitemaps format, and Google Sitemaps got renamed into Sitemaps Protocol. Common for both is they define a <priority></priority> tag which can be used by the website owner to help search engines determine which pages are most important.

Quick Facts About Priority Values

  • Priority values are considered to be relative within your website.
  • Valid priority values are: 0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0.
  • Usually your most important page URL will have priority value: 1.0.
  • Priority of pages in the XML sitemap should decrease after importance.

Priority Values and Search Engine Optimization

Setting priority values can not be considered search engine optimization. You can not set all pages to maximum priority and assume search engines will agree 🙂 That said, here are some thoughts about how priorities in XML sitemaps are used:

  • Pages with high priority values are likely to get indexed faster and crawled more often.
  • Help search engines to decide which url to show if multiple pages from same website rank for search query.
  • While highly speculative, one can imagine that if a website and XML sitemap is highly trusted, you might be able boost rankings in search engine result pages, just like good internal linking within a website can.

Always ensure you have backlinks, i.e. incoming links. This will improve chances that search engines index and crawl all urls listed in your XML sitemap.

How to Set Priorities in XML Sitemaps

Never use the same priority value for all pages, better to leave priority values out of your XML sitemaps! 🙂 You can set priority value for all pages manually, but this quickly gets tedious if you have a large website. Alternatively, if you use A1 Sitemap Generator, you can have all priority values calculated. It does so by counting and weighing importance of all internal links within a website and converting this into a priority value for each URL in the XML sitemap.

Final steps for your XML sitemap

At this point, you can choose to upload your XML Sitemap. You can also ping and update it to various search engines such as Google. If you haven’t already, it is also a good idea to create a robots.txt file with a reference to your generated XML sitemap.

XML site maps tutorial - view site map

FTP Upload Generated Sitemaps

After you have created your XML sitemap, image sitemap, video sitemap etc. you will need to upload the generated sitemaps to your site. While you can use your normal favourite FTP client software for uploading files, you can also use the integrated FTP upload functionality in A1 Sitemap Generator. When done with configuration, simply upload all your site map files.

How to Setup FTP Upload

If you are not used to FTP upload files, you should check the instructions and information your webhost provides.

XML sitemaps FTP upload

When configuring the FTP directory upload path, you should remember that many web servers have the files that can be accessed from the outside (i.e. those that can be seen through browser) placed in a sub directory, e.g. named htdocs, public, html or similar. You should refer to your webhost for this information, but a way to sometimes identify the correct directory it is to try find where you have an index file for your domain.

Notice: If you are uploading sitemaps for addon domains, your upload directory path will usually look like this /html/ or /public/

Concerning host, just remember that it is not not necessary to write www in front of your FTP domain name.

  • Press the Upload now button to start uploading the site map file you just built.
  • Press the Upload all button to have all created site map files uploaded.

You can view progress while upload happens.

Search Engines, Robots Text File and XML Sitemaps

Since the birth of internet search engines, the robots.txt file has been how webmasters could let search engines like Google know what content should get crawled and indexed. However, as part of Google Sitemaps, later named XML Sitemaps Protocol, the usage was expanded with Sitemaps Autodiscovery. It is now possible for webmaster to direct search engines to the website XML sitemap. The moment a search engine has found your website and the robots.txt file, it will also know where to find your XML sitemap.

Submit Your XML Sitemaps

In the beginning XML sitemaps submission required you had created and verified a Google Webmaster Tools account. You also had to submit your sitemap files manually. Now, instead of submitting XML sitemaps to all search engines individually, you can be done with them all in seconds.

XML sitemaps and robots.txt

After crawling your website and creating the XML sitemap, the sitemap generator program can create the robots.txt file with correct and complete path to your XML sitemap.

Generated robots.txt file

As see from above, to add XML sitemaps autodiscovery to a robots.txt file, add the fully qualified XML sitemap file path like this: Sitemap:

Example robots.txt for XML Sitemaps Autodiscovery

User-agent: *

If you have created a sitemap index file, you can also reference that:
User-agent: *
Sitemap: Sitemap:

Manual XML Sitemap Submission

There are some good reasons to submit your XML sitemaps manually the first time, e.g. to get acquainted with the different search engine and webmaster tools available:

Advanced Manage and Submit Sitemaps

In the beginning no search engines supported cross submit multiple websites in one XML sitemap file. However, now most include support for new ways of managing sitemaps across multiple sites. Requirement is you need to verify ownership of all websites:

  • Sitemaps protocol: Cross sitemaps submit and manage using robots.txt.
  • Google: More website verification methods than sitemaps protocol defines.

Ping Search Engines Using XML Sitemaps Protocol

First appearing in Google sitemaps and later in XML Sitemaps Protocol, sitemap ping is a way to inform search engines that you have updated your XML sitemap files. Pinging is not required for search engines to discover changes in XML sitemaps, but it will speed the process.

Some sitemap generator tools like ours have builtin support for pinging.

XML sitemaps ping

To add more search engines like Google and Bing, click the Add common pings button. If you have scanned a website and generated an XML sitemap, A1 Sitemap Generator will automatically generate the correct ping URLs for you. It is possible to add more sitemap ping URLs if wanted. When you are done, click the Ping now button.

Video Demonstrating How to Create XML Sitemaps

We have uploaded a video to YouTube that shows how you can make xml sitemaps:

Besides explaining the benefits of sitemaps, it also shows how to scan websites, create xml sitemaps, FTP upload sitemaps and sitemap ping search engines!

Note: You can also check this shorter building sitemaps video.

General Information About XML Sitemaps Protocol

Here is a collection of articles containing general information about XML sitemaps:
General summary of what sitemaps are.
The official specification/protocol for XML sitemaps.
All sitemaps formats explained including code examples. Explains the differences between RSS feeds, ROR files and XML sitemaps.
Help page by Google about video sitemaps – an XML sitemaps extension.
Help page by Google about image sitemaps – an XML sitemaps extension.

A1 Sitemap Generator Tutorials

Tutorials specific on how to use our sitemapping software to create various kinds of XML sitemaps:
Tutorial on how to create standard XML sitemaps.
Tutorial on how to create video sitemaps.
Tutorial on how to create image sitemaps.

Get Help With XML Sitemaps and Related Webmaster Problems

Webmaster forums where you can ask for help incase you run into problems:

A1 Sitemap Generator
Forum at WebHelpForums for helpping webmasters specificly on how to use A1 Sitemap Generator.

Website Crawling and Indexing
Forum at WebHelpForums for general webmaster help on website crawling, indexing and sitemaps.

Leave a Reply