Categories
josephscott

Yahoo, Google and Microsoft agree on Sitemaps Spec

Yahoo, Google and Microsoft (Windows Live / MSN Search) have all agreed to the use sitemaps protocol. Sitemaps.org has information on how sitemaps work. This is being announced all over the place:

Just in case you aren’t familiar with sitemaps, they are XML files that describe the different URLs available on your site. The details are available at http://www.sitemaps.org/protocol.html.

An important aspect of this is that you can ping search engines to let them know that your sitemap has been updated. From the FAQ on sitemaps.org:

Q: What do I do after I create my Sitemap?

Once you have created the Sitemap file and placed it on your webserver, you submit it to search engines that support the Sitemaps protocol. The search engines can then retrieve your Sitemap and make the URLs available to their crawlers. Refer to each search engine’s documentation for more information about submitting to them. You can also submit your Sitemap using an HTTP request (replace <searchengine_URL> with the URL provided by the search engine):
Issue your request to the following URL:

<searchengine_URL>/ping?sitemap=sitemap_url

For example, if your Sitemap is located at http://www.example.com/sitemap.gz, your URL will become:

<searchengine_URL>/ping?sitemap=http://www.example.com/sitemap.gz

URL encode everything after the /ping?sitemap=:

<searchengine_URL>/ping?sitemap=http%3A%2F%2Fwww.example.com%2Fsitemap.gz

You can issue the HTTP request using wget, curl, or another mechanism of your choosing. A successful request will return an HTTP 200 response code; if you receive a different response, you should resubmit your request. The HTTP 200 response code only indicates that the search engine has received your Sitemap, not that the Sitemap itself or the URLs contained in it were valid. An easy way to do this is to set up an automated job to generate and submit Sitemaps on a regular basis.

Note: If you are providing a Sitemap index file, you only need to issue one HTTP request that includes the location of the Sitemap index file; you do not need to issue individual requests for each Sitemap listed in the index.

So I tried to ping for updates to this blog at:

Unfortunately none of those URLs seem to be working. Both Yahoo! and Google return error pages. The MSN Search URL returned an error indicating it didn’t like the format of my sitemap, with Google has been consuming without a problem for months.

It has been more than two years since I first wrote about the idea of pinging search engines for site updates. The implementation that I had in mind was different, but the concept is the same. Nice to see this finally get some traction at the big three search engines.

4 replies on “Yahoo, Google and Microsoft agree on Sitemaps Spec”

Leave a Reply

Your email address will not be published. Required fields are marked *