Sitemap and indexing

A page can only appear in Google once Google has found it and added it to its index. Usually that happens by itself via links, but not always quickly or completely. New pages, deeply tucked-away articles or hundreds of product pages can go unnoticed for a long time. A sitemap helps.

What is a sitemap?

A sitemap is a list of all the pages on your site that you want in Google, in a format search engines can read directly. For each page it records when it was last changed. That tells Google which pages exist and which ones need another look.

How it works in Obelisk

Obelisk creates and maintains the sitemap fully automatically. You'll find it at /sitemap.xml on your own domain, and the robots.txt file already points to it, so search engines pick it up by themselves. A few more things Obelisk handles for you:

• A sitemap per language. If you have a multilingual site, each language gets its own sitemap.
• Dynamic pages are included. Add an article, product or vacancy as a record and its page appears in the sitemap straight away.
• Always an up-to-date modification date, so Google can see what's new.
• No noindex pages. Obelisk leaves those out; read more in canonical and noindex.

Submitting to Google

Being found automatically works, but it's faster and gives more insight through Google Search Console. Submit your sitemap there by entering the address /sitemap.xml. Search Console then shows for each page whether it's indexed, and if not, why not. How to set up Search Console is covered in measuring with Search Console.

A concrete example

An estate agent puts new properties online as records every week. Without a sitemap, Google would only find those pages once something linked to them, sometimes weeks later. Because Obelisk adds every new property to the sitemap immediately, and the sitemap is submitted in Search Console, new listings are often in Google within a few days. For stock that sells fast, that makes a real difference.

When does a page not get indexed?

• The page is set to noindex, deliberately or by accident.
• The content is too thin or very similar to another page.
• No other page links to it; work on your internal links.
• The page is brand new and Google hasn't visited yet.

Common mistakes

Thinking a sitemap guarantees a place in Google (it only helps with discovery; quality decides whether you rank), never looking at Search Console, and making important pages reachable only through a search box. A sitemap is a safety net, not a replacement for a good structure with links.