A sitemap is a file that lists the URLs of your website. It serves as a guide for Google or Bing’s crawlers to navigate your pages. Without this file, some parts of your site may remain invisible in search results, even if their content is relevant. Understanding how it works allows you to fix indexing issues that many website owners overlook.
What Google no longer reads in your XML sitemap
Have you ever seen tags like priority or changefreq in an XML sitemap file? Google ignores them. These two fields, although present in the protocol specification, have no effect on crawl frequency or the ranking of your pages.
On the other hand, the lastmod tag remains the only freshness signal used by Google. It indicates the actual last modification date of a page. If you update it with each publication or correction, the crawlers know which URLs to revisit as a priority.
A common pitfall: some tools regenerate the sitemap by updating the lastmod date of all pages, even those that haven’t changed. This behavior muddles the signal. Google eventually stops trusting this tag on your site. Ensure that your sitemap generator only updates lastmod on pages that have actually been modified. To explore this topic with concrete examples, the site pepseo.fr in detail describes best configuration practices.
Sitemap and crawl budget: why large sites can’t do without it
Let’s take a simple example. Your site contains several thousand pages. Googlebot has a limited amount of time to crawl each site. This time, called crawl budget, varies based on your domain’s authority and your server’s speed.
Without a sitemap, the bot discovers your pages by following internal links. If a page is poorly linked to the rest of the site (no link from the menu, no contextual link), it may never be crawled. The XML sitemap compensates for this problem by explicitly declaring each URL.
For sites with fewer than a hundred pages, with a correct internal linking structure, the sitemap remains useful but less critical. Large or complex sites depend directly on it for their deep pages to be indexed.
Orphan pages and invisible content
An orphan page is a page that is not linked by any other page on the site. It exists on the server, but no navigation path leads to it. Googlebot will only find it if it is included in the sitemap.
Do you regularly publish content? Each new article needs to be discovered. The sitemap speeds up this process by signaling new URLs as soon as they are published.
Submitting a sitemap to Google: the two methods that still work
Until 2023, it was possible to “ping” Google to signal a new sitemap. This method no longer works. Google deprecated its ping endpoint in June 2023, and URLs like /ping?sitemap= now return a 404 error.
Two methods remain valid:
- Submit the sitemap via Google Search Console, in the Sitemaps report. You receive feedback on the number of discovered URLs and any formatting errors.
- Declare the sitemap in the robots.txt file with the Sitemap: directive followed by the URL of the file. This approach ensures automatic discovery, even without manual intervention in Search Console.
The combination of both offers the best result. Search Console provides you with diagnostics, while robots.txt ensures permanent discovery by all compliant bots.
HTML sitemap and XML sitemap: two files, two audiences
The XML sitemap is aimed at search engines. It contains technical data: URL, modification date, language. Its format is structured to be read by machines.
The HTML sitemap, on the other hand, is a standard page on your site, visible to visitors. It displays a list of your sections and pages in the form of clickable links. Its role is to improve navigation for users looking for a specific page.
The XML sitemap remains the priority for SEO. The HTML is a useful complement for user experience, but it does not carry the same weight with Google. If you have to choose where to start, the XML file comes first.
What the XML sitemap should contain
The rule is simple: include only the pages you want to appear in search results. Pages with a noindex tag, login pages, and internal search result pages do not belong in the sitemap.
- Main pages (home, categories, services, blog articles) should be included.
- Only canonical URLs: if multiple URLs lead to the same content, only the canonical version appears in the file.
- Images and videos can have specific sitemaps, useful for appearing in Google Images or Google Videos.
Common errors that reduce the effectiveness of a sitemap
A poorly configured sitemap can do more harm than good. Declaring URLs that return a 404 error degrades Googlebot’s trust in your file. If the bot finds that half of your URLs are broken, it slows down the crawling of the rest.
Another common error: a sitemap that exceeds the size limit. The protocol allows a maximum of 50,000 URLs per file, with a size not exceeding 50 MB. Beyond that, it must be divided into multiple files and grouped via a sitemap index.
Finally, do not confuse sitemap with indexing guarantee. Submitting a URL does not force Google to index it. The search engine decides on its own whether the page deserves to be included in its index, based on its quality and relevance. The sitemap facilitates discovery, not ranking.
A well-maintained sitemap file, updated with each actual content modification and submitted through the right channels, remains one of the simplest technical levers to implement to improve a website’s visibility. The work does not stop at its creation: regularly check its content in Search Console to identify erroneous URLs and adjust the file accordingly.
