MkDocs sitemap.xml file generation
Here at OctoPerf we love static website generators. Even if our load testing tool is made completely dynamic by AngularJS, our website and blog are generated using Jekyll.
The OctoPerf documentation is created with MkDocs. This wonderful tool allowed us to concentrate on the documentation content as the markdown syntax is really easy to comprehend.
Moreover, the responsive themes let us integrate the documentation directly into our performance testing tool:
When we first wrote it, MkDocs version was 0.12.0. It did not support sitemap.xml file generation. This guide helps migrating to the latest version (0.14.0).
Sitemaps¶
The format and Sitemap protocol are quite old as they were initiated by Google in 2005. This is a solution to provide the search engine crawlers (Google, Bing, etc.) a plan of your site. A Sitemap is a comprehensive list of your URLs in XML format. These robots can then identify and fetch all available pages, according to information provided in the file.
sitemap.xml¶
The concept of this format is extremely simple: you create an XML file that lists the pages on your site, as well as some information about them (update frequency, crawl priority, etc.):
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9 http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd" xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://api.octoperf.com/doc/</loc>
<lastmod>2015-06-15</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>https://api.octoperf.com/doc/account/</loc>
<lastmod>2015-06-15</lastmod>
<changefreq>daily</changefreq>
</url>
</urlset>
You point out its presence to the search engine through an administration interface. Then the search engine robots read then it and use the proposed URLs to exhaustively index your site in depth.
- Using a Sitemap does not guarantee that the search engines will index all listed pages. They remain master of the way they index websites (It still eases this process).
- A Sitemap file is not a guarantee that your site will be better positioned. This tool is only an index tool and not a ranking tool.
- Finally, a Sitemap does not replace the crawling of your website by search engine robots, done by following the web pages links. Both methods are quite complementary indeed.
Webmaster tools¶
Your sitemap.xml file can be submitted to Google using Google Webmaster Tools. It can be submitted to Bing via their Bing Webmaster Tools.
MkDocs¶
MkDocs is a static site generator. Its goal is to build projects documentation. Documentation source files are written in Markdown, and configured with a single YAML configuration file. It is quite easy to install. MkDocs is currently still in development, and evolves quickly. When we started to write OctoPerf's documentation, MkDoc was in version 0.12.0 and did not support sitemap.xml file generation. The version 0.13.0 added this feature.
But it also came with changes in the configuration file (mkdocs.yml) format.
What is your version?¶
First you may display your current MkDocs version:
mkdocs --version
If it is above 0.13.0, you may already have a sitemap.xml generated and available at /sitemap.xml.
Otherwise, you should upgrade MkDocs version.
Version upgrade¶
To upgrade MkDocs, simply run the following command in a shell:
pip install mkdocs --upgrade
pip stands for Python Index Package and is a package manager for Python. It is mandatory to install MkDocs so you probably already have it installed on your machine.
If you get an error like OSError: [Errno 13] Permission denied: '/usr/local/bin/mkdocs' and you are running on Ubuntu, you may prefix the command with sudo.
Once upgraded, you can type mkdocs --version to ensure that you use the latest version.
Configuration migration¶
The mkdocs.yml multilevel pages format changed for the better. It was redundant and it is now more succinct.
E.g. our load testing tool documentation configuration file included:
pages:
- ['index.md', 'Getting Started','Introduction']
- ['account/index.md', 'Getting Started','Account']
- ['project/index.md', 'Getting Started','Project']
- ['design/record.md', 'Design','Recording']
- ['design/index.md', 'Design', 'Virtual Users']
- ['design/logicActions.md', 'Design', 'Logical Actions']
- ['design/httpActions.md', 'Design', 'HTTP Actions']
- ['design/postProcessors.md', 'Design', 'Post Processors']
- ['variable/index.md', 'Design','Variables']
- ['server/index.md', 'Design','Servers']
- ['correlation/index.md', 'Design','Correlation Rules']
- ['scenario/index.md', 'Execution', 'Scenarios']
- ['analysis/index.md', 'Analysis','Bench Reports']
- ['glossary.md', 'Glossary']
- ['privacy.md', 'Privacy Policy']
- ['tos.md', 'Terms Of Service']
We had to repeat each section ('Getting Started', 'Design', etc.) multiple times.
Now you define such hierarchy in a clearer way:
pages:
- Getting Started:
- 'Introduction': 'index.md'
- 'Account': 'account/index.md'
- 'Project': 'project/index.md'
- Design:
- 'Recording': 'design/record.md'
- 'Virtual Users': 'design/index.md'
- 'Logical Actions': 'design/logicActions.md'
- 'HTTP Actions': 'design/httpActions.md'
- 'Post Processors': 'design/postProcessors.md'
- 'Variables': 'variable/index.md'
- 'Servers': 'server/index.md'
- 'Correlation Rules': 'correlation/index.md'
- Execution:
- 'Scenarios': 'scenario/index.md'
- Analysis:
- 'Bench Reports': 'analysis/index.md'
- Glossary: 'glossary.md'
- Privacy Policy: 'privacy.md'
- Terms Of Service: 'tos.md'
Tips¶
One last thing with withMkDocs, to avoid broken links you can add the line strict: true in your mkdocs.yml file. MkDocs will check for internal links (links between your documentation pages) and won't run or build if there is a broken one.
Unfortunately it does not check for images URLs.
Another great tool to check for broken links (both internal and external) is the Chrome extension Check My Links.