Crawlabout

Website Exploration Tool

See your website
the way the web sees it.

Enter an address – Crawlabout walks through every page and shows you what visitors and search engines find: broken links, redirects, SEO issues from 48 checks, the external services the pages load, the structure of the site and how its pages link to each other. It searches the content of all pages and can take the whole website offline.

Download Source on GitHub

Free and open source (MIT) · Linux, macOS (Apple Silicon) and Windows · runs locally, no account, no cloud

Crawlabout overview: key figures, status of all URLs, SEO health, response times, clicks from the start page and external services

Everything you need to know about your website

Crawlabout behaves like a browser: it sends the same headers as Firefox, keeps cookies and follows the browser's referrer rules, so servers answer it the way they answer your visitors.

One input

Only the start URL is needed. Crawlabout finds all internal pages by itself and checks each URL exactly once.

All kinds of links

Internal pages, links to other websites (including where redirects end up) and files: images, PDFs, CSS, JavaScript, fonts, videos.

Overview with charts

Status of all URLs, response times, clicks from the start page, status codes, size by MIME type and external services. Click any figure to see the matching URLs.

External services

Which hosts your pages load fonts, statistics, videos, maps or scripts from – Google Fonts, Tag Manager, YouTube … – and on how many pages. Handy for the privacy policy.

Content search

Search the text or the HTML source of all pages as you type: an old phone number, the former company name, a forgotten tracking code. Regular expressions included.

Sortable, filterable list

URL, status, MIME type, title, kind, clicks, size, time and SEO issues – filter by every value, open details with redirect chain, SEO data and the pages that link to a URL.

Link graph

The whole site as a network you can zoom, move and click: which pages link to each other, where clusters and dead ends are, how many clicks a page is from the start.

Sitemap and robots.txt

The URLs of your sitemap are checked too – pages that are listed there but linked nowhere show up as orphans. URLs blocked by robots.txt are marked.

Compare runs

Compare the results with an earlier report: new and removed URLs, changed status codes, new and fixed SEO issues.

Export and re-check

HTML and PDF reports with every chart of the overview and the details of each URL; JSON and CSV exports with all data – of the whole site or just the filtered list. The download starts at once, a bar shows the progress. Import a report later, search it and re-check only the URLs that failed – or everything.

Password protected sites

HTTP Basic authentication is supported. Credentials are only sent to the checked site and never stored in reports.

Large websites

Made for 100,000 URLs and more: the list comes in pages, filters and search answer at once, the link graph groups itself by directory, and nothing slows the browser down while the check runs.

48 SEO checks

Every internal page is analysed while it is checked – no extra requests, no account, nothing leaves your machine. The overview shows the SEO health of all pages and the length of titles and descriptions; the results list shows and filters the issues per URL, and the details explain each one with what exactly was found.

  • Indexability: noindex, missing or broken canonicals, robots.txt, sitemap entries with errors or noindex, orphan pages.
  • Content: missing, duplicate, too long or too short titles and meta descriptions, H1 and heading structure, duplicate content.
  • Links: redirect chains and loops, internal links to redirects, broken #anchors, mixed content, deep and rarely linked pages.
  • Markup: images without alt text or too large, hreflang, JSON-LD, viewport, lang, social media previews.
  • Site: security headers, certificate expiry, URL hygiene.
SEO health of the pages and length of titles and meta descriptions

Site structure and sitemap.xml

The Sitemap tab shows every internal page as a tree by its URL path – with title, last modification date and the number of pages below each folder. A click on a page shows its details.

  • sitemap.xml for Google: ready for the Search Console, with lastmod from the server's Last-Modified header.
  • Only what belongs there: pages with noindex, another canonical URL or blocked by robots.txt are marked and left out.
  • As HTML: the tree as nested list, e.g. as a starting point for a sitemap page.
Site structure as tree with export of sitemap.xml and HTML

The link graph

The Graph tab draws every page as a dot and every link as a line – and builds itself up while the check runs. Zoom, move it around, pin dots where you want them, or open it over the whole window.

  • Hide the menu: links that are on almost every page (navigation, footer) can be left out – what remains is how your content links to each other.
  • Focus on a page: a click shows only the page, the pages it links to and the shortest click path from the start page – why is this page five clicks deep?
  • By directory: colour the dots by /blog/, /shop/ … or group each directory into one dot, with lines as thick as the links between them – a site with more than 2,500 URLs is grouped from the start.
  • Find problems: colour by status or clicks from the start page, search for a page, show files and external links too; a click opens the details.
Link graph of a website, coloured by directory

Save websites offline

Crawlabout downloads all pages of a website with their images, stylesheets, scripts, fonts and documents and rewrites the links, so the copy can be browsed from disk – or downloaded as ZIP.

  • Download list: add several websites, each with its own settings, and download them at the same time. Edit entries later, export the list to a JSON file and import it again.
  • Flexible target folders: placeholders like <host> and <date>, default ~/Downloads/crawlabout/<host>.
  • Project sites: a site in a sub-directory (like user.github.io/project) ends up directly in the target folder.
  • Error pages included: pages answering 404 or 500 are saved anyway when they have content; the download list marks them and lists every URL with an error.
  • Update instead of starting over: a second run only downloads what changed and removes what is gone; a stopped download continues where it ended.
  • You decide what goes in: files on other websites, the pages of the sitemap, robots.txt and rules like -/private/* or -*.zip; requests per second and bandwidth can be limited.
  • External resources: images, CSS and scripts from CDNs are included.
Download list with several offline copies

Install

Crawlabout is a single, statically linked binary without dependencies. Start it without arguments and it opens its interface in your browser. The interface is available in English and German and follows your system language.

Debian, Ubuntu (x86_64)

curl -LO https://github.com/dlehmann/crawlabout/releases/latest/download/crawlabout-amd64.deb
sudo apt install ./crawlabout-amd64.deb

The package installs the program, a launcher in the application menu and the icon. For ARM boards (Raspberry Pi 4/5 and others) use crawlabout-arm64.deb.

Arch Linux (x86_64)

curl -LO https://github.com/dlehmann/crawlabout/releases/latest/download/crawlabout-x86_64.pkg.tar.zst
sudo pacman -U crawlabout-x86_64.pkg.tar.zst

For ARM (Arch Linux ARM) use crawlabout-aarch64.pkg.tar.zst.

Other Linux distributions

curl -LO https://github.com/dlehmann/crawlabout/releases/latest/download/crawlabout-linux-amd64.tar.gz
tar -xzf crawlabout-linux-amd64.tar.gz
sudo install -m755 crawlabout-*/crawlabout /usr/local/bin/

For ARM use crawlabout-linux-arm64.tar.gz.

macOS (Apple Silicon)

curl -LO https://github.com/dlehmann/crawlabout/releases/latest/download/crawlabout-darwin-arm64.tar.gz
tar -xzf crawlabout-darwin-arm64.tar.gz
xattr -d com.apple.quarantine crawlabout-*/crawlabout   # the binary is not signed
sudo install -m755 crawlabout-*/crawlabout /usr/local/bin/

Windows (x86_64)

Download crawlabout-windows-amd64.exe and start it – a single file, no installer. The console window that opens with it stops Crawlabout when you close it. The .exe is not signed: if Windows shows “Windows protected your PC”, choose “More info” → “Run anyway”.

From source

sudo pacman -S go git            # Arch Linux
sudo apt install golang-go git   # Debian 13 or newer (needs Go ≥ 1.24)

git clone https://github.com/dlehmann/crawlabout.git
cd crawlabout
make install PREFIX=$HOME/.local   # binary, desktop launcher and icon

Without the browser

crawlabout -report report.json https://example.com      # check, write a JSON report
crawlabout -report report.html https://example.com      # … or a readable report (.html, .pdf, .csv)
crawlabout -recheck report.json                          # re-check the failed URLs
crawlabout -report new.json -compare report.json https://example.com  # what changed?
crawlabout -search "old-company.example" https://example.com    # pages that still mention it
crawlabout -mirror ./example-offline https://example.com # offline copy

All releases with checksums: github.com/dlehmann/crawlabout/releases