The bot in your logs is called Metrimato-Scanner
It runs the technical site scan that Metrimato customers can turn on for their own site. If you see it in your logs, this page tells you what it does and how to control it.
Metrimato-Scanner/1.0 (+https://metrimato.com/scanner)The address in the user agent points here, so the bot can always be identified from a log line alone. Metrimato-Scanner never fetches a site that has not installed Metrimato's tracking snippet.
What it does, and what it does not
One scan fetches at most 500 pages and waits 0.3 seconds between requests, or longer if robots.txt asks for it. It reads HTML and nothing else.
- It reads the sitemap first
- The address list comes from robots.txt and the sitemap. Only when there is no sitemap does it follow links from the front page.
- It obeys robots.txt
- Disallowed paths are not fetched, and a Crawl-delay is honoured. A bot that ignores robots.txt is malware, whoever ordered it.
- It fetches HTML only
- Images, video, stylesheets and scripts are never requested, and a response larger than two megabytes is cut off.
- It stores no page content
- The markup is parsed in memory into lengths, counts and addresses, and then discarded. Duplicate pages are matched by a hash, not by keeping the text.
- It submits no forms and follows no links off the site
- Only GET requests, only to the site's own domains. It carries no cookies and no credentials.
- It runs weekly at most
- A site is scanned once a week, and a scan started by hand is limited on top of that.
How to turn it away
The bot obeys robots.txt like any other crawler, so a group naming Metrimato-Scanner is all it takes. The change applies on the next scan.
If the bot is fetching a site you own and you did not ask for it, tell us and we will stop it and check how the domain was verified.
User-agent: Metrimato-Scanner
Disallow: /Who can have a site scanned
Not anyone. A scan is traffic to somebody's server, so the right to order one has to be earned, not typed into a form.
- The domain must belong to a Metrimato site
- Addresses are read from the site's own domains, never from a form field. The tracking snippet on that domain is what proves ownership.
- Internal addresses are refused
- Every hostname is resolved and checked before the connection is made, and again at every redirect. A domain pointing at a private or link-local address is not fetched.
- The pace is capped, not negotiable
- One request at a time per site, with a delay in between. The scan is meant to be the bot nobody notices.