SEOfinity site scan crawler

If you found SEOfinity-SiteScan/1.0 (+https://tpg-seo-app.fly.dev/crawler) in your server logs, this page explains what it is and how to control it.

Who runs it

SEOfinity is a Shopify app made by Digital Experience Leaders GmbH. Its site scan checks a store's own public pages for SEO problems: status codes, redirects, titles, meta descriptions, canonical URLs, noindex tags, headings and broken links.

It only runs when a merchant who installed SEOfinity starts a scan in the app. It only visits that store's own domain. It never crawls other websites.

How it finds pages

  • It reads robots.txt first, then sitemap.xml.
  • It follows internal links from the home page, nearest pages first. Depth and the number of pages are limited.
  • It checks links on the store's own domain to find broken ones. It sends a light HEAD request, and a normal GET only if the server does not accept HEAD.

How it behaves

  • At most 1 request per second per site, and only one scan per store at a time.
  • It follows Crawl-delay in your robots.txt for SEOfinity-SiteScan or *, up to 10 seconds.
  • It follows Disallow rules for SEOfinity-SiteScan, or your User-agent: * rules if there is no group for it.
  • A robots.txt group for SEOfinity also applies to it.
  • If your site answers 429 (too many requests) or 503 (unavailable), it waits as long as your Retry-After header asks (up to 90 seconds), then continues at the same pace. After 8 failed requests in a row, the scan stops.
  • It never opens cart, checkout, account or login pages. It never submits forms and never sends cookies.
  • It never asks for a storefront password. Stores behind a password page are skipped.

What it stores

Only the facts it extracts from each page, such as the status code, title, meta description, headings, canonical URL and robots tags. It does not keep copies of your pages. Results are deleted after 30 days, and right away if the store removes the app and Shopify asks us to erase its data.

How to block or slow it

Block it completely:

User-agent: SEOfinity-SiteScan
Disallow: /

Slow it down to one request every 5 seconds:

User-agent: SEOfinity-SiteScan
Crawl-delay: 5

You can also block the user agent SEOfinity-SiteScan in your firewall or CDN. If your site refuses the request (403, 429 or a bot challenge), the scan stops and the merchant sees that it was blocked; if requests are silently dropped, the merchant sees that your site could not be reached.

Where requests come from

Requests come from our hosting provider, Fly.io. The IP addresses are not fixed, so please identify the crawler by its user agent and its signature.

How to verify it

Every request is signed with Web Bot Auth (HTTP Message Signatures, Ed25519). Each request carries Signature-Agent, Signature-Input and Signature headers, so your CDN or firewall can confirm it really comes from SEOfinity and not from someone copying our user agent. Our public keys are published at https://tpg-seo-app.fly.dev/.well-known/http-message-signatures-directory.

Contact

Questions, complaints, or a request to stop scans of your domain: apps@the-platform-group.com. Please include your domain and an example log line.