Skip to content
/ Docs

SitepagerBot and User Agent

Sitepager is a website testing tool for marketing teams and agencies. It checks visual changes, broken links, SEO, performance, and accessibility in one run.

When Sitepager tests a website, it loads the pages and files needed for the scan. By default, its requests include the Sitepager identity below, so you can recognize Sitepager traffic in your logs and allow it through your website security.

Sitepager user agent

Section titled “Sitepager user agent”

By default, Sitepager uses the user agent of the browser running the test and appends its Sitepager identity:

<browser user agent> SitepagerBot/1.0 (<workspace-id>; sitepager)

The browser portion varies depending on the browser and device used for the test. <workspace-id> is the Sitepager workspace that made the request.

SitepagerBot is the part of the user agent that identifies Sitepager. When setting up a rule on your site, match:

SitepagerBot

Match this part rather than the complete user agent, because the browser and workspace portions can vary.

You do not need to configure a custom user agent for Sitepager to identify itself.

Custom user agents

Section titled “Custom user agents”

If you configure a custom bot identifier, Sitepager keeps the same identity format and replaces the final sitepager label:

<browser user agent> SitepagerBot/1.0 (<workspace-id>; <bot-identifier>)

SitepagerBot continues to match these requests.

If you provide a complete custom user agent instead, Sitepager sends that value exactly as supplied and does not add SitepagerBot. In that case, set up your rule to match the value you supplied.

Custom user-agent settings change the requests Sitepager makes to your pages. Sitepager still checks robots.txt as SitepagerBot, so custom user agents do not bypass your SitepagerBot robots.txt rules.

See Custom User Agent for configuration options.

Control SitepagerBot with robots.txt

Section titled “Control SitepagerBot with robots.txt”

Sitepager looks only at robots.txt rules written for User-agent: SitepagerBot. Rules for User-agent: * or other bots do not apply to Sitepager.

This means you can keep search engines blocked from a staging site and still test that site with Sitepager.

Allow Sitepager

Section titled “Allow Sitepager”

You do not need to add anything. Sitepager scans normally unless you write rules for SitepagerBot. To allow it explicitly:

User-agent: SitepagerBot
Allow: /

Block Sitepager completely

Section titled “Block Sitepager completely”
User-agent: SitepagerBot
Disallow: /

Block a folder or path

Section titled “Block a folder or path”
User-agent: SitepagerBot
Disallow: /private/

Allow part of a blocked path

Section titled “Allow part of a blocked path”
User-agent: SitepagerBot
Disallow: /private/
Allow: /private/test/

How Sitepager reads these rules

Section titled “How Sitepager reads these rules”
  • Only rules for User-agent: SitepagerBot apply to Sitepager.
  • Allow and Disallow are both supported.
  • * matches any characters.
  • Use $ to match the end of the path and query string.
  • Sitepager matches the path and query string, and paths are case-sensitive. For example, /Private and /private are different.
  • If several rules match, the most specific rule wins.
  • If equally specific Allow and Disallow rules match, Allow wins.
  • A custom user agent does not bypass these rules. Sitepager always checks robots.txt as SitepagerBot.
  • If robots.txt is missing or cannot be reached, Sitepager continues the scan.

What happens in a scan

Section titled “What happens in a scan”

If robots.txt blocks Sitepager from the entire scan, Sitepager stops before scanning pages and tells you that the site blocks Sitepager.

If robots.txt blocks only some URLs, Sitepager skips those URLs and scans the rest.

Web Bot Auth

Section titled “Web Bot Auth”

Sitepager publishes its Web Bot Auth public key directory at:

https://api-home.sitepager.io/.well-known/http-message-signatures-directory

Web Bot Auth allows websites to verify that a signed request really came from Sitepager. The directory above publishes the keys a website needs to check that.

Allow Sitepager through your firewall

Section titled “Allow Sitepager through your firewall”

If your site is blocking Sitepager, allow requests whose user agent contains:

SitepagerBot

If you use a complete custom user agent, allow the exact user-agent value you configured instead.

See Allow Sitepager Through Your Firewall for steps for Cloudflare, Webflow, Framer, Vercel, Netlify, and custom servers.

Support

Section titled “Support”

If you need help identifying Sitepager traffic or allowing it through your site, contact support@sitepager.io.

Sitepager is operated by Shaztra Inc.