Skip to content

Site Auditor

Crawl a whole site, re-crawl or cancel, estimate cost and read page-level issues.

The Site Auditor crawls an entire site (not a single page) and reports per-page technical health plus site-wide issues: broken internal/external links, duplicate content, indexability, canonical clusters, and more. It is a pass-through bridge over the same engine that powers the in-app Site Auditor — the agent surface reuses the existing crawl pipeline with no scoring changes.

Pricing. A crawl costs 1 credit per 5 pages, based on the project’s max_pages (e.g. a 500-page crawl ≈ 100 credits). The credit model is charge-on-success only:

  • The up-front check on create / crawl is an affordability gate (returns 402 insufficient_credits if the org cannot afford the crawl). It reserves nothing.
  • Credits are charged once when a crawl completes successfully (status completed), based on the pages actually crawled.
  • A crawl that is cancelled or ends in error charges nothing — because nothing was reserved, there is nothing to refund.

This is identical to the in-app Site Auditor’s credit behavior — the bridge reuses the same crawl launcher, which deducts credits on successful completion based on actual pages crawled. (This differs from the Page Auditor’s flat reserve-then-charge model because the two features bill differently: Page Auditor is a flat per-audit charge, Site Auditor is metered per actual pages crawled. The agent surface mirrors each feature’s existing billing, not a single shared model.)

Use GET /site-auditor/credit-estimate?max_pages=N to preview the cost before starting.

Crawls run asynchronously. Create a project with POST /site-auditor/projects (which also starts the first crawl), then poll GET /site-auditor/projects/:project_id until status is completed or error. Large crawls can take several minutes.

Scope Grants
site-auditor:write Create projects + start crawls via POST /site-auditor/projects and POST /site-auditor/projects/:project_id/crawl
site-auditor:read List/read projects and estimate cost via the GET /site-auditor/* endpoints

POST /api/agent/v1/site-auditor/projects

Create a Site Auditor project and immediately start the first full-site crawl. Returns with status: "crawling".

Required scope: site-auditor:write

Field Type Required Description
domain string Yes Domain or root URL to crawl (e.g. "example.com"). A leading http(s):// and trailing slash are stripped. Max 2000 chars.
name string No Human-readable project name. Max 255 chars.
client_id UUID No Optional client to associate the project with. Must belong to your organization.
max_pages number No Max pages to crawl (default 500, 1–1000). Values outside the shared Site Auditor limit are rejected before work starts. Drives the credit cost.
max_depth number No Max crawl depth from the root (default 10, 1–50).
check_external_links boolean No Whether to validate external links (default false).
Terminal window
curl -X POST https://yourapp.com/api/agent/v1/site-auditor/projects \
-H "Authorization: Bearer rk_live_YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{
"domain": "example.com",
"name": "Example full-site audit",
"max_pages": 500
}'
{
"project_id": "8c2a...uuid",
"client_id": null,
"name": "Example full-site audit",
"domain": "example.com",
"status": "crawling",
"max_pages": 500,
"max_depth": 10,
"check_external_links": false,
"total_pages_found": 0,
"pages_crawled": 0,
"pages_analyzed": 0,
"issues_found": 0,
"created_at": "2026-06-14T15:00:00.000Z",
"estimated_credits": 100
}
Status Code When
400 invalid_input Missing/invalid fields or empty domain
402 insufficient_credits Org cannot afford the crawl for max_pages
404 not_found client_id does not exist or belongs to another organization

POST /api/agent/v1/site-auditor/projects/:project_id/crawl

Start a fresh crawl on an existing project (reuses the project’s stored crawl configuration). Prior crawl data is cleared; user-set page triage is carried over by URL.

Required scope: site-auditor:write

Status Code When
402 insufficient_credits Org cannot afford the crawl
404 not_found Project does not exist or belongs to another organization
409 conflict A crawl is already in progress (status is crawling or analyzing)

POST /api/agent/v1/site-auditor/projects/:project_id/cancel

Abort an in-progress crawl and flip the project status to cancelled. Only valid while the crawl is running (status is crawling or analyzing). No credits are charged for a cancelled run. Returns the updated project. If the crawl finished between your status check and the cancel call, the request fails with 409 conflict rather than clobbering the completed project.

Required scope: site-auditor:write

Status Code When
400 invalid_input No crawl in progress to cancel (status is not crawling/analyzing)
404 not_found Project does not exist or belongs to another organization
409 conflict The crawl finished before the cancel could be applied

GET /api/agent/v1/site-auditor/projects/:project_id

Fetch a project without forcing its complete inventory into every call. Poll with ?view=status until terminal. The default view=summary returns project, findings totals, and verification totals. Use view=full for paginated page and issue records.

Required scope: site-auditor:read

{
"project": {
"project_id": "8c2a...uuid",
"client_id": null,
"name": "Example full-site audit",
"domain": "example.com",
"status": "completed",
"max_pages": 500,
"total_pages_found": 312,
"pages_crawled": 312,
"pages_analyzed": 312,
"issues_found": 47,
"broken_internal_links": 3,
"broken_external_links": 1,
"duplicate_content_pairs": 2,
"unique_content_percent": 96,
"started_at": "2026-06-14T15:00:01.000Z",
"completed_at": "2026-06-14T15:04:22.000Z",
"created_at": "2026-06-14T15:00:00.000Z",
"updated_at": "2026-06-14T15:04:22.000Z"
},
"findings_summary": { /* severity and lifecycle totals */ },
"verification_summary": {
"unverified": 12,
"semantics": "unverified means the finding was detected from crawl evidence but has not been rechecked against a later live fetch; it does not mean the initial detector lacked evidence"
},
"view": "summary"
}

For view=full, paginate inventories independently with page_limit, page_offset, issue_limit, and issue_offset. Limits default to 5 and max at 100. Nested keys are canonical snake_case; use naming=legacy only for compatibility with the older mixed-casing payload.

Status Meaning Credits status
pending Created, not yet started Not charged
crawling / analyzing Crawl is in flight Not charged
completed Crawl finished successfully Charged (1 credit / 5 pages crawled)
cancelled Crawl was cancelled No charge
error Crawl failed No charge
Status Code When
400 invalid_input Missing or malformed project_id
404 not_found Project does not exist or belongs to another organization

GET /api/agent/v1/site-auditor/projects

List projects for your organization, most recent first.

Required scope: site-auditor:read

Parameter Type Default Description
client_id UUID — Filter by client
status enum — Filter by pending, crawling, analyzing, completed, error, or cancelled
limit number 50 Max 100
offset number 0 For pagination
{
"projects": [
{
"project_id": "8c2a...uuid",
"domain": "example.com",
"status": "completed",
"pages_crawled": 312,
"issues_found": 47,
"created_at": "2026-06-14T15:00:00.000Z",
"completed_at": "2026-06-14T15:04:22.000Z"
}
],
"pagination": { "limit": 50, "offset": 0, "count": 1, "total": 1, "has_more": false }
}

GET /api/agent/v1/site-auditor/credit-estimate

Estimate the credit cost of a crawl. Read-only, charges nothing.

Required scope: site-auditor:read

Parameter Type Default Description
max_pages number 500 Pages to estimate for (1 credit / 5 pages)
{ "max_pages": 500, "estimated_credits": 100 }