Site Auditor
Crawl a whole site, re-crawl or cancel, estimate cost and read page-level issues.
Site Auditor
Section titled “Site Auditor”The Site Auditor crawls an entire site (not a single page) and reports per-page technical health plus site-wide issues: broken internal/external links, duplicate content, indexability, canonical clusters, and more. It is a pass-through bridge over the same engine that powers the in-app Site Auditor — the agent surface reuses the existing crawl pipeline with no scoring changes.
Pricing. A crawl costs 1 credit per 5 pages, based on the project’s max_pages (e.g. a 500-page crawl ≈ 100 credits). The credit model is charge-on-success only:
- The up-front check on create / crawl is an affordability gate (returns
402 insufficient_creditsif the org cannot afford the crawl). It reserves nothing. - Credits are charged once when a crawl completes successfully (status
completed), based on the pages actually crawled. - A crawl that is cancelled or ends in error charges nothing — because nothing was reserved, there is nothing to refund.
This is identical to the in-app Site Auditor’s credit behavior — the bridge reuses the same crawl launcher, which deducts credits on successful completion based on actual pages crawled. (This differs from the Page Auditor’s flat reserve-then-charge model because the two features bill differently: Page Auditor is a flat per-audit charge, Site Auditor is metered per actual pages crawled. The agent surface mirrors each feature’s existing billing, not a single shared model.)
Use GET /site-auditor/credit-estimate?max_pages=N to preview the cost before starting.
Crawls run asynchronously. Create a project with POST /site-auditor/projects (which also starts the first crawl), then poll GET /site-auditor/projects/:project_id until status is completed or error. Large crawls can take several minutes.
Scopes
Section titled “Scopes”| Scope | Grants |
|---|---|
site-auditor:write |
Create projects + start crawls via POST /site-auditor/projects and POST /site-auditor/projects/:project_id/crawl |
site-auditor:read |
List/read projects and estimate cost via the GET /site-auditor/* endpoints |
Site Auditor — Create project & crawl
Section titled “Site Auditor — Create project & crawl”POST /api/agent/v1/site-auditor/projects
Create a Site Auditor project and immediately start the first full-site crawl. Returns with status: "crawling".
Required scope: site-auditor:write
Request body
Section titled “Request body”| Field | Type | Required | Description |
|---|---|---|---|
domain |
string | Yes | Domain or root URL to crawl (e.g. "example.com"). A leading http(s):// and trailing slash are stripped. Max 2000 chars. |
name |
string | No | Human-readable project name. Max 255 chars. |
client_id |
UUID | No | Optional client to associate the project with. Must belong to your organization. |
max_pages |
number | No | Max pages to crawl (default 500, 1–1000). Values outside the shared Site Auditor limit are rejected before work starts. Drives the credit cost. |
max_depth |
number | No | Max crawl depth from the root (default 10, 1–50). |
check_external_links |
boolean | No | Whether to validate external links (default false). |
Example request
Section titled “Example request”curl -X POST https://yourapp.com/api/agent/v1/site-auditor/projects \ -H "Authorization: Bearer rk_live_YOUR_KEY" \ -H "Content-Type: application/json" \ -d '{ "domain": "example.com", "name": "Example full-site audit", "max_pages": 500 }'Response (201 Created)
Section titled “Response (201 Created)”{ "project_id": "8c2a...uuid", "client_id": null, "name": "Example full-site audit", "domain": "example.com", "status": "crawling", "max_pages": 500, "max_depth": 10, "check_external_links": false, "total_pages_found": 0, "pages_crawled": 0, "pages_analyzed": 0, "issues_found": 0, "created_at": "2026-06-14T15:00:00.000Z", "estimated_credits": 100}Errors
Section titled “Errors”| Status | Code | When |
|---|---|---|
| 400 | invalid_input |
Missing/invalid fields or empty domain |
| 402 | insufficient_credits |
Org cannot afford the crawl for max_pages |
| 404 | not_found |
client_id does not exist or belongs to another organization |
Site Auditor — Re-crawl project
Section titled “Site Auditor — Re-crawl project”POST /api/agent/v1/site-auditor/projects/:project_id/crawl
Start a fresh crawl on an existing project (reuses the project’s stored crawl configuration). Prior crawl data is cleared; user-set page triage is carried over by URL.
Required scope: site-auditor:write
Errors
Section titled “Errors”| Status | Code | When |
|---|---|---|
| 402 | insufficient_credits |
Org cannot afford the crawl |
| 404 | not_found |
Project does not exist or belongs to another organization |
| 409 | conflict |
A crawl is already in progress (status is crawling or analyzing) |
Site Auditor — Cancel crawl
Section titled “Site Auditor — Cancel crawl”POST /api/agent/v1/site-auditor/projects/:project_id/cancel
Abort an in-progress crawl and flip the project status to cancelled. Only valid while the crawl is running (status is crawling or analyzing). No credits are charged for a cancelled run. Returns the updated project. If the crawl finished between your status check and the cancel call, the request fails with 409 conflict rather than clobbering the completed project.
Required scope: site-auditor:write
Errors
Section titled “Errors”| Status | Code | When |
|---|---|---|
| 400 | invalid_input |
No crawl in progress to cancel (status is not crawling/analyzing) |
| 404 | not_found |
Project does not exist or belongs to another organization |
| 409 | conflict |
The crawl finished before the cancel could be applied |
Site Auditor — Get project
Section titled “Site Auditor — Get project”GET /api/agent/v1/site-auditor/projects/:project_id
Fetch a project without forcing its complete inventory into every call. Poll with ?view=status until terminal. The default view=summary returns project, findings totals, and verification totals. Use view=full for paginated page and issue records.
Required scope: site-auditor:read
Response
Section titled “Response”{ "project": { "project_id": "8c2a...uuid", "client_id": null, "name": "Example full-site audit", "domain": "example.com", "status": "completed", "max_pages": 500, "total_pages_found": 312, "pages_crawled": 312, "pages_analyzed": 312, "issues_found": 47, "broken_internal_links": 3, "broken_external_links": 1, "duplicate_content_pairs": 2, "unique_content_percent": 96, "started_at": "2026-06-14T15:00:01.000Z", "completed_at": "2026-06-14T15:04:22.000Z", "created_at": "2026-06-14T15:00:00.000Z", "updated_at": "2026-06-14T15:04:22.000Z" }, "findings_summary": { /* severity and lifecycle totals */ }, "verification_summary": { "unverified": 12, "semantics": "unverified means the finding was detected from crawl evidence but has not been rechecked against a later live fetch; it does not mean the initial detector lacked evidence" }, "view": "summary"}For view=full, paginate inventories independently with page_limit, page_offset, issue_limit, and issue_offset. Limits default to 5 and max at 100. Nested keys are canonical snake_case; use naming=legacy only for compatibility with the older mixed-casing payload.
Status values
Section titled “Status values”| Status | Meaning | Credits status |
|---|---|---|
pending |
Created, not yet started | Not charged |
crawling / analyzing |
Crawl is in flight | Not charged |
completed |
Crawl finished successfully | Charged (1 credit / 5 pages crawled) |
cancelled |
Crawl was cancelled | No charge |
error |
Crawl failed | No charge |
Errors
Section titled “Errors”| Status | Code | When |
|---|---|---|
| 400 | invalid_input |
Missing or malformed project_id |
| 404 | not_found |
Project does not exist or belongs to another organization |
Site Auditor — List projects
Section titled “Site Auditor — List projects”GET /api/agent/v1/site-auditor/projects
List projects for your organization, most recent first.
Required scope: site-auditor:read
Query parameters
Section titled “Query parameters”| Parameter | Type | Default | Description |
|---|---|---|---|
client_id |
UUID | — | Filter by client |
status |
enum | — | Filter by pending, crawling, analyzing, completed, error, or cancelled |
limit |
number | 50 | Max 100 |
offset |
number | 0 | For pagination |
Response
Section titled “Response”{ "projects": [ { "project_id": "8c2a...uuid", "domain": "example.com", "status": "completed", "pages_crawled": 312, "issues_found": 47, "created_at": "2026-06-14T15:00:00.000Z", "completed_at": "2026-06-14T15:04:22.000Z" } ], "pagination": { "limit": 50, "offset": 0, "count": 1, "total": 1, "has_more": false }}Site Auditor — Credit estimate
Section titled “Site Auditor — Credit estimate”GET /api/agent/v1/site-auditor/credit-estimate
Estimate the credit cost of a crawl. Read-only, charges nothing.
Required scope: site-auditor:read
Query parameters
Section titled “Query parameters”| Parameter | Type | Default | Description |
|---|---|---|---|
max_pages |
number | 500 | Pages to estimate for (1 credit / 5 pages) |
Response
Section titled “Response”{ "max_pages": 500, "estimated_credits": 100 }