Skip to main content

Get Crawl Status

1 min read
get/crawl/{id}
Request example
Response
get/crawl/{id}

Authorization

Authorizationstringheaderrequired#
Bearer authentication header of the form Bearer <token>, where <token> is your API key.

Path parameters

idstringrequired#
The ID of the crawl job

Responses

application/json
Successful response
statusstring#
The current status of the crawl. Can be scraping, completed, or failed.
totalinteger#
The total number of pages that were attempted to be crawled.
completedinteger#
The number of pages that have been successfully crawled.
creditsUsedinteger#
The number of credits used for the crawl.
expiresAtstring#
The date and time when the crawl will expire.
createdAtstring#
The date and time when the crawl was started.
completedAtstring#
The date and time when the crawl finished. Present only when the crawl is in a terminal state (completed, failed, or cancelled).
durationnumber#
Crawl duration in seconds. For terminal crawls, this is the elapsed time from createdAt to completedAt. For in-progress crawls, it is the elapsed time from createdAt to now.
nextstring#
The URL to retrieve the next 10MB of data. Returned if the crawl is not completed or if the response is larger than 10MB.
dataobject[]#
The data of the crawl.
Show child attributes
markdownstring#
pagesobject[]#
Physical per-page markdown for PDFs. Present only when the request set the pages PDF parser option.
Show child attributes
pageNumberinteger#
1-based physical PDF page number.
markdownstring#
blocksobject[]#
Per-page typed layout blocks for PDFs. Present only when the request set the blocks PDF parser option.
Show child attributes
pageNumberinteger#
1-based physical PDF page number.
widthnumber#
Page render width in px — the anchor for denormalizing bbox coordinates. Null for pages that never rendered.
heightnumber#
Page render height in px. Null for pages that never rendered.
statusstring#
Page-level rollup: ok | partial | failed.
itemsobject[]#
Show child attributes
idstring#
Stable within a response: p<page>.b<index in reading order>.
typestring#
Block type: title, section_header, text, table, formula, figure, caption, page_number, page_header, page_footer. New types may appear over time.
labelstring#
Raw layout-model label, passthrough for forward compatibility.
bboxnumber[]#
[x0, y0, x1, y1] normalized 0-1 relative to the page width/height. Multiply by the page width/height to get pixel coordinates. Null when the page has no known dimensions.

Items: 4–4

contentstring#
Markdown fragment this block contributed to the document markdown.
markdownSpaninteger[]#
[start, end) character offsets into the document markdown covering this block's fragment. Null when a post-processing transform rewrote the fragment.

Items: 2–2

readingOrderinteger#
sourcestring#
Pipeline path that produced the block (for example native_text, layout_ocr, tsr, formula_model, full_page).
confidenceobject#
Show child attributes
layoutnumber#
Layout-model detection score (0-1). Null when the page bypassed layout analysis.
ocrnumber#
Text confidence when the source path provides one; null otherwise.
htmlstring#
HTML version of the content on page if includeHtml is true
rawHtmlstring#
Raw HTML content of the page if includeRawHtml is true
rawBase64string#
The Base64-encoded original HTTP response body if rawBase64 is in formats. A bare Base64 string, not a data URI. The MIME type is in metadata.contentType.
screenshotstring#
Screenshot of the page if includeScreenshot is true
metadataobject#
Show child attributes
titlestring · string[]#
Title extracted from the page, can be a string or array of strings
Show child attributes
descriptionstring · string[]#
Description extracted from the page, can be a string or array of strings
Show child attributes
languagestring · string[]#
Language extracted from the page, can be a string or array of strings
Show child attributes
sourceURLstring#
The original URL that was requested. May differ from the page's final URL if redirects occurred.
urlstring#
The final URL of the page after all redirects have been followed.
keywordsstring · string[]#
Keywords extracted from the page, can be a string or array of strings
Show child attributes
ogLocaleAlternatestring[]#
Alternative locales for the page
<any other metadata> string#
statusCodeinteger#
The status code of the page
numPagesinteger#
For PDF inputs, the number of pages parsed (capped by the parsers maxPages option).
totalPagesinteger#
For PDF inputs, the document's true page count before any maxPages capping. Omitted when it cannot be determined; a totalPages greater than numPages indicates the result was truncated.
errorstring#
The error message of the page
concurrencyLimitedboolean#
Whether this scrape was throttled due to team concurrency limits
concurrencyQueueDurationMsnumber#
Time in milliseconds the request waited in the concurrency queue. Only present when concurrencyLimited is true.