Crawl

1 min read

post/crawl

Request example

curl --request POST \
  --url https://api.firecrawl.dev/v2/crawl \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '{
  "url": "<string>",
  "prompt": "<string>",
  "excludePaths": [
    "<string>"
  ],
  "includePaths": [
    "<string>"
  ],
  "maxDiscoveryDepth": 1,
  "sitemap": "include",
  "ignoreQueryParameters": false,
  "regexOnFullURL": false,
  "limit": 10000,
  "crawlEntireDomain": false,
  "allowExternalLinks": false,
  "allowSubdomains": false,
  "ignoreRobotsTxt": false,
  "robotsUserAgent": "<string>",
  "delay": 1,
  "maxConcurrency": 1,
  "webhook": {
    "url": "<string>",
    "headers": {},
    "metadata": {},
    "events": [
      "completed"
    ]
  },
  "scrapeOptions": {
    "formats": [
      "markdown"
    ],
    "onlyMainContent": true,
    "onlyCleanContent": false,
    "includeTags": [
      "<string>"
    ],
    "excludeTags": [
      "<string>"
    ],
    "maxAge": 172800000,
    "minAge": 1,
    "headers": {},
    "waitFor": 0,
    "mobile": false,
    "skipTlsVerification": true,
    "timeout": 60000,
    "parsers": [
      "pdf"
    ],
    "actions": [
      {
        "type": "wait",
        "milliseconds": 1
      }
    ],
    "location": {
      "country": "US",
      "languages": [
        "en-US"
      ]
    },
    "removeBase64Images": true,
    "blockAds": true,
    "proxy": "auto",
    "storeInCache": true,
    "lockdown": false,
    "redactPII": false,
    "profile": {
      "name": "<string>",
      "saveChanges": true
    },
    "threatProtection": {
      "mode": "off",
      "riskScoreThreshold": 75,
      "blacklist": [
        "<string>"
      ],
      "whitelist": [
        "<string>"
      ],
      "blockedTlds": [
        "<string>"
      ],
      "failurePolicy": "open"
    },
    "auditMetadata": {
      "username": "<string>"
    }
  },
  "zeroDataRetention": false
}'

Response

{
  "success": true,
  "id": "<string>",
  "url": "<string>"
}

post/crawl

Are you an AI agent that needs a Firecrawl API key? See firecrawl.dev/agent-onboarding/SKILL.md for automated onboarding instructions.

Authorization

Authorizationstringheaderrequired#

Bearer authentication header of the form Bearer <token>, where <token> is your API key.

Body

application/json

urlstringrequired#

The base URL to start crawling from

promptstring#

A prompt to use to generate the crawler options (all the parameters below) from natural language. Explicitly set parameters will override the generated equivalents.

excludePathsstring[]#

URL pathname regex patterns that exclude matching URLs from the crawl. For example, if you set "excludePaths": ["blog/.*"] for the base URL firecrawl.dev, any results matching that pattern will be excluded, such as https://www.firecrawl.dev/blog/firecrawl-launch-week-1-recap.

includePathsstring[]#

URL pathname regex patterns that include matching URLs in the crawl. Only the paths that match the specified patterns will be included in the response. Note: the starting URL is also checked against these patterns — if it does not match, the crawl may return 0 pages. For example, if you set "includePaths": ["blog/.*"] for the base URL firecrawl.dev/blog, only pages under /blog/ will be included in the results, such as https://www.firecrawl.dev/blog/firecrawl-launch-week-1-recap.

maxDiscoveryDepthinteger#

Maximum depth to crawl based on discovery order. The root site and sitemapped pages has a discovery depth of 0. For example, if you set it to 1, and you set sitemap: 'skip', you will only crawl the entered URL and all URLs that are linked on that page.

sitemapenum<string>#

Sitemap mode when crawling. If you set it to 'skip', the crawler will ignore the website sitemap and only crawl the entered URL and discover pages from there onwards. If you set it to 'only', the crawler will only crawl URLs from the sitemap (plus the start URL) and will not discover links from HTML.

Default: "include"

Options:skipincludeonly

ignoreQueryParametersboolean#

Do not re-scrape the same path with different (or none) query parameters

Default: false

regexOnFullURLboolean#

When true, includePaths and excludePaths regex patterns are matched against the full URL (including query parameters) instead of just the URL pathname. Useful when you need to filter URLs based on query strings.

Default: false

limitinteger#

Maximum number of pages to crawl. Default limit is 10000.

Default: 10000

crawlEntireDomainboolean#

Allows the crawler to follow internal links to sibling or parent URLs, not just child paths.

false: Only crawls deeper (child) URLs.
→ e.g. /features/feature-1 → /features/feature-1/tips ✅
→ Won't follow /pricing or / ❌

true: Crawls any internal links, including siblings and parents.
→ e.g. /features/feature-1 → /pricing, /, etc. ✅

Use true for broader internal coverage beyond nested paths.

Default: false

allowExternalLinksboolean#

Allows the crawler to follow links to external websites. External links are followed one hop (the links found on those external pages are not crawled). Links pointing to an external site's homepage (a root URL with no path) are skipped and reported in Get Crawl Errors with the code EXTERNAL_LINK; redirects to an external homepage are skipped for the same reason.

Default: false

allowSubdomainsboolean#

Allows the crawler to follow links to subdomains of the main domain.

Default: false

ignoreRobotsTxtboolean#

Ignore the website's robots.txt rules. Enterprise only — contact support@firecrawl.com to enable.

Default: false

robotsUserAgentstring#

Custom User-Agent string for robots.txt evaluation. When set, robots.txt is fetched with this User-Agent and allow/disallow rules are matched against it instead of the default. Enterprise only — contact support@firecrawl.com to enable.

delaynumber#

Delay in seconds between scrapes. This helps respect website rate limits. Setting this forces concurrency to 1.

maxConcurrencyinteger#

Maximum number of concurrent scrapes. This parameter allows you to set a concurrency limit for this crawl. If not specified, the crawl adheres to your team's concurrency limit.

webhookobject#

A webhook specification object.

Show child attributes

urlstringrequired#

The URL to send the webhook to. This will trigger for crawl started (crawl.started), every page crawled (crawl.page) and when the crawl is completed (crawl.completed or crawl.failed). The response will be the same as the /scrape endpoint.

headersobject#

Headers to send to the webhook URL.

Show child attributes

{key}string#

metadataobject#

Custom metadata that will be included in all webhook payloads for this crawl

eventsenum<string>[]#

Type of events that should be sent to the webhook URL. (default: all)

scrapeOptionsobject#

Show child attributes

formatsobject[]#

Output formats to include in the response. You can specify one or more formats, either as strings (e.g., 'markdown') or as objects with additional options (e.g., { type: 'json', schema: {...} }). Some formats require specific options to be set. Example: ['markdown', { type: 'json', schema: {...} }].

Default: ["markdown"]

Show child attributes

One of

typeenum<string>required#

Options:markdown

typeenum<string>required#

Options:summary

typeenum<string>required#

Options:html

typeenum<string>required#

Options:rawHtml

typeenum<string>required#

Options:rawBase64

typeenum<string>required#

Options:links

typeenum<string>required#

Options:images

typeenum<string>required#

Options:screenshot

fullPageboolean#

Whether to capture a full-page screenshot (ignores viewport.height) or limit to the current viewport.

Default: false

qualityinteger#

The quality of the screenshot, from 1 to 100. 100 is the highest quality.

viewportobject#

Show child attributes

widthintegerrequired#

The width of the viewport in pixels

heightintegerrequired#

The height of the viewport in pixels

typeenum<string>required#

Options:json

schemaobject#

The schema to use for the JSON output. Must conform to JSON Schema.

promptstring#

The prompt to use for the JSON output

checkPromptInjectionboolean#

When enabled, scans the scraped page content for prompt injection attempts before running the extraction. If an injection is detected, the request fails with a 403 and error code SCRAPE_PROMPT_INJECTION_DETECTED. Adds 4 credits when the check runs. Defaults to false.

Default: false

typeenum<string>required#

Options:changeTracking

modesenum<string>[]#

The mode to use for change tracking. 'git-diff' provides a detailed diff, and 'json' compares extracted JSON data.

schemaobject#

Schema for JSON extraction when using 'json' mode. Defines the structure of data to extract and compare. Must conform to JSON Schema.

promptstring#

Prompt to use for change tracking when using 'json' mode. If not provided, the default prompt will be used.

tagstring#

Tag to use for change tracking. Tags can separate change tracking history into separate "branches", where change tracking with a specific tagwill only compare to scrapes made in the same tag. If not provided, the default tag (null) will be used.

Default: null

typeenum<string>required#

Options:branding

typeenum<string>required#

Options:product

typeenum<string>required#

Options:menu

typeenum<string>required#

Options:audio

typeenum<string>required#

Options:video

typeenum<string>required#

Options:question

questionstringrequired#

The question to answer about the page. Maximum 10,000 characters.

Length: max 10000

typeenum<string>required#

Options:highlights

querystringrequired#

The text-selection query to run against the page. Maximum 10,000 characters.

Length: max 10000

onlyMainContentboolean#

Only return the main content of the page excluding headers, navs, footers, etc. This is a deterministic HTML-level filter applied before markdown is generated; no LLM is involved.

Default: true

onlyCleanContentboolean#

Beta. Run an additional LLM-based pass over the generated markdown to remove residual boilerplate that onlyMainContent can miss (cookie banners, ad blocks, social share widgets, breadcrumbs, newsletter signups, comment sections, related-article lists). Headings, lists, tables, code blocks, image references, and inline links are preserved. Can be combined with onlyMainContent (the most common setup) or used on its own. Skipped with a warning when the markdown exceeds the cleaning model's output token limit (the original markdown is preserved). Not supported on zero-data-retention requests.

Default: false

includeTagsstring[]#

Tags to include in the output.

excludeTagsstring[]#

Tags to exclude from the output.

maxAgeinteger#

Returns a cached version of the page if it is younger than this age in milliseconds. If a cached version of the page is older than this value, the page will be scraped. If you do not need extremely fresh data, enabling this can speed up your scrapes by 500%. Defaults to 2 days.

Default: 172800000

minAgeinteger#

When set, the request only checks the cache and never triggers a fresh scrape. The value is in milliseconds and specifies the minimum age the cached data must be. If matching cached data exists, it is returned instantly. If no cached data is found, a 404 with error code SCRAPE_NO_CACHED_DATA is returned. Set to 1 to accept any cached data regardless of age.

headersobject#

Headers to send with the request. Can be used to send cookies, user-agent, etc.

waitForinteger#

Specify a delay in milliseconds before fetching the content, allowing the page sufficient time to load. This waiting time is in addition to Firecrawl's smart wait feature.

Default: 0

mobileboolean#

Set to true if you want to emulate scraping from a mobile device. Useful for testing responsive pages and taking mobile screenshots.

Default: false

skipTlsVerificationboolean#

Skip TLS certificate verification when making requests.

Default: true

timeoutinteger#

Timeout in milliseconds for the request. Minimum is 1000 (1 second). Default is 60000 (60 seconds). Maximum is 300000 (300 seconds).

Default: 60000

Range: 1000–300000

parsersobject[]#

Controls how files are processed during scraping. When "pdf" is included (default), the PDF content is extracted and converted to markdown format, with billing based on the number of pages (1 credit per page). When an empty array is passed, the PDF file is returned in base64 encoding with a flat rate of 1 credit for the entire PDF.

Default: ["pdf"]

Show child attributes

One of

typeenum<string>required#

Options:pdf

modeenum<string>#

PDF parsing mode. "fast": text-based extraction only (embedded text, fastest). "auto" (default): attempts fast extraction first, falls back to OCR if needed. "ocr": forces OCR parsing on every page.

Default: "auto"

Options:fastautoocr

maxPagesinteger#

Maximum number of pages to parse from the PDF. Must be a positive integer up to 10000.

Range: 1–10000

pagesboolean#

Include physical per-page markdown alongside the document markdown. Populates the pages field on the document as an array of { pageNumber, markdown }. No additional cost.

Default: false

blocksboolean#

Include per-page typed layout blocks alongside the document markdown. Populates the blocks field on the document: typed blocks (title, section_header, text, table, formula, figure, caption, ...) with normalized bounding boxes, reading order, character-span links into the markdown, and per-block confidence. No additional cost.

Default: false

pageMarkersboolean#

Annotate page breaks in the document markdown: pages are joined with \n\n---\n\n\n\n, where N is the 1-based physical page of the content that follows. Markers appear between pages only (no leading marker for page 1), and numbering may skip pages merged across a page break — use pages: true when every physical page is needed. No new response field; no additional cost.

Default: false

actionsobject[]#

Actions to perform on the page before grabbing the content

Show child attributes

One of

typeenum<string>required#

Take a screenshot. The links will be in the response's actions.screenshots array.

Options:screenshot

fullPageboolean#

Whether to capture a full-page screenshot (ignores viewport.height) or limit to the current viewport.

Default: false

qualityinteger#

The quality of the screenshot, from 1 to 100. 100 is the highest quality.

viewportobject#

Show child attributes

widthintegerrequired#

The width of the viewport in pixels

heightintegerrequired#

The height of the viewport in pixels

typeenum<string>required#

Click on an element

Options:click

selectorstringrequired#

Query selector to find the element by

allboolean#

Clicks all elements matched by the selector, not just the first one. Does not throw an error if no elements match the selector.

Default: false

typeenum<string>required#

Write text into an input field, text area, or contenteditable element. Note: You must first focus the element using a 'click' action before writing. The text will be typed character by character to simulate keyboard input.

Options:write

textstringrequired#

Text to type

typeenum<string>required#

Press a key on the page

Options:press

keystringrequired#

Key to press

typeenum<string>required#

Scroll the page or a specific element

Options:scroll

directionenum<string>#

Direction to scroll

Default: "down"

Options:updown

selectorstring#

Query selector for the element to scroll

typeenum<string>required#

Scrape the current page content, returns the url and the html.

Options:scrape

typeenum<string>required#

Execute JavaScript code on the page

Options:executeJavascript

scriptstringrequired#

JavaScript code to execute

typeenum<string>required#

Generate a PDF of the current page. The PDF will be returned in the actions.pdfs array of the response.

Options:pdf

formatenum<string>#

The page size of the resulting PDF

Default: "Letter"

Options:A0A1A2A3A4A5A6LetterLegalTabloidLedger

landscapeboolean#

Whether to generate the PDF in landscape orientation

Default: false

scalenumber#

The scale multiplier of the resulting PDF

Default: 1

locationobject#

Location settings for the request. When specified, this will use an appropriate proxy if available and emulate the corresponding language and timezone settings. Defaults to 'US' if not specified.

Show child attributes

countrystring#

ISO 3166-1 alpha-2 country code (e.g., 'US', 'AU', 'DE', 'JP')

Default: "US"

Pattern: ^[A-Z]{2}$

languagesstring[]#

Preferred languages and locales for the request in order of priority. Defaults to the language of the specified location. See https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Language

removeBase64Imagesboolean#

Removes all base 64 images from the markdown output, which may be overwhelmingly long. This does not affect html or rawHtml formats. The image's alt text remains in the output, but the URL is replaced with a placeholder.

Default: true

blockAdsboolean#

Enables ad-blocking and cookie popup blocking.

Default: true

proxyenum<string>#

Specifies the type of proxy to use.

basic: Proxies for scraping sites with none to basic anti-bot solutions. Fast and usually works.
enhanced: Enhanced proxies for scraping sites with advanced anti-bot solutions. Slower, but more reliable on certain sites. Billed at the same credit cost as basic.
auto: Firecrawl will automatically retry scraping with enhanced proxies if the basic proxy fails. Enhanced proxies carry no credit surcharge, so either way only the regular cost is billed.

Default: "auto"

Options:basicenhancedauto

storeInCacheboolean#

If true, the page will be stored in the Firecrawl index and cache. Setting this to false is useful if your scraping activity may have data protection concerns. Using some parameters associated with sensitive scraping (e.g. actions, headers) will force this parameter to be false.

Default: true

lockdownboolean#

If true, serves the request from Firecrawl's cache only and never makes an outbound request to the target URL. Designed for compliance-constrained or air-gapped environments where the scrape request itself could leak sensitive information. On cache miss, returns a 404 with error code SCRAPE_LOCKDOWN_CACHE_MISS (the URL is never logged on miss). Lockdown requests are treated as zero data retention. Default maxAge is extended to 2 years so existing cached pages remain eligible. Billed at 5 credits on hit, 1 credit on cache miss.

Default: false

redactPIIboolean · object#

Redact personally identifiable information from returned markdown. Pass true to use defaults, or an object to tune mode, entities, and replacement style.

Default: false

Show child attributes

One of

modeenum<string>#

Redaction strategy. accurate is model-only and optimized for precision, aggressive increases recall with additional heuristics, and fast uses heuristics without the model call.

Default: "accurate"

Options:accurateaggressivefast

entitiesenum<string>[]#

Restrict redaction to these entity buckets. If omitted, all supported entities are redacted.

replaceStyleenum<string>#

tag replaces spans with placeholders like <EMAIL>, mask replaces characters with *, and remove deletes the span text.

Default: "tag"

Options:tagmaskremove

profileobject#

Enable persistent browser storage across scrape and interact sessions. Pass a profile when scraping to preserve cookies, localStorage, and session data. Sessions with the same profile name share browser state.

Show child attributes

namestringrequired#

A name for the profile. Scrapes with the same name share browser state (cookies, localStorage, sessions).

Length: 1–128

saveChangesboolean#

When true, browser state is saved back to the profile when the interact session stops. Set to false to load existing data without writing. Only one saving session is allowed at a time.

Default: true

threatProtectionobject#

Per-request Threat Protection override. Fields you provide replace the corresponding fields of your organization's policy for this request only; omitted fields keep their organization-level values. Requires Threat Protection to be enabled for your team (enterprise feature) — otherwise the request is rejected with a 403. If your organization has disabled request overrides, any request that includes this object is rejected with a 403. If Threat Protection is enforced for your team, mode may not be set to off.

Show child attributes

modeenum<string>#

URL scanning mode for this request. normal checks URLs against Google Web Risk (+2 credits per URL scanned).

Options:offnormal

riskScoreThresholdinteger#

Normalized risk score (0–100) at or above which a classifier verdict blocks the URL. Lower is stricter.

Range: 0–100

blackliststring[]#

Domains to always block, as plain domains (example.com) or wildcard globs (*.example.com). No protocol, path, or port.

Items: max 1000

whiteliststring[]#

Domains to always allow, as plain domains or wildcard globs. Wins over every other rule.

Items: max 1000

blockedTldsstring[]#

Top-level domains to block outright, lowercase without the leading dot (e.g. zip).

Items: max 1000

failurePolicyenum<string>#

What to do when the classifier can't be reached: closed blocks the request, open allows it.

Options:openclosed

auditMetadataobject#

User attribution included with SIEM logging events when SIEM Logging is enabled for the organization.

Show child attributes

usernamestringrequired#

The username associated with the request.

Length: max 1024

zeroDataRetentionboolean#

If true, this will enable zero data retention for this crawl. To enable this feature, please contact help@firecrawl.dev

Default: false

Responses

application/json

Successful response

successboolean#

idstring#

urlstring#

Parse Get Crawl Status