Crawl

1 min read

post/crawl

Request example

curl --request POST \
  --url https://api.firecrawl.dev/v1/crawl \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '{
  "url": "<string>",
  "excludePaths": [
    "<string>"
  ],
  "includePaths": [
    "<string>"
  ],
  "regexOnFullURL": false,
  "maxDepth": 10,
  "maxDiscoveryDepth": 1,
  "ignoreSitemap": false,
  "ignoreQueryParameters": false,
  "limit": 10000,
  "allowBackwardLinks": false,
  "crawlEntireDomain": false,
  "allowExternalLinks": false,
  "allowSubdomains": false,
  "delay": 1,
  "maxConcurrency": 1,
  "webhook": {
    "url": "<string>",
    "headers": {},
    "metadata": {},
    "events": [
      "completed"
    ]
  },
  "scrapeOptions": {
    "onlyMainContent": true,
    "includeTags": [
      "<string>"
    ],
    "excludeTags": [
      "<string>"
    ],
    "maxAge": 0,
    "headers": {},
    "waitFor": 0,
    "mobile": false,
    "skipTlsVerification": false,
    "timeout": 30000,
    "parsePDF": true,
    "jsonOptions": {
      "schema": {},
      "systemPrompt": "<string>",
      "prompt": "<string>",
      "checkPromptInjection": false
    },
    "actions": [
      {
        "type": "wait",
        "milliseconds": 1,
        "selector": "#my-element"
      }
    ],
    "location": {
      "country": "US",
      "languages": [
        "en-US"
      ]
    },
    "removeBase64Images": true,
    "blockAds": true,
    "proxy": "basic",
    "storeInCache": true,
    "threatProtection": {
      "mode": "off",
      "riskScoreThreshold": 75,
      "blacklist": [
        "<string>"
      ],
      "whitelist": [
        "<string>"
      ],
      "blockedTlds": [
        "<string>"
      ],
      "failurePolicy": "open"
    },
    "formats": [
      "markdown"
    ],
    "changeTrackingOptions": {
      "modes": [
        "git-diff"
      ],
      "schema": {},
      "prompt": "<string>",
      "tag": null
    }
  },
  "zeroDataRetention": false
}'

Response

{
  "success": true,
  "id": "<string>",
  "url": "<string>"
}

post/crawl

Note: A new v2 version of this API is now available with improved features and performance.

Authorization

Authorizationstringheaderrequired#

Bearer authentication header of the form Bearer <token>, where <token> is your API key.

Body

application/json

urlstringrequired#

The base URL to start crawling from

excludePathsstring[]#

URL pathname regex patterns that exclude matching URLs from the crawl. For example, if you set "excludePaths": ["blog/.*"] for the base URL firecrawl.dev, any results matching that pattern will be excluded, such as https://www.firecrawl.dev/blog/firecrawl-launch-week-1-recap.

includePathsstring[]#

URL pathname regex patterns that include matching URLs in the crawl. Only the paths that match the specified patterns will be included in the response. For example, if you set "includePaths": ["blog/.*"] for the base URL firecrawl.dev, only results matching that pattern will be included, such as https://www.firecrawl.dev/blog/firecrawl-launch-week-1-recap.

regexOnFullURLboolean#

When true, includePaths and excludePaths regex patterns are matched against the full URL (including query parameters) instead of just the URL pathname. Useful when you need to filter URLs based on query strings.

Default: false

maxDepthinteger#

Maximum absolute depth to crawl from the base of the entered URL. Basically, the max number of slashes the pathname of a scraped URL may contain.

Default: 10

maxDiscoveryDepthinteger#

Maximum depth to crawl based on discovery order. The root site and sitemapped pages has a discovery depth of 0. For example, if you set it to 1, and you set ignoreSitemap, you will only crawl the entered URL and all URLs that are linked on that page.

ignoreSitemapboolean#

Ignore the website sitemap when crawling

Default: false

ignoreQueryParametersboolean#

Do not re-scrape the same path with different (or none) query parameters

Default: false

limitinteger#

Maximum number of pages to crawl. Default limit is 10000.

Default: 10000

allowBackwardLinksbooleandeprecated#

⚠️ DEPRECATED: Use 'crawlEntireDomain' instead. Allows the crawler to follow internal links to sibling or parent URLs, not just child paths.

Default: false

crawlEntireDomainboolean#

Allows the crawler to follow internal links to sibling or parent URLs, not just child paths.

false: Only crawls deeper (child) URLs.
→ e.g. /features/feature-1 → /features/feature-1/tips ✅
→ Won't follow /pricing or / ❌

true: Crawls any internal links, including siblings and parents.
→ e.g. /features/feature-1 → /pricing, /, etc. ✅

Use true for broader internal coverage beyond nested paths.

Default: false

allowExternalLinksboolean#

Allows the crawler to follow links to external websites.

Default: false

allowSubdomainsboolean#

Allows the crawler to follow links to subdomains of the main domain.

Default: false

delaynumber#

Delay in seconds between scrapes. This helps respect website rate limits.

maxConcurrencyinteger#

Maximum number of concurrent scrapes. This parameter allows you to set a concurrency limit for this crawl. If not specified, the crawl adheres to your team's concurrency limit.

webhookobject#

A webhook specification object.

Show child attributes

urlstringrequired#

The URL to send the webhook to. This will trigger for crawl started (crawl.started), every page crawled (crawl.page) and when the crawl is completed (crawl.completed or crawl.failed). The response will be the same as the /scrape endpoint.

headersobject#

Headers to send to the webhook URL.

Show child attributes

{key}string#

metadataobject#

Custom metadata that will be included in all webhook payloads for this crawl

eventsenum<string>[]#

Type of events that should be sent to the webhook URL. (default: all)

scrapeOptionsobject#

Show child attributes

onlyMainContentboolean#

Only return the main content of the page excluding headers, navs, footers, etc.

Default: true

includeTagsstring[]#

Tags to include in the output.

excludeTagsstring[]#

Tags to exclude from the output.

maxAgeinteger#

Returns a cached version of the page if it is younger than this age in milliseconds. If a cached version of the page is older than this value, the page will be scraped. If you do not need extremely fresh data, enabling this can speed up your scrapes by 500%. Defaults to 0, which disables caching.

Default: 0

headersobject#

Headers to send with the request. Can be used to send cookies, user-agent, etc.

waitForinteger#

Specify a delay in milliseconds before fetching the content, allowing the page sufficient time to load.

Default: 0

mobileboolean#

Set to true if you want to emulate scraping from a mobile device. Useful for testing responsive pages and taking mobile screenshots.

Default: false

skipTlsVerificationboolean#

Skip TLS certificate verification when making requests

Default: false

timeoutinteger#

Timeout in milliseconds for the request

Default: 30000

parsePDFboolean#

Controls how PDF files are processed during scraping. When true, the PDF content is extracted and converted to markdown format, with billing based on the number of pages (1 credit per page). When false, the PDF file is returned in base64 encoding with a flat rate of 1 credit total.

Default: true

jsonOptionsobject#

JSON options object

Show child attributes

schemaobject#

The schema to use for the extraction (Optional). Must conform to JSON Schema.

systemPromptstring#

The system prompt to use for the extraction (Optional)

promptstring#

The prompt to use for the extraction without a schema (Optional)

checkPromptInjectionboolean#

When enabled, scans the scraped page content for prompt injection attempts before running the extraction. If an injection is detected, the request fails with a 403 and error code SCRAPE_PROMPT_INJECTION_DETECTED. Adds 4 credits when the check runs. Defaults to false.

Default: false

actionsobject[]#

Actions to perform on the page before grabbing the content

Show child attributes

One of

typeenum<string>required#

Wait for a specified amount of milliseconds

Options:wait

millisecondsinteger#

Number of milliseconds to wait

Range: >= 1

selectorstring#

Query selector to find the element by

typeenum<string>required#

Take a screenshot. The links will be in the response's actions.screenshots array.

Options:screenshot

fullPageboolean#

Whether to capture a full-page screenshot or limit to the current viewport.

Default: false

qualityinteger#

The quality of the screenshot, from 1 to 100. 100 is the highest quality.

typeenum<string>required#

Click on an element

Options:click

selectorstringrequired#

Query selector to find the element by

allboolean#

Clicks all elements matched by the selector, not just the first one. Does not throw an error if no elements match the selector.

Default: false

typeenum<string>required#

Write text into an input field, text area, or contenteditable element. Note: You must first focus the element using a 'click' action before writing. The text will be typed character by character to simulate keyboard input.

Options:write

textstringrequired#

Text to type

typeenum<string>required#

Press a key on the page

Options:press

keystringrequired#

Key to press

typeenum<string>required#

Scroll the page or a specific element

Options:scroll

directionenum<string>#

Direction to scroll

Default: "down"

Options:updown

selectorstring#

Query selector for the element to scroll

typeenum<string>required#

Scrape the current page content, returns the url and the html.

Options:scrape

typeenum<string>required#

Execute JavaScript code on the page

Options:executeJavascript

scriptstringrequired#

JavaScript code to execute

typeenum<string>required#

Generate a PDF of the current page. The PDF will be returned in the actions.pdfs array of the response.

Options:pdf

formatenum<string>#

The page size of the resulting PDF

Default: "Letter"

Options:A0A1A2A3A4A5A6LetterLegalTabloidLedger

landscapeboolean#

Whether to generate the PDF in landscape orientation

Default: false

scalenumber#

The scale multiplier of the resulting PDF

Default: 1

locationobject#

Location settings for the request. When specified, this will use an appropriate proxy if available and emulate the corresponding language and timezone settings. Defaults to 'US' if not specified.

Show child attributes

countrystring#

ISO 3166-1 alpha-2 country code (e.g., 'US', 'AU', 'DE', 'JP')

Default: "US"

Pattern: ^[A-Z]{2}$

languagesstring[]#

Preferred languages and locales for the request in order of priority. Defaults to the language of the specified location. See https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Language

removeBase64Imagesboolean#

Removes all base 64 images from the output, which may be overwhelmingly long. The image's alt text remains in the output, but the URL is replaced with a placeholder.

Default: true

blockAdsboolean#

Enables ad-blocking and cookie popup blocking.

Default: true

proxyenum<string>#

Specifies the type of proxy to use.

basic: Proxies for scraping sites with none to basic anti-bot solutions. Fast and usually works.
enhanced: Enhanced proxies for scraping sites with advanced anti-bot solutions. Slower, but more reliable on certain sites. Billed at the same credit cost as basic.
auto: Firecrawl will automatically retry scraping with enhanced proxies if the basic proxy fails. Enhanced proxies carry no credit surcharge, so either way only the regular cost is billed.

If you do not specify a proxy, Firecrawl will default to basic.

Options:basicenhancedauto

storeInCacheboolean#

If true, the page will be stored in the Firecrawl index and cache. Setting this to false is useful if your scraping activity may have data protection concerns. Using some parameters associated with sensitive scraping (actions, headers) will force this parameter to be false.

Default: true

threatProtectionobject#

Per-request Threat Protection override. Fields you provide replace the corresponding fields of your organization's policy for this request only; omitted fields keep their organization-level values. Requires Threat Protection to be enabled for your team (enterprise feature) — otherwise the request is rejected with a 403. If your organization has disabled request overrides, any request that includes this object is rejected with a 403. If Threat Protection is enforced for your team, mode may not be set to off.

Show child attributes

modeenum<string>#

URL scanning mode for this request. normal checks URLs against Google Web Risk (+2 credits per URL scanned).

Options:offnormal

riskScoreThresholdinteger#

Normalized risk score (0–100) at or above which a classifier verdict blocks the URL. Lower is stricter.

Range: 0–100

blackliststring[]#

Domains to always block, as plain domains (example.com) or wildcard globs (*.example.com). No protocol, path, or port.

Items: max 1000

whiteliststring[]#

Domains to always allow, as plain domains or wildcard globs. Wins over every other rule.

Items: max 1000

blockedTldsstring[]#

Top-level domains to block outright, lowercase without the leading dot (e.g. zip).

Items: max 1000

failurePolicyenum<string>#

What to do when the classifier can't be reached: closed blocks the request, open allows it.

Options:openclosed

formatsenum<string>[]#

Formats to include in the output.

Default: ["markdown"]

changeTrackingOptionsobject#

Options for change tracking (Beta). Only applicable when 'changeTracking' is included in formats. The 'markdown' format must also be specified when using change tracking.

Show child attributes

modesenum<string>[]#

The mode to use for change tracking. 'git-diff' provides a detailed diff, and 'json' compares extracted JSON data.

schemaobject#

Schema for JSON extraction when using 'json' mode. Defines the structure of data to extract and compare. Must conform to JSON Schema.

promptstring#

Prompt to use for change tracking when using 'json' mode. If not provided, the default prompt will be used.

tagstring#

Tag to use for change tracking. Tags can separate change tracking history into separate "branches", where change tracking with a specific tagwill only compare to scrapes made in the same tag. If not provided, the default tag (null) will be used.

Default: null

zeroDataRetentionboolean#

If true, this will enable zero data retention for this crawl. To enable this feature, please contact help@firecrawl.dev

Default: false

Responses

application/json

Successful response

successboolean#

idstring#

urlstring#

Get Batch Scrape Errors Get Crawl Status