CrawlRequest

Web ScrapingAPIAI AgentsMCPData Extraction

Properties

Name	Type	Description
url	string	The page to start from. An absolute http or https URL.
limit	integer	The most pages to crawl, 1 to 100,000.
maxDepth	integer	How many links deep to follow from the start page. 0 is the start page alone.
maxConcurrency	integer	How many pages to fetch at once, 1 to 50.
includePaths	array	Only crawl URLs whose path and query match one of these regexes, e.g. `^/blog/`. Empty means every path.
excludePaths	array	Skip URLs whose path and query match any of these regexes. Wins over `includePaths`.
allowExternalLinks	boolean	Follow links onto other domains. Off by default, since with a high `limit` a crawl can reach many unrelated sites.
allowBackwardLinks	boolean	Follow links above the starting path on the same host.
ignoreSitemap	boolean	Find pages by following links only, without reading the site's sitemap.
ignoreQueryParameters	boolean	Treat URLs that differ only in their query string as the same page.
deduplicateSimilarURLs	boolean	Collapse URLs that differ only in tracking parameters or a trailing slash.
delay	integer	Minimum milliseconds between requests to the same host.
respectRobots	boolean	Honor the site's robots.txt for this job.
scrapeOptions	object	How each page is fetched and what comes back: the same options /v1/scrape takes.
webhook	object	Where to send job events as the crawl progresses.

View JSON Schema on GitHub

JSON Schema

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://artifacts.apis.io/snoopscan-com/json-schema/snoopscan-com-crawl-request-schema.json",
  "title": "CrawlRequest",
  "x-generated": "2026-10-31",
  "x-method": "derived",
  "x-generator": "derive-json-schema.py",
  "x-source": "openapi/snoopscan-com-openapi.json#/components/schemas/CrawlRequest",
  "properties": {
    "url": {
      "type": "string",
      "title": "Url",
      "description": "The page to start from. An absolute http or https URL."
    },
    "limit": {
      "type": "integer",
      "maximum": 100000.0,
      "minimum": 1.0,
      "title": "Limit",
      "description": "The most pages to crawl, 1 to 100,000.",
      "default": 100
    },
    "maxDepth": {
      "type": "integer",
      "maximum": 20.0,
      "minimum": 0.0,
      "title": "Maxdepth",
      "description": "How many links deep to follow from the start page. 0 is the start page alone.",
      "default": 3
    },
    "maxConcurrency": {
      "type": "integer",
      "maximum": 50.0,
      "minimum": 1.0,
      "title": "Maxconcurrency",
      "description": "How many pages to fetch at once, 1 to 50.",
      "default": 5
    },
    "includePaths": {
      "items": {
        "type": "string"
      },
      "type": "array",
      "title": "Includepaths",
      "description": "Only crawl URLs whose path and query match one of these regexes, e.g. `^/blog/`. Empty means every path."
    },
    "excludePaths": {
      "items": {
        "type": "string"
      },
      "type": "array",
      "title": "Excludepaths",
      "description": "Skip URLs whose path and query match any of these regexes. Wins over `includePaths`."
    },
    "allowExternalLinks": {
      "type": "boolean",
      "title": "Allowexternallinks",
      "description": "Follow links onto other domains. Off by default, since with a high `limit` a crawl can reach many unrelated sites.",
      "default": false
    },
    "allowBackwardLinks": {
      "type": "boolean",
      "title": "Allowbackwardlinks",
      "description": "Follow links above the starting path on the same host.",
      "default": false
    },
    "ignoreSitemap": {
      "type": "boolean",
      "title": "Ignoresitemap",
      "description": "Find pages by following links only, without reading the site's sitemap.",
      "default": false
    },
    "ignoreQueryParameters": {
      "type": "boolean",
      "title": "Ignorequeryparameters",
      "description": "Treat URLs that differ only in their query string as the same page.",
      "default": false
    },
    "deduplicateSimilarURLs": {
      "type": "boolean",
      "title": "Deduplicatesimilarurls",
      "description": "Collapse URLs that differ only in tracking parameters or a trailing slash.",
      "default": true
    },
    "delay": {
      "type": "integer",
      "maximum": 60000.0,
      "minimum": 0.0,
      "title": "Delay",
      "description": "Minimum milliseconds between requests to the same host.",
      "default": 0
    },
    "respectRobots": {
      "type": "boolean",
      "title": "Respectrobots",
      "description": "Honor the site's robots.txt for this job.",
      "default": false
    },
    "scrapeOptions": {
      "$ref": "#/$defs/ScrapeOptions",
      "description": "How each page is fetched and what comes back: the same options /v1/scrape takes."
    },
    "webhook": {
      "anyOf": [
        {
          "$ref": "#/$defs/WebhookConfig"
        },
        {
          "type": "null"
        }
      ],
      "description": "Where to send job events as the crawl progresses."
    }
  },
  "additionalProperties": false,
  "type": "object",
  "required": [
    "url"
  ],
  "$defs": {
    "CaptchaCheckboxAction": {
      "properties": {
        "type": {
          "type": "string",
          "const": "captchaCheckbox",
          "title": "Type"
        },
        "frameSelector": {
          "type": "string",
          "title": "Frameselector",
          "default": "iframe[src*=\"/recaptcha/\"][src*=\"/anchor\"]"
        },
        "selector": {
          "type": "string",
          "title": "Selector",
          "default": "#recaptcha-anchor"
        },
        "successSelector": {
          "type": "string",
          "minLength": 1,
          "title": "Successselector"
        },
        "timeoutMs": {
          "type": "integer",
          "maximum": 15000.0,
          "minimum": 500.0,
          "title": "Timeoutms",
          "default": 8000
        },
        "captureEvidence": {
          "type": "boolean",
          "title": "Captureevidence",
          "description": "Return a screenshot from before and after the click.",
          "default": false
        }
      },
      "additionalProperties": false,
      "type": "object",
      "required": [
        "type",
        "successSelector"
      ],
      "title": "CaptchaCheckboxAction",
      "description": "Click a challenge checkbox once, then wait for `successSelector` to appear."
    },
    "ChangeTrackingFormat": {
      "properties": {
        "type": {
          "type": "string",
          "const": "changeTracking",
          "title": "Type"
        },
        "modes": {
          "items": {
            "type": "string",
            "enum": [
              "git-diff",
              "json"
            ]
          },
          "type": "array",
          "title": "Modes"
        }
      },
      "additionalProperties": false,
      "type": "object",
      "required": [
        "type"
      ],
      "title": "ChangeTrackingFormat"
    },
    "ClickAction": {
      "properties": {
        "type": {
          "type": "string",
          "const": "click",
          "title": "Type"
        },
        "selector": {
          "type": "string",
          "title": "Selector"
        }
      },
      "additionalProperties": false,
      "type": "object",
      "required": [
        "type",
        "selector"
      ],
      "title": "ClickAction"
    },
    "ExecuteJavascriptAction": {
      "properties": {
        "type": {
          "type": "string",
          "const": "executeJavascript",
          "title": "Type"
        },
        "script": {
          "type": "string",
          "title": "Script"
        }
      },
      "additionalProperties": false,
      "type": "object",
      "required": [
        "type",
        "script"
      ],
      "title": "ExecuteJavascriptAction"
    },
    "JsonFormat": {
      "properties": {
        "schema": {
          "anyOf": [
            {
              "additionalProperties": true,
              "type": "object"
            },
            {
              "type": "null"
            }
          ],
          "title": "Schema",
          "description": "Your own JSON Schema for the fields you want. Give this or `template`, not both."
        },
        "template": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "title": "Template",
          "description": "A named template instead of a schema — `product`, `article`, `jobPosting` and more. GET /v1/templates lists them with their fields."
        },
        "type": {
          "type": "string",
          "const": "json",
          "title": "Type"
        },
        "prompt": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "title": "Prompt"
        }
      },
      "additionalProperties": false,
      "type": "object",
      "required": [
        "type"
      ],
      "title": "JsonFormat"
    },
    "Location": {
      "properties": {
        "country": {
          "type": "string",
          "maxLength": 2,
          "minLength": 2,
          "title": "Country"
        },
        "languages": {
          "anyOf": [
            {
              "items": {
                "type": "string"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "title": "Languages"
        }
      },
      "additionalProperties": false,
      "type": "object",
      "required": [
        "country"
      ],
      "title": "Location"
    },
    "PressAction": {
      "properties": {
        "type": {
          "type": "string",
          "const": "press",
          "title": "Type"
        },
        "key": {
          "type": "string",
          "title": "Key"
        }
      },
      "additionalProperties": false,
      "type": "object",
      "required": [
        "type",
        "key"
      ],
      "title": "PressAction"
    },
    "ProxyMode": {
      "type": "string",
      "enum": [
        "none",
        "datacenter",
        "residential",
        "mobile",
        "auto"
      ],
      "title": "ProxyMode"
    },
    "ScrapeAction": {
      "properties": {
        "type": {
          "type": "string",
          "const": "scrape",
          "title": "Type"
        }
      },
      "additionalProperties": false,
      "type": "object",
      "required": [
        "type"
      ],
      "title": "ScrapeAction"
    },
    "ScrapeOptions": {
      "properties": {
        "formats": {
          "items": {
            "anyOf": [
              {
                "type": "string",
                "enum": [
                  "markdown",
                  "html",
                  "rawHtml",
                  "links",
                  "media",
                  "summary",
                  "screenshot",
                  "json",
                  "network"
                ]
              },
              {
                "$ref": "#/$defs/ScreenshotFormat"
              },
              {
                "$ref": "#/$defs/JsonFormat"
              },
              {
                "$ref": "#/$defs/ChangeTrackingFormat"
              }
            ]
          },
          "type": "array",
          "title": "Formats",
          "description": "What to return: `markdown`, `html`, `rawHtml`, `links`, `media`, `summary`, `screenshot`, or a `json` object carrying your schema. Ask for several in one call."
        },
        "onlyMainContent": {
          "type": "boolean",
          "title": "Onlymaincontent",
          "description": "Return the article and drop navigation, headers, footers and cookie banners. Turn it off to keep the whole page.",
          "default": true
        },
        "includeTags": {
          "items": {
            "type": "string"
          },
          "type": "array",
          "title": "Includetags",
          "description": "CSS selectors to keep even when they sit outside the main content."
        },
        "excludeTags": {
          "items": {
            "type": "string"
          },
          "type": "array",
          "title": "Excludetags",
          "description": "CSS selectors to drop before the page is read."
        },
        "maxAge": {
          "type": "integer",
          "minimum": 0.0,
          "title": "Maxage",
          "description": "Serve a cached copy if one was taken within this many milliseconds. A cache hit costs nothing. Set `0` to force a fresh fetch.",
          "default": 0
        },
        "storeInCache": {
          "type": "boolean",
          "title": "Storeincache",
          "description": "Keep this response in the cache so a later request can be served free.",
          "default": true
        },
        "waitFor": {
          "type": "integer",
          "maximum": 120000.0,
          "minimum": 0.0,
          "title": "Waitfor",
          "description": "Milliseconds to wait after the page loads before reading it, for content that loads late. Renders the page in a browser, and is billed as a browser page.",
          "default": 0
        },
        "timeout": {
          "type": "integer",
          "maximum": 300000.0,
          "minimum": 1000.0,
          "title": "Timeout",
          "description": "Give up on this request after this many milliseconds.",
          "default": 90000
        },
        "actions": {
          "items": {
            "oneOf": [
              {
                "$ref": "#/$defs/WaitAction"
              },
              {
                "$ref": "#/$defs/ClickAction"
              },
              {
                "$ref": "#/$defs/CaptchaCheckboxAction"
              },
              {
                "$ref": "#/$defs/WriteAction"
              },
              {
                "$ref": "#/$defs/PressAction"
              },
              {
                "$ref": "#/$defs/ScrollAction"
              },
              {
                "$ref": "#/$defs/ScreenshotAction"
              },
              {
                "$ref": "#/$defs/ScrapeAction"
              },
              {
                "$ref": "#/$defs/ExecuteJavascriptAction"
              }
            ]
          },
          "type": "array",
          "title": "Actions",
          "description": "Steps to run in a browser before the page is read: click, type, scroll, wait or `captchaCheckbox`. A `captchaCheckbox` step needs `successSelector`."
        },
        "captchaHandling": {
          "type": "string",
          "enum": [
            "auto",
            "off"
          ],
          "title": "Captchahandling",
          "description": "Click a challenge checkbox once if one appears. Set `off` to never click.",
          "default": "auto"
        },
        "captchaEvidence": {
          "type": "boolean",
          "title": "Captchaevidence",
          "description": "Return screenshots from before and after an automatic checkbox click.",
          "default": false
        },
        "headers": {
          "additionalProperties": {
            "type": "string"
          },
          "type": "object",
          "title": "Headers",
          "description": "Extra headers to send. `User-Agent` and other browser-identity headers are set for you and cannot be changed."
        },
        "mobile": {
          "type": "boolean",
          "title": "Mobile",
          "description": "Read the page as a phone would see it.",
          "default": false
        },
        "location": {
          "anyOf": [
            {
              "$ref": "#/$defs/Location"
            },
            {
              "type": "null"
            }
          ],
          "description": "Fetch from a country, in its language, e.g. `{\"country\": \"US\"}`."
        },
        "proxy": {
          "$ref": "#/$defs/ProxyMode",
          "description": "Leave on `auto` and the route is chosen for you.",
          "default": "auto"
        },
        "blockAssets": {
          "type": "boolean",
          "title": "Blockassets",
          "description": "Skip images, fonts and media when the page is rendered in a browser. Faster.",
          "default": true
        },
        "removeBase64Images": {
          "type": "boolean",
          "title": "Removebase64Images",
          "description": "Strip inline base64 images out of the returned markdown and HTML.",
          "default": true
        },
        "parsers": {
          "items": {
            "type": "string",
            "const": "pdf"
          },
          "type": "array",
          "title": "Parsers",
          "description": "Which document parsers may run, for example `pdf`."
        }
      },
      "additionalProperties": false,
      "type": "object",
      "title": "ScrapeOptions",
      "description": "How a page is fetched and what comes back: every /v1/scrape field except `url`."
    },
    "ScreenshotAction": {
      "properties": {
        "type": {
          "type": "string",
          "const": "screenshot",
          "title": "Type"
        },
        "fullPage": {
          "type": "boolean",
          "title": "Fullpage",
          "default": false
        }
      },
      "additionalProperties": false,
      "type": "object",
      "required": [
        "type"
      ],
      "title": "ScreenshotAction"
    },
    "ScreenshotFormat": {
      "properties": {
        "type": {
          "type": "string",
          "const": "screenshot",
          "title": "Type"
        },
        "fullPage": {
          "type": "boolean",
          "title": "Fullpage",
          "default": false
        },
        "quality": {
          "type": "integer",
          "maximum": 100.0,
          "minimum": 1.0,
          "title": "Quality",
          "default": 80
        }
      },
      "additionalProperties": false,
      "type": "object",
      "required": [
        "type"
      ],
      "title": "ScreenshotFormat"
    },
    "ScrollAction": {
      "properties": {
        "type": {
          "type": "string",
          "const": "scroll",
          "title": "Type"
        },
        "direction": {
          "type": "string",
          "enum": [
            "up",
            "down"
          ],
          "title": "Direction",
          "default": "down"
        },
        "amount": {
          "type": "integer",
          "maximum": 50.0,
          "minimum": 1.0,
          "title": "Amount",
          "default": 1
        }
      },
      "additionalProperties": false,
      "type": "object",
      "required": [
        "type"
      ],
      "title": "ScrollAction"
    },
    "WaitAction": {
      "properties": {
        "type": {
          "type": "string",
          "const": "wait",
          "title": "Type"
        },
        "milliseconds": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "title": "Milliseconds"
        },
        "selector": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "title": "Selector"
        }
      },
      "additionalProperties": false,
      "type": "object",
      "required": [
        "type"
      ],
      "title": "WaitAction"
    },
    "WebhookConfig": {
      "properties": {
        "url": {
          "type": "string",
          "title": "Url"
        },
        "events": {
          "items": {
            "type": "string",
            "enum": [
              "started",
              "page",
              "completed",
              "failed"
            ]
          },
          "type": "array",
          "title": "Events"
        },
        "headers": {
          "additionalProperties": {
            "type": "string"
          },
          "type": "object",
          "title": "Headers"
        }
      },
      "additionalProperties": false,
      "type": "object",
      "required": [
        "url"
      ],
      "title": "WebhookConfig"
    },
    "WriteAction": {
      "properties": {
        "type": {
          "type": "string",
          "const": "write",
          "title": "Type"
        },
        "selector": {
          "type": "string",
          "title": "Selector"
        },
        "text": {
          "type": "string",
          "title": "Text"
        }
      },
      "additionalProperties": false,
      "type": "object",
      "required": [
        "type",
        "selector",
        "text"
      ],
      "title": "WriteAction"
    }
  }
}

Work with this as data

Every JSON Schema here is available over the APIs.io API and to AI agents over MCP.

MCP server

One button, every client — Claude, Cursor, VS Code and the rest.

https://apis.io/mcp

Tools for schemas

4 MCP tools reach this

find_json_schemasBrowse and filter every JSON Schema in the catalog.
apis_io_searchSTART HERE — APIs, providers and tags for one query, each with its total.
resolveTurn a domain, URL or GitHub org into the provider it belongs to.
find_cohortsEvery scored population of providers in the catalog.

All 92 tools →

Call it yourself

curl for this page

This JSON Schema

curl "https://apis.io/api/v1/json-schemas/snoopscan-com-crawl-request"

All schemas

curl "https://apis.io/api/v1/json-schemas?limit=25"

Discovery needs no key. Ratings and market analysis are Pro.

Get an API key

Free tier, no form to fill in. Signing in shares your email address with us — we store it to create your key and to recognise you if you sign in with another provider. See our Privacy Policy and Terms.

GitHub Google LinkedIn

A second provider on the same verified email joins the account you already have.