PipesHub

PipesHub Crawling Jobs API

Endpoints for scheduling, managing, and monitoring data crawling jobs for enterprise connectors. The Crawling Manager uses BullMQ (Redis-based queue) to schedule and execute crawling jobs that sync data from external connectors (Google Drive, OneDrive, Slack, Jira, etc.) into PipesHub's search index. **Key Features:** - Schedule recurring crawls (hourly, daily, weekly, monthly) or one-time crawls - Pause and resume crawling jobs without losing configuration - Priority-based job execution (1-10 scale) - Automatic retry with exponential backoff on failures - Per-connector job isolation and tracking **Access Control:** - Team-scoped connectors require admin privileges - Personal-scoped connectors can only be managed by the creator - All operations require valid JWT authentication **Schedule Types:** - `hourly`: Run every X hours at specified minute - `daily`: Run once per day at specified time - `weekly`: Run on specified days of the week - `monthly`: Run on specified day of the month - `custom`: Use cron expression for complex schedules - `once`: Run once at a specific future time

GitHub OpenAPI

Operations 8

POST /crawlingManager/{connector}/{connectorId}/schedule Schedule a crawling job #
GET /crawlingManager/{connector}/{connectorId}/schedule Get crawling job status #
DELETE /crawlingManager/{connector}/{connectorId}/remove Remove a crawling job #
GET /crawlingManager/schedule/all Get all crawling job statuses #
DELETE /crawlingManager/schedule/all Remove all crawling jobs #
POST /crawlingManager/{connector}/{connectorId}/pause Pause a crawling job #
POST /crawlingManager/{connector}/{connectorId}/resume Resume a crawling job #
GET /crawlingManager/stats Get queue statistics #

Documentation

📖
Documentation
https://docs.pipeshub.com/developer/api-reference

Specifications

⚙
OpenAPI
https://raw.githubusercontent.com/api-evangelist/pipeshub/refs/heads/main/openapi/pipeshub-crawling-jobs-api-openapi.yml

Schemas & Data

📊
JSONSchema
https://raw.githubusercontent.com/api-evangelist/pipeshub/refs/heads/main/json-schema/pipeshub-get-record-by-id-response-schema-schema.json
📊
JSONSchema
https://raw.githubusercontent.com/api-evangelist/pipeshub/refs/heads/main/json-schema/pipeshub-record-schema.json
📊
JSONSchema
https://raw.githubusercontent.com/api-evangelist/pipeshub/refs/heads/main/json-schema/pipeshub-knowledge-hub-nodes-response-schema.json
📊
JSONSchema
https://raw.githubusercontent.com/api-evangelist/pipeshub/refs/heads/main/json-schema/pipeshub-agent-create-conversation-request-schema.json
📊
JSONSchema
https://raw.githubusercontent.com/api-evangelist/pipeshub/refs/heads/main/json-schema/pipeshub-conversation-schema.json
📊
JSONSchema
https://raw.githubusercontent.com/api-evangelist/pipeshub/refs/heads/main/json-schema/pipeshub-create-conversation-request-schema.json

Work with this as data

Every API here is available over the APIs.io API and to AI agents over MCP.

MCP server

One button, every client — Claude, Cursor, VS Code and the rest.

https://apis.io/mcp

Tools for apis

7 MCP tools reach this
  • find_apisBrowse and filter every API in the catalog.
  • get_api_artifactsOne API's artifacts, grouped by type.
  • get_openapiThe primary OpenAPI for this API.
  • find_similar_apisAPIs that look like this one.
  • apis_io_searchSTART HERE — APIs, providers and tags for one query, each with its total.
  • resolveTurn a domain, URL or GitHub org into the provider it belongs to.
  • find_cohortsEvery scored population of providers in the catalog.
All 92 tools →

Call it yourself

curl for this page
This API 
curl "https://apis.io/api/v1/apis/pipeshub:pipeshub-crawling-jobs-api"
All apis 
curl "https://apis.io/api/v1/apis?limit=25"

Discovery needs no key. Ratings and market analysis are Pro.

Get an API key

Free tier, no form to fill in. Signing in shares your email address with us — we store it to create your key and to recognise you if you sign in with another provider. See our Privacy Policy and Terms.

GitHub Google LinkedIn

A second provider on the same verified email joins the account you already have.

OpenAPI Specification

pipeshub-crawling-jobs-api-openapi.yml Raw ↑ 
openapi: 3.2.0
info:
  title: Pipeshub Crawling Jobs API
  version: 1.0.0
  contact:
    name: API Support
    email: support@pipeshub.com
  description: 'Operations tagged Crawling Jobs across 2 of this provider''s published API definitions: pipeshub-openapi.yaml, pipeshub-openapi.yml. Each path carries the servers of the definition it was published in.'
servers:
- url: '{instance_url}/api/v1'
  description: Base API URL
  variables:
    instance_url:
      default: https://app.pipeshub.com
      description: Base server URL (without /api/v1)
- url: '{instance_url}'
  description: Root URL (used for MCP endpoints mounted at /mcp)
  variables:
    instance_url:
      default: https://app.pipeshub.com
      description: Base server URL
security:
- bearerAuth: []
- oauth2: []
tags:


# --- truncated at 32 KB (40 KB total) ---
# Full source: https://raw.githubusercontent.com/api-evangelist/pipeshub/refs/heads/main/openapi/pipeshub-crawling-jobs-api-openapi.yml