# PipesHub Crawling Jobs API

**Canonical:** https://apis.io/apis/pipeshub/pipeshub-crawling-jobs-api/  
**Provider:** PipesHub — https://apis.io/providers/pipeshub/  
**Base URL:** https://app.pipeshub.com/api/v1  
**Documentation:** https://docs.pipeshub.com/developer/api-reference

PipesHub Crawling Jobs API is one of 51 APIs that [PipesHub](https://apis.io/providers/pipeshub/) publishes on the [APIs.io](https://apis.io/) network, described by a machine-readable OpenAPI specification. This API exposes 6 JSON Schema definitions. Tagged areas include Crawling Jobs. The published artifact set on APIs.io includes an OpenAPI specification, API documentation, and 6 JSON Schemas.

Endpoints for scheduling, managing, and monitoring data crawling jobs for enterprise connectors. The Crawling Manager uses BullMQ (Redis-based queue) to schedule and execute crawling jobs that sync data from external connectors (Google Drive, OneDrive, Slack, Jira, etc.) into PipesHub's search index. **Key Features:** - Schedule recurring crawls (hourly, daily, weekly, monthly) or one-time crawls - Pause and resume crawling jobs without losing configuration - Priority-based job execution (1-10 scale) - Automatic retry with exponential backoff on failures - Per-connector job isolation and tracking **Access Control:** - Team-scoped connectors require admin privileges - Personal-scoped connectors can only be managed by the creator - All operations require valid JWT authentication **Schedule Types:** - `hourly`: Run every X hours at specified minute - `daily`: Run once per day at specified time - `weekly`: Run on specified days of the week - `monthly`: Run on specified day of the month - `custom`: Use cron expression for complex schedules - `once`: Run once at a specific future time

## Operations (8)

| Method | Path | Summary |
|---|---|---|
| POST | `/crawlingManager/{connector}/{connectorId}/schedule` | Schedule a crawling job |
| GET | `/crawlingManager/{connector}/{connectorId}/schedule` | Get crawling job status |
| DELETE | `/crawlingManager/{connector}/{connectorId}/remove` | Remove a crawling job |
| GET | `/crawlingManager/schedule/all` | Get all crawling job statuses |
| DELETE | `/crawlingManager/schedule/all` | Remove all crawling jobs |
| POST | `/crawlingManager/{connector}/{connectorId}/pause` | Pause a crawling job |
| POST | `/crawlingManager/{connector}/{connectorId}/resume` | Resume a crawling job |
| GET | `/crawlingManager/stats` | Get queue statistics |

## Machine-readable artifacts (8)

- **OpenAPI** — https://raw.githubusercontent.com/api-evangelist/pipeshub/refs/heads/main/openapi/pipeshub-crawling-jobs-api-openapi.yml
- **Documentation** — https://docs.pipeshub.com/developer/api-reference
- **JSONSchema** — https://raw.githubusercontent.com/api-evangelist/pipeshub/refs/heads/main/json-schema/pipeshub-get-record-by-id-response-schema-schema.json
- **JSONSchema** — https://raw.githubusercontent.com/api-evangelist/pipeshub/refs/heads/main/json-schema/pipeshub-record-schema.json
- **JSONSchema** — https://raw.githubusercontent.com/api-evangelist/pipeshub/refs/heads/main/json-schema/pipeshub-knowledge-hub-nodes-response-schema.json
- **JSONSchema** — https://raw.githubusercontent.com/api-evangelist/pipeshub/refs/heads/main/json-schema/pipeshub-agent-create-conversation-request-schema.json
- **JSONSchema** — https://raw.githubusercontent.com/api-evangelist/pipeshub/refs/heads/main/json-schema/pipeshub-conversation-schema.json
- **JSONSchema** — https://raw.githubusercontent.com/api-evangelist/pipeshub/refs/heads/main/json-schema/pipeshub-create-conversation-request-schema.json

## Other PipesHub APIs (12)

- [PipesHub Agents API](https://apis.io/apis/pipeshub/pipeshub-agents-api/)
- [PipesHub AI Models Providers API](https://apis.io/apis/pipeshub/pipeshub-ai-models-providers-api/)
- [PipesHub Authentication Configuration API](https://apis.io/apis/pipeshub/pipeshub-authentication-configuration-api/)
- [PipesHub Configuration Manager API](https://apis.io/apis/pipeshub/pipeshub-configuration-manager-api/)
- [PipesHub Connector API](https://apis.io/apis/pipeshub/pipeshub-connector-api/)
- [PipesHub Connector Configuration API](https://apis.io/apis/pipeshub/pipeshub-connector-configuration-api/)
- [PipesHub Connector Control API](https://apis.io/apis/pipeshub/pipeshub-connector-control-api/)
- [PipesHub Connector Filters API](https://apis.io/apis/pipeshub/pipeshub-connector-filters-api/)
- [PipesHub Connector Instances API](https://apis.io/apis/pipeshub/pipeshub-connector-instances-api/)
- [PipesHub Connector OAuth API](https://apis.io/apis/pipeshub/pipeshub-connector-oauth-api/)
- [PipesHub Connector Registry API](https://apis.io/apis/pipeshub/pipeshub-connector-registry-api/)
- [PipesHub Connectors API](https://apis.io/apis/pipeshub/pipeshub-connectors-api/)

## Tags

Crawling Jobs

---

Profiled by [API Evangelist](https://apievangelist.com) and published on [APIs.io](https://apis.io/apis/pipeshub/pipeshub-crawling-jobs-api/). The API's provider profile, Kin Score and agent-readiness rating are at https://apis.io/providers/pipeshub/.