usewrit-app Datasets API

openapi: 3.2.0
info:
  title: Writ Agent Datasets API
  version: 0.1.0
  license:
    name: MIT
    identifier: MIT
  description: 'Loopback HTTP API of the Writ local agent (`writ-agentd`), the daemon embedded in the

    Writ desktop app and the self-host build. This document is generated against the Rust

    daemon source (`writ-agent/src/local/`), which is the wire truth.


    ## Transport


    - Plain HTTP on `127.0.0.1:8131` (default; `WRIT_PORT` env overrides).

    - Optional HTTPS twin on `127.0.0.1:8132`, served with a certificate minted by a

    per-install, name-constrained local CA (`~/.writ/tls/ca.pem`). Clients must trust that

    CA file (or pin it) to use the TLS port; the payloads are identical on both ports.

    - The daemon enforces a DNS-rebind guard: `Host`/`Origin` must be loopback. Talk to

    `127.0.0.1`, not `localhost`.


    ## Errors


    Domain errors are JSON `{"error": "", "code": ""}` (see the

    `Error` schema). Codes include `bad_request`, `unauthorized`, `forbidden`, `not_found`,

    `vault_locked` (HTTP 423), `too_many_requests`, `device_capacity` (HTTP 409),

    `captcha_required` (HTTP 402), `internal`.


    **Caveat:** some 4xx bodies are `text/plain`, not JSON — axum framework rejections

    (e.g. a JSON body that fails to deserialize returns

    `Failed to deserialize the JSON body into the target type: missing field ''url''` as

    plain text with a 4xx status). Clients must tolerate non-JSON error bodies.


    ## List envelopes


    The wire is deliberately documented as-is (SDKs normalize; this spec does not):


    - `{"data": [...], "count": n}` — workflows, personas, persona runs, secrets, files, keys

    - `{"data": [...], "count": n, "total": n}` — runs

    - **bare JSON array** — monitors, monitor selectors, extractors, automations,

    `/v1/changes/recent`

    - bespoke objects — `/v1/monitors/{id}/changes`, `/v1/data`, the workflow-data table'
servers:
- url: http://127.0.0.1:8131
  description: Plain-HTTP loopback listener (default port; WRIT_PORT overrides).
- url: https://127.0.0.1:8132
  description: HTTPS twin of the same API, terminated with a certificate from the per-install local CA at ~/.writ/tls/ca.pem. Trust that CA (or pin it) in your HTTP client.
security:
- bearerAuth: []
tags:
- name: Datasets
  description: Unified read-only catalogue over everything that has accumulated extracted data — crawl and workflow origins alike, keyed by the same id (`source_type` tells them apart). Records/export/search accept `?format=json|csv|markdown|html`; a non-json format returns rendered TEXT instead of the JSON envelope, and markdown/html are CONTENT-AWARE (a crawl's pages render as documents, structured data as a table). Mirrors the cloud `/api/v1/datasets/*` surface (tagged `Public API · Datasets`), so a dataset reads the same locally or forwarded to the fleet.
paths:
  /v1/datasets:
    get:
      operationId: listDatasets
      tags:
      - Datasets
      summary: The unified dataset catalogue
      description: 'Every data source that has accumulated extracted data, framed as a dataset: local data-bearing workflows plus (on a cloud-linked daemon) forwarded cloud crawl datasets, merged into one list. `source_type` is `crawl` for a whole-site crawl, otherwise `workflow`; `origin` marks where the dataset lives. Read a dataset''s rows with `GET /v1/datasets/{id}/records`.'
      responses:
        '200':
          description: Catalogue entries under a `datasets` key (NOT the `{data, count}` envelope).
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/DatasetList'
        '401':
          $ref: '#/components/responses/Unauthorized'
  /v1/datasets/search:
    get:
      operationId: searchDatasets
      tags:
      - Datasets
      summary: Global full-text search across every dataset
      description: 'Full-text search over every local dataset at once, backed by the SQLite FTS5 index (not a bounded scan), so it stays fast and does not miss older data. Terms are space-separated, ANDed, case-insensitive and prefix-matched; a record must contain EVERY term. Results come back newest-first, each tagged with the dataset it came from and a `highlight` snippet. Extracted data covers both JSON fields and markdown (a crawl page is a record with a `markdown` field), so both are searched.


        A non-json `format` renders the matches as TEXT instead of the JSON envelope — the per-result `dataset` tag and `highlight` snippet exist only in the JSON shape. `truncated` flags that the candidate-run cap (500) was hit.'
      parameters:
      - $ref: '#/components/parameters/DatasetSearchQuery'
      - $ref: '#/components/parameters/DatasetSearchLimit'
      - $ref: '#/components/parameters/DatasetSearchOffset'
      - $ref: '#/components/parameters/DatasetFormat'
      responses:
        '200':
          $ref: '#/components/responses/DatasetSearchResponse'
        '400':
          $ref: '#/components/responses/BadRequest'
        '401':
          $ref: '#/components/responses/Unauthorized'
  /v1/datasets/{id}:
    parameters:
    - $ref: '#/components/parameters/DatasetId'
    get:
      operationId: getDataset
      tags:
      - Datasets
      summary: Dataset metadata + inferred schema
      description: 'One dataset''s metadata and inferred schema: its columns, per-column `facets` (inferred type, non-empty count, numeric min/max, a distinct pick-list when low-cardinality), total `row_count`, the `run_count` behind it, and `truncated` when the run-scan cap bounded the answer.


        On a cloud-linked daemon a crawl dataset that is absent locally FORWARDS to the cloud dataset, and the forwarded body is the cloud facets shape — which also carries `collections` (the nested array paths you can pivot to). A purely local dataset does not return `collections`.'
      responses:
        '200':
          description: The dataset's metadata + schema.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/DatasetMeta'
        '401':
          $ref: '#/components/responses/Unauthorized'
        '404':
          $ref: '#/components/responses/NotFound'
  /v1/datasets/{id}/records:
    parameters:
    - $ref: '#/components/parameters/DatasetId'
    get:
      operationId: getDatasetRecords
      tags:
      - Datasets
      summary: Page through a dataset's records
      description: 'The dataset''s records as one sortable / searchable / filterable table (the flat grid). A run that extracted a LIST contributes one record per item; each record carries the run it came from and, with `include_inputs`, that run''s inputs as `input.` columns. Identical query semantics to `GET /v1/workflows/{id}/data` (`view=all`).


        A non-json `format` serves THIS SAME PAGE rendered as text — pagination still applies, so use `/v1/datasets/{id}/export` for the whole set. markdown/html are content-aware: a crawl''s pages render as documents, structured data as a table.


        On a cloud-linked daemon a crawl dataset absent locally forwards to the fleet and the response is re-keyed to this same shape.'
      parameters:
      - $ref: '#/components/parameters/DatasetQ'
      - $ref: '#/components/parameters/DatasetFilter'
      - $ref: '#/components/parameters/DatasetFilters'
      - $ref: '#/components/parameters/DatasetSortBy'
      - $ref: '#/components/parameters/DatasetSortDir'
      - name: limit
        in: query
        description: Page size (clamped to 1..=500; default 50).
        schema:
          type: integer
          default: 50
      - name: offset
        in: query
        schema:
          type: integer
          default: 0
      - $ref: '#/components/parameters/DatasetIncludeInputs'
      - $ref: '#/components/parameters/DatasetCollection'
      - $ref: '#/components/parameters/DatasetFormat'
      responses:
        '200':
          description: The records page — the JSON envelope for `format=json` (the default), or the rendered page as TEXT for any other format.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/DatasetRecords'
            text/csv:
              schema:
                type: string
                description: '`format=csv` — RFC-4180 CSV of this page.'
            text/markdown:
              schema:
                type: string
                description: '`format=markdown` — content-aware: documents when the records carry a long-form content column, otherwise a table.'
            text/html:
              schema:
                type: string
                description: '`format=html` — the markdown render as HTML. Scraped third-party content is escaped and link schemes allowlisted; served `nosniff` under a script-less CSP.'
        '400':
          $ref: '#/components/responses/BadRequest'
        '401':
          $ref: '#/components/responses/Unauthorized'
        '404':
          $ref: '#/components/responses/NotFound'
  /v1/datasets/{id}/export:
    parameters:
    - $ref: '#/components/parameters/DatasetId'
    get:
      operationId: exportDataset
      tags:
      - Datasets
      summary: Download a dataset's full records
      description: 'Downloads the dataset''s full (search / sort / filter applied, UN-paginated) records, bounded by the run-scan cap, framed as an attachment. The bytes are identical to `GET /v1/workflows/{id}/data/export`, which this delegates to.


        NOTE the default here is `csv`, NOT `json` — the one place the dataset surface departs from the `format=json` default of every other dataset read. markdown/html are content-aware: a crawl''s pages download as readable documents, structured data as a table.'
      parameters:
      - $ref: '#/components/parameters/DatasetExportFormat'
      - $ref: '#/components/parameters/DatasetQ'
      - $ref: '#/components/parameters/DatasetFilter'
      - $ref: '#/components/parameters/DatasetFilters'
      - $ref: '#/components/parameters/DatasetSortBy'
      - $ref: '#/components/parameters/DatasetSortDir'
      - $ref: '#/components/parameters/DatasetIncludeInputs'
      - $ref: '#/components/parameters/DatasetCollection'
      responses:
        '200':
          description: 'The exported records as a download (`Content-Disposition: attachment`).'
          content:
            text/csv:
              schema:
                type: string
                description: '`format=csv` (the default) — RFC-4180 CSV; run-meta columns then data columns.'
            application/json:
              schema:
                type: array
                description: '`format=json` — one object per record.'
                items:
                  type: object
            text/markdown:
              schema:
                type: string
                description: '`format=markdown` — content-aware documents or a table.'
            text/html:
              schema:
                type: string
                description: '`format=html` — the markdown render as safe HTML.'
        '400':
          $ref: '#/components/responses/BadRequest'
        '401':
          $ref: '#/components/responses/Unauthorized'
        '404':
          $ref: '#/components/responses/NotFound'
  /v1/datasets/{id}/search:
    parameters:
    - $ref: '#/components/parameters/DatasetId'
    get:
      operationId: searchDataset
      tags:
      - Datasets
      summary: Full-text search within one dataset
      description: 'The same full-text find as `GET /v1/datasets/search` — same terms semantics, same ranking, same result shape — scoped to this dataset''s id.


        On a cloud-linked daemon a crawl dataset absent locally forwards the query to the fleet''s data table; that forwarded body is the cloud table shape, and `offset` is not forwarded.'
      parameters:
      - $ref: '#/components/parameters/DatasetSearchQuery'
      - $ref: '#/components/parameters/DatasetSearchLimit'
      - $ref: '#/components/parameters/DatasetSearchOffset'
      - $ref: '#/components/parameters/DatasetFormat'
      responses:
        '200':
          $ref: '#/components/responses/DatasetSearchResponse'
        '400':
          $ref: '#/components/responses/BadRequest'
        '401':
          $ref: '#/components/responses/Unauthorized'
        '404':
          $ref: '#/components/responses/NotFound'
components:
  parameters:
    DatasetSearchLimit:
      name: limit
      in: query
      description: Page size over the ranked hits (clamped to 1..=200; default 50).
      schema:
        type: integer
        default: 50
    DatasetIncludeInputs:
      name: include_inputs
      in: query
      description: Also surface run input values as `input.<name>` columns.
      schema:
        type: boolean
        default: false
    DatasetQ:
      name: q
      in: query
      description: Global substring filter across extracted fields (and inputs, when included).
      schema:
        type: string
    DatasetSearchQuery:
      name: q
      in: query
      required: true
      description: Space-separated keywords — ANDed, case-insensitive, prefix-matched; a record must contain EVERY term. A query with no usable terms returns an empty result set rather than an error.
      schema:
        type: string
    DatasetSearchOffset:
      name: offset
      in: query
      schema:
        type: integer
        default: 0
    DatasetExportFormat:
      name: format
      in: query
      description: Output shape, defaulting to `csv` on this route (every other dataset read defaults to `json`). `csv`/`json` return the tabular records; `markdown`/`html` render them and are CONTENT-AWARE — a crawl's pages export as readable documents, structured data as a table. An unknown value is a 400.
      schema:
        type: string
        enum:
        - csv
        - json
        - markdown
        - html
        default: csv
    DatasetSortBy:
      name: sort_by
      in: query
      description: A data column, `input.<name>`, or `run_at` / `status` / `duration_ms` / `run_id`.
      schema:
        type: string
    DatasetCollection:
      name: collection
      in: query
      description: Pivot to a nested array path (e.g. `posts.items`), one row per item. ACCEPTED BUT IGNORED by the daemon — nested collections are scoped client-side by the desktop app, and only the cloud `/api/v1/datasets/*` twin pivots server-side. Passing it here changes nothing rather than erroring.
      schema:
        type: string
    DatasetFilters:
      name: filters
      in: query
      description: Structured smart filters — a JSON array of clauses, e.g. `[{"col":"price","op":"between","min":10,"max":50}]`.
      schema:
        type: string
    DatasetId:
      name: id
      in: path
      required: true
      description: Dataset id — the same id as the workflow/crawl that produced the data (`GET /v1/datasets` lists them).
      schema:
        type: integer
        format: int64
    DatasetFormat:
      name: format
      in: query
      description: 'Output shape. `json` (the default) returns the documented JSON envelope; ANY other value returns rendered TEXT instead — `csv` as RFC-4180 CSV, and `markdown`/`html` CONTENT-AWARE: a dataset whose records carry a long-form content column (a crawl''s pages carry `markdown`) renders as DOCUMENTS — one section per record, heading + source link + body — while anything else renders as a TABLE mirroring the CSV columns. An unknown value is a 400, never a silent fall back to JSON.'
      schema:
        type: string
        enum:
        - json
        - csv
        - markdown
        - html
        default: json
    DatasetFilter:
      name: filter
      in: query
      description: Per-column filter `column:substring` (repeatable).
      schema:
        type: array
        items:
          type: string
      explode: true
    DatasetSortDir:
      name: sort_dir
      in: query
      schema:
        type: string
        enum:
        - asc
        - desc
        default: desc
  schemas:
    Dataset:
      type: object
      description: One entry in the unified dataset catalogue.
      required:
      - id
      - name
      - source_type
      - run_count
      - last_updated
      properties:
        id:
          type: integer
          format: int64
          description: The id of the workflow/crawl that produced the data.
        name:
          type:
          - string
          - 'null'
        source_type:
          type: string
          description: '`crawl` for a whole-site crawl dataset, otherwise `workflow`.'
          enum:
          - crawl
          - workflow
        run_count:
          type: integer
          description: Distinct successful runs that contributed at least one record.
        last_updated:
          type:
          - string
          - 'null'
          description: Newest contributing run's timestamp.
        origin:
          type: string
          description: Where the dataset lives — present on a cloud-linked daemon, which merges forwarded cloud crawl datasets into this list.
    DatasetRecords:
      type: object
      description: '`GET /v1/datasets/{id}/records` with `format=json` — a page of the dataset''s records. Reuses the workflow-data row shape.'
      required:
      - dataset
      - columns
      - declared
      - records
      - total
      - scanned_runs
      - truncated
      - limit
      - offset
      properties:
        dataset:
          $ref: '#/components/schemas/DatasetRef'
        columns:
          type: array
          items:
            type: string
        declared:
          type: boolean
          description: The columns come from the workflow's declared output fields.
        records:
          type: array
          items:
            $ref: '#/components/schemas/DataTableRow'
        total:
          type:
          - integer
          - 'null'
          description: Matching records before pagination.
        scanned_runs:
          type:
          - integer
          - 'null'
        truncated:
          type: boolean
        limit:
          type:
          - integer
          - 'null'
        offset:
          type:
          - integer
          - 'null'
    DataTableRow:
      type: object
      description: 'One table row: run-meta at the top level, data columns nested under `fields`. Lens views (`view=latest|run`) add a `lineage` sibling (`uid`, `change`, `changed_fields` — present only when change is `changed` —, `changed_leaf_count`, `first_seen_at`, `last_seen_at`, `versions`, `source`, and `prev_run_id` on run views).'
      required:
      - run_id
      - run_at
      - status
      - record_index
      - fields
      properties:
        run_id:
          type: integer
          format: int64
        run_at:
          type:
          - string
          - 'null'
        status:
          type:
          - string
          - 'null'
        record_index:
          type: integer
        fields:
          type: object
          description: The extracted data columns (redacted; a column absent for a row is null).
          additionalProperties: true
        lineage:
          type: object
          description: Lens views only.
          additionalProperties: true
    DatasetSearchResult:
      type: object
      description: '`GET /v1/datasets/search` and `GET /v1/datasets/{id}/search` with `format=json`.'
      required:
      - query
      - terms
      - results
      - total
      - truncated
      - scanned_runs
      properties:
        query:
          type: string
          description: The query as received (trimmed).
        terms:
          type: array
          description: The parsed terms — ANDed, case-insensitive, prefix-matched.
          items:
            type: string
        results:
          type: array
          description: Matched records, newest-first.
          items:
            $ref: '#/components/schemas/DatasetSearchHit'
        total:
          type: integer
          description: Matching records before pagination.
        truncated:
          type: boolean
          description: The candidate-run cap (500) was hit, so older matches may be missing.
        scanned_runs:
          type:
          - integer
          - 'null'
    DatasetSearchHit:
      type: object
      description: One matched record, tagged with the dataset it came from.
      required:
      - dataset
      - run_id
      - run_at
      - fields
      - highlight
      properties:
        dataset:
          $ref: '#/components/schemas/DatasetRef'
        run_id:
          type:
          - integer
          - 'null'
          format: int64
        run_at:
          type:
          - string
          - 'null'
        fields:
          type: object
          description: The matched record's extracted columns.
          additionalProperties: true
        highlight:
          description: A snippet around the first matching term, or null when none could be built (a forwarded cloud search never carries one).
          oneOf:
          - type: object
            required:
            - field
            - snippet
            properties:
              field:
                type: string
                description: The column the snippet was taken from.
              snippet:
                type: string
          - type: 'null'
    DatasetRef:
      type: object
      description: The dataset a record page or search hit belongs to.
      required:
      - id
      - name
      - source_type
      properties:
        id:
          type:
          - integer
          - 'null'
          format: int64
        name:
          type:
          - string
          - 'null'
        source_type:
          type: string
          enum:
          - crawl
          - workflow
    DatasetMeta:
      type: object
      description: A dataset's metadata + inferred schema. A cloud crawl dataset forwarded by a linked daemon returns the cloud facets shape, which additionally carries `collections`.
      required:
      - id
      - name
      - source_type
      - columns
      - facets
      - row_count
      - run_count
      - truncated
      properties:
        id:
          type: integer
          format: int64
        name:
          type: string
        source_type:
          type: string
          enum:
          - crawl
          - workflow
        columns:
          type: array
          items:
            type: string
        facets:
          type: object
          description: Per-column facet descriptors (inferred type, non-empty count, numeric min/max, distinct pick-list when low-cardinality).
          additionalProperties: true
        collections:
          type: array
          description: Nested array paths (e.g. `posts.items`) you can pivot to with `collection`. Only on a FORWARDED cloud dataset — a local dataset omits this key.
          items:
            type: string
        row_count:
          type: integer
        run_count:
          type: integer
          description: Runs actually scanned to build this schema.
        truncated:
          type: boolean
          description: The run-scan cap bounded the answer.
    Error:
      type: object
      description: 'The daemon''s domain error body. NOTE: some 4xx responses (axum framework rejections) are `text/plain` instead of this JSON shape. Certain codes attach extra fields (e.g. `device_capacity` adds `would_use_units_per_min` / `budget_units_per_min`; `captcha_required` adds `captcha_type` / `url`).'
      required:
      - error
      - code
      properties:
        error:
          type: string
          description: 'Human-readable message (e.g. `not found: workflow 999999`).'
        code:
          type: string
          description: 'Stable machine code: `bad_request`, `unauthorized`, `forbidden`, `not_found`, `vault_locked`, `too_many_requests`, `captcha_required`, `device_capacity`, `cloud_unreachable`, `db_error`, `io_error`, `json_error`, `internal`, …'
      additionalProperties: true
    DatasetList:
      type: object
      description: '`GET /v1/datasets` — a bespoke `{datasets}` envelope, NOT `{data, count}`.'
      required:
      - datasets
      properties:
        datasets:
          type: array
          items:
            $ref: '#/components/schemas/Dataset'
  responses:
    NotFound:
      description: The addressed resource does not exist.
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/Error'
    Unauthorized:
      description: Missing/invalid bearer token, or the Origin/Host guard rejected the request.
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/Error'
    DatasetSearchResponse:
      description: The matches — the JSON envelope for `format=json` (the default), or the matches rendered as TEXT for any other format. The per-result `dataset` tag and `highlight` snippet exist only in the JSON shape.
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/DatasetSearchResult'
        text/csv:
          schema:
            type: string
            description: '`format=csv` — RFC-4180 CSV of the matched records.'
        text/markdown:
          schema:
            type: string
            description: '`format=markdown` — content-aware: matched documents render as documents, structured matches as a table.'
        text/html:
          schema:
            type: string
            description: '`format=html` — the markdown render as safe HTML.'
    BadRequest:
      description: 'Invalid input. Domain validation errors use the JSON Error shape; axum body-deserialize rejections are `text/plain` with a bare message (e.g. `Failed to deserialize the JSON body into the target type: missing field ''url''`).'
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/Error'
        text/plain:
          schema:
            type: string
  securitySchemes:
    bearerAuth:
      type: http
      scheme: bearer
      description: "Every route (including `/v1/agent` and `/v1/health`) requires\n`Authorization: Bearer <token>`. Token families:\n\n- `wlt_…` — the runtime token from `runtime.json` (full access, used by the desktop\n  UI and SDK discovery).\n- `wlk_…` — a scoped API key minted via `POST /v1/keys` (CSV scopes `read|run|admin`;\n  `/v1/keys` itself requires the full-access token).\n- `wlo_…` — an OAuth 2.1 access token from the daemon's local authorization server\n  (`run` scope).\n\nTreat the token as an opaque string. The daemon additionally enforces a loopback\nOrigin/Host DNS-rebind guard.\n"