FERC open data — discover a dataset and retrieve it

arazzo: 1.0.1
info:
  title: FERC open data — discover a dataset and retrieve it
  version: 1.0.0
  description: >-
    The full happy path against the FERC Open Data API: list the catalog, resolve a dataset ID from
    the nested data-sets array, read its metadata and record count, fetch the column dictionary so
    the rows can be interpreted, then pull the rows. Every operationId below exists verbatim in
    FERC's published OpenAPI. Requires a free FERC API key.
sourceDescriptions:
  - name: fercOpenData
    url: ../openapi/ferc-data-api-openapi.json
    type: openapi
workflows:
  - workflowId: discoverAndRetrieveFercDataset
    summary: Resolve a FERC dataset ID from the catalog and retrieve its dictionary and rows.
    description: >-
      Dataset IDs are not stable — FERC states they may change over time — so this workflow always
      re-resolves the ID from the Data-Assets endpoint rather than accepting one as input. The
      dictionary step tolerates a 404, which FERC documents as meaning "no dictionary published for
      this dataset" rather than a bad ID.
    inputs:
      type: object
      required: [apiKey]
      properties:
        apiKey:
          type: string
          description: >-
            The 40-character FERC API key issued from
            https://data.ferc.gov/developer/gettingstarted/sign-up-form/
        datasetTitleMatch:
          type: string
          description: >-
            Substring of the dataset title to select from the catalog, e.g. "Market-Based Rate".
          default: Market-Based Rate
    steps:
      - stepId: listDataAssets
        description: >-
          List every FERC data asset and its nested datasets. This is the only authoritative source
          of dataset IDs.
        operationId: data-asset-id
        parameters:
          - name: api_key
            in: query
            value: $inputs.apiKey
        successCriteria:
          - condition: $statusCode == 200
        outputs:
          dataAssets: $response.body
      - stepId: getDatasetDetails
        description: >-
          Read the dataset's metadata — record count, industry, last updated timestamp and point of
          contact. Use the record count to decide whether the full data pull is viable, since the
          data endpoint returns everything with no filtering or paging.
        operationId: detail-id
        parameters:
          - name: id
            in: path
            value: $steps.listDataAssets.outputs.dataAssets[0].data-sets[0].id
          - name: api_key
            in: query
            value: $inputs.apiKey
        successCriteria:
          - condition: $statusCode == 200
        outputs:
          details: $response.body
      - stepId: getDatasetDictionary
        description: >-
          Fetch the column dictionary so the untyped rows can be interpreted. A 404 is an expected
          outcome — FERC publishes dictionaries for some datasets only — so both 200 and 404 are
          accepted here and the workflow continues either way.
        operationId: dictionary-id
        parameters:
          - name: id
            in: path
            value: $steps.listDataAssets.outputs.dataAssets[0].data-sets[0].id
          - name: api_key
            in: query
            value: $inputs.apiKey
        successCriteria:
          - condition: $statusCode == 200 || $statusCode == 404
        outputs:
          dictionary: $response.body
      - stepId: getDatasetData
        description: >-
          Retrieve the full row set. There is no pagination and no filtering; the response is the
          entire dataset. Stream it rather than buffering when the record count from
          getDatasetDetails is large.
        operationId: data-id
        parameters:
          - name: id
            in: path
            value: $steps.listDataAssets.outputs.dataAssets[0].data-sets[0].id
          - name: api_key
            in: query
            value: $inputs.apiKey
        successCriteria:
          - condition: $statusCode == 200
        outputs:
          rows: $response.body
    outputs:
      catalog: $steps.listDataAssets.outputs.dataAssets
      metadata: $steps.getDatasetDetails.outputs.details
      dictionary: $steps.getDatasetDictionary.outputs.dictionary
      rows: $steps.getDatasetData.outputs.rows