FERC open data — discover a dataset and retrieve it

arazzo: 1.0.1
info:
  title: FERC open data — discover a dataset and retrieve it
  version: 1.0.0
  description: >-
    The full happy path against the FERC Open Data API: list the catalog, resolve a dataset ID from
    the nested data-sets array, read its metadata and record count, fetch the column dictionary so
    the rows can be interpreted, then pull the rows. Every operationId below exists verbatim in
    FERC's published OpenAPI. Requires a free FERC API key.
sourceDescriptions:
- name: dataApi
  url: ../openapi/ferc-data-api-openapi.yml
  type: openapi
- name: dataAssetsApi
  url: ../openapi/ferc-data-assets-api-openapi.yml
  type: openapi
- name: detailsApi
  url: ../openapi/ferc-details-api-openapi.yml
  type: openapi
- name: dictionaryApi
  url: ../openapi/ferc-dictionary-api-openapi.yml
  type: openapi
workflows:
- workflowId: discoverAndRetrieveFercDataset
  summary: Resolve a FERC dataset ID from the catalog and retrieve its dictionary and rows.
  description: >-
    Dataset IDs are not stable — FERC states they may change over time — so this workflow always
    re-resolves the ID from the Data-Assets endpoint rather than accepting one as input. The
    dictionary step tolerates a 404, which FERC documents as meaning "no dictionary published for
    this dataset" rather than a bad ID.
  inputs:
    type: object
    required: [apiKey]
    properties:
      apiKey:
        type: string
        description: >-
          The 40-character FERC API key issued from
          https://data.ferc.gov/developer/gettingstarted/sign-up-form/
      datasetTitleMatch:
        type: string
        description: >-
          Substring of the dataset title to select from the catalog, e.g. "Market-Based Rate".
        default: Market-Based Rate
  steps:
  - stepId: listDataAssets
    description: >-
      List every FERC data asset and its nested datasets. This is the only authoritative source
      of dataset IDs.
    operationId: data-asset-id
    parameters:
    - name: api_key
      in: query
      value: $inputs.apiKey
    successCriteria:
    - condition: $statusCode == 200
    outputs:
      dataAssets: $response.body
  - stepId: getDatasetDetails
    description: >-
      Read the dataset's metadata — record count, industry, last updated timestamp and point of
      contact. Use the record count to decide whether the full data pull is viable, since the
      data endpoint returns everything with no filtering or paging.
    operationId: detail-id
    parameters:
    - name: id
      in: path
      value: $steps.listDataAssets.outputs.dataAssets[0].data-sets[0].id
    - name: api_key
      in: query
      value: $inputs.apiKey
    successCriteria:
    - condition: $statusCode == 200
    outputs:
      details: $response.body
  - stepId: getDatasetDictionary
    description: >-
      Fetch the column dictionary so the untyped rows can be interpreted. A 404 is an expected
      outcome — FERC publishes dictionaries for some datasets only — so both 200 and 404 are
      accepted here and the workflow continues either way.
    operationId: dictionary-id
    parameters:
    - name: id
      in: path
      value: $steps.listDataAssets.outputs.dataAssets[0].data-sets[0].id
    - name: api_key
      in: query
      value: $inputs.apiKey
    successCriteria:
    - condition: $statusCode == 200 || $statusCode == 404
    outputs:
      dictionary: $response.body
  - stepId: getDatasetData
    description: >-
      Retrieve the full row set. There is no pagination and no filtering; the response is the
      entire dataset. Stream it rather than buffering when the record count from
      getDatasetDetails is large.
    operationId: data-id
    parameters:
    - name: id
      in: path
      value: $steps.listDataAssets.outputs.dataAssets[0].data-sets[0].id
    - name: api_key
      in: query
      value: $inputs.apiKey
    successCriteria:
    - condition: $statusCode == 200
    outputs:
      rows: $response.body
  outputs:
    catalog: $steps.listDataAssets.outputs.dataAssets
    metadata: $steps.getDatasetDetails.outputs.details
    dictionary: $steps.getDatasetDictionary.outputs.dictionary
    rows: $steps.getDatasetData.outputs.rows