Prometheus Push Batch Job Metrics to the Pushgateway

arazzo: 1.0.1
info:
  title: Prometheus Push Batch Job Metrics to the Pushgateway
  summary: Publish a short-lived job's results so Prometheus can scrape them after the job exits.
  description: >-
    Batch jobs finish and disappear long before Prometheus comes round to scrape
    them, so they push their results to a Pushgateway that holds the metrics on
    their behalf. The workflow confirms the Pushgateway is up, atomically
    replaces the job's metric group with the run's results, adds the completion
    timestamp as a follow-up push, and reads the stored groups back to prove the
    metrics are there for Prometheus to collect. Every step spells out its
    request inline so the flow can be read and executed without opening the
    underlying OpenAPI description.
  version: 1.0.0
sourceDescriptions:
- name: lifecycleApi
  url: ../openapi/prometheus-lifecycle-api-openapi.yml
  type: openapi
- name: metricsApi
  url: ../openapi/prometheus-metrics-api-openapi.yml
  type: openapi
- name: statusApi
  url: ../openapi/prometheus-status-api-openapi.yml
  type: openapi
workflows:
- workflowId: push-batch-job-metrics
  summary: Push a completed batch job's metrics and verify they are stored.
  description: >-
    PUT and POST differ in a way that matters here. The PUT replaces the whole
    group atomically, clearing any metric left behind by a previous run that
    this run did not emit. The POST then adds to that group without disturbing
    what the PUT just wrote.
  inputs:
    type: object
    required:
    - jobName
    - metricsPayload
    - completionPayload
    properties:
      jobName:
        type: string
        description: >-
          The job label for this metric group, and the primary grouping key
          (e.g. batch-importer).
      metricsPayload:
        type: string
        description: >-
          The run's metrics in Prometheus text exposition format, including HELP
          and TYPE lines. Must end with a trailing newline.
      completionPayload:
        type: string
        description: >-
          A follow-up metric in text exposition format, typically the
          last-success timestamp used to alert on jobs that stop running.
  steps:
  - stepId: checkPushgatewayStatus
    description: >-
      Read the Pushgateway build information. Recording the version alongside a
      push is worth the one request, because text-format and OpenMetrics
      handling has changed across Pushgateway releases.
    operationId: getStatus
    successCriteria:
    - condition: $statusCode == 200
    - condition: $response.body#/status == "success"
    outputs:
      version: $response.body#/data/version
      revision: $response.body#/data/revision
  - stepId: confirmReady
    description: >-
      Confirm the Pushgateway is ready to accept pushes before the job discards
      its in-memory results.
    operationId: checkReady
    successCriteria:
    - condition: $statusCode == 200
  - stepId: replaceJobMetrics
    description: >-
      Atomically replace every metric for this job with the results of this run.
      Using PUT rather than POST is what stops a metric emitted by yesterday's
      run, and not by today's, from lingering forever and being scraped as if it
      were current.
    operationId: replaceJobMetrics
    parameters:
    - name: job
      in: path
      value: $inputs.jobName
    requestBody:
      contentType: text/plain
      payload: $inputs.metricsPayload
    successCriteria:
    - condition: $statusCode == 200
  - stepId: addCompletionMetric
    description: >-
      Add the completion metric to the group with POST, which leaves the metrics
      written by the previous step untouched.
    operationId: pushJobMetrics
    parameters:
    - name: job
      in: path
      value: $inputs.jobName
    requestBody:
      contentType: text/plain
      payload: $inputs.completionPayload
    successCriteria:
    - condition: $statusCode == 200
  - stepId: verifyMetricsStored
    description: >-
      Read back every stored metric group and confirm this job's group is
      present with a fresh lastPushedAt. This is the only positive confirmation
      the job gets that its telemetry survived its own exit.
    operationId: listAllMetrics
    successCriteria:
    - condition: $statusCode == 200
    - condition: $response.body#/status == "success"
    outputs:
      metricGroups: $response.body#/data
      firstGroupLabels: $response.body#/data/0/labels
      firstGroupLastPushedAt: $response.body#/data/0/lastPushedAt
      firstGroupMetricFamilies: $response.body#/data/0/metricFamilies
  outputs:
    pushgatewayVersion: $steps.checkPushgatewayStatus.outputs.version
    metricGroups: $steps.verifyMetricsStored.outputs.metricGroups
    lastPushedAt: $steps.verifyMetricsStored.outputs.firstGroupLastPushedAt