Trainings are dedicated workloads that are specifically used for training models. They are by design preemptible workloads because they are used in unattended sessions where the scientists and researchers do not need to be present. These workloads are allowed to utilize resources over the project's quota.
openapi: 3.0.3
info:
version: latest
description: '# Introduction
The NVIDIA Run:ai Control-Plane API reference is a guide that provides an easy-to-use programming interface for adding various tasks to your application, including workload submission, resource management, and administrative operations.
NVIDIA Run:ai APIs are accessed using *bearer tokens*. To obtain a token, you need to create a **Service account** through the NVIDIA Run:ai user interface.
To create a service account, in your UI, go to Access → Service Accounts (for organization-level service accounts) or User settings → Access Keys (for user access keys), and create a new one.
After you have created a new service account, you will need to assign it access rules.
To assign access rules to the service account, see [Create access rules](https://run-ai-docs.nvidia.com/saas/infrastructure-setup/authentication/accessrules#create-or-delete-rules).
Make sure you assign the correct rules to your service account. Use the [Roles](https://run-ai-docs.nvidia.com/saas/infrastructure-setup/authentication/roles) to assign the correct access rules.
To get your access token, follow the instructions in [Request a token](https://run-ai-docs.nvidia.com/saas/reference/api/rest-auth/#request-an-api-token).
'
title: NVIDIA Run:ai Access Keys Trainings API
x-logo:
url: https://api.redocly.com/registry/raw/runai-xq8/saas/latest/public/runai-logo-api.png
altText: NVIDIA Run:ai
href: https://run.ai
license:
name: NVIDIA Run:ai
url: https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-software-license-agreement/
servers:
- url: https://app.run.ai
security:
- bearerAuth: []
tags:
- name: Trainings
description: "Trainings are dedicated workloads that are specifically used for training models. \nThey are by design preemptible workloads because they are used in unattended sessions where the \nscientists and researchers do not need to be present. These workloads are allowed to utilize \nresources over the project's quota. \n"
paths:
/api/v1/workloads/trainings:
post:
summary: Create a training.
operationId: create_training1
description: Create a training workload using container related fields.
tags:
- Trainings
requestBody:
content:
application/json:
schema:
$ref: '#/components/schemas/TrainingCreationRequest'
responses:
'202':
description: Request completed successfully.
content:
application/json:
schema:
$ref: '#/components/schemas/Training1'
'400':
$ref: '#/components/responses/400SubmissionErrorV2'
'401':
$ref: '#/components/responses/401Unauthorized'
'403':
$ref: '#/components/responses/403Forbidden'
'503':
$ref: '#/components/responses/503ServiceUnavailable'
/api/v1/workloads/trainings/{workloadId}:
delete:
summary: Delete a training.
operationId: delete_training
description: Delete a training using a workload id.
tags:
- Trainings
parameters:
- $ref: '#/components/parameters/WorkloadId'
responses:
'202':
$ref: '#/components/responses/202Accepted'
'401':
$ref: '#/components/responses/401Unauthorized'
'403':
$ref: '#/components/responses/403Forbidden'
'404':
$ref: '#/components/responses/404NotFound'
'500':
$ref: '#/components/responses/500InternalServerError'
'503':
$ref: '#/components/responses/503ServiceUnavailable'
get:
summary: Get training data.
operationId: get_training
description: Retrieve training details using a workload id.
tags:
- Trainings
parameters:
- $ref: '#/components/parameters/WorkloadId'
responses:
'200':
description: Executed successfully.
content:
application/json:
schema:
$ref: '#/components/schemas/Training1'
'401':
$ref: '#/components/responses/401Unauthorized'
'403':
$ref: '#/components/responses/403Forbidden'
'404':
$ref: '#/components/responses/404NotFound'
'500':
$ref: '#/components/responses/500InternalServerError'
'503':
$ref: '#/components/responses/503ServiceUnavailable'
/api/v1/workloads/trainings/{workloadId}/suspend:
post:
summary: Suspend a training.
operationId: suspend_training
description: Suspend a training from running using a workload id.
tags:
- Trainings
parameters:
- $ref: '#/components/parameters/WorkloadId'
responses:
'202':
$ref: '#/components/responses/202Accepted'
'401':
$ref: '#/components/responses/401Unauthorized'
'403':
$ref: '#/components/responses/403Forbidden'
'404':
$ref: '#/components/responses/404NotFound'
'500':
$ref: '#/components/responses/500InternalServerError'
'503':
$ref: '#/components/responses/503ServiceUnavailable'
/api/v1/workloads/trainings/{workloadId}/resume:
post:
summary: Resume a training.
operationId: resume_training
description: Resume a training that was suspended using a workload id.
tags:
- Trainings
parameters:
- $ref: '#/components/parameters/WorkloadId'
responses:
'202':
$ref: '#/components/responses/202Accepted'
'401':
$ref: '#/components/responses/401Unauthorized'
'403':
$ref: '#/components/responses/403Forbidden'
'404':
$ref: '#/components/responses/404NotFound'
'500':
$ref: '#/components/responses/500InternalServerError'
'503':
$ref: '#/components/responses/503ServiceUnavailable'
components:
schemas:
S3Common:
properties:
bucket:
description: The name of the bucket. (mandatory)
type: string
minLength: 1
example: my-bucket
nullable: true
path:
description: Local path within the workload to which the S3 bucket will be mapped. (mandatory)
type: string
minLength: 1
example: /container/my-bucket
nullable: true
url:
description: The url of the S3 service provider. The default is the URL of the Amazon AWS S3 service.
type: string
minLength: 1
example: https://s3.amazonaws.com
nullable: true
nullable: true
type: object
Category:
description: Specify the workload category assigned to the workload. Categories are used to classify and monitor different types of workloads within the NVIDIA Run:ai platform.
type: string
nullable: true
pattern: .*
PvcFieldsNonUpdatable:
properties:
existingPvc:
description: Verify existing PVC. PVC is assumed to exist when set to `true`. If set to `false`, the PVC will be created, if it does not exist.
type: boolean
default: false
nullable: true
claimName:
description: Name for the PVC. Allow referencing it across workloads. If not provided, a name based on the workload name and scope will be auto-generated.
type: string
minLength: 1
maxLength: 63
example: my-claim
nullable: true
pattern: .*
readOnly:
description: Permit only read access to PVC.
type: boolean
default: false
nullable: true
ephemeral:
description: Use `true` to set PVC to ephemeral. If set to `true`, the PVC will be deleted when the workload is stopped. Not supported for inference workloads.
type: boolean
default: false
example: false
nullable: true
claimInfo:
$ref: '#/components/schemas/ClaimInfo'
dataSharing:
description: use `true` to share the PVC data to all projects under the selected scope.
type: boolean
default: false
example: false
nullable: true
nullable: true
type: object
Probe:
type: object
properties:
initialDelaySeconds:
description: Number of seconds after the container has started before liveness or readiness probes are initiated.
type: integer
format: int32
minimum: 0
nullable: true
periodSeconds:
description: How often (in seconds) to perform the probe.
type: integer
format: int32
minimum: 1
nullable: true
timeoutSeconds:
description: Number of seconds after which the probe times out.
type: integer
format: int32
minimum: 1
nullable: true
successThreshold:
description: Minimum consecutive successes for the probe to be considered successful after having failed.
type: integer
format: int32
minimum: 1
nullable: true
failureThreshold:
description: When a probe fails, the number of times to try before giving up.
type: integer
format: int32
minimum: 1
nullable: true
handler:
$ref: '#/components/schemas/ProbeHandler'
nullable: true
ExposedUrl:
description: A URL for accessing the workload.
properties:
container:
description: The port that the container running the workload exposes. (mandatory)
type: integer
format: int32
example: 8080
nullable: true
url:
description: The URL for connecting to the container port. If not specified, the URL will be auto-generated by the system..
type: string
pattern: .*
example: https://my-url.com
nullable: true
authorizationType:
$ref: '#/components/schemas/AuthorizationType'
authorizedUsers:
description: List of users or service accounts that are allowed to access the URL. Note that authorizedUsers and authorizedGroups are mutually exclusive.
type: array
items:
type: string
pattern: .*
example:
- user-a
- user-b
nullable: true
authorizedGroups:
description: List of groups that are allowed to access the URL. Note that authorizedUsers and authorizedGroups are mutually exclusive.
type: array
items:
type: string
pattern: .*
example:
- group-a
- group-b
nullable: true
toolType:
description: The tool type that runs on this container port.
type: string
example: jupyter
nullable: true
pattern: .*
toolName:
description: A name describing the tool that runs on this url.
type: string
example: my-pytorch
nullable: true
pattern: .*
name:
description: Unique name to identify the instance. primarily used for policy locked rules.
type: string
minLength: 1
example: url-instance-a
nullable: true
pattern: .*
exclude:
description: Use 'true' in case the instance is defined in defaults of the policy, and you wish to exclude it from the workload.
type: boolean
example: false
nullable: true
nullable: true
type: object
WorkloadId2:
description: A unique ID of the workload.
type: string
format: uuid
NodeAffinityRequired:
type: object
description: If the affinity requirements specified by this field are not met at scheduling time, the pod will not be scheduled onto the node. If the affinity requirements specified by this field cease to be met at some point during pod execution (e.g. due to an update), the system may or may not try to eventually evict the pod from its node.
properties:
nodeSelectorTerms:
description: A list of node selector terms. The terms are ORed.
type: array
items:
$ref: '#/components/schemas/NodeSelectorTerm'
nullable: true
UidGidSource:
description: Indicate the way to determine the user and group ids of the container. The options are a. `fromTheImage` - user and group ids are determined by the docker image that the container runs. this is the default option. b. `custom` - user and group ids can be specified in the environment asset and/or the workload creation request. c. `idpToken` - user and group IDs are automatically taken from the identity provider (IdP) token (available only in SSO-enabled installations). For more information, see [User Identity](https://docs.run.ai/latest/admin/runai-setup/config/non-root-containers/).
type: string
enum:
- fromTheImage
- fromIdpToken
- custom
nullable: true
SecretInstance2:
allOf:
- $ref: '#/components/schemas/StorageInstanceName'
- $ref: '#/components/schemas/Secret5'
- $ref: '#/components/schemas/ExcludeField'
nullable: true
type: object
Labels:
description: Set of labels to populate into the container running the workload.
type: array
items:
$ref: '#/components/schemas/Label'
nullable: true
Error:
required:
- code
- message
properties:
code:
type: integer
minimum: 100
maximum: 599
message:
type: string
details:
type: string
example:
code: 400
message: Bad request - Resource should have a name
ExcludeField:
properties:
exclude:
description: Use 'true' in case the item is defined in defaults of the policy, and you wish to exclude it from the workload.
type: boolean
default: false
example: false
nullable: true
type: object
nullable: true
TerminationGracePeriod:
description: Duration in seconds the pod needs to terminate gracefully upon probe failure. The grace period is the duration in seconds after the processes running in the pod are sent a termination signal and the time when the processes are forcibly halted with a kill signal. Set this value longer than the expected cleanup time for your process. Value must be non-negative integer. The value zero indicates stop immediately via the kill signal (no opportunity to shut down).
format: int64
type: integer
minimum: 0
example: 20
nullable: true
CpuMemoryLimit:
description: Limitations on the CPU memory to allocate for this workload (1G, 20M, .etc). The system guarantees that this workload will not be able to consume more than this amount of memory. The workload will receive an error when trying to allocate more memory than this limit.
type: string
pattern: ^([+]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$
example: 30M
nullable: true
GitInstance:
allOf:
- $ref: '#/components/schemas/StorageInstanceName'
- $ref: '#/components/schemas/GitCommon'
- $ref: '#/components/schemas/GitPassword'
- $ref: '#/components/schemas/ExcludeField'
- type: object
properties:
secretRef:
$ref: '#/components/schemas/GitSecretRef'
nullable: true
nullable: true
type: object
PvcAddedAttrValues:
description: an optional array of key-values pairs that are written as annotations on the created PVC. the allowed attributes are determined according to the storage class configuration (see k8s-objects-tracker for further info).
type: array
items:
$ref: '#/components/schemas/PvcAddedAttrValue'
PvcClaimSize:
description: 'Requested size for the PVC. Mandatory when existingPvc is false. Recommended sizes: TB/GB/MB/TIB/GIB/MIB'
type: string
pattern: ^([+]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$
example: 1G
nullable: true
Ports:
description: Set of container ports that the workload exposes.
type: array
items:
$ref: '#/components/schemas/Port'
nullable: true
Toleration:
description: Toleration details.
properties:
name:
description: The name of the toleration.
type: string
minLength: 1
nullable: true
pattern: .*
operator:
$ref: '#/components/schemas/TolerationOperator'
key:
description: The taint key that the toleration applies to. (mandatory)
type: string
nullable: true
pattern: .*
value:
description: The taint value the toleration matches to. Mandatory if operator is Exists, forbidden otherwise.
type: string
nullable: true
pattern: .*
effect:
$ref: '#/components/schemas/TolerationEffect'
seconds:
description: The period of time the toleration tolerates the taint. Valid only if effect is NoExecute. taint.
type: integer
minimum: 1
nullable: true
exclude:
description: Use 'true' in case the label is defined in defaults of the policy, and you wish to exclude it from the workload.
type: boolean
example: false
nullable: true
nullable: true
type: object
ClusterId:
description: The id of the cluster.
type: string
format: uuid
example: 71f69d83-ba66-4822-adf5-55ce55efd210
S3AccessKey:
properties:
accessKeySecret:
description: Name of the secret containing credentials of the S3 bucket. Used for private S3 buckets.
type: string
minLength: 1
example: my-access-key-secret
nullable: true
pattern: ^[a-z0-9]([a-z0-9-]*[a-z0-9])?$
secretKeyOfAccessKeyId:
description: The key to use for loading the access key id from the secret. The default is `AccessKeyId`. For more information, see [Credentials access keys](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
type: string
minLength: 1
example: AccessKeyId
nullable: true
pattern: .*
secretKeyOfSecretKey:
description: The key to use for loading the secret key from the secret. The default is `SecretKey`. For more information, see [Credentials access keys](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
type: string
minLength: 1
example: SecretKey
nullable: true
pattern: .*
nullable: true
type: object
SecretFieldsNonUpdatable:
properties:
secret:
description: The name of the Secret resource. (mandatory)
type: string
minLength: 1
nullable: true
nullable: true
type: object
Pvc:
allOf:
- $ref: '#/components/schemas/PvcFieldsUpdatable'
- $ref: '#/components/schemas/PvcFieldsNonUpdatable'
Stdin:
description: Whether this container should allocate a buffer for stdin in the container runtime. If this is not set, reads from stdin in the container will always result in EOF
type: boolean
nullable: true
Capabilities:
description: Add POSIX capabilities to running containers. Defaults to the default set of capabilities granted by the container runtime.
type: array
items:
$ref: '#/components/schemas/Capability'
example:
- CHOWN
- KILL
nullable: true
GpuMemoryRequest:
description: Required if and only if gpuRequestType is memory. States the GPU memory to allocate for the created workload, per GPU device. Note that the workload will not be scheduled unless the system can guarantee this amount of GPU memory to the workload.
type: string
pattern: ^([+]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$
example: 10M
nullable: true
HostPath:
properties:
path:
description: Local path within the controller to which the host volume will be mapped. (mandatory)
type: string
minLength: 1
example: /container/directory
nullable: true
pattern: .*
readOnly:
description: Force the volume to be mounted with read-only permissions. Defaults to false.
type: boolean
default: true
nullable: true
mountPath:
description: The path that the host volume will be mounted to when in use. (mandatory)
type: string
minLength: 1
example: /local/directory
nullable: true
pattern: .*
mountPropagation:
$ref: '#/components/schemas/HostPathMountPropagation'
nullable: true
type: object
SecretFieldsUpdatable:
properties:
mountPath:
description: Local path within the workload to which the Secret will be mapped to. (mandatory)
type: string
minLength: 1
nullable: true
defaultMode:
$ref: '#/components/schemas/DefaultMode'
nullable: true
type: object
HostNetwork:
description: Whether to enable host networking. Default to false.
type: boolean
example: false
nullable: true
EnvironmentVariable:
description: Details of an environment variable which is populated into the container.
properties:
name:
description: The name of the environment variable. (mandatory)
type: string
minLength: 1
example: HOME
nullable: true
pattern: .*
value:
description: The value of the environment variable. (mutually exclusive with secret, userCredential, configMap and podFieldRef)
type: string
example: /home/my-folder
nullable: true
pattern: .*
secret:
$ref: '#/components/schemas/EnvironmentVariableSecret'
configMap:
$ref: '#/components/schemas/EnvironmentVariableConfigMap'
podFieldRef:
$ref: '#/components/schemas/EnvironmentVariablePodFieldReference'
userCredential:
$ref: '#/components/schemas/EnvironmentVariableUserCredential'
exclude:
description: Use 'true' in case the environment variable is defined in defaults of the policy, and you wish to exclude it from the workload.
type: boolean
example: false
nullable: true
description:
description: Description of the environment variable.
type: string
example: Home directory of the user.
nullable: true
pattern: .*
nullable: true
type: object
LargeShmRequest:
description: A large /dev/shm device to mount into a container running the created workload. An shm is a shared file system mounted on RAM.
type: boolean
example: false
nullable: true
RunAsGid:
description: The group id to run the entrypoint of the container which executes the workspace. Default to the value specified in the environment asset `runAsGid` field (optional). Use only when the source uid/gid of the environment asset is not `fromTheImage`, and `overrideUidGidInWorkspace` is enabled.
type: integer
format: int64
example: 30
nullable: true
EnvironmentVariableSecret:
description: Details of the secret and key use to populate the environment variable
properties:
name:
description: The name of the secret resource. (mandatory)
type: string
minLength: 1
example: postgress_secret
nullable: true
pattern: ^[a-z0-9]([a-z0-9-]*[a-z0-9])?$
key:
description: The key in the secret resource. (mandatory)
type: string
minLength: 1
example: POSTGRES_PASSWORD
nullable: true
pattern: .*
nullable: true
type: object
TrainingSpec:
description: The specifications of the training to be created.
properties:
spec:
$ref: '#/components/schemas/TrainingSpecSpec'
Annotations:
description: Set of annotations to populate into the container running the workload.
type: array
items:
$ref: '#/components/schemas/Annotation'
nullable: true
WorkloadDesiredPhase:
description: The desired phase of the workload.
type: string
enum:
- Running
- Stopped
- Deleted
ProbeHandler:
description: The action taken to determine the health of the container. (mandatory)
type: object
properties:
httpGet:
description: An action based on HTTP Get requests.
type: object
properties:
path:
description: Path to access on the HTTP server, defaults to /.
type: string
pattern: ^(\x2F[a-zA-Z0-9\-_.\x2F]*)?$
example: /
nullable: true
port:
description: Number of the port to access on the container.
type: integer
format: int32
minimum: 1
maximum: 65535
nullable: true
host:
description: Host name to connect to, defaults to the pod IP.
type: string
format: hostname
example: example.com
nullable: true
pattern: .*
scheme:
$ref: '#/components/schemas/ProbeHandlerScheme'
nullable: true
MatchExpressionOperator:
description: Represents a key's relationship to a set of values (mandatory).
type: string
enum:
- In
- NotIn
- Exists
- DoesNotExist
- Gt
- Lt
TolerationEffect:
description: The taint effect to match. (mandatory)
type: string
enum:
- NoSchedule
- NoExecute
- PreferNoSchedule
- Any
nullable: true
GitPassword:
properties:
passwordSecret:
description: Secret containing the credentials of the repository (needed for non public repository which requires authentication). (deprecated)
type: string
minLength: 1
example: my-password-secret
nullable: true
secretKeyOfUser:
description: The key to use for loading the user name from the secret. The default is `User`. (deprecated)
type: string
minLength: 1
example: User
nullable: true
secretKeyOfPassword:
description: The key to use for loading the password from the secret. The default is `Password`. (deprecated)
type: string
minLength: 1
example: Password
nullable: true
nullable: true
type: object
ImagePullPolicy:
description: Image pull policy. Defaults to `Always` if `:latest` tag is specified, otherwise it is `IfNotPresent`.
type: string
minLength: 1
enum:
- Always
- Never
- IfNotPresent
nullable: true
AllowPrivilegeEscalation:
description: Allow the container running the workload and all launched processes to gain additional privileges after the workload starts. For more information consult the User Identity in Container guide at https://docs.run.ai/admin/runai-setup/config/non-root-containers/
type: boolean
example: false
nullable: true
HostPathItems:
description: Set of host paths to use in the workload.
type: array
items:
$ref: '#/components/schemas/HostPathInstance'
nullable: true
WorkloadName:
description: The name of the workload.
type: string
minLength: 1
example: my-workload-name
pattern: .*
ClaimInfo:
description: Claim information for the newly created PVC. The information should not be provided when attempting to use existing PVC.
properties:
size:
$ref: '#/components/schemas/PvcClaimSize'
storageClass:
description: Storage class name to associate with the PVC. This parameter may be omitted if there is a single storage class in the system, or you are using the default storage class. For more information, see [Storage class](https://kubernetes.io/docs/concepts/storage/storage-classes).
type: string
minLength: 1
example: my-storage-class
nullable: true
pattern: ^[a-z0-9]([a-z0-9-]*[a-z0-9])?$
accessModes:
$ref: '#/components/schemas/PvcAccessModes'
volumeMode:
$ref: '#/components/schemas/PvcVolumeMode'
addedAttrValues:
$ref: '#/components/schemas/PvcAddedAttrValues'
nullable: true
type: object
DepartmentId2:
description: The id of the department.
type: string
minLength: 1
example: 2
pattern: .*
HostPathMountPropagation:
description: Share this volumes mount with other containers. If set to HostToContainer, this volume mount will receive all subsequent mounts that are mounted to this volume or any of its subdirectories. In case of multiple hostPath entries, this field should have the same value for all of them.
type: string
enum:
- None
- HostToContainer
nullable: true
StorageInstanceName:
properties:
name:
description: unique name to identify the instance. primarily used for policy locked rules.
type: string
minLength: 1
example: storage-instance-a
nullable: true
nullable: true
type: object
EnvironmentVariables:
description: Set of environment variables to populate into the container running the workload.
type: array
items:
$ref: '#/components/schemas/EnvironmentVariable'
nullable: true
CpuCoreRequest:
description: CPU units to allocate for the created workload (0.5, 1, .etc). The workload will receive at least this amount of CPU. Note that the workload will not be scheduled unless the system can guarantee this amount of CPUs to the workload.
format: double
type: number
example: 0.5
nullable: true
minimum: 0
Preemptibility:
description: Specifies whether the workload can be preempted by higher-priority workloads. Valid values are preemptible and non-preemptible. If explicitly set, this value takes precedence. If not set, the system derives the preemptibility from the priorityClassName field, ensuring backward compatibility. Each workload type has a default preemptibility. To view the default preemptibility for each workload type, use the GET /workload-types endpoint.
type: string
minLength: 1
enum:
- preemptible
- non-preemptible
nullable: true
RelatedUrls:
description: Set of URLs that are related to the workload.
type: array
items:
$ref: '#/components/schemas/RelatedUrl'
nullable: true
SupplementalGroups:
description: Comma separated list of groups that the user running the container belongs to, in addition to the group indicated by runAsGid. Use only when the source uid/gid of the environment asset is not `fromTheImage`, and `overrideUidGidInWorkspace` is enabled. Using an empty string implies reverting the supplementary groups of the image.
type: string
example: 2,3,5,8
nullable: true
pattern: .*
DataVolume:
properties:
id:
description: The unique identifier of the data volume. (mandatory)
type: string
format: uuid
example: 123e4567-e89b-12d3-a456-426614174000
nullable: true
mountPath:
description: The path where the data volume will be mounted. (mandatory)
type: string
example: /mnt/data
nullable: true
pattern: .*
nullable: true
type: object
ConfigMapItems:
description: Set of config map volumes to use in the workload
type: array
items:
$ref: '#/components/schemas/ConfigMapInstance'
# --- truncated at 32 KB (75 KB total) ---
# Full source: https://raw.githubusercontent.com/api-evangelist/runai/refs/heads/main/openapi/runai-trainings-api-openapi.yml