Scale AI Model Templates V3 (Beta) API

openapi: 3.2.0
info:
  title: EGP API V4 Model Templates V3 (Beta) API
  description: This is the parent API for all EGP APIs. If you are looking for the EGP API, please go to https://api.egp.scale.com/docs.
  contact:
    name: Scale Generative AI Platform
    url: https://scale.com/genai-platform
  version: 0.1.0
servers:
- url: https://api.egp.scale.com
security: []
tags:
- name: Model Templates V3 (Beta)
paths:
  /v4/model-templates:
    post:
      tags:
      - Model Templates V3 (Beta)
      summary: Create Model Template
      description: '### Description

        Creates a model template.


        ### Details


        Model templates serve 2 purposes. First, they provide common scaffolding that is static across multiple models. Second, they expose several variables that can be injected at model creation time to customize the model.


        For example, a model template can define a docker image that contains code to run a HuggingFace or SentenceTransformers model. This docker image code also accepts environment variables that can be set to swap out the model weights or model name.


        Two of the most important fields required to create a model template are the `model_creation_parameters_schema` and `model_request_parameters_schema` fields.


        The `model_creation_parameters_schema` field defines the schema for parameters that can be injected at model creation time. For example, if the schema contains a `model_weights_uri` field, which expects a string, when a model is created from this template, the user can provide a URI to a model weights file that can be used to swap out the model weights used by the model.


        The `model_request_parameters_schema` field defines the schema for parameters that can be injected by an end user at model execution time. For example, if the schema contains a `model_request_parameters` field, which expects a string, when a model is executed, the user can provide a string that will be passed to the model for inference.'
      operationId: POST-V4-/model-templates
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/ModelTemplateRequest'
      responses:
        '200':
          description: Successful Response
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ModelTemplateResponse'
        '422':
          description: Validation Error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/HTTPValidationError'
      x-source: https://docs.gp.scale.com/reference/v4/model-templates-v3-beta/create-model-template
    get:
      tags:
      - Model Templates V3 (Beta)
      summary: List Model Templates
      description: '### Description

        Lists all model templates accessible to the user.


        ### Details

        This API can be used to list model templates. If a user has access to multiple accounts, all model templates from all accounts the user is associated with will be returned.'
      operationId: GET-V4-/model-templates
      parameters:
      - name: page
        in: query
        required: false
        schema:
          type: integer
          minimum: 1
          description: Page number for pagination to be returned by the given endpoint. Starts at page 1
          default: 1
          title: Page
        description: Page number for pagination to be returned by the given endpoint. Starts at page 1
      - name: limit
        in: query
        required: false
        schema:
          type: integer
          maximum: 10000
          minimum: 1
          description: Maximum number of artifacts to be returned by the given endpoint. Defaults to 100 and cannot be greater than 10k.
          default: 100
          title: Limit
        description: Maximum number of artifacts to be returned by the given endpoint. Defaults to 100 and cannot be greater than 10k.
      responses:
        '200':
          description: Successful Response
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/PaginatedModelTemplateResponse'
        '422':
          description: Validation Error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/HTTPValidationError'
      x-source: https://docs.gp.scale.com/reference/v4/model-templates-v3-beta/list-model-templates
  /v4/model-templates/{model_template_id}:
    delete:
      tags:
      - Model Templates V3 (Beta)
      summary: Delete Model Template
      description: '### Description

        Deletes a model template


        ### Details

        This API can be used to delete a model template by ID. To use this API, pass in the `id` that was returned from your Create Model Template API call as a path parameter.'
      operationId: DELETE-V4-/model-templates/{model_template_id}
      parameters:
      - name: model_template_id
        in: path
        required: true
        schema:
          title: Model Template Id
          type: string
      responses:
        '200':
          description: Successful Response
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/DeleteResponse'
        '422':
          description: Validation Error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/HTTPValidationError'
      x-source: https://docs.gp.scale.com/reference/v4/model-templates-v3-beta/delete-model-template
    get:
      tags:
      - Model Templates V3 (Beta)
      summary: Get Model Template
      description: '### Description

        Gets the details of a model template


        ### Details

        This API can be used to get information about a single model template by ID. To use this API, pass in the `id` that was returned from your Create Model Template API call as a path parameter.


        Review the response schema to see the fields that will be returned.'
      operationId: GET-V4-/model-templates/{model_template_id}
      parameters:
      - name: model_template_id
        in: path
        required: true
        schema:
          title: Model Template Id
          type: string
      responses:
        '200':
          description: Successful Response
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ModelTemplateResponse'
        '422':
          description: Validation Error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/HTTPValidationError'
      x-source: https://docs.gp.scale.com/reference/v4/model-templates-v3-beta/get-model-template
components:
  schemas:
    ModelTemplateRequest:
      properties:
        name:
          type: string
          title: Name
        endpoint_type:
          $ref: '#/components/schemas/ModelEndpointType'
        model_type:
          $ref: '#/components/schemas/ModelType'
        vendor_configuration:
          $ref: '#/components/schemas/LaunchVendorConfiguration'
        model_creation_parameters_schema:
          $ref: '#/components/schemas/ParameterSchema'
          description: The field names and types of available parameter fields which may be specified during model creation
        model_request_parameters_schema:
          $ref: '#/components/schemas/ParameterSchema'
          description: The field names and types of available parameter fields which may be specified in a model execution API's `model_request_parameters` field.
        endpoint_protocol:
          $ref: '#/components/schemas/LaunchEndpointProtocol'
          description: The name of the calling convention expected by the Launch model endpoint
        account_id:
          type: string
          title: Account Id
          description: The ID of the account that owns the given entity.
          can_patch: false
      type: object
      required:
      - name
      - endpoint_type
      - model_type
      - vendor_configuration
      - account_id
      title: ModelTemplateRequest
    IdentifierTypeEnum:
      type: string
      enum:
      - user
      - service_account
      title: IdentifierTypeEnum
    RequiredResources:
      properties:
        cpus:
          type: integer
          title: Cpus
          default: 3
        memory:
          type: string
          title: Memory
          default: 8Gi
        storage:
          type: string
          title: Storage
          default: 16Gi
        gpus:
          type: integer
          title: Gpus
          default: 0
        gpu_type:
          $ref: '#/components/schemas/GPUType'
      type: object
      title: RequiredResources
    LaunchVendorConfiguration:
      properties:
        vendor:
          type: string
          const: LAUNCH
          title: Vendor
          default: LAUNCH
        bundle_config:
          $ref: '#/components/schemas/ModelBundleConfiguration'
        endpoint_config:
          $ref: '#/components/schemas/CreateModelEndpointConfig'
        fine_tuning_job_bundle_config:
          $ref: '#/components/schemas/FineTuningBundleConfiguration'
      type: object
      required:
      - bundle_config
      title: LaunchVendorConfiguration
      description: "Configuration for launching a model using the Launch service which is an internal and\nself-hosted service developed by Scale that deploys models on Kubernetes.\n\nAttributes:\n    vendor: The vendor of the model template\n    bundle_config: The bundle configuration of the model template\n    endpoint_config: The endpoint configuration of the model template"
    FineTuningBundleConfiguration:
      properties:
        registry:
          type: string
          title: Registry
        image:
          type: string
          title: Image
        tag:
          type: string
          title: Tag
        command:
          items:
            type: string
          type: array
          title: Command
        env:
          additionalProperties:
            type: string
          type: object
          title: Env
        mount_location:
          title: Mount Location
          description: The filesystem location where the fine tuning job's configuration will be available when it is started.
          default: /workspace/launch_specific/config.json
          type: string
        training_dataset_schema_type:
          $ref: '#/components/schemas/TrainingDatasetORMSchemaTypeEnum'
          description: Optionally set required training and validation dataset schema
        resources:
          $ref: '#/components/schemas/RequiredResources'
      type: object
      required:
      - registry
      - image
      - tag
      title: FineTuningBundleConfiguration
    PaginatedModelTemplateResponse:
      properties:
        items:
          items:
            $ref: '#/components/schemas/ModelTemplateResponse'
          type: array
          title: Items
          description: The data returned for the current page.
        total_item_count:
          type: integer
          title: Total Item Count
          description: The total number of items of the query
        current_page:
          type: integer
          title: Current Page
          description: The current page number.
        items_per_page:
          type: integer
          title: Items Per Page
          description: The number of items per page.
      type: object
      required:
      - items
      - total_item_count
      - current_page
      - items_per_page
      title: PaginatedModelTemplateResponse
    ModelEndpointType:
      type: string
      enum:
      - SYNC
      - ASYNC
      - STREAMING
      - BATCH
      title: ModelEndpointType
      description: "An enum representing the different types of model endpoint types supported.\n\nAttributes:\n    SYNC: Denotes that the model endpoint type is sync.\n    ASYNC: Denotes that the model endpoint type is async.\n    STREAMING: Denotes that the model endpoint type is streaming.\n    BATCH: Denotes that the model endpoint type is batch."
    HTTPValidationError:
      properties:
        detail:
          items:
            $ref: '#/components/schemas/ValidationError'
          type: array
          title: Detail
      type: object
      title: HTTPValidationError
    LaunchEndpointProtocol:
      type: string
      enum:
      - SGP
      - COHERE
      - VLLM
      title: LaunchEndpointProtocol
    TrainingDatasetORMSchemaTypeEnum:
      type: string
      enum:
      - GENERATION
      - RERANKING_QUESTIONS
      title: TrainingDatasetORMSchemaTypeEnum
    CreateModelEndpointConfig:
      properties:
        cpus:
          type: integer
          title: Cpus
          default: 3
        memory:
          type: string
          title: Memory
          default: 8Gi
        storage:
          type: string
          title: Storage
          default: 16Gi
        gpus:
          type: integer
          title: Gpus
          default: 0
        gpu_type:
          $ref: '#/components/schemas/GPUType'
        min_workers:
          type: integer
          title: Min Workers
          default: 0
        max_workers:
          type: integer
          title: Max Workers
          default: 1
        per_worker:
          type: integer
          title: Per Worker
          description: 'The maximum number of concurrent requests that an individual worker can

            service. Launch automatically scales the number of workers for the endpoint so that

            each worker is processing ``per_worker`` requests, subject to the limits defined by

            ``min_workers`` and ``max_workers``.


            - If the average number of concurrent requests per worker is lower than

            ``per_worker``, then the number of workers will be reduced. - Otherwise,

            if the average number of concurrent requests per worker is higher than

            ``per_worker``, then the number of workers will be increased to meet the elevated

            traffic.


            Here is our recommendation for computing ``per_worker``:


            1. Compute ``min_workers`` and ``max_workers`` per your minimum and maximum

            throughput requirements. 2. Determine a value for the maximum number of

            concurrent requests in the workload. Divide this number by ``max_workers``. Doing

            this ensures that the number of workers will "climb" to ``max_workers``.'
          default: 10
        endpoint_type:
          $ref: '#/components/schemas/ModelEndpointType'
          default: ASYNC
        high_priority:
          title: High Priority
          default: false
          type: boolean
      type: object
      title: CreateModelEndpointConfig
    ModelTemplateResponse:
      properties:
        name:
          type: string
          title: Name
        endpoint_type:
          $ref: '#/components/schemas/ModelEndpointType'
        model_type:
          $ref: '#/components/schemas/ModelType'
        vendor_configuration:
          $ref: '#/components/schemas/LaunchVendorConfiguration'
        model_creation_parameters_schema:
          $ref: '#/components/schemas/ParameterSchema'
          description: The field names and types of available parameter fields which may be specified during model creation
        model_request_parameters_schema:
          $ref: '#/components/schemas/ParameterSchema'
          description: The field names and types of available parameter fields which may be specified in a model execution API's `model_request_parameters` field.
        endpoint_protocol:
          $ref: '#/components/schemas/LaunchEndpointProtocol'
          description: The name of the calling convention expected by the Launch model endpoint
        id:
          type: string
          title: Id
          description: The unique identifier of the entity.
        created_at:
          type: string
          format: date-time
          title: Created At
          description: The date and time when the entity was created in ISO format.
        account_id:
          type: string
          title: Account Id
          description: The ID of the account that owns the given entity.
          can_patch: false
        created_by_user_id:
          type: string
          title: Created By User Id
          description: The user who originally created the entity.
        created_by_identity_type:
          $ref: '#/components/schemas/IdentifierTypeEnum'
          description: The type of identity that created the entity.
      type: object
      required:
      - name
      - endpoint_type
      - model_type
      - vendor_configuration
      - id
      - created_at
      - account_id
      - created_by_user_id
      - created_by_identity_type
      title: ModelTemplateResponse
    DeleteResponse:
      properties:
        success:
          type: boolean
          title: Success
        count:
          type: integer
          title: Count
      type: object
      required:
      - success
      - count
      title: DeleteResponse
    ParameterSchemaField:
      properties:
        name:
          type: string
          title: Name
        type:
          anyOf:
          - type: string
            const: str
          - type: string
            const: int
          - type: string
            const: float
          - type: string
            const: bool
          title: Type
        description:
          title: Description
          type: string
        required:
          title: Required
          type: boolean
      type: object
      required:
      - name
      - type
      - description
      - required
      title: ParameterSchemaField
    GPUType:
      type: string
      enum:
      - nvidia-tesla-t4
      - nvidia-ampere-a10
      - nvidia-ampere-a100
      - nvidia-ampere-a100e
      - nvidia-hopper-h100
      - nvidia-hopper-h100-1g20gb
      - nvidia-hopper-h100-3g40gb
      title: GPUType
    ParameterSchema:
      properties:
        parameters:
          items:
            $ref: '#/components/schemas/ParameterSchemaField'
          type: array
          title: Parameters
      type: object
      required:
      - parameters
      title: ParameterSchema
    ModelBundleConfiguration:
      properties:
        registry:
          type: string
          title: Registry
        image:
          type: string
          title: Image
        tag:
          type: string
          title: Tag
        command:
          items:
            type: string
          type: array
          title: Command
        env:
          additionalProperties:
            type: string
          type: object
          title: Env
        streaming_command:
          title: Streaming Command
          items:
            type: string
          type: array
        readiness_initial_delay_seconds:
          type: integer
          title: Readiness Initial Delay Seconds
          default: 120
        healthcheck_route:
          type: string
          title: Healthcheck Route
          default: /readyz
        predict_route:
          type: string
          title: Predict Route
          default: /predict
        streaming_predict_route:
          title: Streaming Predict Route
          default: /generate_streaming
          type: string
      type: object
      required:
      - registry
      - image
      - tag
      title: ModelBundleConfiguration
    ModelType:
      type: string
      enum:
      - COMPLETION
      - CHAT_COMPLETION
      - AGENT
      - EMBEDDING
      - RERANKING
      - GENERIC
      - BUNDLE
      title: ModelType
      description: "An enum representing the different types of models supported.\n\nAttributes:\n    COMPLETION: Denotes that the model type is completion.\n    CHAT_COMPLETION: Denotes that the model type is chat completion.\n    AGENT: Denotes that the model type is agent.\n    EMBEDDING: Denotes that the model type is embedding.\n    RERANKING: Denotes that the model type is reranking.\n    GENERIC: Denotes that the model type is generic.\n    BUNDLE: \"Not to be used directly - type to surface a model bundle in the UI - TODO: Explicitly type a bundle"
    ValidationError:
      properties:
        loc:
          items:
            anyOf:
            - type: string
            - type: integer
          type: array
          title: Location
        msg:
          type: string
          title: Message
        type:
          type: string
          title: Error Type
        input:
          title: Input
        ctx:
          type: object
          title: Context
          additionalProperties: true
      type: object
      required:
      - loc
      - msg
      - type
      title: ValidationError
  securitySchemes:
    APIKeyHeader:
      type: apiKey
      in: header
      name: x-api-key
x-assembled-from: Per-operation OpenAPI fragments embedded verbatim in docs.gp.scale.com reference pages; source spec https://api.sgp.scale.com/openapi-versions/v4/openapi.json (404 when assembled)