Macrometa Index API

openapi: 3.0.0
info:
  title: Macrometa API Reference Activity Metrics Index API
  version: 0.17.17
  description: API reference for the Macrometa Global Data Network.
  license:
    name: Macrometa License, Version 2.0
servers:
- url: https://api-play.paas.macrometa.io
  description: GDN API
host: api-play.paas.macrometa.io
security:
- ApiKeyAuth: []
- BearerAuth: []
tags:
- name: Index
paths:
  /api/hss/v1/index:
    get:
      tags:
      - Index
      summary: List all indexes
      description: Get the list of all indexes.
      operationId: get_all_indexes_api_hss_v1_index_get
      responses:
        '200':
          description: Successfully returned indexes.
          content:
            application/json:
              schema:
                items:
                  $ref: '#/components/schemas/SearchIndex'
                type: array
                title: Response getIndexList
        '401':
          description: 'Error: Unauthorized.'
          content:
            application/json:
              example:
                detail: Unauthorized.
        '500':
          description: 'Error: Internal server error.'
          content:
            application/json:
              example:
                message: Unable to fetch indexes.
      security:
      - APIKeyHeader: []
    post:
      tags:
      - Index
      summary: Create an index
      description: 'An index is a bucket where all searchable documents are collected.

        We support indexing text documents, JSON documents, and PDFs.  An index can be created in one of two ways: a lexical search index or a

        semantic search index.


        A lexical search index is based on the exact match of query terms or phrases within the text.  It operates on

        the surface level, looking for specific sequences of characters or work forms without considering the deeper meaning of those words.

        Use the *LexicalSearchIndex* schema in the REST API body to create a lexical search index.


        A semantic search index goes beyond the exact match, aiming to understand the meaning and context of the query terms and documents.  It

        seeks to interpret the intent behind a query and find conceptually related documents, even if they don’t contain the exact query terms.

        Use the *SemanticSearchIndex* schema in the REST API body to create a semantic search index.'
      operationId: create_index_api_hss_v1_index_post
      requestBody:
        content:
          application/json:
            schema:
              oneOf:
              - $ref: '#/components/schemas/SemanticSearchIndex-Input'
              - $ref: '#/components/schemas/LexicalSearchIndex-Input'
              title: Index
              discriminator:
                propertyName: type
                mapping:
                  semantic: '#/components/schemas/SemanticSearchIndex-Input'
                  lexical: '#/components/schemas/LexicalSearchIndex-Input'
        required: true
      responses:
        '201':
          description: Successfully created index.
          content:
            application/json:
              schema:
                anyOf:
                - $ref: '#/components/schemas/SemanticSearchIndex-Input'
                - $ref: '#/components/schemas/LexicalSearchIndex-Input'
                title: Response createIndex
        '400':
          description: 'Error: Bad request.'
          content:
            application/json:
              example:
                detail: Invalid index definition.
        '401':
          description: 'Error: Unauthorized.'
          content:
            application/json:
              example:
                detail: Unauthorized.
        '409':
          description: 'Error: Conflict.'
          content:
            application/json:
              example:
                detail: Index already exists.
        '500':
          description: 'Error: Internal server error.'
          content:
            application/json:
              example:
                message: Unable to create index.
        '422':
          description: Validation Error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/HTTPValidationError'
      security:
      - APIKeyHeader: []
  /api/hss/v1/index/{index_name}:
    get:
      tags:
      - Index
      summary: Get an index
      description: Get details of an index. This will include the complete index configuration.
      operationId: get_index_api_hss_v1_index__index_name__get
      security:
      - APIKeyHeader: []
      parameters:
      - name: index_name
        in: path
        required: true
        schema:
          type: string
          title: Name of the index to fetch
      responses:
        '200':
          description: Successfully returned index.
          content:
            application/json:
              schema:
                anyOf:
                - $ref: '#/components/schemas/SemanticSearchIndex-Output'
                - $ref: '#/components/schemas/LexicalSearchIndex-Output'
                - $ref: '#/components/schemas/SemanticSearchIndex-Input'
                - $ref: '#/components/schemas/LexicalSearchIndex-Input'
                title: Response getIndex
        '401':
          description: 'Error: Unauthorized.'
          content:
            application/json:
              example:
                detail: Unauthorized.
        '404':
          description: 'Error: Not Found.'
          content:
            application/json:
              example:
                detail: Index `example_search` not found.
        '500':
          description: 'Error: Internal server error.'
          content:
            application/json:
              example:
                message: Unable to fetch index 'example_search'.
        '422':
          description: Validation Error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/HTTPValidationError'
    delete:
      tags:
      - Index
      summary: Delete an index
      description: Delete an index.
      operationId: remove_index_api_hss_v1_index__index_name__delete
      security:
      - APIKeyHeader: []
      parameters:
      - name: index_name
        in: path
        required: true
        schema:
          type: string
          description: Name of the index to fetch.
          title: Index Name
        description: Name of the index to fetch.
      - name: force
        in: query
        required: false
        schema:
          type: boolean
          title: Force delete
          description: This will try to delete all related resources ignoring intermediate errors.
          default: false
        description: This will try to delete all related resources ignoring intermediate errors.
      responses:
        '204':
          description: Successfully deleted index.
        '401':
          description: 'Error: Unauthorized.'
          content:
            application/json:
              example:
                detail: Unauthorized.
        '404':
          description: 'Error: Not Found.'
          content:
            application/json:
              example:
                detail: Index not found.
        '500':
          description: 'Error: Internal server error.'
          content:
            application/json:
              example:
                message: Unable to delete index.
        '422':
          description: Validation Error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/HTTPValidationError'
  /api/hss/v1/index/{index_name}/reindex:
    post:
      tags:
      - Index
      summary: Trigger re-indexing on the source.
      operationId: reindex_api_hss_v1_index__index_name__reindex_post
      security:
      - APIKeyHeader: []
      parameters:
      - name: index_name
        in: path
        required: true
        schema:
          type: string
          title: Index Name
      - name: truncate
        in: query
        required: false
        schema:
          type: boolean
          default: false
          title: Truncate
      responses:
        '200':
          description: Successfully re-indexed.
          content:
            application/json:
              schema: {}
        '401':
          description: 'Error: Unauthorized.'
          content:
            application/json:
              example:
                detail: Unauthorized.
        '404':
          description: 'Error: Not Found.'
          content:
            application/json:
              example:
                detail: Index not found.
        '422':
          description: 'Error: Unprocessable entity.'
          content:
            application/json:
              example:
                detail:
                - type: json_invalid
                  loc:
                  - body
                  - 72
                  msg: JSON decode error
                  input: {}
                  ctx:
                    error: Expecting ',' delimiter
        '500':
          description: 'Error: Internal server error.'
          content:
            application/json:
              example:
                message: Unable to reindex.
  /api/hss/v1/index/{index_name}/sources/{connection_name}:
    post:
      tags:
      - Index
      summary: Attach connection to index
      operationId: attach_source_api_hss_v1_index__index_name__sources__connection_name__post
      security:
      - APIKeyHeader: []
      parameters:
      - name: index_name
        in: path
        required: true
        schema:
          type: string
          title: Index Name
      - name: connection_name
        in: path
        required: true
        schema:
          type: string
          title: Connection Name
      responses:
        '200':
          description: Successfully attached connection.
          content:
            application/json:
              schema:
                type: array
                items:
                  type: string
                title: Response attachConnection
        '401':
          description: 'Error: Unauthorized.'
          content:
            application/json:
              example:
                detail: Unauthorized.
        '404':
          description: 'Error: Not Found.'
          content:
            application/json:
              example:
                detail: Index not found.
        '422':
          description: Validation Error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/HTTPValidationError'
  /api/hss/v1/index/{index_name}/sources:
    get:
      tags:
      - Index
      summary: Return list of attached connections
      description: Get the list of connections attached to the index.
      operationId: get_attached_sources_api_hss_v1_index__index_name__sources_get
      security:
      - APIKeyHeader: []
      parameters:
      - name: index_name
        in: path
        required: true
        schema:
          type: string
          title: Index Name
      responses:
        '200':
          description: Successfully returned attached connections.
          content:
            application/json:
              schema:
                type: object
                title: Response getAttachedConnections
              example:
                s3:
                  is_init: true
                  total_documents_to_process: 104
                  processed_documents: 35
                  pipeline_start_timestamp: 1712135397406
                  current_process_start_timestamp: 1712135403297
        '401':
          description: 'Error: Unauthorized.'
          content:
            application/json:
              example:
                detail: Unauthorized.
        '404':
          description: 'Error: Not Found.'
          content:
            application/json:
              example:
                detail: Index not found.
        '422':
          description: Validation Error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/HTTPValidationError'
  /api/hss/v1/index/{index_name}/sources/{source_name}:
    delete:
      tags:
      - Index
      summary: Remove connection attached to the index
      description: 'Remove a connection from the index.  This removes all the documents from the index that were added by the connection.

        And this stops these documents from being indexed in the future.'
      operationId: drop_source_api_hss_v1_index__index_name__sources__source_name__delete
      security:
      - APIKeyHeader: []
      parameters:
      - name: index_name
        in: path
        required: true
        schema:
          type: string
          title: Index Name
      - name: source_name
        in: path
        required: true
        schema:
          type: string
          title: Source Name
      responses:
        '200':
          description: Successful Response
          content:
            application/json:
              schema: {}
        '204':
          description: Successfully remove connection.
        '401':
          description: 'Error: Unauthorized.'
          content:
            application/json:
              example:
                detail: Unauthorized.
        '404':
          description: 'Error: Not Found.'
          content:
            application/json:
              example:
                detail: Index not found.
        '500':
          description: 'Error: Internal server error.'
          content:
            application/json:
              example:
                message: Unable to remove connection.
        '422':
          description: Validation Error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/HTTPValidationError'
components:
  schemas:
    PreprocessorConfig:
      properties:
        clean:
          allOf:
          - $ref: '#/components/schemas/CleanConfig'
          description: Configuration for Clean Preprocessor.
          default:
            enable: false
            replace_unicode_quotes: true
            bytes_string_to_string: true
            clean_bullets: true
            clean_extra_whitespace: true
            clean_dashes: true
            clean_trailing_punctuation: false
            clean_non_ascii_chars: true
            clean_ordered_bullets: true
            remove_punctuation: false
            group_broken_paragraphs: true
        extract:
          allOf:
          - $ref: '#/components/schemas/ExtractConfig'
          description: Configuration for JavaScript Extract Preprocessor.
          default:
            enable: false
            logic: function process(data) { return data; }
        chunk:
          allOf:
          - $ref: '#/components/schemas/ChunkConfig'
          description: Configuration for Chunk Preprocessor.
          default:
            enable: false
      type: object
      title: PreprocessorConfig
    SearchIndex:
      properties:
        name:
          type: string
          title: Name
        connections:
          items:
            type: string
          type: array
          title: Connections
          description: List of datasource connections to populate the index.
          default: []
          examples:
          - - s3_wiki
            - akamai_wiki
        type:
          type: string
          title: Type
          description: Type of the Index.
          examples:
          - semantic
          - lexical
      type: object
      required:
      - type
      title: SearchIndex
    ChunkConfig:
      properties:
        enable:
          type: boolean
          title: Enable
          description: Indicates whether to enable chunking. If set to `False`, chunking will be disabled, and other chunk configurations will be ignored.
          default: false
        max_characters:
          anyOf:
          - type: integer
            exclusiveMinimum: 0.0
          - type: 'null'
          title: Max Characters
          description: Chunks elements text and text_as_html (if present) into chunks of length n characters (hard max)
        multipage_sections:
          anyOf:
          - type: boolean
          - type: 'null'
          title: Multipage Sections
          description: If True, sections can span multiple pages.
        combine_text_under_n_chars:
          anyOf:
          - type: integer
            exclusiveMinimum: 0.0
          - type: 'null'
          title: Combine Text Under N Chars
          description: Combines elements (for example a series of titles) until a section reaches a length of n characters. Defaults to `max_characters` which combines chunks whenever space allows.Specifying 0 for this argument suppresses combining of small chunks. Note this value is capped at the `new_after_n_chars` value since a value higher than that would not change this parameter's effect.
        new_after_n_chars:
          anyOf:
          - type: integer
            exclusiveMinimum: 0.0
          - type: 'null'
          title: New After N Chars
          description: Cuts off new sections once they reach a length of n characters (soft max). Defaults to `max_characters` when not specified, which effectively disables any soft window. Specifying 0 for this argument causes each element to appear in a chunk by itself (although an element with text longer than `max_characters` will be still be split into two or more chunks).
        overlap:
          anyOf:
          - type: integer
            exclusiveMinimum: 0.0
          - type: 'null'
          title: Overlap
          description: Specifies the length of a string (`tail`) to be drawn from each chunk and prefixed to the next chunk as a context-preserving mechanism. By default, this only applies to split-chunks where an over sized element is divided into multiple chunks by text-splitting.
        overlap_all:
          anyOf:
          - type: boolean
          - type: 'null'
          title: Overlap All
          description: When `True`, apply overlap between 'normal' chunks formed from whole elements and not subject to text-splitting. Use this with caution as it entails a certain level of 'pollution' of otherwise clean semantic chunk boundaries.
      type: object
      title: ChunkConfig
    DistanceType:
      type: string
      enum:
      - COSINE
      - IP
      - L2
      title: DistanceType
    ExtractConfig:
      properties:
        enable:
          type: boolean
          title: Enable
          description: Indicates whether to enable extracting. If set to `False`, extracting will be disabled, and other extract configurations will be ignored.
          default: false
        logic:
          type: string
          title: Logic
          description: 'JavaScript based data extracting function logic, accepting a `data` argument and returning either a JSON object or a string.Please note: The type of the `data` argument will depend on the ingested data. If the ingested data is structured (JSON or CSV), `data` will be a map/dictionary type. If the ingested data is unstructured (text, PDF, etc.), `data` will be an array of strings.'
          default: function process(data) { return data; }
          examples:
          - function process(data) { return data; }
      type: object
      title: ExtractConfig
    FlatIndexType:
      properties:
        type:
          type: string
          enum:
          - flat
          const: flat
          title: Type
          description: Type of the vector index.
          default: flat
      type: object
      title: FlatIndexType
      description: 'A Flat index, also known as a brute-force index, involves comparing the query vector directly against

        every vector in the dataset to find the most similar items. While this method is simple and can be very

        accurate since it checks every possibility, it''s also computationally expensive and slow, especially as

        the dataset size grows. It is best suited for smaller datasets or scenarios where maximum accuracy is

        required and computational resources or time are not limiting factors.'
    HTTPValidationError:
      properties:
        detail:
          items:
            $ref: '#/components/schemas/ValidationError'
          type: array
          title: Detail
      type: object
      title: HTTPValidationError
    LexicalSearchIndex-Output:
      properties:
        name:
          type: string
          title: Name
          description: Name of the Index.
          examples:
          - wiki_index
        connections:
          items:
            type: string
          type: array
          title: Connections
          description: List of datasource connections to populate the index.
          default: []
          examples:
          - - s3_wiki
            - akamai_wiki
        preprocessor:
          allOf:
          - $ref: '#/components/schemas/PreprocessorConfig'
          description: The preprocessing function and parameters for input data before adding data to the index.
          default:
            clean:
              bytes_string_to_string: true
              clean_bullets: true
              clean_dashes: true
              clean_extra_whitespace: true
              clean_non_ascii_chars: true
              clean_ordered_bullets: true
              clean_trailing_punctuation: false
              enable: false
              group_broken_paragraphs: true
              remove_punctuation: false
              replace_unicode_quotes: true
            extract:
              enable: false
              logic: function process(data) { return data; }
            chunk:
              enable: false
        facet_fields:
          anyOf:
          - items:
              type: string
            type: array
          - type: 'null'
          title: Facet Fields
          description: Facet fields are attributes or properties of documents in a dataset that can be used to categorize and filter search results. List of fields in the document applicable for filtering results.
          default: []
          examples:
          - - doc_name
            - content
        sort_fields:
          anyOf:
          - additionalProperties:
              $ref: '#/components/schemas/SortOrderType'
            type: object
          - type: 'null'
          title: Sort Fields
          description: "List of fields in the document applicable for sorting results and direction. \n        Should be provided in `{ field_name: asc|desc }` format.  \n        **Note: Setting index level sorting affects\n        ranking and similarity score based ordering of search results.**"
          default: {}
          example:
            created_date: desc
            doc_name: asc
        index_job_interval:
          type: integer
          minimum: 0.0
          title: Index Job Interval
          description: Set the time interval, in seconds, between each indexing job. Use 0 to index just once.
          default: 86400
          examples:
          - 3600
          - 86400
        type:
          type: string
          enum:
          - lexical
          const: lexical
          title: Type
          description: Type of the Index.
          default: lexical
        search_config:
          additionalProperties:
            $ref: '#/components/schemas/AnalyzerType'
          type: object
          title: Search Config
          description: "A dictionary of fields to search on and the analyzers to use.\n    Analyzers enable you to break search inputs into sets of sub-values that search views\n    can use to improve searching.  Should be provided in `{ field_name: analyzer_name }` format.  \n    **NOTE: If this property is not provided for an index with structured documents, indexed\n    documents are treated as unstructured documents. For unstructured documents ex: pdf, html,\n    provide `{\"content\": <analyzer_name>}` in the dictionary.\n    if not `{\"content\": \"text_en\"}` is assumed**"
          default:
            content: text_en
          example:
            content: text_en
            doc_name: identity
      type: object
      required:
      - name
      title: LexicalSearchIndex
    AnalyzerType:
      type: string
      enum:
      - identity
      - text_en
      - text_de
      - text_es
      - text_fi
      - text_fr
      - text_it
      - text_nl
      - text_no
      - text_pt
      - text_ru
      - text_sv
      title: AnalyzerType
      description: Analyzers.
    SemanticSearchIndex-Input:
      properties:
        name:
          type: string
          title: Name
          description: Name of the Index.
          examples:
          - wiki_index
        connections:
          items:
            type: string
          type: array
          title: Connections
          description: List of datasource connections to populate the index.
          default: []
          examples:
          - - s3_wiki
            - akamai_wiki
        preprocessor:
          allOf:
          - $ref: '#/components/schemas/PreprocessorConfig'
          description: The preprocessing function and parameters for input data before adding data to the index.
          default:
            clean:
              bytes_string_to_string: true
              clean_bullets: true
              clean_dashes: true
              clean_extra_whitespace: true
              clean_non_ascii_chars: true
              clean_ordered_bullets: true
              clean_trailing_punctuation: false
              enable: false
              group_broken_paragraphs: true
              remove_punctuation: false
              replace_unicode_quotes: true
            extract:
              enable: false
              logic: function process(data) { return data; }
            chunk:
              enable: false
        facet_fields:
          anyOf:
          - items:
              type: string
            type: array
          - type: 'null'
          title: Facet Fields
          description: Facet fields are attributes or properties of documents in a dataset that can be used to categorize and filter search results. List of fields in the document applicable for filtering results.
          default: []
          examples:
          - - doc_name
            - content
        sort_fields:
          anyOf:
          - additionalProperties:
              $ref: '#/components/schemas/SortOrderType'
            type: object
          - type: 'null'
          title: Sort Fields
          description: "List of fields in the document applicable for sorting results and direction. \n        Should be provided in `{ field_name: asc|desc }` format.  \n        **Note: Setting index level sorting affects\n        ranking and similarity score based ordering of search results.**"
          default: {}
          example:
            created_date: desc
            doc_name: asc
        index_job_interval:
          type: integer
          minimum: 0.0
          title: Index Job Interval
          description: Set the time interval, in seconds, between each indexing job. Use 0 to index just once.
          default: 86400
          examples:
          - 3600
          - 86400
        type:
          type: string
          enum:
          - semantic
          const: semantic
          title: Type
          description: Type of the Index.
          default: semantic
        distance_type:
          allOf:
          - $ref: '#/components/schemas/DistanceType'
          description: The distance type defines the distance metrics or similarity metrics functions used to quantify the similarity or dissimilarity (distance) between two vectors. It is used to find approximate nearest neighbors when performing a search on an index.
          default: COSINE
        indexed_fields:
          items:
            type: string
          type: array
          title: Indexed Fields
          description: "List of fields in the document to be indexed.  \n    **NOTE: Make sure to provide this field when indexing structured docuements.**"
          default: []
          examples:
          - - content
            - title
        model_name:
          type: string
          title: Model Name
          description: The name of the model to use for generating embeddings.
          examples:
          - sentence-transformers/all-MiniLM-L6-v2
        index_config:
          oneOf:
          - $ref: '#/components/schemas/IvfIndexType'
          - $ref: '#/components/schemas/HnswIndexType'
          - $ref: '#/components/schemas/FlatIndexType'
          title: Index Config
          description: Defines the indexing strategies to use to search items.
          discriminator:
            propertyName: type
            mapping:
              flat: '#/components/schemas/FlatIndexType'
              hnsw: '#/components/schemas/HnswIndexType'
              ivf: '#/components/schemas/IvfIndexType'
        quantization_config:
          oneOf:
          - $ref: '#/components/schemas/NoneQuantization'
          - $ref: '#/components/schemas/PqQuantization'
          - $ref: '#/components/schemas/SqQuantization'
          title: Quantization Config
          description: Configure the precision of the numbers used to represent model parameters or embeddings. This technique maps continuous or high-precision values to smaller discrete values, effectively compressing the embeddings' size and reducing the memory and computational requirements for storing and processing them.
          discriminator:
            propertyName: type
            mapping:
              none: '#/components/schemas/NoneQuantization'
              pq: '#/components/schemas/PqQuantization'
              sq: '#/components/schemas/SqQuantization'
      type: object
      required:
      - name
      - model_name
      - index_config
      - quantization_config
      title: SemanticSearchIndex
    NoneQuantization:
      properties:
        type:
          type: string
          enum:
          - none
          const: none
          title: Type
          description: Type of quantization.
          default: none
      type: object
      title: NoneQuantization
    SqQuantization:
      properties:
        type:
          type: string
          enum:
          - sq
          const: sq
          title: Type
          description: Type of quantization.
          default: sq
        bits:
          type: integer
          title: Bits
          description: The number of output bits after quantization.
          examples:
          - 8
      type: object
      required:
      - bits
      title: SqQuantization
      description: 'In Scalar quantization (SC), each value is quantized independently, which is straightforward but may

        not capture the correlations between the embeddings'' different dimensions.'
    IvfIndexType:
      properties:
        type:
          type: string
          enum:
          - ivf
          const: ivf
          title: Type
          description: Type of the vector index.
          default: ivf
        n_lists:
          type: integer
          title: N Lists
          description: The number of partitions to use.
          examples:
          - 2
        n_probe:
          type: integer
          title: N Probe
          description: Number of probes during search.
          examples:
          - 8
          min: 1
        sample:
          type: number
          title: Sample
          description: Sampling percentage between 0 and 1,
          examples:
          - 0.8
          max: 1
          min: 0
      type: object
      required:
      - n_lists
      - n_probe
      - sample
      title: IvfIndexType
      description: 'In an Inverted File Index approach, the dataset is first partitioned into smaller clusters or groups using a

        clustering algorithm like k-means. During a search, the query vector is first compared to the cluster centroids

        to identify the clusters most likely to be clusters where similar items might be found, reducing the number

        of comparisons needed. This approach is suitable for larger datasets where there is a need for a balance

        between search speed and accuracy. It narrows down the search space to a few relevant items.'
    PqQuantization:
      properties:
        type:
          type: string
          enum:
          - pq
          const: pq
          title: Type
          description: Type of quantization.
          default: pq
        bits:
          type: integer
          title: Bits
          description: The number of output bits after quant

# --- truncated at 32 KB (46 KB total) ---
# Full source: https://raw.githubusercontent.com/api-evangelist/macrometa/refs/heads/main/openapi/macrometa-index-api-openapi.yml