Antfly Index Management API

Create and manage indexes for full-text search, vector similarity search, and multimodal content. ## Index Types Antfly supports two primary index types: ### `full_text` - Full-Text Search Index BM25-based full-text search using Antfly's native Zig full-text index (a segmented, Lucene-style inverted index). Automatically uses schema information for field mappings. **Configuration**: ```json { "type": "full_text", "name": "search_idx", "mem_only": false } ``` **Options**: - `mem_only` (optional): If true, stores index in memory only (useful for testing) **Features**: - BM25 ranking algorithm - Field-specific queries (`title:computer`) - Boolean operators (`AND`, `OR`, `NOT`) - Range queries (`year:>2020`) - Phrase queries (`"exact phrase"`) - Uses schema's `x-antfly-types` for field analyzers - Respects `x-antfly-include-in-all` for cross-field search ### `embeddings` - Vector Similarity Index Approximate k-nearest neighbors (AKNN) index for semantic similarity search using vector embeddings. **Performance Features**: - Hardware-accelerated distance calculations using SIMD instructions - x86/AMD: AVX2 and AVX-512 optimizations for vector operations - ARM: NEON and SME (Scalable Matrix Extension) acceleration - RaBitQ quantization for reduced memory footprint and faster search - No GPU required - optimized for CPU-based deployments **Basic Configuration**: ```json { "type": "embeddings", "name": "embedding_idx", "field": "description", "embedder": { "provider": "ollama", "model": "all-minilm", "url": "http://localhost:11434" } } ``` **Index Modes**: **Managed indexes** (`external: false`, default) - Antfly owns embedding generation - Must specify `field` or `template` - Must specify `embedder` - Reject manual `_embeddings` writes for that index **External indexes** (`external: true`) - Your application owns embedding generation - Must not specify `field`, `template`, or `embedder` - Dense external indexes must specify `dimension` - Vectors are written directly through the document `_embeddings` field **Managed Index Fields**: - `field`: Field to embed using JSONPath syntax (e.g., `"description"`, `"metadata.content"`, `"$.user.bio"`) - `template`: Handlebars template for custom prompt generation - `embedder`: Embedder configuration for generating vectors - `dimension`: Optional for dense indexes; inferred from the embedder if omitted - `mem_only`: Store index in memory only **External Index Fields**: - `external: true` - `dimension`: Required for dense indexes - `sparse: true`: Use sparse vectors supplied through `_embeddings` **External Dense Example**: ```json { "type": "embeddings", "name": "manual_dense_idx", "external": true, "dimension": 384 } ``` **External Sparse Example**: ```json { "type": "embeddings", "name": "manual_sparse_idx", "external": true, "sparse": true } ``` **Writing Manual Embeddings**: - Manual `_embeddings` writes are only accepted for `external: true` indexes - Dense vectors use arrays: `"_embeddings": { "manual_dense_idx": [0.1, 0.2, 0.3] }` - Sparse vectors use token-weight maps: `"_embeddings": { "manual_sparse_idx": { "7": 1.5, "42": 2.0 } }` - Explicit delete uses `null`: `"_embeddings": { "manual_dense_idx": null }` - Omitting `_embeddings[]` is a no-op; it does not delete the stored vector ## Embedder Providers Managed index creation supports `antfly`, `ollama`, `openai`, and `bedrock` through the purpose-specific `IndexEmbedderConfig`. Other providers in the broader `EmbedderConfig` are available only on surfaces whose runtime can execute them. ### Antfly Inference ```json { "provider": "antfly", "model": "BAAI/bge-small-en-v1.5" } ``` ### Ollama (Local Models) ```json { "provider": "ollama", "model": "all-minilm", "url": "http://localhost:11434" } ``` Popular models: `all-minilm`, `nomic-embed-text`, `mxbai-embed-large` ### OpenAI ```json { "provider": "openai", "model": "text-embedding-3-small", "api_key": "sk-..." } ``` Models: `text-embedding-3-small` (1536d), `text-embedding-3-large` (3072d), `text-embedding-ada-002` (1536d) ### AWS Bedrock ```json { "provider": "bedrock", "model": "amazon.titan-embed-text-v1", "region": "us-east-1" } ``` Models: `amazon.titan-embed-text-v1`, `amazon.titan-embed-text-v2`, `cohere.embed-english-v3`, `cohere.embed-multilingual-v3` ## Multimodal Indexing `EmbeddingsIndexConfig` does not accept a `summarizer`. For two-stage multimodal indexing, generate a textual description into a document field before indexing and configure the managed index against that field. If the application already owns the final vectors, use an `external: true` index and write them through `_embeddings`. ## Template-Based Prompts Customize how documents are converted to text before embedding using Handlebars templates: ```json { "type": "embeddings", "dimension": 384, "field": "description", "template": "Product: {{title}}\nCategory: {{category}}\nDescription: {{description}}", "embedder": { "provider": "ollama", "model": "all-minilm" } } ``` **Template System:** - **Syntax**: Handlebars templating (https://handlebarsjs.com/guide/) - **Caching**: Templates are automatically cached with configurable TTL (default: 5 minutes) - **Context**: Templates receive the full document as context **Basic Syntax:** - `{{fieldName}}`: Insert field value - `{{#if field}}...{{/if}}`: Conditional logic - `{{#each array}}...{{/each}}`: Iterate arrays - `{{#unless condition}}...{{/unless}}`: Inverse conditional - `{{@index}}`, `{{@key}}`, `{{@first}}`, `{{@last}}`: Loop variables **Built-in Helpers:** 1. **scrubHtml** - Remove script/style tags and extract clean text from HTML ```handlebars {{scrubHtml html_content}} ``` - Removes `` and `` tags - Adds newlines after block elements (p, div, h1-h6, li, etc.) - Returns plain text with preserved readability 2. **eq** - Equality comparison for conditionals ```handlebars {{#if (eq status "active")}}Active{{/if}} {{#if (eq @key "special")}}Special field{{/if}} ``` 3. **media** - GenKit dotprompt media directive for multimodal content ```handlebars {{media url=imageDataURI}} {{media url=this.image_url}} {{media url="https://example.com/image.jpg"}} {{media url="s3://endpoint/bucket/image.png"}} {{media url="file:///path/to/image.jpg"}} ``` **Supported URL Schemes:** - `data:` - Base64 encoded data URIs (e.g., `data:image/jpeg;base64,...`) - `http://` / `https://` - Web URLs with automatic content type detection - `file://` - Local filesystem paths - `s3://` - S3-compatible storage (format: `s3://endpoint/bucket/key`) **Automatic Content Processing:** - **Images**: Downloaded, resized (if needed), converted to data URIs - **PDFs**: Text extracted or first page rendered as image - **HTML**: Readable text extracted using Mozilla Readability **Security Controls:** Downloads are protected by content security settings: - Allowed host whitelist - Private IP blocking (prevents SSRF attacks) - Download size limits (default: 100MB) - HTTP downloads time out after 30 seconds by default; zero disables the deadline - Image dimension limits (default: 2048px, auto-resized) See: https://antfly.io/docs/configuration#security--cors 4. **encodeToon** is not available in these templates. It is a helper of the retrieval agent's `document_renderer`, which renders documents into the generation prompt as TOON by default. Without a template, the `field` value is used directly, or the entire document is serialized. ## Field Extraction with JSONPath The `field` parameter supports JSONPath syntax for extracting values from nested document structures: ```json { "type": "embeddings", "field": "metadata.description", "embedder": {"provider": "ollama", "model": "all-minilm"} } ``` **JSONPath Examples**: - `"title"` - Top-level field - `"user.bio"` - Nested field (auto-prepends `$.` if not present) - `"$.metadata.content"` - Explicit JSONPath - `"$.items[0].name"` - Array indexing The extracted value must be a string. Use `template` for complex transformations or combining multiple fields. ## Configuration Examples ### Simple Text Embedding ```json { "type": "embeddings", "field": "content", "embedder": { "provider": "ollama", "model": "all-minilm" } } ``` Note: `dimension` is omitted and will be inferred from the `all-minilm` model (384). ### Text Embedding with Custom Template ```json { "type": "embeddings", "dimension": 1536, "template": "{{title}}: {{description}}", "embedder": { "provider": "openai", "model": "text-embedding-3-small" } } ``` ### Production OpenAI Setup ```json { "type": "embeddings", "dimension": 1536, "field": "body", "embedder": { "provider": "openai", "model": "text-embedding-3-small", "api_key": "${OPENAI_API_KEY}" } } ``` ## Best Practices - **Choose dimension wisely**: Higher dimensions improve accuracy but increase storage/compute costs - **Use templates for structured data**: Combine multiple fields into meaningful prompts - **Local vs Cloud**: Ollama for development/privacy, cloud providers for production/scale - **Field specification**: Specify `field` when embedding a single field to avoid whole-document serialization ## Implementing Autocomplete Search To implement autocomplete/search-as-you-type functionality: **1. Define schema with search_as_you_type type:** ```json { "document_schemas": { "product": { "schema": { "type": "object", "properties": { "name": { "type": "string", "x-antfly-types": ["text", "search_as_you_type"] } } } } } } ``` **2. Create full-text index:** ```json { "indexes": { "search_idx": { "type": "full_text" } } } ``` **3. Query with bool-prefix multi-match:** ```json { "full_text_search": { "multi_match": { "query": "lap", "type": "bool_prefix", "fields": ["name"] } } } ``` The `search_as_you_type` type creates shingle and prefix subfields that match partial input as users type. For example, "lap" matches "laptop", "lapel", etc. ## Reindexing Data What happens when you need to reindex? Antfly handles reindexing through async enrichment, automatically rebuilding indexes in the background. Here's what to expect when reindexing data (after adding a new index or changing configuration): ### Adding a New Index to Existing Table Antfly handles reindexing automatically through async enrichment: **1. Add the index:** ```bash POST /tables/{tableName}/indexes/{indexName} { "type": "embeddings", "embedder": {...} } ``` **2. Enrichment happens automatically:** - Existing documents are enriched in the background - New writes are enriched immediately - Leader-only processing prevents duplicate work - Check index status to monitor progress **3. Monitor progress:** ```bash GET /tables/{tableName}/indexes/{indexName} ``` Look at `shard_status` for enrichment progress per shard. ### Error Handling for Async Enrichment When embedding generation or enrichment fails, Antfly handles errors gracefully: **Retry Behavior:** - Failed enrichments are automatically retried with exponential backoff - Leader-only processing ensures no duplicate work during retries - Enrichment continues for other documents even if some fail **Monitoring Failures:** - Check index status (`GET /tables/{table}/indexes/{index}`) for error counts - Prometheus metrics expose enrichment failure rates - Failed documents remain searchable via full-text index (if configured) **Common Failure Scenarios:** 1. **Embedding API Unavailable** - Enrichment retries automatically when service recovers - Documents remain indexed (without embeddings) for full-text search - Use `sync_level: enrichments` to precompute enrichments before committing. A synchronous producer failure rejects the write. Failures detected after a document has committed retain it and may report `committed_repair_required` so operators can repair the derived state. 2. **Rate Limiting** - Automatic backoff and retry - Consider using local Ollama to avoid rate limits - Or increase API quota with embedding provider 3. **Model Errors (invalid input, size limits)** - Logged with document ID for investigation - Retries stop after max attempts - Document remains in table without enrichment ### Changing Index Configuration To change index configuration: 1. Drop the existing index: `DELETE /tables/{table}/indexes/{index}` 2. Recreate with new config: `POST /tables/{table}/indexes/{index}` 3. Existing data is automatically re-enriched **Note:** Full-text indexes (`full_text`) reindex immediately on creation and automatically rebuild when schema changes are made. Vector indexes (`embeddings`) enrich asynchronously in the background. ### Backup Before Major Reindexing For large datasets, backup before reindexing: ```bash POST /tables/{tableName}/backup { "backup_id": "before-reindex-2025-01", "location": "s3://mybucket/backups/products" } ``` ## Graph Indexes and Edge TTL Graph indexes (`graph`) enable relationship modeling and traversal queries. Edge TTL is under development. The stateful runtime currently stamps edge artifacts and selects live projected contributors for adjacency reads, but authoritative cleanup, metric invalidation, and serverless visibility are not complete. Do not rely on graph TTL in production until those pieces are implemented. ### Graph Index Overview Graph indexes store directed edges between documents, enabling queries like: - Finding all connections from a document - Traversing multi-hop paths - Filtering relationships by type **Basic graph index:** ```json { "type": "graph", "name": "social_graph" } ``` ### Edge TTL Configuration Graph indexes accept an index-level `ttl.duration` policy. The legacy `ttl_duration` field is an alias; do not specify both. ```json { "type": "graph", "name": "recent_interactions", "ttl": {"duration": "7d"} } ``` The duration uses the same format as document TTL, such as `30s`, `5m`, `24h`, or `7d`. Each source contribution receives a server-assigned creation timestamp. Stateful graph reads filter expired contributions and select the highest-priority live source when sources overlap. Graph TTL is currently incomplete. There is no authoritative edge cleanup worker, published graph metric invalidation at a deadline, or serverless segment expiration filter. See `zig/GRAPH.md` for the design and current implementation status.