Antfly Data Operations API

Perform data operations including batch inserts/deletes, linear merge sync, backup/restore, and individual key lookups. ## Batch Operations Efficiently insert or delete multiple documents in a single request. The batch endpoint accepts both inserts (map of key-value pairs) and deletes (array of keys). ## Distributed Transactions Antfly provides atomic cross-shard write transactions using a coordinator-based 2-phase commit (2PC) protocol. When a batch operation spans multiple shards, Antfly automatically executes it as a distributed transaction. **How it works:** 1. Metadata server allocates HLC timestamp and selects coordinator shard 2. Coordinator writes transaction record, participants write intents 3. After all intents succeed, coordinator commits transaction 4. Participants are notified asynchronously to resolve intents 5. Recovery loop ensures notifications complete even after coordinator failure **Features:** - **Automatic**: No special API required - just use the batch endpoint - **Atomic**: All writes across all shards commit or abort together - **Recoverable**: Coordinator failures are handled via recovery loops - **Efficient**: ~20ms latency for cross-shard transactions **Performance:** - Single-shard batches: < 5ms latency - Cross-shard transactions: ~20ms latency - Intent resolution: < 30 seconds worst-case (via recovery loop) **Guarantees:** - All writes succeed or all fail (atomicity) - Coordinator failure is recoverable (new leader resumes notifications) - Idempotent resolution (duplicate notifications are safe) **Use Cases:** - Updating related records across shards (e.g., user profile + preferences) - Multi-table inserts that must succeed together - Bulk imports requiring all-or-nothing semantics ## Linear Merge (Data Sync) Synchronize and keep Antfly in sync with external data sources like Shopify, Postgres, S3, or any sorted record source. Also known as: data synchronization, database sync, incremental sync, e-commerce sync. Both source and Antfly must be sorted by the same key. Performs three-way merge: inserts new records, updates changed records, deletes records absent from source. **How it works:** 1. Query external source with pagination (sorted by key) 2. Send sorted page to linear merge endpoint 3. Antfly merges: upserts present records, deletes Antfly records absent from page 4. Repeat for next page - no sync state required between pages **Use Cases:** - **Postgres/MySQL**: `SELECT * FROM products ORDER BY id LIMIT 1000 OFFSET 0` - Sync production DB to Antfly for hybrid search - Run periodically (hourly/daily) to stay in sync - **Shopify API**: `GET /admin/api/2024-01/products.json?order=id&limit=250` - Sync e-commerce catalog with cursor pagination - Antfly becomes searchable product index - **S3 Data Lake**: Sorted JSON files (`data/0001.json`, `data/0002.json`, ...) - Batch import from data warehouse exports - Process files in order, send contents page by page - **Databricks**: `SELECT * FROM delta_table ORDER BY key LIMIT 10000` - Sync data warehouse tables to Antfly - Enable low-latency search over warehouse data **Benefits:** - **Stateless**: No cursors or checkpoints - restart from any page - **Idempotent**: Safe to re-run if interrupted - **Efficient**: Stream comparison, no random access needed **WARNING**: Not safe for concurrent merges with overlapping ranges. Single-client sync API only. ## Backup and Restore Create backups to various storage backends: - `file:///path/to/backup` - Local filesystem - `s3://bucket/path` - Amazon S3 Restore operations rebuild tables from backup snapshots. ## Key Lookups Direct key-value lookups for retrieving individual documents by their unique key.