ARCANA incremental sync#

Re-uploading and re-indexing an entire knowledge base on every run is slow. This guide shows how to push only the files that changed and re-embed only those — the pattern a production ingestion pipeline uses against ARCANA.

The idea#

ARCANA’s index trigger (generate_index) is whole-arcana — there is no per-file index call. But the server skips files already at index_status == "INDEXED", and uploading a file resets it to NOT_INDEXED. So the efficient pattern is:

  1. Work out which local files are new or changed (your code — ARCANA exposes no content hash).

  2. Upload only those: new files POST, existing files PUT. They flip to NOT_INDEXED; everything else stays INDEXED.

  3. Trigger one generate_index — the server re-embeds only the NOT_INDEXED files, i.e. exactly what you just uploaded.

A no-op re-run uploads nothing and indexes nothing.

Note

This relies on the server’s skip-INDEXED behavior (documented on generate_index in the API Reference). If that ever changes, the calls stay correct but a single generate_index would re-embed the whole arcana.

Recipe 1: sync_directory with a SHA-256 policy#

sync_directory does the plumbing — glob the directory, list the remote files, apply your decisions, and trigger one index — while you supply a select callback that decides, per file, whether to "upload" (new), "replace" (changed), or "skip". Change-detection (here SHA-256 against a local manifest you persist between runs) lives entirely in that callback.

import hashlib
import json
from pathlib import Path

from saia_python import SAIAClient

client = SAIAClient()
ARCANA = "my-kb"
MANIFEST = Path("arcana_manifest.json")


def sha256(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()


manifest = json.loads(MANIFEST.read_text()) if MANIFEST.exists() else {}
digests: dict[str, str] = {}  # filled by select(), reused by on_result()


def select(path: Path, remote: dict | None) -> str:
    digests[path.name] = sha256(path)
    if remote is None:
        return "upload"                          # not on the arcana yet (POST)
    if digests[path.name] != manifest.get(path.name):
        return "replace"                         # contents changed (PUT)
    return "skip"                                # unchanged


def on_result(path: Path, entry: dict) -> None:
    # entry == {"file", "status", ["error"]}; status is uploaded/replaced/
    # skipped/failed. Persist the digest only for files we actually pushed.
    if entry["status"] in ("uploaded", "replaced"):
        manifest[path.name] = digests[path.name]


report = client.arcana.sync_directory(
    ARCANA,
    "docs/",
    select=select,
    pattern="*.md",
    prune=True,            # delete remote files with no local counterpart
    on_result=on_result,
)
MANIFEST.write_text(json.dumps(manifest, indent=2, sort_keys=True))

print(report["uploaded"], report["replaced"], report["skipped"])

sync_directory triggers generate_index once, and only when something actually changed (an upload, replace, or prune). Because select returns "upload" vs "replace", the call chooses POST (new) vs PUT (existing) per file, so the sync is idempotent regardless of what is already on the arcana.

Recipe 2: explicit list + one index (provenance-heavy pipelines)#

When the unit of change is coarser than a file (e.g. a source document that re-splits into many files), or you need a transaction-log entry per upload, decide the file set upstream and push it with upload_files, then trigger one index yourself. on_result streams each file’s outcome so you can record provenance (e.g. a git SHA) as the upload happens.

TERMINAL = {"INDEXED", "ERROR", "NOT_INDEXED"}
transaction_log: list[dict] = []


def push_changed(client, arcana: str, candidates: list[Path], manifest: dict):
    # 1. Your change-detection decides the set.
    changed = [p for p in candidates if sha256(p) != manifest.get(p.name)]
    if not changed:
        print("nothing changed — skipping upload + index")
        return None

    # 2. Don't trigger a reindex while one is already running — the server
    #    rejects a concurrent whole-arcana reindex. Defer if busy.
    info = client.arcana.get(arcana).get("index_info") or {}
    status = info.get("index_status")
    if status and status not in TERMINAL:
        print(f"arcana busy (status={status}); re-run when idle")
        return None

    # 3. Upload only the changed files; record provenance per file.
    def record(path: Path, entry: dict) -> None:
        transaction_log.append(
            {
                "file": entry["file"],
                "status": entry["status"],
                "sha256": sha256(path),
                "error": entry.get("error"),
            }
        )

    client.arcana.upload_files(arcana, changed, overwrite=True, on_result=record)

    # 4. One whole-arcana index — re-embeds only the files just uploaded.
    return client.arcana.generate_index(arcana, wait=True, timeout=1800)

Note

upload_files uses a single overwrite mode for the whole batch (True → PUT, False → POST), and a PUT against a file that isn’t on the arcana yet fails. So overwrite=True is correct only when every file in the set already exists. For a mixed set, either call list_files() once and split into a POST batch (new) and a PUT batch (existing), or use Recipe 1 — its select picks POST/PUT per file automatically.

Reliability#

  • Transport-drop tolerance. generate_index treats a dropped trigger connection (RemoteDisconnected / 504 / read timeout) as “the server accepted the work” and falls through to polling the index state, which is authoritative. With wait=True it polls until the status is terminal, tolerating a transient failure on an individual poll; only the overall timeout ends the wait, and the raised TimeoutError names the last status and/or transport error so a stuck reindex is diagnosable.

  • Fail-fast everywhere else. Every other ARCANA call carries a default (connect, read) timeout, so a stalled list_files / upload / get raises requests.Timeout instead of hanging. In a batch, on_result still records the per-file failure and the loop continues.

  • One reindex at a time. Triggering a whole-arcana reindex while one is already running is rejected by the server, so guard with the index_status check shown in Recipe 2 before triggering, and defer if it is non-terminal.

When to use which#

  • Recipe 1 (``sync_directory``) — local files map roughly 1:1 to KB files and you keep a per-file manifest. It handles POST/PUT/skip, pruning, and the single index for you.

  • Recipe 2 (``upload_files`` + ``generate_index``) — selection happens upstream (coarser than per-file), or you need per-file provenance / transaction logging and explicit control over when the index fires.

Both keep change-detection in your code, because ARCANA exposes no content hash. Server-side scoped indexing and a content_sha256 field that would move that detection onto the server are on the Roadmap.