Tokenizers#

Download the open-weight models’ tokenizers and measure prompts offline: token counts, the special/structural-token overhead a chat template adds, the subword fertility of your text, and the token distribution across a whole RAG corpus. This is a local capability built on Hugging Face transformers; it talks to the Hugging Face Hub, not the SAIA API, so it needs no SAIA key.

Availability#

The feature lives in saia_python.tokenizer and is an opt-in extra so the heavy libraries never burden a plain install:

pip install saia-python[tokenizer]

That pulls transformers (chat-template engine), huggingface-hub (downloads), tiktoken (the OpenAI encodings), and sentencepiece. The imports are lazy: importing saia_python without the extra still works — only the tokenizer calls require it, and they raise a clear ImportError pointing at the extra if it is missing.

Why a built-in catalogue is needed#

The SAIA GET /models listing says which models exist but not where each tokenizer lives. Its per-model payload is id / name / input / output / status / demand, with no repository link. The GWDG model catalogue publishes the Hugging Face repositories only in human-readable form. GWDG_MODEL_REPOS captures that mapping, and TokenizerService.available_repos annotates the live model list with it (proprietary external models, e.g. GPT-5.x, o3, … map to None; for those, use tiktoken instead).

Tokenizer files are cached under ~/saia_python/tokenizers/ by default, overridable per call (cache_dir=) or globally via the SAIA_TOKENIZER_DIR environment variable. Only the tokenizer files are fetched — never the model weights — so a download is a few megabytes, not gigabytes.

Authenticated and gated downloads#

Set an HF_TOKEN (in the environment or a .env / .saia_env file beside your SAIA key) for higher Hub rate limits; load_hf_token() resolves it automatically. A few catalogue repos are gated (e.g. google/medgemma-27b-it, under Google’s Health AI terms): a token alone is not enough, but one must also accept the licence once on the model’s Hugging Face page, signed in as the token account. Otherwise the download raises an expressive GatedRepoAccessError carrying the licence URL and the exact setup steps; in a batch download the model is recorded as None and the run continues.

Capabilities#

Counting chat-template tokens#

chat_template_tokens() applies a model’s own chat template to a role/content conversation and returns a ChatTokenCount. The system prompt (and, optionally, the user turn) may be given inline or read from a ``.txt`` / ``.md`` file. It is deliberately tolerant: a missing user turn, an empty conversation, or any shape a strict template rejects degrades to a best-effort count with a recorded warning rather than raising.

The result separates the raw text from the template’s scaffolding:

Field	Meaning
`num_tokens`	Full chat-templated length, including special/structural tokens (BOS/EOS, role markers such as `<\|im_start\|>`, the generation prompt).
`num_text_tokens`	Tokens from the raw message content alone (no special tokens).
`overhead_tokens`	`num_tokens - num_text_tokens` — what the template’s scaffolding adds.
`overhead_ratio_text`	Overhead relative to the text tokens (`overhead / num_text_tokens`).
`overhead_ratio_total`	Overhead as a share of the templated total (`overhead / num_tokens`).
`fertility` / `fertility_with_special`	Subword fertility (tokens per word) of the raw text, and of the full templated prompt.

from saia_python import chat_template_tokens

r = chat_template_tokens(
    "openai-gpt-oss-120b",
    system="You are a careful assistant. Cite sources.",
    user="Summarise the attached report.",
)
print(r.num_tokens, "tokens;", r.overhead_tokens, "from special tokens")
print(f"overhead is {r.overhead_ratio_total:.0%} of the templated total, "
      f"{r.overhead_ratio_text:.0%} on top of the text")
print("subword fertility:", round(r.fertility, 3))

# The system prompt may instead be read from a file:
r = chat_template_tokens("openai-gpt-oss-120b", system_file="system_prompt.md")

chat_template_length(), special_token_overhead(), and subword_fertility() (with an include_special flag) are thin wrappers when you only need one number.

Sizing a RAG corpus#

token_distribution() walks a directory recursively, tokenizes every text file (and estimates a token cost for each image), and returns a TokenDistribution: per-file counts plus aggregate statistics (total, mean, median, percentiles, and a breakdown by_kind()). Useful for sizing a knowledge base against a model’s tokenizer; for example the embedding model qwen3-embedding-4b, which ARCANA’s RAG pipeline uses internally.

from saia_python import token_distribution

dist = token_distribution("path/to/markdown", "qwen3-embedding-4b")
print(dist.summary())
print(dist.by_kind())     # tokens grouped by text vs. image

External OpenAI models#

The externally hosted, proprietary models (GPT-5.x, o3, …) have no downloadable tokenizer. count_tiktoken_tokens() counts their tokens through tiktoken instead, using OPENAI_TIKTOKEN_ENCODINGS.

From a client#

Everything above is also reachable from a SAIAClient via client.tokenizers (a TokenizerService), which additionally knows the live model list — e.g. client.tokenizers.download_all() downloads every open-weight model’s tokenizer, and client.tokenizers.available_repos() shows the id → repository mapping.