Tokenizers#

Tokenizer support for the GWDG open-weight models. Requires the optional [tokenizer] extra:

pip install saia-python[tokenizer]

Why a built-in catalogue#

The GET /models listing does not carry a repository link — its per-model payload is id / name / input / output / owned_by / created / demand / status only. The mapping from a GWDG model id to its upstream Hugging Face tokenizer is published only in the human-readable model catalogue, so it is captured here in GWDG_MODEL_REPOS. available_repos() annotates the live model list from that catalogue (proprietary external models, which have no downloadable tokenizer, map to None).

Tokenizers download — files only, never weights — into ~/saia_python/tokenizers/ by default (override per call or via the SAIA_TOKENIZER_DIR environment variable).

Authenticated downloads (`HF_TOKEN`)#

Public tokenizers download anonymously, but a Hugging Face token lifts the Hub’s anonymous rate limit (the “sending unauthenticated requests to the HF Hub” warning) and is required for gated repos. A token is resolved by load_hf_token() from, in order: the HF_TOKEN / HUGGING_FACE_HUB_TOKEN / HUGGINGFACE_TOKEN environment variable, then a .env then .saia_env file in the working directory, then the home directory — and is applied automatically by every download function.

First-time setup:

Create a read token at https://huggingface.co/settings/tokens .
Add it next to your SAIA key, in .env or .saia_env:
```
HF_TOKEN=hf_xxxxxxxxxxxxxxxxxxxxxxxx
```

It is then picked up automatically by every download.

A few catalogue repos are gated (e.g. google/medgemma-27b-it, under Google’s Health AI terms). For those a valid token is not enough — you must also accept the licence once on the model’s Hugging Face page, signed in as the token account; otherwise the download raises GatedRepoAccessError (HTTP 403) — an expressive error carrying the licence URL and the exact setup steps. In a download_all_tokenizers() run a gated model is instead recorded as None and the batch continues; pass verbose=True to see the licence hint.

Sizing a RAG corpus#

token_distribution() walks a directory recursively, tokenizes every text file (and estimates a token cost for each image), and returns per-file counts plus aggregate statistics — for sizing a RAG corpus against a model’s tokenizer (e.g. the embedding model qwen3-embedding-4b, which ARCANA’s RAG pipeline uses internally).

API#

Tokenizer support for the GWDG open-weight models.

The SAIA / Chat AI GET /models endpoint lists which models are available but does not expose where each model’s tokenizer lives — its per-model payload carries id, name, input/output modalities, status and demand, but no repository link. The mapping from a GWDG model id to its upstream Hugging Face repository is only published, in human-readable form, in the model catalogue at https://docs.hpc.gwdg.de/services/ai-services/chat-ai/models/index.html .

This module captures that mapping (GWDG_MODEL_REPOS) and builds the tooling on top of it:

resolve_repo() — translate a GWDG model id (or display name, or a full org/name repo) into its Hugging Face repository.
download_tokenizer() / download_all_tokenizers() — fetch only the tokenizer files (never the weights) into a local cache, defaulting to ~/saia_python/tokenizers/ and overridable per call or via the SAIA_TOKENIZER_DIR environment variable.
load_tokenizer() — load a downloaded tokenizer through transformers.AutoTokenizer.
chat_template_tokens() — apply a model’s chat template to a role/content conversation (the system prompt may be supplied inline or read from a .txt / .md file), and report the resulting token ids, the length, the overhead contributed by special / structural tokens versus the raw text, and the subword fertility. It is deliberately tolerant: a missing user turn (or any other gap a strict chat template would reject) degrades to a best-effort count with a recorded warning rather than raising.

The heavy third-party libraries (transformers, huggingface_hub, tiktoken) are an opt-in extra — install with pip install saia-python[tokenizer] — and are imported lazily so that importing this module (and the package as a whole) never requires them.

saia_python.tokenizer.GWDG_MODEL_REPOS: dict[str, str] = {'apertus-70b-instruct-2509': 'RedHatAI/Apertus-70B-Instruct-2509-FP8-dynamic', 'deepseek-r1-distill-llama-70b': 'deepseek-ai/DeepSeek-R1-Distill-Llama-70B', 'devstral-2-123b-instruct-2512': 'mistralai/Devstral-2-123B-Instruct-2512', 'e5-mistral-7b-instruct': 'intfloat/e5-mistral-7b-instruct', 'gemma-4-31b-it': 'google/gemma-4-31B-it', 'glm-4.7': 'zai-org/GLM-4.7-FP8', 'internvl3.5-30b-a3b': 'OpenGVLab/InternVL3_5-30B-A3B-HF', 'medgemma-27b-it': 'google/medgemma-27b-it', 'meta-llama-3.1-8b-instruct': 'nvidia/Llama-3.1-8B-Instruct-FP8', 'mistral-large-3-675b-instruct-2512': 'mistralai/Mistral-Large-3-675B-Instruct-2512-NVFP4', 'openai-gpt-oss-120b': 'openai/gpt-oss-120b', 'qwen3-30b-a3b-instruct-2507': 'Qwen/Qwen3-30B-A3B-Instruct-2507-FP8', 'qwen3-coder-30b-a3b-instruct': 'Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8', 'qwen3-embedding-4b': 'Qwen/Qwen3-Embedding-4B', 'qwen3-omni-30b-a3b-instruct': 'Qwen/Qwen3-Omni-30B-A3B-Instruct', 'qwen3.5-122b-a10b': 'Qwen/Qwen3.5-122B-A10B-GPTQ-Int4', 'qwen3.5-397b-a17b': 'Qwen/Qwen3.5-397B-A17B-GPTQ-Int4', 'qwen3.6-35b-a3b': 'Qwen/Qwen3.6-35B-A3B-FP8', 'teuken-7b-instruct-research': 'openGPT-X/Teuken-7B-instruct-research-v0.4'}#: Mapping of GWDG API model id → Hugging Face org/name repository.

saia_python.tokenizer.OPENAI_TIKTOKEN_ENCODINGS: dict[str, str] = {'gpt-4.1': 'o200k_base', 'gpt-4.1-mini': 'o200k_base', 'gpt-5': 'o200k_base', 'gpt-5-mini': 'o200k_base', 'gpt-5-nano': 'o200k_base', 'gpt-5.1': 'o200k_base', 'gpt-5.2': 'o200k_base', 'gpt-5.4': 'o200k_base', 'gpt-5.4-mini': 'o200k_base', 'gpt-5.4-nano': 'o200k_base', 'gpt-5.5': 'o200k_base', 'o3': 'o200k_base', 'o3-mini': 'o200k_base'}#: Best-effort tiktoken encoding for the externally hosted OpenAI models, which have no downloadable tokenizer. Used by count_tiktoken_tokens().

saia_python.tokenizer.DEFAULT_TOKENIZER_DIR: Path = PosixPath('/home/runner/saia_python/tokenizers')#: Default directory for downloaded tokenizers (~/saia_python/tokenizers/).

saia_python.tokenizer.load_hf_token(path: str | PathLike[str] | None = None) → str | None[source]#

Discover a Hugging Face access token for authenticated downloads.

An HF token raises the Hub’s anonymous rate limits and is required for gated or private repositories. Resolution order:

path — an explicit .env-style file (any of the HF_TOKEN / HUGGING_FACE_HUB_TOKEN / HUGGINGFACE_TOKEN keys).
Those same environment variables.
A .env then .saia_env file in the current working directory, then in the home directory.

Returns None when no token is configured — public repositories still download anonymously (just rate-limited), so a missing token is not an error.

Note

This is independent of huggingface_hub’s own discovery (its cached huggingface-cli login token / HF_TOKEN env). When this returns None, the Hub library’s own discovery still applies.

saia_python.tokenizer.available_open_models() → list[str][source]#

Return the GWDG open-weight model ids known to this module.

These are the keys of GWDG_MODEL_REPOS — the models for which a tokenizer repository is published and can be downloaded.

saia_python.tokenizer.resolve_repo(model: str) → str[source]#

Translate a GWDG model into its Hugging Face org/name repository.

Accepts, in order of preference:

A GWDG API model id (e.g. "openai-gpt-oss-120b") — exactly as returned by GET /models / passed as "model" in API calls.
A full org/name Hugging Face repo (anything containing /) — used verbatim, so callers can point at a model this module does not list yet.
A catalogue display name (e.g. "GPT OSS 120B") or a loose spelling of an id — matched after normalisation (case / punctuation insensitive).

Parameters:: model – The model id, display name, or org/name repository.
Returns:: The Hugging Face repository as "org/name".
Raises:: ValueError – If model cannot be resolved to a known repository.

saia_python.tokenizer.repo_url(model: str) → str[source]#: Return the full https://huggingface.co/... URL for model.

exception saia_python.tokenizer.GatedRepoAccessError[source]#

A tokenizer download was blocked because the repository is gated.

The repository exists and its tokenizer files are present, but Hugging Face denied access (HTTP 403): a gated repo requires accepting its licence on the model page and an HF_TOKEN for the accepting account. A few catalogue entries are gated — e.g. google/medgemma-27b-it (Google’s Health AI terms). The original huggingface_hub error is preserved as the exception cause (__cause__).

saia_python.tokenizer.tokenizer_dir(cache_dir: str | PathLike[str] | None = None) → Path[source]#

Resolve the directory tokenizers are cached in.

Resolution order: the explicit cache_dir argument, then the SAIA_TOKENIZER_DIR environment variable, then DEFAULT_TOKENIZER_DIR (~/saia_python/tokenizers/). ~ is expanded in all cases.

saia_python.tokenizer.download_tokenizer(model: str, *, cache_dir: str | PathLike[str] | None = None, repo: str | None = None, token: str | None = None, force: bool = False) → Path[source]#

Download a model’s tokenizer files into the local cache.

Only the tokenizer-relevant files are fetched (see _TOKENIZER_FILE_PATTERNS); the model weights are never downloaded, so this stays in the low-megabytes range. Files land in <cache_dir>/<org>/<name>/.

Parameters:

model – A GWDG model id, display name, or org/name repo (see resolve_repo()).
cache_dir – Where to store the files. Defaults to tokenizer_dir() (~/saia_python/tokenizers/).
repo – Explicit Hugging Face repository, bypassing resolve_repo().
token – A Hugging Face access token, for gated/private repos and to lift the anonymous rate limit. When omitted, it is resolved via load_hf_token() (HF_TOKEN env, then .env / .saia_env); if that finds nothing, huggingface_hub’s own discovery (cached login) still applies.
force – Re-download even if the files already exist locally.

Returns:

The local directory containing the tokenizer files.

Raises:

ImportError – If the [tokenizer] extra is not installed.

Download the tokenizers for many models, tolerating per-model failures.

Convenience wrapper that loops download_tokenizer() over a model list. With models=None it covers every open-weight model in GWDG_MODEL_REPOS. A model that fails (e.g. a gated repo without a token, or a transient network error) is recorded as None instead of aborting the whole batch.

To drive this from the live set of available models, fetch the ids first:

from saia_python import SAIAClient, download_all_tokenizers
ids = SAIAClient().models.list_ids()
download_all_tokenizers(models=ids)

Parameters:

models – Model ids/names to download. Defaults to all open-weight models.
cache_dir – Cache directory (see tokenizer_dir()).
token – Hugging Face token. Resolved once via load_hf_token() when omitted (HF_TOKEN env, then .env / .saia_env).
force – Re-download even if cached.
verbose – Print a per-model success/failure line.

Returns:

A mapping of input model id → local directory (or None on failure).

saia_python.tokenizer.load_tokenizer(model: str, *, cache_dir: str | os.PathLike[str] | None = None, repo: str | None = None, token: str | None = None, download: bool = True, trust_remote_code: bool = False, **from_pretrained_kwargs: Any) → PreTrainedTokenizerBase[source]#

Load a model’s tokenizer via transformers.AutoTokenizer.

The tokenizer files are taken from the local cache, downloading them first when missing (unless download=False). Loaded tokenizers are cached per process, so repeated calls for the same model are cheap.

Parameters:

model – A GWDG model id, display name, or org/name repo.
cache_dir – Cache directory (see tokenizer_dir()).
repo – Explicit Hugging Face repository, bypassing resolve_repo().
token – Hugging Face token for gated repos.
download – Download the files if they are not already cached. When False and the cache is empty, loading raises.
trust_remote_code – Forwarded to AutoTokenizer.from_pretrained — a few tokenizers ship a custom class and need this.
**from_pretrained_kwargs – Extra keyword arguments forwarded verbatim to AutoTokenizer.from_pretrained.

Returns:

A transformers.PreTrainedTokenizerBase instance.

Raises:

ImportError – If the [tokenizer] extra is not installed.

class saia_python.tokenizer.ChatTokenCount(model: str | None, repo: str | None, num_tokens: int, num_text_tokens: int, overhead_tokens: int, overhead_ratio_text: float, overhead_ratio_total: float, num_words: int, fertility: float, fertility_with_special: float, token_ids: list[int], tokens: list[str], rendered: str | None, add_generation_prompt: bool, warnings: list[str] = <factory>)[source]#

The result of tokenizing a chat conversation against a model template.

Variables:

model (str | None) – The model the conversation was tokenized for (if known).
repo (str | None) – The resolved Hugging Face repository (if known).
num_tokens (int) – Length of the full chat-templated prompt, including the special and structural tokens the template inserts (BOS/EOS, role markers such as <|im_start|>, the generation prompt, …).
num_text_tokens (int) – Tokens contributed by the raw message content alone, encoded without any special tokens. The “pure text” baseline.
overhead_tokens (int) – num_tokens - num_text_tokens — how many tokens the chat template’s special/structural scaffolding adds.
overhead_ratio_text (float) – overhead_tokens / num_text_tokens — the overhead relative to the raw text (e.g. 0.5 = the scaffolding adds half again as many tokens as the text itself). nan if no text tokens.
overhead_ratio_total (float) – overhead_tokens / num_tokens — the fraction of the full prompt that is special/structural overhead (always in [0, 1] for normal templates). nan if the prompt is empty.
num_words (int) – Whitespace-delimited word count across all message contents.
fertility (float) – Subword fertility of the raw text — num_text_tokens / num_words (tokens per word, special tokens excluded).
fertility_with_special (float) – num_tokens / num_words — fertility including the template’s special/structural tokens.
token_ids (list[int]) – The full templated token ids.
tokens (list[str]) – The token strings for token_ids (empty if the tokenizer cannot map ids back to tokens).
rendered (str | None) – The chat template rendered to text (None if rendering was not available).
add_generation_prompt (bool) – Whether a trailing generation prompt is actually present in token_ids / rendered. This is the effective value: it is forced to False if the requested generation prompt had to be dropped on a fallback path, so it never disagrees with the counted tokens.
warnings (list[str]) – Non-fatal issues — e.g. the chat template rejected the conversation (missing user turn) and a fallback was used.

summary() → str[source]#: Return a one-line human-readable summary.

saia_python.tokenizer.count_words(text: str) → int[source]#: Count whitespace-delimited words — the denominator for fertility.

Apply a model’s chat template to a conversation and measure it.

Builds a role/content conversation — either from an explicit messages list or from the system / user / assistant convenience arguments (the system prompt, and optionally the user turn, may be read from a .txt / .md file) — applies the model’s chat template, and reports the full ChatTokenCount: the templated length, the raw text length, the special/structural-token overhead between them, and the subword fertility.

It is deliberately forgiving. A missing user turn, an empty conversation, or any other shape a strict template rejects is degraded to a best-effort count with an entry in ChatTokenCount.warnings — it never raises on account of a missing chat-template part.

Either model (the tokenizer is loaded for you) or tokenizer (a pre-loaded transformers tokenizer, reused as-is) must be given.

Parameters:

model – A GWDG model id, display name, or org/name repo. Used to load the tokenizer when tokenizer is not supplied.
messages – An explicit [{"role": ..., "content": ...}, ...] list. If given, the convenience arguments below are ignored.
system – Inline system-prompt text.
system_file – Path to a text file whose contents are the system prompt.
user – Inline user-turn text.
user_file – Path to a text file whose contents are the user turn.
assistant – Inline assistant-turn text.
add_generation_prompt – Append the template’s generation prompt (the tokens that cue the model to start replying). Default True.
tokenizer – A pre-loaded tokenizer to use instead of loading one.
cache_dir – Cache directory (see tokenizer_dir()).
repo – Explicit Hugging Face repository, bypassing resolve_repo().
token – Hugging Face token for gated repos.

Returns:

A ChatTokenCount.

Raises:

ValueError – If neither model nor tokenizer is provided, or if a message body is given both inline and as a file.

saia_python.tokenizer.chat_template_length(model: str | None = None, messages: list[dict[str, str]] | None = None, **kwargs: Any) → int[source]#

Return only the templated token length (see chat_template_tokens()).

A thin wrapper that discards everything but ChatTokenCount.num_tokens.

saia_python.tokenizer.special_token_overhead(model: str | None = None, messages: list[dict[str, str]] | None = None, **kwargs: Any) → int[source]#

Return only the special/structural-token overhead.

The number of tokens the chat template adds on top of the raw message text — see ChatTokenCount.overhead_tokens.

saia_python.tokenizer.subword_fertility(model: str | None = None, messages: list[dict[str, str]] | None = None, *, include_special: bool = False, **kwargs: Any) → float[source]#

Return the subword fertility (tokens per word) of a conversation.

Fertility is tokens / words. The include_special flag selects the numerator:

include_special=False (default): the raw-text tokens only — the genuine subword fertility of the content (ChatTokenCount.fertility).
include_special=True: the full chat-templated length, so the template’s special/structural tokens are counted too (ChatTokenCount.fertility_with_special).

Returns nan when the conversation contains no words.

saia_python.tokenizer.count_tiktoken_tokens(text: str, *, model: str | None = None, encoding: str | None = None) → int[source]#

Count tokens with tiktoken — for the externally hosted OpenAI models.

The proprietary models GWDG relays (GPT-5.x, o3, …) have no downloadable Hugging Face tokenizer, but their byte-pair encoding is available through tiktoken. This counts the tokens of a plain string under that encoding.

Parameters:

text – The text to tokenize.
model – An OpenAI model id; its encoding is looked up in OPENAI_TIKTOKEN_ENCODINGS, then via tiktoken.encoding_for_model.
encoding – An explicit tiktoken encoding name (e.g. "o200k_base") that overrides model.

When neither model nor encoding resolves to a known encoding (including the no-argument call), the count falls back to "o200k_base", the encoding shared by the current OpenAI models this module targets.

Returns:: The token count.
Raises:: ImportError – If the [tokenizer] extra is not installed.

saia_python.tokenizer.DEFAULT_TEXT_EXTENSIONS: frozenset[str] = frozenset({'.c', '.cpp', '.csv', '.go', '.htm', '.html', '.java', '.js', '.json', '.jsonl', '.log', '.markdown', '.md', '.mdx', '.ndjson', '.org', '.py', '.rs', '.rst', '.sh', '.srt', '.tex', '.text', '.toml', '.ts', '.tsv', '.txt', '.vtt', '.xml', '.yaml', '.yml'})#: Extensions treated as UTF-8 text and tokenized as content (RAG chunks, docs).

saia_python.tokenizer.DEFAULT_IMAGE_EXTENSIONS: frozenset[str] = frozenset({'.bmp', '.gif', '.jpeg', '.jpg', '.png', '.tif', '.tiff', '.webp'})#: Extensions treated as images (token cost estimated from pixel dimensions).

class saia_python.tokenizer.FileTokenCount(path: str, kind: str, num_tokens: int, num_words: int, size_bytes: int, note: str | None = None)[source]#

Token count for one file in a token_distribution() scan.

Variables:

path (str) – Path relative to the scanned root (POSIX separators).
kind (str) – "text", "image", or "skipped".
num_tokens (int) – Token count — text tokens for text files, an estimate for images (see _estimate_image_tokens()), 0 when skipped or uncounted.
num_words (int) – Whitespace word count (0 for images / skipped).
size_bytes (int) – File size on disk.
note (str | None) – Why a file was skipped or how its count was derived (e.g. "image estimate"), else None.

class saia_python.tokenizer.TokenDistribution(model: str | None, repo: str | None, root: str, files: list[FileTokenCount], include_special: bool = False)[source]#

Token statistics over a directory of files (e.g. a RAG corpus).

Returned by token_distribution(). The per-file rows are in files; the aggregates below summarise the counted ones (text and image, excluding skipped files).

property counted: list[FileTokenCount]#: Files that contributed a token count (text + image).

property token_counts: list[int]#: The per-file token counts of the counted files.

property num_files: int#: Total files iterated (including skipped).

by_kind() → dict[str, int][source]#: File counts grouped by kind (text / image / skipped).

by_extension() → dict[str, dict[str, int]][source]#: Per-extension {files, tokens} rollup (counted files only).

stats() → dict[str, float][source]#: Summary statistics of the per-file token counts.

histogram(bins: int = 10) → list[tuple[float, float, int]][source]#: Bin the per-file token counts into (low, high, count) tuples.

summary() → str[source]#: Return a one-line human-readable summary.

saia_python.tokenizer.token_distribution(directory: str | PathLike[str], model: str | None = None, *, tokenizer: Any | None = None, recursive: bool = True, include_special: bool = False, text_extensions: frozenset[str] | set[str] = frozenset({'.c', '.cpp', '.csv', '.go', '.htm', '.html', '.java', '.js', '.json', '.jsonl', '.log', '.markdown', '.md', '.mdx', '.ndjson', '.org', '.py', '.rs', '.rst', '.sh', '.srt', '.tex', '.text', '.toml', '.ts', '.tsv', '.txt', '.vtt', '.xml', '.yaml', '.yml'}), image_extensions: frozenset[str] | set[str] = frozenset({'.bmp', '.gif', '.jpeg', '.jpg', '.png', '.tif', '.tiff', '.webp'}), count_images: bool = True, image_patch_size: int = 28, max_bytes: int | None = None, follow_symlinks: bool = False, cache_dir: str | PathLike[str] | None = None, repo: str | None = None, token: str | None = None, verbose: bool = False) → TokenDistribution[source]#

Tokenize every file under a directory and summarise the distribution.

Walks directory (recursively by default, into subdirectories), tokenizes each text file as raw content (no chat template — these are documents / RAG chunks), estimates a token cost for each image, and returns a TokenDistribution with per-file rows and aggregate statistics. Built for sizing a RAG corpus against a model’s tokenizer (e.g. the embedding model qwen3-embedding-4b).

Either model or a preloaded tokenizer must be given.

Parameters:

directory – The root directory to scan.
model – A GWDG model id / display name / org/name repo, used to load the tokenizer when tokenizer is not supplied.
tokenizer – A preloaded tokenizer to use instead of loading one.
recursive – Descend into subdirectories (default True).
include_special – Encode text with the tokenizer’s special tokens (default False — count the pure content).
text_extensions – Lower-case suffixes treated as UTF-8 text.
image_extensions – Lower-case suffixes treated as images.
count_images – Estimate image token costs (default True). When False, images are recorded as skipped.
image_patch_size – Patch size for the image estimate (see _estimate_image_tokens()).
max_bytes – Skip text files larger than this many bytes (None = no limit), recording them as skipped.
follow_symlinks – Follow symlinked files/dirs (default False).
cache_dir – Forwarded to load_tokenizer() when a tokenizer must be loaded.
repo – Forwarded to load_tokenizer() when a tokenizer must be loaded.
token – Forwarded to load_tokenizer() when a tokenizer must be loaded.
verbose – Print a per-file line as it is processed.

Returns:

A TokenDistribution.

Raises:

ValueError – If neither model nor tokenizer is given.
NotADirectoryError – If directory is not a directory.

class saia_python.tokenizer.TokenizerService(models_service: Any | None = None, *, cache_dir: str | PathLike[str] | None = None)[source]#

Tokenizer access bound to a SAIAClient.

Reached as client.tokenizers. Thin, stateful glue over the module-level functions that additionally knows the client’s live model list, so it can annotate it with repositories and download the whole available set.

Parameters:

models_service – The client’s ModelsService, used to fetch the live model list.
cache_dir – Default cache directory for this service’s downloads (see tokenizer_dir()).

available_repos(*, live: bool = True) → dict[str, str | None][source]#

Map each available model id to its tokenizer repository.

This is the answer to “does the models endpoint expose the repository?” — it does not: the live /models payload carries no repo field, so each live id is annotated here from the published catalogue (GWDG_MODEL_REPOS). Ids with no known repository — notably the externally hosted proprietary models — map to None.

Parameters:: live – Annotate the live /models listing. When False (or when no models service / the call fails), the static catalogue is returned instead.
Returns:: A mapping of model id → "org/name" repository (or None).

download(model: str, **kwargs: Any) → Path[source]#: Download one model’s tokenizer (see download_tokenizer()).

download_all(*, open_only: bool = True, **kwargs: Any) → dict[str, Path | None][source]#

Download tokenizers for every available model.

Wraps the live available-models listing and download_tokenizer(). With open_only=True (default) only the models with a known downloadable repository are attempted, so the proprietary external models are skipped rather than reported as failures.

Parameters:

open_only – Restrict to models with a known tokenizer repository.
**kwargs – Forwarded to download_all_tokenizers() (token, force, verbose, cache_dir).

Returns:

A mapping of model id → local directory (or None on failure).

load(model: str, **kwargs: Any) → PreTrainedTokenizerBase[source]#: Load one model’s tokenizer (see load_tokenizer()).

chat_template_tokens(model: str | None = None, messages: list[dict[str, str]] | None = None, **kwargs: Any) → ChatTokenCount[source]#: Tokenize a conversation (see chat_template_tokens()).

token_distribution(directory: str | PathLike[str], model: str | None = None, **kwargs: Any) → TokenDistribution[source]#: Tokenize a directory of files (see token_distribution()).

Tokenizers#

Why a built-in catalogue#

Authenticated downloads (HF_TOKEN)#

Sizing a RAG corpus#

API#

Authenticated downloads (`HF_TOKEN`)#