Zephon is a data loader for text and multimodal training. Tokenize, pack, and mix data as you train, while preserving the same global batches across supported changes in GPU count (elastic determinism) and processing parallelism.
$ uv add zephonPython 3.10+ · Apache 2.0
Start on 16 GPUs and resume on 8 without quietly changing the data at each training step. Zephon preserves the same global batches even when your pipeline splits, packs, and reorders samples.
Keep the data recipe and global batch size fixed, and configure elastic loading for the GPU counts you plan to use.
How elastic determinism worksExperimenting with training data often means repeatedly preprocessing and writing out slightly different copies of a dataset.
For example, changes to tokenization or packing can require another pass over the corpus, adding compute, storage, and time before training begins.
Zephon moves more of that work into training, reducing repeated materialization and shortening the path from a new data recipe to an experiment.
Compose data loading, preprocessing, mixing, and batching in one Python pipeline.
In this example, we load WikiText-2 and AG News from Hugging Face, target a 50:50 token mix, and pack the tokenized text into training batches.
Run uv add "zephon[hf]" transformers. Zephon downloads and caches the public data for you.
example.pyexcerpt
Full runnable example ↓
from zephon import Pipeline
from zephon.io import Dataset
from zephon.work import StaticMixtureWorkSource, TokenEstimation
if __name__ == "__main__":
wiki = Dataset.from_path(
"wiki", "hf://Salesforce/wikitext/wikitext-2-raw-v1/train"
)
news = Dataset.from_path("news", "hf://fancyzhx/ag_news/default/train")
source = StaticMixtureWorkSource(
datasets=[wiki, news],
mixture={"wiki": 0.5, "news": 0.5},
token_estimation=TokenEstimation(), # Enable token-aware mixing since sources are strings
seed=42,
)
pipeline = (
Pipeline(source)
.options(
io_options={"cache": {"enabled": True}},
)
.tokenize(tokenizer_id="gpt2", field="text", parallelism=4)
.pack_flat(max_length=2048, algorithm="wrap")
.batch(microbatch_size=8)
)
batch = next(iter(pipeline))
print(batch.to_training())First batchplayback
$ uv run example.py
Batch output from a local run. Startup shown as an illustrated playback.
Read more about Zephon’s basic concept in the ZUG
Zephon User Guide
Zephon comes with a set of operators needed for foundation model training, but also supports your own custom stateful operators.
Zephon is independent of any fileformat. Read Parquet, JSONL, MDS, Vortex, or LitData - and a new format integrates seamlessly.
Zephon can target mixing ratios on the token-level even when input samples are provided as strings with different token/byte ratios.
Scale operators independently. You can even change the execution substrate - a switch from process to thread-based data loading is just a small change away.
Load from local and distributed filesystems, S3, GCS, or Azure, with optional prefetching.
Zephon pipelines are Python iterables. Use them with any training framework you want.