fsdp

Train speculative decoding models effortlessly and port them smoothly to SGLang serving.

A scalable, agentic-first, and HuggingFace-native RL framework for research (9k lines).

Best practices & guides on how to write distributed pytorch training code

From teacher to tiles — a from-scratch LLM distillation & serving engine: custom Triton/CUDA kernels, FSDP distillation, paged-KV continuous batching, speculative decoding, a Rust gateway, a JAX oracle, and interpretability tooling.

Fault tolerance for PyTorch (HSDP, LocalSGD, DiLoCo, Streaming DiLoCo)

A PyTorch native library for training speculative decoding models

Find why PyTorch training is slow, compare runs, and catch performance regressions in CI.

Educational distributed training and inference library for heterogeneous local hardware: Mac minis, Raspberry Pis and GPUs over plain Python sockets. FSDP, GRPO, parameter-server, expert and data parallelism.

Write once, run anywhere; ezpz 🍋

Research platform for model training, evaluation, and experimentation across architectures, benchmarks, and recipes.

Fast and easy distributed model training examples.

Billus 大模型技能库：面向 LLM、VL、多模态与图像生成模型的训练、调参、剪枝、量化与工程化技能库

Forge kernels — fused Triton kernels for faster, leaner LLM fine-tuning. One-call patching into Hugging Face models via forge.patch(model), with FSDP2 multi-GPU support and a growing set of kernels and architectures. Every speedup backed by a committed benchmark. Apache-2.0.

A minimal, hackable pre-training stack for GPT-style language models

A script for training the ConvNextV2 on CIFAR10 dataset using the FSDP technique for a distributed training scheme.

Simple and efficient implementation of 671B DeepSeek V3 that trainable with FSDP+EP and minimal requirement of 256x A100/H100, targeted for HuggingFace ecosystem

A minimal, hackable pre-training stack for GPT-style language models

Minimal yet high performant code for pretraining llms. Attempts to implement some SOTA features. Implements training through: Deepspeed, Megatron-LM, and FSDP. WIP

Framework, Model & Kernel Optimizations for Distributed Deep Learning - Data Hack Summit

Distributed fine-tuning on Kubernetes that survives mid-run pod eviction and resumes bit-identically.