OpenRLHF
High-performance open-source RLHF framework built on Ray, DeepSpeed, and vLLM.
About
OpenRLHF is an open-source framework for reinforcement learning from human feedback and agentic RL on language and vision-language models. Built on Ray for distributed scaling with DeepSpeed and vLLM under the hood, it uses an agent-based execution model that separates actor, reward, reference, and critic roles across GPUs. It supports PPO, REINFORCE++, GRPO, RLOO, DPO and iterative DPO, KTO, rejection sampling, LoRA RL, and multi-turn VLM RL.
The project is Apache 2.0 licensed, highly active (v0.10 shipped in spring 2026 with multi-turn VLM RL), and holds roughly ten thousand GitHub stars per 2026 third-party snapshots. It is widely cited as one of the most production-ready open RLHF stacks, claiming 3-4x the throughput of older DeepSpeed-Chat style setups thanks to vLLM-accelerated generation.
Honest limitations: this is a serious distributed-training tool, not a plug-and-play trainer; running it well means managing Ray clusters, GPU topology, and reward-model quality yourself. Performance claims are project-reported, and cutting-edge RL methods evolve fast enough that any framework snapshot can age quickly.
What it can do
- PPO, GRPO, DPO, KTO, and rejection-sampling RL pipelines
- Ray-based distributed execution with vLLM-accelerated rollout generation
- Multi-turn and vision-language RLHF with reward-model ensembling
Setup
Clone the GitHub repo, install the Python package with Ray and DeepSpeed dependencies, and launch a training script on your cluster.
Prerequisites
A multi-GPU cluster with Ray set up, plus a base model and preference/reward data.
Use cases
Works with
Trust score — 9/10
Scored 9/10 (verified tier, metrics-v1). Evidence: Verified via vendor site/docs/GitHub and reputable coverage, wave 81; Trending score 86/100 from researcher assessment.
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.