DBRX
132B fine-grained MoE that set the 2024 bar for open models
About
DBRX was Databricks' March 27, 2024 answer to the open-model race: a 132-billion-parameter mixture-of-experts model with 16 fine-grained experts and 4 active per token (36B active), pretrained on 12 trillion tokens of text and code with a 32K context window. The finer routing gives 65x more expert combinations than the 8-expert designs of Mixtral and Grok-1, and the team reported it beating Llama 2 70B, Mixtral, and Grok-1 across language, programming, and math benchmarks. It was trained in about two and a half months on 3,072 H100s for a reported $10M, built entirely on Databricks' own stack. The license is the Databricks Open Model License rather than Apache 2.0, so commercial users should read the terms. The original Hugging Face checkpoint was later closed, with community re-uploads carrying it forward.
Specs
Context window
Pricing
Free quota
API access
Weights
Released
Benchmarks
Team-reported: outperformed Llama 2 70B, Mixtral 8x7B, and Grok-1 on language, programming, and math (team-reported)
Provider-reported unless a source says otherwise. Benchmarks are marketing-adjacent by nature — treat them as directional.
Use cases
Trust score — 8/10
Scored 8/10 (verified tier, metrics-v1). Evidence: Released 2024-03-27 per Wikipedia and Databricks; Databricks Open Model License (not Apache) — caveat captured; 132B total / 36B active, 16 experts, 12T tokens, 32K context (team specs); Benchmark wins over Llama 2 70B / Mixtral / Grok-1 are team-reported; Original HF checkpoint closed; community re-upload noted.