Falcon 180B

Released September 6, 2023 by Abu Dhabi's Technology Innovation Institute, Falcon 180B was the largest openly released language model of its time: 180 billion parameters trained on 3.5 trillion tokens of RefinedWeb — a 5-trillion-token deduplicated, aggressively filtered Common Crawl derivative. It topped the Hugging Face Open LLM Leaderboard, beat Llama 2 70B on MMLU, and matched PaLM 2-Large on a range of benchmarks.

Falcon's thesis was data curation over architecture novelty: proof that meticulously cleaned public web data could rival curated corpora. The license is the caveat — unlike the Apache-2.0 Falcon 7B/40B, the 180B weights ship under a model-specific Falcon license with an acceptable-use policy and restrictive commercial terms (notably excluding certain hosting uses). At 320GB+ in full precision, it also demands serious hardware.