SOLAR-10.7B

SOLAR-10.7B is a 10.7-billion-parameter language model from South Korea's Upstage, introduced in December 2023 alongside a paper on depth up-scaling (DUS). The technique concatenates and continues pre-training base models, in this case built on the Llama 2 architecture with Mistral 7B weights integrated into upscaled layers.

The model drew attention for outperforming models up to 30B parameters and even Mixtral 8x7B on several benchmarks despite its compact size. Released under Apache 2.0, SOLAR-10.7B became a popular fine-tuning base and a showcase for efficient scaling methods from the Korean AI ecosystem.