CodeFuse-CodeLlama-34B (MFTCoder)

MFTCoder is Ant Group's CodeFuse team framework for multi-task fine-tuning of code LLMs, and CodeFuse-CodeLlama-34B is its showcase model. The framework's pitch is that fine-tuning one model jointly across code tasks — code generation, translation, comments, test generation — beats separate per-task fine-tunes and beats naive mixed-data training, while handling data imbalance and differing convergence speeds across tasks. It accepts CodeLlama and Qwen bases and was presented at KDD 2024.

The model itself scored 74.4% pass@1 on HumanEval in the team's tests, ahead of GPT-4's 67% zero-shot, and the broader CodeFuse line reported HumanEval wins across sizes including CodeFuse-DeepSeek-33B at 78.7%. The framework, training data, and models are open sourced, and the CodeFuse project reported over 1.5 million model downloads on Hugging Face and ModelScope. The headline comparisons are vendor-run, but the KDD peer review adds credibility to the method.

• HumanEval: CodeFuse-CodeLlama-34B 74.4% pass@1 vs GPT-4 67% zero-shot (provider-reported, paper)
• HumanEval: CodeFuse-DeepSeek-33B 78.7% (provider-reported, MFTCoder repo table)
• HumanEval: CodeFuse-13B 37.1% (provider-reported, CodeFuse paper)