MagicoderS
The hardened Magicoder follow-up: OSS-Instruct plus Evol-Instruct, beating GPT-3.5 and Gemini Ultra on HumanEval at 7B.
About
MagicoderS is the stronger sibling in the Magicoder line, trained on OSS-Instruct combined with the Evol-Instruct recipe rather than OSS-Instruct alone. The flagship MagicoderS-DS-6.7B pairs a DeepSeek-Coder base with 240M tokens of the combined data, and the team reported it scoring 76.8% on HumanEval, ahead of GPT-3.5-turbo-1106 (72.6%) and Gemini Ultra (74.4%). That made it one of the most headline-grabbing small open models of early 2024.
As with the base line, weights and data are MIT-licensed and published on GitHub and Hugging Face, and the comparisons were run through the authors' own harness with numbers reported in the ICML 2024 paper. It is a research artifact rather than a product: there is no hosted API, and newer open models have since moved past its scores, but the OSS-Instruct method itself remains widely cited.
Specs
Context window
Pricing
Free quota
API access
Weights
Released
Benchmarks
- HumanEval: MagicoderS-DS 6.7B 76.8% vs GPT-3.5-turbo-1106 72.6% and Gemini Ultra 74.4% (provider-reported, repo README and paper)
- HumanEval+: MagicoderS-DS 6.7B 70.7% pass@1; MBPP+: 75.7% (provider-reported, paper Table 4)
Provider-reported unless a source says otherwise. Benchmarks are marketing-adjacent by nature — treat them as directional.
Use cases
Trust score — 7/10
Scored 7/10 (verified tier, metrics-v1). Evidence: same ICML'24 paper and MIT-licensed artifacts as Magicoder; the GPT-3.5/Gemini-Ultra comparison numbers are published in the repo README and the paper. Strengths: genuinely strong small-model scores, full reproducibility. Docked for: (1) all headline comparisons are author-run against closed models nobody else can re-run identically; (2) research artifact, no hosted API, superseded by later open models.
How the trust score is built
Full methodology →- Adoption · stars/downloads, log-scaled vs peers
- 7.0/10
- Maintenance · recency, releases, issue load
- 6.0/10
- Openness · license + docs
- 10.0/10
- Manual review · sanity read + red-flag check
- 7.0/10
- · Same ICML'24 paper; GPT-3.5/Gemini-Ultra comparison published in repo README
- · Headline comparisons are author-run against closed models