OpenAI's New Open gpt-oss Models vs o4-mini: A Real-World Comparison

OpenAI just made history by open-sourcing their first language models in over six years. The gpt-oss series includes two reasoning models: a 20B parameter model comparable to o3-mini, and a 120B model that supposedly rivals o4-mini. As champions of open source AI, we were excited to see how these models perform in practice.

At Together AI, we believe the future of AI is open source. That's why we immediately added the gpt-oss models to our platform and decided to put the 120B model head-to-head against o4-mini across five practical tests. Instead of relying solely on benchmarks, we wanted to show our developer community how these models compare in real-world scenarios.

Why gpt-oss Models Align with Our Mission

These models represent exactly what we've been advocating for:

• Complete model ownership - Download, fine-tune, and deploy however you want
• No vendor lock-in - Apache 2.0 license means true freedom
• Exceptional value - 100x cheaper than Claude Opus 4.1 with competitive performance
• State-of-the-art capabilities - Strong reasoning, agentic abilities, and structured outputs
• Open innovation - The AI community can build, improve, and customize freely

Our Testing Methodology

We used our own chat.together.ai interface running gpt-oss-120B against ChatGPT with o4-mini. While not comprehensive scientific benchmarks, these practical tests give developers a feel for real-world performance - the kind of tasks our customers tackle daily.

Test 1: Terminal Snake Game Development

The Challenge: Build a functional snake game that runs in the terminal.Results:‍

• o4-mini: Generated code that compiled but failed functionally˙- snake only moved horizontally despite arrow key inputs
• gpt-oss-120B on Together AI: Created a fully working snake game with proper controls, collision detection, and game-over mechanics

This test highlights something we see regularly: open source models often excel at practical code generation tasks.

‍Winner: gpt-oss-120B ✓

Test 2: Creative SVG Generation

The Challenge: Generate an SVG of a pelican riding a bicycle.‍

Results:‍

• o4-mini: Produced a clean, well-structured SVG with accurate spatial relationships
• gpt-oss-120B: Created functional SVG but with physics issues - pelican appeared to float above the bicycle

Creative tasks can be challenging, and this shows areas where different models have varying strengths.

‍Winner: o4-mini ✓ (gpt-oss gets 0.5 points for partial success)

Test 3: Advanced Instruction Following

The Challenge: Rewrite the US Declaration of Independence's first two paragraphs in cyberpunk style while preserving historical references.

We wanted an objective evaluation, so we used powerful reasoning models (including DeepSeek R1, also available on Together AI) to judge both outputs.

‍Results: Both reasoning models unanimously selected gpt-oss-120B for:

• Superior balance of complex requirements
• Better historical accuracy
• Clearer, more engaging presentation
• More effective style integration

This demonstrates the sophisticated instruction-following capabilities that make open source models viable for complex enterprise use cases.

‍Winner: gpt-oss-120B ✓

Test 4: Mathematical Reasoning

The Challenge: Classic algebra word problem with chickens and cows (196 legs, 68 heads total).

‍Results: Both models correctly solved the problem, arriving at 38 chickens and 30 cows with clear mathematical reasoning.

Winner: Tie ✓✓

Test 5: Web-Enhanced Information Synthesis

The Challenge: Research and summarize a current NSA program status in under 200 words.

‍Results: Both models demonstrated strong capabilities:

• Effective web search integration
• Accurate synthesis of multiple sources
• Proper adherence to word limits and formatting requirements

Winner: Tie ✓✓

Final Results: Open Source Delivers

• gpt-oss-120B on Together AI: 4.5/5
• o4-mini: 3/5

The Open Source Advantage in Action

This comparison reinforces why we're bullish on open source AI. gpt-oss-120B delivers competitive performance while offering:

• Full customization rights - Fine-tune for your specific use cases
• Cost efficiency - Run inference at fraction of proprietary model costs
• Deployment flexibility - Host on Together Cloud, your VPC, or on-premise
• No usage restrictions - Build commercial applications without limitations

Experience gpt-oss on Together AI

Ready to try these groundbreaking open source models? We've optimized gpt-oss for peak performance on our platform:

✅ Fastest inference speeds - Our inference engine delivers 4x faster performance than standard implementations

✅ Competitive pricing - Up to 11x lower costs compared to proprietary alternatives

✅ Easy integration - OpenAI-compatible APIs for seamless migration

✅ Multiple deployment options - Serverless, dedicated, or private cloud

‍Try gpt-oss models now →‍

What This Means for Developers

These results show that high-quality AI capabilities are becoming democratized. Whether you're a startup building your first AI feature or an enterprise scaling mission-critical applications, open source models like gpt-oss offer a compelling alternative to proprietary solutions.

At Together AI, we're committed to making these cutting-edge open source models accessible, fast, and cost-effective for every developer. As the ecosystem continues evolving, we'll keep bringing you the latest and greatest open source innovations.Ready to build with open source AI? Create your Together AI account and start experimenting with gpt-oss models today.