Together AI delivers fastest inference for the top open-source models

Over the past few months, our team has been laser-focused on one goal: making our inference platform the fastest place to run the world’s best open-source models. Today, we're excited to share results that speak for themselves.

Across multiple independent benchmarks from Artificial Analysis, our platform now consistently ranks #1 in output speed among GPU-based providers for the most demanding open-source models—including GPT-OSS-20B, GPT-OSS-120B, Qwen-3-235B-Instruct, Qwen-3-Coder-480B, Kimi-K2-Instruct, DeepSeek-R1, and DeepSeek-V3.1. On several models, we’re now delivering up to 2x better output speed than competing providers.

Performance at this level doesn’t come from a single change — it’s the result of coordinated improvements across hardware, kernels, runtime engine tuning, speculative decoding, and draft model training pipeline. In this post, we’ll share the story behind how we achieved it.

1. Next-Gen GPU hardware with engine optimization.

A major portion of our performance gains comes from a fully modernized inference engine built to exploit the latest GPU hardware, optimized kernels, and emerging low-bit quantization formats such as FP4. Instead of optimizing isolated layers, we re-architected the entire stack—compute kernels, memory layout, execution graphs, and scheduling—to work together as a unified high-efficiency system.

Hardware-aware execution on the latest GPUs

Our engine is tuned specifically for the NVIDIA Blackwell architecture, including the NVIDIA GB200 NVL72. This includes optimized paths for low-precision compute (FP8, FP4), high-bandwidth data movement, and near-zero-overhead scheduling that maximizes utilization across all compute tiers. We don’t just run on fast hardware—we structure execution around it to extract its full capability in real workloads.

Together Kernels

We built and integrated a new generation of high-performance GPU kernels designed for NVIDIA Blackwell architecture, enabling us to fully leverage massive bandwidth. This includes our optimized FlashAttention-4 kernels, fused MoE kernels that combine routing and expert FFNs, and more. Together, these hardware-aware kernels dramatically improve throughput for large models and are a key driver of the performance uplift we now see across real-world workloads.

2. Turbo optimization suite

Quantization

A key part of our speed gains comes from our ability to quantize large model weights to low-bit formats—FP8, FP4 (nvfp4 or mxfp4), and hybrid precision—while remaining effectively lossless in practice. Our pipeline performs architecture-aware calibration, fine-grain block-wise scaling, and selective mixed-precision on sensitive paths, allowing us to retain target-model quality even at extreme compression levels. Combined with a runtime built for low-bit execution—including fused FP4/FP8 kernels, quantized KV-cache, and Blackwell-optimized memory layouts—we achieve major latency and throughput improvements without sacrificing accuracy. This near-lossless quantization capability is foundational to the up to faster inference speeds we now deliver across the largest open-source models.

Speculator algorithm

One of the biggest unlocks for our recent performance leap has been our work on production-grade speculative decoding algorithms.

Speculative decoding is not new, but making it reliable across data domains and consistently faster in a multi-tenant serverless environment is extremely difficult. Our implementation includes:

• Training-efficient algorithms that enable us to get higher performance per training flops.
• Training high-accuracy draft models optimized specifically for each target model.
• Adaptive acceptance strategies that maximize speed while preserving output quality.
• Fail-safe fallback mechanisms that keep latency predictable under load.

This unlocks substantial gains—especially for models like Kimi or Qwen3, where our SpecDec stack provides nearly double the output speed. Check our ATLAS blog and the world’s fastest Blackwell inference for more details.

Large-scale speculator training

To support the largest modern LLMs, we built a fully scalable draft-model training pipeline. This is the backbone that lets us deploy high-quality speculative decoders for models that do not come with off-the-shelf speculators, as well as improving upon existing speculators.

Our company developed:

• Scalable training framework that supports high-performance speculator algorithms for target models as large as 1T parameters and beyond.
• Curriculum-based training, post-training recipe, and data-mixing strategies for draft models to match the target model’s stylistic and structural outputs.
• Alignment evaluation framework to test and iterate draft model quality with respect to target models.
• High-performance architectures and pre-trained base models that can be adapted as speculators for many target models.

The result: draft models that achieve high acceptance rates and speed, resulting in the world’s fastest inference speed on NVIDIA Blackwell architecture.

What’s Next?

We’re committed to making open-source AI models not only accessible, but also leading performance and planet-level scalability. These latest benchmarks are a milestone—but not the finish line.

We’re already working on:

• Even faster generation for downstream domains.
• New generation strategies beyond speculative decoding.
• Expanded support for hybrid quantization.

And of course, continuing to push inferencing performance forward.

Contact us

If you are interested in exploring NVIDIA GB200 NVL72 or other Blackwell GPUs for your workloads, or would like to learn more about how our world-class inference optimization works, we invite you to get in touch with our Customer Experience team.