RedPajama provides a set of leading open-source foundation models built on the largest-ever open pre-training dataset.
01 RedPajama-Data-30TLearn more
The largest open-source pre-training dataset, used by over 500 leading generative AI models. This dataset and the open research approach used to create the RedPajama models is helping to advance the frontier of open-source AI.
02 RedPajama-7BLearn more
A suite of fully open-source base, instruction-tuned, and chat models.
The instruct model is the highest scoring open model on HELM benchmarks, making it ideal for a wide range of tasks. It outperforms LLaMA-7B and state-of-the-art open models such as Falcon-7B (Base and Instruct) and MPT-7B (Base and Instruct) on HELM by 2-9 points.
Innovations that make training and inference faster, more scalable, and reliable.
01 FlashAttention-2Learn more
This update to FlashAttention is now broadly used by all transformer models, speeds up training and fine-tuning of LLMs by up to 9x and achieves 72% model FLOPs utilization for training on NVIDIA A100s.
02 Sub-quadratic model architectures
03 Cocktail SGDLearn more
One of the key challenges in training generative AI models is networking. To enable faster, more reliable training that can run in a distributed environment, we created Cocktail SGD – a set of optimizations that reduces network communication by 117x.
Read the latest research from our team and academic partners