Home » Technology » Elon Musk Teases Grok 3 Launch: 10X More Power on 100K Nvidia GPUs!

Elon Musk Teases Grok 3 Launch: 10X More Power on 100K Nvidia GPUs!

Photo of author

By Harper Westfield

Elon Musk Teases Grok 3 Launch: 10X More Power on 100K Nvidia GPUs!

Photo of author

By Harper Westfield

Elon Musk has disclosed that the pretraining of xAI’s new Grok 3 large language model (LLM) required ten times the computational power of its predecessor, Grok 2. While specifics weren’t provided, Musk indicated that the training likely occurred on the Colossus supercluster, which is equipped with approximately 100,000 Nvidia H100 GPUs.

In a recent post on X, Musk announced, “Grok 3 is on the horizon, with pretraining now finished using 10X the compute of Grok 2.”

The timing of this development suggests that the Colossus supercomputer, known for its massive array of Nvidia H100 GPUs, was indeed utilized. This aligns with earlier speculations about its purpose for high-scale AI initiatives. The tenfold increase in computing power from Grok 2 to Grok 3 highlights the substantial upgrade in infrastructure used for the latest model. It is believed that the data for training Grok 3 was sourced from X platform users.

While the details of the computational resources for Grok 2 remain under wraps, it’s clear that it operated on a less robust system compared to Grok 3. Nevertheless, Grok 2 still ran on a powerful setup, though it pales in comparison to the latest advancements.

Firms like xAI are compelled to develop supercomputing systems like Colossus to stay competitive against giants such as OpenAI, Google DeepMind, and Anthropic. These enhanced training capabilities enable quicker development and deployment of sophisticated models like Grok 3 or GPT-4, which are designed with hundreds of billions of parameters and require trillions of floating-point operations. Plans are already underway to double Colossus’s capacity to 200,000 H100 and H200 GPUs in the near future, setting the stage for the next iteration of Grok to be pre-trained on an even larger scale.

See also  Comcast's L4S Tech Slashes Latency by 78%: Testing Starts in Select Cities!

xAI is also aiming to eventually build out Colossus to over a million GPUs. This future version of the supercomputer will facilitate the training of LLMs with trillions of parameters, potentially resulting in unprecedented accuracy that surpasses the capabilities of Grok 3 or GPT-4. Beyond sheer size, these future models are expected to exhibit enhanced reasoning abilities, edging closer to the elusive goal of achieving artificial general intelligence—a key ambition for leaders in the AI space like xAI and OpenAI.

Similar Posts

Rate this post
Share this :

Leave a Comment