Reportedly Elon Musk’s xAI plans a tenfold expansion of its Colossus data center to incorporate more than 1 million GPUs in an effort to close the gap with rivals such as Google, OpenAI and Anthropic - FT [**Note: Colossus believed to be the largest supercomputer in the world and currently operating a cluster of 100-200K Nvidia GPUs]
Thursday, December 5, 2024 1:34:23 AMEST
- Work has already begun to increase the size of the facility in Memphis, Tennessee
- The chips are used to train Musk’s chatbot Grok, which is less advanced and has fewer users than market-leader ChatGPT or Google’s Gemini.
**Reminder: on Dec 2nd, NVDA Reportedly Elon Musk directly approached Nvidia CEO Huang, offering to pay a premium for Grok AI to be prioritized for GB200 delivery - DigiTimes
**Note: on June 3rd, Elon Musk: Given the pace of technology improvement, it’s not worth sinking 1GW of power into H100s; The xAI 100K H100 liquid-cooled training cluster will be online in a few months; Next big step would probably be ~300K B200s with CX8 networking next summer.
in Apr 2024, the X account "The Technology Brother" posted that Mark Zuckerberg's Meta had amassed one of the largest stashes of the H100 GPUs in the world, amounting to around 350K units.
- in Mar 2024, NVIDIA's CEO Jensen Huang revealed that the company's latest B200 GPUs would likely be priced at between $30,000 and $40,000. This means that xAI could be about to spend ~$9B on acquiring around 300K units of the B200 GPU
- In April's interview with Norway wealth fund CEO Nicolai Tangen on Twitter/X spaces, which was beset with multiple crashes and glitches, Musk said training the Grok 2 model takes about 20,000 Nvidia H100 GPUs. Also said that training the Grok 3 model and beyond will require 100,000 Nvidia H100s.
~
- The chips are used to train Musk’s chatbot Grok, which is less advanced and has fewer users than market-leader ChatGPT or Google’s Gemini.
**Reminder: on Dec 2nd, NVDA Reportedly Elon Musk directly approached Nvidia CEO Huang, offering to pay a premium for Grok AI to be prioritized for GB200 delivery - DigiTimes
**Note: on June 3rd, Elon Musk: Given the pace of technology improvement, it’s not worth sinking 1GW of power into H100s; The xAI 100K H100 liquid-cooled training cluster will be online in a few months; Next big step would probably be ~300K B200s with CX8 networking next summer.
in Apr 2024, the X account "The Technology Brother" posted that Mark Zuckerberg's Meta had amassed one of the largest stashes of the H100 GPUs in the world, amounting to around 350K units.
- in Mar 2024, NVIDIA's CEO Jensen Huang revealed that the company's latest B200 GPUs would likely be priced at between $30,000 and $40,000. This means that xAI could be about to spend ~$9B on acquiring around 300K units of the B200 GPU
- In April's interview with Norway wealth fund CEO Nicolai Tangen on Twitter/X spaces, which was beset with multiple crashes and glitches, Musk said training the Grok 2 model takes about 20,000 Nvidia H100 GPUs. Also said that training the Grok 3 model and beyond will require 100,000 Nvidia H100s.