Zuckerberg recently announced that they are going big with Llama3. It is trained with 600,000 H100/A100 GPUs worth around $20B. If Llama3 achieves AGI, Meta might make it open-sourced.
It is almost a year now since Meta first announced their LLM Llama. Now, it's a new year with newer promises...