Tesla's New FSD Training Cluster With 10,000 GPUs Comes Online

Tesla is set to turn on its new NVIDIA-based AI-training cluster

NVIDIA

Today, Tesla is turning on its massive new system to advance its Full Self-Driving (FSD) technology. The system has many technicalities, so Tesla enthusiast and investor, Sawyer Merritt, shared insights on X.com.

Tesla uses Graphic Processing Units (GPUs) to train its neural networks that are used to solve AI-based problems. Training Tesla's AI involves using vast amounts of video to teach the system how to operate a vehicle autonomously and safely. NVIDIA, a leading GPU manufacturer, recently released a new chip, the H100, which offers significantly higher performance than its predecessor, the A100. However, the H100 doesn't come cheap. NVIDIA's latest training-focused GPU costs about $30,000, and Tesla needs more than just a handful.

Tesla's new training cluster will be powered by 10,000 H100 NVIDIA GPUs, bringing its cost to hundreds of millions of dollars for the GPUs alone.

However, due to the high demand for this advanced chip, NVIDIA cannot produce enough H100 units to meet Tesla's, and the industry's growing demand.

The Power of Dojo

As a result, Tesla is investing over $1 billion to develop its own supercomputer, Dojo. Dojo will have a custom-designed chip to train Tesla's neural nets that will be leveraged in FSD and Optimus. Elon Musk, Tesla’s CEO, indicated that if NVIDIA could supply enough chips, the creation of Dojo might not have been necessary. Yet, the current situation has made it imperative for Tesla to take matters into its own hands.

Merritt underscored that the computational power required to train Tesla's FSD technology is a significant challenge. However, Tesla is actively addressing this issue. Musk announced that the company plans to invest over $2 billion in 2023 and again in 2024 to enhance its computational capabilities. This investment is necessary to develop FSD technology and signals Tesla’s commitment to innovation and leadership in the automotive industry.

Groundbreaking Data Centers

Tesla is not only developing its own supercomputer but is also in the process of designing what it describes as its "1st of its kind Data Centers." A recent job listing for a Senior Engineering Program Manager in Austin, Texas, where Tesla's Giga Factory operates, reveals the complexity of this initiative.

The establishment of these data centers is integral to Tesla's broader vision. The company’s need for extensive computing resources is paramount, especially when processing and analyzing large amounts of video data for its self-driving software. The Dojo supercomputer is already a testament to their commitment in this area.

Follow

Follow

Follow

Tesla Releases FSD V12.6.1 for Model 3 & Model Y

Not a Tesla App

Last night, Tesla released software update 2024.45.25.15, which includes FSD V12.6.1. This update adds support for all HW3 vehicles, including the Model 3 and Model Y. We’re excited to see the continued support for HW3 owners. 

FSD V12.6.1

V12.6.1 is now going wide, according to Ashok Elluswamy, Tesla’s VP of AI. This update is going to the Model 3 and Model Y for the first time - as only the Model S and Model X were included in FSD V12.6. 

V12.6 is a big step forward for HW3 - it includes End-to-End on Highway, Improved City Streets Behavior, and Smoother and More Accurate Tracking - all contributing towards a better, smoother, and more comfortable build of FSD. You can read our comparison between FSD V12.6 and V13.2.2 here. 

In short, FSD V12.6 performs considerably closer to V13 than V12.5.4.2 - which is a massive improvement. It performs as well as the Cybertruck version of FSD V13, which is still missing a few features when compared to other HW4 vehicles, but it’s a great sign for HW3. A lot of the improvements can be pointed to in the improvements to lane selection and decision-making - the vehicle tends to hesitate far less on V12.6, meaning the ride is a lot smoother. Many early V12.6 testers mentioned that it felt more like V13-mini than anything else.

Legacy Model S & X

We haven’t seen this update hit any legacy Model S and Model X vehicles just yet. We’re not sure whether Ashok’s statement of “generally” applies here - but it should. If you do get the update, please let us know.

Legacy Model S and Model X vehicles are still on an older FSD build and potentially won’t see another FSD update for a little while longer. While they do have the same FSD hardware as other vehicles, there are enough hardware differences that require a build specifically for these vehicles.

Release Date

FSD V12.6.1 is going out now to the redesigned Model S and X with HW3 and all Model 3 and Model Y vehicles with HW3. The initial wave went out last night, and we expect to see more later today or tomorrow. If this release ends up going “wide,” we should see much larger waves go out next week.

Follow

Follow

Follow

Tesla Issues Physical & OTA Recall for Hardware 4 Computer

Not a Tesla App

Tesla has issued a recall for some vehicles due to issues linked to a revision of the Hardware 4 (HW4) computer that was introduced this past November. This recall affects all vehicles with the revised hardware except for the Cybertruck, as it uses a slightly modified version of the HW4 system.

Many vehicle owners have reported a complete failure of their Autopilot computer, which not only leaves the vehicle without FSD and Autopilot but also disables many of the active safety features that make a Tesla what it is.

Let’s take a look at this recall - because the recall is partially physical and partially a software recall.

HW4 Failure

Tesla has identified the failure and flagged it to the U.S. NHTSA (but not Transport Canada or Euro NCAP at the time of publishing) as a recall.

Tesla mentions that the vehicle’s FSD computer circuit board may short circuit, resulting in the loss of the rearview camera image. The recall itself is focused on the rearview camera - as not having a functional rearview camera breaks a Federal Motor Vehicle Safety Standard. However, it does impact more than just the rearview camera.

As previously mentioned, users have experienced the full loss of Autopilot and FSD on these vehicles, as well as reduced traction control and removal of safety features such as automatic emergency braking, collision avoidance assist, and forward collision warnings. 

While the vehicle remains driveable, it is not necessarily a good idea to drive the vehicle with many of these features, especially with the reduced traction control noted by many people. We’ve already noted that while Tesla has been sending tow trucks to pick up some vehicles, it is also prompting owners to drive the vehicle to their service center in some cases.

Remedy

Unlike most of Tesla’s recalls, this one isn’t entirely an OTA recall, owing to the fact that people have already experienced circuit board shorts and are already waiting for replacements.

The software portion of the recall has already been mostly completed for any vehicles with Tesla software update 2024.45.25.3 or greater. However, a significant portion of the tracked fleet remains on update 2024.44.25.2 (approximately 20%), so we recommend updating your HW4 vehicle whenever you receive the notification.

On the hardware side, Tesla is immediately identifying and replacing hardware that has had circuit board failure. Tesla is also reaching out to customers who have expressed intermittent issues with Autopilot features, active safety features, or vehicle cameras that don't display correctly. If you have experienced any of these issues in a late 2024 vehicle, we recommend submitting a service ticket with Tesla to ensure that your vehicle isn’t affected by this recall.

Follow

Follow

Follow