DeepSeek Bypasses CUDA: A New Challenge for NVIDIA’s Moat?

Just as NVIDIA was recovering from the $4 trillion plunge caused by DeepSeek-R1, it faces new pressure. According to hardware media Tom’s Hardware, the latest hot topic is that DeepSeek has bypassed CUDA, using a lower-level programming language for optimization.

This time, more details from the DeepSeek-V3 paper have been uncovered. Mirae Asset Securities Research (Mirae Asset) analyzed that the hardware efficiency of V3 is 10 times higher than that of Meta, which can be summarized as “they rebuilt everything from scratch”.

When training DeepSeek-V3 using NVIDIA’s H800 GPU, they modified 20 of the 132 streaming multiprocessors (SMs) to handle server-to-server communication instead of computation tasks.

This effectively bypassed the hardware’s communication speed limitations.

This operation was implemented using NVIDIA’s PTX (Parallel Thread Execution) language, not CUDA. PTX runs at a level close to assembly language, allowing fine-grained optimizations such as register allocation and Thread/Warp-level adjustments.

This kind of programming is very complex and difficult to maintain, so the industry generally uses high-level programming languages like CUDA.

In other words, they have taken optimization to the extreme.

Some netizens commented that if a group of people bypass CUDA and use PTX because they think it is too slow, they must be former quantitative traders.

An Amazon engineer raised a soul-searching question: Is CUDA still a moat? Such top-tier labs can effectively utilize any GPU.

Some even began to imagine what would happen if “New Source God” DeepSeek open-sourced a CUDA alternative……

Now imagine if DeepSeek open sources a CUDA alternative

— Saurabh Kumar (@drummatick) January 28, 2025

So, is this really the case?

Did DeepSeek Really Bypass CUDA?

First, it is important to clarify that PTX is still a technology within NVIDIA’s GPU architecture. It is the intermediate representation in the CUDA programming model, used to connect CUDA high-level language code with GPU low-level hardware instructions.

PTX is similar to assembly language, with code looking something like this:

△ From tinkerd.net

In the actual compilation process, CUDA code is first compiled into PTX code, which is then compiled into machine code (SASS, Streaming ASSembler) for the target GPU architecture.

CUDA provides a high-level programming interface and toolchain to simplify developers’ work. PTX, as an intermediate layer, acts as a bridge between high-level languages and low-level hardware.

Additionally, this two-step compilation process also gives CUDA programs cross-architecture compatibility and portability.

Conversely, directly writing PTX code, as DeepSeek did, is not only very complex but also difficult to port to different GPU models.

Industry practitioners have said that code optimized for H100 may not work well, or may not work at all, on other models.

So, DeepSeek’s PTX-level optimization does not mean they have completely left the CUDA ecosystem, but it does show they have the ability to optimize other GPUs.

In fact, we can see that DeepSeek has already closely collaborated with teams from AMD, Huawei, and others, providing support for other hardware ecosystems in the first place.

One More Thing

Some have suggested that making AI proficient in writing assembly language is a direction for AI self-improvement.

We do not know if DeepSeek internally used AI to assist in writing PTX code—

But we have just witnessed DeepSeek-R1’s code significantly improving the running speed of large model inference frameworks.

In the llama.cpp project, a new PR request uses SIMD instructions (which allow a single instruction to process multiple data) to significantly improve the running speed of WebAssembly on specific dot product functions. The submitter said:

99% of the code in this PR was written by DeepSeek-R1. All I did was develop tests and write prompts (with some trial and error). Yes, this PR aims to prove that large models can now write good low-level code and even optimize their own code.

The founder of the llama.cpp project checked this code and said it was “more than expected.”

This blew up way more than expected. Make sure to follow @ngxson‘s work – he is doing a lot of great stuff for llama.cpp.

Shout out to the great team at @huggingface for supporting ggml!

— Georgi Gerganov (@ggerganov) January 27, 2025