This post is a classic! Also recommended: Horace also gave a related talk (covering the high-level picture of modern ML Systems) at Jane Street in Dec 2024 <a href="https://www.youtube.com/watch?v=139UPjoq7Kw" rel="nofollow">https://www.youtube.com/watch?v=139UPjoq7Kw</a>
One thing people seems not to acknowledge, and this post made it super clear is that NVIDIA kept their lead extremely well in a few years of very high growth. The TFLOPs, the bandwidth, the interconnect mentioned in this post continues to grow at exponential rate with no sign of stopping yet. This is a 30-year-old incumbent reminding you. The willingness to compete from NVIDIA is just simply remarkable.
I feel like there is no portable advice for performance. A torch model exported as onnx is a different model.<p>That onnx model run using onnxruntime with cuda ep is a different model than the one run with TRT ep.<p>And even among the same runtime, depending on the target hardware and the memory available during tuning, the model behaves differently. It is a humongous mess
I'd want to see more about the failure modes. Production systems need graceful degradation more than optimal performance.
> in the time that Python can perform a single FLOP, an A100 could have chewed through 9.75 million FLOPS<p>wild
Why are we comparing a programing language and a GPU. This is a category error. Programing languages do not do any operations. They perform no FLOPs, they are the thing the FLOPs are performing.<p>"The I7-4770K and preform 20k more Flops than C++" is an equally sensible statement (i.e. not)
> Why are we comparing a programing language and a GPU.<p>You are taking the statement too literally and forgetting it's a figure of speech, specifically metonymy.<p>When the author says it's millions of flops faster in a gpu than in an interpreteted programming language, it's not comparing them directly, but algorithms that run in them, so the substitution is the algorithms for the tools used to implement/run them.<p>It makes sense if you say "running similar logic -- like multiplying vectors and matrices -- on the CPU is millions of flops slower then on the GPU". There is no category error there.
the sentence is ambiguous because "Python" can mean python + a certain library and even a different Python implementation<p>but I find it illuminating to compare what a certain hardware can do in principle (what is possible) vs what I can "reach" as programmer within a certain system/setup<p>in this case NVIDIA A100 vs "Python" that does not reach a A100 (without the help of CUDA and PyTorch)<p>another analogy:<p>I find it useful to be able to compare what the fastest known way is to move a container from A to B using a certain vehicle (e.g. truck) and how that compares to how fast a person that can not drive that truck can do it + variants of it (on foot, using a cargo bike, using a boat via waterway, …)<p>I'm also interested in how much energy is needed, how much the hw costs and so on<p>Often there are many ways to do things, comparing is a great starting point for learning more
> This is a category error.<p>Okay, but surely you know what they actually mean right, or are you being willfully obtuse? They are comparing CPython (the <i>main</i> python implementation)'s implementation that runs on the CPU with a kernel running on the GPU.
I’m not 100%, in context. Sorry for the big quote:<p>> Overhead is when your code is spending time doing anything that's not transferring tensors or computing things. For example, time spent in the Python interpreter? Overhead. Time spent in the PyTorch framework? Overhead. Time spent launching CUDA kernels (but not executing them)? Also... overhead.<p>> The primary reason overhead is such a pernicious problem is that modern GPUs are really fast. An A100 can perform 312 trillion floating point operations per second (312 TeraFLOPS). <i>In comparison, Python is really slooooowwww. Benchmarking locally, Python can perform 32 million additions in one second.</i><p>> That means that in the time that Python can perform a single FLOP, an A100 could have chewed through 9.75 million FLOPS.<p>> <i>Even worse, the Python interpreter isn't even the only source of overhead - frameworks like PyTorch also have many layers of dispatch before you get to your actual kernel.</i> If you perform the same experiment with PyTorch, we can only get 280 thousand operations per second. Of course, tiny tensors aren't what PyTorch is built for, but... if you are using tiny tensors (such as in scientific computing), you might find PyTorch incredibly slow compared to C++.<p>Emphasis mine.<p>It’s all a bit jumbled up. I get that he was going for an informal tone and this isn’t exactly a benchmark. But I’m still not sure, based on the second emphasized part I think the “bad” measurements are coming from Python+PyTorch but with too-small workloads, and dispatching to CPU, maybe? But the first one looks like naive Python loops.
This statement makes zero sense
re comments:<p>yes of course this is apples to oranges but that's kind of the point<p>it shows the vast span between specialized hardware throughput IFF you can use an A100 at its limit vs overhead of one of the most popular programming languages in use today that eventually does the "same thing" on a CPU<p>the interesting thing is why that is so<p>CPU vs GPU (latency vs throughput), boxing vs dense representation, interpreter overhead, scalar execution, layers upon layers, …
Which, lets be honest, is probably still being orchestrated by Python somewhere.<p>Python is 9.75 million times faster than Python.
I was researching if there was much benefit to using Rust or C++ over Python for AI, and turns out, the GPU doesn't care once the instructions are in because its an entirely different spec running on the GPU. The only thing you might save on is "startup" costs of getting your code into the GPU I guess? I assume that time cost is miniscule though, once its all in memory, nobody cares that you spent any time "booting it up" any more than how long Windows takes these days.
As long as you don't keep calling out to the CPU, that is.<p>Tool calling, searches, cache movement if used, and even debug steps all stall the GPU waiting for the CPU.<p>There was a test of turning one of the under 1B Qwen3+ models into a kernel that didn't stall by the CPU as one GPU pass that saw quite a bit f perf lift over vLLM, I believe, showing this is an issue still.<p>Its been a month, so I don't remember more details than this.
you can port anything python is doing with a couple prompts into rust/c++, including parity validation. when the barrier to migrating is that thin, you are losing money and time even continuing to talk about it. python is miserably slow, so dont let it touch any part of your system. no snakes in the house.
Pytorch dataloaders are often horribly inefficient, a lot of stuff there can benefit from Rust/C++
Single core vs multi core accounts for much of this
Not really. GPU many cores, at least for fp32, gives you 2 to 4 order of magnitudes compared to high speed CPU.<p>The rest will be from "python float" (e.g. not from numpy) to C, which gives you already 2 to 3 order of magnitude difference, and then another 2 to 3 from plan C to optimized SIMD.<p>See e.g. <a href="https://github.com/Avafly/optimize-gemm" rel="nofollow">https://github.com/Avafly/optimize-gemm</a> for how you can get 2 to 3 order of magnitude just from C.
How does x.cos().cos() work faster than doing two cos calls separately? Like the first cos call returns a tensor either way, the only difference is that it's not assigned to a variable. But how is it even possible know that difference in python?
It’s really not a concept you can express in idiomatic Python very easily. This comes from the actual generated assembly involving copies from global GPU memory into registers (slow, bandwidth saturates quickly) and back in between the cosines. If you can avoid the intermediate roundtrip that cuts the cost approximately in half.
Yeah, that part should not be read literally; `x.cos().cos()` and `x1 = x.cos(); x2 = x1.cos()` both launch the same number of kernels (two in unfused/eager mode, one in fused/torch.compile, see this test notebook [1]). I think the author chained the two cos calls to symbolize the idea of combining them (without exposing the intermediate result), but chaining the two cos calls doesn't literally trigger operator fusion.<p>[1] <a href="https://colab.research.google.com/drive/13a4Y-ko6QLMPAhBz64cRkdqxOT7LFHaF?usp=sharing" rel="nofollow">https://colab.research.google.com/drive/13a4Y-ko6QLMPAhBz64c...</a>
Deep learning is just glorified linear algebra. Master the progression: Feed-forward CNN RNN LSTM Attention. You don't even need a GPU to understand the climax; Karpathy’s llama2.c implements a full transformer inference engine in just ~300 lines of C using SIMD pragmas for CPU execution.
I wish more people pursued that approach to teaching neural networks.<p>First teach what the network does and why, writing it as a loopy, inference-only Python function. Explain training only in an abstract way, E.G. with the "take a random weight, twist it a little and see if the loss improves" algorithm. This lets you focus on the architecture and on why it is what it is.<p>Then, teach the intuitions behind derivatives and gradient descent. You don't need the entirety of calculus, there's no benefit to knowing how a sequence or limit works if you ) only want to understand neural networks. With autograd, you won't be manually doing derivatives of weird functions either, so intuitive understanding is a lot more important than doing dozens of traditional calculus exercises on paper like it's the 1800s. You could probably explain the little bit of calculus you need in an hour or two, even to somebody with a 12-year-old's understanding of math and a good bit of programming knowledge.<p>Only when people understand the training and inference, implemented with loops and descriptive variable names, teach the tensor, explain how a modern CPU and GPU works (because many programmers still think a modern computer is just a much faster 6502), and then teach the tricks we use to make it fast.
So you created a new account to blatantly plagiarize another comment from this same page? What's even going on here?
Needs 2022 in title
Right now, all I know how to do is pull models from Hugging Face, but someday I want to build my own small LLM from scratch
If you want a written resource I have a blog post about the mathematics behind building a feed forward from scratch, <a href="https://max-amb.github.io/blog/the_maths_behind_the_mlp/" rel="nofollow">https://max-amb.github.io/blog/the_maths_behind_the_mlp/</a>. Kinda focuses on translation from individual components to matrix operations.
If you aren't already aware, Karpathy has several videos that could get you there in a few hours <a href="https://www.youtube.com/@AndrejKarpathy" rel="nofollow">https://www.youtube.com/@AndrejKarpathy</a>
very thanks!
Also check out his nanochat repo. I used the repo, claude and shadeform to train my own mini model for about $300. Would have been less but I screwed up and let the cloud gpu rental run for a few hours even though the training run errored out.<p>Of course the model was dumber than GPT2 but still it was a great learning experience.
It’s just linear algebra. Work your way from feed forward to CNN to RNN to LSTM to attention then maybe a small inference engine. Kaparthy’s llama2.c is only ~300 lines on the latter and it pragma simds so you don’t need fancy GPUs
[flagged]
>For example, getting good performance on a dataset with deep learning also involves a lot of guesswork. But, if your training loss is way lower than your test loss, you're in the "overfitting" regime, and you're wasting your time if you try to increase the capacity of your model.<p><a href="https://arxiv.org/abs/1912.02292" rel="nofollow">https://arxiv.org/abs/1912.02292</a>
Generally, posting a link-only reply without further elaboration comes across as a bit rude. Are you providing support for the above point? Refuting it? You felt compelled to comment, a few words to indicate what you’re actually trying to say would go a long way.