Writing triton kernels with Daniel Han from Unsloth + linear algebra
Linear algebra notes
Today I wrote some more linear algebra notes, on linear dependence, rank, basis and dimension. In the process I also wrote about vector spaces. I really begin to feel the value of these notes, both as an external memory to refer to later, but also as a learning process allowing me to really understand to summarize and re-explain.
Daniel Han
As part of S2S we had a guest lecture by Daniel Han from Unsloth on Triton and custom kernels. It was super interesting because it was working from first principles, and that’s the way I work. Instead of diving into triton’s DSL etc, he took a pen and a stack of paper and wrote neural network graphs of computations, deriving them by hand and explaining how custom training kernels were written as a way to speed-up backprop by using derivation tricks, using our knowledge of calculus, caching computations and results to achieve a better result than a torch.compile or autograd.
Light mode
I also took way too much time to get an OK-tier light mode for this website.