I just recently watched some (not all) of this video "coding a machine learning library in c from scratch" and seems like he's going through a similar process in this blog as this video. I would recommend watching the video to get an idea of what the fundamentals of a ML library look like. From someone who has recently been getting interested in actually writing ML code and trying to make sense of it myself (from the perspective of just a typical backend engineer) it was very interesting to see. Previously my experience with ML libs (PyTorch specific) was writing my own Mini-GPT and training it on a small dataset using my own GPU (5090). Cool to see the behind the scenes and took away some o the handwaveyness... <a href="https://www.youtube.com/watch?v=hL_n_GljC0I" rel="nofollow">https://www.youtube.com/watch?v=hL_n_GljC0I</a>
If one wants to add the capability to reason about shape and shape compatibility, Barry Jay's FiSh would be an interesting detour.<p><a href="https://web.archive.org/web/20111015133833/http://www-staff.it.uts.edu.au/~cbj/Publications/shapes.html" rel="nofollow">https://web.archive.org/web/20111015133833/http://www-staff....</a><p>This was used in his shape aware language FiSh, for dealing with multidimensional arrays. Shape compatibilities were statically type checked, if I recall correctly. Shapes were also used to optimize the loops.<p>[Programming in FISh] <a href="https://link.springer.com/article/10.1007/s100090050037" rel="nofollow">https://link.springer.com/article/10.1007/s100090050037</a><p>[Towards Dynamic Shaping] <a href="https://www.researchgate.net/publication/265975794_Towards_D" rel="nofollow">https://www.researchgate.net/publication/265975794_Towards_D</a>...
Cool, but I find rather than just shapes and indexes, tensors with labels are much easier to use and reason about. E.g.:<p><pre><code> {
{user:bob, movie:"Heat"}:0.1,
{user:alice, movie:"Frozen"}:0.9,
{user:carol, movie:"Top Gun"}:0.3,
}
</code></pre>
<a href="https://docs.vespa.ai/en/ranking/tensor-user-guide.html" rel="nofollow">https://docs.vespa.ai/en/ranking/tensor-user-guide.html</a>
> A tensor is nothing but a flat array of numbers, plus some metadata telling you how to interpret those numbers as a multi-dimensional object.<p>Erm... many would disagree. I think what he means is just a multidimensional array.
Not in a general mathematical sense, of course. Rather carthesian tensors. Seems like for most libraries it's `numpy.ndarray`: <a href="https://docs.tinygrad.org/tensor/" rel="nofollow">https://docs.tinygrad.org/tensor/</a> or <a href="https://docs.pytorch.org/docs/main/tensors.html" rel="nofollow">https://docs.pytorch.org/docs/main/tensors.html</a>
To add to the confusion, Mathematicians and Physicists often disagree on what that word means. A Physicist's tensor is often a mathematicians tensor field.<p>An appropriate tensor can certainly be represented by a multidimensional array but not all multidimensional arrays are tensors. It depends on how that multidimensional array participates with others.<p>For example, (lat, long) is not a 2d vector although it can be represented as a 2 dimensional array.
Why does does the diagram say tensors are 3D?