2 comments
Looks very nice, but I can't find numerical gradient checks, which is helpful when verifying that backward pass is correct:<p><a href="https://github.com/markusheimerl/gpt/blob/main/transformer/attention/attention.c" rel="nofollow">https://github.com/markusheimerl/gpt/blob/main/transformer/a...</a>
I deleted the numerical checks a while back after confirming the backward pass is correct to keep the code base lean - running <a href="https://github.com/markusheimerl/gpt/blob/main/transformer/attention/test.c" rel="nofollow">https://github.com/markusheimerl/gpt/blob/main/transformer/a...</a> is also somewhat of a confirmation that the backward pass is correct, since an analytically incorrect backward pass cant fit perfectly to synthetic data.