8 comments

  • bri3d9 hours ago
    Linked in the Bugzilla thread is a really nice in depth investigation of the same issue with high register aliases in a similar algorithm (Huffman coding) but in an entirely different product: <a href="https:&#x2F;&#x2F;fgiesen.wordpress.com&#x2F;2025&#x2F;05&#x2F;21&#x2F;oodle-2-9-14-and-intel-13th-14th-gen-cpus&#x2F;" rel="nofollow">https:&#x2F;&#x2F;fgiesen.wordpress.com&#x2F;2025&#x2F;05&#x2F;21&#x2F;oodle-2-9-14-and-in...</a> .<p>It&#x27;s concerning that Intel don&#x27;t seem to have been responsive to anyone with respect to this issue and it doesn&#x27;t appear to have an official errata yet, although Raptor Lake was the Intel CPU with voltage issues and basically random bit rot so I suppose it&#x27;s hard to tell if this is a silicon level errata caused by bad design or by some kind of post-manufacturing damage. Raptor Lake in general causes enough non-reproducible noise that I believe Firefox gave up on automated crash reports from it ( <a href="https:&#x2F;&#x2F;bugzilla.mozilla.org&#x2F;show_bug.cgi?id=1975808" rel="nofollow">https:&#x2F;&#x2F;bugzilla.mozilla.org&#x2F;show_bug.cgi?id=1975808</a> ).<p>EDIT: I read that Oodle article (which is SO good!) again and realized that their customer-provided reproduction of the bug was directly linked to boost clock speeds (the customer said that overclocking by 5% made it happen entirely reliably), so this is definitely not a &quot;the architecture has a 100% bug in it&quot; but rather some deeper issue with clock propagation that appears at edge cases.
    • m1324 hours ago
      Read the Oodle article in full, fantastic investigation indeed!<p>It also looks like there&#x27;s a slight difference in the unwanted effect both companies have reported, despite the bug being seemingly triggered the same way (mov touching the high byte):<p>- Oodle reports that a low byte is occasionally stored in the intended location.<p>- Mozilla&#x27;s fix suggests that a full 16-bit value is stored instead, corrupting an adjacent variable! This could have much more serious consequences.<p>Technically, this could still be the same exact bug. I found no mention of the order the output buffer was accessed in by the Huffman decoder debugged in the Oodle report, and, since it was a contiguous buffer, it&#x27;s easy to mistake an occasional out-of-bounds copy there for a copy from a wrong location. But if both analyses are correct, the behavior of high byte accesses on Raptor Lake is way less predictable than those fixes suggest. Haven&#x27;t managed to find an official erratum from Intel.
    • mtlmtlmtlmtl6 hours ago
      It&#x27;s very interesting because my 13900K has worked like a dream from day one and still to this day. Never had any of the voltage issues, never had any abnormal crashes in Firefox or any other software. I was undervolting it for a long while, so I wonder if somehow that saved me from the voltage issues before they were fixed?
      • Numerlor1 hour ago
        Undervolting would definitely help, and is the actual fix. The current Intel fixes were mostly just for the symptoms, as the main issue is high voltage+power when pushing high clocks, but they can&#x27;t actually fix that as it&#x27;d downgrade the advertised clocks the cpus were sold with
      • nubinetwork1 hour ago
        My 1360p and 13400 seem fine too. I applied the microcode and firmware updates when they came out... but I&#x27;m guessing it didn&#x27;t affect all skus equally for whatever magical reason.
      • whizzter3 hours ago
        I remember Puget systems pointed to this same thing when they analyzed the issues back in the day when it was blowing up.<p><a href="https:&#x2F;&#x2F;www.pugetsystems.com&#x2F;blog&#x2F;2024&#x2F;08&#x2F;02&#x2F;puget-systems-perspective-on-intel-cpu-instability-issues&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.pugetsystems.com&#x2F;blog&#x2F;2024&#x2F;08&#x2F;02&#x2F;puget-systems-p...</a>
      • nekzn1 hour ago
        My 13900K was affected by the widespread voltage issue and had to be replaced, but since then I have had zero problems with it.
  • Polizeiposaune9 hours ago
    Details of the errata from a comment in the diff:<p>&quot;Write both dist bytes as a single 2-byte store. This avoids the `movb %ch, [mem]` instruction pattern (store from high-byte register alias) that LLVM otherwise emits when dist arrives as a wide register. That pattern triggers the Intel Raptor Lake CPU errata, causing silent 2-byte stores that corrupt the adjacent `len` byte.&quot;
    • whadawha9 hours ago
      How did this get past validation at Intel?<p>This is worse than <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Pentium_FDIV_bug" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Pentium_FDIV_bug</a>
      • bri3d9 hours ago
        There&#x27;s another blog post going into more depth about the issue here: <a href="https:&#x2F;&#x2F;fgiesen.wordpress.com&#x2F;2025&#x2F;05&#x2F;21&#x2F;oodle-2-9-14-and-intel-13th-14th-gen-cpus&#x2F;" rel="nofollow">https:&#x2F;&#x2F;fgiesen.wordpress.com&#x2F;2025&#x2F;05&#x2F;21&#x2F;oodle-2-9-14-and-in...</a> where they speculate that it seems to relate to both other clock-related instability on specific Raptor Lake parts and possibly the overarching voltage control problems that the platform had early on; I can&#x27;t tell entirely from the bug reports whether the behavior reliably reproduces on 100% of Raptor Lakes but the indicators I&#x27;m reading point to that it doesn&#x27;t. It is concerning that Intel didn&#x27;t get back to Mozilla about it though, since it&#x27;s certainly a lot more than a one off.
      • userbinator8 hours ago
        &quot;validation? what validation?&quot;<p><a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=27244941">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=27244941</a><p>Edit: you should probably read the article I linked first.
    • moffkalast3 hours ago
      Common Raptor Lake L, add it to the pile of oxidation and overvoltage faults. This has to be the most faulty generation in Intel&#x27;s entire recent manufacturing history.
    • dmitrygr9 hours ago
      modifying source to avoid an assembly isntr isn&#x27;t a fix... this need a compiler fix most likely, or a microcode fix, if possible.
      • whadawha8 hours ago
        Anyone have knowledge of whether microcode can be patched on consumer grade Intel CPUs?
        • altairprime8 hours ago
          <a href="https:&#x2F;&#x2F;github.com&#x2F;intel&#x2F;intel-linux-processor-microcode-data-files" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;intel&#x2F;intel-linux-processor-microcode-dat...</a><p><pre><code> $ echo 1 &gt; &#x2F;sys&#x2F;devices&#x2F;system&#x2F;cpu&#x2F;microcode&#x2F;reload </code></pre> Hot-swappable, even. TIL!
          • xxs2 hours ago
            of course, it&#x27;s hot-swap material, the microcode is 1st deployed by the bios, then the OS can apply changes as well.<p>Just that it&#x27;s writable by $ (not #) feels awkward.
        • bri3d8 hours ago
          Yes? It is regularly; both the firmware or the OS can deliver updates depending on configuration. The Raptor Lake CPUs in question have gone through an enormous number of microcode revisions already due to quite famous voltage scaling issues; it&#x27;s unclear if this errata is fallout from or related to a similar root cause or just another issue with the processor.
        • __patchbit__7 hours ago
          At boot time, the following package provides the latest Intel CPU microcode data files on NetBSD.<p><pre><code> sysutils&#x2F;intel-microcode-netbsd </code></pre> dmesg shows<p><pre><code> cpu 0: ucode 0xf0-&gt;0xf6 cpu 1: ucode 0xf0-&gt;0xf6</code></pre>
        • throwaway20374 hours ago
          Why is this downvoted? (At the time of writing, the text is grey, so it has at least a few downvotes.)<p>This is a good question. As others have noted below, yes, and sometimes you can see kernel logging on start-up when the microcode is loaded.
          • varispeed4 hours ago
            Classic HN elitism. Oh this peasant doesn&#x27;t know the basics and dares to ask question that I deem stupid? DOWNVOTE
  • codedokode2 hours ago
    I looked at the Raptor Lake errata [1] and it looks pretty scary. What if someones builds an exploit on these errors?<p>This is why CPU designers should aim for simplicity. This is why RISC-V vector extension, which requires complicated logic, can become a source of implementation errors.<p>[1] <a href="https:&#x2F;&#x2F;edc.intel.com&#x2F;content&#x2F;www&#x2F;us&#x2F;en&#x2F;design&#x2F;products&#x2F;platforms&#x2F;details&#x2F;raptor-lake-s&#x2F;13th-generation-core-processor-specification-update&#x2F;errata-details&#x2F;" rel="nofollow">https:&#x2F;&#x2F;edc.intel.com&#x2F;content&#x2F;www&#x2F;us&#x2F;en&#x2F;design&#x2F;products&#x2F;plat...</a>
    • adrian_b2 hours ago
      All Intel, AMD or Arm-based CPUs and any other modern CPUs have dozens of errata, even if some CPU vendors keep them secret, instead of publishing them, as they should.<p>Fortunately, most of the erroneous behaviors are triggered only by very unlikely combinations of circumstances, some of which may even be impossible to happen in user programs, but only in operating system kernels.<p>Nevertheless, from time to time there are also serious errata, like the one discussed here, which can be triggered even by ordinary user programs. Sometimes, like here, such errata can be avoided by compilers patched to not generate the buggy instructions for the affected CPU models, assuming that it is known with certainty on which model of computer the compiled program will be executed (or using code dispatch at run time, based on the CPU model).<p>Simplicity in the CPU hardware may reduce the probability of hardware bugs, but it increases the probability of software bugs, because the missing hardware features must be implemented at a much greater cost in software, like in the case with the missing integer overflow detection of RISC-V, which causes most RISC-V programs to omit overflow checks, increasing the chances of undetected bugs.
      • camel-cdr17 minutes ago
        &gt; Simplicity in the CPU hardware may reduce the probability of hardware bugs, but it increases the probability of software bugs, because the missing hardware features must be implemented at a much greater cost in software, like in the case with the missing integer overflow detection of RISC-V, which causes most RISC-V programs to omit overflow checks, increasing the chances of undetected bugs.<p>Since I&#x27;ve got a SpacemiT K3 board my self now, I though I test it again:<p>I compiled microjs with both tinycc and chibicc, which where both compiled for the target platform with and without -ftrapv:<p><pre><code> Slowdown Zen1: tinycc: 1.34%, chibicc: -0.3% (slight speedup somehow?) Slowdown X100: tinycc: 0.1%, chibicc: 3.4% </code></pre> Last time I did full clang: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=47328214#47342362">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=47328214#47342362</a> And there was minimal slowdown (sometimes speedup) on x86, Arm and RISC-V. It was pointed out that llvm mostly uses size_t, however chibicc and tinycc use int as their default type, so there should be lots of overflow checking.
      • codedokode7 minutes ago
        And Rust omits overflow check under the same excuse, although overflow was a reason that allowed multiple Linux kernel vulnerabilities.
  • robin_reala8 hours ago
    Also worth reading this thread on the subject: <a href="https:&#x2F;&#x2F;mas.to&#x2F;@gabrielesvelto&#x2F;116630047156991279" rel="nofollow">https:&#x2F;&#x2F;mas.to&#x2F;@gabrielesvelto&#x2F;116630047156991279</a><p><i>Regarding the Raptor Lake bug I received a couple of messages from confused users that had read articles on Tomshardware and Neowin. They asked about erratas and microcode updates which puzzled me, because that was part of my early investigation into the bug and we know that the failure is not caused by a known errata and microcode updates cannot fix broken CPUs. So why did they ask? As it turns out it was slop. Both articles are 100% slop full of confusing and inaccurate claims.</i>
    • websg-x4 hours ago
      Because it&#x27;s a known problem. It&#x27;s called Vmin Shift Instability issues. The affected CPUs are broken. One needs to RMA the CPUs. Intel also extended CPU warranty for 2 more years.<p>Because there still are many broken CPUs out in the wild. Firefox works around the crash so the broken CPUs won&#x27;t flood the channel with crash reports.
    • samlinnfer1 hour ago
      Sorry it&#x27;s still not clear what he means? When a CPU is &quot;broken&quot;, is it already failing or is it &quot;broken&quot; in the sense it will fail?<p>For example:<p>Does he mean all existing 13th&#x2F;14th gen CPUs (prior to Intel&#x27;s discovery of the vmin issue) are broken in the sense that they are susceptible to damage and can only be replaced.<p>OR<p>Does he mean that the microcode updates, applied by Intel to existing CPUs that are susceptible to damage, will only slow degradation and the CPUs will eventually fail and can only be replaced.<p>OR<p>Is he saying the 13th&#x2F;14th gen CPUs which have already sustained damage, cannot be fixed by microcode updates.
      • websg-x1 minute ago
        Only the desktop 8p+16e cores(13600&#x2F;700&#x2F;900,14600&#x2F;700&#x2F;900)k are susceptible. The CPUs are safe with the fixed bios&#x2F;microcode. The notebook version of 8p+16e CPUs are also safe from vmin shift since notebook computer cannot go insane with voltage.<p>If the CPU is damaged already, the new microcode wont fixed the problem. It broken. You have to RMA the CPU.<p>The vmin shift instability is fixed. There are no new report of mass failures of 13th&#x2F;14th gen CPUs after the new bios&#x2F;microcode release.
  • userbinator9 hours ago
    WTF, Intel? This is reminding me of a very similar bug from 9 years ago: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=14630183">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=14630183</a><p>Clearly Intel needs to do far more extensive regression-testing, with things like demoscene productions --- especially the extremely size-optimised ones that can exercise the edge-cases much better than the usual &quot;compiler slop&quot;.
    • hsbauauvhabzb8 hours ago
      Intel knowingly sold defective cpus and denied the defect until reports hit critical mass. I don’t think they care.
      • userbinator8 hours ago
        &quot;knowingly&quot; is meaningless, as otherwise they wouldn&#x27;t even bother releasing errata lists; it&#x27;s more likely that they underestimated the severity or their planned obsolescence calculations happened to be more statistically favourable than reality.<p><a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=41041855">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=41041855</a>
        • close042 hours ago
          &gt; &quot;knowingly&quot; is meaningless<p>You’re intentionally muddying the waters with meaningless philosophy. Even the law makes the difference between “knowingly” (with knowledge, intention, premeditation) and “mistake”. They didn’t knowingly break the CPU but they knowingly launched it despite their own internal findings, and knowingly blamed others when this came out.<p>But here you are claiming that the company must deserve the benefit of the doubt on their intent.<p>No, “knowingly” is most definitely not “meaningless”. And anyone who’s not naive or bad intentioned should make that difference and take note. Every time a company gets away with it because an army of philosophers washes away any guilt or plays it down with meaningless distinctions it becomes one more reason to do it again. <i>Knowingly</i>.
        • hsbauauvhabzb7 hours ago
          The link you provided doesn’t match your comment, one of the comments in that thread points out that Intel blamed motherboards during the early stages.
        • moffkalast2 hours ago
          Yes let&#x27;s give a 50 billion dollar corporation flooded with MBAs the benefit of the doubt lmao. If GamersNexus can be believed, it was extremely deliberate to try and sweep it all under the rug.<p>I don&#x27;t think they&#x27;ve clarified which exact models of which serial number ranges are affected with oxidation to this day. It should&#x27;ve been a recall.
  • mike_hock9 hours ago
    Uh ... working around this in each and every piece of software sounds like a non-starter? Intel should be on the hook to fix this.
    • Polizeiposaune9 hours ago
      Use of the &quot;h&quot; register slices (bits 8..15) by compilers is thankfully pretty rare -- otherwise this would have been noticed much sooner!<p>Agner Fog&#x27;s optimization guide says &quot;Any use of the high 8-bit registers AH, BH, CH, DH should be avoided because it can cause false dependences and less efficient code.&quot;
      • userbinator8 hours ago
        <i>Use of the &quot;h&quot; register slices (bits 8..15) by compilers is thankfully pretty rare</i><p>That&#x27;s unfortunate, because it&#x27;s precisely why things like this will keep happening.<p><i>Agner Fog&#x27;s optimization guide says &quot;Any use of the high 8-bit registers AH, BH, CH, DH should be avoided because it can cause false dependences and less efficient code.&quot;</i><p>The sad vicious cycle of compilers not exercising the hardware, and then the hardware designers not paying attention. Using the high 8-bit registers and &quot;implicitly merging&quot; them is one of the ways to reduce the number of instructions and thus improve size optimisation.
        • fuhsnn5 hours ago
          &gt; The sad vicious cycle of compilers not exercising the hardware<p>There could theoretically be instruction selection passes that are biased toward rare instructions, specialized for fuzzing hardware, I&#x27;m surprised Intel doesn&#x27;t already do that.
  • charcircuit8 hours ago
    Hopefully this bug is getting handled upstream in a microcode update or a compiler fix to avoid emitting such instructions. Just a comment mentioning that you should not emit a particular instruction is not a strong guarantee.
    • progval5 hours ago
      According to <a href="https:&#x2F;&#x2F;bugzilla.mozilla.org&#x2F;show_bug.cgi?id=1950764#c23" rel="nofollow">https:&#x2F;&#x2F;bugzilla.mozilla.org&#x2F;show_bug.cgi?id=1950764#c23</a> , it is not getting fixed.
  • WaylandYang6 hours ago
    [flagged]