i9-14900Ks the last bastard

Intel has been costing me money for a while now. To be fair, it has also made me money in the stock market with those delicious dividends, so apparently this is a toxic relationship with benefits.

The fact is, you've probably heard about the i9-14900K disaster. Unless you were high or in some kind of trance, you probably heard something about it. These processors became a massive headache for a lot of people because of a microcode-related degradation problem.

And guess what: I bought eight of them at the end of 2023.

Man, I've never won the lottery. But somehow, when it comes to getting lucky with defective hardware, I'm fucking gifted.

And the story of those eight doesn't get any prettier. It gets worse.

At the time, I imported the processors. Four arrived. Four disappeared in transit. I had to get into endless arguments with the logistics system until, many months later, they finally agreed to reimburse me so I could buy replacements.

Honestly, if I had known the headache those little beauties were going to give me, I would've asked them to set fire to the ship they came on.

The problem had a weird name:

Kernel-Power 41 (0x00020001)

Something like the name of Elon Musk's fortieth child.

Basically, the bug worked like this:

PC uses very little power. PC crashes.

This shit nearly drove Intel into bankruptcy.

I managed to replace seven of the eight processors that failed, but I refused to replace one of them. That one started failing two weeks ago, roughly a year and a half after the last one had been replaced.

And somehow this final CPU pissed me off more than all the others.

It had survived this long.

I needed to cure it.

I was going to cure it.

Knowing the problem had something to do with power consumption, blah blah blah, and knowing that my BIOS was properly updated while the problem still persisted, I came up with a plan.

The machine

Two things on that list matter more than they seem.

The microcode was already newer than 0x12B, which was Intel's mitigation for the degradation problem affecting 13th- and 14th-generation CPUs.

And HVCI being enabled meant the Windows hypervisor was active all the time, a detail that would later explain the error code.

Event Viewer had recorded three reboots on August 28, 29 and 30.

All three showed Kernel-Power 41, followed by bugcheck 0x00020001, which is HYPERVISOR_ERROR.

Two initial checks eliminated some obvious suspects.

No WHEA events. Windows had not recorded a single hardware error in the previous 72 hours. That weighed against a PCIe bus failure or a classic CPU machine-check.

Nothing in the minutes before the crashes. I filtered the System and Application logs for the three minutes before each crash, looking for errors and warnings.

Nothing.

All three crashes were instantaneous. The system didn't even have time to leave a trail before falling over.

That already told me something.

A driver that slowly shits itself usually leaves evidence behind, normally some kind of timeout.

Here, there was nothing.

The minidumps

Windows stores minidumps in:

C:\Windows\Minidump

But the folder requires administrative privileges even to list its contents, and the machine didn't have a debugger installed.

So I solved both problems at once.

I installed WinDbg with:

winget install Microsoft.WinDbg

Then I used the cdb.exe bundled with it and pointed it at Microsoft's symbol server:

set _NT_SYMBOL_PATH=srv*C:\symbols*https://msdl.microsoft.com/download/symbols
cdb.exe -z <dump>.dmp -c "!analyze -v; q"

The first result was specific enough to look like an answer:

BUGCHECK_CODE:  20001
Arg1: 0000000000000011, An unhandled exception occurred in the hypervisor.

STACK_TEXT:
    nt!KeBugCheckEx
    nt!HvlSkCrashdumpCallbackRoutine
    nt!KiProcessNMI
    nt!KxNmiInterrupt
    nt!KiNmiInterrupt
    nt!HvcallInitiateHypercall
    nt!HvlRequestProcessorHalt
    nt!PpmIdleExecuteTransition
    nt!PoIdle
    nt!KiIdleLoop

MODULE_NAME: intelppm
IMAGE_NAME: intelppm.sys
FAILURE_BUCKET_ID: 0x20001_1_..._intelppm!HvRequestIdle

The stack is read from bottom to top.

The processor enters an idle state through KiIdleLoop and PoIdle. Intel's power-management driver asks the core to sleep through intelppm and HvlRequestProcessorHalt. The call reaches the hypervisor, an NMI fires, and the whole thing turns into a bugcheck.

The machine was crashing while sitting there doing nothing.

Two of the three crashes happened in the middle of the night, while the PC was idle.

The pattern that changed the diagnosis

If the story had ended there, the obvious conclusion would've been:

The problem is intelppm.sys.

Except the crashes kept happening, and the error codes started changing.

That last one was the useful bastard.

The stack showed a completely ordinary file operation:

NtCreateFile
→ IopCreateFile
→ ObpLookupObjectName
→ IofCallDriver

And then an NMI interrupted it.

The process running at the time was Resilio Sync, which had done absolutely nothing wrong. It just happened to be the poor asshole standing next to the body when the police arrived.

This is the reasoning that finally cracked the case:

A broken driver usually fails in the same way.

Different bugcheck codes appearing in completely unrelated subsystems point to the layer underneath all of them.

The CPU screws up a calculation wherever it happens to be working, and the crash gets blamed on whatever happened to be nearby.

The NVIDIA driver crash is the most misleading example.

Graphics code is heavy and runs constantly, so it's often where an unstable CPU trips first. A lot of people end up replacing their GPU because of this.

And HYPERVISOR_ERROR follows the same logic.

With HVCI enabled, the hypervisor is always active. When the CPU produces an invalid result, the hypervisor may simply be the first thing to notice and bring the system down.

The hypervisor was the detector.

It wasn't the culprit.

What it wasn't

The dead ends are worth documenting because they may save someone else from wasting the same damn time.

Outdated microcode.
The machine was already on 0x147, newer than Intel's 0x12B mitigation. There wasn't another magical update left to fix this.

Minimum processor state.
Windows was already configured with the minimum processor state at 100%, and the crashes still happened.

This is a common point of confusion.

That setting controls the P-state, meaning frequency.

The failure was happening when entering a C-state, meaning the core was going to sleep.

Different layers. One does not protect you from the other.

ParkControl and similar tools.
For the same reason, they wouldn't help. Those tools change core parking and frequency scaling, not C-states.

And no software utility is going to repair an electrical failure.

Memory.
I ran MemTest86 for an entire night and got exactly zero errors.

That test wasn't optional. I was on the fourth processor, and when four parts fail in basically the same way, it's perfectly reasonable to start wondering whether some other component is the actual asshole.

With the RAM clean, the CPU moved right back to the top of the suspect list.

Factory BIOS defaults.
I reflashed the BIOS and loaded Intel's defaults.

That did have a real effect, but only a partial one.

The crash frequency went from three crashes in 48 hours to one every four days.

It spread them out.

It didn't cure anything.

Vmin shift

At that point, the whole picture matched the known degradation problem affecting Intel's 13th- and 14th-generation processors, the so-called Vmin shift.

The chip degrades over time and starts needing more voltage than it used to need in order to remain stable.

The problem shows up first where voltage is lowest, which is exactly when the CPU is idle.

That explained both situations:

At first, it crashed while idle, when voltage dropped.

Later, as degradation progressed, it also began crashing under load.

The fix

The solution was to give back the voltage margin that degradation had taken away.

A small increase in core voltage.

On the ASRock BIOS, the path was:

OC Tweaker → Voltage Configuration

Then:

• CPU Core/Cache Voltage → Offset Mode
• Offset Voltage (mV) → 50
• CPU Core/Cache Load-Line Calibration → Level 3
• Intel Performance profile kept enabled:
  • PL1: 125 W
  • PL2: 253 W
  • ICCMAX: 307 A

Two stupid little details cost me an entire failed attempt.

First, the field is in millivolts, not volts.

Entering 0.050 doesn't work.

The correct value is:

50

Second, changing the mode to Offset is not enough.

On my first attempt, I changed the mode but left the actual value field on Auto, which means that in practice no offset gets applied at all.

I found that out in HWMonitor, which showed:

IA Voltage Offset     0 mV
Voltage 0             +0.00 Volts (IA Offset)

Checking whether the setting actually took effect is just as important as changing the damn setting.

Making the evidence collect itself

I didn't want to depend on noticing every crash manually, so I created a Windows Scheduled Task triggered by the Kernel-Power 41 event and configured it to run as SYSTEM.

Whenever a crash happens, it records the date, bugcheck code and parameters in a log, then copies the minidump to an accessible folder.

That way, the evidence is already waiting for me before I even sit down at the machine.

And then... nothing

The baseline was September 11 at 11:28, on the first boot with the offset actually active.

Eleven days later, the system log had recorded zero bugchecks.

No Event 1001.

No new minidumps.

Three Kernel-Power 41 events did appear on September 16, 17 and 19, but all of them had:

Bugcheck=0x0
PowerButton=0

That's the signature of a power loss, and those were shutdowns I had caused myself.

A real crash always left a non-zero bugcheck code and a dump.

The full progression looked like this:

So, after eight cursed processors, seven replacements, disappearing shipments, random bugchecks, fake suspects, hypervisor errors, NVIDIA getting framed for a crime it didn't commit, and more time staring at Event Viewer than any human being should willingly spend...

the last bastard finally shut up.