GPU has fallen off the bus? Read the log upwards from Xid 79.

Xid 79 is the last line of the story, not the first

The NVIDIA driver does not watch the PCIe link. It finds out the GPU is gone the next time it reads a GPU register and gets nonsense back. By then, the event that broke the link may be seconds old, and Linux may already have logged it.

That changes how you read the log. Most people search the Xid line and stop. The useful evidence is usually just above it: an AER report on the GPU’s upstream port, a Downstream Port Containment event, a link-down message or a failed resume. Those lines come from the PCIe side of the conversation, and they can tell you which kind of failure you have.

What “GPU has fallen off the bus” actually means

When nothing answers a PCIe read, the CPU side completes it with all-ones data: 0xFFFFFFFF. The Linux error-recovery documentation describes drivers spotting a dead device exactly this way. NVIDIA’s open kernel modules show the driver’s version of the test: when a register read returns all ones, it re-reads a GPU identification register, and if that is all ones too, it reports Xid 79 and marks the GPU as lost.

The same mechanism produces lspci’s “unknown header type 7f”: the header-type byte reads as 0xff, and lspci masks off the multi-function bit. If you see both messages for the same address, they are two views of one event. The NVIDIA header type 7f guide covers the driver and Secure Boot branches that look similar but are not bus failures.

NVIDIA’s Xid catalog says the event is often caused by hardware failures on the PCIe link, and also lists failing GPU hardware and driver issues. Its advice points the same way as this guide: “Reviewing system event logs and kernel PCI event logs may provide additional indications.” NVIDIA’s own driver README even documents a distribution kernel bug that produced this exact message. Losing PCIe access is a common end state for very different problems, so your job is to find which branch you are on.

Triage: an AER-first sequence

1. Save the log from the boot that failed.

   # After a reboot, the failed boot is usually -1. Check with --list-boots.
   journalctl --list-boots | tail -n 3
   sudo journalctl -k -b -1 --no-pager > failed-boot.txt

   Do this first. A reboot, a driver reload or a reinstall can rotate the evidence away. If the journal is not persistent on your distribution, save dmesg output before rebooting.

2. Find the GPU and the port above it.

   lspci -D -PP -nn -d 10de:

   The address before the last slash is the GPU’s link partner, for example 0000:00:01.1 in 0000:00:01.1/01:00.0. Errors on that port matter as much as errors on the GPU itself.

3. Read upwards from the Xid line.

   grep -n -E "Xid|fallen off|AER|DPC|pcieport|Link Down|link down|Surprise" failed-boot.txt

   Note the line number of the first Xid 79, then look for the first PCIe message that mentions the GPU or its port before it. Record the time gap between them.

4. Check the AER counters on both ends.

   for dev in 0000:01:00.0 0000:00:01.1; do
     grep -H . /sys/bus/pci/devices/$dev/aer_dev_* 2>/dev/null
   done

   After a reboot these start again from zero, so they tell you about this boot. Take a snapshot now and another after you reproduce the failure. Correctable errors climbing under load are a signal-quality clue even before anything falls off. The PCIe AER guide explains each severity and how to read a “PCIe Bus Error” line.

5. Make sure AER is not switched off.

   cat /proc/cmdline
   sudo dmesg | grep -iE "AER|_OSC"

   If the command line contains pci=noaer, Linux will not report the errors you are looking for. On some servers, firmware handles PCIe errors itself and reports them through its own hardware-error records instead. Either way, an empty AER search is not proof of a clean link.

6. Write down what the GPU was doing.

   Sustained load, idle, a few seconds after resume, a display hot-plug, or a VM shutting down: the timing is half the diagnosis. Note temperatures, the power supply, any riser and the exact driver and kernel versions.

7. If the GPU is still visible, read its own link counters.

   nvidia-smi -q | grep -i replay

   On a GPU that is still reachable, for example after a reboot and before the next failure, Replays Since Reset counts PCIe packet replays. NVIDIA documents that a replay rollover retrains the link. A count that climbs under load is the GPU-side view of the same signal-quality clue as correctable AER errors.

8. Collect NVIDIA’s report, then compare.

   sudo nvidia-bug-report.sh

   It writes nvidia-bug-report.log.gz in the current directory. If it hangs, NVIDIA suggests --safe-mode --extra-system-data. Review the bundle before sharing, because it contains system details. Attach it to an NVIDIA report with your own short timeline.

Match the evidence to a branch

None of these patterns proves a cause on its own. Each one tells you which experiment to run next.

If several GPUs share a PCIe switch and all of them fall off at once, look upstream of the switch: its link, its power and its port. One card failing alone points at that card’s own path.

D3cold: when the GPU is switched off on purpose

PCI power states run from D0 (on) to D3. In D3hot the device keeps its power and can be woken through configuration space. In D3cold the platform removes its main power altogether. Laptops with hybrid graphics use this to turn the discrete GPU off when nothing needs it, and the NVIDIA driver supports it as runtime D3 power management. Link power states such as ASPM L1 are a different kind of sleep; the ASPM guide separates the two.

That makes resume a delicate moment. A GPU that is unpowered or still in reset answers reads with all ones, just like one with a broken link. If it does not come back cleanly when the platform restores power, the driver’s check can report the same message, even though nothing physical failed.

cd /sys/bus/pci/devices/0000:01:00.0
grep -H . power/control power/runtime_status power_state d3cold_allowed
cat /proc/driver/nvidia/gpus/0000:01:00.0/power

The last file reports the driver’s runtime D3 status. NVIDIA’s README says its default setting enables fine-grained runtime D3 on Ampere-or-newer notebooks and leaves it off on desktops, so this branch is mostly a laptop story. NVIDIA’s docs do not single out D3cold as a cause of Xid 79; treat it as a hypothesis to test, not a diagnosis.

If the failures cluster after idle or resume, run one controlled comparison: keep the GPU powered for one session by writing on to its power/control file, which stops the kernel from runtime-suspending it, then repeat the trigger and compare. A write to that file lasts until reboot. Treat a clean run as a clue, not a fix, and check the laptop maker’s firmware notes before making it permanent; it costs battery life.

Annotated example: reading a failure from the top

[ 8123.402117] pcieport 0000:00:01.1: AER: Uncorrectable (Fatal) error message received from 0000:00:01.1
[ 8123.402130] pcieport 0000:00:01.1: AER: PCIe Bus Error: severity=Uncorrectable (Fatal), type=Data Link Layer, (Receiver ID)
[ 8123.402135] pcieport 0000:00:01.1: AER:   device [1022:1483] error status/mask=00000020/00000000
[ 8123.402139] pcieport 0000:00:01.1: AER:    [ 5] SDES                   (First)
#  ^ 1. The root port above the GPU saw the link go down unexpectedly
#       (SDES = Surprise Down). The first clue is on the port, not the GPU.
[ 8124.118850] NVRM: Xid (PCI:0000:01:00): 79, pid=2817, name=python3, GPU has fallen off the bus.
[ 8124.118861] NVRM: GPU 0000:01:00.0: GPU has fallen off the bus.
#  ^ 2. About 0.7 s later, the driver's next register read came back
#       all ones. The Xid reports the consequence.

The order tells the story. The port saw the link drop first, so the GPU did not “crash” and take the link with it; something interrupted the link or the card’s power. The next experiments are the physical ones: power connectors, the supply, a riser if there is one, and another slot. If the first message had been a long run of correctable errors, signal quality would be the stronger lead. If there were no PCIe messages and the failure came straight after resume, the D3cold branch would come first.

Getting the GPU back, and what not to do

A GPU that has fallen off the bus rarely comes back without a restart; NVIDIA’s catalog lists a bare-metal restart as the immediate action for Xid 79. Save the log, stop the workloads and reboot. If a warm reboot does not bring the GPU back, do a full power-off as the system maker describes, because a warm restart may not remove power from the card. Then confirm the fix with the original trigger, not just a clean boot.

• Don’t add pci=noaer. It hides the errors that tell you which branch you are on. The failure stays; the evidence goes.
• Don’t switch power management off globally on a hunch. Note that pcie_aspm=off does not even do what its name suggests: the kernel documentation says it leaves the firmware’s ASPM configuration unchanged, as the ASPM guide explains. Use one controlled comparison on the affected device, and keep it only if the evidence supports it.
• Don’t reinstall the driver first. A driver cannot fix a link that has gone down. Reinstalling is the right move only when the GPU is reachable and the module will not load.
• Don’t change three things at once. One change, one reproduction, one saved log. Otherwise a “fix” is just a coincidence you can’t repeat.

For servers with many GPUs, record which slot, switch and power rail each GPU uses. A failure that follows the slot rather than the card is a platform question; one that follows the card is a card question.

Questions

Is Xid 79 always a hardware failure?

No. It means the driver lost PCIe access to the GPU. A failing card is one cause, but so are power delivery, a riser or slot, signal quality, a failed resume from a low-power state and driver or firmware faults. The kernel messages logged before the Xid usually narrow it down.

Can a power supply cause “GPU has fallen off the bus”?

NVIDIA’s catalog does not name the power supply for Xid 79, but a card that loses power drops its link, and the driver then reports the GPU as gone. Evidence for this branch is a failure under heavy load, often with a Surprise Down or link-down message on the upstream port first. Check the connectors and compare with a known-good supply.

What is D3cold, and can it cause Xid 79?

D3cold is the PCI power state in which the platform removes a device’s main power, used for the discrete GPU in many hybrid-graphics laptops. A GPU that is unpowered or still in reset answers reads with all ones, so a resume that goes wrong can produce the same message. NVIDIA does not document this as a specific cause; test it with one controlled comparison.

Does pcie_aspm=off fix GPU has fallen off the bus?

Not reliably, and it may not do what you expect. The kernel documentation says pcie_aspm=off makes Linux leave the firmware’s ASPM configuration unchanged rather than forcing ASPM off. It is also global. Use any power-management change as one controlled comparison, and keep it only if the original trigger stops failing.

What are NVIDIA Xid errors?

Xid messages are error reports the NVIDIA driver writes to the kernel log, each with a number. They cover memory errors, GPU exceptions, driver timeouts and bus loss. NVIDIA’s Xid catalog lists each code with its likely causes. Xid 79 is the bus-loss code.

Can I recover the GPU without rebooting?

Rarely. Once the GPU stops answering on PCIe, the driver marks it lost, and NVIDIA’s Xid catalog lists restarting the machine as the immediate action for Xid 79. Sometimes a full power-off is needed. Save the kernel log before you restart, because it holds the evidence for the cause.