Mining rig PCIe risers: x1 is enough. So why does your rig still crash?

How many PCIe lanes do you need for mining?

Usually one per GPU. A mining kernel loads its working data into the GPU’s own memory and grinds on it there. The host’s job is bookkeeping: hand the card a new job, collect the occasional share, report the hashrate.

Michael Bedford Taylor’s 2013 history of Bitcoin hardware puts it bluntly: in a GPU rig, the motherboard, processor, disk and RAM are “all but unused”. The same paper describes the trick early miners used to exploit that: a cheap riser cable that adapts the card’s x16 connector to a x1 slot, so one inexpensive board can feed five or six cards. Michael Bedford Taylor, “Bitcoin and the Age of Bespoke Silicon”, CASES 2013 (PDF).

Mining traffic after start-up is tiny next to those numbers. Loading 4 GB of mining data at start takes about 16 seconds even at Gen1 x1, and it happens once per run. Bandwidth matters again when the same card does AI inference or gaming; check that before you buy an ex-mining card.

The aha: your x1 riser has the bandwidth. Its cable and power may not have the margin

A USB-style mining riser puts a small board in the slot, a cable, and another board under the GPU. The cable looks like USB 3.0, but it carries PCIe signals, not USB. Never plug a riser into a USB port.

Every connector and every centimetre of cheap cable spends signal margin that the link was never designed to give away. PCIe is built to survive the odd bad bit: the receiver spots a corrupted symbol or packet, and the link layer quietly retransmits. Nothing crashes. The kernel’s AER guide says correctable errors have no impact on the interface’s functionality. Linux kernel: PCI Express Advanced Error Reporting guide.

That is the trap. A riser can be marginal for days while the rig “works”, quietly logging corrected errors. Then heat, vibration or a power dip pushes it past what retransmission can fix. The link retrains or drops, and the GPU vanishes. NVIDIA’s Xid catalog says Xid 79, “GPU has fallen off the bus”, is often caused by hardware failures on the PCIe link. NVIDIA: Xid catalog, including Xid 79 (GPU has fallen off the bus).

Power is the other half. Taylor records that early rigs pulled too much 12 V current through the slots and overloaded the motherboard’s power connector, so builders fed the risers straight from the power supply. Modern risers have their own power input for that reason. A loose, overloaded or adapter-chained riser feed can produce the same symptom as a bad cable.

What do AER “Physical Layer” corrected errors mean?

Linux names each correctable error and assigns it a layer. The short names below come from the kernel’s AER code; older kernels and documentation use longer labels such as “Receiver Error”. Linux kernel source: AER error names, layers and log rate limits.

Two details make these counters useful. First, an error is counted by the side that saw it. The kernel’s ABI notes say a problem endpoint may report zero while its link partner, often the root port above it, does the counting. Check both ends. Linux kernel ABI: PCIe AER statistics in sysfs.

Second, recent kernels rate-limit AER log messages per device, but the sysfs statistics keep counting. A quiet dmesg is not proof of a clean link. Linux kernel: PCI Express Advanced Error Reporting guide.

Mining rig crash triage on Linux and HiveOS

Hive OS is built on Ubuntu Linux, so these standard tools work from its shell. Hiveon: what Hive OS is. The dashboard tells you a GPU was lost; the kernel log and AER counters tell you why. Every command below only reads state. Replace the example addresses with yours.

1. Map each GPU to a bus address and a riser.

   lspci -nn | grep -i -e vga -e '3d controller'
   lspci -tv

   Look for one line per GPU, and label each riser with its address, such as 05:00.0. The tree shows which root port each card hangs off. A card missing from lspci has dropped off and not come back. If it reappears as rev ff or “unknown header type 7f”, read what unknown header type 7f means.

2. Read the trained link while the rig is mining.

   sudo lspci -vv -s 05:00.0 | grep -e 'LnkCap:' -e 'LnkSta:'

   Width x1 is expected on a x1 riser. Many GPUs lower their link speed when idle, which NVIDIA documents for its cards, so read it under load. NVIDIA: nvidia-smi documentation, PCIe link and replay counters. A speed below the slot setting while mining, or a link that changes between runs, is a quality clue. The glossary entry on link width and speed explains the two.

3. Count corrected errors on every device.

   for f in /sys/bus/pci/devices/*/aer_dev_correctable; do
     echo "$(basename "$(dirname "$f")") $(grep TOTAL_ERR_COR "$f")"
   done
   # Root port totals, for example the port above 05:00.0:
   cat /sys/bus/pci/devices/0000:00:1c.4/aer_rootport_total_err_cor

   Look for one device, or the root port above it, whose total keeps climbing while the others stay flat. If the files don’t exist, the device or platform isn’t exposing AER statistics to Linux. Treat that as “not captured”, not as zero. The AER glossary entry covers what a report can and can’t prove.

4. Read the kernel log around the crash.

   sudo dmesg -T | grep -i -e 'AER' -e 'PCIe Bus Error' -e 'Xid' -e 'fallen off'

   Look for bursts of severity=Corrected, type=Physical Layer naming one address before a crash (recent kernels print Correctable), and for NVIDIA’s Xid 79. If the rig rebooted and your system keeps a persistent journal, journalctl -k -b -1 shows the previous boot.

5. Read the GPU’s own replay counter.

   nvidia-smi -q | grep -i -e 'Replays Since Reset' -e 'Replay Number Rollovers'
   # amdgpu cards:
   cat /sys/bus/pci/devices/0000:05:00.0/pcie_replay_count

   A replay count that climbs on one card says its link is retransmitting. NVIDIA documents rollovers as forcing a retrain; amdgpu reports the total of NAKs generated and received. NVIDIA: nvidia-smi documentation, PCIe link and replay counters, Linux kernel: amdgpu sysfs attributes, including pcie_replay_count.

6. Swap one part, then measure for the same time.

   Note every counter, mine for a fixed hour, and note them again. Then swap exactly one thing: the riser cable for a known-good one, the riser board, its power lead, or the slot. Repeat the hour. If the errors move with the part, you’ve found it. If they stay with the slot, look at the board.

7. Buy margin, not bandwidth.

   Many mining motherboards let you force each slot to Gen1 or Gen2 in firmware setup. At x1, Gen1 still moves about 250 MB/s each way, and slower signalling gives the same cable more margin. Compare the error rate before and after. Feed each riser from the power supply with the connector its maker specifies, and keep a GPU and its riser on the same supply, a common rule among rig builders.

Annotated example: one bad riser in a six-GPU rig (synthetic)

This example is synthetic. The addresses, IDs and counts are illustrative, not captured from a real rig. It shows the pattern to look for.

# SYNTHETIC: totals from the loop in step 3, after one hour of mining
0000:00:1c.0 TOTAL_ERR_COR 0
0000:00:1c.4 TOTAL_ERR_COR 0
0000:03:00.0 TOTAL_ERR_COR 0
0000:04:00.0 TOTAL_ERR_COR 1
0000:05:00.0 TOTAL_ERR_COR 1873   <- one card, still climbing
0000:06:00.0 TOTAL_ERR_COR 0

# SYNTHETIC: kernel log for the same card (older-kernel wording)
pcieport 0000:00:1c.4: AER: Corrected error received: 0000:05:00.0
nvidia 0000:05:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
nvidia 0000:05:00.0:   device [10de:xxxx] error status/mask=00000001/00002000
nvidia 0000:05:00.0:    [ 0] RxErr
NVRM: Xid (PCI:0000:05:00): 79, GPU has fallen off the bus.

• TOTAL_ERR_COR 1873 on one card while its neighbours sit near zero: the problem is local to that card’s path, not the rig’s software.
• type=Physical Layer and RxErr: the receiver saw bad symbols. That is signal quality, the riser path’s territory.
• status/mask=00000001: bit 0 is set, which is Receiver Error. xxxx stands in for the card’s device ID.
• Xid 79 after the burst: the link went down and the driver lost the card. Read the error trail before the Xid, not just the Xid.

Next step: swap that card’s riser cable, rerun the same hour and compare. If the count stays flat under the same load, the cable was the weak link. If the errors follow the card to another riser, test the card on its own; the used-GPU checks work for that too.

When it isn’t the riser

• Flat AER counters, still crashing: look at clocks, undervolts and temperatures. An unstable memory overclock can crash a GPU without a single link error.
• Errors on every card at once: suspect something they share: the power supply, a shared cable or the motherboard.
• The card won’t come back after a crash: NVIDIA GPU missing or showing unknown header type 7f covers evidence and recovery.
• You want a second look at the evidence: import your lspci output and kernel log into the free Console workbench and compare a good capture with a bad one.

Questions

How many PCIe lanes do you need for a mining rig?

For most GPU mining, one lane per GPU is enough. Mining kernels keep their data in the card’s memory, so after start-up the PCIe link carries little traffic. A Gen1 x1 link still moves about 250 MB/s each way. More lanes matter for AI, gaming and video work, not for the hashing loop.

Do x1 risers reduce mining hashrate?

A working x1 link shouldn’t cap the hashing loop, because the work happens in VRAM. What does cost hashrate is crashes and restarts. If one card underperforms, check its AER and replay counters and its kernel log before blaming the lane count.

Why do my GPUs keep dropping off in HiveOS?

Hive OS reports that a GPU is lost; the evidence is often in Linux itself. Check the aer_dev_correctable counters and dmesg for corrected Physical Layer errors or NVIDIA Xid 79 on that card’s address. Then swap one part at a time: riser cable, riser board, power lead, slot.

Are AER corrected errors harmful?

One corrected error is harmless by design: the link retransmits and carries on. A steadily climbing count on one device means the link has little margin left, and it can come before a dropped GPU. Watch the rate, not the single event.

Should I set PCIe to Gen1 or Gen2 for a mining rig?

It is a reasonable test. A lower speed gives a marginal riser cable more signal margin, and at x1 even Gen1 has more bandwidth than the hashing loop needs. Compare error counts per hour before and after. If the errors don’t drop, look at power or a damaged part.

What does “GPU has fallen off the bus” mean?

It is NVIDIA’s Xid 79: the driver can no longer reach the GPU. NVIDIA says it is often caused by hardware failures on the PCIe link, and sometimes by failing GPU hardware or driver issues. On a rig, read the AER errors logged before it to see which link was struggling.