PCIe AER errors: count them before you silence them.

What PCIe AER errors mean: corrected, non-fatal and fatal

PCIe has a built-in error reporter called Advanced Error Reporting, or AER. When a device notices something wrong on its link, it sends an error message up to the Root Port above it, and Linux prints what it found. Every report carries a severity. The severity tells you who has already cleaned up the mess.

Kernels up to 6.7 spell the first row “Corrected”; 6.8 and later say “Correctable”. Same event, new word. The kernel's AER driver guide describes the classes and the recovery flow.

The type= part names the layer of the PCIe stack that caught the problem:

• Physical Layer means RxErr, a receiver error: bits arrived that didn't decode cleanly. Start with signal integrity: the slot, riser, cable, connectors, retimers and link speed.
• Data Link Layer covers BadTLP, BadDLLP, Rollover and Timeout. A packet failed its CRC check, or an acknowledgement didn't arrive in time, so the packet was sent again.
• Transaction Layer is nearly everything else: advisory NonFatalErr, CorrIntErr and HeaderOF on the corrected side; completion timeouts, unsupported requests and malformed packets on the uncorrectable side.

The aha: a corrected error is a retry you didn’t notice

Here's the part most forum answers skip. PCIe was designed on the assumption that bits will occasionally flip. Every packet carries a sequence number and a CRC. The receiver acknowledges good packets and rejects bad ones. The sender keeps a copy of everything in flight in a replay buffer until it's acknowledged.

When a packet arrives damaged, the receiver flags it, the sender replays it, and the software above never notices. That's what “corrected” means. Not “nothing happened”, but something went wrong and the link paid to fix it. Each replay costs a little bandwidth and latency. If the same packet has to be replayed too many times, the replay counter rolls over (Rollover) and the link retrains.

So one corrected error is a non-event. The interesting question is the rate:

• A burst tied to boot, resume or hot-plug, then silence: note it, and check whether it repeats on the next transition.
• A steady trickle that grows with load, temperature or link speed: the link is running close to the edge of its margin.
• A rate that changes when you swap the slot, riser or cable: you've found the weak part.

Two kernel details make the count more trustworthy than the log.

The log is rate-limited. Since Linux 6.16, AER messages are throttled per device: by default 10 messages every 5 seconds for correctable errors, and separately for non-fatal ones. Fatal errors are never throttled. In current kernel source, the counters and the trace event are still updated when a line is suppressed, so dmesg can undercount while sysfs keeps score.

The counter lives on whoever saw the error. A receiver error is counted by the device whose receiver caught it, which is often the link partner. The sysfs documentation warns that a problem endpoint can show all zeros while its Root Port counts the errors. Always read both ends of the link.

How to read a “PCIe Bus Error” line

Here's a corrected receiver error in the format most forum posts show. It's a synthetic example built from the kernel's print format: an FPGA endpoint at 01:00.0, sitting below Root Port 00:1c.0.

# Synthetic example, Linux 6.7 and earlier
pcieport 0000:00:1c.0: AER: Corrected error received: 0000:01:00.0
xdma 0000:01:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
xdma 0000:01:00.0:   device [10ee:9038] error status/mask=00000001/00002000
xdma 0000:01:00.0:    [ 0] RxErr

1. Line 1 comes from the Root Port. It's the messenger: it received an error message, and the address after “received” is the device that sent it.
2. Line 2 gives the severity, the layer and the agent. Receiver ID means the reporting device is the one whose receiver saw bad bits. Those bits were sent by the other end of the link, so the reporter is the victim, not necessarily the culprit.
3. Line 3 gives vendor and device IDs, then two raw registers. Status 00000001 is bit 0, Receiver Error. Mask 00002000 is bit 13, Advisory Non-Fatal Error, which the PCIe specification masks by default.
4. Line 4 decodes each set status bit by name. You may also see (First) on this line. The first-error pointer belongs to uncorrectable errors, so don't read anything into it on a corrected one.

On 6.8 and later, the same event reads slightly differently:

# Synthetic example, Linux 6.8 and later
pcieport 0000:00:1c.0: AER: Correctable error message received from 0000:01:00.0
xdma 0000:01:00.0: PCIe Bus Error: severity=Correctable, type=Physical Layer, (Receiver ID)

The layout keeps evolving in newer kernels, so search for the error name (RxErr, BadTLP) rather than exact spacing. Uncorrectable reports follow the same shape, and some include a TLP Header line: the raw header of the packet that caused the error. The kernel AER guide shows an example.

How to triage PCIe AER errors, step by step

These commands only read state. Replace 01:00.0 and 00:1c.0 with your device and the port above it.

1. Collect every AER report from this boot. Don't filter by priority: the “received” line is logged at info level and corrected details at warning level, so journalctl -p err hides both.

   sudo journalctl -k -b --no-pager | grep -E -A4 'AER: .*received'

   Look for: the device address after “received”, the severity and the error names.

2. Identify both ends of the link. An error belongs to a link, and a link has two ends.

   lspci -nn -s 01:00.0
   lspci -PP -s 01:00.0

   Look for: a path such as 00:1c.0/01:00.0. The first address is the port on the other end of the link.

3. Read the counters on both ends. These files exist on AER-capable devices. The Root Port total includes everything reported from below it.

   cat /sys/bus/pci/devices/0000:01:00.0/aer_dev_correctable
   cat /sys/bus/pci/devices/0000:00:1c.0/aer_dev_correctable
   cat /sys/bus/pci/devices/0000:00:1c.0/aer_rootport_total_err_cor

   Look for: which device counts, and which error name keeps climbing.

4. Turn the count into a rate. Sample once a minute while you idle, then while you run the real workload.

   DEV=/sys/bus/pci/devices/0000:00:1c.0
   while sleep 60; do
     echo "$(date +%T) $(awk '/TOTAL_ERR_COR/ {print $2}' "$DEV/aer_dev_correctable")"
   done

   Look for: errors per minute, idle versus loaded. Counters start from zero at every boot, so compare like with like.

5. Check what the link actually trained to. Run lspci as root; otherwise it can't read the capabilities.

   sudo lspci -vv -s 01:00.0 | grep -E 'LnkCap:|LnkSta:|CESta:|UESta:'

   Look for: a LnkSta speed or width below LnkCap (lspci may append “downgraded”), and + flags in CESta or UESta. Those flags are sticky: they mean “seen since last cleared”, not “happening now”. For a field-by-field tour, see lspci -vvv explained. For a narrow link, read XDMA PCIe link stuck at x1. To decode a raw dump bit by bit, paste sudo lspci -xxxx output into the config space decoder.

6. Change one thing, then recount. Reseat the card, try another slot, remove a riser or extension cable, update firmware, or test a lower link speed if your firmware offers one. If the errors cluster around idle periods, test ASPM on just this link with the ASPM guide. Keep the workload and the time window the same.

   Look for: a rate that moves with exactly one change.

7. Keep a history with rasdaemon. rasdaemon records the kernel's RAS trace events, including AER, so you get timestamps that survive reboots. Its upstream service runs it with recording enabled.

   sudo systemctl enable --now rasdaemon
   sudo ras-mc-ctl --summary      # rasdaemon 0.8.x
   sudo ras-mc-ctl db --summary   # newer releases

   Look for: AER events you can line up against workload, temperature or firmware changes.

Is “PCIe Bus Error: severity=Corrected” harmful?

Not by itself. The kernel's guide says correctable errors don't affect the interface's function and the protocol recovers without losing data. But “harmless today” is not the same as “healthy”. Pay attention when:

• The rate keeps climbing over days or weeks.
• You see Data Link Layer names (BadTLP, BadDLLP, Timeout, Rollover). The link is replaying packets, and replays cost throughput.
• The same link has also trained narrower or slower than it should.
• Uncorrectable errors show up on the same device, even rarely.
• The errors started after a hardware change: a new card, riser, cable or retimer, or a move to a faster link generation.

Risers, extension cables and adapter boards add connectors to the signal path. Every connector is another place for signal quality to suffer, so they're the first thing to swap.

Should you boot with pci=noaer?

The kernel parameter documentation describes pci=noaer in a single line: it disables the use of PCIe Advanced Error Reporting. That's precisely what it does, and precisely what it costs.

Use it only after you've measured: the rate is stable, you've ruled out the cheap physical causes, and the log volume itself is causing harm. On 6.16 and newer, try the per-device rate-limit files first (/sys/bus/pci/devices/…/aer/correctable_ratelimit_interval_ms and correctable_ratelimit_burst). They trim the noise and keep the counters running.

Don't confuse it with pcie_aspm=off, either. That one changes how the link manages power instead of hiding reports. The ASPM guide compares the two.

Questions

What does “AER: Corrected error received” mean?

A Root Port received a correctable-error message from the device named after “received”. The hardware has already fixed the problem, typically by resending a packet. The lines that follow name the layer and the error bit. Kernels 6.8 and later print “Correctable error message received from” instead.

Is “PCIe Bus Error: severity=Corrected, type=Physical Layer” harmful?

Not immediately: no data was lost. It reports a receiver error, which points at signal quality on that link. Count how often it happens. A rare event is noise; a steady or rising rate deserves a look at the slot, riser, cable and link speed.

What is the difference between corrected and uncorrectable PCIe errors?

Corrected errors are fixed by the hardware with no software help. Uncorrectable non-fatal errors make one transaction unreliable, and the driver decides whether a reset is needed. Fatal errors make the whole link unreliable, so Linux resets the port above the device.

Does pci=noaer fix PCIe errors?

No. It stops Linux from using AER, so the messages, counters and AER-based recovery go away while the underlying errors continue. Measure first. If you were told to try pcie_aspm=off instead, read the ASPM guide before you do.

How do I count PCIe AER errors on Linux?

Read aer_dev_correctable, aer_dev_nonfatal and aer_dev_fatal under /sys/bus/pci/devices/ for both ends of the link, or run rasdaemon to keep a history across reboots. The sysfs counters reset at boot.

Why don’t I see any AER messages when my device misbehaves?

Linux handles AER only when firmware grants it control through ACPI _OSC, so check dmesg for _OSC lines. On firmware-first systems, errors may appear as [Hardware Error] records instead. And a device that drops off the bus may never get the chance to report anything; unknown header type 7f covers that case.