PCIe BARs explained: every device needs an address, and the space runs out.

What is a PCIe BAR?

Before a driver can talk to a device, the device needs an address. Not a bus number, but a range of physical addresses where the CPU's loads and stores land on the card instead of in RAM. That's memory-mapped I/O (MMIO). The Base Address Registers are how a device asks for it.

An ordinary endpoint has six 32-bit BAR registers in its configuration header, at offsets 0x10 to 0x24: BAR0 to BAR5. Bridges get two, plus the windows covered below. Each BAR describes one window, and its low bits say what kind:

A 64-bit BAR uses two consecutive registers for its address. That's why lspci often lists Regions 0, 1, 3 and 5. The “missing” 2 and 4 are the upper halves of 64-bit BARs, not absent features. Here's a synthetic excerpt in lspci's format for a GPU-like device:

# Synthetic example: sudo lspci -vv -s 01:00.0 (excerpt)
Region 0: Memory at f6000000 (32-bit, non-prefetchable) [size=16M]
Region 1: Memory at e0000000 (64-bit, prefetchable) [size=256M]
Region 3: Memory at f0000000 (64-bit, prefetchable) [size=32M]
Region 5: I/O ports at e000 [size=128]

• Region 0 holds registers. It's non-prefetchable because reading a status register can change it.
• Region 1 is a 64-bit, prefetchable window into device memory. It uses BAR1 and BAR2.
• Region 3 is another 64-bit window, using BAR3 and BAR4.
• Region 5 is legacy I/O port space.

The kernel exposes the same information in sysfs. Each line of the resource file holds the start, end and flags of one resource: BAR0 to BAR5 first, then the expansion ROM, then any further resources. A line of zeros is an unused BAR. The resource0 to resource5 files let privileged user space map a BAR directly, as the kernel sysfs guide explains.

cat /sys/bus/pci/devices/0000:01:00.0/resource

How is a PCIe BAR size determined?

A device never states its BAR size in bytes. Software discovers it with a trick that dates back to conventional PCI:

1. Save the BAR's value and turn off address decoding.
2. Write all ones to the BAR.
3. Read it back. The device hard-wires the low address bits to zero, so they stay zero.
4. Clear the flag bits, invert and add one. That's the size. Restore the original value.

Two consequences matter later. Sizes are always powers of two. And a BAR must be naturally aligned: a 256 MiB BAR can only start on a 256 MiB boundary. So a free gap can be big enough in total and still not fit.

Linux sizes BARs during enumeration and logs each one. On 6.8 and later, that looks like pci 0000:01:00.0: BAR 0 [mem 0xf6000000-0xf6ffffff]. Don't try the all-ones trick by hand with setpci on a live device: you'll pull its registers out from under the driver.

A raw configuration dump shows each BAR's base address and flags, but not its size. That's also what the free config space decoder reports from sudo lspci -xxxx output: type, 64-bit pairing and the prefetchable bit, tied to each register offset.

The aha: a BAR doesn’t need free memory, it needs a free window

Here's what makes “no space” so confusing: your machine may have terabytes of unused physical address space. That isn't the question. Every BAR has to fit inside the window of the bridge above it. That bridge's window has to fit inside its parent's window, and so on, up to the host bridge, whose windows come from firmware.

Each bridge has three windows, and they aren't interchangeable:

• The I/O window holds I/O BARs, in 4 KiB steps.
• The memory window holds non-prefetchable BARs. It's 32-bit only, in 1 MiB steps. So every non-prefetchable BAR behind a bridge must live below 4 GiB, even if the BAR itself is 64-bit.
• The prefetchable memory window may be 64-bit. Behind a bridge, it's the only route above 4 GiB.

Here's the port above the device from the previous example, again as a synthetic excerpt:

# Synthetic example: sudo lspci -vv -s 00:01.0 (excerpt)
Memory behind bridge: f6000000-f6ffffff [size=16M] [32-bit]
Prefetchable memory behind bridge: 00000000e0000000-00000000f1ffffff [size=288M] [64-bit]

The 32-bit window holds Region 0. The prefetchable window holds Regions 1 and 3: 256 MiB plus 32 MiB is 288 MiB. The window is 64-bit capable, but here it sits below 4 GiB. Now imagine the card asks for a 16 GiB Region 1 instead. It can't fit in the hole below 4 GiB, so unless that window moves above 4 GiB, the BAR has nowhere to go.

Hot-plug ports make the squeeze worse. By default, Linux reserves only 2 MiB for each memory window behind them, for devices that might arrive later.

What does Above 4G decoding do?

Below 4 GiB, the address map is crowded. RAM, firmware, interrupt controllers, PCIe configuration space and every 32-bit BAR all need room there, so the platform leaves only a limited MMIO hole below the 4 GiB line.

Above 4G Decoding (the exact name varies by vendor) is the firmware switch for 64-bit MMIO. With it on, firmware can assign 64-bit BARs above 4 GiB, where there's plenty of room. Turn it on when:

• a device has a BAR bigger than the hole can hold, such as a multi-gigabyte GPU or accelerator window, or a large FPGA BAR;
• you install many devices that each ask for several windows. Multi-GPU rigs are the classic case;
• you want Resizable BAR. Intel's Resizable BAR guide says to enable both Above 4G Decoding and Re-Size BAR Support, with CSM disabled and UEFI boot enabled.

Check what firmware actually gave Linux:

sudo dmesg | grep 'root bus resource'
sudo grep -E 'PCI Bus|0000:01:00' /proc/iomem

Look for: a host bridge window that starts at or above 0x100000000 (4 GiB). If every window ends below it, 64-bit BARs have nowhere to go up there. Run these as root: /proc/iomem shows zeros to ordinary users.

Resizable BAR on Linux

Resizable BAR lets a device advertise several legal sizes for a BAR, so the system can choose a bigger one when there's room. The kernel's parameter documentation notes, for example, that some graphics hardware can resize a BAR to allow access to all of its VRAM.

sudo lspci -vv -s 03:00.0 | grep -A2 'Resizable BAR'
cat /sys/bus/pci/devices/0000:03:00.0/resource0_resize

# Synthetic example output from the first command
Capabilities: [bb0 v1] Physical Resizable BAR
	BAR 0: current size: 256MB, supported: 256MB 512MB 1GB 2GB 4GB 8GB

Compare the current size with the supported list. Some drivers resize at probe time: amdgpu tries to grow its VRAM BAR and logs Not enough PCI address space for a large BAR. when it can't.

Kernels since late 2022 also expose resourceN_resize files. Reading one returns a bitmap of supported sizes, where bit 0 is 1 MB. Writing a bit number requests that size. The ABI documentation says all drivers must be unbound from the device first, and that success isn't guaranteed (sysfs ABI).

How to fix “can’t assign; no space” on Linux

1. Read the exact failure. The wording depends on the kernel version.

   sudo dmesg | grep -E "can't assign|no space|failed to assign"

   A synthetic example of both formats:

   # Linux 6.8 and later
   pci 0000:01:00.0: BAR 1 [mem size 0x400000000 64bit pref]: can't assign; no space
   pci 0000:01:00.0: BAR 1 [mem size 0x400000000 64bit pref]: failed to assign
   # Linux 6.7 and earlier
   pci 0000:01:00.0: BAR 1: no space for [mem size 0x400000000 64bit pref]
   pci 0000:01:00.0: BAR 1: failed to assign [mem size 0x400000000 64bit pref]

   Look for: which BAR, its size (0x400000000 is 16 GiB) and its flags. 64bit pref means it could live above 4 GiB if a window allowed it. Bridge windows can fail the same way, as bridge window [mem size …]: can't assign; no space.

2. Confirm the damage.

   sudo lspci -vv -s 01:00.0 | grep -E 'Region|Expansion ROM'

   Look for: <unassigned> where an address should be. The driver can't enable a device whose BAR has no address.

3. Find the window it needed.

   lspci -PP -s 01:00.0
   sudo lspci -vv -s 00:01.0 | grep 'behind bridge'

   Look for: the size of the prefetchable window, and whether it sits above 4 GiB.

4. Check firmware's windows. Use the root bus resource and /proc/iomem commands from the Above 4G section.

   Look for: no host bridge window above 4 GiB. If so, enable Above 4G Decoding, update the firmware and do a full power cycle. Change one firmware setting at a time.

5. Let Linux rebalance the bridges. Boot once with pci=realloc, by editing the kernel line from the boot menu. It reallocates bridge windows when firmware made them too small for the devices behind them. Make it permanent only if it helps.

   Look for: new assigned lines for the BAR and its bridge windows.

6. Reserve space behind hot-plug ports. For Thunderbolt and USB4 enclosures or hot-plug slots, pci=hpmmioprefsize=SIZE reserves a larger prefetchable window, and pci=hpmmiosize=SIZE a larger non-prefetchable one. Size them for the largest BAR you expect. Options combine with commas, for example pci=realloc,hpmmioprefsize=SIZE.

7. If you design the device, ask for less. On an FPGA, keep register BARs small and non-prefetchable, and put large memory windows in 64-bit prefetchable BARs when the hardware really has no read side effects. Don't request more space than you use. Avoid pci=nocrs except for debugging: the kernel documentation says that if you need it, you should report a bug.

If the card isn't in lspci at all, BARs aren't your problem yet. Start with FPGA missing from lspci or unknown header type 7f.

Questions

What is a PCIe BAR and how is its size determined?

A BAR is a register in the device's configuration header that requests a window of memory or I/O address space. To size it, software writes all ones, reads back, clears the flag bits, inverts the value and adds one. The result is a power of two, and the window must be aligned to it.

Should I enable Above 4G decoding?

Enable it when a device needs large or many 64-bit BARs, when you want Resizable BAR, or when Linux reports “no space” for a 64-bit prefetchable BAR. Change one firmware setting at a time and keep a way back, such as a firmware settings reset.

What does “can't assign; no space” mean in dmesg?

Linux couldn't find a free, correctly aligned gap for that BAR inside the bridge window above it. The device is visible, but that BAR has no address, so the driver can't use it. Kernels before 6.8 print “BAR n: no space for” instead.

What is MMIO?

Memory-mapped I/O exposes device registers or memory at physical addresses, so the CPU reaches the device with ordinary loads and stores. BARs define where those addresses are. MMIO isn't RAM: a read can have side effects, which is why register BARs are non-prefetchable.

Does Linux support Resizable BAR?

Yes. Drivers such as amdgpu resize their BAR at probe time when address space allows, and kernels since late 2022 expose resourceN_resize files in sysfs. Firmware still has to leave enough 64-bit space for the larger size.

What does pci=realloc do?

It lets Linux reallocate bridge windows when firmware made them too small for the devices behind them. That can fix “no space” errors on platforms with tight firmware allocations. It can't create space that firmware never gave the host bridge.