What does this do? I perused the README but I'm still clueless on what the purpose and use-case is for ReBAR.<p>(Posting this under the assumption others will also appreciate a bit of quick context.)
Since about 5 or 6 years ago, PCIe Video Cards advertise a capability known as Resizeable BAR. PCI Devices like GPUs require some memory to use for PCI MMIO, which is directly visible on the CPU Address Space. As a side note, this Address Space is shared with RAM, and anyone that was around when having 4 GiB RAM with a 32 Bit OS was common (Earlier than 2010 or so) knows that you only saw about 3.25 GiB RAM or so because of sharing the Address Space with PCI MMIO, and those fortunate enough to have used SLI usually saw even less than that, like 2.87 GiB RAM.<p>GPUs has been using a 256 MiB PCI MMIO window regardless of how much VRAM they actually have since... nearly forever? At least since PCIe is a thing, since I recall than AGP Aperture Size was seteable in era accurate BIOSes. PCIe 3.0 specification introduced a feature known as Resizeable BAR, where the PCI Device can tell a compatible Firmware how much MMIO it actually wants. GPUs uses that to tell a ReBAR capable UEFI Firmware that it wants more MMIO (Usually as big as the GPU VRAM), or uses legacy 256 MiB otherwise.
GPUs have been using a 256MB window since they started coming with 256MB of VRAM, as it would otherwise be a pointless waste of address space to have a window larger than the actual amount of VRAM. Previously, they would have a window exactly equal to the size of VRAM (likely rounded up to the next power of 2.)
There would also be a block of memory mapped hardware registers also needing address space that you have to poke to make the GPU actually do GPU things, instead of just being an expensive way to add extra memory to a system, no?<p>And if my experience from embedded development is in any way transferable, they're probably fairly spread out and probably takes a fairly big chunk of address space too.
Just to make sure I understand: Is the resizable BAR/MMIO a RAM buffer for PCie packets? Or does it have some deeper integration with DMA or something? I assume it's not like memory mapped peripherals on an AXI bus which is why you need the buffer?
No idea on the AXI Bus you're talking about, so can't make comparisons.<p>MMIO (Memory Mapped I/O) is essentially memory (Whenever RAM or ROM) from OTHER devices that is directly visible on the CPU Address Space. My understanding is that from the CPU side, MMIO is mostly transparent (Except for the massive increase in latency) because it gets used like if it was interacting with its own workspace with regular instructions like MOV.<p>What PCIe ReBAR changes is that before, you could only see a 256 MiB window onto the GPU VRAM, so there was an added overhead since the GPU may need to relocate things from inside that window somewhere else on its total VRAM (So yes, it may be interpreted as if what you see from the CPU side is just some kind of exchange buffer).
I believe the best way to describe how it operates is comparing it to EMS (Expanded Memory) from the DOS days since it also worked with a similar, if not the same idea. You could only see a portion of the total memory from what was installed on the EMS card (A 128 KiB window located on the upper part of the 1 MiB address space from the 8086 CPU), so you had to switch which Page (Region) of the memory was visible, adding a lot of overhead and most likely requiring an additional buffer in main RAM to move data from one Page to another. However, since I have no knowledge if the GPUs really work like that I can't confirm. I never knew whenever the 256 MiB is "fixed" (You always see the same Region) or if you can decide which section of the VRAM to make visible.
Just to close the loop, AXI bus[1] refers to a very common implementation, in the gateware/verilog, of a MIMO.<p>[1] <a href="https://en.wikipedia.org/wiki/Advanced_eXtensible_Interface" rel="nofollow">https://en.wikipedia.org/wiki/Advanced_eXtensible_Interface</a>
It’s also similar to how mmap() on 32-bit systems maps files larger than 2GB.
It's just memory mapping the peripheral.<p>For historic reasons (e.g. 32 bit address spaces, plus the need to reserve the space for multiple pci peripherals) it has been a narrow, movable aperture.<p>Resizable BAR lets the size of the aperture be chosen (which is usually chosen to allow all of VRAM to fit in and be directly accessible).
It's hard to explain without going into a few low-level details of PCI Express, but let me try.<p>Most PCI devices expose some memory and/or I/O ports to the CPU. That memory (or I/O ports) is mapped to somewhere in the address space visible to the CPU. Besides the memory and I/O ports, all PCI devices also expose a separate set of configuration registers; among these registers, there are the Base Address Registers (BARs), which configure where the memory or I/O ports is mapped.<p>Here's an example output from "lspci -vv" for a GPU:<p><pre><code> Region 0: Memory at 7c00000000 (64-bit, prefetchable) [size=8G]
Region 2: Memory at 7e00000000 (64-bit, prefetchable) [size=256M]
Region 4: I/O ports at f000 [size=256]
Region 5: Memory at fca00000 (32-bit, non-prefetchable) [size=1M]
Expansion ROM at fcb00000 [disabled] [size=128K]
</code></pre>
Note that regions 0 and 2 are above the 4GB addressable by old 32-bit CPUs. To be compatible with these old CPUs, this card and many others like it allow the firmware (and/or the operating system) to choose not only where the memory is mapped, but also its size. We can see this in the same "lspci -vv" output for this GPU:<p><pre><code> Capabilities: [200 v1] Physical Resizable BAR
BAR 0: current size: 8GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB
BAR 2: current size: 256MB, supported: 2MB 4MB 8MB 16MB 32MB 64MB 128MB 256MB
</code></pre>
Older systems which do not understand this extended capability will still treat these regions as fixed size, probably with the first size in this list (256MB for region 0, 2MB for region 2). Newer systems can tell the device to "resize" the BAR to a bigger size, which obviously needs the first region to be placed above the 4GB barrier since it's too big.<p>Why is this useful? This particular GPU has 8GB of VRAM; it's quite obvious that region 0 is a direct view into that VRAM. When using the maximum BAR size, the CPU can directly read and write anywhere into the VRAM; when using a smaller BAR, the CPU can only see a small window into the VRAM, and has to use less direct methods to access it.<p>(As an aside: go right now and do a "sudo lspci -vv" on your computer, if you see a Resizable BAR capability which isn't using the maximum size, you can probably gain a bit more speed for free by going into the BIOS and enabling "Resizable BAR" and/or "Above 4G decoding". If you can't find these options, well, AFAIU that's what this project is all about..)
> (As an aside: go right now and do a "sudo lspci -vv" on your computer, if you see a Resizable BAR capability which isn't using the maximum size, you can probably gain a bit more speed for free by going into the BIOS and enabling "Resizable BAR" and/or "Above 4G decoding". If you can't find these options, well, AFAIU that's what this project is all about..)<p>IIRC Linux doesn't need resizable bar enabled in the BIOS since the kernel will resize the bar if supported by the GPU, Windows however relies on the UEFI doing it which is where it being enabled in the BIOS is needed.
Resizable Base Address Register. In short, rather than the 256MB of mapped memory available to the CPU for any PCI device, ReBAR capable devices can map larger memory to the CPU’s addressable memory space. Without ReBAR or sufficient tricks (that Intel GPUs lack) you have to shuffle 256MB chunks around between GPU and CPU.
ReBar's commercial name is AMD Smart Access Memory, it allows a PCIe device such as a GPU to map more VRAM to the system at once, which improves performance by reducing access overhead. This generated much fanfare in the early 2020s after AMD officially supported it in the newly released AMD Zen 3 CPUs with RX6800 series GPUs. It was marketed as a new technology to boost GPU/gaming performance. What AMD did was just rebranding an obscure feature in the PCIe specification [1]. As shown by this project, it was actually supported by the PCIe controller since Sandy Bridge, just disabled in the firmware. For a decade nobody bothered to use it. Presumably, AMD saw an opportunity and enabled it, presumably after validating the hardware and fixing any driver compatibility problems.<p>[1] This is nothing new in the tech industry. Intel rebrands DVFS as SpeedStep, IOMMU as VT-d, AMD rebrands the NX bit as Enhanced Virus Protection, etc.
SAM wasn't just about the resizable BAR capability but also about driver adjustments to make efficient use of the larger BAR.
I feel part of the "Smart Access Memory" name is actually the driver features and paths to actually make use of it.<p>It's not like ReBar is a single toggle "Make Things Faster", but a different option in how it can map gpu memory to the cpu. The driver still needs to use it - and decide where it's use vs the "staging buffer" approach would actually be be benefitial.
IOMMU branding at least makes a little bit of sense. AMDs and Intels virtualisation ISA are completely different, including their IOMMU.
It allows the CPU to access all GPU VRAM. This improves gaming performance, especially for AMD GPUs.
> what the purpose and use-case is for ReBAR<p>Literally the second sentence in the repo:<p>> This provides performance benefits and is even required for Intel Arc GPUs to function optimally.