NVIDIA Enables Beta Support For Virtualization On GeForce GPUs

With the latest Game Ready Driver, NVIDIA has enabled GPU Passthrough beta support for Windows virtual machines on GeForce GPUs. Prior to driver 465.89, only Quadro NVIDIA cards supported GPU passthrough. For Linux users, this technology will allow playing Windows-only games in your VM; or, if you’re a developer, you can more easily test both Windows and Linux code from a single machine, accelerating development.

GeForce RTX 30 Series

GPU passthrough has been used for a long time, and there were ways to enable it on GeForce graphics cards; however, it was never officially supported. Technically speaking, the technology always has been available in the silicon chip for GeForce, but it was not enabled in the drivers, causing the popular “error 43” on Windows virtual machines. Using a software hack has been possible for a long time to enabled GPU passthrough on the consumer GeForce cards and vGPU capabilities. More about vGPU, in this article.

NVIDIA Game Ready Driver 465.89

On March 30 of this year, NVIDIA released a new Game Ready Driver, 465.89. This driver supports brand-new video games and offers a fix to open issues presented in many other games and apps. However, something particularly exciting about this new release is found under Gaming Technology. Here, three import features have been announced: Resizable BAR the GeForce RTX 30 Series, support for OpenCL 3.0, and beta support for virtualization on GeForce GPUs.

Resizable BAR

Resizable BAR allows the CPU to access all of a PCIe device’s memory at once, rather than the previous limitation of 256MB per cycle. Smart Access Memory, SAM (AMD’s term for the technology) was announced last year by the company and has already been implemented.

OpenCL 3.0

Added support for OpenCL 3.0, the latest major version of OpenCL maintaining backward compatibility with OpenCL 1.2. NVIDIA OpenCL 3.0 continues to support existing OpenCL 1.2 functionality and Khronos and vendor extensions that are already supported with NVIDIA OpenCL 1.2 drivers.

Beta Support for virtualization on GeForce GPUs.

Also, starting with this driver, NVIDIA has enabled GPU passthrough (not vGPU, a.k.a. SR-IOV for GPU sharing) on all GeForce graphics cards based on Kepler architecture and newer for Windows virtual machines. This is the technology we are focusing on in this article. Even though Resizable BAR was one of the most expected features for the new RTX generation and gamers, virtualization on GeForce GPUs was the most expected addition for “HomeLabers” and Linux users.

Windows Virtual Machine Beta Support for GeForce

If you’re primarily a Linux user, you can now enable GeForce GPU passthrough on a Windows Virtual Machine (VM). NVIDIA GPU passthrough was only reserved for Enterprise or workstation-class GPUs. The company didn’t allow GeForce customers to use this technology into a Windows VM running in their consumer-grade computers. However, many customers were actually using GeForce cards as workstation GPUs.

On consumer cards, the silicon was never the issue with GPU passthrough; it was the software (driver) that locked out users. Previously, to use GPU passthrough, users would have to spend a substantial amount of money for an NVIDIA Tesla or Quadro card. GPU passthrough beta support for GeForce is a meaningful victory for those wanting to run Windows games from within a virtual machine while maintaining their Linux (physical) host and are using a GeForce graphics card.

The main target for this technology is Linux users wanting to multi-task GPU-demanding applications. There are a few GeForce use cases where this functionality is beneficial, such as:

GeForce customers wanting to run a Linux host and be able to launch a Windows virtual machine (VM) to play games
Game developers wanting to test code in both Windows and Linux on one machine

GeForce virtualization (beta) is supported on R465 or higher drivers. It is enabled on all GeForce/TITAN GPUs supported by this driver (Kepler and later for desktop; Maxwell and later for laptops) on Windows. Windows 10 is the only supported guest operating system for this GeForce virtualization setup.

With GeForce GPU Passthrough, you can’t leverage the same GPU used by the host OS for virtualization. One GPU is required for the Linux host OS, and one GPU is required for the Windows virtual machine. Also, It is important to remember that GPU passthrough supports only one virtual machine. vGPU (SR-IOV) or a shared GPU Passthrough for multiple VMs are not supported on GeForce. Suppose you want to enable various VMs to have direct access to a single GPU or want the GPU to assign virtual functions to multiple virtual machines. In that case, you will need to use a workstation or enterprise GPUs.

GPU Passthrough Error: Code 43

If you ever tried to run a Windows VM from a Linux machine, before the Game Ready Driver 465.89, the drivers in the guest OS would return the popular “Code 43”. This error only showed up in Windows (Guest) OS when the GeForce driver detected that the OS was actually running on a VM. The support for virtualization on GeForce GPUs means that you no longer have to conceal your VM; the drivers can be installed, and this error code should be gone.

It is also perfectly fine to mention that NVIDIA deliberately engineered this error code for the GeForce graphics cards. It is not manifested on Tesla and Quadro GPUs, the company’s enterprise-grade cards. The reason is that NVIDIA certainly needs to separate the consumers from the enterprise-grade segment. Some enterprise features don’t make much sense for the consumer (lower-performance) video cards, but enabling some of them would result in a significant sales loss for NVIDIA’s enterprise (expensive) cards.

GPU passthrough vs. vGPU

NVIDIA 465.89

Many users are still confused about GPU Passthrough and the technical capabilities offered by NVIDIA. It is important to revisit what the virtualization on GeForce GPU is and what is not. The term GPU Passthrough might sound pretty self-explanatory for many GPU enthusiasts; yet, it is usually confused with NVIDIA virtual GPU (vGPU).

Both GPU Passthrough and vGPU are technologies long-time used for enterprise virtualization solutions such as VMware ESXi and Citrix XenServer. And for Tesla and Quadro GPUs. In VMware ESXi, GPU Passthrough is called virtual dedicated graphic acceleration (vDGA), and vGPU is called Virtual shared graphics acceleration (vSGA). vGPU allows enterprise-grade GPUs to be shared across multiple virtual desktops.

VMware offers a third option called Virtual Shared Passthrough Graphics Acceleration. This technology allows a GPU to be shared with multiple users instead of focused on only one user. The difference from vGPU (vSGA in VMware) is that the proprietary VMware 3D driver is not used, and most of the graphics card’s features are supported. Remember that with GPU Passthrough on GeForce cards, you can’t enable multiple VMs to have direct access to a single GPU, nor is the GPU able to assign virtual functions to multiple virtual machines.

Virtual Desktop Infrastructure (VDI) is the best example and scenario for using these GPU technologies. VDI solutions take advantage of virtualization by quickly creating hundreds of VMs clones running on a single powerful server or cluster. Most of the time, vGPU is the best choice for users who require the GPU’s full potential during brief periods. However, vGPU can create bottlenecks, depending on which applications are used and the resources these applications need from the GPU. vGPU is generally used for knowledge workers and, occasionally, for power users. Nowadays, planning VDI solutions without acknowledging vGPU is considered a negligent act.

Nvidia vGPU Hacked for GeForce Cards

Even with GPU Passthrough enabled in the latest NVIDIA drivers, GeForce users still don’t have (officially) the ability to share GPUs across multiple virtual desktops. For vGPU, you will need to use NVIDIA Tesla, Quadro, or RTX enterprise GPUs. So, you cannot trick NVIDIA on this one, or can you?

A couple of months ago, it was revealed an unlock to the vGPU functionality for consumer-grade GPUs. This tool enables the use of GeForce and Quadro GPUs with the NVIDIA vGPU software. NVIDIA vGPU normally only supports a few Tesla GPUs, but since some GeForce and Quadro GPUs share the same physical chip as the Tesla, this is only a software limitation for those GPUs. This tool aims to remove this limitation.

The tool and instructions are available at https://github.com/DualCoder/vgpu_unlock.

Final Thoughts

NVIDIA has finally decided to remove the block that they had arbitrarily on GeForce cards. Prior to NVIDIA’s release of driver 465.89, when the guest OS detected a virtual machine, the driver was disabled inside Windows and Linux. With virtualization enabled, GeForce customers on a Linux host PC can now enable GeForce GPU passthrough on a virtual Windows guest OS.

Technically speaking, the support for virtualization on GeForce GPU is not a beta release; but more like NVIDIA just ticking a box to unlock a function that has existed for a long time. However, GPU Passthrough is still limited to a single GPU instance, which means that only a virtual machine will have access to the graphics card exclusively.

As for vGPU, unfortunately, unless you go full Enterprise, NVIDIA still doesn’t support sharing a GPU with multiple virtual machines, the same way that a CPU manufacturer does. If the user needs more virtual machines to access the same GPU, then Tesla or Quadro graphics cards will be required. But this could be subject to change as GPU passthrough. Currently, a software hack is available to try out vGPU capabilities, but of course, its support is not official.

NVIDIA

Engage with StorageReview