Fixing a messed up ESXi Passtrough configuration

Okay, not exactly O365-related but an issue I came across recently that needed to be fixed. You can see this post as some sort of documentation for myself 😉

The situation: we have some costumers running a Dell R730 with VMWare ESXi as a hypervisor running of a SD card, with several virtual machines on either local or shared storage. Recently they purchased some software that needed a dongle as a licensing devices, so this needed to be passed through to one of the VM’s. At first, we decided (because there would be several dongles needed) to passtrough the entire USB controller to the virtual. That’s not quite difficult: in the advanced settings for the hypervisor, select the devices you wish to passtrough, reboot the host and your all set. After that, you can connect a virtual USB-device to the VM so the connected devices should be exposed to the guest… Unfortunately, this didn’t work out. The connected device couldn’t be correctly addressed in the guest, so we decided to undo the passtrough of the USB-controller and connect the USB-device directly to the guest, without passing trough the controller. So, back to the advanced settings, disable the passtrough and reboot the host once again.

Once back up, we couldn’t select any USB-devices to connect to the guest. So, let’s start up Putty and SSH in to the host to check if there are any devices available: lsusb

Nothing was returned. No devices, but also no USB-controllers. Almost as if the passtrough was still active. Back to the GUI to check the config. Passtrough still enabled on those USB-controllers, while we disabled that just a few minutes ago. After some Googling, we found an article about passing trough USB-controllers when running ESXi from an SD-card. Long story short: the SD-slot is attached to the same USB-controller. When booting, ESXi can read from this SD-card and load the configuration in memory, but as soon as the passtrough is initiated the SD-card becomes unavailable to the hypervisor and everything you change to the configuration can’t be written back to the card, and is lost at reboot.

Most solutions on userforums to this issue state that reinstalling ESXi will resolve it. Of course it does, and of course your VM’s will remain intact and you can easily restart them when the hypervisor is restored, but I wanted a real fix. I came up with booting the server, trough iDRAC, from a bootable Linux-disk. In this case I used Knoppix, but any bootable Linux distro will do.

The SD-card will be available as a device in the shell you booted to. In my case, the partition that holds the right files was /dev/sdb5, but your milage may vary. The file we need is is the state.tgz file. Just look for this file. I’ll mount the partition to a directory on the bootable Linux distro.

1
2
mkdir /mnt/sdb5mount 
/dev/sdb5 /mnt/sdb5

We’ll copy the state.tgz file from the SDCard to a temp directory on our running distro, and extract the local.tgz file from the state.tgz file. Next, we’ll unpack the local.tgz file.

1
2
3
4
cp state.tgz /tmp/state.tgz
cd /tmp
tar xzf state.tgz
tar xzf local.tgz

No, we’ll have an etc directory in our temp directory. We’ll navigate to the esx.conf file and edit it.

1
2
cd /tmp/etc/VMware
vim esx.conf

In the esx.conf file, you can find the devices with their respective GUID, with owner = passtrough. Remove the entire line that states the owner, and save and close the file. After that, we’ll navigate back up to the temp directory to repack the directory to local.tgz and repack local.tgz to state.tgz

1
2
3
cd /tmp
tar czf local.tgz etc
tar czf state.tgz local.tgz

Before writing back the adjusted state.tgz to the SD-card, we’ll want to backup the current file. So, navigate back to the SD-card, backup the current file and replace it with the adjusted one.

1
2
3
cd /mnt/sdb5
cp state.tgz state.tgz.old
cp /tmp/state.tgz /mnt/sdb5/state.tgz

To clean everything up nicely, we’ll navigate back to the temp folder, so we can unmounts the SD-card before rebooting the host.

1
2
3
cd /tmp
umount /mnt/sdb5
shutdown -r now

After this, the server will boot back to ESXi and the passtrough of the USB-controllers will be undone so we can connect the USB-dongle directly to the guest.

Keep in mind, that some of the command used in this example require you to be root, so depending on the distribution you are booting from you might need to use sudo.