I ran into an issue tonight that kept me up for a few hours. A customer is running Cisco UCS solution, Cisco Nexus 1000v, Netapp SAN and VMware vSphere 5.
We had to re-acknowledge the Cisco UCS chassis because of a faulty IOM card in the system. When you re-acknowledge the chassis, the chassis is disconnected and the fabric interconnects then discovers the chassis through each link again. As you can image there is a few seconds of downtime here.
Just to be safe we powered down all virtual machines and VMware ESXi hosts.
The chassis successfully rediscovered and we powered all our ESXi hosts up.
We connected directly to an ESXi host, as vCenter wasn’t powered on yet. First we powered up the domain controller, followed by the SQL server and lastly the vCenter server.
The issue
As we powered on the vCenter server we noticed that the server had no network connectivity. I edited the virtual machine and could see that the option was selected to “Connect at power on” but the “Connected” option was not ticked. I tried to tick the Connected option and click ok. As the setting was being applied to the virtual machine I got the error: Invalid configuration for device ‘0’
The Fix
To cut a long story short, I’m going to jump straight to the issue we found and what we did to fix it.
This particular vCenter Server was connected to port 169 of the Cisco Nexus 1000v. Looking in the datastore to where the virtual machine resides and then under .dvsData – vDS Switch UUID folder, we can see the config files for each port. The vCenter Server’s file, also named 169, was missing. It actually appeared under another datastore.
I needed to copy the 169 config file from the other datastore to the correct datastore. I then rebooted the host and was able to make changed on the vCenter Server.
Resources
We also came across this blog post from psvmware, with some powercli scripts which searches your ESXi hosts for missing network config files.
Lastly, for further explanation on the .vdsData folder and its use especially in HA, please see this VMware vSphere Blog Post here
Be the first to comment