Changing a Nexus 9K switch from ACI to NX-OS mode without an APIC controller

Cisco ACI -just like any other beta platform- had an infancy period, which for the Nexus 9K line meant to have a series of licensing schemas, one of those being the dual ACI/NX-OS license. By default, Cisco would ship the switches with ACI-OS but there would be occasions where you would need to change back to NX-OS, which is very straightforward following the official documentation that uses the APIC controller to do so but, what happens you DON’T have an APIC controller to perform the “downgrade”? it does get trickier but it is possible.

For your reference, this process was performed in a Nexus 93180YC-EX switch and for the sake of simplicity, we will use a flash drive to load the NX-OS software into the switch. Unfortunately, the ACI-OS has a bug which makes it unable to read FAT file systems. To overcome this we can use the loader> mode which is not affected by this issue. If you can format the flash drive in ext4 (with a Macbook or another Unix-based OS) you can save time and performs steps 6 and 7 only.

1 Load the NX-OS file into the flash drive and boot hit CTRL+C to interrupt the boot process and access the loader> mode

2 Set the  cmdline recoverymode=1  parameter to instruct the switch to stop in another mode called switch(boot) mode.

3 Run the  dir  command and confirm you have the system can read the flash drive. You should see the NX-OS .bin file. The flash drive volume should be mounted as usb2: (or sometimes as usb1:)

4 Boot NX-OS directly from the flash drive using the  boot usb2:[your-NX-OS-bin-file.bin]  command. The switch will boot and then stop at the switch(boot) mode. Run the  load-nxos  command and the switch will boot. If it refuses to boot or shows any other error message, run the  init system  command first and then  load-nxos . I’ve seen it working in both ways and the reason why is not yet clear to me (thanks TAC…)

5 You will see the classic NX-OS initial configuration wizard, set a password for admin and then log into the switch. Don’t lock yourself out!

6 The flash drive should be mounted as usb1:, check its content and then copy the NX-OS .bin file to the bootflash: directory with the  copy usb1:[your-NX-OS-bin-file.bin] bootflash:  command.

7 Go to the global configuration mode and set the boot path with the  boot nxos bootflash:[your-NX-OS-bin-file.bin]  command and reload the switch. You should see NX-OS loading without any issues. Sayonara ACI.

In case something goes wrong after the reload and you are stuck in the loader> mode, especially if you are getting the “no autoboot or failed autoboot. failing to loader” message, try with the following additional steps.

8 Check that you have the NX-OS bin file both at the flash drive and the bootflash with the  dir  command.

9 If the file is indeed in the flash drive, boot NX-OS from there using the  boot [your-NX-OS-bin-file.bin]  command. 

9 Try with step 7 again, setting the boot path in NX-OS and reloading. While not the most common issue nowadays, there’s some weird magic with corrupted OS files as these may fail at boot time but manually booting would usually work. Just in case, while NX-OS has booted, run a checksum on the NX-OS file with the  show file bootflash:/[your-NX-OS-bin-file.bin] md5sum  command. If the hash doesn’t match the one at software.cisco.com, download the .bin file again and make sure you validate the checksum in every step of the process. 

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.