Working with Exadata is awesome as it extends the knowledge of the DBA (DMA) a lot as we are now in charge of many things that were not in our scope before like . . . patching a Cisco switch !
You may indeed have to patch the Exadata Cisco Switch to update its version but also to enable SSH access as the switch is only telnet accessible by default.
I've also added this Cisco Switch patching procedure in the How to patch / Upgrade to 18c an Exadata procedure.
We don't do it very often but it is always good to have real life procedure handy.
0/ A Cisco Switch Introduction
Let's start by clarifying what the Cisco switch is and what is its purpose.
0.1/ Where is the Cisco Switch located ?
The Cisco Switch is located in the U21 Rack unit as you can see on the below Exadata Rack layout (from the Oracle documentation):
0.2/ What is it used for ?
The Cisco Switch (aka the Ethernet Switch) is used to connect each Exadata component for an administrative purpose; in short, it is for the DBAs to access the ILOMs and the Infiniband Switches (the IB switches have no dedicated ILOMs, the ILOMs are on the IB Switches themselves, you can find more information here). You can find more information about the Cisco Switch in the official documentation.
I found the below image self explanatory:
I found the below image self explanatory:
1/ Access the switch using telnet
Before anything, you'll need to find the hostname / IP address of this switch. You'll find this information in the /opt/oracle.SupportTools/onecommand/databasemachine.xml file. This file comes from OEDA when installing a new Exadata.
I recommend using the exa-racklayout.sh script to quickly get this information:
Here, you can quickly see that the Cisco switch is located in the U21 location.
You can also follow the below steps to find this information manually:
You can also get the information with the below command line:
By default, the switch is only telnet accessible; you'll then have to find a server where telnet is installed. For information, telnet is not installed on the database servers so you may want to install telnet on a database server or on any another jump server you would use to connect. In this example, we installed telnet on the database node 1.
I recommend using the exa-racklayout.sh script to quickly get this information:
You can also follow the below steps to find this information manually:
[oracle@exadatadb01 ~]$ vi /opt/oracle.SupportTools/onecommand/databasemachine.xml . . . <ITEM ID="14"> <TYPE>cisco</TYPE> <ADMINNAME>cisco_switch</ADMINNAME> <ADMINIP>10.11.12.13</ADMINIP> <ULOCATION>21</ULOCATION> <UHEIGHT>1</UHEIGHT> </ITEM> . . .Note that we can confirm that the switch is in the U21 rack unit (as per the ULOCATION tag)
You can also get the information with the below command line:
[oracle@exadatadb01 ~]$ awk '{if ($0 ~ /cisco/) { print $0; getline; print $0; getline; print $0}}' /opt/oracle.SupportTools/onecommand/databasemachine.xml <TYPE>cisco</TYPE> <ADMINNAME>cisco_switch</ADMINNAME> <ADMINIP>10.11.12.13</ADMINIP> [oracle@exadatadb01 ~]$
By default, the switch is only telnet accessible; you'll then have to find a server where telnet is installed. For information, telnet is not installed on the database servers so you may want to install telnet on a database server or on any another jump server you would use to connect. In this example, we installed telnet on the database node 1.
[root@exadatadb01 ~]# telnet cisco_switch Trying 10.11.12.13... Connected to cisco_switch. Escape character is '^]'. User Access Verification Password: cisco_switch> enable -- the enable command is to "su - root" Password: cisco_switch# -- here you are root (# in the prompt)Note that you have 2 different passwords to provide and be "enabled" as root on the Switch, they are given by Oracle when they cable the Exadata at first -- if you don't know it, you can always try the default one :)
2/ Pre requisites
2.1/ Configure a TFTP server
Here is a new interesting thing to learn here : a TFTP server !
TFTP stands for Trivial File Transfer Protocol; it is a simple File Transfer Protocol to transfer files from remote hosts with no user nor password. It is used a lot in the network world. You can find more information about it here.
There is no TFTP server installed by default on a database server, you'll then have to install it yourself. The thing is that you'll need to install xinetd to install a TFTP server. As we were not really keen on installing xinetd on a database server, we created a VM and installed the TFTP server on it following this procedure. The below example will assume that the TFTP server IP address is 10.100.100.100.
TFTP stands for Trivial File Transfer Protocol; it is a simple File Transfer Protocol to transfer files from remote hosts with no user nor password. It is used a lot in the network world. You can find more information about it here.
There is no TFTP server installed by default on a database server, you'll then have to install it yourself. The thing is that you'll need to install xinetd to install a TFTP server. As we were not really keen on installing xinetd on a database server, we created a VM and installed the TFTP server on it following this procedure. The below example will assume that the TFTP server IP address is 10.100.100.100.
2.2/ Open a MOS SR to require the firmware
As specified in the note Upgrading firmware / Configuring SSH on Cisco Catalyst 4948 Ethernet Switch (Doc ID 1415044.1), you first have to open a MOS SR to request the new firmware version:
In order to obtain the SSH enabled Cisco IOS firmware, please open an SR using Hardware CSI with distinct product id, name, component and refer to this MOS note.- MOS will then attach the new firmware to the MOS SR with the md5:
Hi Fred, IOS 15.0.2-SG11 for Cisco4948 is uploaded to attachment. cat4500-ipbasek9-mz.150-2.SG11.bin. 17.26 MB ( 18095624 bytes) MD5: 32ade7ea6c81bae54d3e97512ff4e406 SHA512: f19e32fba6566f10ccb508ce7f997b9869fbdbf89d3e3a9026392ad75d226f3e43fd75f5bbf6332130e364b27e72fd1a0f12e6450f0e446d4c95576fa1968044 Please refer to Doc ID 1415044.1 to update the IOS. Thank you!Once you got the new version of the firmware, we can start the pre requisites !
2.3/ Check the version
First of all, let's have a look at what is currently installed:
cisco_switch# show version Cisco IOS Software, Catalyst 4500 L3 Switch Software (cat4500-IPBASE-M), Version 12.2(53)SG5, RELEASE SOFTWARE (fc1) Technical Support: http://www.cisco.com/techsupport Copyright (c) 1986-2011 by Cisco Systems, Inc. Compiled Mon 20-Jun-11 19:33 by prod_rel_team Image text-base: 0x10000000, data-base: 0x11BAFF68 ROM: 12.2(31r)SGA2 Dagobah Revision 226, Swamp Revision 5 cisco_switch uptime is 3 years, 43 weeks, 5 days, 5 hours, 40 minutes -- nice uptime ! System returned to ROM by power-on System restarted at 13:02:31 CST Tue Jan 13 2015 System restarted at 13:02:31 CST Tue Jan 13 2015 System image file is "bootflash:cat4500-ipbase-mz.122-53.SG5.bin" -- this is the current firmware cisco WS-C4948 (MPC8245) processor (revision 0) with 262144K bytes of memory. Processor board ID FOX1543G8VT MPC8245 CPU at 266Mhz, Fixed Module Last reset from PowerUp 1 Virtual Ethernet interface 48 Gigabit Ethernet interfaces 511K bytes of non-volatile configuration memory. Configuration register is 0x2101 cisco_switch#
2.4/ Check the FS
bootflash is the equivalent of /boot on a Linux system, it needs at leats 20 MB free on it for the new firmware version (45 MB is free here):
cisco_switch# show file systems File Systems: Size(b) Free(b) Type Flags Prefixes * 60817408 45173068 flash rw bootflash: - - opaque rw system: - - opaque rw tmpsys: - - opaque ro crashinfo: 524280 523724 flash rw cat4000_flash: - - opaque rw null: - - opaque ro tar: - - network rw tftp: - - opaque ro profiler: - - opaque wo syslog: 524280 518558 nvram rw nvram: - - network rw rcp: - - network rw http: - - network rw ftp: - - opaque ro cns: cisco_switch#
2.5/ Check what is in bootflash
Let's have a look at what is in the bootflash directory; we'll need the name of the current image later on to backup the current configuration:
cisco_switch# dir bootflash: Directory of bootflash:/ 1 -rwx 15644212 Apr 3 2012 08:07:05 -05:00 cat4500-ipbase-mz.122-53.SG5.bin 60817408 bytes total (45173068 bytes free) cisco_switch#
3/ Update the firmware, configure SSH and (optionally) disable telnet
3.1/ Backup the current configuration
Before upgrading the switch per se, we will start by making a backup of the current configuration.
And here is where TFTP is kind of tricky:
Let's just touch the file on the TFTP server side:
cisco_switch#configure terminal Enter configuration commands, one per line. End with CNTL/Z. cisco_switch(config)#no boot system cisco_switch(config)#boot system bootflash:cat4500-ipbase-mz.122-53.SG5.bin cisco_switch(config)#^Z -- This is CTRL + Z cisco_switch#copy running-config startup-config all Destination filename [startup-config]? -- just press ENTER here Compressed configuration from 24500 bytes to 7674 bytes[OK] 24500 bytes copied in 2.372 secs (10329 bytes/sec) cisco_switch#copy running-config bootflash:cisco_switch-confg-before-ssh Destination filename [cisco_switch-confg-before-ssh]? -- press ENTER to accept this name or put another one 6135 bytes copied in 1.120 secs (5478 bytes/sec) cisco_switch#dir bootflash: Directory of bootflash:/ 1 -rwx 15644212 Apr 3 2012 08:07:05 -05:00 cat4500-ipbase-mz.122-53.SG5.bin 2 -rwx 18095624 Nov 29 2018 18:35:59 -06:00 cat4500-ipbasek9-mz.150-2.SG11.bin 3 rw 6135 Nov 29 2018 19:15:04 -06:00 cisco_switch-confg-before-ssh 60817408 bytes total (27071052 bytes free) cisco_switch#Now that we have our cisco_switch-confg-before-ssh backup file, we want to transfer it to the TFTP server as an external backup.
And here is where TFTP is kind of tricky:
cisco_switch#copy bootflash:cisco_switch-confg-before-ssh tftp: Address or name of remote host []? 10.100.100.100 Destination filename [cisco_switch-confg-before-ssh]? -- press ENTER to accept this name or put another one TFTP: error code 1 received - 18025 %Error opening tftp://10.100.100.100/cisco_switch-confg-before-ssh (No such file or directory) cisco_switch#You may think that this error is a permission issue on the TFTP server side. Well no it isn't. Actually, you need to create the file you want to transfer on the TFTP server before transfering it !
Let's just touch the file on the TFTP server side:
user@vm:/Users/user 0 > touch /private/tftpboot/cisco_switch-confg-before-ssh user@vm:/Users/user 0 > ls -l /private/tftpboot/ total 35352 -rwxrwxrwx 1 root root 0 Nov 29 19:37 cisco_switch-confg-before-ssh -r--r--r--@ 1 root root 18095624 Nov 20 15:19 cat4500-ipbasek9-mz.150-2.SG11.bin user@vm:/Users/user 0 >And then you can backup your file on the TFTP server:
cisco_switch#copy bootflash:cisco_switch-confg-before-ssh tftp: Address or name of remote host []? 10.100.100.100 Destination filename [cisco_switch-confg-before-ssh]? -- press ENTER to accept this name or put another one (it should be the exact name you previously touched) 6135 bytes copied in 1.020 secs (6015 bytes/sec) cisco_switch#Easy . . . when you know the trick !
3.2/ Transfer the new Cisco IOS SSH-capable firmware to switch's boot flash
Let's now transfer the new firmware from the TFTP server to the switch.
- Execute a md5 of the file to be sure it is same as the one mentioned in the MOS SR
- Verify the file for integrity
- Execute a md5 of the file to be sure it is same as the one mentioned in the MOS SR
user@vm:/Users/user > md5 /private/tftpboot/cat4500-ipbasek9-mz.150-2.SG11.bin MD5 (/private/tftpboot/cat4500-ipbasek9-mz.150-2.SG11.bin) = 32ade7ea6c81bae54d3e97512ff4e406 user@vm:/Users/user >- Transfer the file to the switch
cisco_switch# copy tftp: bootflash: Address or name of remote host []? 10.100.100.100 Source filename []? cat4500-ipbasek9-mz.150-2.SG11.bin Destination filename [cat4500-ipbasek9-mz.150-2.SG11.bin]? -- just press ENTER here to keep the same name as the original Accessing tftp://10.11.12.13/cat4500-ipbasek9-mz.150-2.SG11.bin... Loading cat4500-ipbasek9-mz.150-2.SG11.bin from 10.11.12.13 (via Vlan1): !.!!.!!!!.!.!.!.!.!.!.!.!.!.!..!.!!!!!.!!!!!!!.!!.!!!!!!.!!!.!!.!!!!!.!!!!!!!!!!!!!!!!!.!!.!!.!!!.!!.!!!!!!!.!!!.!.!.!!.!.!!.!.!.! [OK - 18095624 bytes] 18095624 bytes copied in 1901.348 secs (9517 bytes/sec) cisco_switch#dir bootflash: Directory of bootflash:/ 1 -rwx 15644212 Apr 3 2012 08:07:05 -05:00 cat4500-ipbase-mz.122-53.SG5.bin 2 -rwx 18095624 Nov 29 2018 18:35:59 -06:00 cat4500-ipbasek9-mz.150-2.SG11.bin 60817408 bytes total (27077316 bytes free) cisco_switch#Note that the transfer is slow, it took 1900 seconds (~30 minutes) to transfer a 20 MB file.
- Verify the file for integrity
cisco_switch# verify cat4500-ipbasek9-mz.150-2.SG11.bin CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC File system hash verification successful. cisco_switch#Note: the only message you would get here is that the verification is successful or not. You won't be mentioned about the md5.
3.3/ Prepare Cisco 4948 to boot with new IOS firmware
We are now ready to apply the patch to the Switch, first let's set up the reboot with the new firmware configuration.
cisco_switch#configure terminal Enter configuration commands, one per line. End with CNTL/Z. cisco_switch(config)# config-register 0x2102 cisco_switch(config)# no boot system cisco_switch(config)# boot system bootflash:cat4500-ipbasek9-mz.150-2.SG11.bin cisco_switch(config)# ^Z -- This is CTRL + Z cisco_switch# show run | include boot boot-start-marker boot system bootflash:cat4500-ipbasek9-mz.150-2.SG11.bin boot-end-marker snmp-server enable traps entity-diag boot-up-fail hm-test-recover hm-thresh-reached scheduled-test-fail cisco_switch# copy running-config startup-config all Destination filename [startup-config]? -- Press ENTER here 24502 bytes copied in 2.404 secs (10192 bytes/sec) cisco_switch# write memory Building configuration... Compressed configuration from 6137 bytes to 2600 bytes[OK] cisco_switch#
3.4/ Reboot the Cisco 4948 switch with the new firmware
There will be an outage on the management network for all connected devices (including all storage cells, database servers, ILOMs, and InfiniBand switches) for a minute or two while the switch reboots. A management network outage should not cause an application outage as the databases should all remain available and functioning normally.
cisco_switch# reload Proceed with reload? [confirm] -- Press ENTER when ready Connection closed by foreign host. [oracle@exadatadb01]- Connect and check the new versions
[oracle@exadatadb01] telnet cisco_switch Trying 10.11.12.13... Connected to cisco_switch. Escape character is '^]'. User Access Verification Password: cisco_switch> enable Password: cisco_switch# show version Cisco IOS Software, Catalyst 4500 L3 Switch Software (cat4500-IPBASEK9-M), Version 15.0(2)SG11, RELEASE SOFTWARE (fc2) Technical Support: http://www.cisco.com/techsupport Copyright (c) 1986-2016 by Cisco Systems, Inc. Compiled Mon 24-Oct-16 10:00 by prod_rel_team Image text-base: 0x10000000, data-base: 0x12097BA8 ROM: 12.2(31r)SGA2 Dagobah Revision 226, Swamp Revision 5 cisco_switch uptime is 1 minute System returned to ROM by reload System restarted at 19:50:30 CST Thu Nov 29 2018 System image file is "bootflash:cat4500-ipbasek9-mz.150-2.SG11.bin" -- This is the new firmware . . . cisco WS-C4948 (MPC8245) processor (revision 0) with 262144K bytes of memory. Processor board ID FOX1543G8VT MPC8245 CPU at 266Mhz, Fixed Module Last reset from Reload 1 Virtual Ethernet interface 48 Gigabit Ethernet interfaces 511K bytes of non-volatile configuration memory. Configuration register is 0x2102 cisco_switch#
3.5/ Configure SSH access
We can now configure SSh access to the Switch.
cisco_switch# conf terminal Enter configuration commands, one per line. End with CNTL/Z. cisco_switch(config)#crypto key generate rsa The name for the keys will be: cisco_switch.tops.gdi Choose the size of the key modulus in the range of 360 to 2048 for your General Purpose Keys. Choosing a key modulus greater than 512 may take a few minutes. How many bits in the modulus [512]: 768 % Generating 768 bit RSA keys, keys will be non-exportable...[OK] cisco_switch(config)#username admin password 0 xxxxx -- Put the password you want here cisco_switch(config)#line vty 0 4 cisco_switch(config-line)#transport input all cisco_switch(config-line)#exit cisco_switch(config)#aaa new-model cisco_switch(config)#ip ssh time-out 60 cisco_switch(config)#ip ssh authentication-retries 3 cisco_switch(config)#ip ssh version 2 cisco_switch(config)#^Z -- This is CTRL + Z cisco_switch# show ip ssh SSH Enabled - version 2.0 Authentication timeout: 60 secs; Authentication retries: 3 cisco_switch#
3.6/ Verify SSH
Let's double check that the SSH access is now working as expected:
[oracle@exadatadb01] ssh admin@cisco_switch no matching cipher found: client arcfour,aes128-ctr,aes192-ctr,aes256-ctr server aes128-cbc,3des-cbc,aes192-cbc,aes256-cbc [oracle@exadatadb01]If you face the same issue as above, you can use the below workaround:
[oracle@exadatadb01] ssh admin@cisco_switch -c aes256-cbc The authenticity of host 'cisco_switch (10.11.12.13)' can't be established. RSA key fingerprint is 64:e4:3d:41:74:2e:ea:70:3e:dc:c0:3f:f4:89:60:3d. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'cisco_switch,10.11.12.13' (RSA) to the list of known hosts. Password: cisco_switch>Or fix the issue as described in the note "SSH Connection fail with "fatal: no hostkey alg" or "fatal: no matching cipher found". (Doc ID 1633094.1)"
3.7/ Disable telnet access (optional)
I haven't applied this disable telnet procedure in reality, the procedure is validated by Oracle support though.
cisco_switch# conf terminal Enter configuration commands, one per line. End with CNTL/Z. cisco_switch(config)# cisco_switch(config)# line vty 0 4 cisco_switch(config-line)# transport input ssh cisco_switch(config-line)# exit cisco_switch(config)# (type CTRL+Z here to end) If there are more input lines in your operational Cisco firmware, then apply SSH to remaining lines as well. Verify the number of transport lines in 'show running' output. cisco_switch(config)# line vty 5 15 cisco_switch(config-line)# transup port input ssh cisco_switch(config-line)# exit cisco_switch(config-line)# end
3.8/ Save configuration changes
Now let's save the changes we made in the configuration.
cisco_switch> enable Password: cisco_switch# copy running-config startup-config all Destination filename [startup-config]? -- Press ENTER here cisco_switch# dir bootflash: Directory of bootflash:/ 1 -rwx 15644212 Apr 3 2012 08:07:05 -05:00 cat4500-ipbase-mz.122-53.SG5.bin 2 -rwx 18095624 Nov 29 2018 18:35:59 -06:00 cat4500-ipbasek9-mz.150-2.SG11.bin 3 -rw- 6135 Nov 29 2018 19:15:04 -06:00 cisco_switch-confg-before-ssh 60817408 bytes total (27071052 bytes free) cisco_switch# copy running-config bootflash:cisco_switch-confg-with-ssh Destination filename [cisco_switch-confg-with-ssh]? -- Press ENTER here or use another name 6170 bytes copied in 0.972 secs (6348 bytes/sec) cisco_switch# write memory Building configuration... Compressed configuration from 6239 bytes to 2635 bytes[OK] cisco_switch# dir bootflash: Directory of bootflash:/ 1 -rwx 15644212 Apr 3 2012 08:07:05 -05:00 cat4500-ipbase-mz.122-53.SG5.bin 2 -rwx 18095624 Nov 29 2018 18:35:59 -06:00 cat4500-ipbasek9-mz.150-2.SG11.bin 3 -rw- 6135 Nov 29 2018 19:15:04 -06:00 cisco_switch-confg-before-ssh 4 -rw- 6170 Nov 29 2018 20:10:49 -06:00 cisco_switch-confg-with-ssh 60817408 bytes total (27064752 bytes free) cisco_switch#
3.9/ copy the backup config file to the TFTP server
cisco_switch# copy bootflash:cisco_switch-confg-with-ssh tftp: Address or name of remote host []? 10.100.100.100 Destination filename [cisco_switch-confg-with-ssh]? -- Press ENTER or use another name !! 6170 bytes copied in 1.076 secs (5734 bytes/sec) cisco_switch#
3.10/ Reboot the switch
I recommend rebooting the switch to be sure that everything will come online as expected after all the changes we made.
cisco_switch#reload
span class="fred_command">Proceed with reload? [confirm] -- Press ENTER when ready
Connection to cisco_switch closed by remote host.
Connection to cisco_switch closed.
[oracle@exadatadb01]
3.11/ Test SSH access again
[oracle@exadatadb01] ssh admin@cisco_switch -c aes256-cbc Password: cisco_switch> show version Cisco IOS Software, Catalyst 4500 L3 Switch Software (cat4500-IPBASEK9-M), Version 15.0(2)SG11, RELEASE SOFTWARE (fc2) Technical Support: http://www.cisco.com/techsupport Copyright (c) 1986-2016 by Cisco Systems, Inc. Compiled Mon 24-Oct-16 10:00 by prod_rel_team Image text-base: 0x10000000, data-base: 0x12097BA8 ROM: 12.2(31r)SGA2 Dagobah Revision 226, Swamp Revision 5 cisco_switch uptime is 3 minutes System returned to ROM by reload System restarted at 20:16:41 CST Thu Nov 29 2018 System image file is "bootflash:cat4500-ipbasek9-mz.150-2.SG11.bin" Last reload reason: Reload command . . . cisco WS-C4948 (MPC8245) processor (revision 0) with 262144K bytes of memory. Processor board ID FOX1543G8VT MPC8245 CPU at 266Mhz, Fixed Module Last reset from Reload 1 Virtual Ethernet interface 48 Gigabit Ethernet interfaces 511K bytes of non-volatile configuration memory. Configuration register is 0x2102 cisco_switch>We can still SSH to the switch, everything works as expected, congratulations !
4/ The rollback procedure
It is always good to have a rollback procedure in case of something goes wrong. I have never applied the below procedure but it is validated by Oracle support.
-- Set the switch to boot with the previous firmware
cisco_switch#configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
cisco_switch#config-register 0x2102
cisco_switch#no boot system
cisco_switch#boot system bootflash:cat4500-ipbase-mz.122-53.SG5.bin -- This points to the previous firmware
cisco_switch#
cisco_switch# (type CTRL-Z here to end)
cisco_switch#show run | include boot
boot-start-marker
boot system bootflash:cat4500-ipbase-mz.122-53.SG5.bin
boot-end-marker
cisco_switch#copy running-config startup-config all
cisco_switch#write memory
Building configuration...
Compressed configuration from 6725 bytes to 2261 bytes[OK]
--- Reboot the switch
cisco_switch# reload
Enjoy !
-- Set the switch to boot with the previous firmware cisco_switch#configure terminal Enter configuration commands, one per line. End with CNTL/Z. cisco_switch#config-register 0x2102 cisco_switch#no boot system cisco_switch#boot system bootflash:cat4500-ipbase-mz.122-53.SG5.bin -- This points to the previous firmware cisco_switch# cisco_switch# (type CTRL-Z here to end) cisco_switch#show run | include boot boot-start-marker boot system bootflash:cat4500-ipbase-mz.122-53.SG5.bin boot-end-marker cisco_switch#copy running-config startup-config all cisco_switch#write memory Building configuration... Compressed configuration from 6725 bytes to 2261 bytes[OK] --- Reboot the switch cisco_switch# reload
No comments:
Post a Comment