Exadata: Infiniband Switches ILOM -- stop, start, restart, status

Database nodes and storage servers in an Exadata have a dedicated ILOM running independantly from the OS. Each ILOM is a PCI card adapter (number 12 on the below picture which comes from this oracle documentation, you'll also find in this documentation the description of all the below numbers).

But this is a different story for the Infiniband Switches; indeed, the IB switches have a software ILOM (and not a hardware ILOM) which you access using the spsh command (you'll find an example in this post).

So when this ILOM is hung (which can hang your IB switches patching for example), you can restart it easily; first of all, we can check its status:

[root@exa01sw-ib3 ~]# service ilom status
ILOM stack is partly started with 16 processes.
ILOM daemons that failed to start are :  stdiscoverer  <== something wrong here
However, ILOM stack subsystem is locked..... OK!
[root@exa01sw-ib3 ~]#

And then restart the ILOM:

[root@exa01sw-ib3 ~]# service ilom restart
Stopping ILOM stack
Stopping Servicetags listener: stlistener.
. . .
Running ntpdate...
29 Oct 00:10:25 ntpdate[5508]: step time server 10.55.111.125 offset 0.001666 sec
Starting Servicetags listener: stlistener.
Starting platform_logger
[root@exa01sw-ib3 ~]# service ilom status
ILOM stack is running.
[root@exa01sw-ib3 ~]#

You can also stop/start the ILOM instead of restart; let's do that on another switch:

[root@exa01sw-ib2 ~]# service ilom stop
Stopping ILOM stack
. . .
Stopping capidirect daemon: capidirectd  Done
Created dump file: /coredump/sp_trace/reboot/dump.gz
[root@exa01sw-ib2 ~]# service ilom start
Creating home directories
Updating FW version
. . .
Running ntpdate...
29 Oct 00:06:45 ntpdate[8892]: step time server 10.55.111.125 offset -0.000279 sec
Starting Servicetags listener: stlistener.
Starting platform_logger
[root@exa01sw-ib2 ~]#

That's all folks, another issue easily fixed !

Exadata : restart SSH on a storage cell with no SSH access

Few days ago, I was preparing an Exadata upgrade to 18c, I then started by checking the versions of each component and I got an error that the cell storage 2 was unreachable :

[root@exadatadb01]# ./exa-versions.sh

                Cluster is a X5-2 Elastic Rack HC 8TB

           -- Database Servers

   exadatadb01            exadatadb02          exadatadb03      exadatadb04
--------------------------------------------------------------------------------
  12.2.1.1.7.180506   12.2.1.1.7.180506   12.2.1.1.7.180506   12.2.1.1.7.180506
--------------------------------------------------------------------------------


Unable to connect to cells: ['exadatacel02']
           -- Cells

   exadatacel01          exadatacel03        exadatacel04        exadatacel05        exadatacel06
----------------------------------------------------------------------------------------------------
  12.2.1.1.7.180506   12.2.1.1.7.180506   12.2.1.1.7.180506   12.2.1.1.7.180506   12.2.1.1.7.180506
----------------------------------------------------------------------------------------------------


         -- Infiniband Switches

    exadatasw-ib2      exadatasw-ib3
----------------------------------------
       2.2.9-3             2.2.9-3
----------------------------------------

[root@exadatadb01]#

I then tried to ping (with success) and login (with no success):

[root@exadatadb01]# ping exadatacel02
PING exadatacel02 (10.11.12.13) 56(84) bytes of data.
64 bytes from exadatacel02 (10.11.12.13): icmp_seq=1 ttl=64 time=0.169 ms
64 bytes from exadatacel02 (10.11.12.13): icmp_seq=2 ttl=64 time=0.153 ms
64 bytes from exadatacel02 (10.11.12.13): icmp_seq=3 ttl=64 time=0.152 ms
^C
--- exadatacel02 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2939ms
rtt min/avg/max/mdev = 0.152/0.158/0.169/0.007 ms
[root@exadatadb01]#  ssh exadatacel02
ssh: connect to host exadatacel02 port 22: Connection refused
[root@exadatadb01]#

It looked like the cell was still up and running (there was no other alert) but SSH was dead. I could have rebooted the cell from the ILOM but it may have been overreacted as the cell seemed to work fine.
Anyhow, I needed to connect to the cell to check what was going on.

But how to connect there with no SSH ?

Let's start by connecting to the cell ILOM (indeed, this is the "closest" to the cell we can go as the cell is not SSHable) :

[root@exadatadb01]# ssh exadatacel02-ilom
Password:
Oracle(R) Integrated Lights Out Manager
Version 4.0.2.26.a r123797
Copyright (c) 2018, Oracle and/or its affiliates. All rights reserved.
Warning: HTTPS certificate is set to factory default.
Hostname: exadatacel02-ilom
->

But now, how to get to the cell itself?

Let's start the console to get even closer to the cell itself:

->start /sp/console
Are you sure you want to start /SP/console (y/n)?y
Serial console started.  To stop, type ESC (

And here is the trick : "CTRL + D" will give you the cell prompt to login ! (pressing ENTER as well)

-> ^D
exadatacel02 login: root
Password:
[root@exadatacel02 ~]#

I could then check that SSH was indeed dead:

[root@exadatacel02 ~]# service sshd status
openssh-daemon dead but pid file exists
[root@exadatacel02 ~]#

Also check that all the disks were online and then working properly:

[root@exadatacel02 ~]# cellcli
CellCLI: Release 12.2.1.1.7 - Production on Sun Dec 09 17:17:36 CST 2018
Copyright (c) 2007, 2016, Oracle and/or its affiliates. All rights reserved.
CellCLI> list griddisk attributes name,asmmodestatus
         DATA_CD_00_exadatacel02      ONLINE
         DATA_CD_01_exadatacel02      ONLINE
         DATA_CD_02_exadatacel02      ONLINE
         DATA_CD_03_exadatacel02      ONLINE
         DATA_CD_04_exadatacel02      ONLINE
         DATA_CD_05_exadatacel02      ONLINE
         DATA_CD_06_exadatacel02      ONLINE
         DATA_CD_07_exadatacel02      ONLINE
         DATA_CD_08_exadatacel02      ONLINE
         DATA_CD_09_exadatacel02      ONLINE
         DATA_CD_10_exadatacel02      ONLINE
         DATA_CD_11_exadatacel02      ONLINE
         DBFS_DG_CD_02_exadatacel02   ONLINE
         DBFS_DG_CD_03_exadatacel02   ONLINE
         DBFS_DG_CD_04_exadatacel02   ONLINE
         DBFS_DG_CD_05_exadatacel02   ONLINE
         DBFS_DG_CD_06_exadatacel02   ONLINE
         DBFS_DG_CD_07_exadatacel02   ONLINE
         DBFS_DG_CD_08_exadatacel02   ONLINE
         DBFS_DG_CD_09_exadatacel02   ONLINE
         DBFS_DG_CD_10_exadatacel02   ONLINE
         DBFS_DG_CD_11_exadatacel02   ONLINE
         RECO_CD_00_exadatacel02      ONLINE
         RECO_CD_01_exadatacel02      ONLINE
         RECO_CD_02_exadatacel02      ONLINE
         RECO_CD_03_exadatacel02      ONLINE
         RECO_CD_04_exadatacel02      ONLINE
         RECO_CD_05_exadatacel02      ONLINE
         RECO_CD_06_exadatacel02      ONLINE
         RECO_CD_07_exadatacel02      ONLINE
         RECO_CD_08_exadatacel02      ONLINE
         RECO_CD_09_exadatacel02      ONLINE
         RECO_CD_10_exadatacel02      ONLINE
         RECO_CD_11_exadatacel02      ONLINE
CellCLI>

And then restart SSH:

[root@exadatacel02 ~]# service sshd status
openssh-daemon dead but pid file exists
[root@exadatacel02 ~]# service sshd start
Starting sshd: [  OK  ]
[root@exadatacel02 ~]# service sshd status
openssh-daemon (pid  16249) is running...
[root@exadatacel02 ~]#

Then, re running my exa-versions script, I could verify that everything was in good shape before starting to upgrade this Exadata to 18c !

[root@exadatadb01]# ./exa-versions.sh

                Cluster is a X5-2 Elastic Rack HC 8TB

           -- Database Servers

      exadatadb01        exadatadb02          exadatadb03         exadatadb04
--------------------------------------------------------------------------------
  12.2.1.1.7.180506   12.2.1.1.7.180506   12.2.1.1.7.180506   12.2.1.1.7.180506
--------------------------------------------------------------------------------


           -- Cells

      exadatacel01      exadatacel02         exadatacel03        exadatacel04         exadatacel05      exadatacel06
------------------------------------------------------------------------------------------------------------------------
  12.2.1.1.7.180506   12.2.1.1.7.180506   12.2.1.1.7.180506   12.2.1.1.7.180506   12.2.1.1.7.180506   12.2.1.1.7.180506
------------------------------------------------------------------------------------------------------------------------


         -- Infiniband Switches

    exadatasw-ib2      exadatasw-ib3
----------------------------------------
       2.2.9-3             2.2.9-3
----------------------------------------


[root@exadatadb01]#

Enjoy !

Exadata : How to take an ILOM snapshot with the command line

When working with Exadata, you will eventually have to provide an ILOM snapshot to support.

The procedure is described in this documentation but honestly it was not really clear to me before I did it as the procedure does not show real examples.

Before describing a real example from an ILOM snapshot taken from a production cell, one point is worth to be mentioned here :

• A full ILOM snapshot (which is the one Oracle support will most likely ask you) may (yes, "may") reset the host as per the documentation :

Note - Using this option might reset the host operating system.

"Reset the host" meaning rebooting the host.

I did it few times on production cells and they have never been rebooted but this is something to keep in mind if you are asked to take a full ILOM snapshot of a database server. Indeed, a cell reboot would be transparent but this is a different story with a database server.

Unless a full snapshot is specifically required by Oracle Support, I recommend to take a normal ILOM snapshot instead of a full which is 99% of the time enough to troubleshoot an issue (and no resest host risk).

Having said that, one more thing before describing the procedure itself is to check what is needed to proceed :

• The name of the target you want to take the snapshot on; in my example I'll take it on "myclustercel07" 
• An access to the target's ILOM: here "myclustercel07-ilom"
• The ILOM root password (default is welcome1)
• The IP address of the host you want to store the ILOM on (it will be 10.11.12.13 in the below example); I personally use the database server 1 for this purpose : "myclusterdb01"; just ping it to get its IP as the ILOM won't resolve the hostname

First of all, let's connect and set the snapshot type to full :

  [root@myclusterdb01 ~]# ssh myclustercel07-ilom
  Password:
  Oracle(R) Integrated Lights Out Manager
  Version 3.2.7.30.a r112904
  Copyright (c) 2016, Oracle and/or its affiliates. All rights reserved.
  Warning: HTTPS certificate is set to factory default.
  Hostname: myclustercel07-ilom
  -> set /SP/diag/snapshot dataset=full    # or "normal" for a non full ILOM snapshot
  
  Set 'dataset' to 'full'
  ->

Then start the ILOM snapshot using the IP of the target system we will put the ILOM on and its root password (it'll copy the ILOM snapshot in /tmp in the below example) :

  -> set /SP/diag/snapshot dump_uri=sftp://root@10.11.12.13/tmp
  Collecting a "full" dataset may reset the host. Are you sure (y/n)? y
  Enter remote user password: ************
  Set 'dump_uri' to 'sftp://root@10.11.12.13/tmp'
  
  -> 

An alternative syntax with the password in the command line is also an option :

  -> set /SP/diag/snapshot dump_uri=sftp://root:root_password@10.11.12.13/tmp
  Collecting a "full" dataset may reset the host. Are you sure (y/n)? y
  Set 'dump_uri' to 'sftp://root@10.11.12.13/tmp'
  
  -> 

Now that the ILOM snapshot has been started, you can monitor it using the below command :

  -> show /SP/diag/snapshot
  
   /SP/diag/snapshot
      Targets:
  
      Properties:
          dataset = full
          dump_uri = (Cannot show property)
          encrypt_output = false
          result = Running
  
      Commands:
          cd
          set
          show
  
  ->

After few minutes you should see the ILOM snapshot as completed :

  -> show /SP/diag/snapshot
  
   /SP/diag/snapshot
      Targets:
      Properties:
          dataset = full
          dump_uri = (Cannot show property)
          encrypt_output = false
          result = Collecting data into sftp://root@10.11.12.13/tmp/myclustercel07-ilom_1133FMM02D_2018-02-04T23-18-06.zip
                   Snapshot Complete.
                   Done.
  
      Commands:
          cd
          set
          show
  ->

This is actually quite a small file easy to transfer to MOS :

  [root@myclusterdb01 ~]# du -sh /tmp/myclustercel07-ilom_1133FMM02D_2018-02-04T23-18-06.zip
  2.5M    /tmp/myclustercel07-ilom_1133FMM02D_2018-02-04T23-18-06.zip
  [root@myclusterdb01 ~]#[root@myclusterdb01 ~]#

Et voila !

Exadata : How to reboot a database server using its ILOM

It may happen that, during an Exadata patching for example, an Exadata database server does not reboot properly then you cannot connect any more and this makes your patch to hang forever. You'll then have to reboot it manually. There is nothing tricky here but it is always good to have the procedure handy when it happens.
For this purpose we will be using its ILOM which is the administration console each Exadata component has. Be sure to have :

• The database server ILOM IP (usually <dbserver-name>-ilom like <mycluster>db02-ilom)
• The ILOM root's password (in case of, the default password is welcome1)

Please note that I will describe the command line way of rebooting a server using its ILOM here and not the graphical way. Indeed, depending on how you have to connect to the servers (firewalls, jump servers, sudo, etc...) you may not be able to access the ILOM graphical interface.


Please find the procedure below :

[root@myclusterdb01 ~]# ssh myclusterdb04-ilom
Warning: Permanently added the RSA host key for IP address '10.191.84.24' to the                                                                            list of known hosts.
Password
Oracle(R) Integrated Lights Out Manager
Version 3.2.8.25 r114493
Copyright (c) 2016, Oracle and/or its affiliates. All rights reserved.
Warning: HTTPS certificate is set to factory default.
Hostname: myclusterdb04-ilom

-> reset /SYS
Are you sure you want to reset /SYS (y/n)? y
Performing hard reset on /SYS
->

This would have started a hard reboot of the myclusterdb04 database server.

You can then connect to the console to have a look at what is happening  (the server boot logs) :

-> start /sp/console
Are you sure you want to start /SP/console (y/n)? y
Serial console started.  To stop, type ESC (
. . .
[INFO] /usr/sbin/ipmitool user set name 4 iu_ngtmh
[INFO] /usr/sbin/ipmitool user set password 4 ********
[INFO] Executing: /usr/bin/mstflint -y -d /proc/bus/pci/40/00.0 -i /var/log/exadatatmp/firmware/ActualFirmwareFiles/fw-ConnectX3-rel-2_35_5532-15-7046442_7092757.bin  burn

    Current FW version on flash:  2.11.1280
    New FW version:               2.35.5532

[INFO] run /usr/sbin/ipmitool cmd to set /SP/users/iu_ngtmh/role=aucro
Burning FS2 FW image without signatures - 7[INFO] export IPMI_PASSWORD=********
[INFO] /usr/sbin/ipmiflash -v -I lanplus -H 10.191.84.24 -U iu_ngtmh -E write /var/log/exadatatmp/firmware/ActualFirmwareFiles/ILOM-3_2_10_22_a_r121452-Sun_Server_X4-2.pkg force script config delaybios warning=0
Burning FS2 FW image without signatures - OK
Restoring signature                     - OK
[INFO] Waiting for the service processor to finish firmware upgrade for up to 1200 seconds.
. . .

Give few minutes to the server to reboot and you're done.

Please keep in mind that :

• Unlike an Infiniband Switch, you do not have to use the spsh command to jump into the ILOM shell as you are using the dedicated ILOM IP address to connect to
• Note that you have to use this weird ILOM syntax to quit the console : ESC and then "("

Hope it helps

Exadata : How to manually reboot an Infiniband Switch

It may happen that, during an Exadata patching session or not, you'll have to manually reboot an Infiniband (IB) Switch; it is kind of easy, there's just few things to know :

• Do NOT reboot both Exadata Switches at the same time -- you'll get into lots of trouble
• Each Exadata component has a dedicated ILOM that can be accessed by a dedicated IP address (please find an example with a database server here) but not the IB Swicthes; an IB Switch ILOM is embedded within the Switch itself and has to be accessed using the ILOM shell with the "spsh" command and then use the "reset /SP" command to reboot the Switch as shown below

[root@myclusterdb01 ~]# ssh myclustersw-ib3

Last login: Wed Dec 20 17:58:46 2017 from myclusterdb01.mydomain.com
You are now logged in to the root shell.
It is recommended to use ILOM shell instead of root shell.
All usage should be restricted to documented commands and documented
config files.
To view the list of documented commands, use "help" at linux prompt.
[root@myclustersw-ib3 ~]# spsh

Oracle(R) Integrated Lights Out Manager
Version ILOM 3.0 r47111
Copyright (c) 2012, Oracle and/or its affiliates. All rights reserved.
->  reset /SP
Are you sure you want to reset /SP (y/n)? y

Performing reset on /SP Broadcast message from root (Sun Jan 28 20:14:42 2018):
The system is going down for reboot NOW!
-> Connection to myclustersw-ib3 closed by remote host.
Connection to myclustersw-ib3 closed.
[root@myclusterdb01 ~]#

Give it few minutes and you're done !