Exadata: re-image a Database Server to 18c (Introduction)

You may be in some situation where you will have to re-image an Exadata database server. The procedure is not very well documented and we faced many issues doing this with my friend Franky.
I will describe in this blog the detailed procedure we followed to achieve this (with complications).

0/ Disclaimer

Some scenarios didn't work for us; it does not mean that you cannot make them work for you. It also does not mean that what is described on the official documentation and that hasn't worked for us does not work at all; it may work for you (may be :)).

We did this in April 2019, re-imaging few new X7-2 19c database servers to 18c with Oct 18 Bundle to be added to X6s. This is important as it seems that some steps described in the documentation may have worked in versions 11 (12 ?) but do not work any more.

As the procedure is very detailed, I have split it in 4 parts:

• Introduction and boot from an ISO
• Re-image the database server
• OS configuration
• Extend your Grid Infrastructure

1/ The official documentation

You can find the official documentation in the MOS note "Bare Metal Restore Procedure for Compute Nodes on an Exadata Environment (Doc ID 1084360.1)". The note does not describe the procedure itself but it points to this PDF which contains the Bare Metal Restore Procedure. This document also explains how to remove a damaged database server in case of you want to re-image after a database server replacement. I won't go on this side but stick to the procedure to re-image a database server whatever the reason you want to re-image it.

Few ways of re-imaging a database server are described in the PDF; I will roughly describe these procedures below:

• 1/ Boot on an USB key Scenario 1:
• Plug an USB key on the back of a healthy database server (not the new one)
• Ask Oracle support the makeImageMedia.sh script
• Create a bootable image of a good system on the USB key using makeImageMedia.sh
• Remove the USB key from the server
• Plug the USB key on the new server
• Reboot the new server which should boot on it and follow the instructions to re-image your database server
makeImageMedia.sh was for 11 and 12 versions, it does not apply to 18+ versions.

• 2/ Boot on an USB key Scenario 2:
• Download the ISO of the version you want provided by Oracle; example: "Patch 28666212 - Database server ISO image (18.1.9.0.0.181006)"
• Plug an USB key on the back of the server where is the downloaded ISO. It can be an healhty one or the new one, do what is easiest for you.
• Copy the ISO to the USB key with dd
• If you haven't plugged the USB key on the new server, unplug it from the server you plugged it to and plug it on the new server
• Reboot the new server which which should boot on the USB key and follow the instructions to re-image your database server

We tried this solution but the installation could not finish successfully, we were supposed to have a message asking us to unplug the USB key and reboot (according to the doc) but this never happened and we were like stuck in a kind of reboot on USB key loop.

• 3/ Boot on a PXE server: :
The procedure is described in How to setup a PXE Boot Server to Re-Image an Exadata Compute Node (Doc ID 1577323.1). Feel free to have a look, having worked with a TFTP server when upgrading the Exadata Cisco swicth, I was not really keen on going that way so we avoided it.

• 4/ Boot from NFS + ISO :
• Download the ISO of the version you want provided by Oracle; example: "Patch 28666212 - Database server ISO image (18.1.9.0.0.181006) and save it on a NFS accessible from the ILOM."
• Restart the system and boot from the CD
We tried this solution but we have never been able to boot on the CD; we could mount it, Oracle support found no issue in our procedure but we were never able to boot on it. May be due to a firewall, not sure. I am pretty positive on the fact that this should be working though.

• 5/ Boot on an ISO mounted on the ILOM from a laptop/jump server:
• Download the ISO of the version you want provided by Oracle; example: "Patch 28666212 - Database server ISO image (18.1.9.0.0.181006) and save it on your laptop / jump server."
• Restart the system and boot from the CD
This is the solution which worked for us and that we followed. This is the procedure I will be describing in detail in this blog.

2/ Mount and Boot on the ISO

2.1 Verify there is no ISO mounted on NFS

The first thing to do is to verify that you have no ISO saved in the Host Storage Device (ILOM GUI menu "Remote Control" => "Host Storage Device") -- this would prevent to boot from the ISO mounted on your laptop / jump server (as it happened to us). FYI, this is the GUI screen you use to mount the ISO from a NFS when using the scenario 4 described earlier.

Here the status should "Disabled".

2.2/ Set the next boot device to CDROM

From the ILOM GUI Menu "Host Management" => "Host Control" set the "Next Boot Device" to "CDROM".

You can also do it manually from the ILOM command line:

-> set boot_device=cdrom
-> show host
 /HOST
    Targets:
        console
        diag
        network
    Properties:
        boot_device = cdrom
        generate_host_nmi = (Cannot show property)
->

2.3/ Launch Remote Console

Now, launch the remote console from ILOM GUI menu "Remote Control" => "Redirection"

You may be asked about JAVA plugin of your browser, accept and the Remote Console will start.

2.4/ Select your ISO

We will now choose the ISO to boot from, first use menu "KVMS" => "Storage..."

In the below screen, proceed as below:

• Use the "Add..." button to select your ISO located on your laptop/jump server
• Deselect "SSL enabled"
• Click on "Connect"

And you should have the below output:

Now click "OK" to continue.

2.5/ Reboot the host

Let's reboot the host using the GUI ILOM (Menu "Host Management" => "Power Control"):

Or manually using the ILOM command line:

-> reset /system
Are you sure you want to reset /System (y/n)? y
Performing hard reset on /System
->

2.6 Booting . . .

You should soon see the boot screen on the Remote Console:

No rush here, do not touch your keyboard, just wait . . . it may take a bit of time before you can see the next screen; remember we are booting from network here.

2.7/ Install Oracle Exadata

You will finally get the install Exadata boot menu with one choice: "Install Oracle Exadata"

You're now ready to jump to part 2 and start re-imaging your Exadata database server !

Quick links : part 1 / part 2 / part 3 / part 4