Twitter

Exadata: re-image a Database Server to 18c (Introduction)

You may be in some situation where you will have to re-image an Exadata database server. The procedure is not very well documented and we faced many issues doing this with my friend Franky.
I will describe in this blog the detailed procedure we followed to achieve this (with complications).

0/ Disclaimer

Some scenarios didn't work for us; it does not mean that you cannot make them work for you. It also does not mean that what is described on the official documentation and that hasn't worked for us does not work at all; it may work for you (may be :)).

We did this in April 2019, re-imaging few new X7-2 19c database servers to 18c with Oct 18 Bundle to be added to X6s. This is important as it seems that some steps described in the documentation may have worked in versions 11 (12 ?) but do not work any more.

As the procedure is very detailed, I have split it in 4 parts:

1/ The official documentation

You can find the official documentation in the MOS note "Bare Metal Restore Procedure for Compute Nodes on an Exadata Environment (Doc ID 1084360.1)". The note does not describe the procedure itself but it points to this PDF which contains the Bare Metal Restore Procedure. This document also explains how to remove a damaged database server in case of you want to re-image after a database server replacement. I won't go on this side but stick to the procedure to re-image a database server whatever the reason you want to re-image it.

Few ways of re-imaging a database server are described in the PDF; I will roughly describe these procedures below:
  • 1/ Boot on an USB key Scenario 1:
    • Plug an USB key on the back of a healthy database server (not the new one)
    • Ask Oracle support the makeImageMedia.sh script
    • Create a bootable image of a good system on the USB key using makeImageMedia.sh
    • Remove the USB key from the server
    • Plug the USB key on the new server
    • Reboot the new server which should boot on it and follow the instructions to re-image your database server
    makeImageMedia.sh was for 11 and 12 versions, it does not apply to 18+ versions.

  • 2/ Boot on an USB key Scenario 2:
    • Download the ISO of the version you want provided by Oracle; example: "Patch 28666212 - Database server ISO image (18.1.9.0.0.181006)"
    • Plug an USB key on the back of the server where is the downloaded ISO. It can be an healhty one or the new one, do what is easiest for you.
    • Copy the ISO to the USB key with dd
    • If you haven't plugged the USB key on the new server, unplug it from the server you plugged it to and plug it on the new server
    • Reboot the new server which which should boot on the USB key and follow the instructions to re-image your database server
We tried this solution but the installation could not finish successfully, we were supposed to have a message asking us to unplug the USB key and reboot (according to the doc) but this never happened and we were like stuck in a kind of reboot on USB key loop.

  • 3/ Boot on a PXE server: :
  • The procedure is described in How to setup a PXE Boot Server to Re-Image an Exadata Compute Node (Doc ID 1577323.1). Feel free to have a look, having worked with a TFTP server when upgrading the Exadata Cisco swicth, I was not really keen on going that way so we avoided it.

  • 4/ Boot from NFS + ISO :
    • Download the ISO of the version you want provided by Oracle; example: "Patch 28666212 - Database server ISO image (18.1.9.0.0.181006) and save it on a NFS accessible from the ILOM."
    • Restart the system and boot from the CD
    We tried this solution but we have never been able to boot on the CD; we could mount it, Oracle support found no issue in our procedure but we were never able to boot on it. May be due to a firewall, not sure. I am pretty positive on the fact that this should be working though.

  • 5/ Boot on an ISO mounted on the ILOM from a laptop/jump server:
    • Download the ISO of the version you want provided by Oracle; example: "Patch 28666212 - Database server ISO image (18.1.9.0.0.181006) and save it on your laptop / jump server."
    • Restart the system and boot from the CD
    This is the solution which worked for us and that we followed. This is the procedure I will be describing in detail in this blog.


2/ Mount and Boot on the ISO


2.1 Verify there is no ISO mounted on NFS

The first thing to do is to verify that you have no ISO saved in the Host Storage Device (ILOM GUI menu "Remote Control" => "Host Storage Device") -- this would prevent to boot from the ISO mounted on your laptop / jump server (as it happened to us). FYI, this is the GUI screen you use to mount the ISO from a NFS when using the scenario 4 described earlier.
Here the status should "Disabled".

2.2/ Set the next boot device to CDROM

From the ILOM GUI Menu "Host Management" => "Host Control" set the "Next Boot Device" to "CDROM".

You can also do it manually from the ILOM command line:
-> set boot_device=cdrom
-> show host
 /HOST
    Targets:
        console
        diag
        network
    Properties:
        boot_device = cdrom
        generate_host_nmi = (Cannot show property)
->

2.3/ Launch Remote Console

Now, launch the remote console from ILOM GUI menu "Remote Control" => "Redirection"
You may be asked about JAVA plugin of your browser, accept and the Remote Console will start.

2.4/ Select your ISO

We will now choose the ISO to boot from, first use menu "KVMS" => "Storage..."

In the below screen, proceed as below:
  • Use the "Add..." button to select your ISO located on your laptop/jump server
  • Deselect "SSL enabled"
  • Click on "Connect"
And you should have the below output:

Now click "OK" to continue.

2.5/ Reboot the host

Let's reboot the host using the GUI ILOM (Menu "Host Management" => "Power Control"):


Or manually using the ILOM command line:
-> reset /system
Are you sure you want to reset /System (y/n)? y
Performing hard reset on /System
->

2.6 Booting . . .

You should soon see the boot screen on the Remote Console:

No rush here, do not touch your keyboard, just wait . . . it may take a bit of time before you can see the next screen; remember we are booting from network here.

2.7/ Install Oracle Exadata

You will finally get the install Exadata boot menu with one choice: "Install Oracle Exadata"

You're now ready to jump to part 2 and start re-imaging your Exadata database server !



Quick links : part 1 / part 2 / part 3 / part 4

8 comments:

  1. Very interesting article series, thanks for sharing.

    ReplyDelete
  2. Hi @unknowndba,

    Thanks for the helpful article. Does this method work with X2 db version 11.2?

    Regards

    ReplyDelete
    Replies
    1. Yes it should work *** BUT *** there should no be ISO released by Oracle for this version so you should ask MOS for “computeImageMaker” and then build your ISO yourself.

      Having said that, 11 is very old, you should upgrade to 18c (not 19c as /boot is too small on X2 to upgrade to 19c).

      Delete
  3. Hi, im trying to re-image an exadata x5 with 19.3.9 image from pxe boot server, i have been reset the dbnode. then the dbnode has started the booting and freezes on NFS:Registering the id_resolver key type. do you know how its wrong??

    thanks

    ReplyDelete
    Replies
    1. Hi,

      I could never have that to be working either, this is why I used the method I described in this blog (mount the ISO on the ILOM and boot on it).

      Delete
  4. Very useful article. Good work

    ReplyDelete
  5. Hi there, I find the information very useful and I wish to seek help as I am the process of getting a Exadata X4 for my testing.. Hence I am not sure I can engage someone in billable time to assist for reimaging for Exadata X4. Do let me know my email address jktlee86@gmail.com. Thank you.

    ReplyDelete

awk Tutorial -- 3 -- IF THEN ELSE

This blog is part of an awk Tutorial, you'll find the whole list of section already covered here . Every language has an IF THEN EL...