Twitter

Exadata: re-image a Database Server to 18c (re-image the database server)

Part 1 left you with the "Install Oracle Exadata" boot screen:
Press ENTER and continue !

3/ A First boot . . .

We can now spectate a First boot . . .
[   51.629520] bnxt_en 0000:18:00.0 eth1: Broadcom BCM57417 NetXtreme-E 10GBase-T Ethernet found at mem 381c00210000, node addr 00:10:e0:de:c5:2d
[   51.659340] bnxt_en 0000:18:00.0 eth1: PCIe: Speed 8.0GT/s Width x8
[   51.689599] bnxt_en 0000:18:00.1 eth2: Broadcom BCM57417 NetXtreme-E 10GBase-T Ethernet found at mem 381c00200000, node addr 00:10:e0:de:c5:2e
[   51.723264] bnxt_en 0000:18:00.1 eth2: PCIe: Speed 8.0GT/s Width x8
[   51.758581] bnxt_en 0000:af:00.0 eth3: Broadcom BCM57414 NetXtreme-E 10Gb/25Gb Ethernet found at mem 386c01210000, node addr b0:26:28:2d:c2:e0
[   51.787327] bnxt_en 0000:af:00.0 eth3: PCIe: Speed 8.0GT/s Width x8
[   51.818394] bnxt_en 0000:af:00.1 eth4: Broadcom BCM57414 NetXtreme-E 10Gb/25Gb Ethernet found at mem 386c01200000, node addr b0:26:28:2d:c2:e8
[   51.850085] bnxt_en 0000:af:00.1 eth4: PCIe: Speed 8.0GT/s Width x8
[INFO     ] Image id in memory: 1538869894
[INFO     ] Waiting for USB devices to be ready...may take few minutes
[INFO     ] Waiting for the virtual cdrom availability. It can take up to 1 minute ...
Interesting point here: eth3 and eth4 are the 10 Gb (and then possibly 25Gb from X7) network connections to the outside world whereas it was eth4 and eth5 in models < X7.
eth0 . . .
Ethernet interface (eth0,1,2,3) [eth0]: eth0
IP Address of this host: 10.11.12.13
Netmask of this host: 255.255.254.0
Default gateway: 10.11.12.1
[INFO     ] Check eth0 for 1000Mb/s or 10000Mb/s speed and Full duplex mode

4/ A second boot. . .

The system will now reboot again:
Saving random seed:  [  OK  ]
Syncing hardware clock to system time [  OK  ]
Turning off swap:  [  OK  ]
Turning off quotas:  [  OK  ]
Unmounting file systems:  [  OK  ]
init: Re-executing /sbin/init
Please stand by while rebooting the system...
[ 2627.922883] reboot: Restarting system
You'll see the boot screen again:
Copyright (C) 2018 Oracle and/or its affiliates. All rights reserved.
Version 2.19.1268. Copyright (C) 2018 American Megatrends, Inc.
BIOS Date: 08/17/2018 14:01:39 Ver: 41040200
Press F2 to run Setup (CTRL+E on serial keyboard)
Press F8 for BBS Popup (CTRL+P on serial keyboard)
Press F12 for network boot (CTRL+N on serial keyboard)
Boot Mode = UEFI
You may wonder if that second reboot would boot again on the ISO as this is what we configured, right ? answer is no and the boot process has changed the boot device back to default:
-> show host
 /HOST
    Targets:
        console
        diag
        network
    Properties:
        boot_device = default
        generate_host_nmi = (Cannot show property)
->

5/ A third boot . . .

And the system reboots again . .
Turning off quotas:  [  OK  ]
Unmounting file systems:  [  OK  ]
init: Re-executing /sbin/init
Please stand by while rebooting the system...
[  545.285702] reboot: Restarting system

6/ Configuration

You will finally be prompted for the configuration of the system.

6.1/ DNS

Add your DNS servers:
Nameserver: 10.100.101.102
Add more nameservers (y/n) [n]: y
Nameserver: 10.200.201.202
Add more nameservers (y/n) [n]: y
Nameserver: 10.50.51.52
Add more nameservers (y/n) [n]: n

6.2/ Server Timezone

Setting up local time...

1) Andorra
. . .
12) Austria
13) Australia
14) Aruba
15) Ã…land Islands
16) Azerbaijan
Select country by number, [n]ext, [l]ast: l
If you want an US TZ, press "l" for the last page and then "b" for back and US are "233"

6.3/ NTP

Set your NTP(s) server(s):
The current NTP server(s):
Do you want to change it (y/n) [n]: y
qualified hostname or ip address for NTP server. Press enter if none: 10.11.22.33
Continue adding more ntp servers (y/n) [n]: n

6.4/ Network interfaces

Then you are prompted to set up the network interfaces
Network interfaces
Name     Bonding  Speed    Status  Bridge  IP address  Netmask  Gateway  Net type  Hostname
ib0                        UNCONF
ib1                        UNCONF
eth0                       UNCONF
eth1                       UNCONF
eth2                       UNCONF
eth3                       UNCONF
eth4                       UNCONF
bondeth0 None              UNCONF
bondeth1 None              UNCONF
bondeth2 None              UNCONF
Select interface name to configure or press Enter to continue:
Let's start by ib0
Select interface name to configure or press Enter to continue: ib0
Selected interface. ib0
IP address or up or none: 192.168.1.2
Netmask: 255.255.252.0
Fully qualified hostname or none: newnode-priv1.domain.com
Continue configuring or re-configuring interfaces? (y/n) [y]:  y
Network interfaces
Name     Bonding  Speed    Status  Bridge  IP address   Netmask       Gateway  Net type  Hostname
ib0                        UP              192.168.1.2 255.255.252.0          Private   newnode-priv1.domain.com
ib1                        UNCONF
. . .
Continue to define the needed network interface to end with something like this:
Network interfaces
Name     Bonding   Speed    Status  Bridge     IP address    Netmask       Gateway     Net type   Hostname
ib0                         UP                 192.168.1.2  255.255.252.0             Private    newnode-priv1.domain.com
ib1                         UP                 192.168.1.3 255.255.252.0             Private    newnode-priv2.domain.com
eth0               10000Mb  UP      vmeth0     10.11.12.13 255.255.254.0 10.11.12.1 Management newnode.domain.com
eth1                        UNCONF
eth2                        UNCONF
eth3               default  UP
eth4               default  UP
bondeth0 eth3,eth4 10000Mb  UP      vmbondeth0 10.22.33.44  255.255.254.0 10.22.33.1 Other      newnodevip.domain.com
bondeth1 None               UNCONF
bondeth2 None               UNCONF
Select interface name to configure or press Enter to continue:

6.5/ Hostname

You will then be prompted to define a hostname based in the ones you defined in the network interfaces:
Select canonical hostname from the list below
1: newnode-priv1.domain.com
2: newnode-priv2.domain.com
3: newnode.domain.com
4: newnodevip.domain.com
Canonical fully qualified domain name: 3

6.6/ Default gateway

Default gateway device: bondeth0
Network interfaces
ib1                         UP                 192.168.1.3 255.255.252.0             Private    newnode-priv2.domain.com
eth0               10000Mb  UP      vmeth0     10.11.12.13 255.255.254.0 10.11.12.1 Management newnode.domain.com
eth1                        UNCONF
eth2                        UNCONF
eth3               default  UP
eth4               default  UP
bondeth0 eth3,eth4 10000Mb  UP      vmbondeth0 10.22.33.44  255.255.254.0 10.22.33.1 Other      newnodevip.domain.com
bondeth1 None               UNCONF
bondeth2 None               UNCONF
Is this correct (y/n) [y]: y

6.7 ILOM settings

Here, everything should be properly populated but it is worth double checking.
Do you want to configure basic ILOM settings (y/n) [y]: y
Loading basic configuration settings from ILOM ...
ILOM Fully qualified hostname [nenwnode-ilom.domain.com]:
Inet protocol (IPv4,IPv6) [IPv4]:
ILOM IP address [10.1.2.3]:
ILOM Netmask [255.255.254.0]:
ILOM Gateway or none [10.1.2.1]:
44,10.55.66.77]: ultiple IPs separated by a comma) or none [10.11.12.13,10.22.33. <= This are the DNS, I let the output issue you will have
ILOM Use NTP Servers (enabled/disabled) [enabled]:
1]:  First NTP server. Fully qualified hostname or ip address or none [10.99.1.1 <= This are the NTPs, I let the output issue you will have
ILOM Second NTP server. Fully qualified hostname or ip address or none [none]:
ILOM Vlan id or zero for non-tagged VLAN (0-4079) [0]:     <= We don't do VLAN taggging here

Basic ILOM configuration settings:
Hostname             : newnode-ilom.domain.com
IP Address           : 10.1.2.3
Netmask              : 255.255.254.0
Gateway              : 10.1.2.1
DNS servers          : 10.11.12.13,10.22.33.44,10.55.66.77
Use NTP servers      : enabled
First NTP server     : 10.99.1.1
Second NTP server    : none
Timezone (read-only) : America/Chicago
VLAN id              : 0
Is this correct (y/n) [y]: y

6.8/ Reboot

Now the system will reboot.
(d1) Press F12 for boot menu.
(d1)
(d1) Booting from Hard Disk...
(XEN) irq.c:398: Dom1 callback via changed to Direct Vector 0xf3
(XEN) irq.c:295: Dom1 PCI link 0 changed 5 -> 0
(XEN) irq.c:295: Dom1 PCI link 1 changed 10 -> 0
(XEN) irq.c:295: Dom1 PCI link 2 changed 11 -> 0
(XEN) irq.c:295: Dom1 PCI link 3 changed 5 -> 0

7/A Bug

I am not sure if this bug impacts only the image we used here (18.1.9.0.0.181006) or not but this bug is an annoying one. Also it is a known one as it is clearly mentioned in the ISO README:
4. Due to a bug ipconf will be prompting you twice to enter the network settings.
This means that after the reboot in point 6.8, you will have to go through all the network configuration again (it also means that Oracle doesn't seem to care fixing this kind of annoying issue, or hasn't tested the image ?).

8/ Re-image done !

After this reboot, your system is finally re-imaged !
[root@newnode ~]# imageinfo
Kernel version: 4.1.12-94.8.6.el6uek.x86_64 #2 SMP Mon Aug 6 19:10:09 PDT 2018 x86_64
Image kernel version: 4.1.12-94.8.6.el6uek
Image version: 18.1.9.0.0.181006
Image activated: 2019-04-28 18:11:00 -0500
Image status: success
System partition on device: /dev/mapper/VGExaDb-LVDbSys1
[root@newnode ~]#
I recommend you to verify everything using ibhosts, ifconfig, ethtool, verify your SSH connections to the new node, to the new node's ILOM, etc ...
Everything should be good at that point, you are then almost done. Almost ? yes, almost ! indeed your system is now re-imaged from scratch with no configuration at all which is described in part 3 !



Quick links : part 1 / part 2 / part 3 / part 4

1 comment:

  1. Boa tarde! Por que no Compute Node, informar default gateway como bondeth0 e não eth0? Sempre tem que ser a rede de cliente ao invés da rede de gerencia?

    ReplyDelete

awk Tutorial -- 3 -- IF THEN ELSE

This blog is part of an awk Tutorial, you'll find the whole list of section already covered here . Every language has an IF THEN EL...