ODA Patching from version 19.21 to 19.23

 ODA Patching from version 19.21 to 19.23

This document will help you in ODA patching to upgrade from version 19.21 to version 19.23

If we have ODA box with version 19.17 then we cannot directly patch it to 19.23 version. We must patch the ODA box to version to 19.21 and then we can go on patch 19.23.

To perform ODA patching VM snapshots and RMAN backup is most important; if anything fails then we can restore VMs from the snapshot.

1.       Download 19.23 patches from Oracle site:

https://docs.oracle.com/en/engineered-systems/oracle-database-appliance/19.23/cmtrn/oda-patches.html#GUID-ACB179CD-5901-405C-B732-AD6923D78339

 

                     Patch list:

Oracle Database Appliance Server Patch for the ODACLI/DCS stack (patch 36524605),
Oracle Grid Infrastructure clone files (patch 30403673),
Oracle Database clone files (patch 30403662)

2.       Login to the Oda bare metal (BM) servers and note down the VMs and DB system details.

#virsh list    >>> This command shows VM names and we will be taking VM snapshots for the same before starting patching

 

#odacli list-vms  >>> This command list VMs information like VM names, they are active on which node,current state, target state etc…

 

3.       Identify the vdisk assosiated with the respective VM

#virsh domblklist <VM Name>

 

4.       Here we are taking snapshot of vda and vdb disk so note down the size of the vdisk and here we will ignore hda as it is iso

Note: Bring down respective VM before taking snapshot.

5.       Create VM snapshot for the both the vdisks

6.       Save the VM snapshot to some NFS location

 

Actual Patching process:

Before starting the patching process make sure DB and DB system and crs cluster service should be running.

 

Run below commands from BM server

1.       Update repository with server patch

#unzip p35938481_1923000_Linux-x86-64.zip             >>> The zip file will contain oda-sm-19.23.0.0.0-date-server.zip and readme.html file

 

2.       Update the repository with the server software file

# odacli update-repository -f /tmp/oda-sm-19.23.0.0.0-date-server.zip

 

3.       Confirm that the repository update is successful

                              # odacli describe-job -i <job_ID>           >>> every odacli commands generate some                                         job id

4.       Update DCS admin/Components/Agents

4.1   Update DCS admin

                              # odacli update-dcsadmin -v 19.23.0.0.0

4.2   Update the DCS components

# odacli update-dcscomponents -v 19.23.0.0.0 

4.3   Update the DCS agent

#odacli update-dcsagent -v 19.23.0.0.0

 

5.       Unzip below patches:

Oracle Grid Infrastructure clone files (patch 30403673),
Oracle Database clone files (patch 30403662)

 

6.       Update the repository with the Oracle Grid Infrastructure clone file and the Oracle Database clone file:

# odacli update-repository -f /tmp/odacli-dcs-19.23.0.0.0-date-GI-19.23.0.0.zip

# odacli update-repository -f /tmp/odacli-dcs-19.23.0.0.0-date-DB-19.23.0.0.zip

 

7.       Create prepatch report for server

# odacli create-prepatchreport -s -v 19.23.0.0.0

 

8.       Verify that the patching pre-checks ran successfully

# odacli describe-prepatchreport -i <job ID>        

Note: Fix the warnings and errors mentioned in the report and proceed with the server patching.

 

9.       Apply the server update. 

                              # /opt/oracle/dcs/bin/odacli update-server -v 19.23.0.0.0

 

10.   Confirm that the server update is successful

        # /opt/oracle/dcs/bin/odacli describe-job -i <job_ID>

 

Run below commands from DB server

11.   Update DCS admin/Components/Agents

11.1           Update DCS admin

                              # odacli update-dcsadmin -v 19.23.0.0.0

11.2           Update the DCS components

# odacli update-dcscomponents -v 19.23.0.0.0

11.3           Update the DCS agent

#odacli update-dcsagent -v 19.23.0.0.0

 

12.   Update the repository with the Oracle Grid Infrastructure clone file and the Oracle Database clone file:

# odacli update-repository -f /tmp/odacli-dcs-19.23.0.0.0-date-GI-19.23.0.0.zip

# odacli update-repository -f /tmp/odacli-dcs-19.23.0.0.0-date-DB-19.23.0.0.zip

 

13.   Apply the server update. 

                              # /opt/oracle/dcs/bin/odacli update-server -v 19.23.0.0.0               >>> verify it by                                               using command “#odacli describe-component”

14.   Create prepatch report for DB home

# odacli create-prepatchreport --dbhome --dbhomeid d8307f2e-c126-41da-ab6a-1a7f23c5c074 -v 19.23.0.0.0

We can get dbhome id by using command “#odacli list-databases”

 

15.   Verify the pre-patch report; it should be successful.

Note down the job id and run below command to verify

#odacli describe-jobs -I <job-id>

 

16.   Update db_home

# odacli update-dbhome --id d8307f2e-c126-41da-ab6a-1a7f23c5c074 -v 19.23.0.0.0 -f 

 

17.   Note down the job id and observe the status; it should be successful.

18.   Verify patching by using command:

# odacli describe-component        >>> we can see here version 19.23

Regards,

Kiren Jadhav

Exadata Patching steps from Beginning

 Exadata Patching steps from start to end: 

* Exadata patching doc number : doc id 888828.1

* Patching sequence: Below patching sequence we have to follow while doing exadata patching:

  1. DB/Grid patching

  2. Cell node patching

  3. Compute node patching

  4. Roce/Ibswitch patching 

Patching steps:

1. Open the doc id 888828.1 in MOS (My Oracle Support) to know the latest (N) or N-1 patch information. Identify the exact patch you want to download

2. Copy the patches to the server ( there will be approx 10 files)

3. After copying patches to the server : 

[root@testJan_2022]# ls -lrth

total 30G

-rw-r--r-- 1 root root 3.1G Apr 22 17:05 p33567288_210000_Linux-x86-64_1of10.zip

-rw-r--r-- 1 root root 3.1G Apr 22 17:18 p33567288_210000_Linux-x86-64_3of10.zip

-rw-r--r-- 1 root root 3.1G Apr 22 17:27 p33567288_210000_Linux-x86-64_9of10.zip

-rw-r--r-- 1 root root 1.7G Apr 22 17:33 p33567288_210000_Linux-x86-64_10of10.zip

-rw-r--r-- 1 root root 3.1G Apr 22 18:18 p33567288_210000_Linux-x86-64_2of10.zip

-rw-r--r-- 1 root root 3.1G Apr 22 18:20 p33567288_210000_Linux-x86-64_7of10.zip

-rw-r--r-- 1 root root 3.1G Apr 22 18:24 p33567288_210000_Linux-x86-64_5of10.zip

-rw-r--r-- 1 root root 3.1G Apr 22 18:58 p33567288_210000_Linux-x86-64_8of10.zip

-rw-r--r-- 1 root root 3.1G Apr 22 19:00 p33567288_210000_Linux-x86-64_6of10.zip

-rw-r--r-- 1 root root 3.1G Apr 22 19:06 p33567288_210000_Linux-x86-64_4of10.zip

4. Then unzip all patch files

#unzip p33567288_210000_Linux-x86-64_1of10.zip

.

#unzip p33567288_210000_Linux-x86-64_10of10.zip

or 

#unzip '*.zip'

4. Unzipping files will create tar files something like below:

-rw-r--r-- 1 root root 3.1G Jan 21 15:12 33567288.tar.splitaa

-rw-r--r-- 1 root root 3.1G Jan 21 15:12 33567288.tar.splitab

-rw-r--r-- 1 root root 3.1G Jan 21 15:13 33567288.tar.splitac

-rw-r--r-- 1 root root 3.1G Jan 21 15:13 33567288.tar.splitad

5. Now untar all files to create a common patch repo files:

#cat *.tar.* | tar -xvf -

6. Now unzip patch files from below directory in order to get dbnodeupdate.sh and patchmgr scripts.

#cd /QFSDP/Jan_2022/33567288/Infrastructure/SoftwareMaintenanceTools/DBNodeUpdate/21.211221

#cd /QFSDP/Jan_2022/33567288/Infrastructure/21.2.8.0.0/FabricSwitch

#cd /QFSDP/Jan_2022/33567288/Infrastructure/21.2.8.0.0/ExadataStorageServer_InfiniBandSwitch

7. Once DB team confirm that they are done with the DB/GRID patching then we can start cellnode patching.

8. Before doing actual patching we have to raise the prechecks SR in MOS and need to upload necessary logs like sosreport and sundiag from compute nodes and cell nodes, exachk from one of the compute node and the prechecks logs from compute node and cell nodes.

8.a Below is the cell node prechecks commands:

#cd /QFSDP/Jan_2022/33567288/Infrastructure/21.2.8.0.0/ExadataStorageServer_InfiniBandSwitch/patch_21.2.8.0.0.220114.1

#./patchmgr -cells /opt/oracle.SupportTools/onecommand/cell_group -reset_force

#./patchmgr -cells /opt/oracle.SupportTools/onecommand/cell_group -cleanup

#./patchmgr -cells /opt/oracle.SupportTools/onecommand/cell_group -patch_check_prereq -rolling -ignore_alerts

Note : If error found in prechecks get it rectified with the help of backend team.

The possible error could be:

https://kiranbjadhav.blogspot.com/2022/05/exadata-cell-node-patching-errorusb.html

8.b If there is no error found in prechecks then we can proceed with the actual patching commands.

Actual patching command :

#cd /QFSDP/Jan_2022/33567288/Infrastructure/21.2.8.0.0/ExadataStorageServer_InfiniBandSwitch/patch_21.2.8.0.0.220114.1

#./patchmgr -cells /opt/oracle.SupportTools/onecommand/cell_group -reset_force

#./patchmgr -cells /opt/oracle.SupportTools/onecommand/cell_group -cleanup

#./patchmgr -cells /opt/oracle.SupportTools/onecommand/cell_group -patch -ignore_alerts   

all cells patching will get completed (DB needs to be down here)

If you are doing cellnode patching without DB downtime then 

#./patchmgr -cells /opt/oracle.SupportTools/onecommand/cell_group -patch -rolling -ignore_alerts   

all cells will get patched in rolling mode one by one.

If you want to do manual cell node patching taking one cell at a time then

#./patchmgr -cells /opt/oracle.SupportTools/onecommand/cell_group_1 -patch -rolling -ignore_alerts  ---> 1 cell at a time assuming cell_group_1 has only one cell entry.

#./patchmgr -cells /opt/oracle.SupportTools/onecommand/cell_group_2 -patch -rolling -ignore_alerts and so on

9. Once cell nodes patching completed successfully then we can start with Compute node patching 

9.a Compute node patching prechecks

#cd /QFSDP/Jan_2022/33567288/Infrastructure/SoftwareMaintenanceTools/DBNodeUpdate/21.211221

#./dbnodeupdate.sh -u -l /QFSDP/Jan_2022/33567288/Infrastructure/21.2.8.0.0/ExadataDatabaseServer_OL7/p33665705_212000_Linux-x86-64.zip -v 

The possible error could be:

https://kiranbjadhav.blogspot.com/2022/05/exadata-compute-node-patching-prechecks.html

9.b Actual patching command: 

Prerequisite:

. Once compute node at a time and 

. DB and CRS must be down on the particular compute node

. NFS mount point should be unmounted

#./dbnodeupdate.sh -u -l /QFSDP/Jan_2022/33567288/Infrastructure/21.2.8.0.0/ExadataDatabaseServer_OL7/p33665705_212000_Linux-x86-64.zip

9.c After successful compute node upgrade run below command to finish post steps.

#./dbnodeupdate.sh -c 

10. After successful compute node patching we can do ROCE switch or IBswitch patching

10.a Roce switch patching prechecks:

#cd /QFSDP/Jan_2022/33567288/Infrastructure/21.2.8.0.0/FabricSwitch/patch_switch_21.2.8.0.0.220114.1

#./patchmgr --roceswitches /roceswitches.lst --upgrade --roceswitch-precheck --log_dir /scratchpad/

10.b Actual patching command:

./patchmgr --roceswitches /roceswitches.lst --upgrade --log_dir /scratchpad/

10.c If ibswitches are there instead of Roce switches

#cd /QFSDP/Jan_2022/33567288/Infrastructure/21.2.8.0.0/ExadataStorageServer_InfiniBandSwitch/patch_21.2.8.0.0.220114.1

  ./patchmgr -ibswitches /opt/oracle.SupportTools/onecommand/ibs_group -upgrade -ibswitch_precheck

10.d Actual patching command:

./patchmgr -ibswitches /opt/oracle.SupportTools/onecommand/ibs_group -upgrade

Regards,

Kiran Jaadhav