From oracle 12c onward Bundle patches/PSU patch can be applied to oracle GRID or database home using opatchauto utility which simplifies the patching process.
NOTE - > WHAT IS PROACTIVE BUNDLE PATCH
-> APPLY BUNDLE PATCH TO GRID USING OPATCHAUTO
However recently i faced an issue where due to opatch utility issue, we are unable to use optachauto utility . So we went ahead with the manual patching method. In the below article we will explain all the steps in detail.
ENVIROMENT:
Note - > The patching will be done in rolling fashion. I.e first patch will be applied on node1 , during which node2 will be running fine. and once node1 patching is over and its services are up, we can proceed with patching on node 2.
In this below demo, We have shown the steps for first node ( dbhost-n1).
1. Download the latest bundle patch from oracle support.
This patchset comes with below patches:
2. Copy the patch to both the nodes and unzip there.
cd /dumparea/APR2020BP
unzip p30805478_121020_SOLARIS64.zip
3. Check the recommended opatch version for both db_home and grid_home
For the recommended opatch version, always check the readme file.
A per readme file 12.2.0.19 or later is recommended. Lets check in our environment.
cd $GRID_HOME
/gridapp/app/oracle/product/grid12c/OPatch# ./opatch version
OPatch Version: 12.2.0.1.21
OPatch succeeded.
In our case the the opatch version(12.2.0.21) is latest. So no need to upgrade the opatch utility.
If your opatch version is lower than the recommended, then follow the below link to upgrade your opatch version:
LINK - > How to install/upgrade OPatch in RAC
4. Check for patch conflicts with GRID_HOME:
cd $GRID_HOME/OPatch
./opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /dumparea/APR2020BP/30805478/30691015
./opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /dumparea/APR2020BP/30805478/30703508
./opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /dumparea/APR2020BP/30805478/30703511
./opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /dumparea/APR2020BP/30805478/26983807
cd /gridapp/app/oracle/product/grid12c/OPatch
oracle@dbhost-n1:.$ ./opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /dumparea/APR2020BP/30805478/30691015
Oracle Interim Patch Installer version 12.2.0.1.21
Copyright (c) 2020, Oracle Corporation. All rights reserved.
PREREQ session
Oracle Home : /gridapp/app/oracle/product/grid12c
Central Inventory : /gridapp/app/oraInventory
from : /gridapp/app/oracle/product/grid12c/oraInst.loc
OPatch version : 12.2.0.1.21
OUI version : 12.1.0.2.0
Log file location : /gridapp/app/oracle/product/grid12c/cfgtoollogs/opatch/opatch2020-06-16_15-42-45PM_1.log
Invoking prereq "checkconflictagainstohwithdetail"
Prereq "checkConflictAgainstOHWithDetail" passed.
OPatch succeeded.
oracle@dbhost-n1:$ ./opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /dumparea/APR2020BP/30805478/30703508
Oracle Interim Patch Installer version 12.2.0.1.21
Copyright (c) 2020, Oracle Corporation. All rights reserved.
PREREQ session
Oracle Home : /gridapp/app/oracle/product/grid12c
Central Inventory : /gridapp/app/oraInventory
from : /gridapp/app/oracle/product/grid12c/oraInst.loc
OPatch version : 12.2.0.1.21
OUI version : 12.1.0.2.0
Log file location : /gridapp/app/oracle/product/grid12c/cfgtoollogs/opatch/opatch2020-06-16_15-42-56PM_1.log
Invoking prereq "checkconflictagainstohwithdetail"
Prereq "checkConflictAgainstOHWithDetail" passed.
OPatch succeeded.
oracle@dbhost-n1:...$ ./opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /dumparea/APR2020BP/30805478/30703511
Oracle Interim Patch Installer version 12.2.0.1.21
Copyright (c) 2020, Oracle Corporation. All rights reserved.
PREREQ session
Oracle Home : /gridapp/app/oracle/product/grid12c
Central Inventory : /gridapp/app/oraInventory
from : /gridapp/app/oracle/product/grid12c/oraInst.loc
OPatch version : 12.2.0.1.21
OUI version : 12.1.0.2.0
Log file location : /gridapp/app/oracle/product/grid12c/cfgtoollogs/opatch/opatch2020-06-16_15-43-04PM_1.log
Invoking prereq "checkconflictagainstohwithdetail"
Prereq "checkConflictAgainstOHWithDetail" passed.
OPatch succeeded.
oracle@dbhost-n1:.$ ./opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /dumparea/APR2020BP/30805478/26983807
Oracle Interim Patch Installer version 12.2.0.1.21
Copyright (c) 2020, Oracle Corporation. All rights reserved.
PREREQ session
Oracle Home : /gridapp/app/oracle/product/grid12c
Central Inventory : /gridapp/app/oraInventory
from : /gridapp/app/oracle/product/grid12c/oraInst.loc
OPatch version : 12.2.0.1.21
OUI version : 12.1.0.2.0
Log file location : /gridapp/app/oracle/product/grid12c/cfgtoollogs/opatch/opatch2020-06-16_15-43-12PM_1.log
Invoking prereq "checkconflictagainstohwithdetail"
Prereq "checkConflictAgainstOHWithDetail" passed.
OPatch succeeded.
If any conflict is reported, then those need to be taken care.
4. Check for patch conflicts with DB_HOME:
./opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /dumparea/APR2020BP/30805478/30691015
./opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /dumparea/APR2020BP/30805478/30703508
If any conflict is reported, then those need to be taken care.
5. Check whether sufficient space available for patching in GRID_HOME
% cat /tmp/patch_list_gihome.txt
/dumparea/APR2020BP/30805478/26983807
/dumparea/APR2020BP/30805478/30703511
/dumparea/APR2020BP/30805478/30703508
/dumparea/APR2020BP/30805478/30691015
-- space checker:
--------------------------
oracle@dbhost-n1:.../oracle/product/grid12c/OPatch$ ./opatch prereq CheckSystemSpace -phBaseFile /tmp/patch_list_gihome.txt
Oracle Interim Patch Installer version 12.2.0.1.21
Copyright (c) 2020, Oracle Corporation. All rights reserved.
PREREQ session
Oracle Home : /gridapp/app/oracle/product/grid12c
Central Inventory : /gridapp/app/oraInventory
from : /gridapp/app/oracle/product/grid12c/oraInst.loc
OPatch version : 12.2.0.1.21
OUI version : 12.1.0.2.0
Log file location : /gridapp/app/oracle/product/grid12c/cfgtoollogs/opatch/opatch2020-06-16_15-44-55PM_1.log
Invoking prereq "checksystemspace"
Prereq "checkSystemSpace" passed.
OPatch succeeded.
5. Check whether sufficient space available for patching in DB_HOME:
$cat /tmp/patch_list_dbhome.txt
/dumparea/APR2020BP/30805478/30703508
/dumparea/APR2020BP/30805478/30691015
$ $ORACLE_HOME/OPatch/opatch prereq CheckSystemSpace -phBaseFile /tmp/patch_list_dbhome.txt
Oracle Interim Patch Installer version 12.2.0.1.21
Copyright (c) 2020, Oracle Corporation. All rights reserved.
PREREQ session
Oracle Home : /oracle/app/oracle/product/12.1.0.2/dbhome_1
Central Inventory : /gridapp/app/oraInventory
from : /gridapp/app/oracle/product/grid12c/oraInst.loc
OPatch version : 12.2.0.1.21
OUI version : 12.1.0.2.0
Log file location : /oracle/app/oracle/product/12.1.0.2/dbhome_1/cfgtoollogs/opatch/opatch2020-06-16_15-44-55PM_1.log
Invoking prereq "checksystemspace"
Prereq "checkSystemSpace" passed.
OPatch succeeded.
6. shutdown the services running on db_home:
Shutdown database instance and listener services running on this node1
export ORACLE_HOME=/oracle/app/oracle/product/12.1.0.2/dbhome_1
export PATH=$ORACLE_HOME/bin:$PATH
srvctl stop home -o /oracle/app/oracle/product/12.1.0.2/dbhome_1 -s /dumparea/db_prepatch_stat.txt -n dbhost-n1
7. Prepare the CRS for patching:
This script need to be executed from root user.
root#/gridapp/app/oracle/product/grid12c/crs/install/rootcrs.sh -prepatch
root@dbhost-n1:~# /gridapp/app/oracle/product/grid12c/crs/install/rootcrs.sh -prepatch
Using configuration parameter file: /gridapp/app/oracle/product/grid12c/crs/install/crsconfig_params
Oracle Clusterware active version on the cluster is [12.1.0.2.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [4267482287].
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'dbhost-n1'
CRS-2673: Attempting to stop 'ora.crsd' on 'dbhost-n1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'dbhost-n1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'dbhost-n1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'dbhost-n1'
CRS-2673: Attempting to stop 'ora.LISTENER_TEST.lsnr' on 'dbhost-n1'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'dbhost-n1'
CRS-2673: Attempting to stop 'ora.REDOA.dg' on 'dbhost-n1'
CRS-2673: Attempting to stop 'ora.OCRVD.dg' on 'dbhost-n1'
CRS-2673: Attempting to stop 'ora.MGMT.dg' on 'dbhost-n1'
CRS-2673: Attempting to stop 'ora.cvu' on 'dbhost-n1'
CRS-2673: Attempting to stop 'ora.REDOB.dg' on 'dbhost-n1'
CRS-2673: Attempting to stop 'ora.oc4j' on 'dbhost-n1'
CRS-2677: Stop of 'ora.cvu' on 'dbhost-n1' succeeded
CRS-2672: Attempting to start 'ora.cvu' on 'dbhost-n2'
CRS-2677: Stop of 'ora.OCRVD.dg' on 'dbhost-n1' succeeded
CRS-2677: Stop of 'ora.MGMT.dg' on 'dbhost-n1' succeeded
CRS-2676: Start of 'ora.cvu' on 'dbhost-n2' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'dbhost-n1' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'dbhost-n1'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'dbhost-n1' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'dbhost-n1'
CRS-2677: Stop of 'ora.LISTENER_TEST.lsnr' on 'dbhost-n1' succeeded
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'dbhost-n1'
CRS-2673: Attempting to stop 'ora.LISTENER_GSM.lsnr' on 'dbhost-n1'
CRS-2677: Stop of 'ora.DATA.dg' on 'dbhost-n1' succeeded
CRS-2677: Stop of 'ora.LISTENER_GSM.lsnr' on 'dbhost-n1' succeeded
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'dbhost-n1' succeeded
CRS-2673: Attempting to stop 'ora.dbhost-n1.vip' on 'dbhost-n1'
CRS-2677: Stop of 'ora.scan2.vip' on 'dbhost-n1' succeeded
CRS-2672: Attempting to start 'ora.scan2.vip' on 'dbhost-n2'
CRS-2677: Stop of 'ora.REDOB.dg' on 'dbhost-n1' succeeded
CRS-2677: Stop of 'ora.REDOA.dg' on 'dbhost-n1' succeeded
CRS-2673: Attempting to stop 'ora.ARCH.dg' on 'dbhost-n1'
CRS-2677: Stop of 'ora.scan3.vip' on 'dbhost-n1' succeeded
CRS-2672: Attempting to start 'ora.scan3.vip' on 'dbhost-n2'
CRS-2677: Stop of 'ora.ARCH.dg' on 'dbhost-n1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'dbhost-n1'
CRS-2677: Stop of 'ora.asm' on 'dbhost-n1' succeeded
CRS-2677: Stop of 'ora.dbhost-n1.vip' on 'dbhost-n1' succeeded
CRS-2672: Attempting to start 'ora.dbhost-n1.vip' on 'dbhost-n2'
CRS-2677: Stop of 'ora.oc4j' on 'dbhost-n1' succeeded
CRS-2672: Attempting to start 'ora.oc4j' on 'dbhost-n2'
CRS-2676: Start of 'ora.dbhost-n1.vip' on 'dbhost-n2' succeeded
CRS-2676: Start of 'ora.scan2.vip' on 'dbhost-n2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'dbhost-n2'
CRS-2676: Start of 'ora.scan3.vip' on 'dbhost-n2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'dbhost-n2'
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'dbhost-n2' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'dbhost-n2' succeeded
CRS-2676: Start of 'ora.oc4j' on 'dbhost-n2' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'dbhost-n1'
CRS-2677: Stop of 'ora.ons' on 'dbhost-n1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'dbhost-n1'
CRS-2677: Stop of 'ora.net1.network' on 'dbhost-n1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'dbhost-n1' has completed
CRS-2677: Stop of 'ora.crsd' on 'dbhost-n1' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'dbhost-n1'
CRS-2673: Attempting to stop 'ora.evmd' on 'dbhost-n1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'dbhost-n1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'dbhost-n1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'dbhost-n1'
CRS-2677: Stop of 'ora.storage' on 'dbhost-n1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'dbhost-n1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'dbhost-n1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'dbhost-n1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'dbhost-n1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'dbhost-n1' succeeded
CRS-2677: Stop of 'ora.asm' on 'dbhost-n1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'dbhost-n1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'dbhost-n1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'dbhost-n1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'dbhost-n1'
CRS-2677: Stop of 'ora.crf' on 'dbhost-n1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'dbhost-n1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'dbhost-n1'
CRS-2677: Stop of 'ora.cssd' on 'dbhost-n1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'dbhost-n1'
CRS-2677: Stop of 'ora.gipcd' on 'dbhost-n1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'dbhost-n1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2020/06/17 14:10:54 CLSRSC-4012: Shutting down Oracle Trace File Analyzer (TFA) Collector.
2020/06/17 14:11:44 CLSRSC-4013: Successfully shut down Oracle Trace File Analyzer (TFA) Collector.
2020/06/17 14:11:46 CLSRSC-347: Successfully unlock /gridapp/app/oracle/product/grid12c
8. Apply the patches to grid home:
oracle@dbhost-n1:.../oracle/product/grid12c/OPatch$ ./opatch apply -oh /gridapp/app/oracle/product/grid12c -local /dumparea/APR2020BP/30805478/26983807
Oracle Interim Patch Installer version 12.2.0.1.21
Copyright (c) 2020, Oracle Corporation. All rights reserved.
Oracle Home : /gridapp/app/oracle/product/grid12c
Central Inventory : /gridapp/app/oraInventory
from : /gridapp/app/oracle/product/grid12c/oraInst.loc
OPatch version : 12.2.0.1.21
OUI version : 12.1.0.2.0
Log file location : /gridapp/app/oracle/product/grid12c/cfgtoollogs/opatch/opatch2020-06-17_14-12-53PM_1.log
Verifying environment and performing prerequisite checks...
The following patch(es) are duplicate patches with patches installed in the Oracle Home.
[ 26983807]
You have already installed same patch(es) with same UPI(s) or same version(s).
These patch(es) will be skipped.
OPatch Session completed with warnings.
Log file location: /gridapp/app/oracle/product/grid12c/cfgtoollogs/opatch/opatch2020-06-17_14-12-53PM_1.log
OPatch completed with warnings.
oracle@dbhost-n1:.../oracle/product/grid12c/OPatch$ ./opatch apply -oh /gridapp/app/oracle/product/grid12c -local /dumparea/APR2020BP/30805478/30703511
Oracle Interim Patch Installer version 12.2.0.1.21
Copyright (c) 2020, Oracle Corporation. All rights reserved.
Oracle Home : /gridapp/app/oracle/product/grid12c
Central Inventory : /gridapp/app/oraInventory
from : /gridapp/app/oracle/product/grid12c/oraInst.loc
OPatch version : 12.2.0.1.21
OUI version : 12.1.0.2.0
Log file location : /gridapp/app/oracle/product/grid12c/cfgtoollogs/opatch/opatch2020-06-17_14-13-14PM_1.log
Verifying environment and performing prerequisite checks...
OPatch continues with these patches: 30703511
Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/gridapp/app/oracle/product/grid12c')
Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying interim patch '30703511' to OH '/gridapp/app/oracle/product/grid12c'
Patching component oracle.usm, 12.1.0.2.0...
Patch 30703511 successfully applied.
Sub-set patch [29423125] has become inactive due to the application of a super-set patch [30703511].
Please refer to Doc ID 2161861.1 for any possible further required actions.
Log file location: /gridapp/app/oracle/product/grid12c/cfgtoollogs/opatch/opatch2020-06-17_14-13-14PM_1.log
OPatch succeeded.
oracle@dbhost-n1:.../oracle/product/grid12c/OPatch$ ./opatch apply -oh /gridapp/app/oracle/product/grid12c -local /dumparea/APR2020BP/30805478/30703508
Oracle Interim Patch Installer version 12.2.0.1.21
Copyright (c) 2020, Oracle Corporation. All rights reserved.
Oracle Home : /gridapp/app/oracle/product/grid12c
Central Inventory : /gridapp/app/oraInventory
from : /gridapp/app/oracle/product/grid12c/oraInst.loc
OPatch version : 12.2.0.1.21
OUI version : 12.1.0.2.0
Log file location : /gridapp/app/oracle/product/grid12c/cfgtoollogs/opatch/opatch2020-06-17_14-13-44PM_1.log
Verifying environment and performing prerequisite checks...
OPatch continues with these patches: 30703508
Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/gridapp/app/oracle/product/grid12c')
Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying interim patch '30703508' to OH '/gridapp/app/oracle/product/grid12c'
Patching component oracle.rdbms, 12.1.0.2.0...
Patching component oracle.has.crs, 12.1.0.2.0...
Patching component oracle.has.db, 12.1.0.2.0...
Patching component oracle.has.common, 12.1.0.2.0...
Patching component oracle.has.deconfig, 12.1.0.2.0...
Patching component oracle.has.common.cvu, 12.1.0.2.0...
Patching component oracle.has.cvu, 12.1.0.2.0...
Patching component oracle.has.rsf, 12.1.0.2.0...
Patch 30703508 successfully applied.
Sub-set patch [29509318] has become inactive due to the application of a super-set patch [30703508].
Please refer to Doc ID 2161861.1 for any possible further required actions.
Log file location: /gridapp/app/oracle/product/grid12c/cfgtoollogs/opatch/opatch2020-06-17_14-13-44PM_1.log
OPatch succeeded.
oracle@dbhost-n1:.../oracle/product/grid12c/OPatch$ ./opatch apply -oh /gridapp/app/oracle/product/grid12c -local /dumparea/APR2020BP/30805478/30691015
Oracle Interim Patch Installer version 12.2.0.1.21
Copyright (c) 2020, Oracle Corporation. All rights reserved.
Oracle Home : /gridapp/app/oracle/product/grid12c
Central Inventory : /gridapp/app/oraInventory
from : /gridapp/app/oracle/product/grid12c/oraInst.loc
OPatch version : 12.2.0.1.21
OUI version : 12.1.0.2.0
Log file location : /gridapp/app/oracle/product/grid12c/cfgtoollogs/opatch/opatch2020-06-17_14-15-56PM_1.log
Verifying environment and performing prerequisite checks...
OPatch continues with these patches: 29972716 30364137 30691015
Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/gridapp/app/oracle/product/grid12c')
Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying sub-patch '29972716' to OH '/gridapp/app/oracle/product/grid12c'
ApplySession: Optional component(s) [ oracle.ctx, 12.1.0.2.0 ] , [ oracle.rdbms.lbac, 12.1.0.2.0 ] not present in the Oracle Home or a higher version is found.
Patching component oracle.ldap.rsf, 12.1.0.2.0...
Patching component oracle.network.rsf, 12.1.0.2.0...
Patching component oracle.rdbms.rsf, 12.1.0.2.0...
Patching component oracle.rdbms, 12.1.0.2.0...
Patching component oracle.rdbms.deconfig, 12.1.0.2.0...
Patching component oracle.has.crs, 12.1.0.2.0...
Patching component oracle.rdbms.rman, 12.1