Beginners Guide to Upgrading and Patching Oracle RAC

Patch and Patch Set: Overview

Oracle issues product fixes for its software called patches. They are associated with particular releases and versions of Oracle products. The patching cycle involves downloading patches, applying patches, and verifying the applied patch to ensure that the bug fixes present in the patch reflect appropriately.

Patching involves migrating from one version of the software product to another, within a particular release, unlike upgrading which involves moving from one release of a product to another newer release of the software product. When you apply the patch to your Oracle software installation, it updates the executable files, libraries, and object files in the software home directory. The patch application can also update configuration files and Oracle-supplied SQL schemas.

Summary

• Oracle issues product fixes for its software called patches.
• Patches are associated with particular releases and versions of Oracle products.
• The patching cycle involves downloading patches, applying patches, and verifying the applied patch.
• Patching involves migrating from one version of the software product to another, within a particular release.
• When a patch is applied to an Oracle software installation, it updates the executable files, libraries, and object files in the software home directory. The patch application can also update configuration files and Oracle-supplied SQL schemas.

Types of Patches

Patch Type	Description
Interim Patches	Released to fix a bug, or a collection of bugs. Previously called patch set exceptions (PSE), one-off patches, or hot fixes.
Interim Patches (for Security bug fixes)	Released to provide customer-specific security fixes. Previously referred to as a test patch, fix verification binary, or e-fix.
Diagnostic Patches	Mainly help diagnose and verify a fix, or a collection of bugfixes
Bundle Patch Updates	Cumulative collection of fixes for a specific product or component. Previously referred to as a maintenance pack, service pack, cumulative patch, update release, or MLR.
Patch Set Updates (PSU)	Cumulative patch bundles that contain well-tested and proven bug fixes for critical issues. PSUs have limited new content, and do not include any changes that require re- certification.
Security Patch Updates	A cumulative collection of security bug fixes. Previously known as Critical Patch Updates, or CPUs.

Interim patches are bug fixes available to customers in response to specific bugs. They require a particular base release or patch set to be installed before you can apply them. These patches are not versioned and are generally available in a future patch set as well as the next product release. Interim patches are applied by using Enterprise Manager Cloud Control or OPatch, which is included with your Oracle Database installation.

Patch sets updates (PSUs) and patch bundles are mechanisms for delivering fully tested and integrated product fixes. All the fixes in a patch set have been tested and are certified to work with each other. Because a patch set includes only low impact patches, it does not require you to certify applications or tools against the updated Oracle Database software. When you apply a patch set, many different files and utilities are modified. This results in a release number change for your Oracle software. You use OPatch to apply PSUs and Oracle Universal Installer (OUI) to install patch sets. PSUs are rolling installable but patch sets are not.

Obtaining Oracle RAC Patch Sets

The latest patch sets and recommended BPs can be downloaded from the My Oracle Support website at the following URL: http://support.oracle.com/

After signing in to the website, click the “Patches & Updates” tab. For the latest patch sets and patch bundles, click the “Latest Patchsets” link under Oracle Servers and Tools. You can choose from patch sets for product bundles or patch bundles for individual products.

If you know the patch set number, you can click the Number/Name Sun CR ID link. You can enter a single patch set number or a comma-separated list. Select your platform and click the Search button.

To locate the Patch Set notes on My Oracle Support:

1. Log in to My Oracle Support.
2. Select the Patches & Updates tab.
3. Select Quick Links to the Latest Patchsets, Mini Packs, and Maintenance Packs.
4. Under the heading Latest Oracle Server/Tools Patchsets, select Oracle Database.A list of operating systems appears.
5. Place your cursor over the entry that matches your operating system, or use the triangular arrows to search for your operating system. When you place the cursor over the entry for your operating system, for example, Linux x86, a list of database versions appears.
6. Select 12.2.0. The Advanced Search page appears.
7. Scroll to the bottom of this page to see the list of available patch sets.
8. Select the number in the Patch column for the patch set you want to view or download. The Patchset description and download page appears.
9. Click View Readme to see the patch set notes. On this page you can also click Download to download the patch to your computer.
10. If you choose to download the patch, then follow the instructions in the ReadMe file of the patch set to apply the patch set to your software.

To locate patches on the My Oracle Support website:

1. Log in to your account on My Oracle Support.

2. Select the Patches & Updates tab.

3. If you know the patch number, then you can enter it into the Patch Name or Number field, then click Search. If you want to search for all available patches for your system, then select Product or Family (Advanced Search), which is located above the Patch Name or Number field. Supply the following information:

   • Choose the products you want to patch (for example, Oracle Clusterware, Oracle Database, or an individual product such as Universal Installer)
   • Specify the software release for the products you selected, for example, Oracle 12.2.0.1.0.
   • Specify the platform on which the software is installed. Click Search to look for available patches. The Patch Search Results page is displayed.

4. On the Patch Search Results page, select the number of the patch you want to download. A details page for that patch appears on your screen.

5. Click the ReadMe button to view the ReadMe file for the patch, or click Download to download the patch to your local computer.

Downloading Patches

Once you have located the patch you need, you can download it. Click the patch link on the search results page. Locate and click the Download link on the patch summary page, and then click the patch link in the File Download dialog box. Click the Save File button, and then click OK. Specify the directory location for the file, and then click Save.

RAC Patching methods

Patch sets are now installed as out-of-place upgrades to the Grid Infrastructure software (Oracle Clusterware and Automatic Storage Management) and Oracle Database. This reduces the downtime required for planned outages for patching.

OPatch supports 3 different patch methods on a RAC environment:

• Patching RAC as a single instance (All-Node Patch): In this mode, OPatch applies the patch to the local node first, then propagates the patch to all the other nodes, and finally updates the inventory. All instances must be down during the whole patching process.
• Patching RAC using a minimum downtime strategy (Min. Downtime Patch): In this mode, OPatch patches the local node, asks users for a sub-set of nodes, which will be the first subset of nodes to be patched. After the initial subset of nodes are patched, Opatch propagates the patch to the other nodes and finally updates the inventory. The downtime would happen between the shutdown of the second subset of nodes and the startup of the initial subset of nodes patched.
• Patching RAC using a rolling strategy - No down time (Rolling Patch): With this method, there is no downtime. Each node would be patched and brought up while all the other nodes are up and running, resulting in no disruption of the system. Rolling patching strategy incur no downtime, however, some rolling patches may incur downtime due to post-installation steps, i.e. running sql scripts to patch the actual database. Please refer to patch readme to find out whether postinstallation steps requires downtime or not.

Out-of-Place Database Upgrades with OUI

You must install the software for the new Oracle Database release before you can perform the upgrade of Oracle Database. The installation procedure installs the Oracle software into a new Oracle home. This is referred to as an out-of-place upgrade and is different from patch set releases for earlier releases of Oracle Database, where the patch set was always installed in place. The new Oracle Database software is installed by using the OUI. The upgrade process is completed by the Oracle Database Upgrade Assistant.

Out-of-Place Upgrades with OUI

If you are upgrading an Oracle RAC database, then you must perform the following steps:

1. Upgrade Grid Infrastructure on your cluster, if necessary.

2. Follow the instructions in your Oracle operating system–specific documentation to prepare for installation of Oracle Database software.

3. Start the OUI. Select “Upgrade an existing database” on the Select Installation Option page.

4. It is recommended that you run the Pre-Upgrade Information Tool before you upgrade using DBUA, so that you can preview the types of items DBUA checks. The OUI will launch the DBUA when the root scripts have been executed, but you can elect to run DBUA at a later time. Oracle Database 12c release 2 introduces the preupgrade.jar Pre-Upgrade Information Tool. You can run the tool from the operating system command line. In previous Oracle Database releases, the Pre-Upgrade Information Tool was run within SQL*Plus as a SQL file.

$ cd /u01/app/oracle/product/12.2.0/dbhome_1/rdbms/admin
$ $[Earlier_Release_ORACLE_HOME]/jdk/bin/java -jar preupgrade.jar [FILE|TERMINAL] [TEXT|XML] [DIR output_dir]

4. If you do not specify an output directory with DIR, but you defined ORACLE_BASE, the generated scripts and log files are created in ORACLE_BASE/cfgtoollogs/. If you do not specify an output directory and ORACLE_BASE is not defined, the generated scripts and log files are created in ORACLE_HOME/cfgtoollogs/.

Example: Non-CDB In the Source Oracle Home Example

a. Set your user environment variables to point to the earlier release Oracle home.

$ export ORACLE_HOME=/u01/app/oracle/product/12.1.0/dbhome_1
$ export ORACLE_BASE=/u01/app/oracle
$ export ORACLE_SID=sales01
$ export PATH=.:$ORACLE_HOME/bin:$PATH

b. Run the new release Oracle Database Pre-Upgrade Information Tool on the earlier release Oracle Database server (12.2), using the environment settings you have set to the earlier release Oracle home.

 $ORACLE_HOME/jdk/bin/java -jar /u01/app/oracle/product/12.2.0/rdbms/admin/preupgrade.jar TERMINAL TEXT

Example: CDB in a Source Oracle Home

a. Open all the pluggable databases

b. Set your user environment variables to point to the earlier release Oracle home.

$ export ORACLE_HOME=/u01/app/oracle/product/12.1.0/dbhome_1
$ export ORACLE_BASE=/u01/app/oracle
$ export ORACLE_SID=sales01
$ export PATH=.:$ORACLE_HOME/bin:$PATH

c. Run the Pre-Upgrade Information Tool with an inclusion list, using the -c option. In this example, the inclusion list is PDB1 and PDB2, and the command is run on a Linux or UNIX system. The output of the command is displayed to the terminal, and the output is displayed as text.

$ORACLE_HOME/jdk/bin/java -jar /u01/app/oracle/product/12.2.0/rdbms/admin/preupgrade.jar TERMINAL TEXT -c 'pdb1 pdb2'

5. Run the root.sh script as directed by the OUI.

6. If you are ready, use DBUA to complete the upgrade process.

Rolling Patches

A rolling patch allows one node to be patched to the latest version, while other nodes continue to use the older version and provide business service. This methodology is best suited when you are applying one-off patches that support rolling methodology, maintaining high availability of your targets, so when one node is being patched, the other nodes are available for service.

Rolling patches are enabled by using locally accessible, nonshared file system to store the software files. Rolling patches cannot be done when the Oracle software files are stored in a shared cluster file system in which a single copy of the software is shared among all nodes. A single copy requires much less disk space and a single copy to patch or upgrade. However, to patch or upgrade, the software must be stopped. Stopping the Oracle Clusterware software also requires all databases, applications, and services that depend on Oracle Clusterware to be stopped. This technique requires a complete outage with down time to patch or upgrade.

Out-of-place rolling patching involves installing the patch in a new home, modifying the Oracle home of the database, and then restarting instances in a rolling fashion.

OPatch: Overview

OPatch is an Oracle-supplied utility that assists you with the process of applying interim patches to Oracle’s software. Opatch is a Java-based utility that can run on either OUI-based Oracle homes or standalone homes. It works on all operating systems for which Oracle releases software. OPatch is included with the Oracle Clusterware 12c installation.

For large-scale IT environments, patching individual GI/RAC/DB homes may not be practical because patching large numbers of targets manually is both monotonous and error prone. To maintain and deploy Oracle patches across many targets across your organization, you can use Enterprise Manager Cloud Control’s patch automation capability.

Summary

• OPatch is a utility that assists you with the process of applying interim patches to Oracle software.
• OPatch is a Java-based utility that allows the application and rolling back of interim patches.
• OPatch is included with the Oracle Clusterware 12c installation.
• For large IT environments, EM Cloud Control’s patch automation capability can simplify the patching process.

OPatch: General Usage

The opatch utility requires that the ORACLE_HOME environment variable be defined or that the value of ORACLE_HOME be passed as an argument on the command line with the –oh option. For the case of Oracle RAC, ORACLE_HOME refers to the installation directory for Oracle RAC, not the location of other Oracle products that may be installed. In general, ORACLE_HOME refers to the home for the product to be patched.

$ export ORACLE_HOME=/u01/app/oracle/product/12.2.0/dbhome_1
$ opatch command [options]

or

$ opatch command –oh /u01/app/oracle/product/12.2.0/dbhome_1 [options]

The OPatch documentation can be found in the $ORACLE_HOME/OPatch/docs directory. The utility contains help for its syntax by using the –help option as follows:

opatch -help
opatch apply -help
opatch lsinventory -help
opatch rollback -help
opatch prereq -help
opatch util –help

In general, RAC patches can be applied in a rolling fashion—that is, one node at a time. However, it is still important to check each patch for exceptions to this rule. To verify that a patch supports rolling applications, unzip the downloaded patch into a directory of your choosing and, from that directory, issue the following command:

$ORACLE_HOME/OPatch/opatch query -is_rolling_patch

Before Patching with OPatch

The Oracle Patching utility, OPatch, verifies that the ORACLE_HOME environment variable names an actual directory. You should verify that the ORACLE_HOME variable is set to the Oracle home of the product you are trying to patch.

It is best practice to back up the software directory you are patching before performing any patch operation. This applies to Oracle RAC, ASM, or Oracle Clusterware software installation directories. The backup should include the Oracle Inventory directory as well.

If you manually download the patch and use OPatch to install the patch, you must stage the patch on each node. If you use Enterprise Manager to download the patch and you selected all the nodes in your cluster as targets for the patch, then the patch is automatically staged on those nodes.

The opatch binary file is located in the $ORACLE_HOME/OPatch directory. You can either specify this path when executing OPatch or update the PATH environment variable to include the OPatch directory. To change the PATH variable on Linux, use:

$ export PATH=$PATH:$ORACLE_HOME/OPatch

Installing a Rolling Patch with OPatch

1. Verify that Oracle Inventory is properly configured.

[oracle]$ opatch lsinventory

2. Determine if any installed interim patches conflict with a patch to be installed.

[oracle]$ opatch prereq CheckConflictAgainstOHWithDetail –ph ./

3. Shutdown the instance running on the local node.

[oracle]$ srvctl stop instance –db orcl –instance orcl_3

4. Start the patch installation in rolling fashion

[oracle]$ opatch apply

5. Verify patch installation

[oracle]$ opatch lsinventory

Example

Let’s see an example of installing a rolling patch with OPatch.

1. Assume that you have set ORACLE_HOME and PATH for OPatch. Verify that Oracle Inventory is properly configured.

[oracle@host01 ~]$ opatch lsinventory

2. Determine whether any currently installed interim patches conflict with patch to be installed.

[oracle@host01 ]$ opatch prereq
CheckConflicAgainstOHWithDetail –ph ./

3. Shutdown the instance running on the local node (host01)

[oracle@host01 ~]$ srvctl status database -db orcl
Instance orcl_1 is running on node host02
Instance orcl_2 is running on node host03
Instance orcl_3 is running on node host01
[oracle@host01 ~]$ srvctl stop instance -db orcl -instance orcl_3

4. Start the patch installation. When OPatch asks “Is the local system ready for patching?”, answer yes by typing Y and pressing Enter.

[oracle@host01 ]$ opatch apply

When patching is finished on host01, the OPatch dialog will inform you that the instance can be restarted on host01 and will prompt you for the name of the next node to patch.

[oracle@host01 ]$ opatch apply
...
The local system has been patched. You can restart Oracle instances on it.

Patching in rolling mode.

Remaining nodes to be patched:
'host02' 'host03'

[oracle@host01 ~]$ srvctl start instance -db orcl -instance orcl_3
[oracle@host01 ~]$ srvctl stop instance -db orcl -instance orcl_1

[oracle@host01 ]$ opatch apply
What is the next node to be patched?
host02
...
Is the node ready for patching? [y|n]
y
...
The node 'host02' has been patched. You can restart Oracle instances on it.
...
Please shutdown Oracle instances running out of this ORACLE_HOME on 'host03'.
(Oracle Home = '/u01/app/oracle/product/12.2.0/dbhome_1')

[oracle@host01 ~]$ srvctl start instance -db orcl -instance orcl_1
[oracle@host01 ~]$ srvctl stop instance -db orcl -instance orcl_2

[oracle@host01 ]$ opatch apply
Is the node ready for patching? [y|n]
y
...
The node 'host03' has been patched. You can restart Oracle instances on it.
...
OPatch succeeded.

[oracle@host01 ~]$ srvctl start instance -db orcl -instance orcl_2

Ensure the Oracle homes on all three nodes were patched successfully.

[oracle@host01 ~] opatch lsinventory

OPatch Automation

The OPatch utility has automated the patch application for the Oracle Grid Infrastructure home and the Oracle RAC database homes. It operates by querying existing configurations and automating the steps required for patching each Oracle RAC database home of the same version and the GI home.

The utility must be executed by an operating system user with root privileges (usually the user root), and it must be executed on each node in the cluster if the Grid Infrastructure home or Oracle RAC database home is in non-shared storage. The utility should not be run in parallel on the cluster nodes.

Depending on the command-line options specified, one invocation of OPatch can patch the GI home, one or more Oracle RAC database homes, or both GI and Oracle RAC database homes of the same Oracle release version. You can also roll back the patch with the same selectivity.

OPatch Automation: Examples

If you have not installed Oracle Configuration Manager (OCM), the OPatch utility will prompt for your OCM response file when it is run. You should enter a complete path of OCM response file if you already have created this in your environment. If you do not have the OCM response file (ocm.rsp), you should run the emocmrsp command to create it. As the software home owner, execute:

$ [ORACLE_HOME]/OPatch/ocm/bin/emocmrsp

Before executing OPatch, add the directory containing OPatch to the your path:

# export PATH=$PATH:[GI_HOME]/Opatch

To patch GI home and all Oracle RAC database homes of the same version:

# [Grid_home]/OPatch/opatchauto apply [Grid_home] -ocmrf [ocm_response_file]

To patch only the GI home:

# [Grid_home]/OPatch/opatchauto apply  -oh [Grid_home] -ocmrf [ocm_response_file]

To patch one or more Oracle RAC databases and associated GI/RAC homes:

# opatchauto apply [UNZIPPED_PATCH_LOCATION] -database db1, db2 -ocmrf [ocm_response_file]

To roll back the patch from the GI home and each Oracle RAC database home:

# opatchauto rollback [UNZIPPED_PATCH_LOCATION] -ocmrf [ocm_response_file]

To roll back the patch from the GI home:

# opatchauto rollback [UNZIPPED_PATCH_LOCATION] -oh [Grid_home] -ocmrf [ocm_response_file]

To roll back the patch from one or more Oracle RAC database homes:

# opatchauto rollback [UNZIPPED_PATCH_LOCATION] -database db1, db2 -ocmrf [ocm_response_file]

OPatch Log and Trace Files

Logging and tracing is a common aid in debugging. OPatch maintains logs for apply, rollback, and lsinventory operations. Log files are located in Oracle_home/cfgtoollogs/opatch. Each log file is tagged with the time stamp of the operation. Log files are named as opatch_mm-dd-yyyy_hh-mmss.log, where mm-dd-yyyy is the current date and hh-mm-ss is the current time. Each time you run OPatch, a new log file is created.

For example, if a log file is created on May 17, 2016 at 11:55 PM, then it is named as follows:

opatch_05-17-2016_23-55-00.log

OPatch also maintains an index of the commands processed by OPatch and the log files associated with it in the opatch_history.txt file located in the Oracle_home/cfgtoollogs/opatch directory. A sample of the opatch_history.txt file is as follows:

Date & Time : Thu Jun 09 22:07:45 MDT 2016 Oracle Home :
/u01/app/oracle/product/12.2.0/dbhome_1 OPatch Ver. : 12.2.0.1.0 Current Dir :
/u01/app/oracle/product/12.2.0/dbhome_1 Command : lsinventory -xml
/u01/app/oracle/product/12.2.0/dbhome_1 ... Log File :
/u01/app/oracle/product/12.2.0/dbhome_1/cfgtoollogs/opatch/opatch2016-06-09_22-07-57PM_1.log

Queryable Patch Inventory

Using DBMS_QOPATCH, Oracle Database 12c provides a PL/SQL or SQL interface to view the database patches that are installed. The interface provides all the patch information available as part of the OPatch lsinventory -xml command. The package accesses the OUI patch inventory in real time to provide patch and patch meta information. Using this feature, users can:

• Query what patches are installed from SQL*Plus
• Write wrapper programs to create reports and do validation checks across multiple environments
• Check patches installed on cluster nodes from a single location instead of having to log onto each one in turn

Alternative Methods of Patching

OPatch is a commonly used method for patching Oracle software homes, but this is not the only method of patching Oracle software. Here are two possible alternative methods of patching. You might find these methods more appropriate for your organization than using OPatch.

• Oracle Enterprise Manager Cloud Control for Patching Operations: Using Cloud Control with its Provisioning & Patching functionality, you can automate the patching of your Oracle Grid Infrastructure and Oracle RAC software.
• Rapid Home Provisioning, Patching, and Upgrading: Rapid Home Provisioning is a method of deploying software homes to any number of nodes in a data center from a single cluster, and also facilitates patching and upgrading software.