Oracle Exadata Database Recommended Patch (BP3) for Bug 10387939

2.1.1 OPatch Utility Information

You must use the OPatch utility version 11.2.0.1.4 or later to apply this patch. Oracle recommends that you use the latest released OPatch for 11.2 releases, which is available for download from My Oracle Support patch 6880880 by selecting ARU link for the 11.2.0.0.0 release.

1. Download the OPatch utility to a temporary directory.
2. For each Oracle RAC database homes and the GI home that are being patched, run the following commands as the home owner to extract the OPatch utility.

   % unzip <OPATCH-ZIP> -d <ORACLE_HOME>
   % <ORACLE_HOME>/OPatch/opatch version
   

The version output of the previous command should be 11.2.0.1.4 or later.

For information about OPatch documentation, including any known issues, see My Oracle Support Note 293369.1 OPatch documentation list.

2.1.2 OCM Configuration

The OPatch utility will prompt for your OCM (Oracle Configuration Manager) response file (ocm.rsp) when it is run. You should enter a complete path of OCM response file if you already have created this in your environment.

If you do not have the OCM response file and you wish to use one during the patch application then you should run the following command to create it.

As the Grid home owner execute:

% <ORACLE_HOME>/OPatch/ocm/bin/emocmrsp
This will create the ocm.rsp file in the current directory from which emocmrp is executed

2.1.3 Validation of Oracle Inventory

Before beginning patch application, check the consistency of inventory information for GI home and each database home to be patched. Run the following command as respective Oracle home owner to check the consistency.

% <ORACLE_HOME>/OPatch/opatch lsinventory -detail -oh <ORACLE_HOME>

If this command succeeds, it lists the Oracle components that are installed in the home. Save the output so you have the status prior to the patch apply.

If this command fails, contact Oracle Support Services for assistance. We can’t predict why the command failed – may not be because of an improper inventoory.

2.1.4 Unzipping the Exadata Database Recommended Patch (BP3)

The patch application requires explicit user actions to run ‘opatch auto’ command on each node of Oracle clusterware.

The unzipped patch location should have read permission for ORA_INSTALL group in order to patch Oracle homes owned by different owners. The ORA_INSTALL group is the primary group of the user who owns the GI home or the group owner of the Oracle central inventory.

% cd <UNZIPPED_PATCH_LOCATION>
% unzip p10387939_11202_Linux-x86-64.zip

For example, if <UNZIPPED_PATCH_LOCATION> in your environment is /u01/oracle/patches, enter the following commands:

% cd /u01/oracle/patches
% unzip p10387939_11202_Linux-x86-64.zip
% cd /u01/oracle/patches/10387939
(In this readme, <UNZIPPED_PATCH_LOCATION>/10387939 location is referred as <PATH_TO_PATCH_DIRECTORY>.)

2.2 OPatch Automation

The OPatch utility has automated the patch application for the Oracle Grid Infrastructure (GI) home and the Oracle RAC database homes. It operates by querying existing configurations and automating the steps required for patching each Oracle RAC database home of same version and the GI home.

The utility must be executed by an operating system (OS) user with root privileges (usually the user root), and it must be executed on each node in the cluster if the GI home or Oracle RAC database home is in Non-shared storage.

Depending on command line options specified, one invocation of OPatch can patch the GI home, one or more Oracle RAC database homes, or both GI and Oracle RAC database homes of the same Oracle release version. You can also roll back the patch with the same selectivity.

Add the directory containing the OPatch to the $PATH environment variable. For example:

export PATH=$PATH:<GI_HOME>/OPatch

To patch GI home and all Oracle RAC database homes of the same version:

# opatch auto <PATH_TO_PATCH_DIRECTORY>

To patch only the GI home:

# opatch auto <PATH_TO_PATCH_DIRECTORY> -oh <GI_HOME>

To patch one or more Oracle RAC database homes:

# opatch auto <PATH_TO_PATCH_DIRECTORY> -oh <path to RAC database1 home>[,<path of RAC database2 home>]

To roll back the patch from the GI home and each Oracle RAC database home:

# opatch auto <PATH_TO_PATCH_DIRECTORY> -rollback

To roll back the patch from the GI home:

# opatch auto <PATH_TO_PATCH_DIRECTORY> -oh <path to GI home> -rollback

To roll back the patch from the Oracle RAC database home:

# opatch auto <PATH_TO_PATCH_DIRECTORY> -oh <path to RAC database home> -rollback

For more information about opatch auto, see My Oracle Support Note 293369.1 OPatch documentation list.

For detailed patch installation instructions, see Section 2.3, “Patch Installation”.

2.3 Patch Installation

This section will guide you through the steps required to apply the BP3 patch to RAC database homes, the Grid home, or all relevant homes on the cluster. For non-Exadata customers who have ACFS filesystems configured need to follow the My Oracle Support Note 1274453.1 for detailed patch installation instructions.

# opatch auto <PATH_TO_PATCH_DIRECTORY>

# opatch auto <PATH_TO_PATCH_DIRECTORY> -oh <Comma separated Oracle home paths>

# opatch auto <PATH_TO_PATCH_DIRECTORY> -oh <GI Home>

2.5 Patch Deinstallation

You can use the following steps to rollback the BP3 bundle patch. Choose the instructions that apply to your needs. For non-Exadata customers who have ACFS filesystems configured need to follow the My Oracle Support Note 1274453.1 for detailed patch deinstallation instructions.

Case 1: Rolling Back the Oracle RAC Database Homes and GI Homes Together

Note  :  This case should be used on all Exadata Database Machines unless the configuration has been modified.As root user execute the following command on each node of the cluster.

# opatch auto <PATH_TO_PATCH_DIRECTORY> -rollback

Case 2: Rolling Back from the Oracle RAC Database Homes

You should use the following instruction if you prefer to rollback the patch from Non-Shared Oracle RAC databases homes alone.

As root user execute the following command on each node of the cluster.

# opatch auto <PATH_TO_PATCH_DIRECTORY> -oh <Comma separated Oracle home paths> -rollback

Case 3: Rollback from the GI Home Alone

If the GI home is not shared, then use the following instruction to rollback the patch from the GI home.As root user execute the following on each node of the cluster.

# opatch auto <PATH_TO_PATCH_DIRECTORY> -oh <GI Home> -rollback

2.6 Patch PostdeInstallation

After you deinstall the patch from a database home, follow these steps (this only needs to be completed on one instance for an Oracle RAC database):

1. Connect to the database as sys:

   $ sqlplus /nolog
   SQL> connect / as sysdba

2. Verify that an $ORACLE_HOME/rdbms/admin/catbundle_EXA_<database SID>_ROLLBACK.sql file exists for each database associated with the ORACLE_HOME which will be created after execution of Step (2) from “PostInstallation” and execute it as follows:

   SQL> @?/rdbms/admin/catbundle_EXA_<database SID>_ROLLBACK.sql

3. Navigate to the <ORACLE_HOME>/cfgtoollogs/catbundle directory(if ORACLE_BASE is defined then the logs will be created under ORACLE_BASE/cfgtoollogs/catbundle), and check the catbundle_EXA_<database SID>_ROLLBACK_<TIMESTAMP>.log file for any errors like “grep ^ORA <logfile> | sort -u”. Here, the format of the <TIMESTAMP> is YYYYMMDD_HH_MM_SS. If there are errors, refer to Section 3, “Known Issues”.
4. Ensure that you verify the Oracle Inventory and compare the output with the one ran in the Section 2.1.3 and re-apply any patches that were rolled back as part of this patch apply. To verify the inventory, run the following command:

   $ opatch lsinventory

3 Known Issues

For information about OPatch issues, see My Oracle Support Note 293369.1 OPatch documentation list.

Other known issues are as follows.

Issue 1

BUG 10400542 – Possible wrong results when DBM bundle is rolling installed.

Workaround:

Set the parameter below in init.ora before upgrading.

_fix_control=10026972:off

Issue 2

Known Issues for Opatch AutoBug 10339274 – ‘OPATCH AUTO’ FAILED TO APPLY 11202 PATCH ON EXADATA RAC CLUSTER WITH 11201 RAC

Bug 10339251 – MIN OPATCH ISSUE FOR DB HOME SETUP IN EXADATA RAC CLUSTER USING ‘OPATCH AUTO’

These two issues are observed in an environment where lower version database homes coexist with 11202 clusterware and database homes and opatch auto is used to apply the 11202 BP3.

Workaround:

Apply the 11202 BP3 to the 11202 GI Home and Oracle RAC database home as follows:

#opatch auto <UNZIPPED_PATCH_LOCATION> -oh <GI Home path>
#opatch auto <UNZIPPED_PATCH_LOCATION> -oh <11202 ORACLE_HOME1_PATH>,<11202 ORACLE_HOME1_PATH>

Issue 3

You may see the following errors while running the catbundle.sql script or its rollback script. These errors can be safely ignored.

4 References

The following documents are references for this patch.

Note 293369.1 OPatch documentation list

Note 1265683.1 OPatch Troubleshooting/FAQ Guide for Oracle Database Machine

Note 1265700.1 Oracle Patch Assurance – Data Guard Standby-First Patch Apply

Note 1265998.1 Patch Oracle Exadata Database Machine via Oracle Enterprise Manager 11gR1 (11.1.0.1)

Note 224346.1 OPatch – Where Can I Find the Latest Version of OPatch?

Note 1274453.1 Exadata Bundle Patch Install and Deinstall Procedure on ACFS filesystem for non-Exadata Systems

5 Bugs Fixed by This Patch

Oracle Exadata Database Recommended Patch (BP3) contains the following new fixes:

Automatic Storage management10332589             ORA-00600 [kfcInitRq20] when mounting a disk group

10329146             Possible lost writes with multiple DB writers and with multiple disks off lined at the same time

10310299             Possible corrupt extent when a rebalance and disk resync happen at the same time

10299224             Rebalancing a disk group with offline disks could cause corrupt blocks in the  database if the disk was on lined before restarting the database

10245086             Database may be stuck with a wrong extent when files are dropped and added or shrunk and grown quickly

10230571             ORA-600 [17183] when an update to one of the disks that is used for voting file fails

10222719             ASM hangs with processes waiting for the event “NO FREE BUFFERS”

10102506             Disk resync takes a long time even with no stale extents

10094201             Disk off lining is very slow

10084145             ORA-600 [1427] after interrupting an alter disk group mount of two or more disk groups

10046912             Failing to open a disk in external redundancy on Exadata could cause a hang

8685446               Enhancement to automatically allow a disk group with missing disk(s), when safe, to proceed with normal disk group mount operation

Buffer Cache Management

10355493             SEGSEGV while I/O reaping

10260870             ORA-00600 [kcbo_unlink_q_bg_3] on Active Data Guard

10051966             ORA-00600 [kfk_iodone_oss10] after I/O errors involving direct I/O

Generic

10353054             Wrong result due to loss of semi joins

10261680             Bloom filter predicate on a column added to a table after table creation  returns wrong result

10261072             A grouping sets query returns wrong results due to some groupings getting incorrectly pruned

10245259             Parallel insert on a partitioned table with a +noappend hint corrupts the table’s indexes

10229719             Core dump while using auditing with ORA-07445 in kzaSysAudit

10228393             Core dump with ORA-07445 in qsmqSetupRemoteObjects

10227133             Error after marking packages as hot

10151017             Merge statement is not shared due to insert_direct_load mismatch

10149223             wrong results due to unnesting of sub-query

10113803             ORA-00600 [17053] on dbms_shared_pool.keep with Edition Based Redefinition enabled

10077191             ORA-00600 [kkdlcob-objn-exists] with concurrent queries running with a single session

10026193             ORA-00600 [15740] from the PQ communication layer due to incorrect recovery

9671271               When PARALLEL_FORCE_LOCAL=TRUE & PARALLEL_DEGREE_POLICY=MANUAL are set, the default degree of parallelism is not calculated correctly

9591812               Wait for a cursor mutex in exclusive mode is erroneously recorded as a wait in share mode

High Availability

10314582             RDBMS IPC reply waits when _px_execution_services_enabled is set to false

10129643             Large number of “ksim generic wait event” waits due to busy LCK0

Oracle Space management

10195991             Slower performance on writes when multiple readers are reading a file

9414040               When adding temp files on one node and creating an index on another one,  temp file is being used before it’s been fully created

Oracle Storage Server Service Layer

10221016             ORA-00600 [2116] during database startup

10094949             Slow ASM operations after one cell is down

10113633             Upon switch failure, diskmon may delay declaring cell death

Oracle Transaction Management

10233732             Possible ORA-00600 [k2gUGPC: ptcnt >= tcnt] in distributed transactions in a RAC database

Oracle Utilities

9951423               Data Pump Export not reporting an error message for a transportable  table space operation when the user also specifies the SCHEMAS parameter

Oracle Virtual Operating System Services

10142788             In plsql native mode, shared library memory is not freed resulting in ORA-04030: out of process memory

10127360             ORA-04030 in trace files showing small memory allocations holding relatively large amounts of free memory

10048701             Possible ORA-00600 [kgskrecalc6] when parallel_degree_policy is set to auto and active session limit is active in any consumer group in plan

9924349              ORA-00600 [resplan:tryadd_3] on the cell, while plans are being set on the database

Tracking Bugs

10248523            DATABASE PSU 11.2.0.2.1

10626132            TRACKING BUG FOR 11202 BP3 DISKMON FIX

11074393            Fix for bug 8752691 backed out due to memory corruption issues