Oracle10gR2 Alert E-Mail – Part 2

I really did need to apply patch 5117076 based on the Oracle10gR2 Alert E-Mail. Just to refresh, this is a problem regarding the 10.2.0.2 RDBMS server patchset for Sun Sparc Solaris 64-bit. This post is a little long, but I have tried to include as much detail hoping it will save you from the frustration I encountered by not following the Alert E-Mail.

Onto the gory details. This is what my installation looked like prior to my exercise using “opatch lsinventory”.

Installed Top-level Products (2):

Oracle Database 10g 10.2.0.1.0
Oracle Database 10g Release 2 Patch Set 1 10.2.0.2.0
There are 2 products installed in this Oracle Home.

Interim patches (2) :

Patch 5458753 : applied on Fri Oct 20 15:52:49 CDT 2006
Created on 22 Sep 2006, 08:10:13 hrs PST8PDT
Bugs fixed:
5458753

Patch 5225799 : applied on Tue Jul 25 14:57:14 CDT 2006
Created on 17 Jul 2006, 05:37:39 hrs PST8PDT
Bugs fixed:
5092134, 5099995, 5242650, 5079037, 5079038, 4523125, 5225799, 4669305

I planned to rollback patch 5458753 first, then apply patch 5117016, and then re-apply patch 5458753. I would leave the CPUJul2006 patch (5225799) intact since I intended to apply the CPUOct2006 patch which will automatically rollback CPUJul2006.

My intention to leave CPUJul2006 intact could not be more wrong. Do not take shortcuts in this process. All interim patches must be removed before applying patch 5117016.

Started with the rollback of patch 5458753. That was successfully removed without a problem.

%> opatch rollback -id 5458753

Next, instead of removing patch 5225799 (CPUJul2006), I went on to apply patch 5117016. The patch installed successfully, but this is where my mistake began. All interim patches must be removed prior to installing patch 5117016.

%> cd 5117016
%> opatch apply

Mistakenly, I proceeded by applying patch 5458753 which also installed successfully.

%> cd 5458753
%> opatch apply

Finally, I attempted to install the CPUOct2006 patch (5490848). As expected, this patch removed the CPUJul2006 patch by default. Everything looks okay until the relink phase.

Running make for target ioracle
Make failed to invoke “/usr/ccs/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=/disk01/app/oracle/product/10.2.0/db_1″….
‘Undefined first referenced symbol in file ksqgel /disk01/app/oracle/product/10.2.0/db_1/lib/
/libserver10.a(kkzf.o)
ld: fatal: Symbol referencing errors. No output written
/disk01/app/oracle/product/10.2.0/db_1/rdbms/lib/oracle
make: Fatal error: Command failed for target `/disk01/app/oracle/product/10.2.0/db_1/rdbms/lib/oracle’

Searched Metalink using “libserver10.a(kkzf.o)” and found Note 396649.1 which outlines three solutions to resolve the problem. I chose option #3.

OPTION 3: Extracting libserver10.a from the 10.2.0.2 Patchset

1. Rollback ALL Interim patches installed

2. Extract libserver10.a from 10.2.0.2 patchset staging area

% cd /tmp
% jar xvf <10.2.0.2 patchset staging area>
/Disk1/stage/Patches/oracle.rdbms/10.2.0.2.0/1
/DataFiles/filegroup14.1.1.jar libserver10.a

3. Backup original

% mv $ORACLE_HOME/lib/libserver10.a $ORACLE_HOME/lib/libserver10.a.bad

4. Move extracted libserver10.a to correct location

% mv libserver10.a $ORACLE_HOME/lib/libserver10.a

5. Relink

% relink all

You can now reinstall any Interim patch that your installation requires

This is a snapshot of where I was before beginning the interim patch rollbacks.

Installed Top-level Products (2):

Oracle Database 10g 10.2.0.1.0
Oracle Database 10g Release 2 Patch Set 1 10.2.0.2.0
There are 2 products installed in this Oracle Home.

Interim patches (3) :

Patch 5458753 : applied on Wed Nov 15 14:24:10 CST 2006
Created on 22 Sep 2006, 08:10:13 hrs PST8PDT
Bugs fixed:
5458753

Patch 5117016 : applied on Wed Nov 15 14:19:41 CST 2006
Created on 30 Mar 2006, 12:52:17 hrs US/Pacific
Bugs fixed:
5117016

Patch 5225799 : applied on Tue Jul 25 14:57:14 CDT 2006
Created on 17 Jul 2006, 05:37:39 hrs PST8PDT
Bugs fixed:
5092134, 5099995, 5242650, 5079037, 5079038, 4523125, 5225799, 4669305

Began to rollback the interim patches as specified in Step #1. Rolled back in this order

opatch rollback -id 5458753
opatch rollback -id 5117016
opatch rollback -id 5225799

Finally, all patches removed.

Installed Top-level Products (2):

Oracle Database 10g 10.2.0.1.0
Oracle Database 10g Release 2 Patch Set 1 10.2.0.2.0
There are 2 products installed in this Oracle Home.

There are no Interim patches installed in this Oracle Home.

Followed Steps #2 thru #5 and then applied all the patches again in the correct order.

1. Patch 5117016
2. Patch 5458753
3. Patch 5225799

Finally got back to where I needed to be prior to the mess I created. Unfortunately, I did not install CPUOct2006 because my maintenance window elapsed. Had I followed directions from the beginning, I would have avoided this situation. Now I have to schedule another time to apply CPUOct2006.

Installed Top-level Products (2):

Oracle Database 10g 10.2.0.1.0
Oracle Database 10g Release 2 Patch Set 1 10.2.0.2.0
There are 2 products installed in this Oracle Home.

Interim patches (3) :

Patch 5225799 : applied on Wed Nov 15 15:54:05 CST 2006
Created on 17 Jul 2006, 05:37:39 hrs PST8PDT
Bugs fixed:
5092134, 5099995, 5242650, 5079037, 5079038, 4523125, 5225799, 4669305

Patch 5458753 : applied on Wed Nov 15 15:50:21 CST 2006
Created on 22 Sep 2006, 08:10:13 hrs PST8PDT
Bugs fixed:
5458753

Patch 5117016 : applied on Wed Nov 15 15:48:22 CST 2006
Created on 30 Mar 2006, 12:52:17 hrs US/Pacific
Bugs fixed:
5117016

Advertisements

2 Responses to Oracle10gR2 Alert E-Mail – Part 2

  1. Garr Updegraff says:

    Mike,

    Thank you so much for posting this entry! I ran into precisely this problem on one of our Solaris servers here in the UC Irvine Registrar Office when I tried to install Oracle’s October CPU, and I had no clue was to why opatch failed so badly.

    As it turned out, I had to go back through your directions a couple of times to get the correct sequence. You’re right: all the old interim patches have to be rolled back, the libserver10.a file reinstalled, then Interim Patch p5117016 fix for the libserver10.a location problem, and finally the October/December CPU. Wow, did this security patch suck up time — but not nearly as much as it would have without your posting!

    — Garr

  2. mrothouse says:

    Garr,
    Glad to hear this was useful information and everything worked out well in the end for you. I agree, this patch took way more time and stress than I originally anticipated.

    Mike

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: