below is the 600 entry in the alertlog:
alert.log: Hex dump of Absolute File 14, Block 312821 in trace file /u01/ORAHOME/app/oracle/admin/TIGERS7/bdump/tigers7_dbw0_10999.trc *** Corrupt block relative dba: 0x0384c5f5 (file 14, block 312821) Bad header found during preparing block for write Data in bad block - type: 6 format: 1 rdba: 0x00000384 last change scn: 0xf90b.c5f55f7c seq: 0x9 flg: 0x72 consistency value in tail: 0x0001f90b check value in block header: 0x102, block checksum disabled spare1: 0x6, spare2: 0x2, spare3: 0x0 *** Thu Apr 16 18:32:48 2009 Errors in file /u01/ORAHOME/app/oracle/admin/TIGERS7/bdump/tigers7_dbw0_10999.trc: ORA-00600: internal error code, arguments: [kcbzpb_1], [59033077], [4], [1], [], [], [], [] Thu Apr 16 18:32:49 2009 Errors in file /u01/ORAHOME/app/oracle/admin/TIGERS7/bdump/tigers7_dbw0_10999.trc: ORA-00600: internal error code, arguments: [kcbzpb_1], [59033077], [4], [1], [], [], [], [] DBW0: terminating instance due to error 600 Instance terminated by DBW0, pid = 10999 Thu Apr 16 19:04:58 2009
After that, We have executed dbverify against the identified file and it produced no errors:
DBVERIFY: Release 9.2.0.8.0 - Production on Thu Apr 16 19:31:32 2009 Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved. DBVERIFY - Verification starting : FILE = /u32/ORAINDX/oradata/TIGERS7/indx01.dbf DBVERIFY - Verification complete Total Pages Examined : 1280000 Total Pages Processed (Data) : 0 Total Pages Failing (Data) : 0 Total Pages Processed (Index): 1262823 Total Pages Failing (Index): 0 Total Pages Processed (Other): 8751 Total Pages Processed (Seg) : 0 Total Pages Failing (Seg) : 0 Total Pages Empty : 8426 Total Pages Marked Corrupt : 0 Total Pages Influx : 0 Highest block SCN : 10386833124905 (2418.1602203177)
we do open a sr ,and oracle support suggest to do below query:
ACTION PLAN
===========
1) please describe the sequence of events leading up to the problem
2) please upload the alert.log. ZIP if >2MB. Dot not use RAR.
3) please describe your backup strategy:
a) when was your last valid backup?
b) are you using RMAN to perform this backup?
c) do you have all archivelogs from the last backup to now?
d) was this a hot or cold backup?
4) even if you’re not using RMAN, run the following in RMAN:
$ rman target /
RMAN> backup validate check logical database;
5) Once RMAN validate is completed, run the following in SQL*Plus as SYSDBA:
SQL> select * from v$database_block_corruption;
6) Please run the following query in SQL*Plus as SYSDBA
– db must be in either MOUNT or OPEN mode
– Save the queries to a file, eg. rec_query1.sql, then run it in SQL*Plus
—————– start ——————
set echo on
set pagesize 2000 linesize 200 trimspool on
col name form a60
col status form a10
col dbname form a15
col member form a60
col inst_id form 999
col resetlogs_time form a25
col created form a25
col DB_UNIQUE_NAME form a15
col stat form 9999999999
col thr form 99999
col “Uptime” form a80
spool rec_query1.out
show user
alter session set nls_date_format=’DD-MM-RR hh24:mi:ss’;
select inst_id, instance_name, status,
to_char(STARTUP_TIME,’dd-Mon-yyyy hh24:mi’) || ‘ – ‘ ||
trunc(SYSDATE-(STARTUP_TIME) ) || ‘ day(s), ‘ ||
trunc(24*((SYSDATE-STARTUP_TIME) – trunc(SYSDATE-STARTUP_TIME)))||’ hour(s), ‘ ||
mod(trunc(1440*((SYSDATE-STARTUP_TIME) – trunc(SYSDATE-STARTUP_TIME))), 60) ||’ minute(s), ‘ ||
mod(trunc(86400*((SYSDATE-STARTUP_TIME) – trunc(SYSDATE-STARTUP_TIME))), 60) ||’ seconds’
“Uptime”
from gv$instance
order by inst_id
/
select dbid, name dbname, open_mode, database_role,
to_char(created,’dd-Mon-YYYY hh24:mi:ss’) created,
to_char(resetlogs_time,’dd-Mon-YYYY hh24:mi:ss’) resetlogs_time
from v$database;
archive log list;
select count(*) from v$backup where status = ‘ACTIVE’;
select * from v$log;
select * from v$logfile;
select * from v$recover_file order by 1;
select distinct(status)from v$datafile;
select FILE#,TS# , status, NAME from v$datafile
where status not in (‘SYSTEM’,'ONLINE’)
order by 1;
select fhsta, count(*)
from X$KCVFH group by fhsta;
select min(fhrba_Seq), max(fhrba_Seq)
from X$KCVFH;
select hxfil FILE#,fhsta STAT,fhscn SCN,
fhthr thr, fhrba_Seq SEQUENCE,fhtnm TABLESPACE
from x$kcvfh order by 1;
7) dump the block. Run the following as SYSDBA in SQL*Plus:
SQL> alter session set max_dump_file_size=unlimited;
SQL> oradebug setmypid;
SQL> alter system dump datafile ‘full pathname for file 14′ block 312821;
SQL> oradebug tracefile_name;
==> upload the said trace file
8) run dbv against datafile 14:
$ dbv file=
Alert logfile shows…
Sun Oct 3 23:32:45 2010
Errors in file /orafs/admin/prodfcc/bdump/prodfcc_dbw0_1040460.trc:
ORA-00600: internal error code, arguments: [kcbzpb_1], [42211701], [4], [1], [], [], [], []
Sun Oct 3 23:32:45 2010
Errors in file /orafs/admin/prodfcc/bdump/prodfcc_dbw0_1040460.trc:
ORA-00600: internal error code, arguments: [kcbzpb_1], [42211701], [4], [1], [], [], [], []
DBW0: terminating instance due to error 600
Sun Oct 3 23:32:45 2010
Errors in file /orafs/admin/prodfcc/bdump/prodfcc_ckpt_872652.trc:
ORA-00600: internal error code, arguments: [], [], [], [], [], [], [], []
Sun Oct 3 23:32:47 2010
Errors in file /orafs/admin/prodfcc/bdump/prodfcc_lgwr_1241148.trc:
ORA-00600: internal error code, arguments: [], [], [], [], [], [], [], []
Sun Oct 3 23:32:47 2010
Errors in file /orafs/admin/prodfcc/bdump/prodfcc_pmon_659484.trc:
ORA-00600: internal error code, arguments: [], [], [], [], [], [], [], []
Sun Oct 3 23:33:01 2010
Instance terminated by DBW0, pid = 1040460
Sun Oct 3 23:34:48 2010
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
SCN scheme 3
Using log_archive_dest parameter default value
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up ORACLE RDBMS Version: 9.2.0.1.0.
System parameters with non-default values:
processes = 750
timed_statistics = TRUE
shared_pool_size = 3221225472
sga_max_size = 8490823192
large_pool_size = 33554432
java_pool_size = 33554432
control_files = /fccmisc/control01.ctl, /fccmisc/control02.ctl, /fccmisc/control03.ctl
db_block_size = 8192
db_cache_size = 536870912
compatible = 9.2.0.0.0
Looking at the uploaded alertlog file, we found messages like: “Bad header found during buffer read”.
And the block corruptions were reported in: files: #18, #17, #11 and #6.
Action Plan:
=========
1. Refer to note: 352907.1
Title: Script To Run DBV On All Datafiles Of the Database
And run DBV on all the datafiles and provide the output. SO that we can see the extensiveness of DB-corruption and proceed further.
Datafiles are opened in “read-only” mode by DBV, so it cannot corrupt the contents. (Cannot change the contents of the file being checked.)
DBV can be run against datafiles which are currently opened by a database instance – there is no need to shutdown the database.
After analyzing the trace file and the output of dbv it shows that there is no physical corruption.
This seems to be due to memory corruptions at that point of time. This memory corruptions can happen due to various reasons
including Oracle bugs, OS issues, memory /hardware issues, rogue processes updating Oracle Memory space etc.
Please check the hardware and OS logs if there some error reported at the time when ORA-00600 [kcbzpb_1].
There are many bugs which are fixed in 9.2.0.8 and 10g.
Few Known Bugs:
—————————
NB Bug Fixed Description
P* 6047085 Linux x64-64: SGA corruption / crash following any ORA-7445 See <>
6401576 9.2.0.8.P22 OERI[ktbair1] / ORA-600 [6101] index corruption possible
PI 5496862 10.2.0.3, 11.1.0.6 AIX: Mandatory patch to use Oracle with IBM Technology Level 5 (5300-5) See <>
P 2747111 9.2.0.4, 10.1.0.2 Tru64: RAC instance may crash with OERI:[SKGXPIODONE17] / DBWR dies with OERI:KCBZPB_1
2002197 9.0.1.3, 9.2.0.1 OERI:KCBZPB_1 possible in PQ slave
This is for your information.
Plan to come to latest oracle patchset versions ie 10g or 11g. As you know 9i database version that has moved to sustaining support .
Advantages for being on the latest patchset release:
+ Latest patchset contains fix for all the known issues/Bugs raised in previous releases.
+ Patchsets doesn’t include any new functionality, it’s only for bug fixing
+ New bugs can only be logged against latest patchset release of supported versions
+ Backports can only be requested against latest patchset release of supported versions
+ Most stable release since a lot of bugs are already be fixed in the successive patchsets