ORA-00600:[kole_t2u] [34]

 

The customer reports the error message:
————————————————–
ORA-00600:[kole_t2u], [34], [], [], [], [], [], []
————————————————–
the envrionment of customer are: oracle 10201 single instance+win2k3 r2 x64.
 
there are some clue in trace file:
 
#######################################################################
Dump file e:oracleproduct10.2.0admincdwbdumpcdw_m000_14920.trc
Wed Oct 03 16:00:59 2012
ORACLE V10.2.0.1.0 – 64bit Production vsnsta=0
vsnsql=14 vsnxtr=3
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 – 64bit Production
With the Partitioning, OLAP and Data Mining options
Windows Server 2003 Version V5.2 Service Pack 2
CPU                 : 8 – type 8664, 2 Physical Cores
Process Affinity    : 0x0000000000000000
Memory (Avail/Total): Ph:24113M/65530M, Ph+PgF:33540M/74161M
Instance name: cdw
 
Redo thread mounted by this instance: 1
 
Oracle process number: 59
 
Windows thread id: 14920, image: ORACLE.EXE (m000)
 
 
*** ACTION NAME:(Auto-Flush Slave Action) 2012-10-03 16:00:59.425
*** MODULE NAME:(MMON_SLAVE) 2012-10-03 16:00:59.425
*** SERVICE NAME:(SYS$BACKGROUND) 2012-10-03 16:00:59.425
*** SESSION ID:(967.21972) 2012-10-03 16:00:59.425
*** 2012-10-03 16:00:59.425
ksedmp: internal or fatal error
ORA-00600: internal error: [kole_t2u], [34], [], [], [], [], [], []
Current SQL statement for this session:
INSERT INTO wrh$_sqltext    (sql_id, dbid, sql_text,     command_type, snap_id, ref_count)  SELECT    sqlid_kewrstx, :dbid, sqlfulltext_kewrstx,    cmdtype_kewrstx, :lah_snap_id, 0 ref_count  FROM x$kewrtsqltext
—– Call Stack Trace —–
calling              call     entry                argument values in hex      
location             type     point                (? means dubious value)     
——————– ——– ——————– —————————-
ksedst+55            CALL???  ksedst1+573          94000000000 00001D688
                                                   02CC70AB0 00001E5F0
ksedmp+663           CALL???  ksedst+55            003A750A8 000000000 02CC6F528
                                                   000000000
ksfdmp+19            CALL???  ksedmp+663           000000003 05203F9D0 05203F9D0
                                                   003A975A0
kgerinv+158          CALL???  ksfdmp+19            000000000 000000000 000000000
                                                   000000000
kgesinv+33           CALL???  kgerinv+158          000000000 028C96FD8 000000001
                                                   0000007D0
kgesin+47            CALL???  kgesinv+33           0293FD3B0 0000003E8 027130110
                                                   02CC70EE8
kole_t2u+663         CALL???  kgesin+47            027136A30 05203F9D0
                                                   1FC400000000 000000001
koklwrite+1758       CALL???  kole_t2u+663         02AC80810 02CC70F48 0000003A1
                                                   0000002EA
koklc_write+217      CALL???  koklwrite+1758       02AC80810 000000000 0000003A1
                                                   02CC71FA0
kolaslWrite+798      CALL???  koklc_write+217      000000000 000000000 000000000
                                                   05203F9D0
kolaWrite+332        CALL???  kolaslWrite+798      027136A30 05203F9D0
                                                   FA000000000 00507CC57
koklwrite+2752       CALL???  kolaWrite+332        028C89478 000000000 0000203A1
                                                   0000002EA
qerfxCreateTempClob  CALL???  koklwrite+2752       028C89478 000000000 0000203A1
#####################################################################################
 
key information:
———————————————————————————————
INSERT INTO wrh$_sqltext (sql_id, dbid, sql_text,command_type, snap_id, ref_count) 
SELECT sqlid_kewrstx,:dbid,sqlfulltext_kewrstx,cmdtype_kewrstx, :lah_snap_id, 0 ref_count FROM x$kewrtsqltext
———————————————————————————————-   
MMON slave try to insert into wrh$_sqltext and trigger the error message,mmon is the background process.
so it seems there are do nothing with the application but only some awr/ash job done by mmon process.
 
Yes,indeed,you met the bug and the it's the bug 5017909 following as below:
——————————————————————————————————-
 Incorrect CLOB splits
 
This type of problem is the hardest to detect, and usually requires extensive debugging before the problem can be located, before going down this path it is usually preferable to go through the list of known issues and apply any known patches to rule out any known problems. 
The background of this type of error comes from the fact that CLOB data is sometimes split into more manageable chunks of data of a certain length. It this split is made after a certain number of bytes (for example 1000, or 4000, etc etc), then it could happen that the split happens in the middle of a multibyte codepoint. This then leaves the previous chunk with a incomplete codepoint at the end of the data, and this error can be expected. If this type of problem occurs, it is therefore due to a bug in the way CLOB data is split into chunks. Rather than making the split based on bytes it should always be made based on full characters.
 
Bug 5017909 – fixed in 10.2.0.4 (and higher 10.2 patchsets) and 11.1 (and higher)
Due to a bug in the "cut" in the data as described above, this bug can cause v$sqlarea.sql_fulltext and v$sql.sql_fulltext to contain sql statements in whichinvalid multibyte codepoints are used. These can subsequently lead to a variety of issues, like:
 
(1)SQL Tuning advisor failing with ORA-904, ORA-911 and/or ORA-1740, and ORA-600 [kole_t2u], [34] can be found in the background.
(2)The MMON process periodically running into ORA-600 [kole_t2u], [34] errors
(3)Background processes (like MMON) running into this error when inserting into history tables like wrh$_sqltext
(4)ORA-600 [kole_t2u], [34] errors when using DBMS_STATS.GATHER_FIXED_OBJECTS_STATS
Many of these issues have been individually raised as bugs in the past, but in most cases they go back to the bad data introduced by bug 5017909.Individual patches for some platforms are available on top of 10.1.0.5, 10.2.0.2 and 10.2.0.3.
———————————————————————————————————-
 
It's quite the same situation as (2),(3).
 
you can describe the wrh$_sqltext and x$kewrtsqltext,there is the CLOB type column which may trigger the bug as "incorrect clob split"
 
SQL> desc wrh$_sqltext;
 Name                                                              Null?    Type
 —————————————————————– ——– ——————————————–
 SNAP_ID                                                                              NUMBER
 DBID                                                              NOT NULL   NUMBER
 SQL_ID                                                            NOT NULL VARCHAR2(13)
 SQL_TEXT                                                                           CLOB
 COMMAND_TYPE                                                               NUMBER
 REF_COUNT                                                                  NUMBER
 
SQL> desc x$kewrtsqltext;
 Name                                                              Null?    Type
 —————————————————————– ——– ——————————————–
 ADDR                                                                                        RAW(8)
 INDX                                                                                    NUMBER
 INST_ID                                                                                  NUMBER
 SQLID_KEWRSTX                                                              VARCHAR2(13)
 SQLIDNUM_KEWRSTX                                                           NUMBER
 CMDTYPE_KEWRSTX                                                            NUMBER
 SQLFULLTEXT_KEWRSTX                                                     CLOB
 
solution:
try to opath the patch or update to oracle 10204.
 
References:
Bug 5017909 – V$SQL / V$SQLAREA.SQL_FULLTEXT corrupt for multibyte [ID 5017909.8]
ORA-600 [kole_t2u], [34] – description, bugs, and reasons [ID 734474.1]