Wednesday 2 December 2015

How to Troubleshoot Oracle Wait Event enq: TX - allocate ITL entry

Problem Description
How to Troubleshoot Oracle Wait Event enq: TX - allocate ITL entry

Explanation

ITL (Interested Transaction List) is related to every Data & Index Block. Before a Transaction begin it need to acquire the ITL on the block where the row need to be modified.
There is a small section in the block header of every DATA & INDEX block which has this ITL information (defined by the INITRANS & MAXTRANS parameter).
The ITL structure is created in the block during segment creation as per the INITRANS value and grows dynamically as concurrency occurs to a maximum of MAXTRANS value.
For every single transaction on that block there will be one ITL entry.
For example, if a INITRANS is 5 then there will be 5 ITL rows/entries in each block of that segment.

A session  waits for this event when it is unable to acquire an ITL entry in a block. The primary cause of ITL waits is that free slots in the ITL structure in the block are not available and it is unable to expand the ITL slots.
The lack of slots can be due to
a.       Low setting of the INITRANS & MAXTRANS, which reserves minimal ITL slots initially and place a hard limit on the number of transactions that can have locks on a block respectively
b.      And/Or, the block is so packed that there is no room for the ITL to grow

Troubleshooting Approach/Solution

Step 1

Find the object and the block related to this Wait Event

select a.sid,a.serial#,a.event,a.sql_id,a.ROW_WAIT_OBJ#, a.ROW_WAIT_FILE#,a.ROW_WAIT_BLOCK#,b.owner,b.object_type,b.object_name
from v$session a, dba_objects b
where a.event='enq: TX - allocate ITL entry' and a.ROW_WAIT_OBJ#=b.object_id;

Step 2

Find the INITRANS & MAXTRANS of the object

Table
select owner,table_name,INI_TRANS,MAX_TRANS from dba_tables where owner='SCOTT' and table_name='EMP';

Index
select owner,index_name,INI_TRANS,MAX_TRANS from dba_indexes where owner='SCOTT' and index_name='PK_EMP_IDX';

Step 3

Find the actual ITL slots available in the block header by dumping the block. You need to check the maximum available ITL slots and how many of them are free

              Command to Dump the Block
Alter system dump datafile block ;
              Find the Trace File Name
SELECT value FROM   v$diag_info WHERE  name = 'Default Trace File';

Identify the trace file and check for the ITL section (Note : A sample view of the ITL is provided below)

Itl           Xid                  Uba         Flag  Lck        Scn/Fsc
0x01   0x0000.000.00000000  0x00000000.0000.00  ----    0  fsc 0x0000.00000000
0x02   0x000a.01b.00000232  0x00c158cc.005c.2a  --U-    4  fsc 0x0000.000e593b

Step 4

You may want to suggest the following solution (depending on the scenario) to resolve this issue
a.       Increase INITRANS
b.      Increase PCTFREE
c.       Increase both INITRANS & PCTFREE

Note :
a.       All the above solution need a rebuild to take the impact
b.      Remember ASMM which may be part of your database.

c.       Increasing either INITRANS & PCTFREE is going to impact the block capacity in storing the rows

Wednesday 5 June 2013

Oracle - Row Source Operation


Overview

Generally, in Performance Tuning thoery, Row Source & Record set are two different terminology which some time can be mis-perceived or mis-interpretted. According to me, ROW SOURCE is a superset which has more potential information related to performance statistics which helps to diagnose, troubleshoot in query tuning. On the contrary, RECORDSET is simply a set of rows/records fetched in a particular operation of an execution plan.

Even some Performance Tuning document/book talk about OPTIMIZER which decides the execution plan based on the ROW SOURCE. ROW SOURCE that is being referred here is just record set, i.e. number of rows that may result out of an operation in an execution plan.

Having said what is RECORDSET, ROW SOURCE provides additional information related to reads, writes and importantly "Elapsed Time" for each operation in an execution plan. ROW SOURCE contains execution statistics from performance perspective and is actually associated with an execution of SQL Query.

Where can I find this Row Source information?

Usually this Row Source information is accessed by generating a TKPROF output on a session trace file.
Fo eg. You can trace a session using 10046 event.

Note : Focus is retained to ROW SOURCE hence I am not providing more information on how to trace a session.

The below information is a copy paste.



Rows       Row Source Operation
-------     ---------------------------------------------------
0              LOAD TABLE CONVENTIONAL (cr=14 pr=6 pw=0 time=0 us)
199          NESTED LOOPS (cr=0 pr=0 pw=0 time=43281016 us cost=1 size=548 card=1)
39800      MERGE JOIN CARTESIAN (cr=0 pr=0 pw=0 time=103272 us cost=1 size=496 card=1)
199          HASH JOIN (cr=0 pr=0 pw=0 time=3960 us cost=1 size=466 card=1)
199          HASH JOIN (cr=0 pr=0 pw=0 time=594 us cost=1 size=349 card=1)
199          FIXED TABLE FULL X$KCCFN (cr=0 pr=0 pw=0 time=297 us cost=0 size=310 card=1)
400          FIXED TABLE FULL X$KCBFWAIT (cr=0 pr=0 pw=0 time=199 us cost=0 size=3900 card=100)
200          FIXED TABLE FULL X$KCFIO (cr=0 pr=0 pw=0 time=995 us cost=0 size=11700 card=100)
39800      BUFFER SORT (cr=0 pr=0 pw=0 time=34899 us cost=1 size=3000 card=100)
200          FIXED TABLE FULL X$KCCTS (cr=0 pr=0 pw=0 time=99 us cost=0 size=3000 card=100)
199          FIXED TABLE FIXED INDEX X$KCCFE (ind:1) (cr=0 pr=0 pw=0 time=0 us cost=0 size=52 card=1)


New Feature in 10g

However, starting from 10g onwards, Oracle has provided the following two views using which we can use to obtain the "Row Source Operation" information without tracing.

a. V$SQL_PLAN_STATISTICS
b. V$SQL_PLAN_STATISTICS_ALL

Note : Remember that Oracle will populate these views with these performance statistics only when using STATISTICS_LEVEL=ALL


Eg.
alter session set STATISTICS_LEVEL=ALL;


Example.
select * from table(dbms_xplan.display_cursor('',NULL));
col a form 999999 head "Rows"
col b form a65 head "Row Source Operation"
select
last_output_rows a,
operation||' '||object_name||' (cr='||to_char(last_cr_buffer_gets)||' pr='||to_char(last_disk_reads)||' pw='||to_char(last_disk_writes)||' time='||to_char(last_elapsed_time)||')' b
from v$sql_plan_statistics_all
where sql_id='' and chil_number=
order by id;


How to interpret this Row Source?

I am taking the below simple example to explain this. This example shows only one operation out of an entire execution plan.

Rows           Row Source Operation
-----------   ---------------------------------------------------
10000000   TABLE ACCESS FULL EMPLOYEE (cr=3000 pr=100000 pw=100 time=21000000)

My interpretation from this above example is
"Oracle took 21 seconds (From 9i onwards this number is represented in microseconds) to fetch 10 million rows"

What is the USE of Row Source?

Generally this ROW SOURCE is used to pin point which OPERATION has consumed more time and is an expensive.OPERATION here means the individual operation in an execution plan.

For Eg.
select OPERATION_ID,OPERATION||' 'OBJECT_NAME
from v$sql_plan where sql_id='ild_number=;

Let us assume a query is running long and you are expected to tune this query. However this is a BIG query running into two pages. It might be difficult to tune this query as you need to know the starting point of the problem. In this situation, Row Source can help to identify the operation which is taking a long time.

You need to find out the Row in the Row Source Operation whose "time" is big and drill down from there.

Wednesday 17 April 2013

How DATA Dictionary is created in Oracle?

I feel silly when I presumed the below information was already available in the blog/internet. However later I realized that it is NOT and hence it prompted me to add this blog.

Question : What is DATA Dictionary in an Oracle Database?


Answer : DATA Dictionary is also called METADATA. Oracle documentation tells that METADATA is created by running catalog.sql. While this is partially TRUE, the catalog.sql creates only the view/synonyms and few additional package/procedure/functions..also few more objects. However it certainly doesn't create the actual metadata. These views are created to access the metadata in a readable format. Metadata doesn't reside in the VIEWs rather it resides in the $(dollar) tables. For example, whatever information that you see by querying dba_objects view is majorly sourced from the dollar table called obj$. Likewise dba_tables view is sourced from tab$

In summary all the metadata resides in the DOLLAR table and they are not views.

Few examples,
obj$ - For objects
tab$ - For tables
ts$ - For tablespace
file$ - For datafiles
seg$ - For segments

Question : How & When DATA Dictionary is created in an Oracle Database?


Answer : DATA Dictionary is actually created during the Database creation time. After NOMOUNTING the instance you run the CREATE DATABASE command. This command, after creating the SYSTEM datafile it calls a SQL file ?/rdbms/admin/sql.bsq which creates the metadata



Wednesday 3 April 2013

Different ways to invalidate a cursor in memory for Oracle database

Purpose : Whilst you may have a various different situation and need to invalidate a cursor from the oracle cache/memory at the instance level, I always had the need after pinning a SQL profile.

Note : Pinning a plan through SQL profile doesn't call for an explicit hard parsing.

Simple method/way to invalidate a cursor

1. Flush the SHARED_POOL

Eg.
alter system flush shared_pool;

2. Delete or Gather statistics of the related objects (table/index) which are referenced in the cursor (execution plan) using DBMS_STATS

3. Grant or Revoke privilege on the related objects (table) which are referenced in the cursor

Eg.
I always use the following commands together on the tables which are referenced in the SQL query/cursor

grant select on scott.emp to system;
revoke select on scott.emp from system;

4. Alter the related objects (table/index) which are referenced in the cursor using DDL command.

5. You can also PURGE the cursor from the shared pool

Eg.
exec dbms_shared_pool.purge(',','C');

The above methods helped me in the situation which I described in the PURPOSE section.

However the below alternate methods may also be considered depending on your situation.

6. You can even add a hint to the SQL query to create a new sql_id. Hence it will be a fresh parse.

Eg.
select /* query1 */ empno, ename from emp;
select /* query2 */ empno, ename from emp;


7. Ofcourse, always database BOUNCE will never let you down :-) Just an extreme step.

Keywords : parse, hard, soft, invalid cursor, invalidate cursor, memory, cache, oracle

Friday 18 May 2012

Issues in Service Registration to SCAN listener

Scneario

In a 3 node RAC/Cluster environment, one node (in our case node 2) crashed resulting in relocating the SCAN listener to another available node. However 2 out of 3 SCAN listener shows all the services registered in it except the last one which got relocated.

Hence the client connectivity intermittently experiences ORA-12514 error. i.e. Service not registered/unknown

Cause

Failed over SCAN listener hasn't got all the services registered in it hence whenever a client connection comes to this scan listener it was unable to process/redirect to the correct instance.

Solution

Simply de-register all the services from the remote/scan listener and re-register the same.
This can be achieved using the below simple script which helped us from avoiding a big extended outage.

deregister =>    ALTER SYSTEM SET REMOTE_LISTENER='';
Re-register => ALTER SYSTEM SET REMOTE_LISTENER='plocdb14:50000';

How to pin a desired Plan for a query in Oracle database using SQL Profile

Overview

Starting since 10g people started preferring SQL Profile to pin the plan which seems to be more efficient than the traditional tool - STORED Outline

SQL Profile is more efficient, simple to use, more integrated to AWR.

Personally I feel SQL Profile is more convenient to use and my preferred tool to use as it is very quick and effective.

Methods to use

Method 1 - Below script uses SQL Profile to pin a plan to sqlid provided the plan is either available in memory or in AWR

PROMPT ============================== SQL Profile Inputs ==============================
set serveroutput on
ACCEPT dst_sql_id       PROMPT "Enter Destination SQL_ID: "
ACCEPT src_sql_id       PROMPT "Enter Source SQL_ID: "
ACCEPT plan_hash_value  PROMPT "Enter Source PLAN_HASH_VALUE: "
ACCEPT force_match      PROMPT "Enter FORCE_MATCH option (TRUE/FALSE), with TRUE equivalent to CURSOR_SHARING of FORCE: "
PROMPT ================================================================================
DECLARE
   c_prof_name_prefix  CONSTANT  VARCHAR2(5)  := 'SQLT';
   c_category          CONSTANT  VARCHAR2(10)  := 'DEFAULT';
   c_validate          CONSTANT  BOOLEAN       := TRUE;
   c_replace           CONSTANT  BOOLEAN       := TRUE;
   l_dst_sql_id           VARCHAR2(20)      := '&&dst_sql_id';
   l_src_sql_id           VARCHAR2(20)      := '&&src_sql_id';
   l_plan_hash_value      NUMBER            := &&plan_hash_value;
   l_plan_hash_value_chk  NUMBER;
   l_force_match_str      VARCHAR2(10)      := '&&force_match';
   l_profile_hints        SYS.SQLPROF_ATTR;
   l_dst_sql_text         CLOB;
   l_src_sql_text         CLOB;
   l_signature            NUMBER;
   l_force_match          BOOLEAN;
   l_profile_name         VARCHAR2(30)      := c_prof_name_prefix || '_' || l_dst_sql_id || '_' || l_plan_hash_value;
   l_profile_desc         VARCHAR2(50);
   err_dst_sql_id           EXCEPTION;
   err_src_sql_id           EXCEPTION;
   err_src_plan_hash_value  EXCEPTION;
   CURSOR sc_dst_sql_text IS
   SELECT sql_text
     FROM (SELECT sql_fulltext  sql_text
             FROM gv$sql
            WHERE sql_id = l_dst_sql_id
            UNION ALL
           SELECT sql_text
             FROM dba_hist_sqltext
            WHERE sql_id = l_dst_sql_id
          )
    WHERE ROWNUM = 1;
   CURSOR sc_src_sql_text IS
   SELECT sql_text
     FROM (SELECT sql_fulltext  sql_text
             FROM gv$sql
            WHERE sql_id = l_src_sql_id
            UNION ALL
           SELECT sql_text
             FROM dba_hist_sqltext
            WHERE sql_id = l_src_sql_id
          )
    WHERE ROWNUM = 1;
   CURSOR sc_src_plan_hash_value IS
   SELECT plan_hash_value
     FROM gv$sql_plan
    WHERE sql_id = l_src_sql_id
      AND plan_hash_value = l_plan_hash_value
      AND other_xml IS NOT NULL
    UNION
   SELECT plan_hash_value
     FROM dba_hist_sql_plan
    WHERE sql_id = l_src_sql_id
      AND plan_hash_value = l_plan_hash_value
      AND other_xml IS NOT NULL;
   CURSOR sc_profile_attr IS
   SELECT category
        , type
        , created
        , last_modified
        , status
        , force_matching
        , signature
        , description
     FROM dba_sql_profiles
    WHERE name = l_profile_name;
   CURSOR sc_prof_hints IS
   SELECT SUBSTR(attr_val, 1, 120) hint
     FROM sys.sqlprof$attr
    WHERE signature = l_signature
    ORDER BY category
           , attr#;
BEGIN
   OPEN sc_dst_sql_text;
   FETCH sc_dst_sql_text INTO l_dst_sql_text;
   CLOSE sc_dst_sql_text;
   IF l_dst_sql_text IS NULL THEN
      RAISE err_dst_sql_id;
   END IF;
   OPEN sc_src_sql_text;
   FETCH sc_src_sql_text INTO l_src_sql_text;
   CLOSE sc_src_sql_text;
   IF l_src_sql_text IS NULL THEN
      RAISE err_src_sql_id;
   END IF;
   OPEN sc_src_plan_hash_value;
   FETCH sc_src_plan_hash_value INTO l_plan_hash_value_chk;
   CLOSE sc_src_plan_hash_value;
   IF l_plan_hash_value_chk IS NULL THEN
      RAISE err_src_plan_hash_value;
   END IF;
   SELECT EXTRACTVALUE(VALUE(sub1), '/hint')  AS outline_hints
     BULK COLLECT
     INTO l_profile_hints
     FROM XMLTABLE('/*/outline_data/hint'
                   PASSING (SELECT xmltype(other_xml)  AS xmlval
                              FROM (SELECT other_xml
                                      FROM gv$sql_plan
                                     WHERE sql_id = l_src_sql_id
                                       AND plan_hash_value = l_plan_hash_value
                                       AND other_xml IS NOT NULL
                                     UNION ALL
                                    SELECT other_xml
                                      FROM dba_hist_sql_plan
                                     WHERE sql_id = l_src_sql_id
                                       AND plan_hash_value = l_plan_hash_value
                                       AND other_xml IS NOT NULL
                                   )
                             WHERE ROWNUM = 1
                           )
                  ) sub1;
   IF UPPER(l_force_match_str) = 'TRUE' THEN
      l_force_match := TRUE;
   ELSE
      l_force_match := FALSE;
   END IF;
   l_signature := DBMS_SQLTUNE.SQLTEXT_TO_SIGNATURE(l_dst_sql_text);
   l_profile_desc := l_profile_name || '_' || l_signature;
   DBMS_SQLTUNE.IMPORT_SQL_PROFILE(sql_text     => l_dst_sql_text
                                 , profile      => l_profile_hints
                                 , name         => l_profile_name
                                 , description  => l_profile_desc
                                 , category     => c_category
                                 , validate     => c_validate
                                 , replace      => c_replace
                                 , force_match  => l_force_match
                                  );
   DBMS_OUTPUT.PUT_LINE('SQL Profile ' || l_profile_name || ' successfully created');
   DBMS_OUTPUT.PUT_LINE('................................................................................');
   DBMS_OUTPUT.PUT_LINE('--------------------------------------------------------------------------------');
   DBMS_OUTPUT.PUT_LINE('Destination SQL Text');
   DBMS_OUTPUT.PUT_LINE('--------------------------------------------------------------------------------');
   DBMS_OUTPUT.PUT_LINE(l_dst_sql_text);
   DBMS_OUTPUT.PUT_LINE('--------------------------------------------------------------------------------');
   DBMS_OUTPUT.PUT_LINE('Source SQL Text');
   DBMS_OUTPUT.PUT_LINE('--------------------------------------------------------------------------------');
   DBMS_OUTPUT.PUT_LINE(l_src_sql_text);
   DBMS_OUTPUT.PUT_LINE('--------------------------------------------------------------------------------');
   DBMS_OUTPUT.PUT_LINE('Attributes');
   DBMS_OUTPUT.PUT_LINE('--------------------------------------------------------------------------------');
   FOR x IN sc_profile_attr LOOP
      DBMS_OUTPUT.PUT_LINE('Category:        ' || x.category);
      DBMS_OUTPUT.PUT_LINE('Type:            ' || x.type);
      DBMS_OUTPUT.PUT_LINE('Created:         ' || x.created);
      DBMS_OUTPUT.PUT_LINE('Last Modified:   ' || x.last_modified);
      DBMS_OUTPUT.PUT_LINE('Status:          ' || x.status);
      DBMS_OUTPUT.PUT_LINE('Force Matching:  ' || x.force_matching);
      DBMS_OUTPUT.PUT_LINE('Signature:       ' || x.signature);
      DBMS_OUTPUT.PUT_LINE('Description:     ' || x.description);
   END LOOP; 
   DBMS_OUTPUT.PUT_LINE('--------------------------------------------------------------------------------');
   DBMS_OUTPUT.PUT_LINE('Hints');
   DBMS_OUTPUT.PUT_LINE('--------------------------------------------------------------------------------');
   FOR x IN sc_prof_hints LOOP
      DBMS_OUTPUT.PUT_LINE(x.hint);
   END LOOP;
   DBMS_OUTPUT.PUT_LINE('================================================================================');
EXCEPTION
   WHEN err_dst_sql_id THEN
      DBMS_OUTPUT.PUT_LINE('Error:  Destination SQL_ID "' || l_dst_sql_id || '" not found.');
   WHEN err_src_sql_id THEN
      DBMS_OUTPUT.PUT_LINE('Error:  Source SQL_ID "' || l_src_sql_id || '" not found.');
   WHEN err_src_plan_hash_value THEN
      DBMS_OUTPUT.PUT_LINE('Error:  Source PLAN_HASH_VALUE "' || l_plan_hash_value || '" not found.');
   WHEN OTHERS THEN
      RAISE;
END;
/


Method 2 - There is also a backdoor method to create a SQL profile if the plan is not available in either place, which can be done using SQLT, which again uses IMPORT_SQL_PROFILE with manually created SQLPROF_ATTR (i.e. SQL Profile Attribute)

Below example need to properly converted according to your scenario.

SET DEF ^ ECHO ON TERM ON LIN 2000 TRIMS ON SERVEROUT ON SIZE 1000000 TIM OFF SQLP SQL>;
SPO SQLT_2wskdsyt6w2ty_p4150230984.log;
SET ECHO OFF;
PRO
PRO ... Generating manual custom SQL Profile for SQL text provided
PRO
SET TERM OFF HEA ON TIM OFF;
WHENEVER SQLERROR EXIT SQL.SQLCODE;
VAR signature NUMBER;
DECLARE
  sql_txt VARCHAR2(32767);
  h       SYS.SQLPROF_ATTR;
BEGIN
  sql_txt := q'[SELECT  /*YANTRA*/    YFS_ORDER_HEADER.ORDER_HEADER_KEY  FROM YFS_ORDER_HEADER YFS_ORDER_HEADER          WHERE PAYMENT_STATUS IN ( :"SYS_B_00",:"SYS_B_01",:"SYS_B_02",:"SYS_B_03",:"SYS_B_04",:"SYS_B_05",:"SYS_B_06",:"SYS_B_07",:"SYS_B_08",:"SYS_B_09",:"SYS_B_10",:"SYS_B_11")  AND AUTHORIZATION_EXPIRATION_DATE <= :"SYS_B_12"  AND ENTERPRISE_KEY = :"SYS_B_13"  AND DOCUMENT_TYPE = :"SYS_B_14"  AND NOT EXISTS ( SELECT :"SYS_B_15" FROM YFS_ORDER_HOLD_TYPE WHERE YFS_ORDER_HOLD_TYPE.ORDER_HEADER_KEY = YFS_ORDER_HEADER.ORDER_HEADER_KEY AND HOLD_TYPE IN ( :"SYS_B_16",:"SYS_B_17" ) AND STATUS < :"SYS_B_18") ]';
  h := SYS.SQLPROF_ATTR(
      q'[IGNORE_OPTIM_EMBEDDED_HINTS]',
      q'[OPTIMIZER_FEATURES_ENABLE('10.2.0.4')]',
      q'[OPT_PARAM('optimizer_dynamic_sampling' 1)]',
      q'[OPT_PARAM('optimizer_index_cost_adj' 50)]',
      q'[OUTLINE_LEAF(@"SEL$5DA710D3")]',
      q'[UNNEST(@"SEL$2")]',
      q'[OUTLINE(@"SEL$1")]',
      q'[OUTLINE(@"SEL$2")]',
      q'[INDEX(@"SEL$5DA710D3" mailto:%22YFS_ORDER_HEADER%22@%22SEL$1" ("YFS_ORDER_HEADER"."ENTERPRISE_KEY" "YFS_ORDER_HEADER"."DOCUMENT_TYPE" "YFS_ORDER_HEADER"."PAYMENT_STATUS" "YFS_ORDER_HEADER"."AUTHORIZATION_EXPIRATION_DATE" "YFS_ORDER_HEADER"."ORDER_HEADER_KEY"))]',
      q'[INDEX_RS_ASC(@"SEL$5DA710D3" mailto:%22YFS_ORDER_HOLD_TYPE%22@%22SEL$2" ("YFS_ORDER_HOLD_TYPE"."ORDER_HEADER_KEY" "YFS_ORDER_HOLD_TYPE"."ORDER_LINE_KEY" "YFS_ORDER_HOLD_TYPE"."HOLD_TYPE"))]',
      q'[LEADING(@"SEL$5DA710D3" mailto:%22YFS_ORDER_HEADER%22@%22SEL$1" mailto:%22YFS_ORDER_HOLD_TYPE%22@%22SEL$2%22)]',
      q'[USE_NL(@"SEL$5DA710D3" mailto:%22YFS_ORDER_HOLD_TYPE%22@%22SEL$2%22)]'
  );
  :signature := DBMS_SQLTUNE.SQLTEXT_TO_SIGNATURE(sql_txt);
  DBMS_SQLTUNE.IMPORT_SQL_PROFILE (
    sql_text    => sql_txt,
    profile     => h,
    name        => 'SQLT_2wskdsyt6w2ty_p4150230984',
    description => 'SQLT 2wskdsyt6w2ty p4150230984'||:signature,
    category    => 'DEFAULT',
    validate    => TRUE,
    replace     => TRUE,
    force_match => FALSE /* TRUE:FORCE (match even when different literals in SQL). FALSE:EXACT (similar to CURSOR_SHARING) */ );
END;
/
 

Tuesday 7 December 2010

Key points to consider for Oracle RAC/Cluster in Linux

1. Ensure Max. Open Files is set to 65k for Oracle Cluster.
Reference
Oracle Bug # 8429716
Oracle Metalink Note id # 858279.1

You may see timeout errors like below in the crsd.log file.
Important : Oracle Metalink note id # 732086.1 & Bug 6196746 may misguide you.

2010-12-07 03:14:54.776: [ CRSEVT][1494178112] CAAMonitorHandler :: 0:Action Script /u01/crs/oracle/product/crs/bin/racgwrap(check) timed out for

ora.dlp34db.vip! (timeout=60)
2010-12-07 03:14:54.776: [ CRSAPP][1494178112] CheckResource error for ora.dlp34db.vip error code = -2
2010-12-07 03:14:54.846: [ CRSEVT][1492076864] CAAMonitorHandler :: 0:Could not join /u01/crs/oracle/product/crs/bin/racgwrap(check)
category: 1234, operation: scls_process_join, loc: childcrash, OS error: 0, other: Abnormal termination of the child


2. Increase DIAGWAIT. This avoids failure in dumping log information prior to Node Eviction by giving sufficient time.
Eventually it helps to troubleshoot the root cause of Node Eviction
Reference
Oracle Metalink Note id # 559365.1

crsctl set css diagwait 13 -force

3. Ensure HUGEPAGE is set appropriately.
Oracle Metalink Note Reference
Shell Script to Calculate Values Recommended Linux HugePages / HugeTLB Configuration [ID 401749.1]
HugePages on 64-bit Linux [ID 361468.1]
HugePages on Linux: What It Is... and What It Is Not... [ID 361323.1]

4. Additional settings to fetch server level core information to diagnose the Server crash due to Oracle Cluster
a. Enable CSSD core by following Oracle Metalink Note 605449.1
b. Configure kdump by following Oracle Metalink Note 452067.1