Cloning Oracle Home in RAC

Cloning of Oracle software is an easy and fast way to achieve standardization across organization where all efforts put on one environment and after testing etc that environment could be used as a source of binaries. A simple tar ball could be shipped to all other servers and then untar on destination environment as a new home or replace the existing oracle home based on availability of space on destination environment.

There could be different situations where we could use this method in a little bit twisted way to achieve desired result. Possible scenarios are:

  • Lost filesystem hosting oracle database software, so either new installation or cloning of software from surviving nodes in case of RAC or some other environment in case of standalone environment.
  • Enterprise wide periodic patching activity where cloning could save a lot of efforts by just building an image and clone it across enterprise.
  • Building new environments during migration databases across of datacenters
  • Node edition in RAC also uses cloning technique

Cloning involves following simple steps:

  • Take a backup of central inventory on target servers ( Optional though recommended)

    cd /u01/app

    tar -zcf oraInventory.tar.gz oraInventory

  • Take a backup of necessary configuration files from $ORACLE_HOME/dbs & $ORACLE_HOME/network/admin folder from target server

    mkdir –p /tmp/backup_config_files/network

    mkdir –p /tmp/backup_config_files/dbs

    cd /u01/app/oracle/product/11.2.0.3/dbhome_1

    cp network/admin/tnsnames.ora ../tmp/backup_config_files/network/.

    cp dbs/* /tmp/backup_config_files/dbs/.

  • Create tarball on source server or use exiting approved tarball from source environment. Change directory (cd) to parent directory of source Oracle Home and use tar -zcf <backup_file>.tar.gz <source home>

    Example:

    cd /u01/app/oracle/product/11.2.0.3/

    tar -zcf SOURCE_HOME.tar.gz SOURCE_HOME

    And ship SOURCE_HOME.tar.gz to target server where cloning would take place.

  • In case you want to reuse the same Oracle Home name and location then you must have to detach exiting oracle home from destination server or else cloning would fail. Stop all instances accessing oracle home if you are going to restore new home on same location/name.

    Detaching Oracle Home from Inventory

    ./runInstaller -silent -detachHome ORACLE_HOME=”/u01/app/oracle/product/11.2.0.3/dbhome_1″ ORACLE_HOME_NAME=”DB_HOME” –local

    If we use it without “-local” option then it would detach oracle home for all nodes in cluster.

    ./runInstaller -silent -detachHome ORACLE_HOME=”/u01/app/oracle/product/11.2.0.3/dbhome_1″ ORACLE_HOME_NAME=”DB_HOME”

    where runInstaller location is $ORACLE_HOME/oui/bin

    Entry in oracle inventory before detaching oracle home

    mask1% cd /u01/app/oraInventory/ContentsXML

    mask1% grep -i DB_HOME inventory.xml

    <HOME NAME=”DB_HOME” LOC=”/u01/app/oracle/product/11.2.0.3/dbhome_1″ TYPE=”O” IDX=”4″>

    Detach oracle home from node 1

    mask1% ./runInstaller -silent -detachHome ORACLE_HOME=”/u01/app/oracle/product/11.2.0.3/dbhome_1″ ORACLE_HOME_NAME=”DB_HOME” -local

    Starting Oracle Universal Installer…

    Checking swap space: must be greater than 500 MB. Actual 24575 MB Passed

    The inventory pointer is located at /etc/oraInst.loc

    The inventory is located at /u01/app/oraInventory

    ‘DetachHome’ was successful.

    Entry in oracle inventory after detaching oracle home

    mask1% cd /u01/app/oraInventory/ContentsXML

    mask1% grep -i DB_HOME inventory.xml

    <HOME NAME=”DB_HOME” LOC=”/u01/app/oracle/product/11.2.0.3/dbhome_1″ TYPE=”O” IDX=”4″ REMOVED=”T”/>

    At this stage when if we try to query oracle inventory from detached oracle home then we would get error:

    mask1% cd /u01/app/oracle/product/11.2.0.3/dbhome_1/OPatch

    mask1% ./opatch lsinventory –all

    Inventory load failed… OPatch cannot load inventory for the given Oracle Home.

    Possible causes are:

    Oracle Home dir. path does not exist in Central Inventory

    Oracle Home is a symbolic link

    Oracle Home inventory is corrupted

    LsInventorySession failed: OracleHomeInventory gets null oracleHomeInfo

    OPatch failed with error code 73

    Which means that our current oracle home has no longer been part of inventory, so we are free to remove the binaries and extract our tarball to same or new location.

    Similarly we have detached home from node 2 as well.

    mask2 % ./runInstaller -silent -detachHome ORACLE_HOME=”/u01/app/oracle/product/11.2.0.3/dbhome_1″ ORACLE_HOME_NAME=”DB_HOME” -local

    Starting Oracle Universal Installer…

    Checking swap space: must be greater than 500 MB. Actual 24575 MB Passed

    The inventory pointer is located at /etc/oraInst.loc

    The inventory is located at /u01/app/oraInventory

    ‘DetachHome’ was successful.

    In this case we are cloning to a new location to reduce down time as we want to change the path as well. In case we would have choice to use the existing location then we could choose to perform cloning in rolling fashion to reduce down time to practically

  • Extract tarball to designated location for Oracle Home. Change directory (cd) to parent directory of destination Oracle Home and use tar -zxf <backup_file>.tar.gz and then rename it to the desired name.

    Example:

    cd /u01/app/oracle/product/11.2.0.3/

    tar -zxf SOURCE_HOME.tar.gz

    mv SOURCE_HOME mask

  • Execute clone.pl command to complete the cloning process. In case of RAC you need to perform this operation on all nodes if using –local option or in one shot for full cluster.

    Executing clone.pl command on mask1 node from oracle user

    mask1% perl /u01/app/oracle/product/11.2.0.3/mask/clone/bin/clone.pl ORACLE_HOME=”/u01/app/oracle/product/11.2.0.3/mask” ORACLE_HOME_NAME=”mask_HOME” ORACLE_BASE=”/u01/app/oracle” ‘-O”CLUSTER_NODES={ mask1,mask2}”‘ ‘-O”LOCAL_NODE=mask1″‘

    ./runInstaller -clone -waitForCompletion “ORACLE_HOME=/u01/app/oracle/product/11.2.0.3/mask” “ORACLE_HOME_NAME=mask_HOME” “ORACLE_BASE=/u01/app/oracle” “CLUSTER_NODES={mask1,mask2}” “LOCAL_NODE=mask1” -silent -noConfig -nowait

    Starting Oracle Universal Installer…

    Checking swap space: must be greater than 500 MB. Actual 24575 MB Passed

    Preparing to launch Oracle Universal Installer from /tmp/OraInstall2013-03-22_11-58-10PM. Please wait …Oracle Universal Installer, Version 11.2.0.3.0 Production

    Copyright (C) 1999, 2011, Oracle. All rights reserved.

    You can find the log of this install session at:

    /u01/app/oraInventory/logs/cloneActions2013-03-22_11-58-10PM.log

    .

    Performing tests to see whether nodes mask1 are available

    ……………………………………………………… 100% Done.

    Installation in progress (Friday, March 22, 2013 11:58:23 PM IST)

    ……………………………………………………………………. 79% Done.

    Install successful

    Linking in progress (Friday, March 22, 2013 11:58:30 PM IST)

    Link successful

    Setup in progress (Friday, March 22, 2013 11:59:02 PM IST)

    Setup successful

    End of install phases.(Friday, March 22, 2013 11:59:25 PM IST)

    WARNING:

    The following configuration scripts need to be executed as the “root” user in each new cluster node. Each script in the list below is followed by a list of nodes.

    /u01/app/oracle/product/11.2.0.3/mask/root.sh #On nodes mask1

    To execute the configuration scripts:

    1. Open a terminal window

    2. Log in as “root”

    3. Run the scripts in each cluster node

    The cloning of mask_HOME was successful.

    Please check ‘/u01/app/oraInventory/logs/cloneActions2013-03-22_11-58-10PM.log’ for more details.

    Executing root.sh from root session on mask1

    [root@mask1 ~]# /u01/app/oracle/product/11.2.0.3/mask/root.sh

    Check /u01/app/oracle/product/11.2.0.3/mask/install/root_mask2.lgk.nmk_2013-03-23_12-02-00.log for the output of root script

    [root@mask1 ~]# cat /u01/app/oracle/product/11.2.0.3/mask/install/root_mask2.lgk.nmk_2013-03-23_12-02-00.log

    Performing root user operation for Oracle 11g

    The following environment variables are set as:

    ORACLE_OWNER= oracle

    ORACLE_HOME= /u01/app/oracle/product/11.2.0.3/mask

    Entries will be added to the /etc/oratab file as needed by

    Database Configuration Assistant when a database is created

    Finished running generic part of root script.

    Now product-specific root actions will be performed.

    Finished product-specific root actions.

    Executing clone.pl command on mask2 node from oracle user

    mask1% perl /u01/app/oracle/product/11.2.0.3/mask/clone/bin/clone.pl ORACLE_HOME=”/u01/app/oracle/product/11.2.0.3/mask” ORACLE_HOME_NAME=”mask_HOME” ORACLE_BASE=”/u01/app/oracle” ‘-O”CLUSTER_NODES={mask1,mask2}”‘ ‘-O”LOCAL_NODE=mask2″‘

    ./runInstaller -clone -waitForCompletion “ORACLE_HOME=/u01/app/oracle/product/11.2.0.3/mask” “ORACLE_HOME_NAME=mask_HOME” “ORACLE_BASE=/u01/app/oracle” “CLUSTER_NODES={mask1,mask2}” “LOCAL_NODE=mask2” -silent -noConfig -nowait

    Starting Oracle Universal Installer…

    Checking swap space: must be greater than 500 MB. Actual 24575 MB Passed

    Preparing to launch Oracle Universal Installer from /tmp/OraInstall2013-03-23_12-05-01AM. Please wait …Oracle Universal Installer, Version 11.2.0.3.0 Production

    Copyright (C) 1999, 2011, Oracle. All rights reserved.

    You can find the log of this install session at:

    /u01/app/oraInventory/logs/cloneActions2013-03-23_12-05-01AM.log

    .

    Performing tests to see whether nodes mask1 are available

    ……………………………………………………… 100% Done.

    Installation in progress (Saturday, March 23, 2013 12:05:23 AM IST)

    ……………………………………………………………………. 79% Done.

    Install successful

    Linking in progress (Saturday, March 23, 2013 12:05:30 AM IST)

    Link successful

    Setup in progress (Saturday, March 23, 2013 12:06:02 AM IST)

    Setup successful

    End of install phases.(Saturday, March 23, 2013 12:06:25 AM IST)

    WARNING:

    The following configuration scripts need to be executed as the “root” user in each new cluster node. Each script in the list below is followed by a list of nodes.

    /u01/app/oracle/product/11.2.0.3/mask/root.sh #On nodes mask2

    To execute the configuration scripts:

    1. Open a terminal window

    2. Log in as “root”

    3. Run the scripts in each cluster node

    The cloning of mask_HOME was successful.

    Please check ‘/u01/app/oraInventory/logs/cloneActions2013-03-23_12-05-01AM.log’ for more details.

    Executing root.sh from root session on mask1

    [root@mask2 ~]# /u01/app/oracle/product/11.2.0.3/mask/root.sh

    Check /u01/app/oracle/product/11.2.0.3/mask/install/root_mask2.lgk.nmk_2013-03-23_01-02-00.log for the output of root script

    [root@mask2 ~]# cat /u01/app/oracle/product/11.2.0.3/mask/install/root_mask2.lgk.nmk_2013-03-23_05-02-00.log

    Performing root user operation for Oracle 11g

    The following environment variables are set as:

    ORACLE_OWNER= oracle

    ORACLE_HOME= /u01/app/oracle/product/11.2.0.3/mask

    Entries will be added to the /etc/oratab file as needed by

    Database Configuration Assistant when a database is created

    Finished running generic part of root script.

    Now product-specific root actions will be performed.

    Finished product-specific root actions.

  • Verify the integrity of inventory by querying opatch lsinventory

    mask1 % opatch lsinventory -all_nodes | egrep -i ‘applied|Patch description|node name’

    Node Name : mask1

    Patch 13632653 : applied on Fri Feb 01 23:54:58 CST 2013

    Patch 12552578 : applied on Fri Feb 01 23:54:52 CST 2013

    Patch 13743357 : applied on Fri Feb 01 23:54:48 CST 2013

    Patch 14164849 : applied on Fri Feb 01 23:54:43 CST 2013

    Patch 12317925 : applied on Fri Feb 01 23:54:40 CST 2013

    Patch 14741727 : applied on Fri Feb 01 23:54:37 CST 2013

    Patch 15843238 : applied on Fri Feb 01 23:54:31 CST 2013

    Patch 14837414 : applied on Fri Feb 01 23:54:25 CST 2013

    Patch 13615767 : applied on Fri Feb 01 23:54:21 CST 2013

    Patch 13902963 : applied on Fri Feb 01 23:54:16 CST 2013

    Patch 14793338 : applied on Fri Feb 01 23:54:11 CST 2013

    Patch 14653598 : applied on Fri Feb 01 23:54:07 CST 2013

    Patch 14757709 : applied on Fri Feb 01 23:54:02 CST 2013

    Patch 13714926 : applied on Fri Feb 01 23:53:56 CST 2013

    Patch 14499293 : applied on Fri Feb 01 23:53:52 CST 2013

    Patch 14226599 : applied on Fri Feb 01 23:53:46 CST 2013

    Patch 14307915 : applied on Fri Feb 01 23:52:58 CST 2013

    Patch description: “QUARTERLY DISKMON PATCH FOR EXADATA (OCT 2012 – 11.2.0.3.11) : (14307915)”

    Patch 14275572 : applied on Fri Feb 01 23:52:31 CST 2013

    Patch description: “QUARTERLY CRS PATCH FOR EXADATA (OCT 2012 – 11.2.0.3.11) : (14275572)”

    Patch 14474780 : applied on Fri Feb 01 23:50:57 CST 2013

    Patch description: “QUARTERLY DATABASE PATCH FOR EXADATA (OCT 2012 – 11.2.0.3.11) : (14474780)”

    Patch 14679292 : applied on Fri Feb 01 23:13:09 CST 2013

    Patch 12646746 : applied on Wed Jun 27 18:28:26 CDT 2012

    Patch 12985184 : applied on Wed Jun 27 18:27:32 CDT 2012

    Patch 14029429 : applied on Wed Jun 27 18:27:07 CDT 2012

    Patch 13365700 : applied on Wed Jun 27 18:26:41 CDT 2012

    Patch 13508115 : applied on Wed Jun 27 18:26:21 CDT 2012

    Patch 12977501 : applied on Wed Jun 27 18:25:06 CDT 2012

    Patch 13404129 : applied on Wed Jun 27 18:24:49 CDT 2012

    Patch 13014128 : applied on Wed Jun 27 18:24:24 CDT 2012

    Patch 14058884 : applied on Wed Jun 27 18:19:31 CDT 2012

    Node Name : mask2

    Patch 13632653 : applied on Sat Feb 02 00:18:10 CST 2013

    Patch 12552578 : applied on Sat Feb 02 00:18:05 CST 2013

    Patch 13743357 : applied on Sat Feb 02 00:18:01 CST 2013

    Patch 14164849 : applied on Sat Feb 02 00:17:57 CST 2013

    Patch 12317925 : applied on Sat Feb 02 00:17:54 CST 2013

    Patch 14741727 : applied on Sat Feb 02 00:17:50 CST 2013

    Patch 15843238 : applied on Sat Feb 02 00:17:45 CST 2013

    Patch 14837414 : applied on Sat Feb 02 00:17:40 CST 2013

    Patch 13615767 : applied on Sat Feb 02 00:17:35 CST 2013

    Patch 13902963 : applied on Sat Feb 02 00:17:30 CST 2013

    Patch 14793338 : applied on Sat Feb 02 00:17:25 CST 2013

    Patch 14653598 : applied on Sat Feb 02 00:17:20 CST 2013

    Patch 14757709 : applied on Sat Feb 02 00:17:15 CST 2013

    Patch 13714926 : applied on Sat Feb 02 00:17:10 CST 2013

    Patch 14499293 : applied on Sat Feb 02 00:17:06 CST 2013

    Patch 14226599 : applied on Sat Feb 02 00:17:01 CST 2013

    Patch 14307915 : applied on Sat Feb 02 00:16:17 CST 2013

    Patch description: “QUARTERLY DISKMON PATCH FOR EXADATA (OCT 2012 – 11.2.0.3.11) : (14307915)”

    Patch 14275572 : applied on Sat Feb 02 00:15:50 CST 2013

    Patch description: “QUARTERLY CRS PATCH FOR EXADATA (OCT 2012 – 11.2.0.3.11) : (14275572)”

    Patch 14474780 : applied on Sat Feb 02 00:14:19 CST 2013

    Patch description: “QUARTERLY DATABASE PATCH FOR EXADATA (OCT 2012 – 11.2.0.3.11) : (14474780)”

    Patch 14679292 : applied on Fri Feb 01 23:24:27 CST 2013

    Patch 12646746 : applied on Wed Jun 27 18:28:26 CDT 2012

    Patch 12985184 : applied on Wed Jun 27 18:27:32 CDT 2012

    Patch 14029429 : applied on Wed Jun 27 18:27:07 CDT 2012

    Patch 13365700 : applied on Wed Jun 27 18:26:41 CDT 2012

    Patch 13508115 : applied on Wed Jun 27 18:26:21 CDT 2012

    Patch 12977501 : applied on Wed Jun 27 18:25:06 CDT 2012

    Patch 13404129 : applied on Wed Jun 27 18:24:49 CDT 2012

    Patch 13014128 : applied on Wed Jun 27 18:24:24 CDT 2012

    Patch 14058884 : applied on Wed Jun 27 18:19:31 CDT 2012

  • Copy (or restore in case cloning done on same patch) the necessary configuration files ( i.e. tnsnames.ora, sqlnet.ora & pfile pointing to spfile etc) backed up in step 2.

    cd /u01/app/oracle/product/11.2.0.3/dbhome_1

    cp network/admin/tnsnames.ora ../ mask/network/admin/.

    cp dbs/* ../mask/dbs/.

  • Stop all database instances if not already stopped in step-4

    mask1% srvctl stop database –d MASK

    mask1 % srvctl status database -d MASK

    Instance MASK1 is not running on node mask1

    Instance MASK2 is not running on node mask2

    Backup alert log file and trim the existing one or move it with _bkp suffix so it would easy to notice alerts and message appeared when db starts with new oracle home.

  • As we cloned Oracle Home on new path so need to modify /etc/oratab and and other local environment profile files, as now onwards no reference with respect to DBs switching to new home should go to old oracle home. This step was not required in case we would have cloned Oracle Home on same path.
  • For RAC or Oracle Restart, need to modify db configuration by srvctl command so services would be taken care by CRS/Oracle Restart.

    mask1% srvctl config database -d MASK| grep -i ‘Oracle Home’

    Oracle home: /u01/app/oracle/product/11.2.0.3/dbhome_1

    mask1% srvctl modify database -d MASK -o /u01/app/oracle/product/11.2.0.3/mask

    mask1% srvctl config database -d MASK| grep -i ‘Oracle Home’

    Oracle home: /u01/app/oracle/product/11.2.0.3/mask

  • Start database from new home.

    mask1% srvctl stop database –d MASK

    mask1 % srvctl status database -d MASK

    Instance MASK1 is not running on node mask1

    Instance MASK2 is not running on node mask2

    In case new home has higher patching level then execute respective post patching scripts.

  • Verify alterlog as oracle home should be listed as soon as instances are started.

    mask1% tail –f alert_MASK1.log | grep -i ORACLE_HOME

    ORACLE_HOME = /u01/app/oracle/product/11.2.0.3/mask

    mask2% tail –f alert_MASK2.log | grep -i ORACLE_HOME

    ORACLE_HOME = /u01/app/oracle/product/11.2.0.3/mask

  • Check alert logs for errors and exception. Best practice to do after every change even though you are sure about success of change, to see if anything unusual is cooking behind the screen.

Now your database is successfully switched to new oracle home.

Advertisements

Share Your Comment

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Pierre blog

Pierre Forstmann Oracle Database blog

flashdba

Oracle databases, storage and the high-performance world of flash memory

Future Veterans

Ramblings about Oracle

Ranjeet Srivastava

Smile! You’re at the best blog ever

Kevin Closson's Blog: Platforms, Databases and Storage

Platform, Database and Storage Topics

Real Life Database / SQL Experiences : An Oracle Blog from Vivek Sharma

Being an Oracle Professional, I like to share all my Real Life Performance Tuning Challenges and Experiences. The Content and Views on this site are my own and not necessarily those of Oracle. While, I write on my real life experiences, the resolutions mentioned are solely mine. Comments / Criticisms are always a welcome.

Frits Hoogland Weblog

IT Technology; Oracle, linux, TCP/IP and other stuff I find interesting

OraStory

Dominic Brooks on Oracle Performance, Tuning, Data Quality & Sensible Design ... (Now with added Sets Appeal)

ASM Support Guy

Just Another Crazy Oracle DBA

Exadata Certification

Just Another Crazy Oracle DBA

Carlos Sierra's Tools and Tips

Tools and Tips for Oracle Performance and SQL Tuning

Sangram keshari's Oracle Blog

The Fusion Middleware Administration & Database Administration Blog

Amit Saraswat

Just Another Crazy Oracle DBA

Oracle Scratchpad

Just another Oracle weblog

The Tom Kyte Blog

Just Another Crazy Oracle DBA

Hemant's Oracle DBA Blog

Just Another Crazy Oracle DBA

Uwe Hesse

about Database Technology

Richard Foote's Oracle Blog

Focusing Specifically On Oracle Indexes, Database Administration and Some Great Music

%d bloggers like this: