RAC Full Configuration step by step using third party Storage Openfiler

Configuration RACNODES
----------*********----------

vi /etc/hosts

# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1               localhost.localdomain localhost
::1             localhost6.localdomain6 localhost6


# Public Network - (eth0)
192.168.1.35     rac1.dba.com       rac1
192.168.1.36     rac2.dba.com       rac2

# Private Interconnect - (eth1)
192.168.0.135   rac1-priv.dba.com  rac1-priv
192.168.0.136   rac2-priv.dba.com  rac2-priv

# Public Virtual IP (VIP) addresses for - (eth0)
192.168.1.202   rac1-vip.dba.com  rac1-vip
192.168.1.203   rac2-vip.dba.com  rac2-vip

# Scan IP
192.168.1.188   cluster-scan

#Openfile Ip
192.168.1.100 san.dba.com san


5.1.2. Edit the -sysctl.conf- of orcl1 with the following values.

vi /etc/sysctl.conf

-------------------

kernel.core_uses_pid = 1
kernel.shmmax=4294967295
kernel.sem=256 32000 100 142
fs.file-max = 6815744
kernel.msgmnb = 65535
kernel.msgmni = 2878
net.core.rmem_default = 262144
net.core.rmem_max =  4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
net.ipv4.tcp_wmem = 262144 262144 262144
net.ipv4.tcp_rmem = 262144 262144 262144

fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500

kernel.shmmni = 4096


5.1.3. Edit the -limits.conf- file  of orcl1 with the following values.

vi /etc/security/limits.conf
----------------------------

hard nofile 65535
soft nofile 4096
hard nproc 16384
soft nproc 2047
grid hard nofile 65536
oracle hard nofile 65536

Run the following command to change the current kernel parameters.
/sbin/sysctl -p


5.1.4. Create groups and users

------------------------------

groupadd -g 501 oinstall
groupadd -g 502 dba
useradd -u 503 -g oinstall -G dba oracle
passwd oracle

5.1.5. Install missing packages

-------------------------------

Caution: If you are getting warning
cd /etc/pki/rpm-gpg/
rpm -import RPM-GPG-KEY

If already rpm installed with warning use --force after -Uvh
example rpm -Uvh --force binutils-2.*

# cd /stage/54
# more run54.txt
Copy the output and paste it.
Ensure all rpms installed with out any warning messages.

-------------------------------------------------------------

Create the directories in which the Oracle software will be installed.

mkdir -p  /u01/app/11.2.0/grid

mkdir -p /u01/app/oracle/product/11.2.0/db_1

chown -R oracle:oinstall /u01

chmod -R 775 /u01/

Disable NTP

-----------

Either configure NTP, or make sure it is not configured so the Oracle Cluster Time Synchronization Service (ctssd) can synchronize the times of the RAC nodes. If you want to deconfigure NTP do the following.

# service ntpd stop
Shutting down ntpd:                                        [  OK  ]

# chkconfig ntpd off

# mv /etc/ntp.conf /etc/ntp.conf.orig

# rm /var/run/ntpd.pid

Login as oracle and set the following in .bash_profile

# Oracle Settings
TMP=/tmp; export TMP
TMPDIR=$TMP; export TMPDIR

ORACLE_HOSTNAME=rac2.dba.com; export ORACLE_HOSTNAME
ORACLE_UNQNAME=sales; export ORACLE_UNQNAME
ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE
GRID_HOME=/u01/app/11.2.0/grid; export GRID_HOME
DB_HOME=$ORACLE_BASE/product/11.2.0/db_1; export DB_HOME
ORACLE_HOME=$DB_HOME; export ORACLE_HOME
ORACLE_TERM=xterm; export ORACLE_TERM
BASE_PATH=/usr/sbin:$PATH; export BASE_PATH
PATH=$ORACLE_HOME/bin:$BASE_PATH; export PATH

LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH

if [ $USER = "oracle" ]; then
  if [ $SHELL = "/bin/ksh" ]; then
    ulimit -p 16384
    ulimit -n 65536
  else
    ulimit -u 16384 -n 65536
  fi
fi

alias grid_env='. /home/oracle/grid_env'
alias db_env='. /home/oracle/db_env'

-------------------------------------------------------------------

Create a file called "/home/oracle/grid_env" with the following contents.  Add the following contents to set grid environment.

ORACLE_SID=+ASM1; export ORACLE_SID
ORACLE_HOME=$GRID_HOME; export ORACLE_HOME
PATH=$ORACLE_HOME/bin:$BASE_PATH; export PATH

LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH

----------------------------------------------------------------------

Create a file called "/home/oracle/db_env" with the following contents.  Add the following contents to set database environment.

PATH=$ORACLE_HOME/bin:$BASE_PATH; export PATH
export ORACLE_UNQNAME=sales
LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH

-----------------------------------------------------------------------

Verify bash profiles:
Once the "/home/oracle/grid_env" has been run, you will be able to switch between environments as follows.

$ . grid_env

$ echo $ORACLE_HOME
/u01/app/11.2.0/grid

$ . db_env

$ echo $ORACLE_HOME
/u01/app/oracle/product/11.2.0/db_1

Enableing the ISCSI-TARGET for Both Nodes.

This is status of LUN after mapping. From systems tab iscsi-target.

Client [iscsi initiator] Configuration:
Here we are using a redhat enterprise linux5 as client. In order to use openfiler target as disk, it has to be set as iscsi initiator. For that we need to install the following package.

#rpm -ivh iscsi-initiator-utils-6.2.0.865-0.8.el5.i386.rpm

# service iscsid start

Turning off network shutdown. Starting iSCSI daemon: [ OK ]
[ OK ]

# chkconfig iscsid on
# chkconfig iscsi on

Searching the iscsi target:

# iscsiadm -m discovery -t sendtargets -p 192.168.0.29
You can also use port number if you want. As follows

# iscsiadm -m discovery -t sendtargets -p 192.168.0.29:3260
192.168.0.27:3260,1 iqn.2006-01.com.openfiler:storage.lap

You will get the scanned result as the line above.

Loging in to the iscsi target:

# iscsiadm -m node -T 192.168.0.27:3260,1 iqn.2006-01.com.openfiler:storage.lap -p 192.168.0.29 -l

-l or --login

#fdisk -l

will list the iscsi targets now.

Example:

[root@server ~]# iscsiadm -m discovery -t sendtargets -p 192.168.0.29
192.168.0.29:3260,1 iqn.2006-01.com.openfiler:storage.lap

[root@server ~]# iscsiadm -m node -T iqn.2006-01.com.openfiler:storage.lap -p 192.168.0.29 -l

fdisk -l -> formating newly added asm disks on RAC1

fdisk /dev/sdb -> n,p,1,w
fdisk /dev/sdc -> n,p,1,w
fdisk /dev/sdd -> n,p,1,w
fdisk /dev/sde -> n,p,1,w

# /etc/init.d/oracleasm configure -i
                 -> oracle
                 -> dba
                 -> y,y

# oracleasm init  -> No messages also fine
# oraclasm status -> check the output it is mounted, that is fine.

/usr/sbin/oracleasm createdisk DISK1 /dev/sdb1
/usr/sbin/oracleasm createdisk DISK2 /dev/sdc1
/usr/sbin/oracleasm createdisk DISK3 /dev/sdd1
/usr/sbin/oracleasm createdisk DISK4 /dev/sde1


# oracleasm listdisks

/**************************************************************************
/*  The following is not required and should not be run  - run by advice by seniors only  */
# ./oracleasm stop
# ./oracleasm start
# ./oracleasm exit
#  oracleasm init  -> should show output as it is mounted
# oracleasm scandisks  -> should show up all the DISK* Instantiating...

**************************************************************************

start RAC1
and
start RAC2

Set network adapters
set hostname in neat
change grid_env,db_env if required
ping both priv and host ips

start grid installation
if you not using DNS SERVER then remove this file.
#mv /etc/resolv.conf /etc/resolv.conf_bkp

Thanks Comments Please.