grid infrastructure notes

Subscribe to posts

adding a new Node into the cluster

posted 11 Aug 2011, 07:04 by Alan Nolan-Davies   [ updated 11 Aug 2011, 08:04 ]

Steps to run when adding a new node to the cluster

in this case I am adding a new HP-UX 11.31 node to a cluster which has HP Serviceguard extenstions - this is because our target hosts already had Serviceguard on so we had to add the HP SGEfRAC extension to our build.

cleardown excessive logs from current node that will run the addNode, for example $CRS_HOME/rdbms/audit contains many ASM audit logs

Modify the parameter JRE_MEMORY_OPTIONS to be -mx1024m or even -mx2048m in $CRS_HOME/oui/oraparm.ini
= this was required when I got the error which matched the Oracle Support Note Addnode.sh Fails With PRKC-1025 [ID 1085893.1]

Run the following cluvfy on the cluster node to check that the hardware and operating system on the new host to be added to the cluster is ready (where node3 is the name of the new node to add.

node1/grid$ cluvfy stage -post hwos -n node99

Performing post-checks for hardware and operating system setup 

Checking node reachability...

Node reachability check passed from node "node1"  

Checking user equivalence...

User equivalence check passed for user "grid" 

Checking node connectivity... 

Checking hosts config file... 

ERROR:

PRVF-4190 : Verification of the hosts config file failed 

Node connectivity passed for subnet "10.10.31.0" with node(s) node3

TCP connectivity check passed for subnet "10.10.31.0"

Interfaces found on subnet "10.10.41.0" that are likely candidates for VIP are:

node3 lan41:10.10.41.207 

Interfaces found on subnet "10.10.31.0" that are likely candidates for a private interconnect are:

node3 lan10:10.10.31.57 

Node connectivity check passed 

Check for multiple users with UID value 0 passed 

Post-check for hardware and operating system setup was successful.

Once this has passed all checks you can run a more indepth check by comparing the new node to an existing node in the cluster.

node1/grid$ cluvfy comp peer -refnode node1 -n node3 -orainv oinstall -osdba asmdba -verbose

Verifying peer compatibility

 

Checking peer compatibility...

 

Compatibility check: Physical memory [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      31.9797GB (3.35331E7KB)   31.451GB (3.2978728E7KB)  mismatched

Physical memory check failed

 

Compatibility check: Available memory [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      22.8951GB (2.4007304E7KB)  2.1266GB (2229860.0KB)    mismatched

Available memory check failed

 

Compatibility check: Swap space [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      94.4172GB (9.9003564E7KB)  93.9152GB (9.8477192E7KB)  mismatched

Swap space check failed

 

Compatibility check: Free disk space for "/var/tmp/" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      3.4796GB (3648672.0KB)    3.2417GB (3399192.0KB)    mismatched

Free disk space check failed

 

Compatibility check: User existence for "grid" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      grid(338)                 grid(338)                 matched

User existence for "grid" check passed

 

Compatibility check: Group existence for "oinstall" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      oinstall(200)             oinstall(200)             matched

Group existence for "oinstall" check passed

 

Compatibility check: Group existence for "asmdba" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      asmdba(204)               asmdba(204)               matched

Group existence for "asmdba" check passed

 

Compatibility check: Group membership for "grid" in "oinstall (Primary)" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      yes                       yes                       matched

Group membership for "grid" in "oinstall (Primary)" check passed

 

Compatibility check: Group membership for "grid" in "asmdba" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      yes                       yes                       matched

Group membership for "grid" in "asmdba" check passed

 

Compatibility check: Run level [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      3                         3                         matched

Run level check passed

 

Compatibility check: System architecture [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      ia64                      ia64                      matched

System architecture check passed

 

Compatibility check: Kernel version [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      HP-UX B.11.31             HP-UX B.11.31             matched

Kernel version check passed

 

Compatibility check: Kernel param "ksi_alloc_max" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      33600                     33600                     matched

Kernel param "ksi_alloc_max" check passed

 

Compatibility check: Kernel param "executable_stack" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      0                         0                         matched

Kernel param "executable_stack" check passed

 

Compatibility check: Kernel param "max_thread_proc" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      8000                      8000                      matched

Kernel param "max_thread_proc" check passed

 

Compatibility check: Kernel param "maxdsiz" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      3221225472                3221225472                matched

Kernel param "maxdsiz" check passed

 

Compatibility check: Kernel param "maxdsiz_64bit" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      274877906944              274877906944              matched

Kernel param "maxdsiz_64bit" check passed

 

Compatibility check: Kernel param "maxssiz" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      134217728                 134217728                 matched

Kernel param "maxssiz" check passed

 

Compatibility check: Kernel param "maxssiz_64bit" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      2147483648                2147483648                matched

Kernel param "maxssiz_64bit" check passed

 

Compatibility check: Kernel param "maxuprc" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      27000                     27000                     matched

Kernel param "maxuprc" check passed

 

Compatibility check: Kernel param "msgmni" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      4096                      4096                      matched

Kernel param "msgmni" check passed

 

Compatibility check: Kernel param "msgtql" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      4096                      4096                      matched

Kernel param "msgtql" check passed

 

Compatibility check: Kernel param "ncsize" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      42172                     42172                     matched

Kernel param "ncsize" check passed

 

Compatibility check: Kernel param "maxfiles_lim" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      63488                     63488                     matched

Kernel param "maxfiles_lim" check passed

 

Compatibility check: Kernel param "maxfiles" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      63488                     63488                     matched

Kernel param "maxfiles" check passed

 

Compatibility check: Kernel param "nflocks" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      32788                     32788                     matched

Kernel param "nflocks" check passed

 

Compatibility check: Kernel param "ninode" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      242048                    242048                    matched

Kernel param "ninode" check passed

 

Compatibility check: Kernel param "nkthread" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      250000                    250000                    matched

Kernel param "nkthread" check passed

 

Compatibility check: Kernel param "nproc" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      30000                     30000                     matched

Kernel param "nproc" check passed

 

Compatibility check: Kernel param "semmni" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      8192                      8192                      matched

Kernel param "semmni" check passed

 

Compatibility check: Kernel param "semmns" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      60000                     60000                     matched

Kernel param "semmns" check passed

 

Compatibility check: Kernel param "semmnu" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      27000                     27000                     matched

Kernel param "semmnu" check passed

 

Compatibility check: Kernel param "semvmx" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      32767                     32767                     matched

Kernel param "semvmx" check passed

 

Compatibility check: Package existence for "OS-Core" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      OS-Core-B.11.31-0         OS-Core-B.11.31-0         matched

Package existence for "OS-Core" check passed

 

Compatibility check: Package existence for "ServiceGuard" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      ServiceGuard-A.11.20.00-0  ServiceGuard-A.11.20.00-0  matched

Package existence for "ServiceGuard" check passed

 

Compatibility check: Package existence for "SG-RAC-Tool" [reference node: node1]

  Node Name     Status                    Ref. node status          Comment

  ------------  ------------------------  ------------------------  ----------

  node3      SG-RAC-Tool-A.11.20.00-0  SG-RAC-Tool-A.11.20.00-0  matched

Package existence for "SG-RAC-Tool" check passed

 

Verification of peer compatibility was unsuccessful.

Checks did not pass for the following node(s):

        node3


Again all of these checks should then be marked "matched", however the physical and available memory checks never seem to match and so always marks the test as failed.

But as the difference in memory sizes is hardly worth worrying about, especially the available if the cluster node is in use, I guess we can carry on...

now you are ready to run the addNode.sh command

as this install is not using GNS I have to specify the command like so...

node1/grid$ $CRS_HOME/oui/bin/addNode.sh -silent "CLUSTER_NEW_NODES={node3}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={node3-vip}"

An orainstRoot.sh script needs to be run on the new host (when instructed by the addNode process)

node3/root$ cd /u01/app/oraInventory

node3/root$ ./orainstRoot.sh
Creating the Oracle inventory pointer file (/var/opt/oracle/oraInst.loc)
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.

node3/root$ cd /u01/app/grid/11.2.0

node3/root$ ./root.sh
 

Installing Oracle11g Grid Infrastructure for a Cluster

posted 21 Apr 2010, 09:33 by Alan Nolan-Davies   [ updated 22 Jul 2011, 07:19 ]

Process flow showing tasks per user

ASMCA - incredibly useful GUI management with Oracle ASM 11gR2

posted 25 Feb 2010, 04:25 by Alan Nolan-Davies   [ updated 22 Jul 2011, 07:21 ]

Use ASMCA to manage your ASM Disks and ACFS volumes in Oracle11gR2




$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol03
                      7.7G  4.3G  3.0G  59% /
/dev/mapper/VolGroup00-LogVol02
                      992M   64M  878M   7% /app
/dev/mapper/VolGroup00-LogVol00
                      2.0G  416M  1.5G  23% /var
/dev/sdm1              99M   20M   75M  21% /boot
tmpfs                  32G   17G   16G  52% /dev/shm
/dev/mapper/VolGroup01-LogVol00
                      9.9G  2.1G  7.4G  22% /app/oracle
/dev/mapper/VolGroup01-LogVol01
                      9.9G  6.2G  3.2G  67% /app/grid
/dev/mapper/VolGroup01-LogVol02
                       60G  3.4G   53G   7% /app/oracle/backup
/dev/asm/ora_home1-181
                       10G  4.6G  5.5G  46% /app/oracle/product/11.2.0/dbhome_1
/dev/asm/ora_home3-181
                       10G  2.4G  7.7G  24% /app/oracle/product/10.2.0/dbhome_3
/dev/asm/ora_home4-181
                       10G  2.2G  7.9G  22% /app/oracle/product/10.2.0/dbhome_4
/dev/asm/ora_home6-181
                       10G  2.2G  7.9G  22% /app/oracle/product/10.2.0/dbhome_6
/dev/asm/ora_home5-181
                       10G  4.6G  5.5G  46% /app/oracle/product/11.2.0/dbhome_5
/dev/asm/stage-89      50G   16G   35G  31% /app/oracle/stage
/dev/asm/ora_home2-181
                       10G  9.1G  999M  91% /app/oracle/product/11.2.0/dbhome_2

Oracle ASM 11gR2

posted 19 Feb 2010, 04:25 by Alan Nolan-Davies   [ updated 22 Jul 2011, 07:51 ]

Using Oracle Enterprise Manager Grid Control 10.2.0.5 is not able to see the ACFS volumes and mount points that have been introduced with Oracle ASM 11.2.0.1.0
11gR2 ASM Viewed with 11gR2 DB Console

11gR2 ASM Viewed with 10g Grid Control

The 2 images above show that you will need to use 11gR2 DB Console when managing your Oracle ASM 11.2 or use the Oracle Enterprise Manager Grid Control 11g Version which is 11gR2 aware.

Swap Space requirements

posted 5 Jan 2010, 05:06 by Alan Nolan-Davies   [ updated 22 Jul 2011, 07:57 ]

For Oracle Grid Infrastructure on Linux

Oracle recommends that you set swap space to 1.5 times the amount of RAM for systems with 2 GB of RAM or less. For systems with 2 GB to 16 GB RAM, use swap space equal to RAM. For systems with more than 16 GB RAM, use 16 GB of RAM for swap space.

http://download.oracle.com/docs/cd/E11882_01/install.112/e10812/typinstl.htm#CBHHHJJF


Oracle11g Grid Infrastructure

posted 21 Sep 2009, 06:36 by Alan Nolan-Davies   [ updated 22 Jul 2011, 08:01 ]

Thanks to a very timely release I am able to install Oracle11g Release 2 in the new datacentre that I am currently working on.



Oracle11g Release 2 now incorporates a Grid Infrastructure Product.

Oracle Grid Infrastructure 11.2 which incorporates Oracle Clusterware and Oracle ASM in a single Oracle_Home.

The design is for a Oracle Database Grids in the Data Centre and this move to a Grid Infrastructure installation whocul make things considerably easier.

1-6 of 6