Search this site


    grid infrastructure notes

    Subscribe to posts

    adding a new Node into the cluster

    posted 11 Aug 2011 07:04 by Alan Nolan-Davies   [ updated 11 Aug 2011 08:04 ]

    Steps to run when adding a new node to the cluster

    in this case I am adding a new HP-UX 11.31 node to a cluster which has HP Serviceguard extenstions - this is because our target hosts already had Serviceguard on so we had to add the HP SGEfRAC extension to our build.

    cleardown excessive logs from current node that will run the addNode, for example $CRS_HOME/rdbms/audit contains many ASM audit logs

    Modify the parameter JRE_MEMORY_OPTIONS to be -mx1024m or even -mx2048m in $CRS_HOME/oui/oraparm.ini
    = this was required when I got the error which matched the Oracle Support Note Addnode.sh Fails With PRKC-1025 [ID 1085893.1]

    Run the following cluvfy on the cluster node to check that the hardware and operating system on the new host to be added to the cluster is ready (where node3 is the name of the new node to add.

    node1/grid$ cluvfy stage -post hwos -n node99

    Performing post-checks for hardware and operating system setup 

    Checking node reachability...

    Node reachability check passed from node "node1"  

    Checking user equivalence...

    User equivalence check passed for user "grid" 

    Checking node connectivity... 

    Checking hosts config file... 

    ERROR:

    PRVF-4190 : Verification of the hosts config file failed 

    Node connectivity passed for subnet "10.10.31.0" with node(s) node3

    TCP connectivity check passed for subnet "10.10.31.0"

    Interfaces found on subnet "10.10.41.0" that are likely candidates for VIP are:

    node3 lan41:10.10.41.207 

    Interfaces found on subnet "10.10.31.0" that are likely candidates for a private interconnect are:

    node3 lan10:10.10.31.57 

    Node connectivity check passed 

    Check for multiple users with UID value 0 passed 

    Post-check for hardware and operating system setup was successful.

    Once this has passed all checks you can run a more indepth check by comparing the new node to an existing node in the cluster.

    node1/grid$ cluvfy comp peer -refnode node1 -n node3 -orainv oinstall -osdba asmdba -verbose

    Verifying peer compatibility

     

    Checking peer compatibility...

     

    Compatibility check: Physical memory [reference node: node1]

      Node Name     Status                    Ref. node status          Comment

      ------------  ------------------------  ------------------------  ----------

      node3      31.9797GB (3.35331E7KB)   31.451GB (3.2978728E7KB)  mismatched

    Physical memory check failed

     

    Compatibility check: Available memory [reference node: node1]

      Node Name     Status                    Ref. node status          Comment

      ------------  ------------------------  ------------------------  ----------

      node3      22.8951GB (2.4007304E7KB)  2.1266GB (2229860.0KB)    mismatched

    Available memory check failed

     

    Compatibility check: Swap space [reference node: node1]

      Node Name     Status                    Ref. node status          Comment

      ------------  ------------------------  ------------------------  ----------

      node3      94.4172GB (9.9003564E7KB)  93.9152GB (9.8477192E7KB)  mismatched

    Swap space check failed

     

    Compatibility check: Free disk space for "/var/tmp/" [reference node: node1]

      Node Name     Status                    Ref. node status          Comment

      ------------  ------------------------  ------------------------  ----------

      node3      3.4796GB (3648672.0KB)    3.2417GB (3399192.0KB)    mismatched

    Free disk space check failed

     

    Compatibility check: User existence for "grid" [reference node: node1]

      Node Name     Status                    Ref. node status          Comment

      ------------  ------------------------  ------------------------  ----------

      node3      grid(338)                 grid(338)                 matched

    User existence for "grid" check passed

     

    Compatibility check: Group existence for "oinstall" [reference node: node1]

      Node Name     Status                    Ref. node status          Comment

      ------------  ------------------------  ------------------------  ----------

      node3      oinstall(200)             oinstall(200)             matched

    Group existence for "oinstall" check passed

     

    Compatibility check: Group existence for "asmdba" [reference node: node1]

      Node Name     Status                    Ref. node status          Comment

      ------------  ------------------------  ------------------------  ----------

      node3      asmdba(204)               asmdba(204)               matched

    Group existence for "asmdba" check passed

     

    Compatibility check: Group membership for "grid" in "oinstall (Primary)" [reference node: node1]

      Node Name     Status                    Ref. node status          Comment

      ------------  ------------------------  ------------------------  ----------

      node3      yes                       yes                       matched

    Group membership for "grid" in "oinstall (Primary)" check passed

     

    Compatibility check: Group membership for "grid" in "asmdba" [reference node: node1]

      Node Name     Status                    Ref. node status          Comment

      ------------  ------------------------  ------------------------  ----------

      node3      yes                       yes                       matched

    Group membership for "grid" in "asmdba" check passed

     

    Compatibility check: Run level [reference node: node1]

      Node Name     Status                    Ref. node status          Comment

      ------------  ------------------------  ------------------------  ----------

      node3      3                         3                         matched

    Run level check passed

     

    Compatibility check: System architecture [reference node: node1]

      Node Name     Status                    Ref. node status          Comment

      ------------  ------------------------  ------------------------  ----------

      node3      ia64                      ia64                      matched

    System architecture check passed

     

    Compatibility check: Kernel version [reference node: node1]

      Node Name     Status                    Ref. node status          Comment

      ------------  ------------------------  ------------------------  ----------

      node3      HP-UX B.11.31             HP-UX B.11.31             matched

    Kernel version check passed

     

    Compatibility check: Kernel param "ksi_alloc_max" [reference node: node1]

      Node Name     Status                    Ref. node status          Comment

      ------------  ------------------------  ------------------------  ----------

      node3      33600                     33600                     matched

    Kernel param "ksi_alloc_max" check passed

     

    Compatibility check: Kernel param "executable_stack" [reference node: node1]

      Node Name     Status                    Ref. node status          Comment

      ------------  ------------------------  ------------------------  ----------

      node3      0                         0                         matched

    Kernel param "executable_stack" check passed

     

    Compatibility check: Kernel param "max_thread_proc" [reference node: node1]

      Node Name     Status                    Ref. node status          Comment

      ------------  ------------------------  ------------------------  ----------

      node3      8000                      8000                      matched

    Kernel param "max_thread_proc" check passed

     

    Compatibility check: Kernel param "maxdsiz" [reference node: node1]

      Node Name     Status                    Ref. node status          Comment

      ------------  ------------------------  ------------------------  ----------

      node3      3221225472                3221225472                matched

    Kernel param "maxdsiz" check passed

     

    Compatibility check: Kernel param "maxdsiz_64bit" [reference node: node1]

      Node Name     Status                    Ref. node status          Comment

      ------------  ------------------------  ------------------------  ----------

      node3      274877906944              274877906944              matched

    Kernel param "maxdsiz_64bit" check passed

     

    Compatibility check: Kernel param "maxssiz" [reference node: node1]

      Node Name     Status                    Ref. node status          Comment

      ------------  ------------------------  ------------------------  ----------

      node3      134217728                 134217728                 matched

    Kernel param "maxssiz" check passed

     

    Compatibility check: Kernel param "maxssiz_64bit" [reference node: node1]

      Node Name     Status                    Ref. node status          Comment

      ------------  ------------------------  ------------------------  ----------

      node3      2147483648                2147483648                matched

    Kernel param "maxssiz_64bit" check passed

     

    Compatibility check: Kernel param "maxuprc" [reference node: node1]

      Node Name     Status                    Ref. node status          Comment

      ------------  ------------------------  ------------------------  ----------

      node3      27000                     27000                     matched

    Kernel param "maxuprc" check passed

     

    Compatibility check: Kernel param "msgmni" [reference node: node1]

      Node Name     Status                    Ref. node status          Comment

      ------------  ------------------------  ------------------------  ----------

      node3      4096                      4096                      matched

    Kernel param "msgmni" check passed

     

    Compatibility check: Kernel param "msgtql" [reference node: node1]

      Node Name     Status                    Ref. node status          Comment

      ------------  ------------------------  ------------------------  ----------

      node3      4096                      4096                      matched

    Kernel param "msgtql" check passed

     

    Compatibility check: Kernel param "ncsize" [reference node: node1]

      Node Name     Status                    Ref. node status          Comment

      ------------  ------------------------  ------------------------  ----------

      node3      42172                     42172                     matched

    Kernel param "ncsize" check passed

     

    Compatibility check: Kernel param "maxfiles_lim" [reference node: node1]

      Node Name     Status                    Ref. node status          Comment

      ------------  ------------------------  ------------------------  ----------

      node3      63488                     63488                     matched

    Kernel param "maxfiles_lim" check passed

     

    Compatibility check: Kernel param "maxfiles" [reference node: node1]

      Node Name     Status                    Ref. node status          Comment

      ------------  ------------------------  ------------------------  ----------

      node3      63488                     63488                     matched

    Kernel param "maxfiles" check passed

     

    Compatibility check: Kernel param "nflocks" [reference node: node1]

      Node Name     Status                    Ref. node status          Comment

      ------------  ------------------------  ------------------------  ----------

      node3      32788                     32788                     matched

    Kernel param "nflocks" check passed

     

    Compatibility check: Kernel param "ninode" [reference node: node1]

      Node Name     Status                    Ref. node status          Comment

      ------------  ------------------------  ------------------------  ----------

      node3      242048                    242048                    matched

    Kernel param "ninode" check passed

     

    Compatibility check: Kernel param "nkthread" [reference node: node1]

      Node Name     Status                    Ref. node status          Comment

      ------------  ------------------------  ------------------------  ----------

      node3      250000                    250000                    matched

    Kernel param "nkthread" check passed

     

    Compatibility check: Kernel param "nproc" [reference node: node1]

      Node Name     Status                    Ref. node status          Comment

      ------------  ------------------------  ------------------------  ----------

      node3      30000                     30000                     matched

    Kernel param "nproc" check passed

     

    Compatibility check: Kernel param "semmni" [reference node: node1]

      Node Name     Status                    Ref. node status          Comment

      ------------  ------------------------  ------------------------  ----------

      node3      8192                      8192                      matched

    Kernel param "semmni" check passed

     

    Compatibility check: Kernel param "semmns" [reference node: node1]

      Node Name     Status                    Ref. node status          Comment

      ------------  ------------------------  ------------------------  ----------

      node3      60000                     60000                     matched

    Kernel param "semmns" check passed

     

    Compatibility check: Kernel param "semmnu" [reference node: node1]

      Node Name     Status                    Ref. node status          Comment

      ------------  ------------------------  ------------------------  ----------

      node3      27000                     27000                     matched

    Kernel param "semmnu" check passed

     

    Compatibility check: Kernel param "semvmx" [reference node: node1]

      Node Name     Status                    Ref. node status          Comment

      ------------  ------------------------  ------------------------  ----------

      node3      32767                     32767                     matched

    Kernel param "semvmx" check passed

     

    Compatibility check: Package existence for "OS-Core" [reference node: node1]

      Node Name     Status                    Ref. node status          Comment

      ------------  ------------------------  ------------------------  ----------

      node3      OS-Core-B.11.31-0         OS-Core-B.11.31-0         matched

    Package existence for "OS-Core" check passed

     

    Compatibility check: Package existence for "ServiceGuard" [reference node: node1]

      Node Name     Status                    Ref. node status          Comment

      ------------  ------------------------  ------------------------  ----------

      node3      ServiceGuard-A.11.20.00-0  ServiceGuard-A.11.20.00-0  matched

    Package existence for "ServiceGuard" check passed

     

    Compatibility check: Package existence for "SG-RAC-Tool" [reference node: node1]

      Node Name     Status                    Ref. node status          Comment

      ------------  ------------------------  ------------------------  ----------

      node3      SG-RAC-Tool-A.11.20.00-0  SG-RAC-Tool-A.11.20.00-0  matched

    Package existence for "SG-RAC-Tool" check passed

     

    Verification of peer compatibility was unsuccessful.

    Checks did not pass for the following node(s):

            node3


    Again all of these checks should then be marked "matched", however the physical and available memory checks never seem to match and so always marks the test as failed.

    But as the difference in memory sizes is hardly worth worrying about, especially the available if the cluster node is in use, I guess we can carry on...

    now you are ready to run the addNode.sh command

    as this install is not using GNS I have to specify the command like so...

    node1/grid$ $CRS_HOME/oui/bin/addNode.sh -silent "CLUSTER_NEW_NODES={node3}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={node3-vip}"

    An orainstRoot.sh script needs to be run on the new host (when instructed by the addNode process)

    node3/root$ cd /u01/app/oraInventory

    node3/root$ ./orainstRoot.sh
    Creating the Oracle inventory pointer file (/var/opt/oracle/oraInst.loc)
    Changing permissions of /u01/app/oraInventory.
    Adding read,write permissions for group.
    Removing read,write,execute permissions for world.

    Changing groupname of /u01/app/oraInventory to oinstall.
    The execution of the script is complete.

    node3/root$ cd /u01/app/grid/11.2.0

    node3/root$ ./root.sh
     

    Installing Oracle11g Grid Infrastructure for a Cluster

    posted 21 Apr 2010 09:33 by Alan Nolan-Davies   [ updated 22 Jul 2011 07:19 ]

    Process flow showing tasks per user

    ASMCA - incredibly useful GUI management with Oracle ASM 11gR2

    posted 25 Feb 2010 04:25 by Alan Nolan-Davies   [ updated 22 Jul 2011 07:21 ]

    Use ASMCA to manage your ASM Disks and ACFS volumes in Oracle11gR2




    $ df -h
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/mapper/VolGroup00-LogVol03
                          7.7G  4.3G  3.0G  59% /
    /dev/mapper/VolGroup00-LogVol02
                          992M   64M  878M   7% /app
    /dev/mapper/VolGroup00-LogVol00
                          2.0G  416M  1.5G  23% /var
    /dev/sdm1              99M   20M   75M  21% /boot
    tmpfs                  32G   17G   16G  52% /dev/shm
    /dev/mapper/VolGroup01-LogVol00
                          9.9G  2.1G  7.4G  22% /app/oracle
    /dev/mapper/VolGroup01-LogVol01
                          9.9G  6.2G  3.2G  67% /app/grid
    /dev/mapper/VolGroup01-LogVol02
                           60G  3.4G   53G   7% /app/oracle/backup
    /dev/asm/ora_home1-181
                           10G  4.6G  5.5G  46% /app/oracle/product/11.2.0/dbhome_1
    /dev/asm/ora_home3-181
                           10G  2.4G  7.7G  24% /app/oracle/product/10.2.0/dbhome_3
    /dev/asm/ora_home4-181
                           10G  2.2G  7.9G  22% /app/oracle/product/10.2.0/dbhome_4
    /dev/asm/ora_home6-181
                           10G  2.2G  7.9G  22% /app/oracle/product/10.2.0/dbhome_6
    /dev/asm/ora_home5-181
                           10G  4.6G  5.5G  46% /app/oracle/product/11.2.0/dbhome_5
    /dev/asm/stage-89      50G   16G   35G  31% /app/oracle/stage
    /dev/asm/ora_home2-181
                           10G  9.1G  999M  91% /app/oracle/product/11.2.0/dbhome_2

    Oracle ASM 11gR2

    posted 19 Feb 2010 04:25 by Alan Nolan-Davies   [ updated 22 Jul 2011 07:51 ]

    Using Oracle Enterprise Manager Grid Control 10.2.0.5 is not able to see the ACFS volumes and mount points that have been introduced with Oracle ASM 11.2.0.1.0
    11gR2 ASM Viewed with 11gR2 DB Console

    11gR2 ASM Viewed with 10g Grid Control

    The 2 images above show that you will need to use 11gR2 DB Console when managing your Oracle ASM 11.2 or use the Oracle Enterprise Manager Grid Control 11g Version which is 11gR2 aware.

    Swap Space requirements

    posted 5 Jan 2010 05:06 by Alan Nolan-Davies   [ updated 22 Jul 2011 07:57 ]

    For Oracle Grid Infrastructure on Linux

    Oracle recommends that you set swap space to 1.5 times the amount of RAM for systems with 2 GB of RAM or less. For systems with 2 GB to 16 GB RAM, use swap space equal to RAM. For systems with more than 16 GB RAM, use 16 GB of RAM for swap space.

    http://download.oracle.com/docs/cd/E11882_01/install.112/e10812/typinstl.htm#CBHHHJJF


    Oracle11g Grid Infrastructure

    posted 21 Sep 2009 06:36 by Alan Nolan-Davies   [ updated 22 Jul 2011 08:01 ]

    Thanks to a very timely release I am able to install Oracle11g Release 2 in the new datacentre that I am currently working on.



    Oracle11g Release 2 now incorporates a Grid Infrastructure Product.

    Oracle Grid Infrastructure 11.2 which incorporates Oracle Clusterware and Oracle ASM in a single Oracle_Home.

    The design is for a Oracle Database Grids in the Data Centre and this move to a Grid Infrastructure installation whocul make things considerably easier.

    1-6 of 6

    Sign in|Report Abuse|Print Page|Remove Access|Powered By Google Sites