Oracle Clinic – Maclean Liu的个人技术博客

Owner: 刘相兵(Maclean Liu), Mail&Gtalk: liu.maclean@gmail.com, Mobile: 13764045638, QQ: 47079569, QQ群: 23549328

11gR2 Grid Infrastructure Redundant Interconnect and ora.cluster_interconnect.haip

作者: Maclean Liu , post on April 7th, 2010 , English Version
【本站文章除注明转载外，均为本站原创编译】
转载请注明：文章转载自: Oracle Clinic – Maclean Liu的个人技术博客 [http://www.oracledatabase12g.com/]
本文标题: 11gR2 Grid Infrastructure Redundant Interconnect and ora.cluster_interconnect.haip
本文永久地址: http://www.oracledatabase12g.com/archives/11gr2-grid-infrastructure-redundant-interconnect-and-ora-cluster_interconnect-haip.html

Applies to:

Oracle Server – Enterprise Edition – Version: 11.2.0.2 and later [Release: 11.2 and later ]
Information in this document applies to any platform.

Purpose

This document is intended to explain what is ora.cluster_interconnect.haip resource in 11gR2 Grid Infrastructure.

Scope and Application

This document is intended for RAC Database Administrators and Oracle Support engineers.

11gR2 Grid Infrastructure Redundant Interconnect and ora.cluster_interconnect.haip

Redundant Interconnect without any 3rd-party IP failover technology (bond, IPMP or similar) is supported natively by Grid Infrastructure starting from 11.2.0.2. Multiple private network adapters can be defined either during the installation phase or afterward using the oifcfg. Oracle Database, CSS, OCR, CRS, CTSS, and EVM components in 11.2.0.2 employ it automatically.

Grid Infrastructure can activate a maximum of four private network adapters at a time even if more are defined. The ora.cluster_interconnect.haip resource will start one to four link local HAIP on private network adapters for interconnect communication for Oracle RAC, Oracle ASM, and Oracle ACFS etc.

Grid automatically picks link local addresses from reserved 169.254.*.* subnet for HAIP, and it will not attempt to use any 169.254.*.* address if it’s already in use for another purpose. With HAIP, by default, interconnect traffic will be load balanced across all active interconnect interfaces, and corresponding HAIP address will be failed over transparently to other adapters if one fails or becomes non-communicative. .

The number of HAIP addresses is decided by how many private network adapters are active when Grid comes up on the first node in the cluster . If there’s only one active private network, Grid will create one; if two, Grid will create two; and if more than two, Grid will create four HAIPs. The number of HAIPs won’t change even if more private network adapters are activated later, a restart of clusterware on all nodes is required for new adapters to become effective.

When Oracle Clusterware is fully up, resource haip should show status of ONLINE:

$ $GRID_HOME/bin/crsctl stat res -t -init
..
ora.cluster_interconnect.haip
1 ONLINE ONLINE racnode1

Case 1: Single Private Network Adapter

If multiple physical network adapters are bonded together at the OS level and presented as a single device name, for example bond0, it’s still considered a single network adapter environment. If only one private network adapter is defined, such as eth1 in the example below, one virtual IP will be created by HAIP. Here is what’s expected when Grid is up and running:

$ $GRID_HOME/bin/oifcfg getif
eth1 10.1.0.128 global cluster_interconnect
eth3 10.1.0.0 global public

$ $GRID_HOME/bin/oifcfg iflist -p -n
eth1 10.1.0.128 PRIVATE 255.255.255.128
eth1 169.254.0.0 UNKNOWN 255.255.0.0
eth3 10.1.0.0 PRIVATE 255.255.255.128

Note: subnet 169.254.0.0 on eth1 is started by resource haip.

ifconfig
..
eth1 Link encap:Ethernet HWaddr 00:16:3E:11:11:22
inet addr:10.1.0.168 Bcast:10.1.0.255 Mask:255.255.255.128
inet6 addr: fe80::216:3eff:fe11:1122/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:6369306 errors:0 dropped:0 overruns:0 frame:0
TX packets:4270790 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3037449975 (2.8 GiB) TX bytes:2705797005 (2.5 GiB)

eth1:1 Link encap:Ethernet HWaddr 00:16:3E:11:22:22
inet addr:169.254.167.163 Bcast:169.254.255.255 Mask:255.255.0.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

Instance alert.log (ASM and database):

Private Interface ‘eth1:1′ configured from GPnP for use as a private interconnect.
[name='eth1:1', type=1, ip=169.254.167.163, mac=00-16-3e-11-11-22, net=169.254.0.0/16, mask=255.255.0.0, use=haip:cluster_interconnect/62]
Public Interface ‘eth3′ configured from GPnP for use as a public interface.
[name='eth3', type=1, ip=10.1.0.68, mac=00-16-3e-11-11-44, net=10.1.0.0/25, mask=255.255.255.128, use=public/1]
..
Shared memory segment for instance monitoring created
Picked latch-free SCN scheme 3
..
Cluster communication is configured to use the following interface(s) for this instance
169.254.167.163

Note: interconnect will use virtual private IP 169.254.167.163 instead of real private IP. For pre-11.2.0.2 instance, by default it will still use the real private IP; to take advantage of the new feature, init.ora parameter cluster_interconnects can be updated each time Grid is restarted .

For 11.2.0.2 and above, v$cluster_interconnects will show haip info:

SQL> select name,ip_address from v$cluster_interconnects;

NAME IP_ADDRESS
————— —————-
eth1:1 169.254.167.163

Case 2: Multiple Private Network Adapters

2.1. Default Status

Here is an example of 3 private networks eth1, eth6 and eth7 when Grid is up and running:

$ $GRID_HOME/bin/oifcfg getif
eth1 10.1.0.128 global cluster_interconnect
eth3 10.1.0.0 global public
eth6 10.11.0.128 global cluster_interconnect
eth7 10.12.0.128 global cluster_interconnect

$ $GRID_HOME/bin/oifcfg iflist -p -n
eth1 10.1.0.128 PRIVATE 255.255.255.128
eth1 169.254.0.0 UNKNOWN 255.255.192.0
eth1 169.254.192.0 UNKNOWN 255.255.192.0
eth3 10.1.0.0 PRIVATE 255.255.255.128
eth6 10.11.0.128 PRIVATE 255.255.255.128
eth6 169.254.64.0 UNKNOWN 255.255.192.0
eth7 10.12.0.128 PRIVATE 255.255.255.128
eth7 169.254.128.0 UNKNOWN 255.255.192.0

Note: resource haip started four virtual private IPs, two on eth1, and one on eth6 and eth7

ifconfig
..
eth1      Link encap:Ethernet HWaddr 00:16:3E:11:11:22
inet addr:10.1.0.168 Bcast:10.1.0.255 Mask:255.255.255.128
inet6 addr: fe80::216:3eff:fe11:1122/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:15176906 errors:0 dropped:0 overruns:0 frame:0
TX packets:10239298 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:7929246238 (7.3 GiB) TX bytes:5768511630 (5.3 GiB)

eth1:1    Link encap:Ethernet HWaddr 00:16:3E:11:11:22
inet addr:169.254.30.98 Bcast:169.254.63.255 Mask:255.255.192.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

eth1:2    Link encap:Ethernet HWaddr 00:16:3E:11:11:22
inet addr:169.254.244.103 Bcast:169.254.255.255 Mask:255.255.192.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

eth6      Link encap:Ethernet HWaddr 00:16:3E:11:11:77
inet addr:10.11.0.188 Bcast:10.11.0.255 Mask:255.255.255.128
inet6 addr: fe80::216:3eff:fe11:1177/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:7068185 errors:0 dropped:0 overruns:0 frame:0
TX packets:595746 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2692567483 (2.5 GiB) TX bytes:382357191 (364.6 MiB)

eth6:1    Link encap:Ethernet HWaddr 00:16:3E:11:11:77
inet addr:169.254.112.250 Bcast:169.254.127.255 Mask:255.255.192.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

eth7      Link encap:Ethernet HWaddr 00:16:3E:11:11:88
inet addr:10.12.0.208 Bcast:10.12.0.255 Mask:255.255.255.128
inet6 addr: fe80::216:3eff:fe11:1188/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:6435829 errors:0 dropped:0 overruns:0 frame:0
TX packets:314780 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2024577502 (1.8 GiB) TX bytes:172461585 (164.4 MiB)

eth7:1    Link encap:Ethernet HWaddr 00:16:3E:11:11:88
inet addr:169.254.178.237 Bcast:169.254.191.255 Mask:255.255.192.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

Instance alert.log (ASM and database):

Private Interface ‘eth1:1′ configured from GPnP for use as a private interconnect.
[name='eth1:1', type=1, ip=169.254.30.98, mac=00-16-3e-11-11-22, net=169.254.0.0/18, mask=255.255.192.0, use=haip:cluster_interconnect/62]
Private Interface ‘eth6:1′ configured from GPnP for use as a private interconnect.
[name='eth6:1', type=1, ip=169.254.112.250, mac=00-16-3e-11-11-77, net=169.254.64.0/18, mask=255.255.192.0, use=haip:cluster_interconnect/62]
Private Interface ‘eth7:1′ configured from GPnP for use as a private interconnect.
[name='eth7:1', type=1, ip=169.254.178.237, mac=00-16-3e-11-11-88, net=169.254.128.0/18, mask=255.255.192.0, use=haip:cluster_interconnect/62]
Private Interface ‘eth1:2′ configured from GPnP for use as a private interconnect.
[name='eth1:2', type=1, ip=169.254.244.103, mac=00-16-3e-11-11-22, net=169.254.192.0/18, mask=255.255.192.0, use=haip:cluster_interconnect/62]
Public Interface ‘eth3′ configured from GPnP for use as a public interface.
[name='eth3', type=1, ip=10.1.0.68, mac=00-16-3e-11-11-44, net=10.1.0.0/25, mask=255.255.255.128, use=public/1]
Picked latch-free SCN scheme 3

..
Cluster communication is configured to use the following interface(s) for this instance
169.254.30.98
169.254.112.250
169.254.178.237
169.254.244.103

Note: interconnect communication will use all four virtual private IPs; in case of network failure, as long as there is one private network adapter functioning, all four IPs will remain active.

2.2. When Private Network Adapter Fails

If one private network adapter fails, in this example eth6, virtual private IP on eth6 will be relocated automatically to a healthy adapter, and it is transparent to instances (ASM or database)

$ $GRID_HOME/bin/oifcfg iflist -p -n
eth1 10.1.0.128 PRIVATE 255.255.255.128
eth1 169.254.0.0 UNKNOWN 255.255.192.0
eth1 169.254.128.0 UNKNOWN 255.255.192.0
eth7 10.12.0.128 PRIVATE 255.255.255.128
eth7 169.254.64.0 UNKNOWN 255.255.192.0
eth7 169.254.192.0 UNKNOWN 255.255.192.0

Note: virtual private IP on eth6 subnet 169.254.64.0 relocated to eth7

ifconfig
..
eth1      Link encap:Ethernet HWaddr 00:16:3E:11:11:22
inet addr:10.1.0.168 Bcast:10.1.0.255 Mask:255.255.255.128
inet6 addr: fe80::216:3eff:fe11:1122/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:15183840 errors:0 dropped:0 overruns:0 frame:0
TX packets:10245071 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:7934311823 (7.3 GiB) TX bytes:5771878414 (5.3 GiB)

eth1:1    Link encap:Ethernet HWaddr 00:16:3E:11:11:22
inet addr:169.254.30.98 Bcast:169.254.63.255 Mask:255.255.192.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

eth1:3    Link encap:Ethernet HWaddr 00:16:3E:11:11:22
inet addr:169.254.178.237 Bcast:169.254.191.255 Mask:255.255.192.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

eth7      Link encap:Ethernet HWaddr 00:16:3E:11:11:88
inet addr:10.12.0.208 Bcast:10.12.0.255 Mask:255.255.255.128
inet6 addr: fe80::216:3eff:fe11:1188/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:6438985 errors:0 dropped:0 overruns:0 frame:0
TX packets:315877 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2026266447 (1.8 GiB) TX bytes:173101641 (165.0 MiB)

eth7:2    Link encap:Ethernet HWaddr 00:16:3E:11:11:88
inet addr:169.254.112.250 Bcast:169.254.127.255 Mask:255.255.192.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

eth7:3    Link encap:Ethernet HWaddr 00:16:3E:11:11:88
inet addr:169.254.244.103 Bcast:169.254.255.255 Mask:255.255.192.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

2.3. When Another Private Network Adapter Fails

If another private network adapter is down, in this example eth1, virtual private IP on it will be relocated automatically to other healthy adapter with no impact on instances (ASM or database)

$ $GRID_HOME/bin/oifcfg iflist -p -n
eth7 10.12.0.128 PRIVATE 255.255.255.128
eth7 169.254.64.0 UNKNOWN 255.255.192.0
eth7 169.254.192.0 UNKNOWN 255.255.192.0
eth7 169.254.0.0 UNKNOWN 255.255.192.0
eth7 169.254.128.0 UNKNOWN 255.255.192.0

ifconfig
..
eth7      Link encap:Ethernet HWaddr 00:16:3E:11:11:88
inet addr:10.12.0.208 Bcast:10.12.0.255 Mask:255.255.255.128
inet6 addr: fe80::216:3eff:fe11:1188/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:6441559 errors:0 dropped:0 overruns:0 frame:0
TX packets:317271 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2027824788 (1.8 GiB) TX bytes:173810658 (165.7 MiB)

eth7:1    Link encap:Ethernet HWaddr 00:16:3E:11:11:88
inet addr:169.254.30.98 Bcast:169.254.63.255 Mask:255.255.192.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

eth7:2    Link encap:Ethernet HWaddr 00:16:3E:11:11:88
inet addr:169.254.112.250 Bcast:169.254.127.255 Mask:255.255.192.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

eth7:3    Link encap:Ethernet HWaddr 00:16:3E:11:11:88
inet addr:169.254.244.103 Bcast:169.254.255.255 Mask:255.255.192.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

eth7:4    Link encap:Ethernet HWaddr 00:16:3E:11:11:88
inet addr:169.254.178.237 Bcast:169.254.191.255 Mask:255.255.192.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

2.4. When Private Network Adapter Restores

If private network adapter eth6 is restored, it will be activated automatically as virtual private IPs will be assigned to it:

$ $GRID_HOME/bin/oifcfg iflist -p -n
..
eth6 10.11.0.128 PRIVATE 255.255.255.128
eth6 169.254.128.0 UNKNOWN 255.255.192.0
eth6 169.254.0.0 UNKNOWN 255.255.192.0
eth7 10.12.0.128 PRIVATE 255.255.255.128
eth7 169.254.64.0 UNKNOWN 255.255.192.0
eth7 169.254.192.0 UNKNOWN 255.255.192.0

ifconfig
..
eth6      Link encap:Ethernet HWaddr 00:16:3E:11:11:77
inet addr:10.11.0.188 Bcast:10.11.0.255 Mask:255.255.255.128
inet6 addr: fe80::216:3eff:fe11:1177/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:398 errors:0 dropped:0 overruns:0 frame:0
TX packets:121 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:185138 (180.7 KiB) TX bytes:56439 (55.1 KiB)

eth6:1    Link encap:Ethernet HWaddr 00:16:3E:11:11:77
inet addr:169.254.178.237 Bcast:169.254.191.255 Mask:255.255.192.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

eth6:2    Link encap:Ethernet HWaddr 00:16:3E:11:11:77
inet addr:169.254.30.98 Bcast:169.254.63.255 Mask:255.255.192.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

eth7      Link encap:Ethernet HWaddr 00:16:3E:11:11:88
inet addr:10.12.0.208 Bcast:10.12.0.255 Mask:255.255.255.128
inet6 addr: fe80::216:3eff:fe11:1188/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:6442552 errors:0 dropped:0 overruns:0 frame:0
TX packets:317983 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2028404133 (1.8 GiB) TX bytes:174103017 (166.0 MiB)

eth7:2    Link encap:Ethernet HWaddr 00:16:3E:11:11:88
inet addr:169.254.112.250 Bcast:169.254.127.255 Mask:255.255.192.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

eth7:3    Link encap:Ethernet HWaddr 00:16:3E:11:11:88
inet addr:169.254.244.103 Bcast:169.254.255.255 Mask:255.255.192.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

HAIP Log File

Resource haip is managed by ohasd.bin, resource log is located in $GRID_HOME/log/<nodename>/ohasd/ohasd.log and $GRID_HOME/log/<nodename>/agent/ohasd/orarootagent_root/orarootagent_root.log

L1. Log Sample When Private Network Adapter Fails

In a multiple private network adapter environment, if one of the adapters fails:

ohasd.log

2010-09-24 09:10:00.891: [GIPCHGEN][1083025728]gipchaInterfaceFail: marking interface failing 0x2aaab0269a10 { host ”, haName ‘CLSFRAME_a2b2′, local (nil), ip ’10.11.0.188′, subnet ’10.11.0.128′, mask ’255.255.255.128′, numRef 0, numFail 0, flags 0x4d }
2010-09-24 09:10:00.902: [GIPCHGEN][1138145600]gipchaInterfaceDisable: disabling interface 0x2aaab0269a10 { host ”, haName ‘CLSFRAME_a2b2′, local (nil), ip ’10.11.0.188′, subnet ’10.11.0.128′, mask ’255.255.255.128′, numRef 0, numFail 0, flags 0x1cd }
2010-09-24 09:10:00.902: [GIPCHDEM][1138145600]gipchaWorkerCleanInterface: performing cleanup of disabled interface 0x2aaab0269a10 { host ”, haName ‘CLSFRAME_a2b2′, local (nil), ip ’10.11.0.188′, subnet ’10.11.0.128′, mask ’255.255.255.128′, numRef 0, numFail 0, flags 0x1ed }

orarootagent_root.log

2010-09-24 09:09:57.708: [ USRTHRD][1129138496] {0:0:2} failed to receive ARP request
2010-09-24 09:09:57.708: [ USRTHRD][1129138496] {0:0:2} Assigned IP 169.254.112.250 no longer valid on inf eth6
2010-09-24 09:09:57.708: [ USRTHRD][1129138496] {0:0:2} VipActions::startIp {
2010-09-24 09:09:57.708: [ USRTHRD][1129138496] {0:0:2} Adding 169.254.112.250 on eth6:1
2010-09-24 09:09:57.719: [ USRTHRD][1129138496] {0:0:2} VipActions::startIp }
2010-09-24 09:09:57.719: [ USRTHRD][1129138496] {0:0:2} Reassigned IP: 169.254.112.250 on interface eth6
2010-09-24 09:09:58.013: [ USRTHRD][1082325312] {0:0:2} HAIP: Updating member info HAIP1;10.11.0.128#0;10.11.0.128#1
2010-09-24 09:09:58.015: [ USRTHRD][1082325312] {0:0:2} HAIP: Moving ip ’169.254.112.250′ from inf ‘eth6′ to inf ‘eth7′
2010-09-24 09:09:58.015: [ USRTHRD][1082325312] {0:0:2} pausing thread
2010-09-24 09:09:58.015: [ USRTHRD][1082325312] {0:0:2} posting thread
2010-09-24 09:09:58.016: [ USRTHRD][1082325312] {0:0:2} Thread:[NetHAWork]start {
2010-09-24 09:09:58.016: [ USRTHRD][1082325312] {0:0:2} Thread:[NetHAWork]start }
2010-09-24 09:09:58.016: [ USRTHRD][1082325312] {0:0:2} HAIP: Moving ip ’169.254.244.103′ from inf ‘eth1′ to inf ‘eth7′
2010-09-24 09:09:58.016: [ USRTHRD][1082325312] {0:0:2} pausing thread
2010-09-24 09:09:58.016: [ USRTHRD][1082325312] {0:0:2} posting thread
2010-09-24 09:09:58.016: [ USRTHRD][1082325312] {0:0:2} Thread:[NetHAWork]start {
2010-09-24 09:09:58.016: [ USRTHRD][1082325312] {0:0:2} Thread:[NetHAWork]start }
2010-09-24 09:09:58.016: [ USRTHRD][1082325312] {0:0:2} HAIP: Moving ip ’169.254.178.237′ from inf ‘eth7′ to inf ‘eth1′
2010-09-24 09:09:58.016: [ USRTHRD][1082325312] {0:0:2} pausing thread
2010-09-24 09:09:58.016: [ USRTHRD][1082325312] {0:0:2} posting thread
2010-09-24 09:09:58.017: [ USRTHRD][1082325312] {0:0:2} Thread:[NetHAWork]start {
2010-09-24 09:09:58.017: [ USRTHRD][1116531008] {0:0:2} [NetHAWork] thread started
2010-09-24 09:09:58.017: [ USRTHRD][1116531008] {0:0:2} Arp::sCreateSocket {
2010-09-24 09:09:58.017: [ USRTHRD][1093232960] {0:0:2} [NetHAWork] thread started
2010-09-24 09:09:58.017: [ USRTHRD][1093232960] {0:0:2} Arp::sCreateSocket {
2010-09-24 09:09:58.017: [ USRTHRD][1082325312] {0:0:2} Thread:[NetHAWork]start }
2010-09-24 09:09:58.018: [ USRTHRD][1143847232] {0:0:2} [NetHAWork] thread started
2010-09-24 09:09:58.018: [ USRTHRD][1143847232] {0:0:2} Arp::sCreateSocket {
2010-09-24 09:09:58.034: [ USRTHRD][1116531008] {0:0:2} Arp::sCreateSocket }
2010-09-24 09:09:58.034: [ USRTHRD][1116531008] {0:0:2} Starting Probe for ip 169.254.112.250
2010-09-24 09:09:58.034: [ USRTHRD][1116531008] {0:0:2} Transitioning to Probe State
2010-09-24 09:09:58.034: [ USRTHRD][1093232960] {0:0:2} Arp::sCreateSocket }
2010-09-24 09:09:58.035: [ USRTHRD][1093232960] {0:0:2} Starting Probe for ip 169.254.244.103
2010-09-24 09:09:58.035: [ USRTHRD][1093232960] {0:0:2} Transitioning to Probe State
2010-09-24 09:09:58.050: [ USRTHRD][1143847232] {0:0:2} Arp::sCreateSocket }
2010-09-24 09:09:58.050: [ USRTHRD][1143847232] {0:0:2} Starting Probe for ip 169.254.178.237
2010-09-24 09:09:58.050: [ USRTHRD][1143847232] {0:0:2} Transitioning to Probe State
2010-09-24 09:09:58.231: [ USRTHRD][1093232960] {0:0:2} Arp::sProbe {
2010-09-24 09:09:58.231: [ USRTHRD][1093232960] {0:0:2} Arp::sSend: sending type 1
2010-09-24 09:09:58.231: [ USRTHRD][1093232960] {0:0:2} Arp::sProbe }

2010-09-24 09:10:04.879: [ USRTHRD][1116531008] {0:0:2} Arp::sAnnounce {
2010-09-24 09:10:04.879: [ USRTHRD][1116531008] {0:0:2} Arp::sSend: sending type 1
2010-09-24 09:10:04.879: [ USRTHRD][1116531008] {0:0:2} Arp::sAnnounce }
2010-09-24 09:10:04.879: [ USRTHRD][1116531008] {0:0:2} Transitioning to Defend State
2010-09-24 09:10:04.879: [ USRTHRD][1116531008] {0:0:2} VipActions::startIp {
2010-09-24 09:10:04.879: [ USRTHRD][1116531008] {0:0:2} Adding 169.254.112.250 on eth7:2
2010-09-24 09:10:04.880: [ USRTHRD][1116531008] {0:0:2} VipActions::startIp }
2010-09-24 09:10:04.880: [ USRTHRD][1116531008] {0:0:2} Assigned IP: 169.254.112.250 on interface eth7

2010-09-24 09:10:05.150: [ USRTHRD][1143847232] {0:0:2} Arp::sAnnounce {
2010-09-24 09:10:05.150: [ USRTHRD][1143847232] {0:0:2} Arp::sSend: sending type 1
2010-09-24 09:10:05.150: [ USRTHRD][1143847232] {0:0:2} Arp::sAnnounce }
2010-09-24 09:10:05.150: [ USRTHRD][1143847232] {0:0:2} Transitioning to Defend State
2010-09-24 09:10:05.150: [ USRTHRD][1143847232] {0:0:2} VipActions::startIp {
2010-09-24 09:10:05.151: [ USRTHRD][1143847232] {0:0:2} Adding 169.254.178.237 on eth1:3
2010-09-24 09:10:05.151: [ USRTHRD][1143847232] {0:0:2} VipActions::startIp }
2010-09-24 09:10:05.151: [ USRTHRD][1143847232] {0:0:2} Assigned IP: 169.254.178.237 on interface eth1
2010-09-24 09:10:05.470: [ USRTHRD][1093232960] {0:0:2} Arp::sAnnounce {
2010-09-24 09:10:05.470: [ USRTHRD][1093232960] {0:0:2} Arp::sSend: sending type 1
2010-09-24 09:10:05.470: [ USRTHRD][1093232960] {0:0:2} Arp::sAnnounce }
2010-09-24 09:10:05.470: [ USRTHRD][1093232960] {0:0:2} Transitioning to Defend State
2010-09-24 09:10:05.470: [ USRTHRD][1093232960] {0:0:2} VipActions::startIp {
2010-09-24 09:10:05.471: [ USRTHRD][1093232960] {0:0:2} Adding 169.254.244.103 on eth7:3
2010-09-24 09:10:05.471: [ USRTHRD][1093232960] {0:0:2} VipActions::startIp }
2010-09-24 09:10:05.471: [ USRTHRD][1093232960] {0:0:2} Assigned IP: 169.254.244.103 on interface eth7
2010-09-24 09:10:06.047: [ USRTHRD][1082325312] {0:0:2} Thread:[NetHAWork]stop {
2010-09-24 09:10:06.282: [ USRTHRD][1129138496] {0:0:2} [NetHAWork] thread stopping
2010-09-24 09:10:06.282: [ USRTHRD][1129138496] {0:0:2} Thread:[NetHAWork]isRunning is reset to false here
2010-09-24 09:10:06.282: [ USRTHRD][1082325312] {0:0:2} Thread:[NetHAWork]stop }
2010-09-24 09:10:06.282: [ USRTHRD][1082325312] {0:0:2} VipActions::stopIp {
2010-09-24 09:10:06.282: [ USRTHRD][1082325312] {0:0:2} NetInterface::sStopIp {
2010-09-24 09:10:06.282: [ USRTHRD][1082325312] {0:0:2} Stopping ip ’169.254.112.250′, inf ‘eth6′, mask ’10.11.0.128′
2010-09-24 09:10:06.288: [ USRTHRD][1082325312] {0:0:2} NetInterface::sStopIp }
2010-09-24 09:10:06.288: [ USRTHRD][1082325312] {0:0:2} VipActions::stopIp }
2010-09-24 09:10:06.288: [ USRTHRD][1082325312] {0:0:2} Thread:[NetHAWork]stop {
2010-09-24 09:10:06.298: [ USRTHRD][1131239744] {0:0:2} [NetHAWork] thread stopping
2010-09-24 09:10:06.298: [ USRTHRD][1131239744] {0:0:2} Thread:[NetHAWork]isRunning is reset to false here
2010-09-24 09:10:06.298: [ USRTHRD][1082325312] {0:0:2} Thread:[NetHAWork]stop }
2010-09-24 09:10:06.298: [ USRTHRD][1082325312] {0:0:2} VipActions::stopIp {

2010-09-24 09:10:06.298: [ USRTHRD][1082325312] {0:0:2} NetInterface::sStopIp {
2010-09-24 09:10:06.298: [ USRTHRD][1082325312] {0:0:2} Stopping ip ’169.254.178.237′, inf ‘eth7′, mask ’10.12.0.128′
2010-09-24 09:10:06.299: [ USRTHRD][1082325312] {0:0:2} NetInterface::sStopIp }
2010-09-24 09:10:06.299: [ USRTHRD][1082325312] {0:0:2} VipActions::stopIp }
2010-09-24 09:10:06.299: [ USRTHRD][1082325312] {0:0:2} Thread:[NetHAWork]stop {
2010-09-24 09:10:06.802: [ USRTHRD][1133340992] {0:0:2} [NetHAWork] thread stopping
2010-09-24 09:10:06.802: [ USRTHRD][1133340992] {0:0:2} Thread:[NetHAWork]isRunning is reset to false here
2010-09-24 09:10:06.802: [ USRTHRD][1082325312] {0:0:2} Thread:[NetHAWork]stop }
2010-09-24 09:10:06.802: [ USRTHRD][1082325312] {0:0:2} VipActions::stopIp {
2010-09-24 09:10:06.802: [ USRTHRD][1082325312] {0:0:2} NetInterface::sStopIp {
2010-09-24 09:10:06.802: [ USRTHRD][1082325312] {0:0:2} Stopping ip ’169.254.244.103′, inf ‘eth1′, mask ’10.1.0.128′
2010-09-24 09:10:06.802: [ USRTHRD][1082325312] {0:0:2} NetInterface::sStopIp }
2010-09-24 09:10:06.802: [ USRTHRD][1082325312] {0:0:2} VipActions::stopIp }
2010-09-24 09:10:06.803: [ USRTHRD][1082325312] {0:0:2} USING HAIP[ 0 ]: eth7 – 169.254.112.250
2010-09-24 09:10:06.803: [ USRTHRD][1082325312] {0:0:2} USING HAIP[ 1 ]: eth1 – 169.254.178.237
2010-09-24 09:10:06.803: [ USRTHRD][1082325312] {0:0:2} USING HAIP[ 2 ]: eth7 – 169.254.244.103
2010-09-24 09:10:06.803: [ USRTHRD][1082325312] {0:0:2} USING HAIP[ 3 ]: eth1 – 169.254.30.98

Note: from above, even only NIC eth6 failed, there could be multiple virtual private IP movement among surviving NICs

ocssd.log

2010-09-24 09:09:58.314: [ GIPCNET][1089964352] gipcmodNetworkProcessSend: [network] failed send attempt endp 0xe1b9150 [0000000000000399] { gipcEndpoint : localAddr ‘udp://10.11.0.188:60169′, remoteAddr ”, numPend 5, numReady 1, numDone 0, numDead 0, numTransfer 0, objFlags 0×0, pidPeer 0, flags 0×2, usrFlags 0×4000 }, req 0x2aaab00117f0 [00000000004b0cae] { gipcSendRequest : addr ‘udp://10.11.0.189:41486′, data 0x2aaab0050be8, len 80, olen 0, parentEndp 0xe1b9150, ret gipcretEndpointNotAvailable (40), objFlags 0×0, reqFlags 0×2 }
2010-09-24 09:09:58.314: [ GIPCNET][1089964352] gipcmodNetworkProcessSend: slos op : sgipcnValidateSocket
2010-09-24 09:09:58.314: [ GIPCNET][1089964352] gipcmodNetworkProcessSend: slos dep : Invalid argument (22)
2010-09-24 09:09:58.314: [ GIPCNET][1089964352] gipcmodNetworkProcessSend: slos loc : address not
2010-09-24 09:09:58.314: [ GIPCNET][1089964352] gipcmodNetworkProcessSend: slos info: addr ’10.11.0.188:60169′, len 80, buf 0x2aaab0050be8, cookie 0x2aaab00117f0
2010-09-24 09:09:58.314: [GIPCXCPT][1089964352] gipcInternalSendSync: failed sync request, ret gipcretEndpointNotAvailable (40)
2010-09-24 09:09:58.314: [GIPCXCPT][1089964352] gipcSendSyncF [gipchaLowerInternalSend : gipchaLower.c : 755]: EXCEPTION[ ret gipcretEndpointNotAvailable (40) ] failed to send on endp 0xe1b9150 [0000000000000399] { gipcEndpoint : localAddr ‘udp://10.11.0.188:60169′, remoteAddr ”, numPend 5, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0×0, pidPeer 0, flags 0×2, usrFlags 0×4000 }, addr 0xe4e6d10 [00000000000007ed] { gipcAddress : name ‘udp://10.11.0.189:41486′, objFlags 0×0, addrFlags 0×1 }, buf 0x2aaab0050be8, len 80, flags 0×0
2010-09-24 09:09:58.314: [GIPCHGEN][1089964352] gipchaInterfaceFail: marking interface failing 0xe2bd5f0 { host ‘racnode2′, haName ‘CSS_a2b2′, local 0x2aaaac2098e0, ip ’10.11.0.189:41486′, subnet ’10.11.0.128′, mask ’255.255.255.128′, numRef 0, numFail 0, flags 0×6 }
2010-09-24 09:09:58.314: [GIPCHALO][1089964352] gipchaLowerInternalSend: failed to initiate send on interface 0xe2bd5f0 { host ‘racnode2′, haName ‘CSS_a2b2′, local 0x2aaaac2098e0, ip ’10.11.0.189:41486′, subnet ’10.11.0.128′, mask ’255.255.255.128′, numRef 0, numFail 0, flags 0×86 }, hctx 0xde81d10 [0000000000000010] { gipchaContext : host ‘racnode1′, name ‘CSS_a2b2′, luid ’4f06f2aa-00000000′, numNode 1, numInf 3, usrFlags 0×0, flags 0×7 }
2010-09-24 09:09:58.326: [GIPCHGEN][1089964352] gipchaInterfaceDisable: disabling interface 0x2aaaac2098e0 { host ”, haName ‘CSS_a2b2′, local (nil), ip ’10.11.0.188′, subnet ’10.11.0.128′, mask ’255.255.255.128′, numRef 0, numFail 1, flags 0x14d }
2010-09-24 09:09:58.326: [GIPCHGEN][1089964352] gipchaInterfaceDisable: disabling interface 0xe2bd5f0 { host ‘racnode2′, haName ‘CSS_a2b2′, local 0x2aaaac2098e0, ip ’10.11.0.189:41486′, subnet ’10.11.0.128′, mask ’255.255.255.128′, numRef 0, numFail 0, flags 0×86 }
2010-09-24 09:09:58.327: [GIPCHALO][1089964352] gipchaLowerCleanInterfaces: performing cleanup of disabled interface 0xe2bd5f0 { host ‘racnode2′, haName ‘CSS_a2b2′, local 0x2aaaac2098e0, ip ’10.11.0.189:41486′, subnet ’10.11.0.128′, mask ’255.255.255.128′, numRef 0, numFail 0, flags 0xa6 }
2010-09-24 09:09:58.327: [GIPCHGEN][1089964352] gipchaInterfaceReset: resetting interface 0xe2bd5f0 { host ‘racnode2′, haName ‘CSS_a2b2′, local 0x2aaaac2098e0, ip ’10.11.0.189:41486′, subnet ’10.11.0.128′, mask ’255.255.255.128′, numRef 0, numFail 0, flags 0xa6 }
2010-09-24 09:09:58.338: [GIPCHDEM][1089964352] gipchaWorkerCleanInterface: performing cleanup of disabled interface 0x2aaaac2098e0 { host ”, haName ‘CSS_a2b2′, local (nil), ip ’10.11.0.188′, subnet ’10.11.0.128′, mask ’255.255.255.128′, numRef 0, numFail 0, flags 0x16d }
2010-09-24 09:09:58.338: [GIPCHTHR][1089964352] gipchaWorkerUpdateInterface: created remote interface for node ‘racnode2′, haName ‘CSS_a2b2′, inf ‘udp://10.11.0.189:41486′
2010-09-24 09:09:58.338: [GIPCHGEN][1089964352] gipchaWorkerAttachInterface: Interface attached inf 0xe2bd5f0 { host ‘racnode2′, haName ‘CSS_a2b2′, local 0x2aaaac2014f0, ip ’10.11.0.189:41486′, subnet ’10.11.0.128′, mask ’255.255.255.128′, numRef 0, numFail 0, flags 0×6 }
2010-09-24 09:10:00.454: [ CSSD][1108904256]clssnmSendingThread: sending status msg to all nodes

Note: from above, ocssd.bin won’t fail as long as there’s at least one private network adapter is working

L2. Log Sample When Private Network Adapter Restores

In a multiple private network adapter environment, if one of the failed adapters becomes restored:

ohasd.log

2010-09-24 09:14:30.962: [GIPCHGEN][1083025728]gipchaNodeAddInterface: adding interface information for inf 0x2aaaac1a53d0 { host ”, haName ‘CLSFRAME_a2b2′, local (nil), ip ’10.11.0.188′, subnet ’10.11.0.128′, mask ’255.255.255.128′, numRef 0, numFail 0, flags 0×41 }
2010-09-24 09:14:30.972: [GIPCHTHR][1138145600]gipchaWorkerUpdateInterface: created local bootstrap interface for node ‘eyrac1f’, haName ‘CLSFRAME_a2b2′, inf ‘mcast://230.0.1.0:42424/10.11.0.188′
2010-09-24 09:14:30.972: [GIPCHTHR][1138145600]gipchaWorkerUpdateInterface: created local interface for node ‘eyrac1f’, haName ‘CLSFRAME_a2b2′, inf ’10.11.0.188:13235′

ocssd.log

2010-09-24 09:14:30.961: [GIPCHGEN][1091541312] gipchaNodeAddInterface: adding interface information for inf 0x2aaab005af00 { host ”, haName ‘CSS_a2b2′, local (nil), ip ’10.11.0.188′, subnet ’10.11.0.128′, mask ’255.255.255.128′, numRef 0, numFail 0, flags 0×41 }
2010-09-24 09:14:30.972: [GIPCHTHR][1089964352] gipchaWorkerUpdateInterface: created local bootstrap interface for node ‘racnode1′, haName ‘CSS_a2b2′, inf ‘mcast://230.0.1.0:42424/10.11.0.188′
2010-09-24 09:14:30.972: [GIPCHTHR][1089964352] gipchaWorkerUpdateInterface: created local interface for node ‘racnode1′, haName ‘CSS_a2b2′, inf ’10.11.0.188:10884′
2010-09-24 09:14:30.972: [GIPCHGEN][1089964352] gipchaNodeAddInterface: adding interface information for inf 0x2aaab0035490 { host ‘racnode2′, haName ‘CSS_a2b2′, local (nil), ip ’10.21.0.208′, subnet ’10.12.0.128′, mask ’255.255.255.128′, numRef 0, numFail 0, flags 0×42 }
2010-09-24 09:14:30.972: [GIPCHGEN][1089964352] gipchaNodeAddInterface: adding interface information for inf 0x2aaab00355c0 { host ‘racnode2′, haName ‘CSS_a2b2′, local (nil), ip ’10.11.0.188′, subnet ’10.11.0.128′, mask ’255.255.255.128′, numRef 0, numFail 0, flags 0×42 }
2010-09-24 09:14:30.972: [GIPCHTHR][1089964352] gipchaWorkerUpdateInterface: created remote interface for node ‘racnode2′, haName ‘CSS_a2b2′, inf ‘mcast://230.0.1.0:42424/10.12.0.208′
2010-09-24 09:14:30.972: [GIPCHGEN][1089964352] gipchaWorkerAttachInterface: Interface attached inf 0x2aaab0035490 { host ‘racnode2′, haName ‘CSS_a2b2′, local 0x2aaab005af00, ip ’10.12.0.208′, subnet ’10.12.0.128′, mask ’255.255.255.128′, numRef 0, numFail 0, flags 0×46 }
2010-09-24 09:14:30.972: [GIPCHTHR][1089964352] gipchaWorkerUpdateInterface: created remote interface for node ‘racnode2′, haName ‘CSS_a2b2′, inf ‘mcast://230.0.1.0:42424/10.11.0.188′
2010-09-24 09:14:30.972: [GIPCHGEN][1089964352] gipchaWorkerAttachInterface: Interface attached inf 0x2aaab00355c0 { host ‘racnode2′, haName ‘CSS_a2b2′, local 0x2aaab005af00, ip ’10.11.0.188′, subnet ’10.11.0.128′, mask ’255.255.255.128′, numRef 0, numFail 0, flags 0×46 }
2010-09-24 09:14:31.437: [GIPCHGEN][1089964352] gipchaInterfaceDisable: disabling interface 0x2aaab00355c0 { host ‘racnode2′, haName ‘CSS_a2b2′, local 0x2aaab005af00, ip ’10.11.0.188′, subnet ’10.11.0.128′, mask ’255.255.255.128′, numRef 0, numFail 0, flags 0×46 }
2010-09-24 09:14:31.437: [GIPCHALO][1089964352] gipchaLowerCleanInterfaces: performing cleanup of disabled interface 0x2aaab00355c0 { host ‘racnode2′, haName ‘CSS_a2b2′, local 0x2aaab005af00, ip ’10.11.0.188′, subnet ’10.11.0.128′, mask ’255.255.255.128′, numRef 0, numFail 0, flags 0×66 }
2010-09-24 09:14:31.446: [GIPCHGEN][1089964352] gipchaInterfaceDisable: disabling interface 0x2aaab0035490 { host ‘racnode2′, haName ‘CSS_a2b2′, local 0x2aaab005af00, ip ’10.12.0.208′, subnet ’10.12.0.128′, mask ’255.255.255.128′, numRef 0, numFail 0, flags 0×46 }
2010-09-24 09:14:31.446: [GIPCHALO][1089964352] gipchaLowerCleanInterfaces: performing cleanup of disabled interface 0x2aaab0035490 { host ‘racnode2′, haName ‘CSS_a2b2′, local 0x2aaab005af00, ip ’10.12.0.208′, subnet ’10.12.0.128′, mask ’255.255.255.128′, numRef 0, numFail 0, flags 0×66 }

Miscellaneous

1. If Sun Cluster exists, HAIP fature is disabled in 11.2.0.2

2. The feature does not exist in Windows 11.2.0.2

@doc bug 10155588

Known Issues

Bug 10370797

Issue: HAIP fails to start on AIX

Fixed in: 11.2.0.3, affects AIX only

Symptom:

Output of root script:

CRS-2672: Attempting to start ‘ora.cluster_interconnect.haip’ on ‘racnode1′
CRS-5017: The resource action “ora.cluster_interconnect.haip start” encountered the following error:
Start action for HAIP aborted
CRS-2674: Start of ‘ora.cluster_interconnect.haip’ on ‘racnode1′ failed

$GRID_HOME/log/<hostname>/agent/ohasd/orarootagent_root/orarootagent_root.log

2010-12-04 17:19:54.893: [ USRTHRD][2084] {0:3:37} failed to create arp
2010-12-04 17:19:54.893: [ USRTHRD][2084] {0:3:37} (null) category: -2, operation: ioctl, loc: bpfopen:2,os, OS error: 14, other:
2010-12-04 17:19:54.992: [ USRTHRD][2084] {0:3:37} Arp::sCreateSocket {
2010-12-04 17:19:54.992: [ USRTHRD][2084] {0:3:37} failed to create arp

Bug 10332426

Issue: HAIP fails to start while running rootupgrade.sh

Fixed in: Configuration issue

Symptom:

Output of root script:

CRS-2672: Attempting to start ‘ora.cluster_interconnect.haip’ on ‘racnode1′
CRS-5017: The resource action “ora.cluster_interconnect.haip start”
encountered the following error:
Start action for HAIP aborted
CRS-2674: Start of ‘ora.cluster_interconnect.haip’ on ‘racnode1′ failed

$GRID_HOME/log/<hostname>/gipcd/gipcd.log

2010-12-12 09:41:35.201: [ CLSINET][1088543040] Returning NETDATA: 0 interfaces
2010-12-12 09:41:40.201: [ CLSINET][1088543040] Returning NETDATA: 0 interfaces

Solution:

The cause is mismatch of private network information in OCR and on OS, output of the following should be consistent with each other (network adapter name, subnet and netmask):

oifcfg iflist -p -n oifcfg getif ifconfig

Bug 10363902

Issue: HAIP fails to start if cluster interconnect is on Infiniband or any other network hardware type that has hardware address (MAC) longer than 6 bytes

Fixed in: 11.2.0.3 for Linux and Solaris

Symptom:

Output of root script:

CRS-2672: Attempting to start ‘ora.cluster_interconnect.haip’ on ‘racnode1′
CRS-5017: The resource action “ora.cluster_interconnect.haip start”
encountered the following error:
Start action for HAIP aborted
CRS-2674: Start of ‘ora.cluster_interconnect.haip’ on ‘racnode1′ failed

$GRID_HOME/log/<hostname>/gipcd/gipcd.log

2010-12-07 13:23:08.560: [ USRTHRD][3858] {0:0:62} Arp::sCreateSocket {
2010-12-07 13:23:08.560: [ USRTHRD][3858] {0:0:62} failed to create arp
2010-12-07 13:23:08.561: [ USRTHRD][3858] {0:0:62} (null) category: -2,
operation: ssclsi_aix_get_phys_addr, loc: aixgetpa:4,n, OS error: 2, other:

@ 10380816 – implement of 10363902 on AIX

Bug 10397652

Issue: HAIP does not failover even when private network experiences problem (i.e. switch port disabled or such) as OS is not providing reliable link information

Fixed in: 11.2.0.3

Workaround on AIX is to set “MONITOR” flag for all private network adapters

# ifconfig en1 monitor # ifconfig en1 en1: flags=5e080863,2c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST, GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),PSEG,LARGESEND,CHAIN,MONITOR> inet 192.168.10.83 netmask 0xfffffc00 broadcast 192.168.11.255 inet 169.254.74.136 netmask 0xffff8000 broadcast 169.254.127.255 tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0

1 comment to 11gR2 Grid Infrastructure Redundant Interconnect and ora.cluster_interconnect.haip

admin

April 7, 2011 at 10:29 PM · Reply

How to Setup IPMP as Cluster Interconnect

Applies to:
Oracle Server – Enterprise Edition – Version: 10.1.0.2 to 11.1.0.7 – Release: 10.1 to 11.1
Oracle Solaris on SPARC (64-bit)
Sun Solaris SPARC (64-bit)
Solaris Operating System (SPARC 64-bit)

Updated on 14-Jan-2009
Goal
The goal is to show how IPMP can be used as cluster interconnect in a RAC environment.

For Oracle database 11gR2 follow note 1069584.1

Solution

To start with the IPMP setup use Note 283107.1.
The IP addresses have been changed to show how it works:

- Physical IP : 192.168.0.99
- Test IP for ce0 : 192.168.0.65
- Test IP for ce1 : 192.168.0.66

Oifcfg requires an interface to be used to configure the private interface. This can not be done with IPMP, because you always have two interfaces and the physical IP will be switched to the active interface.

The rest of the documentation is specific to pre-11gR2 releases. This information is not pertinent for 11gR2. Please do not remove cluster_interconnect information from 11gR2 Clusterware releases.

The recommended solution is not to configure any private interface.

The following steps need to done to use IPMP for the cluster interconnect:
1. If the private interface has already been configured delete the interface with ‘oifcfg delif’
2. Set the CLUSTER_INTERCONNECTS parameter in the spfile/init.ora to the physical IP which is swapped by IPMP.
3. Set the CLUSTER_INTERCONNECTS also for your ASM instances

ATTENTION:

Oracle Clusterware must also use the same physical interface, otherwise an interface down will only be recognized by the instances and an instance is evicted after 10 minutes (the mechanism is called IMR). Oracle Clusterware uses the private hostname for communication, so the private hostname in /etc/hosts must be set to the physical IP (192.168.0.99) that is switched from one interface to the other. The same private hostname must also be used in the Oracle Clusterware configuration during installation.

Oracle Clinic – Maclean Liu的个人技术博客

11gR2 Grid Infrastructure Redundant Interconnect and ora.cluster_interconnect.haip

Applies to:

Purpose

Scope and Application