歡迎您光臨本站 註冊首頁

RHCS雙機,node2重啟導致雙機切換,日誌有點看不明白

←手機掃碼閱讀     火星人 @ 2014-03-04 , reply:0

RHCS雙機,node2重啟導致雙機切換,日誌有點看不明白

Jan 11 10:44:58 AMC-S-X3650-2 dhclient: DHCPREQUEST on usb0 to 169.254.95.118 port 67
Jan 11 10:44:58 AMC-S-X3650-2 dhclient: DHCPACK from 169.254.95.118
Jan 11 10:44:58 AMC-S-X3650-2 dhclient: bound to 169.254.95.120 -- renewal in 270 seconds.
Jan 11 10:49:28 AMC-S-X3650-2 dhclient: DHCPREQUEST on usb0 to 169.254.95.118 port 67
Jan 11 10:49:28 AMC-S-X3650-2 dhclient: DHCPACK from 169.254.95.118
Jan 11 10:49:28 AMC-S-X3650-2 dhclient: bound to 169.254.95.120 -- renewal in 274 seconds.
Jan 11 10:54:02 AMC-S-X3650-2 dhclient: DHCPREQUEST on usb0 to 169.254.95.118 port 67
Jan 11 10:54:02 AMC-S-X3650-2 dhclient: DHCPACK from 169.254.95.118
Jan 11 10:54:02 AMC-S-X3650-2 dhclient: bound to 169.254.95.120 -- renewal in 291 seconds.
Jan 11 10:58:53 AMC-S-X3650-2 dhclient: DHCPREQUEST on usb0 to 169.254.95.118 port 67
Jan 11 10:58:53 AMC-S-X3650-2 dhclient: DHCPACK from 169.254.95.118
Jan 11 10:58:53 AMC-S-X3650-2 dhclient: bound to 169.254.95.120 -- renewal in 263 seconds.
Jan 11 11:03:16 AMC-S-X3650-2 dhclient: DHCPREQUEST on usb0 to 169.254.95.118 port 67
Jan 11 11:03:16 AMC-S-X3650-2 dhclient: DHCPACK from 169.254.95.118
Jan 11 11:03:16 AMC-S-X3650-2 dhclient: bound to 169.254.95.120 -- renewal in 242 seconds.
Jan 11 11:07:17 AMC-S-X3650-2 openais: The token was lost in the OPERATIONAL stat
e.
Jan 11 11:07:17 AMC-S-X3650-2 openais: Receive multicast socket recv buffer size
(288000 bytes).
Jan 11 11:07:17 AMC-S-X3650-2 openais: Transmit multicast socket send buffer size
(262142 bytes).
Jan 11 11:07:17 AMC-S-X3650-2 openais: entering GATHER state from 2.
Jan 11 11:07:18 AMC-S-X3650-2 dhclient: DHCPREQUEST on usb0 to 169.254.95.118 port 67
Jan 11 11:07:18 AMC-S-X3650-2 dhclient: DHCPACK from 169.254.95.118
Jan 11 11:07:18 AMC-S-X3650-2 dhclient: bound to 169.254.95.120 -- renewal in 280 seconds.
Jan 11 11:07:22 AMC-S-X3650-2 openais: entering GATHER state from 0.
Jan 11 11:07:22 AMC-S-X3650-2 openais: Creating commit token because I am the rep
.
Jan 11 11:07:22 AMC-S-X3650-2 openais: Saving state aru 57 high seq received 57
Jan 11 11:07:22 AMC-S-X3650-2 openais: Storing new sequence id for ring e834
Jan 11 11:07:22 AMC-S-X3650-2 openais: entering COMMIT state.
Jan 11 11:07:22 AMC-S-X3650-2 openais: entering RECOVERY state.
Jan 11 11:07:22 AMC-S-X3650-2 openais: position member 192.168.70.2:
Jan 11 11:07:22 AMC-S-X3650-2 openais: previous ring seq 59440 rep 192.168.70.1
Jan 11 11:07:22 AMC-S-X3650-2 openais: aru 57 high delivered 57 received flag 1
Jan 11 11:07:22 AMC-S-X3650-2 openais: Did not need to originate any messages in
recovery.
Jan 11 11:07:22 AMC-S-X3650-2 openais: Sending initial ORF token
Jan 11 11:07:22 AMC-S-X3650-2 openais: CLM CONFIGURATION CHANGE
Jan 11 11:07:22 AMC-S-X3650-2 openais: New Configuration:
Jan 11 11:07:22 AMC-S-X3650-2 kernel: dlm: closing connection to node 1
Jan 11 11:07:22 AMC-S-X3650-2 fenced: node1 not a cluster member after 0 sec post_fail_de
lay
Jan 11 11:07:22 AMC-S-X3650-2 openais:     r(0) ip(192.168.70.2)
Jan 11 11:07:22 AMC-S-X3650-2 fenced: fencing node "node1"
Jan 11 11:07:22 AMC-S-X3650-2 openais: Members Left:
Jan 11 11:07:22 AMC-S-X3650-2 openais:     r(0) ip(192.168.70.1)
Jan 11 11:07:22 AMC-S-X3650-2 openais: Members Joined:
Jan 11 11:07:22 AMC-S-X3650-2 openais: CLM CONFIGURATION CHANGE
Jan 11 11:07:22 AMC-S-X3650-2 openais: New Configuration:
Jan 11 11:07:22 AMC-S-X3650-2 openais:     r(0) ip(192.168.70.2)
Jan 11 11:07:22 AMC-S-X3650-2 openais: Members Left:
Jan 11 11:07:22 AMC-S-X3650-2 openais: Members Joined:
Jan 11 11:07:22 AMC-S-X3650-2 openais: This node is within the primary component
and will provide service.
Jan 11 11:07:22 AMC-S-X3650-2 openais: entering OPERATIONAL state.
Jan 11 11:07:22 AMC-S-X3650-2 openais: got nodejoin message 192.168.70.2
Jan 11 11:07:22 AMC-S-X3650-2 openais: got joinlist message from node 2
Jan 11 11:07:32 AMC-S-X3650-2 fenced: agent "fence_ipmilan" reports: Rebooting machine @
IPMI:192.168.70.126...ipmilan: Failed to connect after 10 seconds Failed
Jan 11 11:07:32 AMC-S-X3650-2 fenced: fence "node1" failed
Jan 11 11:07:33 AMC-S-X3650-2 shutdown: shutting down for system halt
Jan 11 11:07:33 AMC-S-X3650-2 pcscd: winscard.c:304:SCardConnect() Reader E-Gate 0 0 Not Found
Jan 11 11:07:33 AMC-S-X3650-2 kernel: gdm-rh-security: segfault at 0000000000000000 rip 0
0002b95b9a8c19c rsp 0000000041bbfeb0 error 4
Jan 11 11:07:34 AMC-S-X3650-2 rgmanager: : <notice> Shutting down Cluster Service Manage
r...
Jan 11 11:07:34 AMC-S-X3650-2 clurgmgrd: <notice> Shutting down
Jan 11 11:07:37 AMC-S-X3650-2 fenced: fencing node "node1"
Jan 11 11:12:58 AMC-S-X3650-2 syslogd 1.4.1: restart.
Jan 11 11:12:58 AMC-S-X3650-2 kernel: klogd 1.4.1, log source = /proc/kmsg started.
Jan 11 11:12:58 AMC-S-X3650-2 kernel: Linux version 2.6.18-164.el5 (mockbuild@x86-003.build.bos
.redhat.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)) #1 SMP Tue Aug 18 15:51:48 EDT 200
9
Jan 11 11:12:58 AMC-S-X3650-2 kernel: Command line: ro root=/dev/VolGroup01/LogVol00 rhgb quiet
Jan 11 11:12:58 AMC-S-X3650-2 kernel: BIOS-provided physical RAM map:
Jan 11 11:12:58 AMC-S-X3650-2 kernel:  BIOS-e820: 0000000000010000 - 000000000009c000 (usable)
Jan 11 11:12:58 AMC-S-X3650-2 kernel:  BIOS-e820: 000000000009c000 - 00000000000a0000 (reserved
)
Jan 11 11:12:58 AMC-S-X3650-2 kernel:  BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved
)
Jan 11 11:12:58 AMC-S-X3650-2 kernel:  BIOS-e820: 0000000000100000 - 000000007d052000 (usable)
Jan 11 11:12:58 AMC-S-X3650-2 kernel:  BIOS-e820: 000000007d052000 - 000000007d116000 (reserved
)
Jan 11 11:12:58 AMC-S-X3650-2 kernel:  BIOS-e820: 000000007d116000 - 000000007d6dc000 (usable)
Jan 11 11:12:58 AMC-S-X3650-2 kernel:  BIOS-e820: 000000007d6dc000 - 000000007d78c000 (reserved
)
Jan 11 11:12:58 AMC-S-X3650-2 kernel:  BIOS-e820: 000000007d78c000 - 000000007f68f000 (usable)
Jan 11 11:12:58 AMC-S-X3650-2 kernel:  BIOS-e820: 000000007f68f000 - 000000007f6df000 (reserved
)
Jan 11 11:12:58 AMC-S-X3650-2 kernel:  BIOS-e820: 000000007f6df000 - 000000007f7df000 (ACPI NVS
)
Jan 11 11:12:58 AMC-S-X3650-2 kernel:  BIOS-e820: 000000007f7df000 - 000000007f7ff000 (ACPI dat
a)
Jan 11 11:12:58 AMC-S-X3650-2 kernel:  BIOS-e820: 000000007f7ff000 - 000000007f800000 (usable)
Jan 11 11:12:58 AMC-S-X3650-2 kernel:  BIOS-e820: 000000007f800000 - 0000000090000000 (reserved
)
Jan 11 11:12:58 AMC-S-X3650-2 kernel:  BIOS-e820: 00000000fc000000 - 00000000fd000000 (reserved
)
Jan 11 11:12:58 AMC-S-X3650-2 kernel:  BIOS-e820: 00000000fed1c000 - 00000000fed20000 (reserved
)
Jan 11 11:12:58 AMC-S-X3650-2 kernel:  BIOS-e820: 00000000ff800000 - 0000000100000000 (reserved
)
Jan 11 11:12:59 AMC-S-X3650-2 kernel:  BIOS-e820: 0000000100000000 - 0000000180000000 (usable)
Jan 11 11:12:59 AMC-S-X3650-2 kernel: DMI 2.5 present.
Jan 11 11:12:59 AMC-S-X3650-2 rpc.statd: Version 1.0.9 Starting
Jan 11 11:12:59 AMC-S-X3650-2 rpc.statd: statd running as root. chown /var/lib/nfs/statd/
sm to choose different user
Jan 11 11:12:59 AMC-S-X3650-2 kernel: SRAT: PXM 0 -> APIC 0 -> Node 0
Jan 11 11:12:59 AMC-S-X3650-2 kernel: SRAT: PXM 0 -> APIC 2 -> Node 0
Jan 11 11:12:59 AMC-S-X3650-2 kernel: SRAT: PXM 0 -> APIC 4 -> Node 0
Jan 11 11:12:59 AMC-S-X3650-2 kernel: SRAT: PXM 0 -> APIC 6 -> Node 0
Jan 11 11:12:59 AMC-S-X3650-2 kernel: SRAT: PXM 1 -> APIC 16 -> Node 1
Jan 11 11:12:59 AMC-S-X3650-2 kernel: SRAT: PXM 1 -> APIC 18 -> Node 1
Jan 11 11:12:59 AMC-S-X3650-2 kernel: SRAT: PXM 1 -> APIC 20 -> Node 1
Jan 11 11:12:59 AMC-S-X3650-2 kernel: SRAT: PXM 1 -> APIC 22 -> Node 1
Jan 11 11:12:59 AMC-S-X3650-2 kernel: SRAT: PXM 0 -> APIC 1 -> Node 0
Jan 11 11:12:59 AMC-S-X3650-2 kernel: SRAT: PXM 0 -> APIC 3 -> Node 0
Jan 11 11:12:59 AMC-S-X3650-2 kernel: SRAT: PXM 0 -> APIC 5 -> Node 0
Jan 11 11:12:59 AMC-S-X3650-2 kernel: SRAT: PXM 0 -> APIC 7 -> Node 0
Jan 11 11:12:59 AMC-S-X3650-2 kernel: SRAT: PXM 1 -> APIC 17 -> Node 1
Jan 11 11:12:59 AMC-S-X3650-2 kernel: SRAT: PXM 1 -> APIC 19 -> Node 1
Jan 11 11:12:59 AMC-S-X3650-2 kernel: SRAT: PXM 1 -> APIC 21 -> Node 1
Jan 11 11:12:59 AMC-S-X3650-2 kernel: SRAT: PXM 1 -> APIC 23 -> Node 1
Jan 11 11:12:59 AMC-S-X3650-2 kernel: SRAT: Node 0 PXM 0 0-80000000
Jan 11 11:12:59 AMC-S-X3650-2 kernel: SRAT: Node 1 PXM 1 100000000-180000000
Jan 11 11:12:59 AMC-S-X3650-2 ccsd: Starting ccsd 2.0.115:
Jan 11 11:12:59 AMC-S-X3650-2 kernel: Bootmem setup node 0 0000000000000000-0000000080000000
Jan 11 11:12:59 AMC-S-X3650-2 ccsd:  Built: Aug  5 2009 08:24:53
Jan 11 11:12:59 AMC-S-X3650-2 kernel: Bootmem setup node 1 0000000100000000-0000000180000000
Jan 11 11:12:59 AMC-S-X3650-2 ccsd:  Copyright (C) Red Hat, Inc.  2004  All rights reserv
ed.
Jan 11 11:12:59 AMC-S-X3650-2 kernel: Memory for crash kernel (0x0 to 0x0) notwithin permissibl
e range
Jan 11 11:12:59 AMC-S-X3650-2 ccsd: cluster.conf (cluster name = amc_cluster, version = 3
0) found.
Jan 11 11:12:59 AMC-S-X3650-2 kernel: disabling kdump
Jan 11 11:13:00 AMC-S-X3650-2 kernel: ACPI: PM-Timer IO Port: 0x588
Jan 11 11:13:00 AMC-S-X3650-2 kernel: ACPI: LAPIC (acpi_id lapic_id enabled)
Jan 11 11:13:00 AMC-S-X3650-2 kernel: Processor #0 7:10 APIC version 21
Jan 11 11:13:00 AMC-S-X3650-2 kernel: ACPI: LAPIC (acpi_id lapic_id enabled)
Jan 11 11:13:00 AMC-S-X3650-2 kernel: Processor #2 7:10 APIC version 21
Jan 11 11:13:00 AMC-S-X3650-2 kernel: ACPI: LAPIC (acpi_id lapic_id enabled)
Jan 11 11:13:00 AMC-S-X3650-2 kernel: Processor #4 7:10 APIC version 21
Jan 11 11:13:00 AMC-S-X3650-2 kernel: ACPI: LAPIC (acpi_id lapic_id enabled)
Jan 11 11:13:00 AMC-S-X3650-2 kernel: Processor #6 7:10 APIC version 21
Jan 11 11:13:00 AMC-S-X3650-2 kernel: ACPI: LAPIC (acpi_id lapic_id enabled)
Jan 11 11:13:00 AMC-S-X3650-2 kernel: Processor #16 7:10 APIC version 21
Jan 11 11:13:00 AMC-S-X3650-2 kernel: ACPI: LAPIC (acpi_id lapic_id enabled)
Jan 11 11:13:00 AMC-S-X3650-2 kernel: Processor #18 7:10 APIC version 21
Jan 11 11:13:00 AMC-S-X3650-2 kernel: ACPI: LAPIC (acpi_id lapic_id enabled)
Jan 11 11:13:00 AMC-S-X3650-2 kernel: Processor #20 7:10 APIC version 21
Jan 11 11:13:00 AMC-S-X3650-2 kernel: ACPI: LAPIC (acpi_id lapic_id enabled)
Jan 11 11:13:00 AMC-S-X3650-2 kernel: Processor #22 7:10 APIC version 21
Jan 11 11:13:00 AMC-S-X3650-2 kernel: ACPI: LAPIC (acpi_id lapic_id enabled)
Jan 11 11:13:00 AMC-S-X3650-2 kernel: Processor #1 7:10 APIC version 21
Jan 11 11:13:00 AMC-S-X3650-2 kernel: ACPI: LAPIC (acpi_id lapic_id enabled)
Jan 11 11:13:00 AMC-S-X3650-2 kernel: Processor #3 7:10 APIC version 21
Jan 11 11:13:00 AMC-S-X3650-2 kernel: ACPI: LAPIC (acpi_id lapic_id enabled)
Jan 11 11:13:00 AMC-S-X3650-2 kernel: Processor #5 7:10 APIC version 21
Jan 11 11:13:00 AMC-S-X3650-2 kernel: ACPI: LAPIC (acpi_id lapic_id enabled)
Jan 11 11:13:00 AMC-S-X3650-2 kernel: Processor #7 7:10 APIC version 21
Jan 11 11:13:00 AMC-S-X3650-2 kernel: ACPI: LAPIC (acpi_id lapic_id enabled)
Jan 11 11:13:00 AMC-S-X3650-2 kernel: Processor #17 7:10 APIC version 21
Jan 11 11:13:00 AMC-S-X3650-2 kernel: ACPI: LAPIC (acpi_id lapic_id enabled)
Jan 11 11:13:00 AMC-S-X3650-2 kernel: Processor #19 7:10 APIC version 21
Jan 11 11:13:00 AMC-S-X3650-2 kernel: ACPI: LAPIC (acpi_id lapic_id enabled)
Jan 11 11:13:00 AMC-S-X3650-2 kernel: Processor #21 7:10 APIC version 21
Jan 11 11:13:00 AMC-S-X3650-2 kernel: ACPI: LAPIC (acpi_id lapic_id enabled)
Jan 11 11:13:00 AMC-S-X3650-2 kernel: Processor #23 7:10 APIC version 21
Jan 11 11:13:00 AMC-S-X3650-2 kernel: ACPI: LAPIC_NMI (acpi_id dfl dfl lint)
Jan 11 11:13:00 AMC-S-X3650-2 kernel: ACPI: IOAPIC (id address gsi_base)
Jan 11 11:13:00 AMC-S-X3650-2 kernel: IOAPIC: apic_id 8, version 32, address 0xfec00000, GSI
0-23
Jan 11 11:13:00 AMC-S-X3650-2 kernel: ACPI: IOAPIC (id address gsi_base)
Jan 11 11:13:00 AMC-S-X3650-2 kernel: IOAPIC: apic_id 9, version 32, address 0xfec80000, GSI
24-47
Jan 11 11:13:00 AMC-S-X3650-2 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
Jan 11 11:13:01 AMC-S-X3650-2 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high leve
l)
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Setting APIC routing to clustered
Jan 11 11:13:01 AMC-S-X3650-2 kernel: ACPI: HPET id: 0x8086a301 base: 0xfed00000
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Using ACPI (MADT) for SMP configuration information
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Nosave address range: 000000000009c000 - 00000000000a0000
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Nosave address range: 00000000000a0000 - 00000000000e0000
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Nosave address range: 00000000000e0000 - 0000000000100000
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Nosave address range: 000000007d052000 - 000000007d116000
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Nosave address range: 000000007d6dc000 - 000000007d78c000
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Nosave address range: 000000007f68f000 - 000000007f6df000
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Nosave address range: 000000007f6df000 - 000000007f7df000
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Nosave address range: 000000007f7df000 - 000000007f7ff000
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Nosave address range: 000000007f800000 - 0000000090000000
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Nosave address range: 0000000090000000 - 00000000fc000000
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Nosave address range: 00000000fc000000 - 00000000fd000000
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Nosave address range: 00000000fd000000 - 00000000fed1c000
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Nosave address range: 00000000fed1c000 - 00000000fed20000
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Nosave address range: 00000000fed20000 - 00000000ff800000
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Nosave address range: 00000000ff800000 - 0000000100000000
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Allocating PCI resources starting at 98000000 (gap: 90000
000:6c000000)
Jan 11 11:13:01 AMC-S-X3650-2 kernel: SMP: Allowing 16 CPUs, 0 hotplug CPUs
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Built 2 zonelists.  Total pages: 1030045
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Kernel command line: ro root=/dev/VolGroup01/LogVol00 rhg
b quiet
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Initializing CPU#0
Jan 11 11:13:01 AMC-S-X3650-2 kernel: PID hash table entries: 4096 (order: 12, 32768 bytes)
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Console: colour VGA+ 80x25
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Dentry cache hash table entries: 524288 (order: 10, 41943
04 bytes)
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Inode-cache hash table entries: 262144 (order: 9, 2097152
bytes)
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Checking aperture...
Jan 11 11:13:01 AMC-S-X3650-2 kernel: ACPI: DMAR not present
Jan 11 11:13:01 AMC-S-X3650-2 kernel: PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Placing software IO TLB between 0x29d0000 - 0x69d0000
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Memory: 4043748k/6291456k available (2547k kernel code, 1
38940k reserved, 1289k data, 208k init)
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Calibrating delay loop (skipped), value calculated using
timer frequency.. 4533.62 BogoMIPS (lpj=2266812)
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Security Framework v1.0.0 initialized
Jan 11 11:13:01 AMC-S-X3650-2 kernel: SELinux:  Initializing.
Jan 11 11:13:01 AMC-S-X3650-2 kernel: selinux_register_security:  Registering secondary module
capability
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Capability LSM initialized as secondary
Jan 11 11:13:01 AMC-S-X3650-2 kernel: Mount-cache hash table entries: 256
Jan 11 11:13:01 AMC-S-X3650-2 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
Jan 11 11:13:01 AMC-S-X3650-2 kernel: CPU: L2 cache: 256K
Jan 11 11:13:02 AMC-S-X3650-2 kernel: CPU: L3 cache: 8192K
Jan 11 11:13:02 AMC-S-X3650-2 kernel: CPU 0/0 -> Node 0
Jan 11 11:13:02 AMC-S-X3650-2 kernel: using mwait in idle threads.
Jan 11 11:13:02 AMC-S-X3650-2 kernel: CPU: Physical Processor ID: 0
Jan 11 11:13:02 AMC-S-X3650-2 kernel: CPU: Processor Core ID: 0
Jan 11 11:13:02 AMC-S-X3650-2 kernel: CPU0: Thermal monitoring enabled (TM1)
Jan 11 11:13:02 AMC-S-X3650-2 kernel: SMP alternatives: switching to UP code
Jan 11 11:13:02 AMC-S-X3650-2 kernel: ACPI: Core revision 20060707
Jan 11 11:13:02 AMC-S-X3650-2 kernel: Using local APIC timer interrupts.
Jan 11 11:13:02 AMC-S-X3650-2 kernel: result 8333842
Jan 11 11:13:02 AMC-S-X3650-2 kernel: Detected 8.333 MHz APIC timer.
Jan 11 11:13:02 AMC-S-X3650-2 kernel: SMP alternatives: switching to SMP code
Jan 11 11:13:02 AMC-S-X3650-2 kernel: Booting processor 1/16 APIC 0x2
Jan 11 11:13:02 AMC-S-X3650-2 kernel: Initializing CPU#1
Jan 11 11:13:02 AMC-S-X3650-2 kernel: Calibrating delay using timer specific routine.. 4533.40
BogoMIPS (lpj=2266701)
Jan 11 11:13:02 AMC-S-X3650-2 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
Jan 11 11:13:02 AMC-S-X3650-2 kernel: CPU: L2 cache: 256K
Jan 11 11:13:02 AMC-S-X3650-2 kernel: CPU: L3 cache: 8192K
Jan 11 11:13:02 AMC-S-X3650-2 kernel: CPU 1/2 -> Node 0
Jan 11 11:13:02 AMC-S-X3650-2 kernel: CPU: Physical Processor ID: 0
--More--(84%)
《解決方案》

什麼原因導致node2重啟了呢:dizzy:
《解決方案》

本帖最後由 yjs_sh 於 2014-01-26 14:03 編輯

可能node1和node2之間的心跳通訊出現問題,因此RHCS會觸發fence。由於心跳相互都不通,因此雙方都會fence對方,但是從日誌里看,node2在fence node1的時候失敗,而node2的重啟應該是由於node1 fence node2造成的。
最終的原因可能是你的node2的網路出問題造成的,你可以檢查一下node1的日誌看看。這個結果應該是rhcs的正常反應。
《解決方案》

Jan 11 00:50:58 AMC-S-X3650-1 last message repeated 8 times
Jan 11 01:15:56 AMC-S-X3650-1 last message repeated 8 times
Jan 11 07:12:18 AMC-S-X3650-1 last message repeated 8 times
Jan 11 10:01:18 AMC-S-X3650-1 last message repeated 9 times
Jan 11 10:01:58 AMC-S-X3650-1 last message repeated 7 times
Jan 11 11:10:05 AMC-S-X3650-1 openais: entering GATHER state from 12.
Jan 11 11:10:05 AMC-S-X3650-1 openais: Creating commit token because I am the rep.
Jan 11 11:10:05 AMC-S-X3650-1 openais: Saving state aru 57 high seq received 57
Jan 11 11:10:05 AMC-S-X3650-1 openais: Storing new sequence id for ring e834
Jan 11 11:10:05 AMC-S-X3650-1 openais: entering COMMIT state.
Jan 11 11:10:15 AMC-S-X3650-1 openais: The token was lost in the COMMIT state.
Jan 11 11:10:15 AMC-S-X3650-1 openais: entering GATHER state from 4.
Jan 11 11:10:20 AMC-S-X3650-1 openais: entering GATHER state from 0.
Jan 11 11:10:20 AMC-S-X3650-1 openais: Creating commit token because I am the rep.
Jan 11 11:10:20 AMC-S-X3650-1 openais: Storing new sequence id for ring e838
Jan 11 11:10:20 AMC-S-X3650-1 openais: entering COMMIT state.
Jan 11 11:10:20 AMC-S-X3650-1 openais: entering RECOVERY state.
Jan 11 11:10:20 AMC-S-X3650-1 openais: position member 192.168.70.1:
Jan 11 11:10:20 AMC-S-X3650-1 openais: previous ring seq 59440 rep 192.168.70.1
Jan 11 11:10:20 AMC-S-X3650-1 openais: aru 57 high delivered 57 received flag 1
Jan 11 11:10:20 AMC-S-X3650-1 openais: Did not need to originate any messages in recovery.
Jan 11 11:10:20 AMC-S-X3650-1 openais: Sending initial ORF token
Jan 11 11:10:20 AMC-S-X3650-1 openais: CLM CONFIGURATION CHANGE
Jan 11 11:10:20 AMC-S-X3650-1 openais: New Configuration:
Jan 11 11:10:20 AMC-S-X3650-1 kernel: dlm: closing connection to node 2
Jan 11 11:10:20 AMC-S-X3650-1 openais:     r(0) ip(192.168.70.1)
Jan 11 11:10:20 AMC-S-X3650-1 fenced: node2 not a cluster member after 0 sec post_fail_delay
Jan 11 11:10:20 AMC-S-X3650-1 openais: Members Left:
Jan 11 11:10:20 AMC-S-X3650-1 fenced: fencing node "node2"
Jan 11 11:10:20 AMC-S-X3650-1 openais:     r(0) ip(192.168.70.2)
Jan 11 11:10:20 AMC-S-X3650-1 openais: Members Joined:
Jan 11 11:10:20 AMC-S-X3650-1 openais: CLM CONFIGURATION CHANGE
Jan 11 11:10:20 AMC-S-X3650-1 openais: New Configuration:
Jan 11 11:10:20 AMC-S-X3650-1 openais:     r(0) ip(192.168.70.1)
Jan 11 11:10:20 AMC-S-X3650-1 openais: Members Left:
Jan 11 11:10:20 AMC-S-X3650-1 openais: Members Joined:
Jan 11 11:10:20 AMC-S-X3650-1 openais: This node is within the primary component and will provide service.
Jan 11 11:10:20 AMC-S-X3650-1 openais: entering OPERATIONAL state.
Jan 11 11:10:20 AMC-S-X3650-1 openais: got nodejoin message 192.168.70.1
Jan 11 11:10:20 AMC-S-X3650-1 openais: got joinlist message from node 1
Jan 11 11:10:20 AMC-S-X3650-1 openais: entering GATHER state from 9.
Jan 11 11:10:25 AMC-S-X3650-1 openais: entering GATHER state from 0.
Jan 11 11:10:25 AMC-S-X3650-1 openais: Creating commit token because I am the rep.
Jan 11 11:10:25 AMC-S-X3650-1 openais: Saving state aru e high seq received e
Jan 11 11:10:25 AMC-S-X3650-1 openais: Storing new sequence id for ring e83c
Jan 11 11:10:25 AMC-S-X3650-1 openais: entering COMMIT state.
Jan 11 11:10:25 AMC-S-X3650-1 openais: entering RECOVERY state.
Jan 11 11:10:25 AMC-S-X3650-1 openais: position member 192.168.70.1:
Jan 11 11:10:25 AMC-S-X3650-1 openais: previous ring seq 59448 rep 192.168.70.1
Jan 11 11:10:25 AMC-S-X3650-1 openais: aru e high delivered e received flag 1
Jan 11 11:10:25 AMC-S-X3650-1 openais: Did not need to originate any messages in recovery.
Jan 11 11:10:25 AMC-S-X3650-1 openais: Sending initial ORF token
Jan 11 11:10:25 AMC-S-X3650-1 openais: CLM CONFIGURATION CHANGE
Jan 11 11:10:25 AMC-S-X3650-1 openais: New Configuration:
Jan 11 11:10:25 AMC-S-X3650-1 openais:     r(0) ip(192.168.70.1)
Jan 11 11:10:25 AMC-S-X3650-1 openais: Members Left:
Jan 11 11:10:25 AMC-S-X3650-1 openais: Members Joined:
Jan 11 11:10:25 AMC-S-X3650-1 openais: CLM CONFIGURATION CHANGE
Jan 11 11:10:25 AMC-S-X3650-1 openais: New Configuration:
Jan 11 11:10:25 AMC-S-X3650-1 openais:     r(0) ip(192.168.70.1)
Jan 11 11:10:25 AMC-S-X3650-1 openais: Members Left:
Jan 11 11:10:25 AMC-S-X3650-1 openais: Members Joined:
Jan 11 11:10:25 AMC-S-X3650-1 openais: This node is within the primary component and will provide service.
Jan 11 11:10:25 AMC-S-X3650-1 openais: entering OPERATIONAL state.
Jan 11 11:10:25 AMC-S-X3650-1 openais: got nodejoin message 192.168.70.1
Jan 11 11:10:25 AMC-S-X3650-1 openais: got joinlist message from node 1
Jan 11 11:10:25 AMC-S-X3650-1 openais: entering GATHER state from 9.
Jan 11 11:10:30 AMC-S-X3650-1 openais: entering GATHER state from 0.
Jan 11 11:10:30 AMC-S-X3650-1 openais: Creating commit token because I am the rep.
Jan 11 11:10:30 AMC-S-X3650-1 openais: Saving state aru c high seq received c
Jan 11 11:10:30 AMC-S-X3650-1 openais: Storing new sequence id for ring e840
Jan 11 11:10:30 AMC-S-X3650-1 openais: entering COMMIT state.
Jan 11 11:10:30 AMC-S-X3650-1 openais: entering RECOVERY state.
Jan 11 11:10:30 AMC-S-X3650-1 openais: position member 192.168.70.1:
Jan 11 11:10:30 AMC-S-X3650-1 openais: previous ring seq 59452 rep 192.168.70.1
Jan 11 11:10:30 AMC-S-X3650-1 openais: aru c high delivered c received flag 1
Jan 11 11:10:30 AMC-S-X3650-1 openais: Did not need to originate any messages in recovery.
Jan 11 11:10:30 AMC-S-X3650-1 openais: Sending initial ORF token
Jan 11 11:10:30 AMC-S-X3650-1 openais: CLM CONFIGURATION CHANGE
Jan 11 11:10:30 AMC-S-X3650-1 openais: New Configuration:
Jan 11 11:10:30 AMC-S-X3650-1 openais:     r(0) ip(192.168.70.1)
Jan 11 11:10:30 AMC-S-X3650-1 openais: Members Left:
Jan 11 11:10:30 AMC-S-X3650-1 openais: Members Joined:
Jan 11 11:10:30 AMC-S-X3650-1 openais: CLM CONFIGURATION CHANGE
Jan 11 11:10:30 AMC-S-X3650-1 openais: New Configuration:
Jan 11 11:10:30 AMC-S-X3650-1 openais:     r(0) ip(192.168.70.1)
Jan 11 11:10:30 AMC-S-X3650-1 openais: Members Left:
Jan 11 11:10:30 AMC-S-X3650-1 openais: Members Joined:
Jan 11 11:10:30 AMC-S-X3650-1 openais: This node is within the primary component and will provide service.
Jan 11 11:10:30 AMC-S-X3650-1 openais: entering OPERATIONAL state.
Jan 11 11:10:30 AMC-S-X3650-1 openais: got nodejoin message 192.168.70.1
Jan 11 11:10:30 AMC-S-X3650-1 openais: got joinlist message from node 1
Jan 11 11:10:30 AMC-S-X3650-1 fenced: fence "node2" success
Jan 11 11:10:31 AMC-S-X3650-1 clurgmgrd: <notice> Taking over service service:amcapache from down member node2
Jan 11 11:10:31 AMC-S-X3650-1 avahi-daemon: Registering new address record for 172.20.16.2 on eth0.
Jan 11 11:10:32 AMC-S-X3650-1 clurgmgrd: <notice> Service service:amcapache started
Jan 11 11:14:24 AMC-S-X3650-1 openais: entering GATHER state from 9.
Jan 11 11:14:24 AMC-S-X3650-1 openais: Creating commit token because I am the rep.
Jan 11 11:14:24 AMC-S-X3650-1 openais: Saving state aru 16 high seq received 16
Jan 11 11:14:24 AMC-S-X3650-1 openais: Storing new sequence id for ring e844
Jan 11 11:14:24 AMC-S-X3650-1 openais: entering COMMIT state.
Jan 11 11:14:24 AMC-S-X3650-1 openais: entering RECOVERY state.
Jan 11 11:14:24 AMC-S-X3650-1 openais: position member 192.168.70.1:
Jan 11 11:14:24 AMC-S-X3650-1 openais: previous ring seq 59456 rep 192.168.70.1
Jan 11 11:14:24 AMC-S-X3650-1 openais: aru 16 high delivered 16 received flag 1
Jan 11 11:14:24 AMC-S-X3650-1 openais: position member 192.168.70.2:
Jan 11 11:14:24 AMC-S-X3650-1 openais: previous ring seq 59448 rep 192.168.70.2
Jan 11 11:14:24 AMC-S-X3650-1 openais: aru d high delivered d received flag 1
Jan 11 11:14:24 AMC-S-X3650-1 openais: Did not need to originate any messages in recovery.
Jan 11 11:14:24 AMC-S-X3650-1 openais: Sending initial ORF token
Jan 11 11:14:24 AMC-S-X3650-1 openais: CLM CONFIGURATION CHANGE
Jan 11 11:14:24 AMC-S-X3650-1 openais: New Configuration:
Jan 11 11:14:24 AMC-S-X3650-1 openais:     r(0) ip(192.168.70.1)
Jan 11 11:14:24 AMC-S-X3650-1 openais: Members Left:
Jan 11 11:14:24 AMC-S-X3650-1 openais: Members Joined:
Jan 11 11:14:24 AMC-S-X3650-1 openais: CLM CONFIGURATION CHANGE
Jan 11 11:14:24 AMC-S-X3650-1 openais: New Configuration:
Jan 11 11:14:24 AMC-S-X3650-1 openais:     r(0) ip(192.168.70.1)
Jan 11 11:14:24 AMC-S-X3650-1 openais:     r(0) ip(192.168.70.2)
Jan 11 11:14:24 AMC-S-X3650-1 openais: Members Left:
Jan 11 11:14:24 AMC-S-X3650-1 openais: Members Joined:
Jan 11 11:14:24 AMC-S-X3650-1 openais:     r(0) ip(192.168.70.2)
Jan 11 11:14:24 AMC-S-X3650-1 openais: This node is within the primary component and will provide service.
Jan 11 11:14:24 AMC-S-X3650-1 openais: entering OPERATIONAL state.
Jan 11 11:14:24 AMC-S-X3650-1 openais: got nodejoin message 192.168.70.1
Jan 11 11:14:24 AMC-S-X3650-1 openais: got nodejoin message 192.168.70.2
Jan 11 11:14:24 AMC-S-X3650-1 openais: got joinlist message from node 1
Jan 11 11:14:24 AMC-S-X3650-1 openais: got joinlist message from node 2
Jan 11 11:14:35 AMC-S-X3650-1 kernel: dlm: connecting to 2
Jan 11 11:14:35 AMC-S-X3650-1 kernel: dlm: got connection from 2

你好,這是node1的日誌信息
《解決方案》

Jan 11 11:10:20 AMC-S-X3650-1 kernel: dlm: closing connection to node 2
Jan 11 11:10:20 AMC-S-X3650-1 openais:     r(0) ip(192.168.70.1)
Jan 11 11:10:20 AMC-S-X3650-1 fenced: node2 not a cluster member after 0 sec post_fail_delay
Jan 11 11:10:20 AMC-S-X3650-1 openais: Members Left:
Jan 11 11:10:20 AMC-S-X3650-1 fenced: fencing node "node2"
以上表明node1無法檢測到node2的心跳后,認為node2已經出現問題,開始fence node2
Jan 11 11:10:30 AMC-S-X3650-1 fenced: fence "node2" success
Jan 11 11:10:31 AMC-S-X3650-1 clurgmgrd: <notice> Taking over service service:amcapache from down member node2
以上表明node1已經成功fence node2,接下來開始接管HA所管理的服務。

RHCS就是這樣的機制,如果一個節點出現問題,那麼其他節點將會fence這個節點。基本上fence就是將其重新啟動,當然還有部分fence是將存儲斷開。
《解決方案》

那node2是為什麼重啟的呢?
《解決方案》

從日誌上看應該可能是node2的網路出現了問題,然後被node1 fence掉了。其實在node2的日誌中也出現了fence node1的記錄,但是沒有成功,那基本就證明了是node2的網路問題。一般fence失敗基本上都是網路問題,除非rhce的fence配置有問題。不過你還可以看看是不是當時服務出了問題,一般rhce會在本地重新啟動服務,3次失敗後會進行切換,如果切換的過程中比如:卸載文件系統失敗,也會自動重啟,不過應該日誌中不會是被fence掉的日誌。
《解決方案》

RHCS垃圾中的戰鬥機

[火星人 ] RHCS雙機,node2重啟導致雙機切換,日誌有點看不明白已經有1075次圍觀

http://coctec.com/docs/service/show-post-3799.html