用VMWARE安裝配置GFS（使用DLM方法）
本文參考文獻：
REDHAT官方網站：
Red Hat GFS 6.1 Administrator's Guide
http://www.redhat.com/docs/manuals/csgfs/browse/rh-gfs-en/index.html
Red Hat Cluster Suite Configuring and Managing a Cluster
http://www.redhat.com/docs/manuals/csgfs/browse/rh-cs-en/index.html
CU網站：
GFS6.1 ON RHAS4 U2安裝文檔
http://www.chinaunix.net/jh/4/746833.html
RedHat GFS 集群文件系統入門和進階 資源帖
http://bbs.chinaunix.net/viewthread.php?tid=777867&extra=page%3D1%26filter%3Ddigest
用VMware GSX和W2K 群集服務實現Exchange群集
http://www.chinaunix.net/jh/34/299757.html
注視：本文後半部分5-10部分基本抄襲suran007 同學的文章，但是去掉了LVM部分，因為沒有這個環境。特此GPL公告。
後面我將寫一個前端LVS，後端GFS的配置文檔，希望對大家有所幫助。
一、測試環境
主機：一台PC,AMD-64位的晶元，1G內存,安裝CentOS-4.4-x86_64版本的操作系統
在這個主機上面安裝了三個虛擬機，全部安裝CentOS-4.4-x86_64版本的操作系統,未進行內核定製，也未打內核補丁，同時安裝了X-windows系統，同時參考
http://www.chinaunix.net/jh/34/299757.html
這篇文章創建了共享磁碟，三個虛擬機使用一個共享磁碟，不過我的網卡一開始就設了兩個
二、需要安裝的包以及順序
包下載地址：
http://mirror.centos.org/centos/4/csgfs/x86_64/RPMS/
2.1. 在所有節點上安裝必須的軟體包，軟體包完整列表請參考GFS6.1用戶手冊 (http://www.redhat.com/docs/manuals/csgfs/browse/rh-gfs-en/ch-install.html#S1-PKG-
INSTALL)
rgmanager — Manages cluster services and resources
system-config-cluster — Contains the Cluster Configuration Tool, used to graphically configure the cluster and the display of the current status of the
nodes, resources, fencing agents, and cluster services
ccsd — Contains the cluster configuration services daemon (ccsd) and associated files
magma — Contains an interface library for cluster lock management
magma-plugins — Contains plugins for the magma library
cman — Contains the Cluster Manager (CMAN), which is used for managing cluster membership, messaging, and notification
cman-kernel — Contains required CMAN kernel modules
dlm — Contains distributed lock management (DLM) library
dlm-kernel — Contains required DLM kernel modules
fence — The cluster I/O fencing system that allows cluster nodes to connect to a variety of network power switches, fibre channel switches, and integrated
power management interfaces
gulm — Contains the GULM lock management userspace tools and libraries (an alternative to using CMAN and DLM).
iddev — Contains libraries used to identify the file system (or volume manager) in which a device is formatted
Also, you can optionally install Red Hat GFS on your Red Hat Cluster Suite. Red Hat GFS consists of the following RPMs:
GFS — The Red Hat GFS module
GFS-kernel — The Red Hat GFS kernel module
gnbd — The GFS Network Block Device module
gnbd-kernel — Kernel module for the GFS Network Block Device
lvm2-cluster — Cluster extensions for the logical volume manager
GFS-kernheaders — GFS kernel header files
gnbd-kernheaders — gnbd kernel header files
2.2 安裝軟體和順序
安裝腳本，install.sh
#!/bin/bash
rpm -ivh kernel-smp-2.6.9-42.EL.x86_64.rpm
rpm -ivh kernel-smp-devel-2.6.9-42.EL.x86_64.rpm
rpm -ivh perl-Net-Telnet-3.03-3.noarch.rpm
rpm -ivh magma-1.0.6-0.x86_64.rpm
rpm -ivh magma-devel-1.0.6-0.x86_64.rpm
rpm -ivh ccs-1.0.7-0.x86_64.rpm
rpm -ivh ccs-devel-1.0.7-0.x86_64.rpm
rpm -ivh cman-kernel-2.6.9-45.4.centos4.x86_64.rpm
rpm -ivh cman-kernheaders-2.6.9-45.4.centos4.x86_64.rpm
rpm -ivh cman-1.0.11-0.x86_64.rpm
rpm -ivh cman-devel-1.0.11-0.x86_64.rpm
rpm -ivh dlm-kernel-2.6.9-42.12.centos4.x86_64.rpm
rpm -ivh dlm-kernheaders-2.6.9-42.12.centos4.x86_64.rpm
rpm -ivh dlm-1.0.1-1.x86_64.rpm
rpm -ivh dlm-devel-1.0.1-1.x86_64.rpm
rpm -ivh fence-1.32.25-1.x86_64.rpm
rpm -ivh GFS-6.1.6-1.x86_64.rpm
rpm -ivh GFS-kernel-2.6.9-58.2.centos4.x86_64.rpm
rpm -ivh GFS-kernheaders-2.6.9-58.2.centos4.x86_64.rpm
#gnbd:
rpm -ivh gnbd-kernel-2.6.9-9.43.centos4.x86_64.rpm
rpm -ivh gnbd-kernheaders-2.6.9-9.43.centos4.x86_64.rpm
rpm -ivh gnbd-1.0.7-0.x86_64.rpm
rpm -ivh gulm-1.0.7-0.x86_64.rpm
rpm -ivh gulm-devel-1.0.7-0.x86_64.rpm
rpm -ivh iddev-2.0.0-3.x86_64.rpm
rpm -ivh iddev-devel-2.0.0-3.x86_64.rpm
rpm -ivh magma-plugins-1.0.9-0.x86_64.rpm
rpm -ivh rgmanager-1.9.53-0.x86_64.rpm
rpm -ivh system-config-cluster-1.0.25-1.0.noarch.rpm
rpm -ivh ipvsadm-1.24-6.x86_64.rpm
rpm ivh piranha-0.8.2-1.x86_64.rpm --nodeps
注視：有些包有依賴關係，使用nodeps開關進行安裝即可，另外因為本人安裝了X-windows，所以gnome的包沒寫在這裡
2.3、修改各個節點上的/etc/hosts文件（每個節點都一樣）
如下：
# cat hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
#127.0.0.1 gfs-node01 localhost.localdomain localhost
192.168.10.1 gfs-node01
192.168.10.2 gfs-node02
192.168.10.3 gfs-node03
192.168.10.1 gnbd-server
#
備註：我把gnbs-server和node01放在了同一台server上，實際運行如果有條件的話中最好分開
三、在node-01上進行分區
#dmesg |grep scsi察看scsi設備，如下：
# dmesg|grep scsi
scsi0 : ioc0: LSI53C1030, FwRev=00000000h, Ports=1, MaxQ=128, IRQ=169
Attached scsi disk sda at scsi0, channel 0, id 0, lun 0
說明認出了scsi設備，然後用fdisk進行了分區，注意，只進行分區，不要格式化,在這裡我使用fdisk分了兩個大小為3G的分區，分別為：
/dev/sda1和/dev/sda2,如下：
# fdisk -l
Disk /dev/hda: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/hda1 * 1 13 104391 83 Linux
/dev/hda2 14 1044 8281507+ 8e Linux LVM
Disk /dev/sda: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 1 366 2939863+ 83 Linux
/dev/sda2 367 732 2939895 83 Linux
#
四、運行system-config-cluster進行配置
(注視「因為system-config-cluster需要X-windows運行，所以第一次測試時安裝了X-windows，實際配置時候只需要根據下面的文件進行修改即可)
增加三個節點，節點的權置全部設置為1，即Quorum值設置為1
三個節點的名稱為:
gfs-node01
gfs-node02
gfs-node03
修改cluster.conf文件，如下：
# cat /etc/cluster/cluster.conf
<?xml version="1.0"?>
<cluster config_version="1" name="alpha_cluster">
<fence_daemon post_fail_delay="0" post_join_delay="3"/>
<clusternodes>
<clusternode name="gfs-node01" votes="1">
<fence>
<method name="1">
<device name="F-Man" nodename="gfs-node01" ipaddr="192.168.10.1"/>
</method>
</fence>
</clusternode>
<clusternode name="gfs-node02" votes="1">
<fence>
<method name="1">
<device name="F-Man" nodename="gfs-node02" ipaddr="192.168.10.2"/>
</method>
</fence>
</clusternode>
<clusternode name="gfs-node03" votes="1">
<fence>
<method name="1">
<device name="F-Man" nodename="gfs-node03" ipaddr="192.168.10.3"/>
</method>
</fence>
</clusternode>
</clusternodes>
<cman/>
<fencedevices>
<fencedevice agent="fence_manual" name="F-Man"/>
</fencedevices>
<rm>
<failoverdomains>
<failoverdomain name="web_failover" ordered="1" restricted="0">
<failoverdomainnode name="gfs-node01" priority="1"/>
<failoverdomainnode name="gfs-node02" priority="2"/>
<failoverdomainnode name="gfs-node03" priority="3"/>
</failoverdomain>
</failoverdomains>
</rm>
</cluster>
#
使用scp命令把這個配置文件copy到02/03節點上
五、 在01/02/03節點上啟動dlm,ccsd,fence等服務
5.1、在三個節點上載入dlm模塊
# modprobe lock_dlm
# modprobe lock_dlm
# modprobe lock_dlm
5.2、啟動ccsd服務
# ccsd
# ccsd
# ccsd
5.3、在兩個節點上啟動集群管理器（cman）
root@one # /sbin/cman_tool join
root@tow # /sbin/cman_tool join
5.4、測試ccsd服務
(注意：ccsd的測試要等cman啟動完成後，然後才可以進行下面的測試
# ccs_test connect
# ccs_test connect
# ccs_test connect
# ccs_test connect 各個節點的返回如下：
node 1：
# ccs_test connect
Connect successful.
Connection descriptor = 0
node 2：
# ccs_test connect
Connect successful.
Connection descriptor = 30
node 3：
# ccs_test connect
Connect successful.
Connection descriptor = 60
5.5、查看節點狀態
cat /proc/cluster/nodes,應該返回
# cat /proc/cluster/nodes
Node Votes Exp Sts Name
1 1 3 M gfs-node01
2 1 3 M gfs-node02
3 1 3 M gfs-node03
#
六、加入fence域：
# /sbin/fence_tool join
# /sbin/fence_tool join
# /sbin/fence_tool join
七、 查看集群狀態
Node 1:
# cat /proc/cluster/status
Protocol version: 5.0.1
Config version: 1
Cluster name: alpha_cluster
Cluster ID: 50356
Cluster Member: Yes
Membership state: Cluster-Member
Nodes: 3
Expected_votes: 3
Total_votes: 3
Quorum: 2
Active subsystems: 1
Node name: gfs-node01
Node ID: 1
Node addresses: 192.168.10.1
Node 2
# cat /proc/cluster/status
Protocol version: 5.0.1
Config version: 1
Cluster name: alpha_cluster
Cluster ID: 50356
Cluster Member: Yes
Membership state: Cluster-Member
Nodes: 3
Expected_votes: 3
Total_votes: 3
Quorum: 2
Active subsystems: 1
Node name: gfs-node02
Node ID: 2
Node addresses: 192.168.10.2
Node3:
# cat /proc/cluster/status
Protocol version: 5.0.1
Config version: 1
Cluster name: alpha_cluster
Cluster ID: 50356
Cluster Member: Yes
Membership state: Cluster-Member
Nodes: 3
Expected_votes: 3
Total_votes: 3
Quorum: 2
Active subsystems: 1
Node name: gfs-node03
Node ID: 3
Node addresses: 192.168.10.3
八、 在gnbd-server 導出設備
8.1、啟動gnbd_serv進程
# /sbin/gnbd_serv –v –n
gnbd_serv: startup succeeded
導出設備
# gnbd_export -v -e gfs -d /dev/sda1 -c
gnbd_export: created GNBD gfs serving file /dev/sda1
#
查看export狀態信息
# gnbd_export -v -l
Server : gfs
--------------------------
file : /dev/sda1
sectors : 5879727
readonly : no
cached : yes
timeout : no
#
8.2、 在01/02/03三個節點導入設備
# modprobe gnbd
# modprobe gnbd
# modprobe gnbd
導入設備
Node1:
# gnbd_import -v -i gnbd-server
gnbd_import: created gnbd device gfs
gnbd_recvd: gnbd_recvd started
#
注視：此處的gnbd-server即是你寫在/etc/hosts裡面的冬冬
Node2:
# gnbd_import -v -i gnbd-server
gnbd_import: created directory /dev/gnbd
gnbd_import: created gnbd device gfs
gnbd_recvd: gnbd_recvd started
#
Node3:
# gnbd_import -v -i gnbd-server
gnbd_import: created directory /dev/gnbd
gnbd_import: created gnbd device gfs
gnbd_recvd: gnbd_recvd started
#
8.3、查看導入狀態信息 （三個節點都要做，我只寫一個節點）
# gnbd_import -v -l
Device name : gfs
----------------------
Minor # : 0
sysfs name : /block/gnbd0
Server : gnbd-server
Port : 14567
State : Close Connected Clear
Readonly : No
Sectors : 5879727
#
九、 建立gfs文件系統並且掛載
9.1、在三個節點載入gfs模塊
# modprobe gfs
# modprobe gfs
# modprobe gfs
9.2、在gnbd-server（我這裡也是node01節點）上建立gfs文件系統
# gfs_mkfs -p lock_dlm -t alpha_cluster:gfs -j 3 /dev/gnbd/gfs
This will destroy any data on /dev/gnbd/gfs.
It appears to contain a GFS filesystem.
Are you sure you want to proceed? y
Device: /dev/gnbd/gfs
Blocksize: 4096
Filesystem Size: 669344
Journals: 2
Resource Groups: 12
Locking Protocol: lock_dlm
Lock Table: alpha_cluster:gfs
Syncing...
All Done
#
9.3、在三個節點掛載文件系統
在三個節點的根目錄下都創建一個名為gfstest的目錄，然後
# mount -t gfs /dev/gnbd/gfs /gfstest
# mount -t gfs /dev/gnbd/gfs /gfstest
# mount -t gfs /dev/gnbd/gfs /gfstest
注視：要在node01掛載文件完全完成後，再在02節點上進行mount,02節點mount完成後，再在03節點進行mount
注視：我在進行node03的mount時出現了如下錯誤
# mount -t gfs /dev/gnbd/gfs /gfstest
mount: wrong fs type, bad option, bad superblock on /dev/gnbd/gfs,
or too many mounted file systems
經查，發現我是按照gfs_mkfs -p lock_dlm -t alpha_cluster:gfs -j 2 /dev/gnbd/gfs做的，是兩個節點的，後來從第九部開始重做並進行了umount，然後再mount就對了，希望對大家有所幫助，再次說明一下：
gfs_mkfs -p lock_dlm -t alpha_cluster:gfs -j 3 /dev/gnbd/gfs
其中的3表示三個節點，實際上的話你是幾個就寫幾個就可以了
十、到此大家就可以玩玩gfs了，不過如果你也是虛擬機器的話，會有點慢阿，祝大家好運
《解決方案》
感謝分享。。。
《解決方案》
謝謝斑竹,以後在實際的配置中有問題還要請教斑竹那
《解決方案》
thanks for sharing. good post !
《解決方案》
樓上版主就是準時啊，00:00
《解決方案》
我剛把GFS所需的RPM包裝完，一切正常，為什麼沒有cluster.conf文件，及cluster文件夾？
《解決方案》
如何增加結點
安裝完成後添加結點，如何增加結點？
《解決方案》
回復 7樓 gb8007 的帖子
在你圖片上的Cluster Nodes上加就可以了，另外你可參照這篇文章
http://www.redhat.com/docs/manuals/csgfs/browse/rh-cs-en/s1-clustertool-overview.html
《解決方案》
原帖由 zhangning_1999 於 2006-10-26 17:13 發表
在你圖片上的Cluster Nodes上加就可以了，另外你可參照這篇文章
http://www.redhat.com/docs/manuals/csgfs/browse/rh-cs-en/s1-clustertool-overview.html
我裝完以後，在管理界面中沒有 Edit Cluster Properties 這一選項，同時Cluster Nodes的左右鍵都沒有反應
我安裝的包為以下的，我是rh4U3，會不會是包裝少了的原因？
ccs-1.0.3-0.i686.rpm
ccs-devel-1.0.3-0.i686.rpm
cman-1.0.4-0.i686.rpm
cman-kernel-smp-2.6.9-43.8.i686.rpm
dlm-1.01.00-1.i686.rpm
dlm-devel-1.01.00-1.i686.rpm
dlm-kernel-smp-2.6.9-41.7.i686.rpm
fence-1.32.6-0.i686.rpm
GFS-6.1.3-0.i386.rpm
GFS-kernel-smp-2.6.9-49.1.i686.rpm
gulm-1.0.6-0.i686.rpm
iddev-2.0.0-3.i686.rpm
kernel-smp-2.6.9-34.EL.i686.rpm
lvm2-2.01.14-2.0.RHEL4.i386.rpm
magma-1.0.1-4.i686.rpm
magma-plugins-1.0.2-0.i386.rpm
perl-Net-Telnet-3.03-3.noarch.rpm
piranha-0.8.1-1.i386.rpm
rgmanager-1.9.43-0.i386.rpm
system-config-cluster-1.0.29-1.0.noarch.rpm
《解決方案》
回復 9樓 gb8007 的帖子
http://www.redhat.com/docs/manuals/csgfs/browse/rh-cs-en/s1-start-clustertool.html,這片文章的全文你先看一遍把
