LB群集--lvs-dr模型

火星人 @ 2014-03-08 , reply:0


lvs簡介: LVS是一個開源的軟體,由畢業於國防科技大學的章文嵩博士於1998年5月創立,可以實現LINUX平台下的簡單負載均衡.LVS是Linux Virtual Server的縮寫,意思是Linux虛擬伺服器.

Lvs 的集群工作模式有3種:VS/NAT,vs/tun,vs/dr.

Lvs的調度演算法:

LVS的演算法分為兩大類:
靜態演算法:只是根據演算法進行調度並不考慮後端REALSERVER的實際連接情況
rr-論調演算法,假如有兩台伺服器A,B,第一個請求給A,第二個給B,第三個給A依次往複
wrr-加權論調,假如有兩台伺服器A,B,A的性能是B的兩倍,則在論調的同時給A上面分配的請求也大致會是B上面的兩倍
dh-假如調度器的後面是兩台緩存伺服器A,B而不是真正的REALSERVER,則會儘可能的把相同請求或者把同一用戶的請求轉發到同一個緩存伺服器上面以提高緩存命中率
sh-假如公司有兩台防火牆讓員工上網,則會把某個員工往外的訪問及向內返回的請求結果定向到同一台防火牆上面,方便防火牆做established的狀態檢測

動態演算法:前端的調度器會根據後端REALSERVER的實際連接情況來分配請求
活動鏈接:當前有數據包傳輸
非活動鏈接:當前連接出於建立狀態但是沒有數據傳輸
lc-同時檢查後端REALSERVER上面活動狀態和非活動狀態的連接數使用(活動連接數*256 非活動連接數)數字小的將接收下次訪問請求
wlc-加權的lc,使用(活動連接數*256 非活動連接數)/權重,數字小的將接收下次訪問請求,是最常用的演算法
sed-不考慮非活動狀態,使用(活動狀態 1)*256,數字小的將接收下次訪問請求, 1主要是為了提高權重大的伺服器的響應能力
nq-假設有兩台伺服器A,B,權重比為10:1,按照sed演算法,只有當A伺服器已經響應了10個請求之時兩者的計算數值才相同,為了避免權重小的伺服器過於空閑,nq沿用sed演算法但是確保讓每個伺服器都不空閑,只有在不考慮非活動連接的情況下nq才能取代wlc演算法


lblc-在dh的基礎上面考慮後台伺服器的負載能力
lblcr-在lblc的基礎上,假設有A,B兩台緩存伺服器,某個用戶第一次訪問被重定向到A,第二次訪問時A負載很大,B過於空閑這時也會打破原來的規則把客戶的第二次訪問重定向給B

Lvs的dr模型

一:Web1的搭建 192.168.0.101

[root@zzu ~]# ifconfig lo:0 192.168.0.100 netmask 255.255.255.255

配置一個vip地址

[root@zzu ~]# ifconfig

eth0 Link encap:Ethernet HWaddr 00:0C:29:4B:61:16

inet addr:192.168.0.101 Bcast:192.168.0.255 Mask:255.255.255.0

inet6 addr: fe80::20c:29ff:fe4b:6116/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:3707 errors:0 dropped:0 overruns:0 frame:0

TX packets:915 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:354625 (346.3 KiB) TX bytes:127781 (124.7 KiB)

Interrupt:67 Base address:0x2000

lo:0 Link encap:Local Loopback

inet addr:192.168.0.100 Mask:255.255.255.255

UP LOOPBACK RUNNING MTU:16436 Metric:1

設置real伺服器的arp選項保證在進行arp解析的時候只有director的vip進行響應

[root@zzu ~]# echo "net.ipv4.conf.all.arp_announce = 2" >>/etc/sysctl.conf

[root@zzu ~]# echo "net.ipv4.conf.lo.arp_announce = 2" >>/etc/sysctl.conf

[root@zzu ~]# echo "net.ipv4.conf.all.arp_ignore = 1" >>/etc/sysctl.conf

[root@zzu ~]# echo "net.ipv4.conf.lo.arp_ignore = 1" >>/etc/sysctl.conf

設置一條特殊的路由,保證在回復客戶端時使用的是vip的地址

[root@zzu ~]# route add -host 192.168.0.100 dev lo:0

[root@zzu ~]# route -n

Kernel IP routing table

Destination Gateway Genmask Flags Metric Ref Use Iface

192.168.0.100 0.0.0.0 255.255.255.255 UH 0 0 0 lo

192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0

169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0

[root@zzu Server]# rpm -ivh httpd-2.2.3-31.el5.i386.rpm

[root@zzu Server]# cd /var/www/html/

[root@zzu html]# vim index.html

web1

[root@zzu html]# links http://192.168.0.101

二:web2的搭建 192.168.0.102

[root@zzu ~]# ifconfig lo:0 192.168.0.100 netmask 255.255.255.255

配置一個vip地址

[root@zzu ~]# ifconfig

eth0 Link encap:Ethernet HWaddr 00:0C:29:4B:61:16

inet addr:192.168.0.102 Bcast:192.168.0.255 Mask:255.255.255.0

inet6 addr: fe80::20c:29ff:fe4b:6116/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:3707 errors:0 dropped:0 overruns:0 frame:0

TX packets:915 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:354625 (346.3 KiB) TX bytes:127781 (124.7 KiB)

Interrupt:67 Base address:0x2000

lo:0 Link encap:Local Loopback

inet addr:192.168.0.100 Mask:255.255.255.255

UP LOOPBACK RUNNING MTU:16436 Metric:1

設置real伺服器的arp選項保證在進行arp解析的時候只有director的vip進行響應

[root@zzu ~]# echo "net.ipv4.conf.all.arp_announce = 2" >>/etc/sysctl.conf

[root@zzu ~]# echo "net.ipv4.conf.lo.arp_announce = 2" >>/etc/sysctl.conf

[root@zzu ~]# echo "net.ipv4.conf.all.arp_ignore = 1" >>/etc/sysctl.conf

[root@zzu ~]# echo "net.ipv4.conf.lo.arp_ignore = 1" >>/etc/sysctl.conf

[root@zzu ~]# route add -host 192.168.0.100 dev lo:0

設置一條特殊的路由,保證在恢復客戶端使使用的是vip的地址

[root@zzu ~]# route -n

Kernel IP routing table

Destination Gateway Genmask Flags Metric Ref Use Iface

192.168.0.100 0.0.0.0 255.255.255.255 UH 0 0 0 lo

192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0

169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0

[root@zzu Server]# rpm -ivh httpd-2.2.3-31.el5.i386.rpm

[root@zzu Server]# cd /var/www/html/

[root@zzu html]# vim index.html

Web2

[root@zzu html]# links http://192.168.0.102

三:director伺服器的搭建

[root@zzu ~]# yum install ipvsadm*

四:lvs-dr模型下rr的測試

[root@zzu ~]# ipvsadm -A 192.168.0.100:80 -s rr

unexpected argument 192.168.0.100:80

[root@zzu ~]# ipvsadm -A -t 192.168.0.100:80 -s rr

[root@zzu ~]# ipvsadm -a -t 192.168.0.100:80 -r 192.168.0.101 -g

[root@zzu ~]# ipvsadm -a -t 192.168.0.100:80 -r 192.168.0.102 -g

[root@zzu ~]# ipvsdm –ln

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port Forward Weight ActiveConn InActConn

TCP 192.168.0.100:80 rr

-> 192.168.0.102:80 Route 1 0 0

-> 192.168.0.101:80 Route 1 0 0

[root@zzu ~]# ipvsadm -ln

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port Forward Weight ActiveConn InActConn

TCP 192.168.0.100:80 rr

-> 192.168.0.102:80 Route 1 0 6

-> 192.168.0.101:80 Route 1 0 6

五:lvs-dr模型下rr的測試 (ppc)

[root@zzu ~]# ipvsadm -A -t 192.168.0.100:80 -s rr

-p 300

[root@zzu ~]# ipvsadm -a -t 192.168.0.100:80 -r 192.168.0.102 -g

[root@zzu ~]# ipvsadm -a -t 192.168.0.100:80 -r 192.168.0.101 –g

[root@zzu ~]# ipvsadm -ln

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port Forward Weight ActiveConn InActConn

TCP 192.168.0.100:80 rr persistent 300

-> 192.168.0.101:80 Route 1 0 0

-> 192.168.0.102:80 Route 1 0 0

[root@zzu ~]# ipvsadm -ln

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port Forward Weight ActiveConn InActConn

TCP 192.168.0.100:80 rr persistent 300

-> 192.168.0.101:80 Route 1 0 5

-> 192.168.0.102:80 Route 1 0 0

[root@zzu ~]# ipvsadm -A -t 192.168.0.100:22 -s rr -p 300

[root@zzu ~]# ipvsadm -a -t 192.168.0.100:22 -r 192.168.0.101 -g

[root@zzu ~]# ipvsadm -a -t 192.168.0.100:22 -r 192.168.0.102 -g

[root@zzu ~]# ipvsadm -ln

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port Forward Weight ActiveConn InActConn

TCP 192.168.0.100:80 rr persistent 300

-> 192.168.0.101:80 Route 1 0 5

-> 192.168.0.102:80 Route 1 0 0

TCP 192.168.0.100:22 rr persistent 300

-> 192.168.0.102:22 Route 1 0 0

-> 192.168.0.101:22 Route 1 0 0

[root@zzu ~]#

[root@zzu ~]# ipvsadm -ln

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port Forward Weight ActiveConn InActConn

TCP 192.168.0.100:80 rr persistent 300

-> 192.168.0.101:80 Route 1 0 5

-> 192.168.0.102:80 Route 1 0 0

TCP 192.168.0.100:22 rr persistent 300

-> 192.168.0.102:22 Route 1 0 10

-> 192.168.0.101:22 Route 1 0 0

六:lvs-dr模型下rr的測試 (pcc)

[root@zzu ~]# ipvsadm -A -t 192.168.0.100:0 -s rr -p 300

[root@zzu ~]# ipvsadm -a -t 192.168.0.100:0 -r 192.168.0.102 -g

[root@zzu ~]# ipvsadm -a -t 192.168.0.100:0 -r 192.168.0.101 -g

[root@zzu ~]# ipvsadm -ln

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port Forward Weight ActiveConn InActConn

TCP 192.168.0.100:0 rr persistent 300

-> 192.168.0.101:0 Route 1 0 0

-> 192.168.0.102:0 Route 1 0 0

[root@zzu ~]# ipvsadm -ln

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port Forward Weight ActiveConn InActConn

TCP 192.168.0.100:0 rr persistent 300

-> 192.168.0.101:0 Route 1 0 9

-> 192.168.0.102:0 Route 1 0 0

七:lvs-dr模型下帶防火牆標記的持續連接(80和443)

將http和https打上標籤

1:搭建https伺服器web1

[root@zzu ~]# yum install openssl*

[root@zzu ~]# cd /etc/pki/

[root@zzu pki]# ll

drwx------ 3 root root 4096 2012-02-08 CA

drwxr-xr-x 2 root root 4096 2012-02-08 nssdb

drwxr-xr-x 2 root root 4096 2012-02-08 rpm-gpg

drwxr-xr-x 5 root root 4096 2012-02-08 tls

[root@zzu pki]# vim tls/openssl.cnf

45 dir = /etc/pki/CA

88 countryName = optional

89 stateOrProvinceName = optional

90 organizationName = optional

136 countryName_default = CN

141 stateOrProvinceName_default = beijing

144 localityName_default = Beijing

2:創建3個目錄和兩個文件

[root@zzu pki]# cd CA

[root@zzu CA]# mkdir certs newcerts crl

[root@zzu CA]# touch index.txt serial

[root@zzu CA]# echo "01" >>serial

[root@zzu CA]# openssl genrsa 1024 >private/cakey.pem

Generating RSA private key, 1024 bit long modulus

..............

.....................................................................

e is 65537 (0x10001)

[root@zzu CA]# openssl req -new -key private/cakey.pem -days 3650 -x509 -out cacert.pem

Country Name (2 letter code) [CN]:

State or Province Name (full name) [beijing]:

Locality Name (eg, city) [beijing]:

Organization Name (eg, company) [My Company Ltd]:qinghua

Organizational Unit Name (eg, section) []:qinghua

Common Name (eg, your name or your server's hostname) []:www.qinghua.com

2:為http辦法證書

[root@zzu ~]# mkdir -pv /etc/httpd/certs

[root@zzu ~]# cd /etc/httpd/certs/

[root@zzu certs]# openssl genrsa 1024 > httpd.key

[root@zzu certs]# openssl req -new -key httpd.key -out httpd.csr

Country Name (2 letter code) [CN]:

State or Province Name (full name) [beijing]:

Locality Name (eg, city) [beijing]:

Organization Name (eg, company) [My Company Ltd]:bjdx

Organizational Unit Name (eg, section) []:sec

Common Name (eg, your name or your server's hostname) []:www.bj.com

[root@zzu certs]# openssl ca -in httpd.csr -out httpd.cert

Using configuration from /etc/pki/tls/openssl.cnf

Check that the request matches the signature

Signature ok

Certificate Details:

Serial Number: 1 (0x1)

Validity

Not Before: Feb 7 13:28:38 2012 GMT

Not After : Feb 6 13:28:38 2013 GMT

Subject:

countryName = CN

stateOrProvinceName = beijing

organizationName = bjdx

organizationalUnitName = sec

commonName = www.bj.com

3:綁緊證書文件

[root@zzu Server]# rpm -ivh distcache-1.4.5-14.1.i386.rpm

Preparing... ########################################### [100%]

1:distcache ########################################### [100%]

[root@zzu Server]# rpm -ivh mod_ssl-2.2.3-31.el5.i386.rpm

Preparing... ########################################### [100%]

1:mod_ssl ########################################### [100%]

[root@zzu ~]# cd /etc/httpd/certs/

[root@zzu certs]# cp /etc/pki/CA/cacert.pem ./

[root@zzu certs]# ll

-rw-r--r-- 1 root root 1168 02-07 21:34 cacert.pem

-rw-r--r-- 1 root root 0 02-07 21:28 httpd.cert

-rw-r--r-- 1 root root 643 02-07 21:27 httpd.csr

-rw-r--r-- 1 root root 887 02-07 21:26 httpd.key

[root@zzu ~]# vim /etc/httpd/conf.d/ssl.conf

112 SSLCertificateFile /etc/http/certs/httpd.cert

119 SSLCertificateKeyFile /etc/http/certs/httpd.key

128 SSLCertificateChainFile /etc/http/certs/cacert.pem

[root@zzu certs]# service httpd restart 重新啟動www服務

Stopping httpd: [FAILED]

Starting httpd: [ OK ]

4.搭建https 伺服器Web2

[root@server2 ~]# mkdir -pv /etc/httpd/certs

mkdir: created directory `/etc/httpd/certs'

[root@server2 ~]# cd /etc/httpd/certs

[root@server2 certs]# ll

total 0

[root@server2 certs]# scp 192.168.0.101:/etc/httpd/certs/* ./

The authenticity of host '192.168.0.101 (192.168.0.101)' can't be established.

RSA key fingerprint is 91:71:d8:d9:f2:63:a6:78:2f:0c:1e:e8:32:aa:55:3c.

Are you sure you want to continue connecting (yes/no)? y

Please type 'yes' or 'no': yes

Warning: Permanently added '192.168.0.101' (RSA) to the list of known hosts.

root@192.168.0.101's password:

cacert.pem 100% 1168 1.1KB/s 00:00

httpd.cert 100% 3082 3.0KB/s 00:00

httpd.csr 100% 643 0.6KB/s 00:00

httpd.key 100% 887 0.9KB/s 00:00

[root@server2 certs]# ll

-rw-r--r-- 1 root root 1168 Apr 30 17:33 cacert.pem

-rw-r--r-- 1 root root 3082 Apr 30 17:33 httpd.cert

-rw-r--r-- 1 root root 643 Apr 30 17:33 httpd.csr

-rw-r--r-- 1 root root 887 Apr 30 17:33 httpd.key

[root@server2 Server]# rpm -ivh distcache-1.4.5-14.1.i386.rpm

[root@server2 Server]# rpm -ivh mod_ssl-2.2.3-31.el5.i386.rpm

[root@server2~]#scp192.168.0.101:/etc/httpd/conf.d/ssl.conf /etc/httpd/conf.d/ssl.conf

root@192.168.0.101's password:

ssl.conf 100% 9655 9.4KB/s 00:00

[root@server2 ~]# service httpd restart

Stopping httpd: [ OK ]

Starting httpd: httpd: apr_sockaddr_info_get() failed for server2

httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName

[ OK ]

5:設置director伺服器

[root@zzu ~]# iptables -t mangle -A PREROUTING -d 192.168.0.100 -p tcp --dport 80 -j MARK --set-mark 1

[root@zzu ~]# iptables -t mangle -A PREROUTING -d 192.168.0.100 -p tcp --dport 443 -j MARK --set-mark 1

[root@zzu ~]# iptables -t mangle -L

Chain PREROUTING (policy ACCEPT)

target prot opt source destination

MARK tcp -- anywhere 192.168.0.100 tcp dpt:http MARK set 0x1

MARK tcp -- anywhere 192.168.0.100 tcp dpt:https MARK set 0x1

[root@zzu ~]# ipvsadm -A -f 1 -s rr -p 1800

[root@zzu ~]# ipvsadm -a -f 1 -r 192.168.0.101 -g

[root@zzu ~]# ipvsadm -a -f 1 -r 192.168.0.102 -g

[root@zzu ~]# ipvsadm -ln

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port Forward Weight ActiveConn InActConn

FWM 1 rr persistent 1800

-> 192.168.0.102:0 Route 1 0 0

-> 192.168.0.101:0 Route 1 0 0

訪問測試:

八:lvs-dr模型下帶防火牆標記的持續連接(ftp的被動方式)

ftp1 伺服器

[root@zzu Server]# rpm -ivh vsftpd-2.0.5-16.el5.i386.rpm

Preparing... ########################################### [100%]

1:vsftpd ########################################### [100%]

[root@zzu ~]# cd /var/ftp/

[root@zzu ftp]# mkdir ftp1

[root@zzu ftp]# ll

total 8

drwxr-xr-x 2 root root 4096 Feb 7 22:27 ftp1

drwxr-xr-x 3 root root 4096 Feb 7 22:26 pub

[root@zzu ~]# vim /etc/vsftpd/vsftpd.conf

12 pasv_min_port=10000

13 pasv_max_port=20000

14 pasv_enable=YES

[root@zzu ftp]# service vsftpd restart

Shutting down vsftpd: [ OK ]

Starting vsftpd for vsftpd: [ OK ]

ftp2伺服器同一

director伺服器的設置

[root@zzu ~]# iptables -t mangle -A PREROUTING -p tcp -d 192.168.0.100/32 --dport 10000:20000 -j MARK --set-mark 21

[root@zzu ~]# iptables -t mangle -A PREROUTING -p tcp -d 192.168.0.100/32 --dport 21 -j MARK --set-mark 21

[root@zzu ~]# iptables -t mangle -L

Chain PREROUTING (policy ACCEPT)

target prot opt source destination

MARK tcp -- anywhere 192.168.0.100 tcp dpts:ndmp:dnp MARK set 0x15

MARK tcp -- anywhere 192.168.0.100 tcp dpt:ftp MARK set 0x15

[root@zzu ~]# ipvsadm -A -f 21 -s rr -p 1800

[root@zzu ~]# ipvsadm -a -f 21 -r 192.168.0.101 -g

[root@zzu ~]# ipvsadm -a -f 21 -r 192.168.0.102 -g

[root@zzu ~]# ipvsadm -ln

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port Forward Weight ActiveConn InActConn

FWM 21 rr persistent 1800

-> 192.168.0.102:0 Route 1 0 0

-> 192.168.0.101:0 Route 1 0 0

[root@zzu ~]# ipvsadm -ln

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port Forward Weight ActiveConn InActConn

FWM 21 rr persistent 1800

-> 192.168.0.102:0 Route 1 0 8

-> 192.168.0.101:0 Route 1 0 0

[root@zzu ~]# ipvsadm –lcn 我們在director上查看鏈接的狀態

IPVS connection entries

pro expire state source virtual destination

TCP 00:19 FIN_WAIT 192.168.0.5:1309 192.168.0.100:21 192.168.0.102:21

TCP 00:19 FIN_WAIT 192.168.0.5:1310 192.168.0.100:10499 192.168.0.102:10499

TCP 00:14 FIN_WAIT 192.168.0.5:1306 192.168.0.100:14859 192.168.0.102:14859

TCP 00:14 FIN_WAIT 192.168.0.5:1305 192.168.0.100:21 192.168.0.102:21

IP 28:19 ERR! 192.168.0.5:0 0.0.0.21:0 192.168.0.102:0




[火星人 via ] LB群集--lvs-dr模型已經有101次圍觀

http://www.coctec.com/docs/linux/show-post-45816.html