?LVS是中國人發起的項目，真是意外呀!大家可以看http://www.douzhe.com/linuxtips/1665.html
我是從最初的HA(高可用性)開始的，別人的例子是用VMWARE，可以做試驗但不能實際應用，我又沒有光纖卡的Share Storage，于是就選用ISCSI，成功后又發現ISCSI+EXT3不能用于LVS，倒最后發現GFS可用，我最終成功配成可實際應用的LVS，前后斷斷續續花了四個月，走了很多彎路。我花了三天時間寫下這篇文章，希望對大家有用。
這里要感謝linuxfans.org、linuxsir.com、chinaunix.com以及其它很多網站，很多資料都是從他們的論壇上找到的。參考文檔及下載點
a.http://www.gyrate.org/misc/gfs.txt
b.http://www.redhat.com/docs/manuals/enterprise/RHEL-3-Manual/cluster-suite/index.html
http://www.redhat.com/docs/manuals/csgfs/admin-guide/index.html
c.ftp://ftp.redhat.com/pub/redhat/linux/updates/enterprise/3ES/en/RHGFS/SRPMS
d.http://distro.ibiblio.org/pub/linux/distributions/caoslinux/centos/3.1/contrib/i386/RPMS/
???????????????? LVS結構圖：??????????????eth0=10.3.1.101????????????eth0:1=10.3.1.254????????????????Load Balance?????????????????? Router??????????????eth1=192.168.1.71????????????eth1:1=192.168.1.1??????????????????|???????? |??????????????????|???????? |????????????????Real1??????Real2?????? eth0=192.168.1.68??eth0=192.168.1.67????????????(eth0 gateway=192.168.1.1)?????? eth1=192.168.0.1---eth1=192.168.0.2??????????????????(雙機互聯線)?????????????????????? |?????????????????????? |??????????????????????GFS???????????????????? ISCSI????????????????Share storage??????????????eth0=192.168.1.124
1.Setup ISCSI Server
Server: PIII 1.4,512M, Dell 1650,Redhat 9,IP=192.168.1.124
從http://iscsitarget.sourceforge.net/下載ISCSI TARGET的Source code
(http://sourceforge.net/project/showfiles.php?group_id=108475&package_id=117141)
我選了iscsitarget-0.3.8.tar.gz,要求kernel 2.4.29
從kernel.org下載kernel 2.4.29,解開編譯重啟后編譯安裝iscsitarget-0.3.8：
#make KERNELSRC=/usr/src/linux-2.4.29
#make KERNELSRC=/usr/src/linux-2.4.29 install
#cp ietd.conf /etc
#vi /etc/ietd.conf

# Example iscsi target configuration## Everything until the first target definition belongs# to the global configuration.# Right now this is only the user configuration used# during discovery sessions:# Users, who can access this target# (no users means anyone can access the target)User iscsiuser 1234567890abcTarget iqn.2005-04.com.my:storage.disk2.sys1.iraw1????????User iscsiuser 1234567890abc????????Lun 0 /dev/sda5 fileio????????Alias iraw1Target iqn.2005-04.com.my:storage.disk2.sys1.iraw2????????User iscsiuser 1234567890abc????????Lun 1 /dev/sda6 fileio????????Alias iraw2Target iqn.2005-04.com.my:storage.disk2.sys2.idisk????????User iscsiuser 1234567890abc????????Lun 2 /dev/sda3 fileio????????Alias idiskTarget iqn.2005-04.com.my:storage.disk2.sys2.icca????????User iscsiuser 1234567890abc????????Lun 3 /dev/sda7 fileio????????Alias icca

說明：password 長度必須不小于12個字符, Alias是別名, 不知為何這個別名在Client端顯示不出來.
分區：我只有一個SCSI盤，所以：

/dev/sda3: Share storage,容量越大越好/dev/sda5: raw1, 建Cluster要的rawdevice, 我給了900M/dev/sda6: raw2, 建Cluster要的rawdevice, 我給了900M/dev/sda7: cca, 建GFS要的,我給了64M(/dev/sda4是Extended分區,在其中建了sda5,6,7)

#Reboot，用service iscsi-target start啟ISCSI server(我覺得比建議的好，可以
用service iscsi-target status看狀態)
2.Setup ISCSI Client(on two real server)
Server: PIII 1.4,512M, Dell 1650,Redhat AS3U4(用AS3U5更好),2.4.21-27.EL
#vi /etc/iscsi.conf

DiscoveryAddress=192.168.1.124OutgoingUsername=iscsiuserOutgoingPassword=1234567890abcUsername=iscsiuserPassword=1234567890abcLoginTimeout=15IncomingUsername=iscsiuserIncomingPassword=1234567890abcSendAsyncTest=yes

#service iscsi restart
#iscsi-ls -l
..., 精簡如下:
/dev/sdb:iraw2
/dev/sdc:iraw1
/dev/sdd:idisk
/dev/sde:icca
注意: 在real server中ISCSI device的順序很重要，兩個real server中一定要一樣，如不一樣
　　　就改ISCSI Server中的設置，多試幾次
3.Install Redhat Cluster suite
先下載Cluster Suite的ISO, AS3的我是從ChinaUnix.net找到的下載點, 安裝clumanager和
redhat-config-cluster。沒有Cluster Suite的ISO也沒關系，從
ftp://ftp.redhat.com/pub/redhat/linux/updates/enterprise/3ES/en/RHCS/SRPMS/下載
clumanager-1.2.xx.src.rpm,redhat-config-cluster-1.0.x.src.rpm,編譯后安裝，應該更好:
#rpm -Uvh clumanager-1.2.26.1-1.src.rpm
#rpmbuild -bs /usr/src/redhat/SPECS/clumanager.spec
#rpmbuild --rebuild --target i686 /usr/src/redhat/SRPMS/clumanager-1.2.26.1-1.src.rpm
還有redhat-config-cluster-1.0.x.src.rpm，也裝好
4.Setup Cluster as HA module
詳細步驟我就不寫了，網上有很多文章，我也是看了別人的文章學會的，不過人家是用VMWARE，
而我是用真的機子+ISCSI，raw device就是/dev/sdb,/dev/sdc, 然后就
mount /dev/sdd /u01, mkfs.ext3 /u01 ......
設好后會發現ISCSI有問題：同時只能有一個Client聯接寫盤，如果
兩個Client同時聯ISCSI的Share Storge,一個Client寫，另一個Client是看不到的，而且此時文
件系統已經破壞了，Client重聯ISCSI時會發現文件是壞的，用fsck也修復不了。
ISCSI真的是雞肋嗎？
　NO！從GOOGLE上我終于查到ISCSI只有用Cluster File System才能真正用于Share Storage!
而Redhat買下的GFS就是一個!
5.Setup GFS on ISCSI
GFS只有Fedora Core4才自帶了，而GFS又一定要用到Cluster Suite產生的/etc/cluster.xml文件，
我沒見FC4有Cluster Suite,真不知Redhat給FC4帶GFS干嘛，饞人嗎？
好，閑話少說，下載：c處的GFS-6.0.2.20-2.src.rpm，按a處的gfs.txt編譯安裝，不過關于
cluster.ccs,fence.ccs,nodes.ccs的設置沒說，看b的文檔，我總算弄出來了，都存在
/root/cluster下，存在別的地方也行，不過我不知道有沒有錯，我沒有光纖卡，文檔又沒講ISCSI
的例子，不過GFS能啟動的。
#cat cluster.ccs

cluster {????????name = "Cluster_1"????????lock_gulm {????????????servers = ["cluster1", "cluster2"]????????????heartbeat_rate = 0.9????????????allowed_misses = 10????????}}

注：name就是Cluster Suite設置的Cluster name, servers就是Cluster member的Hostname，別忘
了加進/etc/hosts;allowed_misses我開始設為1，結果跑二天GFS就會死掉，改為10就沒死過了。
#cat fence.ccs

fence_devices{????????admin {??????????????agent = "fence_manual"????????}}

#cat nodes.ccs

nodes {?? cluster1 {??????ip_interfaces {???????? hsi0 = "192.168.0.1"??????}??????fence {???????? human {????????????admin {?????????????? ipaddr = "192.168.0.1"????????????}???????? }??????}?? }?? cluster2 {??????ip_interfaces {???????? hsi0 = "192.168.0.2"??????}??????fence {???????? human {????????????admin {?????????????? ipaddr = "192.168.0.2"????????????}???????? }??????}?? }}

注：ip就是心跳線的ip
這三個文件建在/root/cluster下，先建立Cluster Configuration System：
a.#vi /etc/gfs/pool0.cfg
poolname pool0
minor 1 subpools 1
subpool 0 8 1 gfs_data
pooldevice 0 0 /dev/sde1
b.#pool_assemble -a pool0
c.#ccs_tool create /root/cluster /dev/pool/pool0
d.#vi /etc/sysconfig/gfs
CCS_ARCHIVE="/dev/pool/pool0"
再Creating a Pool Volume，就是我們要的共享磁盤啦，
a.#vi /etc/gfs/pool1.cfg
poolname pool1
minor 2 subpools 1
subpool 0 128 1 gfs_data
pooldevice 0 0 /dev/sdd1
b.#pool_assemble -a pool1
c.#gfs_mkfs -p lock_gulm -t Cluster_1:gfs1 -j 8 /dev/pool/pool1
d.#mount -t gfs -o noatime /dev/pool/pool1 /u01
下面是個GFS的啟動腳本，注意real1和real2必須同時啟動lock_gulmd進程，第一臺lock_gulmd
會成為Server并等Client的lock_gulmd，幾十秒后沒有響應會fail，GFS啟動失敗。Redhat建議
GFS盤不要寫進/etc/fstab。
#cat gfstart.sh

#!/bin/shdepmod -amodprobe poolmodprobe lock_gulmmodprobe gfssleep 5service iscsi startsleep 20service rawdevices restartpool_assemble -a pool0pool_assemble -a pool1service ccsd startservice lock_gulmd startmount -t gfs /dev/pool/pool1 /s02 -o noatimeservice gfs status
6. Setup Linux LVS
LVS是章文嵩博士發起和領導的優秀的集群解決方案，許多商業的集群產品，比如RedHat的Piranha，Turbolinux公司的Turbo Cluster等，都是基于LVS的核心代碼的。
我的系統是Redhat AS3U4，就用Piranha了。從rhel-3-u5-rhcs-i386.iso安裝piranha-0.7.10-2.i386.rpm,ipvsadm-1.21-9.ipvs108.i386.rpm (http://distro.ibiblio.org/pub/linux/distributions/caoslinux/centos/3.1/contrib/i386/RPMS/) 裝完后service httpd start & service piranha-gui start，就可以從http://xx.xx.xx.xx:3636管理或設置了，當然了，手工改/etc/sysconfig/ha/lvs.cf也一樣。
#cat /etc/sysconfig/ha/lvs.cf

serial_no = 80primary = 10.3.1.101service = lvsrsh_command = sshbackup_active = 0backup = 0.0.0.0heartbeat = 1heartbeat_port = 1050keepalive = 6deadtime = 18network = natnat_router = 192.168.1.1 eth1:1nat_nmask = 255.255.255.0reservation_conflict_action = preemptdebug_level = NONEvirtual lvs1 {???? active = 1???? address = 10.3.1.254 eth0:1???? vip_nmask = 255.255.255.0???? fwmark = 100???? port = 80???? persistent = 60???? pmask = 255.255.255.255???? send = "GET / HTTP/1.0\r\n\r\n"???? expect = "HTTP"???? load_monitor = ruptime???? scheduler = wlc???? protocol = tcp???? timeout = 6???? reentry = 15???? quiesce_server = 1???? server Real1 {???????? address = 192.168.1.68???????? active = 1???????? weight = 1???? }???? server Real2 {???????? address = 192.168.1.67???????? active = 1???????? weight = 1???? }}virtual lvs2 {???? active = 1???? address = 10.3.1.254 eth0:1???? vip_nmask = 255.255.255.0???? port = 21???? send = "\n"???? use_regex = 0???? load_monitor = ruptime???? scheduler = wlc???? protocol = tcp???? timeout = 6???? reentry = 15???? quiesce_server = 0???? server ftp1 {???????? address = 192.168.1.68???????? active = 1???????? weight = 1???? }???? server ftp2 {???????? address = 192.168.1.67???????? active = 1???????? weight = 1???? }}

設置完后service pulse start, 別忘了把相關的client加進/etc/hosts
#iptables -t mangle -A PREROUTING -p tcp -d 10.3.1.254/32 --dport 80 -j MARK --set-mark 100
#iptables -t mangle -A PREROUTING -p tcp -d 10.3.1.254/32 --dport 443 -j MARK --set-mark 100
#iptables -A POSTROUTING -t nat -p tcp -s 10.3.1.0/24 --sport 20 -j MASQUERADE
運行以上三行命令并存入/etc/rc.d/rc.local，用ipvsadm看狀態：
#ipvsadm

IP Virtual Server version 1.0.8 (size=65536)Prot LocalAddress:Port Scheduler Flags??-> RemoteAddress:Port?????????? Forward Weight ActiveConn InActConnTCP??10.3.1.254:ftp wlc??-> cluster2:ftp??????????????????Masq????1??????0??????????0??-> cluster1:ftp??????????????????Masq????1??????0??????????0FWM??100 wlc persistent 60??-> cluster1:0????????????????????Masq????1??????0??????????0??-> cluster2:0????????????????????Masq????1??????0??????????0

注意：a.Firewall Mark可以不要，我反正是加了，文檔說有https的話加上，值我選了100，
b.Virtual IP別加進/etc/hosts，我上過當，80端口時有時無的，
c.eth0:1,eth1:1是piranha產生的，別自己手工設置，我干過這畫蛇添足的事，網上有
些帖子沒說清，最后是看Redhat的文檔才弄清楚的。
d.The LVS router can monitor the load on the various real servers by using
either rup or ruptime. If you select rup from the drop-down menu, each real
server must run the rstatd service. If you select ruptime, each real server
must run the rwhod service.Redhat的原話，就是如選rup的監控模式real server上
都要運行rstatd進程，如選ruptime就要運行rwhod進程。
e.Real Server同Router相聯的網卡的Gateway必須是Router的那塊網卡的VIP，舉本例：
Router的eth1同兩個real server的eth0相聯，如VIP eth1:1=192.168.1.1，則real
server 的eth0的Gateway=192.168.1.1
echo "1" > /proc/sys/net/ipv4/ip_forward
q:在兩臺real server上分別
#vi /usr/local/tomcat/conf/tomcat-users.xml
加下面一行，允許頁面管理：

r:在兩臺real server上分別
#service httpd restart
#/usr/local/tomcat/bin/startup.sh
s:打開http://1092.168.1.68:8080和http://1092.168.1.67:8080，選Tomcat Manager，用
manager/tomcat登錄，虛擬目錄/myjsp和/local應該Start了
在兩臺機子上分別打開網頁http://10.3.1.254，選WEB Local，可以看到一臺顯示：
"test page on real server 1"，另一臺為"test page on real server 2",同時在Router上
ipvsadm可以看到每個real server的聯接數
8.設置FTP服務
#vi /etc/vsftpd/vsftp.conf，在兩臺real server上分別加入以下幾行：

anon_root=/u01/ftprootlocal_root=/u01/ftprootsetproctitle_enable=YES

#service vsftpd start
現在LVM+GFS+ISCSI+TOMCAT就設置好了，我們可以用Apache Jmeter來測試LVM的性能，兩臺機子上分別運行jmeter，都指向10.3.1.254/myjsp/testoracle.jsp，各200個threads同時運行，在Router上用ipvsadm可以監控，Oracle Server的性能可要好，否則大量的http進程會hang在real server上，ipvsadm也會顯示有個real server失去了。測試時real server的CPU idle會降到70%，而Router的CPU idle幾乎不動。
??????????????????????????????????????????????????????文章引用自：大行的BLOG

集群LVS+GFS+ISCSI+TOMCAT