1、Keepalived+LVS 双热互备 Nginx 对 realserver 网站负载均衡体系架构： 在 Keepalived + Nginx 高可靠负载均衡架构中，keepalived 负责实现 High-availability (HA) 功能控制前端机VIP(虚拟网络地址)，当有设备发生故障时，热备服务器可以瞬间将 VIP 自动切换过来，实际运行中体验只有 2 秒钟切换时间， ，后端机 Nginx 实现对 realserver 七层负载均衡功能，日后也可随着业务量增大随意扩展，DNS 负责前端 VIP的负载均衡。硬件环境：vmware 7.1.2 网卡 Host-only 模式接入系统软
2、件环境：两台 DR 安装：centos4.3 + lnmp (linux version 2.6.9-89.31.1.el Red Hat 3.4.6-11) 32 位，分别命名为: LVS_DR_MASTER，LVS_DR_BACKUP；默认 LVS_DR_MASTER 作主机，LVS_DR_BACKUP 作热备；realserver 作为后端应用服务器。Direct Routing：直接路由模式CLIENT：地址：200.200.200.2子网掩码：255.255.255.0默认网关：200.200.200.1LVS_DR_MASTER：ip：200.200.200.10（主服务器）子网掩
3、码：255.255.255.0默认网关：200.200.200.1vip1（LVS_DR_MASTER）：200.200.200.200LVS_DR_BACKUP：ip：200.200.200.11（备服务器）子网掩码：255.255.255.0默认网关：200.200.200.1vip1（LVS_DR_BACKUP）：200.200.200.200RealServer1eth0：200.200.200.20eth0:0：200.200.200.200RealServer2eth0：200.200.200.21eth0:0：200.200.200.199准备工作：分别在每台服务器安装 ngin
4、x关闭所有服务器防火墙分别在每台服务器创建网页显示文件LVS_DR_MASTER：echo “LVS_DR_MASTER 200.200.200.10“ /home/wwwroot/index.htmlLVS_DR_BACKUP：echo “LVS_DR_BACKUP 200.200.200.11“ /home/wwwroot/index.htmlrealserver1：echo “realserver1 200.200.200.20“ /home/wwwroot/index.htmlrealserver2：echo “realserver2 200.200.200.21“ /home/www
5、root/index.html同步服务器的系统时间# ntpdate time.nist.gov8 Dec 11:56:59 ntpdate10531: adjust time server 192.43.244.18 offset 0.009136 sec查看当前 kernels 环境# uname aLinux localhost.localdomain 2.6.9-89.31.1.EL #1 Tue Oct 19 16:47:55 EDT 2010 i686 i686 i386 GNU/Linux软连接当前 kernels 目录到/usr/src/linux ，否则无法支持 IPVS#
6、ln -s /usr/src/kernels/2.6.9-89.31.1.EL-i686/ /usr/src/linux下载：# wget http:/www.linuxvirtualserver.org/software/kernel-2.6/ipvsadm-1.24.tar.gz# wget http:/www.keepalived.org/software/keepalived-1.1.20.tar.gz安装 ipvsadm（lvs 管理查看工具）：# tar zxvf ipvsadm-1.24.tar.gz# cd ipvsadm-1.24# make# make install查看
7、ipvsadm 是否安装正确# watch ipvsadm lnEvery 2.0s: ipvsadm -ln Tue Dec 14 12:59:18 2010IP Virtual Server version 1.2.0 (size=4096)Prot LocalAddress:Port Scheduler Flags- RemoteAddress:Port Forward Weight ActiveConn InActConn安装 keepalived（HA）：# tar zxvf keepalived-1.1.20.tar.gz# cd keepalived-1.1.20注意项Cento
8、s5.0 以下需要修改以下 configure 才能通过。# vi /usr/src/linux/include/linux/types.h/*typedef _u16 _bitwise _sum16;typedef _u32 _bitwise _wsum;*/# ./configure -prefix=/usr/local/keepalived看到提示如下状态Keepalived configuration-Keepalived version : 1.1.20Compiler : gccCompiler flags : -g -O2Extra Lib : -lpopt -lssl -lcr
9、yptoUse IPVS Framework : YesIPVS sync daemon support : YesUse VRRP Framework : YesUse Debug flags : No# make# make install# cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/rc.d/init.d/# cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/# mkdir /etc/keepalived# cp /usr/local/k
10、eepalived/etc/keepalived/keepalived.conf /etc/keepalived/# cp /usr/local/keepalived/sbin/keepalived /usr/sbin/编写 LVS_DR_MASTER keepalived 配置文件# vi /usr/local/keepalived/etc/keepalived/keepalived.confglobal_defsnotification_notification_email_from smtp_server smtp_connect_timeout 30router_id LVS_DEVE
11、L /LVS 负载均衡标识，在一个网络内，它是唯一标识vrrp_script chk_http script “/usr/local/keepalived/nginx_pid.sh“ /监控脚本interval 10 /监控时间，以秒为单位weight 1 /权重值，数值越大权重越高vrrp_instance VI_1state MASTER /实例状态 state ，只有 MASTER,BACKUP 两种必需大写单词interface eth0 /监控网卡virtual_router_id 51/虚拟路由编号，主辅要一致priority 100 /权重值 MASTER 一定要大于 BACKU
12、Padvert_int 1 /检查间隔时间，单位为 1 秒authenticationauth_type PASS /验证类型主要有 PASS、AH 两种，通常使用的类型为 PASS，据说 AH 使用时有问题auth_pass 1111 /验证密码为明文，主从服务器要一致track_script chk_http /执行监控的服务virtual_ipaddress200.200.200.200 /1_vip， 定义虚拟 IP，可以有多个，分行写入#定义 virtual_server (HTTP | 80)virtual_server 200.200.200.200 80 delay_loop
13、6 # service polling 的 delay 时间lb_algo rr #负载调度算法，常用 wlc、rrlb_kind DR #负载均衡转发规则，一般有 DR、NAT、TUNpersistence_timeout 50 #会话保持时间，单位为秒protocol TCP #协议类型(TCP|UDP)#定义 rs1，每一个 rs 都需要下面的一个配置段real_server 200.200.200.20 80 weight 1 #权值 默认 1，0 为失效，值越高权重越高TCP_CHECK #TCP 方式的健康检查connect_timeout 3 #连接超时时间nb_get_retr
14、y 3 #重试次数delay_before_retry 3 #重试间隔connect_port 80 #健康检查端口# 定义 rs2real_server 200.200.200.21 80 weight 1TCP_CHECK connect_timeout 3nb_get_retry 3delay_before_retry 3connect_port 80编写 LVS_DR_BACKUP keepalived 配置文件# vi /usr/local/keepalived/etc/keepalived/keepalived.confglobal_defsnotification_notific
15、ation_email_from smtp_server smtp_connect_timeout 30router_id LVS_DEVELvrrp_script chk_http script “/usr/local/keepalived/nginx_pid.sh“ interval 10weight 1vrrp_instance VI_1state BACKUPinterface eth0virtual_router_id 51priority 99advert_int 1authenticationauth_type PASSauth_pass 1111track_script chk
16、_http /执行监控的服务virtual_ipaddress200.200.200.200 /1_vipvirtual_server 200.200.200.200 80 delay_loop 6lb_algo rrlb_kind DRpersistence_timeout 50protocol TCPreal_server 200.200.200.20 80 weight 1TCP_CHECK connect_timeout 3nb_get_retry 3delay_before_retry 3connect_port 80real_server 200.200.200.21 80 wei
17、ght 1TCP_CHECK connect_timeout 3nb_get_retry 3delay_before_retry 3connect_port 80服务层检查脚本补充当 keepalived 发现当 LVS_DR_MASTER 服务器 nginx 无法正常使用时， keepalived 是无法检测到服务层故障来切换到LVS_DR_BACKUP 服务器，我认为如果 nginx 服务挂掉了，我觉得就很难再起来，所以我把 keepalived 也杀掉了，再有，类似 nagios 服务监控软件也会给你报警。# vi /usr/local/keepalived/nginx_pid.sh#!
18、/bin/shA=ps -C nginx -no-header |wc -l /查看是否有 nginx 进程数并把值赋给变量 Aif $A -eq 0 ;then /如果没有进程则值得为零/usr/local/nginx/sbin/nginxsleep 5if ps -C nginx -no-header |wc -l -eq 0 ;thenkillall keepalived /结束 keepalived 进程fifi提示：要给/usr/local/keepalived/nginx_pid.sh 加入可执行权限# chmod 744 /usr/local/keepalived/nginx_p
19、id.sh注意：一定要在主服务器 keepalived.conf 中相应 LVS_DR_MASTER 地方加入内容（具体见先前配置）1.vrrp_script chk_http script “/usr/local/keepalived/nginx_pid.sh“ /监控脚本interval 10 /监控时间weight 1 /权重值，数值越大权重越高2.track_script chk_http /执行监控的服务启动 keepalived 服务# service keepalived startStarting keepalived: OK 建议使用：# /usr/local/keepali
20、ved/sbin/keepalived -D -f /etc/keepalived/keepalived.conf-D 显示在日志记录-f 指定配置文件目录确认 keepalived 已启动# ps -aux|grep keepalivedWarning: bad syntax, perhaps a bogus -? See /usr/share/doc/procps-3.2.3/FAQroot 5227 0.0 0.2 4896 696 ? Ss 18:15 0:00 keepalived -Droot 5228 0.0 0.4 4948 1276 ? S 18:15 0:00 keepal
21、ived -Droot 5229 0.0 0.4 4948 1036 ? S 18:15 0:00 keepalived -Droot 5654 0.0 0.2 3820 664 pts/1 S+ 18:19 0:00 grep keepalived设置 keepalived 随服务器一起启动# echo “/usr/local/keepalived/sbin/keepalived -D -f /etc/keepalived/keepalived.conf” /etc/rc.d/rc.local所有 realserver 添加一个监听 IP 地址和一条路由：# vi /usr/local/sb
22、in/realserver#!/bin/bash#/usr/local/sbin/realserverSNS_VIP=200.200.200.200. /etc/rc.d/init.d/functionscase “$1“ instart)ifconfig lo:0 $SNS_VIP netmask 255.255.255.255 broadcast $SNS_VIP/sbin/route add -host $SNS_VIP dev lo:0echo “1“ /proc/sys/net/ipv4/conf/lo/arp_ignoreecho “2“ /proc/sys/net/ipv4/co
23、nf/lo/arp_announceecho “1“ /proc/sys/net/ipv4/conf/all/arp_ignoreecho “2“ /proc/sys/net/ipv4/conf/all/arp_announcesysctl -p /dev/null 2;stop)ifconfig lo:0 downroute del $SNS_VIP /dev/null 2;*)echo “Usage: $0 start|stop“exit 1esacexit 0赋给脚本可执行权限# chmod 744 /usr/local/sbin/realserver让脚本最系统一起启动echo “/u
24、sr/local/sbin/realserver start” /etc/rc.d/rc.local增加 /etc/network/interfaces 固定 IP 和轮循 lo:0# vi /etc/sysconfig/network-scripts/ifcfg-lo:0DEVICE=lo:0BOOTPROTO=staticIPADDR=200.200.200.200NETMASK=255.255.255.0ONBOOT=yesGATEWAY=200.200.200.1验证测试1. 当 LVS_DR_MASTER、LVS_DR_BACKUP 服务器 nginx 均正常工作时， CLIENT
25、通过浏览器访问http:/200.200.200.10LVS_DR_MASTER 200.200.200.10http:/200.200.200.11LVS_DR_BACKUP 200.200.200.11http:/200.200.200.20realserver1 200.200.200.20http:/200.200.200.21realserver2 200.200.200.21http:/200.200.200.200realserver1 200.200.200.202. 当 LVS_DR_MASTER 服务器 nginx 出现故障，LVS_DR_BACKUP 正常工作时，CLIE
26、NT 通过浏览器访问http:/200.200.200.10无法访问http:/200.200.200.11LVS_DR_BACKUP 200.200.200.11http:/200.200.200.20realserver1 200.200.200.20http:/200.200.200.21realserver2 200.200.200.21http:/200.200.200.200realserver1 200.200.200.203. 当 LVS_DR_MASTER 正常工作时，LVS_DR_BACKUP 服务器 nginx 出现故障，CLIENT 通过浏览器访问http:/200.2
27、00.200.10LVS_DR_MASTER 200.200.200.10http:/200.200.200.11无法访问http:/200.200.200.20realserver1 200.200.200.20http:/200.200.200.21realserver2 200.200.200.21http:/200.200.200.200realserver1 200.200.200.204. 当 LVS_DR_MASTER、LVS_DR_BACKUP 服务器 nginx 均出现故障时，CLIENT 通过浏览器访问http:/200.200.200.10无法访问http:/200.20
28、0.200.11无法访问http:/200.200.200.20realserver1 200.200.200.20http:/200.200.200.21realserver2 200.200.200.21http:/200.200.200.200无法访问为了配合 LVS 平台，还需如下操作：1、安装 RRDTOOL使用 yum 安装：在/etc/yum.repos.d/目录下新建 dag.repo 文件在文件中输入源地址：# vi dag.repodagname=Dag RPM Repository for Red Hat Enterprise Linuxbaseurl=http:/apt
29、.sw.be/redhat/el$releasever/en/$basearch/daggpgcheck=1gpgkey=http:/ yum -y install rrdtool这样就完成了 RRDTool 安装2、配置 master 端流量 rrd 数据采集# cd /home# wget http:/ tar -zxvf lvs-rrd.tar.gz# mv /home/lvs-rrd /usr/local/keepalived/# vi /usr/local/keepalived/lvs-rrd/lvs.rrd.update根据用户实际情况来修改如下参数RRDTOOL=“/usr/bi
30、n/rrdtool“IPVSADM=“/sbin/ipvsadm“WORKDIR=“/usr/local/keepalived/lvs-rrd/performance“3、定制采集作业*/2 * * * * /usr/local/keepalived/lvs-rrd/lvs.rrd.update 2 /dev/null /dev/null4、LVS 管理平台端会定期来捉取 performance 中的*.rrd 文件，用户可以通过 ssh 或 ftp，系统默认采用 rsync+ssh 方式，配置可参考 实践 rsync+ssh 实现代码同步# ipvsadmProt LocalAddress:
31、Port Scheduler Flags- RemoteAddress:Port Forward Weight ActiveConn InActConnTCP localhost:https wlc persistent 50- localhost:https Route 1 0 0- localhost:https Route 1 0 0TCP localhost:http wlc persistent 50- localhost:http Route 1 0 0- localhost:http Route 3 0 0当有新连接过来时，其相关的计数就会增加。另外通过查看 cat /proc/
32、net/ip_vs_stats 也可以查看 lvs 信息，只不过这里的输出时 16 进制的。# cat /proc/net/ip_vs_statsTotal Incoming Outgoing Incoming OutgoingConns Packets Packets Bytes Bytes594BFC 6335751 0 145F90EC1 0Conns/s Pkts/s Pkts/s Bytes/s Bytes/s0 0 0 0 0问题总结1、make keepalived 时提示：make2: Leaving directory /home/install/lvs/keepalived
33、-1.1.15/keepalived/coremake2: Entering directory /home/install/lvs/keepalived-1.1.15/keepalived/checkgcc -g -O2 -I/usr/src/linux/include -I/include -I/lib -Wall -Wunused -Wstrict-prototypes -D_KRNL_2_6_ -D_WITH_LVS_ -D_HAVE_IPVS_SYNCD_ -D_WITHOUT_VRRP_ -c check_daemon.cIn file included from /usr/src
34、/linux/include/linux/netlink.h:5,from /include/vrrp_netlink.h:35,from check_daemon.c:40:/usr/src/linux/include/linux/types.h:158: error: syntax error before “_sum16“/usr/src/linux/include/linux/types.h:158: warning: type defaults to int in declaration of _sum16/usr/src/linux/include/linux/types.h:15
35、8: warning: data definition has no type or storage class/usr/src/linux/include/linux/types.h:159: error: syntax error before “_wsum“/usr/src/linux/include/linux/types.h:159: warning: type defaults to int in declaration of _wsum/usr/src/linux/include/linux/types.h:159: warning: data definition has no
36、 type or storage classmake2: * check_daemon.o Error 1make2: Leaving directory /home/install/lvs/keepalived-1.1.15/keepalived/checkmake1: * all Error 1make1: Leaving directory /home/install/lvs/keepalived-1.1.15/keepalivedmake: * all Error 2解决：vi /usr/src/linux/include/linux/types.h/*typedef _u16 _bi
37、twise _sum16;typedef _u32 _bitwise _wsum;*/2、master、backup 的/var/log/messagse 显示Nov 23 17:46:41 SN2008-06-070 Keepalived_vrrp: receive an invalid ip number count associated with VRID!Nov 23 17:46:41 SN2008-06-070 Keepalived_vrrp: bogus VRRP packet received on eth0 !Nov 23 17:46:41 SN2008-06-070 Keep
38、alived_vrrp: VRRP_Instance(VI_1) Dropping received VRRP packet.Nov 23 17:46:42 SN2008-06-070 Keepalived_vrrp: receive an invalid ip number count associated with VRID!Nov 23 17:46:42 SN2008-06-070 Keepalived_vrrp: bogus VRRP packet received on eth0 !Nov 23 17:46:42 SN2008-06-070 Keepalived_vrrp: VRRP
39、_Instance(VI_1) Dropping received VRRP packet.解决：修改 virtual_router_id 的值，与网络中不能重复。3、在/var/log/messages 中 Keepalived_vrrp 不生效，即无 vrrp 日志。解决：重新编译源码包，可能是由于修改 types.h 后才 configure。安装 openssl-devel# yum install openssl-devel如果没有安装 openssl-devel 会出现如下报错# ./configure configure: error:! OpenSSL is not properly installed on your system. ! Can not include OpenSSL headers files. !
