1、Oracle RAC 11gR2 三节点安装部署实施步骤(一) 集群规划1. 硬件环境服务器 3 台:DELL PowerEdge R620,2*Xeon E5-2680v2,128G 内存，2*160G SSD 硬盘，2*8Gbps HBA 卡存储:DELL PowerVoult MD3660f,48*1TB 7.2k SAS 硬盘，双控制器，8 光纤口连接方式：通过 SAN 交换机，每台服务器双光纤接入到 SAN 交换机，MD3660f 采用 8 光纤接入到 SAN 交换机。2. 软件环境数据库:oracle11.2.0.3_Linux-x86-64集群件:oracle11.2.0.3_g
2、rid-x86-64操作系统:oracle linux server6.5(二) IP 规划 每个节点一个 public IP每个节点一个 public VIP 每个节点一个 private IP 一个用于集群的 SCAN 名称和一到三个和 public IP 同网段的 IP,本处用一个public IP,VIP 和 private IP 写入/etc/hosts 文件scan 配置本地 /etc/hosts 解析主机名 IP 名 对应网卡 IP 用途rac21 rac21 bond0 10.0.31.5 publicrac21-vip 10.0.31.15 VIPrac21-priv bon
3、d1 10.0.50.5 privaterac22 10.0.31.6 publicrac22-vipbond010.0.31.16 VIPrac22rac22-priv bond1 10.0.50.6 privaterac23 10.0.31.7 publicrac23-vipbond010.0.31.17 VIPrac23rac23-priv bond1 10.0.50.7 privateScanIP rac2-scan bond0 10.0.31.102 Scan IP(三) 安装目录规划建立/opt/oracle 文件系统用来安装 grid、datbase 程序。Oracle 用户:O
4、RACLE_BASE= /opt/oracle/appORACLE_HOME=$ORACLE_BASE/product/11.2.0Grid 用户:ORACLE_BASE= /opt/oracle/grid/appORACLE_HOME= /opt/oracle/grid/11.2.0(四) 存储规划RAID GROUP Raid 类型 LUN 编号 Stripe Size(KB) LUN 大小 (GB) 映射主机 40 128 500 RAC2-Group41 128 500 RAC2-Group42 128 500 RAC2-Group43 128 500 RAC2-Group44 128
5、 500 RAC2-Group45 128 500 RAC2-GroupRG4(7D+1P)RAID 5 46 128 500 RAC2-Group47 128 500 RAC2-Group48 128 500 RAC2-Group49 128 500 RAC2-Group50 128 500 RAC2-Group51 128 500 RAC2-Group52 128 500 RAC2-Group53 128 500 RAC2-Group54 128 500 RAC2-Group55 128 500 RAC2-Group56 128 500 RAC2-Group57 128 500 RAC2-
6、Group58 128 500 RAC2-Group59 128 500 RAC2-Group60 128 500 RAC2-Group61 128 500 RAC2-Group62 128 500 RAC2-Group63 128 500 RAC2-Group64 128 500 RAC2-GroupRG5(7D+1P)RAID 5 65 128 500 RAC2-GroupRAC2-OCR1 71 128 1 RAC2-GroupRAC2-OCR2 72 128 1 RAC2-GroupRAC2-OCR3 73 128 1 RAC2-GroupRAC2-VOT1 74 128 1 RAC2
7、-GroupRAC2-VOT2 75 128 1 RAC2-GroupRAC2-REDO1 79 128 300 RAC2-GroupRAC2-REDO2 80 128 300 RAC2-GroupRG6(2D+2P)RAID10RAC2-REDO3 81 128 300 RAC2-Group(五) 安装实施1. 安装前准备存储环境准备将主机、存储等接入 SAN 交换机,按需划分相应的 zone; 清理所有设备的 WWN 号; 在存储端添加数据库主机,将两节点绑定为一个 Host set; 存储划分 VV,并分配给数据库 Host set。 2. 主机环境准备操作系统安装分别在两个节点安装 o
8、racle linux server6.5 操作系统。2 块 SSD 硬盘设置成 RAID1 镜像。安装参数如下:系统分区/ 50G/boot 500Mswap 16G/opt 79G操作系统安装过程略。3. 在每个节点上(rac21/rac22/rac23)执行 1) 配置 IP1、增加绑定配置# vi /etc/modprobe.d/bonding.confalias bond0 bondingalias bond1 bonding2、加载绑定模块#modprobe bonding3、编辑/etc/sysconfig/network-script/ifcfg-bond0 文件# vi /e
9、tc/sysconfig/network-scripts/ifcfg-bond0DEVICE=bond0TYPE=EthernetONBOOT=yesBOOTPROTO=noneIPADDR=10.0.31.5 (rac22 改成 10.0.31.6，rac23 改成 10.0.31.7)PREFIX=24NETMASK=255.255.255.0NETWORK=10.0.31.0GATEWAY=10.0.31.254DNS1=114.114.114.114DEFROUTE=yesUSERCTL=noBONDING_OPTS=“mode=4 miimon=100“NAME=“bond0“4、编
10、辑/etc/sysconfig/network-script/ifcfg-bond1 文件# vi /etc/sysconfig/network-scripts/ifcfg-bond1DEVICE=bond1TYPE=EthernetONBOOT=yesBOOTPROTO=noneIPADDR=10.0.50.5 (rac22 改成 10.0.50.6，rac23 改成 10.0.50.7)PREFIX=24NETMASK=255.255.255.0NETWORK=10.0.50.0BOARDCAST=10.0.50.255USERCTL=noBONDING_OPTS=“mode=0 miim
11、on=100“NAME=“bond1“5、编辑/etc/sysconfig/network-script/ifcfg-em1 文件# vi /etc/sysconfig/network-scripts/ifcfg-em1DEVICE=em1TYPE=EthernetONBOOT=yesBOOTPROTO=noneIPV4_FAILURE_FATAL=yesIPV6INIT=noUSERCTL=noMASTER=bond0SLAVE=yesNAME=“System em1“6、编辑/etc/sysconfig/network-script/ifcfg-em2 文件# vi /etc/syscon
12、fig/network-scripts/ifcfg-em2DEVICE=em2TYPE=EthernetONBOOT=yesBOOTPROTO=noneIPV4_FAILURE_FATAL=yesIPV6INIT=noUSERCTL=noMASTER=bond0SLAVE=yesNAME=“System em2“7、编辑/etc/sysconfig/network-script/ifcfg-em3 文件# vi /etc/sysconfig/network-scripts/ifcfg-em3DEVICE=em3TYPE=EthernetONBOOT=yesBOOTPROTO=noneIPV4_
13、FAILURE_FATAL=yesIPV6INIT=noUSERCTL=noMASTER=bond1SLAVE=yesNAME=“System em3“8、编辑/etc/sysconfig/network-script/ifcfg-em4 文件# vi /etc/sysconfig/network-scripts/ifcfg-em4DEVICE=em4TYPE=EthernetONBOOT=yesBOOTPROTO=noneIPV4_FAILURE_FATAL=yesIPV6INIT=noUSERCTL=noMASTER=bond1SLAVE=yesNAME=“System em4“9、编辑/
14、etc/sysconfig/network 文件# vi /etc/sysconfig/networkNETWORKING=yesHOSTNAME=rac21 (节点 2 改成 rac22，节点 3 改成 rac23)GATEWAY=10.0.31.25410、配置完成后重启网络服务:#service network restart 2) 编辑/etc/hosts 文件# vi /etc/hosts127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4:1 localhost localhost.
15、localdomain localhost6 localhost6.localdomain610.0.31.8 goldengate10.0.31.5 rac2110.0.31.15 rac21-vip10.0.50.5 rac21-priv10.0.31.6 rac2210.0.31.16 rac22-vip10.0.50.6 rac22-priv10.0.31.7 rac2310.0.31.17 rac23-vip10.0.50.7 rac23-priv10.0.31.102 rac2-scan3) 安装必要的软件包挂载光盘安装软件:迚入操作系统光盘/Server/Packages/ 目录
16、下,找到下列文件 ,拷至/root/patch 目录中:binutils-2.20.51.0.2-5.36.el6.x86_64.rpmcompat-libstdc+-33-3.2.3-69.el6.i686.rpmcompat-libstdc+-33-3.2.3-69.el6.x86_64.rpmelfutils-libelf-0.152-1.el6.i686.rpmelfutils-libelf-0.152-1.el6.x86_64.rpmelfutils-libelf-devel-0.152-1.el6.i686.rpmelfutils-libelf-devel-0.152-1.el6.
17、x86_64.rpmgcc-4.4.7-4.el6.x86_64.rpmgcc-c+-4.4.7-4.el6.x86_64.rpmglibc-2.12-1.132.el6.i686.rpmglibc-2.12-1.132.el6.x86_64.rpmglibc-common-2.12-1.132.el6.x86_64.rpmglibc-devel-2.12-1.132.el6.i686.rpmglibc-devel-2.12-1.132.el6.x86_64.rpmksh-20120801-10.el6.x86_64.rpmlibaio-0.3.107-10.el6.i686.rpmlibai
18、o-0.3.107-10.el6.x86_64.rpmlibaio-devel-0.3.107-10.el6.i686.rpmlibaio-devel-0.3.107-10.el6.x86_64.rpmlibgcc-4.4.7-4.el6.i686.rpmlibgcc-4.4.7-4.el6.x86_64.rpmlibstdc+-4.4.7-4.el6.i686.rpmlibstdc+-4.4.7-4.el6.x86_64.rpmlibstdc+-devel-4.4.7-4.el6.i686.rpmlibstdc+-devel-4.4.7-4.el6.x86_64.rpmlibstdc+-do
19、cs-4.4.7-4.el6.x86_64.rpmmake-3.81-20.el6.x86_64.rpmnumactl-devel-2.0.7-8.el6.i686.rpmnumactl-devel-2.0.7-8.el6.x86_64.rpmsysstat-9.0.4-22.el6.x86_64.rpmunix2dos-2.2-35.el6.x86_64.rpmunixODBC-2.2.14-12.el6_3.i686.rpmunixODBC-2.2.14-12.el6_3.x86_64.rpmunixODBC-devel-2.2.14-12.el6_3.i686.rpmunixODBC-d
20、evel-2.2.14-12.el6_3.x86_64.rpm将上述补丁包和软件上传到/tmp/patch 目录中,并在该目录中执行如下命令进行安装 :# rpm -ivh -nodeps -force /root/patch/* 4) 修改内核参数# vi /etc/sysctl.confkernel.shmmax = 4398046511104kernel.shmall = 1073741824kernel.shmmni = 4096 kernel.sem = 250 32000 100 128 net.ipv4.ip_local_port_range = 9000 65500 net.c
21、ore.rmem_default = 262144 net.core.rmem_max = 4194304 net.core.wmem_default = 262144 net.core.wmem_max = 1048576fs.aio-max-nr = 1048576fs.file-max = 68157445) 修改 oracle 用户限制vi /etc/security/limits.conf 末尾添加:#for Oracle Installoracle soft nofile 2047oracle hard nofile 65536oracle soft nproc 2047oracl
22、e hard nproc 16384oracle soft stack 10240oracle hard stack 32768grid soft nofile 2047grid hard nofile 65536grid soft nproc 2047grid hard nproc 16384grid soft stack 10240grid hard stack 327686) 修改/etc/pam.d/login#vi /etc/pam.d/login 末尾添加:Session reauired /lib64/security/pam_limits.so7) 关闭防火墙的 selinux
23、在每个节点上以 root 用户执行：1、停掉 iptables、selinux 服务 :#service iptables stop #chkconfig iptables off #setenforce 0#getenforce2、修改配置文件:#vi /etc/selinux/configSELINUX=disabledSELINUXTYPE=targeted8) 停用 ntp 服务# service ntpd stop # chkconfig ntpd off # mv /etc/ntp.conf /etc/ntp.conf.bak # rm -rf /etc/ntp.conf9) 更改
24、安装所有者的 ulimit 设置#vi /etc/profile 末尾添加:if /$USER = “oracle“ ; then if /$SHELL = “/bin/ksh“ ; thenulimit -p 16384ulimit -n 65536elseulimit -u 16384 -n 65536fiumask 022 fiexport NLS_LANG=American_America.ZHS16GBK10) 添加 Oracle 用户创建用户:#/usr/sbin/groupadd -g 501 oinstall#/usr/sbin/groupadd -g 502 dba#/usr
25、/sbin/groupadd -g 503 oper#/usr/sbin/groupadd -g 504 asmadmin#/usr/sbin/groupadd -g 505 asmoper#/usr/sbin/groupadd -g 506 asmdba#/usr/sbin/useradd -g oinstall -G dba,asmdba,oper oracle#/usr/sbin/useradd -g oinstall -G asmadmin,asmdba,asmoper,oper,dba grid11) 设置 oracle、grid 用户密码#passwd oracle (设置密码为
26、nebula)#passwd grid (设置密码为 nebula)12) 创建 oracle 安装目录#mkdir p /opt/oracle#chown R oracle:oinstall /opt/oracle#mkdir p /opt/oracle/grid/app#mkdir p /opt/oracle/grid/11.2.0#chown R grid:oinstall /opt/oracle/grid#mkdir p /opt/oracle/app/product/11.2.0#chown R oracle:oinstall /opt/oracle/app#chmod R 755
27、/opt/oracle13) 设置 oracle 用户环境变量以 oracle 用户登录$ vi .bash_profileumask 022export ORACLE_BASE=/opt/oracle/appexport ORACLE_HOME=$ORACLE_BASE/product/11.2.0export ORACLE_SID=APPDBexport NLS_LANG=AMERICAN_AMERICA.UTF8export NLS_DATE_FORMAT=“yyyy-mm-dd hh24:mi:ss“export PATH=.:$PATH:$HOME/bin:$ORACLE_HOME/
28、binexport PATH=$PATH:/usr/bin:/bin:/usr/bin/X11:/usr/local/binexport PATH=$PATH:$ORACLE_BASE/common/oracle/binexport ORACLE_PATH=$PATH:$ORACLE_BASE/common/oracle/sql:.:$ORACLE_HOME/rdbms/adminexport ORACLE_TERM=xtermexport TNS_ADMIN=$ORACLE_HOME/network/adminexport LD_LIBRARY_PATH=$ORACLE_HOME/libex
29、port LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME/oracm/libexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/lib:/usr/lib:/usr/local/libexport CLASSPATH=$ORACLE_HOME/JREexport CLASSPATH=$CLASSPATH:$ORACLE_HOME/jlibexport CLASSPATH=$CLASSPATH:$ORACLE_HOME/rdbms/jlibexport CLASSPATH=$CLASSPATH:$ORACLE_HOME/ne
30、twork/jlibexport THREADS_FLAG=nativeexport TEMP=/tmpexport TMPDIR=/tmp14) 设置 grid 用户环境变量以 grid 用户登录$ vi .bash_profileumask 022export ORACLE_BASE=/opt/oracle/grid/appexport ORACLE_HOME=/opt/oracle/grid/11.2.0export ORACLE_SID=+ASM1 (rac22 改成+ASM2 ，rac23 改成+ASM3)export NLS_LANG=AMERICAN_AMERICA.UTF8ex
31、port NLS_DATE_FORMAT=“yyyy-mm-dd hh24:mi:ss“export PATH=.:$PATH:$HOME/bin:$ORACLE_HOME/binexport PATH=$PATH:/usr/bin:/bin:/usr/bin/X11:/usr/local/binexport PATH=$PATH:$ORACLE_BASE/common/oracle/binexport ORACLE_PATH=$PATH:$ORACLE_BASE/common/oracle/sql:.:$ORACLE_HOME/rdbms/adminexport ORACLE_TERM=xt
32、ermexport TNS_ADMIN=$ORACLE_HOME/network/adminexport LD_LIBRARY_PATH=$ORACLE_HOME/libexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME/oracm/libexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/lib:/usr/lib:/usr/local/libexport CLASSPATH=$ORACLE_HOME/JREexport CLASSPATH=$CLASSPATH:$ORACLE_HOME/jlibexport
33、CLASSPATH=$CLASSPATH:$ORACLE_HOME/rdbms/jlibexport CLASSPATH=$CLASSPATH:$ORACLE_HOME/network/jlibexport THREADS_FLAG=nativeexport TEMP=/tmpexport TMPDIR=/tmpexport PATH=$PATH:$ORACLE_HOME/bin15) 配置 SSH分别以 root、oracle、grid 用户配置三个用户的 SSH 无密码登录。1、分别 rac21、rac22 、rac23 节点以 root 用户登录执行:#mkdir /.ssh #chmo
34、d 700 /.ssh #ssh-keygen -t rsa密码全直接回车#su oracle$mkdir /.ssh $chmod 700 /.ssh $ssh-keygen -t rsa密码全直接回车$exit#su grid$mkdir /.ssh $chmod 700 /.ssh $ssh-keygen -t rsa密码全直接回车$exit2、在 rac21 上以 root 用户执行:#cp /.ssh/id_rsa.pub /.ssh/authorized_keys#ssh rac22 cat /.ssh/id_rsa.pub/.ssh/authorized_keys#ssh rac
35、23 cat /.ssh/id_rsa.pub/.ssh/authorized_keys #scp /.ssh/authorized_keys rac22:/.ssh/authorized_keys#scp /.ssh/authorized_keys rac23:/.ssh/authorized_keys#su - oracle$cp /.ssh/id_rsa.pub /.ssh/authorized_keys$ssh rac22 cat /.ssh/id_rsa.pub/.ssh/authorized_keys$ssh rac23 cat /.ssh/id_rsa.pub/.ssh/auth
36、orized_keys $scp /.ssh/authorized_keys rac22:/.ssh/authorized_keys$scp /.ssh/authorized_keys rac23:/.ssh/authorized_keys$exit#su grid$cp /.ssh/id_rsa.pub /.ssh/authorized_keys$ssh rac22 cat /.ssh/id_rsa.pub/.ssh/authorized_keys$ssh rac23 cat /.ssh/id_rsa.pub/.ssh/authorized_keys $scp /.ssh/authorize
37、d_keys rac22:/.ssh/authorized_keys$scp /.ssh/authorized_keys rac23:/.ssh/authorized_keys3、在 rac21、 rac22 和 rac23 节点上分别用 root、oracle、grid 用户登录，然后分别执行：ssh rac21输入 yesexitssh rac21-priv输入 yesexitssh rac22输入 yesexitssh rac22-priv输入 yesexitssh rac23输入 yesexitssh rac23-priv输入 yesexit4. 刷新磁盘若没有重机器,存储挂载后在系统
38、中会看丌到,用如下命令迚行硬件刷新: #rescan-scsi-bus.sh5. 磁盘多路径配置1) 用 PowerVault MD Storage Manager 做 LUN 映射Disk Group 划分与 Virtual Disks 划分2) 加载多路径模块在每个节点上以 root 用户执行：#scsi-rescan#modprobe dm-multipath#modprobe dm-round-robin3) 启动 multipath 服务在每个节点上以 root 用户执行：#service multipathd restart设置成开机自启动服务#chkconfig level 34
39、5 multipathd on4) 编辑/etc/multipath/bindings 文件，修改虚拟磁盘映射别名在每个节点上以 root 用户执行：#vi /etc/multipath/bindingsrac2-data1 36f01faf000e5fd4000000af33c8ce843rac2-data2 36f01faf000e5fd4000000b083c8cea33rac2-data3 36f01faf000e5fd4700000e8b3f65c57frac2-data4 36f01faf000e5fd4000000b0a3c8cea5arac2-data5 36f01faf000
40、e5fd4700000e8d3f65c5b6rac2-data6 36f01faf000e5fd4000000b0c3c8cea92rac2-data7 36f01faf000e5fd4700000e8f3f65c5e3rac2-data8 36f01faf000e5fd4000000b0e3c8ceabfrac2-data9 36f01faf000e5fd4700000e913f65c60arac2-data10 36f01faf000e5fd4000000b103c8ceae3rac2-data11 36f01faf000e5fd4700000e933f65c62erac2-data12
41、36f01faf000e5fd4000000b123c8ceb09rac2-data13 36f01faf000e5fd4700000e953f65c65crac2-data14 36f01faf000e5fd4700000e763f65c39crac2-data15 36f01faf000e5fd4000000b143c8ceb56rac2-data16 36f01faf000e5fd4700000e973f65c6a3rac2-data17 36f01faf000e5fd4000000b163c8ceb7frac2-data18 36f01faf000e5fd4700000e993f65c
42、6d2rac2-data19 36f01faf000e5fd4000000b183c8cebadrac2-data20 36f01faf000e5fd4700000e9b3f65c6f8rac2-data21 36f01faf000e5fd4000000b1a3c8cebd3rac2-data22 36f01faf000e5fd4700000e9d3f65c720rac2-data23 36f01faf000e5fd4000000b1c3c8cebfbrac2-data24 36f01faf000e5fd4700000e9f3f65c746rac2-data25 36f01faf000e5fd
43、4000000b1e3c8cec1frac2-data26 36f01faf000e5fd4700000ea13f65c772rac2-redo1 36f01faf000e5fd4700000ea73f664f46rac2-redo2 36f01faf000e5fd4000000b263c8d7426rac2-redo3 36f01faf000e5fd4700000ea93f664f83rac2-crs1 36f01faf000e5fd4000000ad83c8ce4fdrac2-crs2 36f01faf000e5fd4700000e5a3f65c048rac2-crs3 36f01faf0
44、00e5fd4000000ada3c8ce526rac2-crs4 36f01faf000e5fd4700000e5c3f65c072rac2-crs5 36f01faf000e5fd4000000adc3c8ce54f5) 清除 multipath 磁盘并重新生成在每个节点上以 root 用户执行：1、重启服务器2、清除磁盘#multipath F3、重新扫描生成磁盘#multipath v2create: rac2-data1 (36f01faf000e5fd4000000af33c8ce843) undef DELL,MD36xxfsize=500G features=0 hwhandl
45、er=0 wp=undef|-+- policy=round-robin 0 prio=1 status=undef| - 1:0:0:1 sdb 8:16 undef ready running-+- policy=round-robin 0 prio=1 status=undef- 2:0:0:1 sdaj 66:48 undef ready runningcreate: rac2-data2 (36f01faf000e5fd4000000b083c8cea33) undef DELL,MD36xxfsize=500G features=0 hwhandler=0 wp=undef|-+-
46、 policy=round-robin 0 prio=1 status=undef| - 1:0:0:2 sdc 8:32 undef ready running-+- policy=round-robin 0 prio=1 status=undef- 2:0:0:2 sdak 66:64 undef ready runningcreate: rac2-data3 (36f01faf000e5fd4700000e8b3f65c57f) undef DELL,MD36xxfsize=500G features=0 hwhandler=0 wp=undef|-+- policy=round-rob
47、in 0 prio=1 status=undef| - 1:0:0:3 sdd 8:48 undef ready running-+- policy=round-robin 0 prio=1 status=undef- 2:0:0:3 sdal 66:80 undef ready runningcreate: rac2-data4 (36f01faf000e5fd4000000b0a3c8cea5a) undef DELL,MD36xxfsize=500G features=0 hwhandler=0 wp=undef|-+- policy=round-robin 0 prio=1 statu
48、s=undef| - 1:0:0:4 sde 8:64 undef ready running-+- policy=round-robin 0 prio=1 status=undef- 2:0:0:4 sdam 66:96 undef ready runningcreate: rac2-data5 (36f01faf000e5fd4700000e8d3f65c5b6) undef DELL,MD36xxfsize=500G features=0 hwhandler=0 wp=undef|-+- policy=round-robin 0 prio=1 status=undef| - 1:0:0:
49、5 sdf 8:80 undef ready running-+- policy=round-robin 0 prio=1 status=undef- 2:0:0:5 sdan 66:112 undef ready runningcreate: rac2-data6 (36f01faf000e5fd4000000b0c3c8cea92) undef DELL,MD36xxfsize=500G features=0 hwhandler=0 wp=undef|-+- policy=round-robin 0 prio=1 status=undef| - 1:0:0:6 sdg 8:96 undef ready running-+- policy=round-robin 0 prio=1 status=undef- 2:0:0:6 sdao 66:128 undef ready runningcreate: rac2
