1、在 ubuntu 8.04 上安装 kerrighed 2.3.0 cluster参考原文：Setting Up A Diskless-boot Kerrighed 2.4.1 Cluster in Ubuntu 8.04本实验中用了三台机器，一台作为server（运行Ubuntu 8.04，内核任意），另外两台作为client。Server不需要安装 Kerrighed内核，他只作为服务器，为Client节点提供网络以及文件系统等支持。Server 通过DHCP 、TFTP服务为Client进行网络安装操作系统，并通过NFS为Client提供全局文件系统支持。Server有两个网卡，一个对外
2、，一个对内。本实验中对外的网卡为eth1，其IP 与本实验无关；对内的网卡为eth0 ，其IP 为192.168.1.1，子网掩码为255.255.255.0。这个安装向导分为两个部分：第一部分是配置服务器，让节点可以使用当前的kernel无盘启动；第二部分是安装与配置Kerrighed 2.3.0 。为了简单，本文大多内容直接原文拷贝，所以是英文，个别地方做了修改，或采用中文说明。另外，重要的补充说明用红字体标示。Part 1: 建立无盘启动的 Ubuntu serverThis will be the basis of our cluster. In order to get a work
3、ing diskless boot server, there are four main components youll need to install: a DHCP server to assign IP addresses to each node, a TFTP server to boot the kernel for each node over the network, an NFS server to allow the nodes to share a filesystem, and a minimal Ubuntu 8.04 installation for them
4、to share. These server components will all run on one box; this box will be your head node or the controller for the cluster. Lets get started!1.1: Setting up the DHCP serverDHCP is what will allow the nodes to get IP addresses from the server. We want to set up and configure a DHCP daemon to run on
5、 the server and give IP addresses only to nodes it recognises, so we will tell the daemon their MAC addresses. First install the DHCP server package with aptitude or apt-get, as root:# aptitude install dhcp3-server补充说明：从此处开始，后续会有很多操作涉及aptitude install或者apt-get install，这些都需要联网操作。那么，在无法上网的情况下，如何安装这些软件
6、呢？一种相对简单的方式是：一次性的把所有的软件安装在另一台上网机上，然后拷贝这些软件到非上网机。此次实验采取的即是这种方法，详细说明见 附录一 。当然，也可以在本地局域网配置一个软件源，只要添加这个源即可，但源非常庞大，下载较麻烦，暂时未使用该方法。修改内网卡的IP为192.168.1.1 ，掩码为255.255.255.0，网关为192.168.1.0，DNS为192.168.1.1。 Check that the DHCP server configuration file, /etc/default/dhcp3-server, contains the correct Ethernet
7、card to listen to DHCP requests from the nodes: this will be the card that does not connect to the Internet. If you put the wrong card into this file, bad things will happen: your server will start listening to DHCP requests on the LAN and denying them all, since they dont come from nodes. Make sure
8、 you know which is the right card! = In this case it is eth0, so make the configuration file look like this:# /etc/default/dhcp3-server #interfaces=“eth0“ Now you need to configure the DHCP daemon to issue addresses only to nodes, and tell it which addresses to give them. Make sure you have the MAC
9、addresses of your nodes Ethernet cards handy - you can get them by issuing the command ifconfig as root from a live DVD or USB stick, for instance, or they may be written on the nodes casing or on stickers stuck to the cards. You can also just boot the cluster and watch the error messages, picking o
10、ut the MACs from the denied requests. Edit /etc/dhcp3/dhcpd.conf, DHCPs daemon configuration file, so it looks like the following. (Dont worry that the file references certain things you havent actually installed yet, like a PXE bootloader; youll put these things in later.)# /etc/dhcp3/dhcpd.conf #
11、General optionsoption dhcp-max-message-size 2048;use-host-decl-names on;deny unknown-clients; # This will stop any non-node machines from appearing on the cluster network.deny bootp;# DNS settingsoption domain-name “kerrighed“; # Just an example name - call it whatever you want.option domain-name-se
12、rvers 192.168.1.1; # The servers IP address, manually configured earlier.# Information about the network setupsubnet 192.168.1.0 netmask 255.255.255.0 option routers 192.168.1.1; # Server IP as above.option broadcast-address 192.168.1.255; # Broadcast address for your network.# Declaring IP addresse
13、s for nodes and PXE infogroup filename “pxelinux.0“; # PXE bootloader. Path is relative to /var/lib/tftpbootoption root-path “192.168.1.1:/nfsroot/kerrighed“; # Location of the bootable filesystem on NFS serverhost kerrighednode1 fixed-address 192.168.1.101; # IP address for the first node, kerrighe
14、dnode1 for example.hardware ethernet 01:2D:61:C7:17:86; # MAC address of the nodes ethernet adapterhost kerrighednode2 fixed-address 192.168.1.102;hardware ethernet 01:2D:61:C7:17:87;server-name “kerrighedserver“; # Name of the server. Call it whatever you like.next-server 192.168.1.1; # Server IP,
15、as above.Now youre done configuring DHCP, so your nodes will be able to get IPs. Its time to add the functionality that will allow the server to transfer a kernel to them afterwards.1.2: Setting up the TFTP server and PXE bootloaderTFTP is the fileserver that will be used by the bootloader to transf
16、er the kernel to your nodes during a PXE boot. We need to install a TFTP server and get a PXE bootloader as part of the syslinux package, so that our nodes will be able to get their operating systems via the cluster server. As root, install the TFTP server package, tftp-hpa, with aptitude or apt-get
17、:# aptitude install tftpd-hpa Open its configuration file, /etc/default/tftpd-hpa, and make sure it uses the following settings. It can be run as a daemon, but it is normally started by the service daemon “inetd“, and it uses the /var/lib/tftpboot directory to get files from:# /etc/default/tftp-hpa
18、#Defaults for tftp-hpaRUN_DAEMON=“yes“OPTIONS=“-l -s /var/lib/tftpboot“ Now we need to configure inetd to run the tftp server. Open its configuration file, /etc/inetd.conf, and change the tftp line to the following. If there is no tftp line, add this:tftp dgram udp wait root /usr/sbin/in.tftpd /usr/
19、sbin/in.tftpd -s /var/lib/tftpboot As root, install syslinux, which is the system required for you to be able to PXE boot the nodes, and copy the PXE bootloader code from it to the TFTP server directory. This is the bootloader you told the DHCP daemon about in its configuration file earlier.# aptitu
20、de install syslinux# cp /usr/lib/syslinux/pxelinux.0 /var/lib/tftpboot Still as root, create a directory to store the default configuration for all the nodes. They will search in this directory for configuration files during the PXE boot process.# mkdir /var/lib/tftpboot/pxelinux.cfg Still as root,
21、copy your current kernel and initrd from /boot to /var/lib/tftpboot/ in order to test the diskless-boot system. If need, replace uname -r with whatever you are using.# cp /boot/vmlinuz-uname -r /boot/initrd.img-uname -r /var/lib/tftpboot/ Create the file /var/lib/tftpboot/pxelinux.cfg/default. This
22、will be the fallback configuration file that the nodes use to PXE boot when they cant find a file specific to their own IP address. Make the file look like this:LABEL linuxDEFAULT vmlinuz-uname -r console=tty1 root=/dev/nfs initrd=initrd.img-uname -r nfsroot=192.168.1.1:/nfsroot/kerrighed ip=dhcp rw
23、Youre done setting up the TFTP and PXE components of the cluster server! Your nodes will now be able to get a kernel and filesystem from the server after theyre given IP addresses. Now you need to add NFS capability.1.3: Setting up the NFS serverThis capability allows for the bootable filesystem tha
24、t the nodes will download over TFTP to be accessed and shared over the network, so that the cluster uses one filesystem. First well install and set up the server that will do this. As root, install the packages nfs-kernel-server and nfs-common, which comprise the NFS server program. Keep your root a
25、uthorisation until youre done working with the NFS server.# apt-get install nfs-kernel-server nfs-common Make a directory to store the bootable filesystem in:# mkdir /nfsroot# mkdir /nfsroot/kerrighed Edit /etc/exports, which configures NFS file transfers. Add the following in order to make NFS expo
26、rt the filesystem that will be stored in the directory you just made:# /etc/exports #/nfsroot/kerrighed 192.168.1.0/255.255.255.0(rw,no_subtree_check,async,no_root_squash) Re-export the file systems, since you just changed how this is done:# exportfs -avrYour NFS server should be up and running. Now
27、 you can add a filesystem for this server to work with.1.4: Setting up the bootable filesystem This isnt as simple as just copying the OS files into another directory - youll need the debootstrap package to install the bootablefilesystem, so install this first (you should still be root.) Once its in
28、stalled, use debootstrap to itself install a minimal Ubuntu Hardy system tothe bootable filesystem directory:# aptitude install debootstrap# debootstrap -arch i386 hardy /nfsroot/kerrighed http:/ 根系统，对于不能联网的机器，解决方法如下。在联网的机器上安装ubuntu8.04 的虚拟机，建立/nfsroot/kerrighed目录后执行以上命令，成功后该目录下将会有一个完整的根目录，拷贝这个目录（最好
29、先压缩再拷贝，防止数据丢失）到非联网机器的相应目录即可。在本实验中，由于后续还要apt-get install一些软件，这些也需要联网，所以应当在联网机上执行完这些所有需要联网的操作之后，再拷贝整个目录到非联网机上。当然，此时在非联网机上就可以掠过这些步骤了，这些需要在联网机上操作的后续步骤如下：Edit /etc/apt/sources.list.# aptitude update$ apt-get install dhcp3-common nfs-common nfsbooted openssh-server$ apt-get install automake autoconf libto
30、ol pkg-config gawk rsync bzip2 gcc-3.3 libncurses5 libncurses5-dev wget lsb-release xmlto patchutils xutils-dev build-essential openssh-server ntp Change the current root of the file system to the bootable filesystem directory (stay chrooted until the guide tells you otherwise.) This is so that you
31、can work with the bootable filesystem directly, as if it were a separate machine, while we make some adjustments to it.# chroot /nfsroot/kerrighed Set the root password for the bootable filesystem. You can use a program called apg, the automated password generator, to create a good one.# passwd Moun
32、t the /proc directory of the current machine as the bootable systems /proc directory, so you can use programs on the bootable FS:# mount -t proc none /proc Edit /etc/apt/sources.list. We want to add access to some extra Ubuntu repositories in order to be able to download the necessary packages for t
33、he FS. Add these lines:deb http:/ hardy partnerdeb http:/ hardy main universe restricted multiversedeb http:/ hardy-security universe main multiverse restricteddeb http:/ hardy-updates universe main multiverse restricteddeb-src http:/ hardy main universe restricted multiversedeb-src http:/ hardy-sec
34、urity universe main multiverse restricteddeb-src http:/ hardy-updates universe main multiverse restricted Update the current package listing from the repositories you just added so youll be able to install things from them:# aptitude update Install some packages that our nodes need for using NFS and
35、 DHCP:$ apt-get install dhcp3-common nfs-common nfsbooted openssh-server Now we need to make it work with NFS. Edit /etc/fstab, the filesystem index, of the bootable filesystem to look like this:# /etc/fstab# proc /proc proc defaults 0 0/dev/nfs / nfs defaults 0 0 Edit /etc/hosts. This is so that th
36、e DHCP server will know which of the PXE-booted nodes get which IP and hostname. Add all your cluster nodes and the server to it. In our example case, it looks like this:# /etc/hosts #127.0.0.1 localhost192.168.1.1 kerrighedserver192.168.1.101 kerrighednode1192.168.1.102 kerrighednode2 Do the follow
37、ing to create a symbolic link which will automount the bootable filesystem as /dev/nfs on the server, when it starts up. This should not collide with other existing services in the directory (e.g. anything that looks like /etc/rcS.d/S34xxxxxxx), so check carefully before you create the link. If ther
38、es a service with a similar name, disable it before you do anything else.$ ln -sf /etc/network/if-up.d/mountnfs /etc/rcS.d/S34mountnfs Edit /etc/network/interfaces and add a line to stop Network Manager from managing the nodes Ethernet cards, as this can cause issues with NFS. Ours looks like the fo
39、llowing:# .# The loopback interface:auto loiface lo inet loopback# The primary network interface, manually configured to protect NFS:iface eth0 inet manual Create a user for the bootable system. Replace with whatever you want to call her. adduser will ask you for her real name andother details, so p
40、lay along.# adduser Ensure that the new user is in the /etc/sudoers file so she can run root commands on the cluster:# /etc/sudoers #User privilege specificationroot ALL=(ALL) ALLALL=(ALL) ALL Exit the root shell, and then exit from the chrooted bootable filesystem. Youre done configuring the bootab
41、le FS and can now test it with yourcommon-or-garden kernel.# exit# exit1.5: Testing the diskless boot system sans Kerrighed Restart the servers on the head node, because youve been messing with them. You need to be root to do this:# /etc/init.d/tftpd-hpa restart# /etc/init.d/dhcp3-server restart# /e
42、tc/init.d/nfs-kernel-server restart Configure the BIOS of each node to have the following boot order: your primary boot device should be PXE, which will usually be described as “network boot“ or “LAN boot“. In certain cases you may need to enable the network cards as boot devices in the BIOS, reboot
43、, and then set the boot priority. Remember also to disable “halt on all errors“, since this can mess up your PXE booting. Boot each of the nodes to see if it works. If so, you should be presented with a login prompt, where you can log-in using the username you defined earlier. When it all works, you
44、re ready to try with a Kerrighed kernel.Part 2: 建立Kerrighed 集群Now that weve got a diskless boot system setup to use as a server, we only need to build the Kerrighed kernel for the nodes to use, put it in the bootable FS, and configure the Kerrighed settings properly in order to have a working SSI (S
45、ingle System Image) cluster.The first thing to do is build the new kernel itself.2.1: Building the Kerrighed kernel Shutdown the nodes. On the server, chroot back into the bootable filesystem, again as root:# chroot /nfsroot/kerrighed Install some basic necessary packages into the bootable filesyste
46、m. These are the dependencies that well need to have on the system in order to successfully compile Kerrighed.$ apt-get install automake autoconf libtool pkg-config gawk rsync bzip2 gcc-3.3 libncurses5 libncurses5-dev wget lsb-release xmlto patchutils xutils-dev build-essential openssh-server ntp Ge
47、t the Kerrighed source ball from INRIAs GForge, and then get a copy of the vanilla 2.6 kernel. This is the kernel that were going to be modifying into a Kerrighed one.# wget -O /usr/src/kerrighed-2.3.0.tar.gz http:/gforge.inria.fr/frs/download.php/23356/kerrighed-2.3.0.tar.gz# wget -O /usr/src/linux
48、-2.6.20.tar.bz2 http:/www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.20.tar.bz2补充说明：此处需要联网操作，对于非联网机器，可在联网机器上下载后拷贝过来。其中kerrighed在www.kerrighed.org上找，linux在www.kernel.org上找。 Change to the /usr/src directory, where the downloaded source tarballs are, and decompress them:# cd /usr/src# tar zxf kerrighed-2.3.0.tar.gz# tar jxf linux-2.6.20.tar.bz2 Look inside the decompressed Kerrighed sources. Theres a bug here (at time of writing) in the modules directory Makefile: open it up with your favourite editor (were using Vim here) and take a look. If the b
