Documentation

Jump to: Setup for image creation | rpm location | post create script | partition list | local files | Creating the image | MAC.info | dhcpd.conf | Installing nodes | adding ssh support | multicast installs | allow any | parted | network device settings | linuxbios | loop back image creation

YACI USE

OTHER SOFTWARE USED BY YACI

-Linux http://www.kernel.org/
-UDP Cast http://udpcast.linux.lu/

# Brief overview
Installs rpms into a chroot'd environment
tar up the chroot image
network boot a node
node nfs mounts, partitions itself, formats, and untar's the image

NOTE: The yaci-ramdisk rpm is not required unless you are doing a multiarchitecture
install, or if for some reason, the ramdisk will not build on your yaci server.

# There are examples in /usr/share/doc/yaci/

VARIABLES FILE
More documentation can be found in the variables file.

The default settings in the variables file are for the following:
- Serial console on port ttyS0, running at 115200
- Hardware is i386 or x86_64
- The node creating the image is of the same hardware type as the nodes to be installed
- PST8PDT timezone

Please be sure the following are correct:
BAUD
USE_SERIAL <- There are a few examples, the default should work
TIMEZONE
DEVICE <- This is the network interface that will be setup after a
node installs

If using an older version of rpm that does not support "--aid" you may have to
update the variables NOSIGNATURE, AID and FORCE

GRUB / ELILO

At the top of /tftpboot/scripts/krappie I have defined the title for grub and elilo, this may or not work by default with your distro name, version, and release will be replace with the rpm's name version and release that owns the kernel being installed.

Back to top

TO CREATE AN IMAGE FOR INSTALL.

1) Create a file (/tftpboot/etc/rpmlist.NODE_TYPE) with a list of rpms that will be installed on the nodes. A easy way to create this list is run `/tftpboot/tools/create_rpmlist` to duplicate your server, or run `rpm -qa` on a running node that you want to duplicate.

YACI now requires that rpm dependencies are met. It uses the --aid option in the newer version of rpm. If you are using an older version of rpm, or are not meeting the dependencies please update /tftpboot/scripts/variables:

###########
# RPM stuff
###########
# Dont check signature on rpm
NOSIGNATURE="--nosignature"
# if using older rpm
#NOSIGNATURE=""
# Use rpm aid
AID="--aid"
# if using older rpm
#AID=""
# Force install of rpms if not making rpm deps.
FORCE="NO"

EXAMPLE:
cat RPMLIST

zip-2.3-8.i386.rpm
zlib-1.1.3-25.7.i386.rpm
zlib-devel-1.1.3-25.7.i386.rpm
zsh-3.0.8-8.i386.rpm

or
zip-2.3-8*
zlib-1.1.3-25.7*
zlib-devel-1.1.3-25.7*
zsh-3.0.8-8*

Because of the way that yaci uses the rpmlist to install a file, we highly discourage the use of any paths in this file.

Back to top

2) Have the rpms located in /tftpboot/rpms

This is where the script create_image looks for rpms by default If you have your rpms in a different location change the variable RPM_DIR in /tftpboot/scripts/variables.

NOTE: YACI will follow symlinks, so creating a rpms directory with symlinks to your favorite rpm hang-out will also work.

Back to top

3) Optionally, you can also have a script to be run in the image after it has been created.

This script will be run in a chroot (see "man chroot") mode on the image. Because of this make sure you write the script as if you are on the live system.

EXAMPLE:
cat POST_CREATE_SCRIPT

chkconfig sshd on
chkconfig ntpd on
rm -rf /var/tmp
ln -sf /tmp /var/tmp
echo "Welcome to my computer" >> /etc/motd

Back to top

4) Create a file called /tftpboot/etc/partition_list.NODE_TYPE From this, the fstab, grub.conf and partitioning info will be created. You can also run `/tftpboot/tools/create_partition_list` to use the partion setup from your running system.

This list is used by /tftpboot/scripts/create_disk_files which looks for partition_list.NODE_TYPE. It creates the files fstab.NODE_TYPE, fstab_mapping.NODE_TYPE, and sfdisk.NODE_TYPE.HARD_DRIVE

The order that the mount points are listed are the partition numbers that they will be on. So given the example, /boot will be on /dev/hda1 and the first swap will be on /dev/hda2. If more then 4 partitions are on a disk, partition 4 will become extended and the forth mount will be mounted on partiton 5. Multiple disks are supported through the device name.

EXAMPLE:
cat /tftpboot/etc/partition_list.other

#Device MountPoint Format SizeMB Bootable
hda /boot ext2 80 *
hda swap swap 2048
hda swap swap 2048
hda / ext2 512
hda /usr ext2 2048
hda /var ext2 1024
hda /opt ext2 1024
hda /tmp ext3 rest


EXAMPLE: for linuxbios you don't want to mount the first partition or format it

cat /tftpboot/etc/partition_list.io.linuxbios

#Device MountPoint Format SizeMB Bootable
hda [none] [none] 10 *
hda /boot ext2 80
hda swap swap 2048
hda swap swap 2048
hda / ext2 512

EXAMPLE: for ia64

#Device MountPoint Format SizeMB Bootable
sda /boot/efi vfat 100 *
sda /boot/diags vfat 400
sda /boot ext3 200
sda swap swap 16384
sda / ext3 2048
sda /usr ext3 4096
sda /var ext3 4096
sda /tmp ext3 rest

Back to top

5) If you would like any files from the system that you are running YACI on to be installed onto the image you can put that info into the file /tftpboot/local/localize or /tftpboot/local/localize.NODE_TYPE or /tftpboot/local/localize.ARCH or /tftpboot/local/localize.NODE_TYPE.ARCH

/tftpboot/local/localize would be used by all nodes being installed /tftpboot/local/localize.NODE_TYPE where NODE_TYPE is equal to what you put in MAC.info will only be read by nodes that match that NODE_TYPE. Same goes for ARCH and NODE_TYPE.ARCH. ARCH is figured out at boot time and is not specified in variables file or MAC.info.

Note in the example you only need the the SOURCE. If you set PERM, UID, or GID, please account for all three attributes

Except for appending to files, the source can be a directory

EXAMPLE:

# Source Destination PERM UID GID

# install /etc/hosts into /etc/hosts with permissions 644, UID=root and GID=root
/etc/hosts /etc/hosts 644 root root

# installs /etc/passwd with current permissions into /etc/passwd in image
/etc/passwd

# installs /etc/inittab into /etc/inittab with current permissions
/etc/inittab /etc/inittab

# installs /etc/group into /etc/group with current permissions
/etc/group /etc/group - - -

# installs /etc/hosts.allow into /etc/hosts.allow with # permissions 644 and user root, and current group
/etc/hosts.allow /etc/group 644 root -

# installs file from /tftpboot/local into image with permsissions 755 foo.bar
somefile /some/place 755 foo bar

# append file /tftpboot/local/services to /etc/services in image
services /etc/services append


Back to top

6) cd /tftpboot/scripts

run ./create_image RPMLIST NODE_TYPE POST_CREATE_SCRIPT

This will do a chroot install of the rpms in /tftpboot/images/NODE_TYPE When this is done a tar file of the image will be created and put in /tftpboot/tarfiles/NODE_TYPE.tgz. The POST_CREATE_SCRIPT is the path to the script from Step 3.

EXAMPLE:
./create_image /etc/myrpmlist io myscript

Back to top

7) Create a file called /tftpboot/etc/MAC.info with the following

NODE_NAME MAC_ADDRESS NODE_TYPE

This is used by the node when it boots to decide which tar file to get and what name to use. If you ran the command `./create_image /etc/myrpmlist io myscript` and wanted tux2 to get this image, "io" is placed next to its name and MAC address

EXAMPLE:
cat /tftpboot/etc/MAC.info

tux1 00:50:43:22:9B:D5 login
tux2 00:50:43:11:79:60 io
tux3 00:50:43:11:95:6B io
tux4 00:50:43:11:A8:1C io
tux5 00:50:43:11:A9:35 io
tux6 00:50:43:11:93:A1 compute

If you have are logging your console output and do not have a running cluster you can set COLLECT_MAC="yes" in /tftpboot/scripts/variables and when a node boots it wil just output MAC_ADDRESS 00:30:48:24:6A:77. The MAC address will be output every 5 seconds.

Also make sure /tftpboot/pxelinux.cfg/default has the right console for output

To do this you would want a dhcpd.conf file like:

subnet 192.168.82.0 netmask 255.255.255.0 {
use-host-decl-names on;
option vendor-encapsulated-options 3c:09:45:74:68:65:72:62:6f:6f:74:ff;
option root-path "/tftpboot";
range 192.168.82.1 192.168.82.50;
filename "pxelinux.0";
}

8) Create /etc/dhcpd.conf
This is based on your nodes doing a PXE boot.

EXAMPLE:

cat /etc/dhcpd.conf

ddns-update-style ad-hoc;
subnet 192.168.15.0 netmask 255.255.255.0 {
use-host-decl-names on;
option vendor-encapsulated-options 3c:09:45:74:68:65:72:62:6f:6f:74:ff;
option root-path "/tftpboot";
host dev25 {
hardware ethernet 00:30:48:11:A3:40;
fixed-address 192.168.15.26;
option host-name "dev25";
filename "pxelinux.0";
}
host dev29 {
hardware ethernet 00:06:5B:3C:D3:FF;
fixed-address 192.168.15.30;
option host-name "dev29";
filename "pxelinux.0";

}
}

Back to top

TO INSTALL NODES

1) Make sure to export /tftpboot read/write on mgmt node. Logging is done in /tftpboot/log, so you need to export with the write option.

EXAMPLE:
cat /etc/exports

/tftpboot 192.168.15.0/24(rw,no_root_squash)

Make sure you have enough nfsd daemons running. You should have one nfsd for each 10 nodes that will be installing.

EXAMPLE:
cat /etc/sysconfig/nfs

RPCNFSDCOUNT=128

Also, make sure that the daemons are running.

/etc/init.d/portmap restart
/etc/init.d/nfs restart

2) Allow connections on mgmt node

EXAMPLE:
cat /etc/hosts.allow

portmap,mountd,in.tftpd : 192.168.15. : ALLOW

3) Make sure tftp is enabled

chkconfig xinetd on
chkconfig tftp on
/etc/init.d/xinetd restart

4) Make sure dhcp is running

/etc/init.d/dhcpd restart

5) The nodes will boot using PXE. They will mount /tftpboot from the mgmt node and look for the file /tftpboot/scripts/node_install If it is there it will run it, else it will drop you to a prompt.

6) When node_install runs it will format the disk, mount the disk, and untar NODE_TYPE.tgz onto the disk. The fstab is installed and grubby will be run to install grub.conf or elilo.conf

7) If the script node_install finds a file called /tftpboot/scripts/post_install it will run it. YACI puts a default post_install script in place for you. It is suggested that you only add to it, do not remove stuff from it.

This can be used to install a /etc/hosts file or anything else that may be needed for the node to come up correctly

The post_install script will setup the network interface that is defined in the variables file with the IP that is given by dhcpd.

post_install will also write a file to /tftpboot/pxelinux.cfg that is the IP in HEX, which will tell the node to boot from the next device This will allow you to put in the bios the boot order of 1- network 2- hard drive. So if you need to re-install, remove the file in /tftpboot/pxelinux.cfg that contains the hostname of the node.

Back to top

SSH

If you add "sshd" to EXTRA_RAMDISK_FILES, and you are dynamicly building your ramdisks ( BUILD_RAMDISK="YES" ), run /tftpboot/tools/connect_node to ssh to the node you are installing


Back to top

MULTICAST

If you are installing < ~128 nodes, this is probably not worth the headache. Fun to play with, but if you need to get it done, use NFS. If you have a GigE connection on the server and 100Mb on the rest of the nodes, it should only take about 10 minutes to push a 600MB tarball to all nodes over NFS.

This is useful if you are doing 100's of nodes, not just one. Unlike installing the image over NFS this can have user interaction. If you pass -n NUMBER multicast_server will automatically start a multicast install. This has not been tested on very many types of clusters. It has been found that a good network switch is needed to install at 100Mb/s, you may try 10mb/s if you have problems. I have had best luck with half the speed of the slowest connection. For example, the compute nodes are 100Mb/s, set the MAX_BITRATE to 50m.

KNOWN_PROBLEMS: YACI logs via NFS, make sure you have 1 NFSD running per 10 nodes that you are installing, even if you are doing a multi-cast install.

1) Set MULTICAST="YES" in /tftpboot/scripts/variables

This will is used by node_install to decide if it should use NFS or multicast

2) Set MAX_BITRATE="50m" to speed of the slowest connection divided by 2, you will be installing over. So with it set to 50m I am saying that my slowest connection if 100Mb/s ( this is also Full Duplex, I have not tried multicast half duplex ).

3) Set INTERFACE="eth0" to the interface that will be installing the cluster This is the interface that is on the node doing the installing

4) Run /tftpboot/tools/multicast_server NODE_TYPE

example: /tftpboot/tools/multicast_server io

This will begin listening for clients. This is run before turning on the clients because not all nodes will connect if the server starts after the clients are looking for it.

5) Boot only the nodes that are of the type you passed to "multicast_server" so in the example I would only boot my io nodes.

NOTE: As the nodes boot they will format their hard drives and get things mounted. If MULTICAST="YES" then the node will look at the size of the image in /tftpboot/tarfiles/NODE_TYPE.tgz and if the node has 2X the amount of RAM then the size of this image, it will put it into RAMFS. If the node doesn't have this much RAM it will look and see if there is enough room in /tmp on the hard drive to put this image. If there is not enough room there, it will fail. UDP Cast ( The multicast server/client YACI uses ) allows for piping the image into tar but this has shown to be slower then NFS since CPU's even at the same MHz do not keep in sync and cause 1000's of re-transmits

6) As the nodes boot, they will begin to run the udp-client and connect to your "multicast_server". The connections start at 0. So if you are installing 128 nodes, expect to see 127 connetions. The multicast will not start until you hit "ENTER" unless you pass "-n" to multicast_server with the number of nodes to wait for then start. Once you do the connected nodes will get the file and put it somewhere, and when they have it all, untar it onto their hard drives and continue with the standared YACI stuff.

Back to top

ALLOW_ANY

1) If you set ALLOW_ANY="yes" in /tftpboot/scripts/variables then any node that connects will get installed with your DEFAULT_IMAGE. Your dhcpd.conf file will need to be setup to give out IP's without known mac addresses. Not sure why you would want this, but here it is.

Back to top

PARTED

1) If you would like to use a GPT disk scheme on your system rather than the standard msdos partitioning scheme, you must set USE_PARTED equal to YES in /tftpboot/scripts/variables. This will allow create_disk_files and node_install perform the actions needed to create the needed partitions.

Back to top

NET_DEV_SETTING

1) By default this is turned off. If you have a network card that will not auto negotiate right, you can try this. There is an example in the variables file.

Back to top

LINUXBIOS NOTES

1) To make a bootable linuxbios image from the yaci kernel run the following: NOTE: you may need to change baud rate from 115200 to something you can do

mkelfImage --command-line="load_ramdisk=1 prompt_ramdisk=0 initrd=initrd.gz \
root=/dev/ram rw ip=dhcp console=ttyS0,115200n8 ide0=ata66" \
--ramdisk="/tftpboot/initrd.gz" \
--kernel=/tftpboot/vmlinuz \
--output=/tftpboot/vmlinuz.elf

Back to top

LOOP_BACK_CREATE

This is useful if building server is one arch and the other node is of a different arch.

1) Uncomment LOOP_BACK_CREATE and set it to the NODE_TYPE Uncomment LOOP_BACK_P0ST_SCRIPT if you would like a script run on in the chrooted env. This is just like the POST_CREATE_SCRIPT that you would pass to create_image. Whatever LOOP_BACK_P0ST_SCRIPT is set to, is what node_install will look for in /tftpboot/scripts/

3) Create the rpm list as $YACI_LOC/etc/$TYPE/rpmlist ie. /tftpboot/etc/io/rpmlist

4) If you set USE_RAM to YES it will build the image in ram. I have no way of knowing how much ram you need so set this option with caution By default the image is created on the server in the default location ( /tftpboot/images/NODE_TYPE.ARCH

By default the image is created on the server in the default location ( /tftpboot/images/$TYPE.$ARCH ) If you set USE_RAM to YES it will build the image in ram. I have no way of knowing how much ram you need so set that option with caution. No matter how you do this the tarfile will be put into $YACI_TARS. This does not install the node, only creates the image

#LOOP_BACK_CREATE="monkey"
#LOOP_BACK_P0ST_SCRIPT="post_script.${LOOP_BACK_CREATE}"
#USE_RAM="YES"