From BCCD 3.0
BCCDv3 -- Basic Information
- Install Instructions
- User Instructions
- Frequently Encountered Problems
- Submitting bug reports
- Submitting feature requests
- Log in as bccd with the password "letmein". Note that ssh is not started by default, but you should change the password with "passwd" if you do want to start it.
- If you run into problems, send email to firstname.lastname@example.org with the error output.
(thanks to Dave Toth for these instructions)
sudo su - rootand enter the BCCD user's password (This makes you the root user - aka the superuser, which allows you to do things like manage flash drives).
- Plug in the flash drive.
dmesgand hit enter. This will show a lot of text including some about a device like
/dev/sdb. If it refers to
sdain place of
sdbfor the remainder of the instructions.
mkdir /tmp/flashdriveand hit enter. This creates a temporary directory for the flashdrive to mount on.
mount -t vfat -o uid=bccd,gid=bccd /dev/sdb1 /tmp/flashdriveand hit enter. This says the directory
/tmp/flashdriveis the top level directory on the flash drive. By copying files to (or from)
/tmp/flashdriveyou are copying them to (or from) the flash drive.
- At this point, you do not need to be the root user anymore until you need to remove the flash drive. You can return to being a normal user by typing
exitand hitting enter in the terminal that you typed in for step 1.
- When you are done using the flash drive, repeat step 1 above to become the root user (aka superuser). Then type
umount -l /tmp/flashdriveand hit enter. Wait a couple seconds and then you can remove the flash drive.
- At this point, you do not need to be the root user anymore. You can return to being a normal user by typing
USB automounting (deprecated)
!!! Warning: this information is deprecated !!!
The BCCD is setup to automatically mount removable USB drives when the drive is inserted. If you're in X, it will pop up a message saying where it mounted the drive (e.g. /mnt/sdg1). If you're not in X, you can figure out what the last device to be mounted was by doing tail -1 /var/log/mount.log. The BCCD won't automatically unmount the drive, so be sure to unmount the drive first, e.g. umount /mnt/sdg1.
BCCDv3 -- Configuring and Customizing
You can supply these options when you start the system up (in the form "linux foo"):
- automode - Don't prompt for anything in boot-up.
- quickboot - Same as above (for now).
- masterserver=server - Merge a client's /cluster/software directory with another server's. Leave blank w/o = to broadcast for server.
- dhcp_masterserver=server - Option for DHCP server to set a pointer to a master software server.
- 5 - Starts X for the bccd user
- recoverdhcp - Run this on a live CD master node to trigger a recovery of the DHCP lease file from another cluster node.
- -x - Start the linuxrc script in debug mode, useful to track down tricky boot problems in the live CD.
- bccdid=abcd - Supply bccdid to change the default bccd-identifier string. This can be used to setup multiple BCCD clusters on a single network. Make sure to select a different IP address later for the BCCD network.
A summary of these options is also available in the F4 menu at boot-up.
For now, both schools are supported:
In the live CD, you can merge /cluster/software from another node (known as the master node) with your own node. This is done through the magic pixie dust of AUFS and DHCP. You should set one live CD system to run DHCP like so:
linux startdhcp dhcp_masterserver=master-server.example.com dhcp_routernic=eth0
Linux dhclient requires an IP address for a router to be passed in, so if you're uncertain of what to provide just provide eth0. Note that the DHCP server need not be the master server; the address of the master server is merely handed out via DHCP. Boot the master server like so:
If your master server is the DHCP server, include `startnfs` on the line with `startdhcp`. On the client systems, boot like so:
and the client system will find the master server using DHCPINFORM packets.
By default, your BCCD client will talk to a server hosted at Earlham to get additional software. This software magically appears at
/mnt/ssh/software if you access anything in that directory. Note that shell searches often don't trigger this automatic mounting; the best way to do it is
There is also a special Modules repository for these remote software packages. By default Modules just points at local software, but you can also load remote software. Type in
modules avail and look at the software packages in the
/usr/local/etc/modules-remote section. Load one of them with
modules load /usr/local/etc/modules-remote/sw-name/sw-version. Note that you'll have to unload the corresponding local software package if it's already loaded.
- To setup your own software repository, first make it available somehow on the network (NFS, sshfs, SMB, etc).
- Add a corresponding autofs entry in /etc/auto.master and /etc/auto.<mapname>.
- Stop autofs with
invoke-rc.d autofs start, make sure the mount directories are present (Debian autofs zorches them on shutdown), and start autofs back up with
invoke-rc.d autofs start. We'll provide documentation later for how to do this.
- Setup a separate modules file in
/usr/local/Modules/3.2.6/modulefiles/modules. Make a copy of the remote-bccd file, and change the name of BCCD_SW_REMOTE to some other name (say, BCCD_SW_MYREMOTE).
- Set the value of the new variable to a different path (say, /mnt/ssh/my-software).
- Change the MODULEPATH variable to point to a new module repository (say, /usr/local/etc/modules-myremote).
- Copy all of /usr/local/etc/modules-remote to /usr/local/etc/modules-myremote.
- Change all the instances of BCCD_SW_REMOTE in modules-myremote to BCCD_SW_MYREMOTE:
find . -type f -print0|xargs -0 perl -wpli -e 's/BCCD_SW_REMOTE/BCCD_SW_MYREMOTE/g'.
bccd-get-mpi- Print the current MPI implementation
bccd-report- Generate debug report for developers
bccd-reset-network- Reset the network to defaults, then prompt for common configuration options
bccd-shutdown- Shutdown a BCCD cluster, including the head node
bccd-sleep-all- Put the entire BCCD cluster to sleep, if possible
bccd-snarfhosts- Generate MPI host list suitable for the currently-loaded implementation of MPI. The file will end up in
bccd-syncdir- Copy one directory from the local node to all nodes in the cluster
bccd-version- Print release and version information for BCCD
module- Use to load and unload various software packages from the shell environment
The keyboard mapping in the BCCD defaults to US English. You can change it by logging in as the bccd user, and running sudo dpkg-reconfigure console-data, hit OK, and then go to "Select keymap from full list". In X, you can set the keymap using setxkbmap country-code, which doesn't require root privileges. A full list of country codes is in /usr/share/X11/xkb/symbols/.
No video on bootup
Confucius say: "The video isn't always the problem when the video doesn't work"
NVidia 8600 GTS and NVidia 8800 GTS
- Recall that these didn't "initialize" the display upon boot. A black screen was rendered irregardless of the vga= and video= options given to the kernel. It turned out that the *motherboard* was the culprit, and supplying "noapic" did the trick (without the noapic, the kernel was panic'ing before initializing the display so there was no way to see the kernel panic message). Serial ports are wonderful things sometimes.
The BCCD is built in North American Eastern time, so the default time zone is EST5EDT. Hardware time is kept in UTC, so just set your TZ variable for the bccd user to something else (e.g. PST8PDT for Pacific time). Look in /usr/share/zoneinfo/ for more options.
You might also find your hardware clock is off. The BCCD is set to use the Debian project's NTP pool by default. If your network's DHCP server provides its own ntp-servers option, a /etc/ntp.conf.dhcp file will be created automatically and NTP will use that instead of the stock file. If your hardware clock is more than a few minutes off, NTP will be unable to slew the time fast enough in most cases. If you don't care about large time corrections, perform these steps to correct your time quickly:
- date # Get the current time
- invoke-rc.d ntp stop # Stop the NTP daemon
- ntpd -gqx # Run a one-off time correction without slew limits
- date # To make sure that the time is now correct
- invoke-rc.d ntp start # Start the NTP server with slew limits
When running in liberated mode, cluster nodes do not need to use pkbcast to broadcast public keys, nor does bccd-allowall need to be run to allow all public keys that pkbcast detected. Nodes check in to the master node's /cluster/home/meta/nodes directory with their hostname as a file, and files that haven't been touched in less than a few minutes are considered stale. For this to work properly, the master node must be at the first IP in a network block (i.e. 192.168.5.1/24). To change what hostname a master node reports, edit the `/etc/cron.d/node-checkin` file on the master node and remove the old /cluster/home/meta/node000 file.
The BCCD uses a framework called Modules to provide user environment management. This system allows users to swap in and out different software packages without having to manage complex shell profiles. It also provides some dependency and conflict management; for instance, you cannot load both MPICH and OpenMPI modules at the same time since their binary names overlap. The BCCD loads most modules out of the box, so in many instances you won't even need to touch modules. If you do, here's some basic commands for working with modules:
- module list - Print out the modules that are currently loaded into the user environment.
- module avail - Print out all modules that are present on the system.
- module load module - Load module named module into the user environment. By default, this will load the version that comes last lexigraphically, unless a symlink to default from the version that should be default is provided. If you want to load a non-default version, simply provide the version after the slash, e.g. gromacs could be represented as gromacs/3.3.3_openmpi.
- module unload module - Unload module named module from the user environment.
- module purge - Unload all modules.
Adding third-party packages
A software package provides a safe, consistent way for a software distributor to install and uninstall their software. The BCCD, being based on Debian, understands its
.deb package format. You can interact with the BCCD's packaging system with
dpkg -i somepkg.deb # Install the package dpkg -r somepkg.deb # Uninstall the package
If you don't have a .deb package, you might be able to use
alien is a Debian utility that will convert non-Debian packages into
.deb files. Optionally, it can automatically install the packages it creates. Currently, alien understands
.rpm (Red Hat), and
.pkg (Solaris) package formats. To install an RPM, you would run
alien -i somepkg.rpm
If you are using a LittleFe, first follow the steps on the LittleFe wiki before returning here.
If you are running on real hardware, you might want to make sure that your BCCD systems are not overheating. BCCD ships with lm-sensors pre-configured. Simply run the
sensors command to see the current system's temperature:
bccd@node000:~$ sensors acpitz-virtual-0 Adapter: Virtual device temp1: +26.8 C (crit = +90.0 C) coretemp-isa-0000 Adapter: ISA adapter Core 0: +43.0 C (high = +105.0 C, crit = +105.0 C) Core 1: +43.0 C (high = +105.0 C, crit = +105.0 C) Core 2: +45.0 C (high = +105.0 C, crit = +105.0 C) Core 3: +45.0 C (high = +105.0 C, crit = +105.0 C)
BCCD ships with PBS/Torque enabled and configured automatically.
See VirtualBox on BCCD for details.
Booting on a Mac
So far, using the Universal USB Installer on Windows is the only reliable way we have found to boot on a Mac. Macs unfortunately have jumped the gun and started using EFI boot loading, which (for all its theoretical advantages) has much less support than the tried-and-true BIOS.
See Booting on a Mac for details as we work on a real solution.
Supplying information to developers
As of the next minor release (likely v3.1.2), BCCD will have the ability to gather all information needed by the developers all in one file. To use it:
You may copy this file to a USB stick or scp it to a system on the local network that does have Internet connectivity, and email it to the developers via email@example.com.
The developer documentation is at on the development page.