From BCCD 3.0

Jump to: navigation, search

If you were looking for information on the test suite, it's been moved here



The BCCD-NG environment is stored entirely in a Subversion repository on hopper. Read-write access is provided over ssh, at


Development branches are under


with the common branch name being your-name-devel.

The BCCDv3 ISO is assembled in a single pass in the bin/ script. It is composed of a series of tests from the perl module, found in trees/usr/local/lib/site_perl/ Tests take different numbers of arguments, so look at the signature of the specific test subroutine to figure that out. Keep in mind that every run_test call will have at least four arguments: the name of the test, the success return code the test should expect (which is left blank in nearly all cases for a safe default), the message the test should print out, and the first (and maybe only) argument for the test.

For historical reasons, the liberation script is at packages/ This probably should be moved to a more sensible location in the future. The script has the same layout as the script. Remember that the liberation should assume no network connectivity, so all files should be fetched in the ISO build. Files that are specific to the liberation can be stored in /root/liberate, which is reference by the $LIBFETCH variable.

For more information on specific tests, see the POD documentation within

Getting started

Creating a new SVN branch

The best way to get started is to make your own SVN branch. To do that, enter a directory on your current system where you want a local check out, and issue a command like this:

  svn cp svn+ssh:// \

You can then checkout or update that branch. To checkout your newly created branch, issue a command like:

  svn co svn+ssh://

You will now want to get setup in Jenkins.

Directory structure


  1. Add or modify a file in ~bccd
    1. All files should be in in packages/etc/skel to support multiple users.
    2. Modify or add the file in skel, and commit the change. All the files will be pulled down automatically.
  2. Change a file in /etc
    1. Add the file to packages/etc.
    2. The placement of files in /etc comes with many dependencies (when a package gets installed, whether a service gets started automatically, etc.). For this reason, you'll have to add a test to fetch the file out of WebDAV, and optionally other tests to change its ownership and permissions. There are several for loops in that you might be able to take advantage of, if your file has similar dependencies as the other files in that loop.
  3. Add a module
    1. "Master" module files are stored in trees/usr/local/Modules/3.2.6/modulefiles. You will probably never have to touch these unless you need to add another module repository.
    2. "Software" module files are stored in trees/usr/local/etc/modules, and trees/usr/local/etc/modules-remote (if you're using remote modules).
    3. Since modules are stored in a subdirectory of tree (trees/usr), all files are fetched at once and you don't have to add a separate test.

Commits etiquette

Follow two rules when you commit:

  1. Always reference a ticket number in the commit message, e.g.
    adding libnet-ip for dhcp dialog script (#358)
  2. Always test your changes before merging from your branch into trunk.

Build process

See Build Process

Build requirements

The following

Note: Some default versions of the File::Temp Perl module are bugged and do not contain the newdir method. To circumvent this issue, install File::Temp using cpan.

Then proceed to setup Jenkins


The BCCD has a number of different configurations, called stages. Applications need to know this so they can react appropriately. The current stage is stored in /etc/bccd-stage. The current stages are these:

A proposed addition is:



/bin/bccd-report is a script that end users may run to gather all information that might be useful in debugging a problem. This script will generate ~bccd/bccd-report.tar.bz2, which maybe transported to another system and emailed to the list manually. The list of gathered information is in #696.

It is designed to have as few dependencies as possible, so that even a seriously impaired system may still generate useful information.

The end user documentation is at the user instructions page.

Regenerating live squashfs

Often it's desirable to generate just the squashfs filesystem, located in the BCCD ISO image at live/filesystem.squashfs. The most common reason would be to test a feature without having to wait for an entire ISO build. These are the steps necessary to do this on a Debian-type system:

  1. Make sure you have the squashfs-tools package installed.
  2. Fetch the extract_squashfs script from SVN.
  3. Extract the ISO image and containing squashfs to separate directories: sudo ./extract_squashfs -i /path/to/bccd.iso
  4. This command will print out two paths - one with "ISO" and one with "SQUASHFS". Take note of both and be sure to clean them up when you are done.
  5. Make edits in the SQUASHFS directory as needed.
    • Generate a new SQUASHFS, stick it on an existing USB stick: rm -f /mnt/usb0/live/filesystem.squashfs && sudo mksquashfs /tmp/squashfs.rand-str/squashfs-root /mnt/usb0/live/filesystem.squashfs -comp xz
    • Generate a new SQUASHFS (see above), but instead stick it in the ISO image directory at live/filesystem.squashfs. Then re-generate the ISO image: genisoimage -pad -l -r -J -v -V BCCDv3-some-svn-rev -no-emul-boot -boot-load-size 4 -boot-info-table -b boot/isolinux/isolinux.bin -c boot/isolinux/ -hide-rr-moved -o /path/to/bccd.iso /tmp/bccd.other-rand-str

Debugging early boot process

Problems early on in the boot process can be difficult to diagnose. It is useful to understand the Linux boot process to know where to begin debugging:

  1. The BIOS (or equivalent like EFI) detects boot devices and assembles them into a list based on some internal heuristic (typically user-specifiable in the BIOS configuration)
  2. For each boot device from the list, the BIOS will attempt to find a boot code in the first sector of the device. The first device that has this code will have it loaded into memory and executed.
  3. In the case of BCCD, this boot code (whether from a CD, USB stick, or hard drive) will point at the boot loader (isolinux in the case of CD, syslinux in the case of USB sticks, and GRUB in the case of hard drives).
  4. The boot loader contains information on where to find the Linux kernel. Additional information is passed as options to the kernel command line, such as the location of the initial RAM disk (initrd), but is not interpreted by the boot loader itself. If found, the boot loader will load the kernel into memory and execute it.
    1. isolinux assumes that it will be placed within a ISO9660 filesystem in the directory /boot/isolinux. It will look for a isolinux.cfg file within that directory, and all paths in that file are relative to /boot/isolinux.
    2. syslinux assumes that it will be placed with a VFAT filesystem in the root directory. It looks for a syslinux.cfg file, and all paths in that file are relative to /.
    3. syslinux 4.05 is known to work for BCCD; if you get problems finding the root filesystem or the BCCD splash screen, you should downgrade.
  5. The kernel will spawn a single process called "swapper", which will initialize critical OS functions like memory management, process scheduling, and hardware detection. Hardware with support built directly into the kernel (as opposed to a loadable module) will be detected during this step.
  6. Once swapper is done bringing the system to a minimally-usable state, it will try to find the initrd specified in the initrd= option in the boot configuration file (isolinux.cfg, syslinux.cfg, etc.) in the same filesystem as the boot loader and kernel. This file is either a gzip'd ext2 filesystem image or a gzip'd cpio archive. It will uncompress and copy the contents of the initrd into a RAM disk, and look for the executable specified by the init= command line option from the boot loader. swapper will fork a process called "init" which will execute the init executable. In practice this is a shell script, but it can be any valid executable.
  7. The init process will configure hardware devices whose modules are not built directly into the kernel, and also perform other tasks that need to be done before the main OS boots. In the case of BCCD, this includes configuring networking for diskless nodes, and configuring the aufs union filesystem for live and diskless nodes. As a final task, it will mount the actual root filesystem. How it finds the root filesystem depends on the mode:
    1. For live systems, it will be detected by mounting every available disk device, and looking for the file specified by the BOOT_IMAGE kernel command line argument (KNOPPIX by default). The first device containing that file is assumed to be the root filesystem.
    2. For liberated systems, it will be specified by the root= kernel command line argument.
    3. For diskless systems, it will be specified by a combination of root=/dev/nfs and nfsroot= being set to the IP address and directory that should be mounted. There is additional logic within the init script to setup the aufs mounts.
  8. When init completes, it will execute a function called pivot_root that will change the root filesystem pointer from the RAM disk to the newly-mounted root filesystem, and then execute /sbin/init within the context of the new root filesystem. At this point RAM disk is destroyed, and the OS completes booting (mounting other filesystems, starting services, networking, etc.).

Debugging problems with the boot code

If you don't even see the boot loader screen (isolinux, syslinux, grub), you should check:

Debugging problems with the boot filesystem

If you see the boot loader, but get some problem immediately after it tries executing the kernel (before the penguins appear):

Debugging problems with the kernel boot process

If you get problems with the kernel (after the penguins appear):


See Systemd


See LightDM

Downgrading syslinux

  1. Fetch syslinux 4.05: curl -k -O -O
  2. Install: dpkg -i syslinux-common_4.05+dfsg-6+deb7u1_all.deb syslinux_4.05+dfsg-6+deb7u1_amd64.deb
  3. Prevent future upgrades from touching syslinux: apt-mark hold syslinux syslinux-common


If you were looking for information on the test suite, it's been moved here

Subversion Best Practices

See Subversion Best Practices for more information on how to use Subversion effectively for BCCD development. and the Perl module

Common variables

Common operations

The following are a list of operations that are common to the `` script. They are somewhat ordered, in that if you can get by with an earlier operation, you should.

Debian CLI

This should always be preferred to any of the other options. Use `apt-get`, `debootstrap`, `debconf`, etc. wherever possible, called from the system test in This might look like

                "Running apt-get install."
                "apt-get install $packages"

Changing files

If you have to, change only part of the file using sed or perl regexen, Again, you can call these from the system tests in This might look like

                "Changing interface in networking."
                "sed -e 's/ifconfig eth0/ifconfig eth1/g' < \ 
$libdir/$projdir/etc/init.d/networking > $libdir/etc/init.d/networking"

Entire file

You can store an entire file in subversion. Fetch it using the revfetch test, which takes the URL to the file, the subversion revison, and where you want to squirt the file in the build. This might look like

              "Fetched checkin cron.",

Entire tree

Use this only if you are fetching a directory tree and not overwriting existing files. There are too many dependency problems involved with overwriting existing files in the build. chdir using the chdir test to the base directory of the destination directory tree. Then use the recrevfetch test, which takes a subversion revision, and a repository URL. This might look like

              "cd $builddir",

              "Fetched bin tools.",

Tar balls

The BCCD-NG development originally used tar balls to distribute changes. This is deprecated for a number of reasons, but where it's still working I haven't touched it. Do what I say and not what I do, and use the options above.

Init scripts

Init scripts are a SysV-based system's way of starting a service at a particular time. Init scripts in the BCCD are located either in packages/etc/init.d or trees/etc/init.d. Remember that files in packages need to be pulled down individually, while files in trees are pulled down en masse. Most init scripts need update-rc.d run, so most should end up in packages. Here's a step-by-step procedure for adding one:

  1. Add your init script to packages/etc/init.d in your SVN branch.
  2. Annotate the LSB block appropriately. For instance, below is the LSB block for packages/etc/init.d/networking. This makes sure that local filesystems are mounted (important for diskless nodes), and that the mountkernfs and bccd-nics init script have already run. It also says that, if available, ifupdown should be run before networking starts, and after it stops. See the Debian wiki for details on how LSB works in Debian.
    # Provides:          networking
    # Required-Start:    mountkernfs $local_fs bccd-nics
    # Required-Stop:     $local_fs
    # Should-Start:      ifupdown
    # Should-Stop:       ifupdown
    # Default-Start:     2 5
    # Default-Stop:      0 1 6
    # Short-Description: Raise network interfaces.
  3. In bin/, add a test to remove the init script's stop and start links. This is necessary so that they can be re-created with the proper dependency information. There are for-loops in, but a singleton test would look like this:
            "Disabled $script.",
            "/usr/sbin/chroot $BUILDDIR /usr/sbin/update-rc.d -f some-script remove",
  4. In bin/, add a test to re-addthe init script's stop and start links based on the new dependency information. There are for-loops in, but a singleton test would look like this:
            "Disabled $script.",
            "/usr/sbin/chroot $BUILDDIR /usr/sbin/update-rc.d -f some-script defaults",
  5. Commit everything to SVN.
  6. Rebuild the ISO.

Building the BCCD ISO

Changes should always be tested before being committed to trunk. This can be done on any Debian system with a checkout of the subversion repository.

  1. cd into branches/livecd-devel/bin/.
  2. script -fc "sudo perl --arch i386 --suite squeeze --outiso /tmp/test-$(whoami).iso --websvn (your branch in the web repository) " /tmp/build-script-$(whoami).log
    • Your branch should look something like this:
      • name)
  3. You can view the output of the test in /tmp/build-script-(your username).log.
    • try grep '^not ok' /tmp/build-script-$(whoami).log
    • If you have problems add --debug --nocleanup to the command line and append &> build-stderr-stdout.log to the command line.

Automated builds

After you get all the prerequisites setup and do a manual build, you can setup automated builds. First fetch from the bin directory of a branch or trunk.

  wget && chmod +x

Then create configuration file that looks like this:


Then setup a cron job that looks like this:

  34 20 * * * /root/ -c /root/auto_build_wrapper.i386.conf

Committing changes to trunk

  1. You'll need a working copy of trunk for this. If you haven't already, go to a folder and run svn co svn+ssh://
    • Get comfortable, this will take a while.
    • If you have a working copy of trunk, make sure it's up-to-date with svn up .
  2. get into the root of your trunk directory
  3. Run svn merge --dry-run -r rev1:rev2 (your development working directory) .. rev1 is the last commit into the trunk, which you can see with svn log | less. rev2 is the new revision that you just committed to livecd-devel. Make sure the files being updated are the files you want updated. Look for any files with a "C" by them. You will have to merge those files by hand in the next step.
  4. svn merge -r rev1:rev2 svn+ssh:// .. Merge any conflicted files by hand, then run svn resolved filename. After everything is OK, run svn commit in the trunk directory, and commit your changes. Make sure that still has its websvn variable pointing into trunk and not livecd-devel.

Adding packages

See Reprepro for details on how the BCCD package repository works.

See Fpm for details on how to build custom packages for BCCD.

Via aptitude

aptitude is easy if there is an existing deb package in the tree on a0. Simply add it to the YAML configuration file bin/packages.conf. Most packages should be added to BASE. EXTRA is for packages that might become optional in the future, and AMD64 is for packages that should only be installed on AMD64 (rather than i386) builds.

Via svn

This option involves adding a "tree" to the subversion repository that contains the software built. Follow these steps to do it:

  1. From the live CD or liberation (liberation is recommended for disk space reasons), build and install the software into a subdirectory of /cluster/software.
  2. Make a symlink from software-xx.yy.zz to software, where xx.yy.zz is a version number.
  3. tar up software-xx.yy.zz and software, and move the tar ball to a host with a local subversion checkout.
  4. cd into the development branch, and then to trees/software/cluster/software.
  5. un-tar the tar ball.
  6. Run svn add software*.
  7. Remove the keywords property (this can cause binaries to become corrupted): find dir -type f ! -regex '.*\/\.svn\/.*' -print0|xargs -0 svn pd svn:keywords
  8. Commit the changes.
  9. Build a test ISO and test the new software.
  10. Merge the changes into the trunk.
  11. Document the process of building the package in the Software Build Documentation page.

Local builds

Using a local package mirror

By default, the build process uses the package mirror located at The script will allow the use of a local debmirror. Supply the hostname to the local repository using the --debmirror option. Don't supply the full URL; the rest of the URL will be constructed later in the script.

Using a local Subversion mirror

The build script fetches files out of Subversion using WebDAV at http://bccd-ng.cluster.earlham/svn/bccd-ng/branch-name, where branch-name would be trunch or branches/livecd-devel. You can supply an alternative WebDAV URL with --websvn.

Editing the splash screen

See the isolinux instructions for changing a boot screen:

Kernel updates

See Kernel updates.

Environment management

As mentioned in the user manual, the BCCD uses Modules to manage a user's environment (variables, aliases, etc.). This provides a Tcl-based framework to manage a user's environment as software packages are added and deleted. Every package installed in /bccd/software should get a module directory in /usr/local/etc/modules, with every version of the package as a separate file in the directory. Every module file is a Tcl script invoked by the modules command. For example, OpenMPI version 1.4.0 has this file in /usr/local/etc/modules/openmpi/1.4.0 (source):


prereq          modules
conflict	lam
conflict	mpich2
set             MOD_OPENMPI              $env(BCCD_SW)/openmpi
set      MOD_OPENMPI_VERSION      1.4.0
module-whatis "OPENMPI"

prepend-path    PATH                $MOD_OPENMPI_DIR/bin/
prepend-path    LD_LIBRARY_PATH     $MOD_OPENMPI_DIR/lib/
prepend-path    MANPATH             $MOD_OPENMPI_DIR/share/man/
prepend-path    -d " " LDFLAGS      -L$MOD_OPENMPI_DIR/lib/
prepend-path    -d " " CPPFLAGS     -I$MOD_OPENMPI_DIR/include/

if { [catch { exec /bin/bccd-snarfhosts $env(HOME)/machines openmpi} msg] } {
	puts "bccd-snarfhosts failed: $::errorInfo"

These module files should be placed in trees/usr/local/etc/modules/software-name/version, which is part of the MODULEPATH search path. module load will load the lexigraphically last software version by default, but you can override this behavior by symlinking the version you want to be default to default.

See the Fpm page for details on building .deb packages to install into Modules.

See the Software Build page for build information for these software packages.

USB automounting

The BCCD uses udev to detect when USB storage devices are plugged in, and mounts them automatically. The udev setup is in [1], and the script that the rule file calls is in [2].


BCCD uses SNMP to gather information from other BCCD nodes. Its unique IANA private enterprise number is 40551.

Licensing & copyright

SVN properties

Todo - Document how to do this in ~/.subversion/config

In every file

Put this at the top of every file, in whatever comment blocks make sense:

  This file is part of BCCD, an open-source live CD for computational science
  Copyright (C) 2010 Andrew Fitz Gibbon, Sam Leeman-Munk, Charlie Peck,
     Skylar Thompson, & Aaron Weeden
  This program is free software: you can redistribute it and/or modify
  it under the terms of the GNU General Public License as published by
  the Free Software Foundation, either version 3 of the License, or
  (at your option) any later version.
  This program is distributed in the hope that it will be useful,
  but WITHOUT ANY WARRANTY; without even the implied warranty of
  GNU General Public License for more details.
  You should have received a copy of the GNU General Public License
  along with this program.  If not, see <>.
Personal tools