Building debs on one machine with Pbuilder

If you have a technical writeup you would like to share, feel free to post it in the articles submission area within.
Post Reply
dandruczyk
LQFP112 - Up with the play
Posts: 100
Joined: Sun Apr 06, 2008 6:30 pm

Building debs on one machine with Pbuilder

Post by dandruczyk »

This mini-tutorial is copyright David J. Andruczyk Jan 1st 2012 based uponvarious howto's scattered across the internet with a little personal experienced tossed in.

Goal: To be able to build debian/ubuntu packages for multiple releases without needed a machine for each release.

Tools used:
1. apt-cacher-ng (to save from having to download the same stuff over and over again)
2. pbuilder with appropriate configuration and setup (see below)
3. Debian or ubuntu machine. This tutorial is based on Ubuntu 10.04 running on an i686 32 bit CPU.

Requirements:
1. Linux machine (64 bit required if you want to make 64 bit packages), this tutorial is based on Ubuntu 10.04, other version/os your mileage will vary.
2. Plenty of space under /var/cache. apt-cache-ng will cache every pkg downloaded into /var/cache/apt-cacher-ng, and pbuilder defaults to using /var/cache/pbuilder as it's root.

NOTE: if you want to build 64 bit packages you MUST be on a 64 bit capable machine with a 64 bit OS, otherwise you're restricted to making 32 bit packages only.

First off install the tools:

Code: Select all

apt-get install apt-cacher-ng pbuilder debootstrap quilt
apt-cacher-ng acts like a debian or ubuntu mirror/proxy listening on port 3142. So you can take an /etc/apt/sources entry and change the hostname to "localhost:3142", or setup apt to use a http proxy of http://localhost:3142 and run apt-get update and it should work, when a request for a file comes in, it will serve it from the cache if present, otherwise it'll download and stream it to the client at the same time populating the cache, thus hte next time around it just uses the cache saving lots of time. The cache is intelligent enough to always give you current stuff.

pbuilder is used to make clean pristine chroot's for each distro and architecture target (for 32/64 bit pkgs), and hence it requires some configuration so things go into the right place as well as handling the special case of sequential package creation where a latter package depends on the former package being available within the buildroot to work (case in point the FreeEMS toolchain compiler depends on binutils which isn't available in the normal repository), so some magic needs to happen in order to make the appropriate stuff available in the build root.

Special steps:

Code: Select all

mkdir /usr/lib/pbuilder/hooks
mkdir /var/cache/pbuilder/repo
create /usr/lib/pbuilder/hooks/D05deps with the following content

Code: Select all

#!/bin/sh
(cd /var/cache/pbuilder/repo; apt-ftparchive packages . > Packages && gzip -c Packages >Packages.gz)
apt-get update
make it executable

Code: Select all

chmod +x /usr/lib/pbuilder/hooks/D05deps
If you are on an "older" linux distro (I'm on 11.04) you might need to create a symlink or two in /usr/share/deboostrap/scripts so that debootstrap knows what to do.

Copy and paste the following into a terminal

Code: Select all

cd /usr/share/debootstrap/scripts
for i in `echo wheezy squeeze lenny` ; do if test ! -L ${i} ; then echo "Creating missing link for ${i}" ; sudo ln -s sid ${i} ; fi ; done
for i in `echo precise oneiric natty maverick lucid hardy` ; do if test ! -L ${i} ; then echo "Creating missing link for ${i}" ; sudo ln -s gutsy ${i} ; fi ; done
Create /etc/pbuilderrc which is the main pbuilder system wide config

Code: Select all

# this is your configuration file for pbuilder.
# the file in /usr/share/pbuilder/pbuilderrc is the default template.
# /etc/pbuilderrc is the one meant for editing.
#
# read pbuilderrc.5 document for notes on specific options.

BASETGZ=/var/cache/pbuilder/base.tgz
BUILDPLACE=/var/cache/pbuilder/build/
OTHERMIRROR="deb file:///var/cache/pbuilder/repo ./"
USEPROC=yes
USEDEVPTS=yes
USEDEVFS=no
BUILDRESULT=/var/cache/pbuilder/result/

REMOVEPACKAGES="lilo"
HOOKDIR="/usr/lib/pbuilder/hooks"
# make debconf not interact with user
export DEBIAN_FRONTEND="noninteractive"
DEBEMAIL=""
# for pbuilder debuild (sudo -E keeps the environment as-is)
BUILDSOURCEROOTCMD="fakeroot"
PBUILDERROOTCMD="sudo -E"

# command to satisfy build-dependencies; the default is an internal shell
# implementation which is relatively slow; there are two alternate
# implementations, the "experimental" implementation,
# "pbuilder-satisfydepends-experimental", which might be useful to pull
# packages from experimental or from repositories with a low APT Pin Priority,
# and the "aptitude" implementation, which will resolve build-dependencies and
# build-conflicts with aptitude which helps dealing with complex cases but does
# not support unsigned APT repositories
PBUILDERSATISFYDEPENDSCMD="/usr/lib/pbuilder/pbuilder-satisfydepends"

#Command-line option passed on to dpkg-buildpackage.
DEBBUILDOPTS=""

#APT configuration files directory
APTCONFDIR=""
# the username and ID used by pbuilder, inside chroot. Needs fakeroot, really
BUILDUSERID=1234
BUILDUSERNAME=pbuilder

# BINDMOUNTS is a space separated list of things to mount
# inside the chroot.
BINDMOUNTS="/var/cache/pbuilder/repo"

# Set the debootstrap variant to 'buildd' type.
# DEBOOTSTRAPOPTS[0]='--variant=buildd'
# or work around bug in debootstrap 3.0.0 (314858)
unset DEBOOTSTRAPOPTS

# Set the PATH I am going to use inside pbuilder: default is "/usr/sbin:/usr/bin:/sbin:/bin:/usr/X11R6/bin"
export PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/X11R6/bin"

# SHELL variable is used inside pbuilder by commands like 'su'; and they need sane values
export SHELL=/bin/bash

# The name of debootstrap command.
DEBOOTSTRAP="debootstrap"

# default file extension for pkgname-logfile
PKGNAME_LOGFILE_EXTENTION="_$(dpkg --print-architecture).build"

# default PKGNAME_LOGFILE
PKGNAME_LOGFILE=""

I chose to also have a PER USER ~/.pbuilderrc script which overrides some of the defaults for some specific cases of using pbuilder, I recommend using this as a baseline and adjusting it to your needs as you see fit.

~/.pbuilderrc

Code: Select all

# Codenames for Debian suites according to their alias. Update these when
# needed.
UNSTABLE_CODENAME="sid"
TESTING_CODENAME="squeeze"
STABLE_CODENAME="lenny"
STABLE_BACKPORTS_SUITE="$STABLE_CODENAME-backports"

# List of Debian suites.
DEBIAN_SUITES=($UNSTABLE_CODENAME $TESTING_CODENAME $STABLE_CODENAME
    "unstable" "testing" "stable")

# List of Ubuntu suites. Update these when needed.
UBUNTU_SUITES=("precise" "oneiric" "natty" "maverick" "lucid" "hardy")

# Mirrors to use. Update these to your preferred mirror.
DEBIAN_MIRROR="ftp.us.debian.org"
UBUNTU_MIRROR="mirrors.kernel.org"

# Optionally use the changelog of a package to determine the suite to use if
# none set.
if [ -z "${DIST}" ] && [ -r "debian/changelog" ]; then
    DIST=$(dpkg-parsechangelog | awk '/^Distribution: / {print $2}')
    # Use the unstable suite for certain suite values.
    if $(echo "experimental UNRELEASED" | grep -q $DIST); then
        DIST="$UNSTABLE_CODENAME"
    fi
fi

# Optionally set a default distribution if none is used. Note that you can set
# your own default (i.e. ${DIST:="unstable"}).
: ${DIST:="$(lsb_release --short --codename)"}

# Optionally change Debian release states in $DIST to their names.
case "$DIST" in
    unstable)
        DIST="$UNSTABLE_CODENAME"
        ;;
    testing)
        DIST="$TESTING_CODENAME"
        ;;
    stable)
        DIST="$STABLE_CODENAME"
        ;;
esac

# Optionally set the architecture to the host architecture if none set. Note
# that you can set your own default (i.e. ${ARCH:="i386"}).
: ${ARCH:="$(dpkg --print-architecture)"}

NAME="$DIST"
if [ -n "${ARCH}" ]; then
    NAME="$NAME-$ARCH"
    DEBOOTSTRAPOPTS=("--arch" "$ARCH" "${DEBOOTSTRAPOPTS[@]}")
fi
BASETGZ="/var/cache/pbuilder/$NAME-base.tgz"
# Optionally, set BASEPATH (and not BASETGZ) if using cowbuilder
# BASEPATH="/var/cache/pbuilder/$NAME/base.cow/"
DISTRIBUTION="$DIST"
BUILDRESULT="/var/cache/pbuilder/$NAME/result/"
APTCACHE="/var/cache/pbuilder/aptcache/$NAME/"
BUILDPLACE="/var/cache/pbuilder/build/"

if $(echo ${DEBIAN_SUITES[@]} | grep -q $DIST); then
    # Debian configuration
    MIRRORSITE="http://$DEBIAN_MIRROR/debian/"
    COMPONENTS="main contrib non-free"
    # This is for enabling backports for the Debian stable suite.
    #if $(echo "$STABLE_CODENAME stable" | grep -q $DIST); then
    #    EXTRAPACKAGES="$EXTRAPACKAGES debian-backports-keyring"
    #    OTHERMIRROR="$OTHERMIRROR | deb http://www.backports.org/debian $STABLE_BACKPORTS_SUITE $COMPONENTS"
    #fi
elif $(echo ${UBUNTU_SUITES[@]} | grep -q $DIST); then
    # Ubuntu configuration
    MIRRORSITE="http://$UBUNTU_MIRROR/ubuntu/"
    COMPONENTS="main restricted universe multiverse"
else
    echo "Unknown distribution: $DIST"
    exit 1
fi

OK, now that pbuilder is configured you will need to init the base build roots to start with. This takes some time.

For 32 bit machines

Code: Select all

for dist in `echo wheezy squeeze lenny precise oneiric natty maverick lucid hardy` ; do sudo DIST=${dist} pbuilder --create --architecture i386 --http-proxy http://localhost:3142 --distribution ${dist}  --basetgz /var/cache/pbuilder/${dist}-i386-base.tgz ; done
For 64 bit machines where you want to be able to make both 32 and 64 bit packages:

Code: Select all

for arch in `echo i386 amd64` ; do for dist in `echo wheezy squeeze lenny precise oneiric natty maverick lucid hardy` ; do sudo DIST=${dist} ARCH=${arch} pbuilder --create --architecture ${arch} --distribution ${dist}  --http-proxy http://localhost:3142 --basetgz /var/cache/pbuilder/${dist}-${arch}-base.tgz ; done ; done
To build something for real:
If you're within a package that has a "debian" or "DEBIAN" dir its a simple command:

Code: Select all

pdebuild --architecture <i386|amd64> --buildresult /tmp --pbuilderroot "sudo DIST=<lenny|squeeze|wheezy|precise|oneiric|natty|maverick|lucid|hardy> ARCH=<i386|amd64>"
This will do all the debian magic and complain as needed if you don't have everything perfectly setup.

When complete the output files will be in /tmp as per the "--buildresult" argument.

See the manual page for pbuilder and pdebuild for additional methods of use.
Last edited by dandruczyk on Tue Feb 28, 2012 2:48 am, edited 5 times in total.
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: Building debs on one machine with Pbuilder

Post by Fred »

Awesome work, Dave! Comments: viewtopic.php?f=35&t=1506
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
Post Reply