Category Archives: Server config

Buildbot custom BuildStep with renderables example

Custom buildsteps need to:
* Accept and take any arguments in the __init__ method
* Call the parent class’ __init__ method
* Declare any attributes that need to be rendered in renderables attribute (this happens after __init__ and before run
* Do the action in the run method and return SUCCESS from this method

Here’s a simple example CustomStep designed to pass arguments directly to another class AnotherClass and call a method on the instance.

from buildbot.process.buildstep import BuildStep
from buildbot.process.results import SUCCESS

class CustomStep(BuildStep):
    name = 'customstep'
    description = 'Do the Thing'
    descriptionDone = 'Done the Thing'

    kwargs = None
    renderables = ['kwargs']

    rendered = True  # attributes are rendered

    def __init__(self, **kwargs):
        self.kwargs = kwargs
        super(CustomStep, self).__init__()

    def run(self):
        c = AnotherClass(**self.kwargs)
        return SUCCESS

And to run the step:

    'id': util.Property('id'),
    'image': util.Property('image'),
    'source': 'someplace',
    'url': util.Interpolate("https://%(prop:branch)"),

If your build raises an exception on the new buildstep but gives no output, check the buildbot master logs. If you get an error TypeError: list indices must be integers, not NoneType with this line: stepsumm += u' (%s)' % Results[self.results] then make sure your run returns SUCCESS.

Mysql service shows stopped when mysql is running after upgrade

In this case, the upgrade changed a line in the service init script /etc/init.d/mysql from if ! mysqladmin ping &>/dev/null;then to if ! $(mysqladmin --no-defaults --socket="$socket" --user=UNKNOWN_MYSQL_USER ping &>/dev/null) ;then.

Printing out $socket revealed it was empty. That value was coming from /usr/sbin/mysqld --print-defaults, which prints out the the configuration defined in the config file. I had not defined socket in my config file (instead just using mysql default socket location), so adding the line socket=/var/run/mysqld/mysqld.sock to /etc/mysql/my.cnf solved the problem.

Recovering VMs that were on local storage after removing host from XenServer pool

When you remove a host from a XenServer pool, the host gets reinitialized, so any VMs running locally get lost.  Luckily, it’s not too hard to recover the vdis from lvm.  Here’s an outline of the steps with some links that have more info / specific commands.

  1. If you can, join the host back to the pool and connect to your shared storage; this way you get the vms (that were moved to the pool when you added the host) and the vdis and only have to match the two together at the end
  2. Navigate to /etc/lvm/backup and find the file with the previous lvm data (the logical volumes should have all of your old vdis / snapshots, and it should have the relevant device path eg /dev/sda3)
  3. Find the current physical volume uuid
  4. Backup the /etc/lvm directory
  5. Modify the old volume group file and replace the old physical volume uuid with the current one
  6. Detach the local storage SR from the XenServer (see link below)
  7. Use vgcfgrestore to restore the old volume group file
  8. If you do vgscan you should see the newer volume group replaced with the old one (the name will be the same as the old one)
  9. Attach local storage SR to the XenServer with the current volume group name
  10. Create a new pbd with the SCSI ID and plug it in (see link below)
  11. Scan the new SR, it should pick up the old vdis but without any meta data.  If you create a new VM and attach these one-by-one as secondary disks, mount them to the new VM and check what they are, then you can rename them and attach back to your vms (that should be sitting in your pool).
  12. Move all the vdis you need over to your new SR, then you can remove your host again


Getting physical volume uuid and finding and modifying the file:
Removing SR:
Adding back local storage as an SR:

Adding mongo-10gen to apt-cacher (and Ubuntu)

On the server:

Add the following line to /etc/apt-cacher/apt-cacher.conf:
path_map = mongodb-10gen

Download the key and serve to clients (I rather add the key to the repo server and have clients download it from there, than connect out and get from the internet):
gpg –keyserver –recv-keys 7F0CEB10
gpg –armor –export 9958C967 >
python -m SimpleHTTPServer 8000


On client:

Create file /etc/apt/sources.list.d/10gen.list with the following contents:
deb http://your.apt-cacher.hostname:3142/mongodb-10gen dist 10gen

Download key from repo server:
wget http://your.apt-cacher.hostname:8000/
apt-key add
apt-get update

That should do it.  Then you can stop the python web server on the repo server.

Moving MS SQL 2008 database location

You cannot change the installation location (so master etc databases), but client databases can be moved like so:

First take the databases offline and move the mdf and ldf files to the new location, then do the following commands:

   Name = “db_name”,
   Filename = ‘Q:\sqldata\db_name.mdf’
   Name = “db_name_log”,
   Filename = ‘Q:\sqldata\db_name_log.LDF’


Taken from:

Getting pypicache running on Ubuntu 10.04

Pypicache is a great way to host a local pypi repository.  Unfortunately, it took some time for me to get it working under Ubuntu 10.04.

Pypicache is written for 2.7+, and ubuntu 10.04 uses 2.6.  Lucikly, the only backwards incompatibility seems to be string formatting.  Sooooo, get a copy of pypicache source and fix all the string formatting in the py files under the pypicache directory (alternatively, clone this: – might be out of date so check first)

Then, while in the directory with your copy of pypicache, run pip install -r requirements.txt –use-mirrors
(Or ‘make init’ if you don’t mind it downloading all the dev requirements too.)

Then ‘make runserver’ to run the server in debug mode, with the target directory /tmp/pypicache

Test the server runs ok and you can use for whatever you want to use it for (in my case, a pip proxy).  Then you can run it using ‘PYTHONPATH=. python -m pypicache.main /tmp/mypackages’

Adding s3ql to apt-cacher

On apt-cacher server

// add repo to apt-cacher /etc/apt-cacher/apt-cacher.conf
path_map = s3ql

// download the key and serve to clients (remember to exit web server process when you are done with the client)
gpg –keyserver –recv-keys 9958C967
gpg –armor –export 9958C967 >
python -m SimpleHTTPServer 8000

On client

// add to /etc/apt/sources.list
# s3ql
deb http://apt.cacher.hostname:3142/s3ql lucid main
// download and install the key
wget http://apt.cacher.hostname:8000/
apt-key add
apt-get update

Running composer in a git hook

If you’re trying to run composer from a git hook but it’s failing with “Failed to clone via git, https and http protocols, aborting”, make sure you unset the GIT_DIR env variable in your script.

Mine looks like this (post-receive hook in bash):

if [ -f “/path/to/composer/directory/composer.phar” ]; then
  unset GIT_DIR && cd “/path/to/composer/directory” && php composer.phar install


And full error:

remote: Loading composer repositories with package information[K
remote: Updating dependencies[K
remote:   – Installing pimple/pimple (dev-master b9f27b)[K
remote:     Cloning b9f27b8dc18c08f00627dec02359b46a24791dc3[K
remote: Initialized empty Git repository in /home/bootstrap/publichtml/dtest/php/lib/vendor/pimple/pimple/.git/[K
remote: Initialized empty Git repository in /home/bootstrap/publichtml/dtest/php/lib/vendor/pimple/pimple/.git/[K
remote: Initialized empty Git repository in /home/bootstrap/publichtml/dtest/php/lib/vendor/pimple/pimple/.git/[K
remote:                                                                                                     [K
remote:   [RuntimeException]                                                                                [K
remote:   Failed to clone via git, https and http protocols, aborting.  [K
remote:                                                                                                     [K
remote:   – git://                                                              [K
remote:     fatal: Not a git repository: ‘.’                                                                [K
remote:                                                                                                     [K
remote:   –                                                            [K
remote:     fatal: Not a git repository: ‘.’                                                                [K
remote:                                                                                                     [K
remote:   –                                                             [K
remote:     fatal: Not a git repository: ‘.’                                                                [K
remote:                                                                                                     [K
remote:                                                                                                     [K
remote: Exception trace:[K
remote:  () at phar:///home/bootstrap/publichtml/dtest/php/lib/composer.phar/src/Composer/Downloader/GitDownloader.php:224[K
remote:  Composer\Downloader\GitDownloader->throwException() at phar:///home/bootstrap/publichtml/dtest/php/lib/composer.phar/src/Composer/Downloader/GitDownloader.php:168[K
remote:  Composer\Downloader\GitDownloader->runCommand() at phar:///home/bootstrap/publichtml/dtest/php/lib/composer.phar/src/Composer/Downloader/GitDownloader.php:37[K
remote:  Composer\Downloader\GitDownloader->doDownload() at phar:///home/bootstrap/publichtml/dtest/php/lib/composer.phar/src/Composer/Downloader/VcsDownloader.php:56[K
remote:  Composer\Downloader\VcsDownloader->download() at phar:///home/bootstrap/publichtml/dtest/php/lib/composer.phar/src/Composer/Downloader/DownloadManager.php:150[K
remote:  Composer\Downloader\DownloadManager->download() at phar:///home/bootstrap/publichtml/dtest/php/lib/composer.phar/src/Composer/Installer/LibraryInstaller.php:151[K
remote:  Composer\Installer\LibraryInstaller->installCode() at phar:///home/bootstrap/publichtml/dtest/php/lib/composer.phar/src/Composer/Installer/LibraryInstaller.php:85[K
remote:  Composer\Installer\LibraryInstaller->install() at phar:///home/bootstrap/publichtml/dtest/php/lib/composer.phar/src/Composer/Installer/InstallationManager.php:132[K
remote:  Composer\Installer\InstallationManager->install() at phar:///home/bootstrap/publichtml/dtest/php/lib/composer.phar/src/Composer/Installer/InstallationManager.php:119[K
remote:  Composer\Installer\InstallationManager->execute() at phar:///home/bootstrap/publichtml/dtest/php/lib/composer.phar/src/Composer/Installer.php:495[K
remote:  Composer\Installer->doInstall() at phar:///home/bootstrap/publichtml/dtest/php/lib/composer.phar/src/Composer/Installer.php:180[K
remote:  Composer\Installer->run() at phar:///home/bootstrap/publichtml/dtest/php/lib/composer.phar/src/Composer/Command/UpdateCommand.php:76[K
remote:  Composer\Command\UpdateCommand->execute() at phar:///home/bootstrap/publichtml/dtest/php/lib/composer.phar/vendor/symfony/console/Symfony/Component/Console/Command/Command.php:239[K
remote:  Symfony\Component\Console\Command\Command->run() at phar:///home/bootstrap/publichtml/dtest/php/lib/composer.phar/vendor/symfony/console/Symfony/Component/Console/Application.php:193[K
remote:  Symfony\Component\Console\Application->doRun() at phar:///home/bootstrap/publichtml/dtest/php/lib/composer.phar/src/Composer/Console/Application.php:96[K
remote:  Composer\Console\Application->doRun() at phar:///home/bootstrap/publichtml/dtest/php/lib/composer.phar/vendor/symfony/console/Symfony/Component/Console/Application.php:106[K
remote:  Symfony\Component\Console\Application->run() at phar:///home/bootstrap/publichtml/dtest/php/lib/composer.phar/src/Composer/Console/Application.php:72[K
remote:  Composer\Console\Application->run() at phar:///home/bootstrap/publichtml/dtest/php/lib/composer.phar/bin/composer:37[K
remote:  require() at /home/bootstrap/publichtml/dtest/php/lib/composer.phar:15[K

Notes on adding new HDD space to XenServer VM using ext3

Ubuntu, for example, will use ext3 by default.  And on XenServer, you can’t boot into gparted (I think it should be possible, but I can’t get it to work).
If your VM is using LVM (eg CentOS), do not do this, there is an easier way.

The following instructions will not delete any data if you have one /root partition with all the data on that you’re increasing, and so long as you start the new partition at the same place on the disk.

Rough instructions:

shut down the VM
backup the VM
using xenserver, add the extra space
boot into recovery mode (have the distro iso in the drive, and go to toolbar > VM > start/shutdown > start in recovery mode
in recovery mode, bring up a shell running from the recovery media (nothing mounted)
fdisk /dev/sda
p # print out the current partition table, take a note of the start of the root partition, the type codes of all partitions, and optionally the size of additional partitions
d # delete all partitions
set up the partitions again (for swap, first add two other partitions – in ubuntu this is sda1 primary and sda2 extended, then label 5 for swap will be available)
t # change type of partitions if necessary
wq # write the partitions

need to restart here?  go back to rescue mode if so

e2fsck -f /dev/sda1
resize2fs /dev/sda1
resize2fs /dev/sda5 (might need to do e2fsck first – try resize2fs and it may tell you to do e2fsck)
(don’t think you need to for /dev/sda2??)

restart the computer, not rescue mode

check fdisk -l shows the partitions

umount /dev/sda5
mkswap /dev/sda5
swapon /dev/sda5

blkid # get partition ids (should also have given you swap id when did mkswap command)
vi /etc/fstab # make sure swap partition is correct, like this:
  UUID=3ddbc973-d84c-4dad-ba09-473f46f72c32 none  swap    swap    sw      0     0
restart, not rescue mode

check swap mounted
free -m #should show swap with some space