Category Archives: Server config

Recovering VMs that were on local storage after removing host from XenServer pool

When you remove a host from a XenServer pool, the host gets reinitialized, so any VMs running locally get lost.  Luckily, it’s not too hard to recover the vdis from lvm.  Here’s an outline of the steps with some links that have more info / specific commands.

  1. If you can, join the host back to the pool and connect to your shared storage; this way you get the vms (that were moved to the pool when you added the host) and the vdis and only have to match the two together at the end
  2. Navigate to /etc/lvm/backup and find the file with the previous lvm data (the logical volumes should have all of your old vdis / snapshots, and it should have the relevant device path eg /dev/sda3)
  3. Find the current physical volume uuid
  4. Backup the /etc/lvm directory
  5. Modify the old volume group file and replace the old physical volume uuid with the current one
  6. Detach the local storage SR from the XenServer (see link below)
  7. Use vgcfgrestore to restore the old volume group file
  8. If you do vgscan you should see the newer volume group replaced with the old one (the name will be the same as the old one)
  9. Attach local storage SR to the XenServer with the current volume group name
  10. Create a new pbd with the SCSI ID and plug it in (see link below)
  11. Scan the new SR, it should pick up the old vdis but without any meta data.  If you create a new VM and attach these one-by-one as secondary disks, mount them to the new VM and check what they are, then you can rename them and attach back to your vms (that should be sitting in your pool).
  12. Move all the vdis you need over to your new SR, then you can remove your host again


Getting physical volume uuid and finding and modifying the file:
Removing SR:
Adding back local storage as an SR:

Adding mongo-10gen to apt-cacher (and Ubuntu)

On the server:

Add the following line to /etc/apt-cacher/apt-cacher.conf:
path_map = mongodb-10gen

Download the key and serve to clients (I rather add the key to the repo server and have clients download it from there, than connect out and get from the internet):
gpg –keyserver –recv-keys 7F0CEB10
gpg –armor –export 9958C967 >
python -m SimpleHTTPServer 8000


On client:

Create file /etc/apt/sources.list.d/10gen.list with the following contents:
deb http://your.apt-cacher.hostname:3142/mongodb-10gen dist 10gen

Download key from repo server:
wget http://your.apt-cacher.hostname:8000/
apt-key add
apt-get update

That should do it.  Then you can stop the python web server on the repo server.

Moving MS SQL 2008 database location

You cannot change the installation location (so master etc databases), but client databases can be moved like so:

First take the databases offline and move the mdf and ldf files to the new location, then do the following commands:

   Name = “db_name”,
   Filename = ‘Q:\sqldata\db_name.mdf’
   Name = “db_name_log”,
   Filename = ‘Q:\sqldata\db_name_log.LDF’


Taken from:

Getting pypicache running on Ubuntu 10.04

Pypicache is a great way to host a local pypi repository.  Unfortunately, it took some time for me to get it working under Ubuntu 10.04.

Pypicache is written for 2.7+, and ubuntu 10.04 uses 2.6.  Lucikly, the only backwards incompatibility seems to be string formatting.  Sooooo, get a copy of pypicache source and fix all the string formatting in the py files under the pypicache directory (alternatively, clone this: – might be out of date so check first)

Then, while in the directory with your copy of pypicache, run pip install -r requirements.txt –use-mirrors
(Or ‘make init’ if you don’t mind it downloading all the dev requirements too.)

Then ‘make runserver’ to run the server in debug mode, with the target directory /tmp/pypicache

Test the server runs ok and you can use for whatever you want to use it for (in my case, a pip proxy).  Then you can run it using ‘PYTHONPATH=. python -m pypicache.main /tmp/mypackages’

Adding s3ql to apt-cacher

On apt-cacher server

// add repo to apt-cacher /etc/apt-cacher/apt-cacher.conf
path_map = s3ql

// download the key and serve to clients (remember to exit web server process when you are done with the client)
gpg –keyserver –recv-keys 9958C967
gpg –armor –export 9958C967 >
python -m SimpleHTTPServer 8000

On client

// add to /etc/apt/sources.list
# s3ql
deb http://apt.cacher.hostname:3142/s3ql lucid main
// download and install the key
wget http://apt.cacher.hostname:8000/
apt-key add
apt-get update

Running composer in a git hook

If you’re trying to run composer from a git hook but it’s failing with “Failed to clone via git, https and http protocols, aborting”, make sure you unset the GIT_DIR env variable in your script.

Mine looks like this (post-receive hook in bash):

if [ -f “/path/to/composer/directory/composer.phar” ]; then
  unset GIT_DIR && cd “/path/to/composer/directory” && php composer.phar install


And full error:

remote: Loading composer repositories with package information[K
remote: Updating dependencies[K
remote:   – Installing pimple/pimple (dev-master b9f27b)[K
remote:     Cloning b9f27b8dc18c08f00627dec02359b46a24791dc3[K
remote: Initialized empty Git repository in /home/bootstrap/publichtml/dtest/php/lib/vendor/pimple/pimple/.git/[K
remote: Initialized empty Git repository in /home/bootstrap/publichtml/dtest/php/lib/vendor/pimple/pimple/.git/[K
remote: Initialized empty Git repository in /home/bootstrap/publichtml/dtest/php/lib/vendor/pimple/pimple/.git/[K
remote:                                                                                                     [K
remote:   [RuntimeException]                                                                                [K
remote:   Failed to clone via git, https and http protocols, aborting.  [K
remote:                                                                                                     [K
remote:   – git://                                                              [K
remote:     fatal: Not a git repository: ‘.’                                                                [K
remote:                                                                                                     [K
remote:   –                                                            [K
remote:     fatal: Not a git repository: ‘.’                                                                [K
remote:                                                                                                     [K
remote:   –                                                             [K
remote:     fatal: Not a git repository: ‘.’                                                                [K
remote:                                                                                                     [K
remote:                                                                                                     [K
remote: Exception trace:[K
remote:  () at phar:///home/bootstrap/publichtml/dtest/php/lib/composer.phar/src/Composer/Downloader/GitDownloader.php:224[K
remote:  Composer\Downloader\GitDownloader->throwException() at phar:///home/bootstrap/publichtml/dtest/php/lib/composer.phar/src/Composer/Downloader/GitDownloader.php:168[K
remote:  Composer\Downloader\GitDownloader->runCommand() at phar:///home/bootstrap/publichtml/dtest/php/lib/composer.phar/src/Composer/Downloader/GitDownloader.php:37[K
remote:  Composer\Downloader\GitDownloader->doDownload() at phar:///home/bootstrap/publichtml/dtest/php/lib/composer.phar/src/Composer/Downloader/VcsDownloader.php:56[K
remote:  Composer\Downloader\VcsDownloader->download() at phar:///home/bootstrap/publichtml/dtest/php/lib/composer.phar/src/Composer/Downloader/DownloadManager.php:150[K
remote:  Composer\Downloader\DownloadManager->download() at phar:///home/bootstrap/publichtml/dtest/php/lib/composer.phar/src/Composer/Installer/LibraryInstaller.php:151[K
remote:  Composer\Installer\LibraryInstaller->installCode() at phar:///home/bootstrap/publichtml/dtest/php/lib/composer.phar/src/Composer/Installer/LibraryInstaller.php:85[K
remote:  Composer\Installer\LibraryInstaller->install() at phar:///home/bootstrap/publichtml/dtest/php/lib/composer.phar/src/Composer/Installer/InstallationManager.php:132[K
remote:  Composer\Installer\InstallationManager->install() at phar:///home/bootstrap/publichtml/dtest/php/lib/composer.phar/src/Composer/Installer/InstallationManager.php:119[K
remote:  Composer\Installer\InstallationManager->execute() at phar:///home/bootstrap/publichtml/dtest/php/lib/composer.phar/src/Composer/Installer.php:495[K
remote:  Composer\Installer->doInstall() at phar:///home/bootstrap/publichtml/dtest/php/lib/composer.phar/src/Composer/Installer.php:180[K
remote:  Composer\Installer->run() at phar:///home/bootstrap/publichtml/dtest/php/lib/composer.phar/src/Composer/Command/UpdateCommand.php:76[K
remote:  Composer\Command\UpdateCommand->execute() at phar:///home/bootstrap/publichtml/dtest/php/lib/composer.phar/vendor/symfony/console/Symfony/Component/Console/Command/Command.php:239[K
remote:  Symfony\Component\Console\Command\Command->run() at phar:///home/bootstrap/publichtml/dtest/php/lib/composer.phar/vendor/symfony/console/Symfony/Component/Console/Application.php:193[K
remote:  Symfony\Component\Console\Application->doRun() at phar:///home/bootstrap/publichtml/dtest/php/lib/composer.phar/src/Composer/Console/Application.php:96[K
remote:  Composer\Console\Application->doRun() at phar:///home/bootstrap/publichtml/dtest/php/lib/composer.phar/vendor/symfony/console/Symfony/Component/Console/Application.php:106[K
remote:  Symfony\Component\Console\Application->run() at phar:///home/bootstrap/publichtml/dtest/php/lib/composer.phar/src/Composer/Console/Application.php:72[K
remote:  Composer\Console\Application->run() at phar:///home/bootstrap/publichtml/dtest/php/lib/composer.phar/bin/composer:37[K
remote:  require() at /home/bootstrap/publichtml/dtest/php/lib/composer.phar:15[K

Notes on adding new HDD space to XenServer VM using ext3

Ubuntu, for example, will use ext3 by default.  And on XenServer, you can’t boot into gparted (I think it should be possible, but I can’t get it to work).
If your VM is using LVM (eg CentOS), do not do this, there is an easier way.

The following instructions will not delete any data if you have one /root partition with all the data on that you’re increasing, and so long as you start the new partition at the same place on the disk.

Rough instructions:

shut down the VM
backup the VM
using xenserver, add the extra space
boot into recovery mode (have the distro iso in the drive, and go to toolbar > VM > start/shutdown > start in recovery mode
in recovery mode, bring up a shell running from the recovery media (nothing mounted)
fdisk /dev/sda
p # print out the current partition table, take a note of the start of the root partition, the type codes of all partitions, and optionally the size of additional partitions
d # delete all partitions
set up the partitions again (for swap, first add two other partitions – in ubuntu this is sda1 primary and sda2 extended, then label 5 for swap will be available)
t # change type of partitions if necessary
wq # write the partitions

need to restart here?  go back to rescue mode if so

e2fsck -f /dev/sda1
resize2fs /dev/sda1
resize2fs /dev/sda5 (might need to do e2fsck first – try resize2fs and it may tell you to do e2fsck)
(don’t think you need to for /dev/sda2??)

restart the computer, not rescue mode

check fdisk -l shows the partitions

umount /dev/sda5
mkswap /dev/sda5
swapon /dev/sda5

blkid # get partition ids (should also have given you swap id when did mkswap command)
vi /etc/fstab # make sure swap partition is correct, like this:
  UUID=3ddbc973-d84c-4dad-ba09-473f46f72c32 none  swap    swap    sw      0     0
restart, not rescue mode

check swap mounted
free -m #should show swap with some space



Adding a new sudo user to CentOs

Our new user will be called batman:

  1. groupadd admins
  2. adduser batman -G admins
  3. passwd batman
  4. visudo (this will open sudoers file in vim)
  5. Add this to the file:
    %admins     ALL=(ALL)     ALL

Any users in the group called ‘admins’ will be sudoers.  So additional users just need to be added to this group.

Getting certificates from Windows p7b certificate files

I was given a binary certificate chain and needed the ASCII versions of the CA certificates.  Unfortunately, I wasn’t able to get openssl to do this, and had to resort to using a Windows machine (on Win7 I got an ‘install certificate’ menu item when right-clicking on the p7b file in windows explorer).  Double click on the file and you’ll be able to navigate to the certificates in the window that opens (with crypto shell extensions).  You can then right-click > all tasks > export for the individual certificates, and export into a der file, which openssl can then convert to pem.

Here are the errors I got from various ssl commands that I tried:

user@ubuntu:~/certs$ openssl x509 -inform der -in Certificate.p7b -out Certificate.pem
unable to load certificate
31083:error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1316:
31083:error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:380:Type=X509_CINF
31083:error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:748:Field=cert_info, Type=X509

user@ubuntu:~/certs$ openssl pkcs7 -print_certs -in Certificate.p7b -out Certificate.pem
unable to load PKCS7 object
31109:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:650:Expecting: PKCS7

user@ubuntu:~/certs$ openssl pkcs12 -in Certificate.p7b
31162:error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1316:
31162:error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:828:
31162:error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:748:Field=version, Type=PKCS12

user@ubuntu:~/certs$ openssl nseq -in Certificate.p7b
Error reading sequence file Certificate.p7b
31475:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:650:Expecting: CERTIFICATE