Tag Archives: xenserver

Recovering VMs that were on local storage after removing host from XenServer pool

When you remove a host from a XenServer pool, the host gets reinitialized, so any VMs running locally get lost.  Luckily, it’s not too hard to recover the vdis from lvm.  Here’s an outline of the steps with some links that have more info / specific commands.

  1. If you can, join the host back to the pool and connect to your shared storage; this way you get the vms (that were moved to the pool when you added the host) and the vdis and only have to match the two together at the end
  2. Navigate to /etc/lvm/backup and find the file with the previous lvm data (the logical volumes should have all of your old vdis / snapshots, and it should have the relevant device path eg /dev/sda3)
  3. Find the current physical volume uuid
  4. Backup the /etc/lvm directory
  5. Modify the old volume group file and replace the old physical volume uuid with the current one
  6. Detach the local storage SR from the XenServer (see link below)
  7. Use vgcfgrestore to restore the old volume group file
  8. If you do vgscan you should see the newer volume group replaced with the old one (the name will be the same as the old one)
  9. Attach local storage SR to the XenServer with the current volume group name
  10. Create a new pbd with the SCSI ID and plug it in (see link below)
  11. Scan the new SR, it should pick up the old vdis but without any meta data.  If you create a new VM and attach these one-by-one as secondary disks, mount them to the new VM and check what they are, then you can rename them and attach back to your vms (that should be sitting in your pool).
  12. Move all the vdis you need over to your new SR, then you can remove your host again


Getting physical volume uuid and finding and modifying the file: http://support.citrix.com/article/CTX128097
Removing SR: http://support.citrix.com/article/CTX131328
Adding back local storage as an SR: http://support.citrix.com/article/CTX121896

Notes on fixing XenServer VDIs

If get the VDI is not available when scanning SR

xe sr-scan uuid=[uuid of the SR]

Should give you a more verbose error message, also
Should give an even more verbose error, eg VDI header error

Get the VDI uuid
xe vdi-list

Forget the VDI and re-scan the SR
xe vdi-forget uuid=[vdi uuid]
xe sr-scan uuid=[sr uuid]

Might fix the issue, otherwise, if need to preserve the data, need to restart the host
If can trash the data, can try to delete the vdi

xe vdi-destroy uuid=[vdi uuid]

Might not work, if not, try restart.  May also need to manually remove the LVM volume

lvremove /dev/VG_XenStorage-[uuid of sr]/VHD-[uuid of vdi]

Then restart the machine or Xen is confused by missing volume

If really need the data, there might be a way to fix broken headers/footers.  If the data on the VDI has issues, though, create a new VDI (larger than original), use dd to copy data from broken VDI to new one, mount new one and use a recovery tool on the VM to recover data.

Instructions taken from post on Citrix Forums by Fabian Baena:

xe vdi-create sr-uuid=bc4c43f3-1321-2b17-bef0-3b58686a8075 name-label=copy virtual-size=210130436096

the sr-uuid is the storage where you want to put the copy. Take note of the uuid that comes up after you execute the command

get the uuid of your xenserver control domain by doing
xe vm-list name-label=Control\ domain\ on\ host:\ <name of your xenserver host> params=uuid

then create the vbd
xe vbd-create vm-uuid=<;vm-uuid you got from the previous command>; vdi-uuid=&lt;vdi uuid you got from the vdi-create command> device=0

plug the vbd
xe vbd-plug uuid=<vbd uuid you got from the previous command>;

then do the copy

dd if=/dev/mapper/VG_XenStorage–bc4c43f3–1321–2b17–bef0–3b58686a8075-VHD–b079b55a–6679–47a1–b2a2–d207a476494e of=/dev/xvda

The copy will take several minutes. When finished unplug the vbd

xe vbd-unplug uuid=<vbd uuid you got from the vbd-create command>
xe vbd-destroy uuid=<vbd uuid you got from the vbd-create command>

Afterwards, you can connect the new vdi to a vm and see if you can recover anything.

Notes on adding new HDD space to XenServer VM using ext3

Ubuntu, for example, will use ext3 by default.  And on XenServer, you can’t boot into gparted (I think it should be possible, but I can’t get it to work).
If your VM is using LVM (eg CentOS), do not do this, there is an easier way.

The following instructions will not delete any data if you have one /root partition with all the data on that you’re increasing, and so long as you start the new partition at the same place on the disk.

Rough instructions:

shut down the VM
backup the VM
using xenserver, add the extra space
boot into recovery mode (have the distro iso in the drive, and go to toolbar > VM > start/shutdown > start in recovery mode
in recovery mode, bring up a shell running from the recovery media (nothing mounted)
fdisk /dev/sda
p # print out the current partition table, take a note of the start of the root partition, the type codes of all partitions, and optionally the size of additional partitions
d # delete all partitions
set up the partitions again (for swap, first add two other partitions – in ubuntu this is sda1 primary and sda2 extended, then label 5 for swap will be available)
t # change type of partitions if necessary
wq # write the partitions

need to restart here?  go back to rescue mode if so

e2fsck -f /dev/sda1
resize2fs /dev/sda1
resize2fs /dev/sda5 (might need to do e2fsck first – try resize2fs and it may tell you to do e2fsck)
(don’t think you need to for /dev/sda2??)

restart the computer, not rescue mode

check fdisk -l shows the partitions

umount /dev/sda5
mkswap /dev/sda5
swapon /dev/sda5

blkid # get partition ids (should also have given you swap id when did mkswap command)
vi /etc/fstab # make sure swap partition is correct, like this:
  UUID=3ddbc973-d84c-4dad-ba09-473f46f72c32 none  swap    swap    sw      0     0
restart, not rescue mode

check swap mounted
free -m #should show swap with some space



Installing CentOS 6.2 on XenServer 6.0

Choose to install from URL, and enter the following URL (or your nearest mirror equivalent): http://mirror01.th.ifl.net/centos/6.2/os/x86_64/

The installer is looking for the following path: isolinux/isolinux.cfg

Choose to start VNC (as per: http://forums.citrix.com/thread.jspa?threadID=302677)

Start up your VNC client (eg tightVNC), connect to <IP>:1, put in your password

Once you’ve finished the installation you can install XenServer Tools:

  1. Put the XS Tools cd into the VM cd drive
  2. mkdir /mnt/xs-tools
  3. mount /dev/xvdd /mnt/xs-tools
  4. bash /mnt/xs-tools/Linux/install.sh

This should auto-detect your OS, install required packages, and ask you to reboot the machine.

Accessing a GUI on Ubuntu XenServer VM

If you try and start the GUI on a paravirtualized Ubuntu VM in XenServer, you’ll get the following error:

Primary device is not PCI
(EE) open /dev/fb0: No such file or directory
(EE) No devices detected

Peter Bats from Citrix said the following:
In a paravirtualized world there is no such thing as a physical console (nor is there a physical CPU, physical memory etc). Hence for completely paravirtualized OSes (with a paravirtualized kernel like Xen) there’s no GUI console.

PS: With the upcoming move/approach of something called PV on HVM (Paravirtualized Linux I/O drivers for a HVM machine, that in this was can profit from the latest developments by Intel/AMD at the hardware levels like EPT/NPT etc.) one will have the option to have both. And a hardware console GUI and good paravirtualized I/O.
This is already available for the RHEL/SLES distros, but not for the general Linux kernel or debian based distributions. There’s work underway to get this included into upstream Linux very soon.


In other words, use VNC for now.

To do that:

  1. Install a GUI onto your Ubuntu server:
    apt-get install ubuntu-desktop (for gnome)
    apt-get install –without-recommends ubuntu-desktop (for gnome without libreoffice and some other stuff)
    apt-get install xubuntu-desktop (xfce)
    apt-get install kubuntu-desktop (KDE)
  2. Install VNC
    apt-get install vnc4server
  3. Set the VNC resolution (whatever resolution you want to see on your desktop machine you’ll be using the VNC client on
    vncserver -geometry 1280×1024 -depth 24
  4. Create a password and VNC server should create some configuration files and start up
  5. Now we need to edit one of the configuration files
    vncserver -kill :1
    vim ~/.vnc/xstartup
  6. Change the file to look like the following (Ubuntu 10.04 and 10.10 only, other versions look different):

    # Uncomment the following two lines for normal desktop:
    exec sh /etc/X11/xinit/xinitrc

    [ -x /etc/vnc/xstartup ] && exec /etc/vnc/xstartup
    [ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources
    xsetroot -solid grey
    vncconfig -iconic &
    x-terminal-emulator -geometry 1280×1024+10+10 -ls -title “$VNCDESKTOP Desktop” &
    x-window-manager &

  7. Save and quit vim
  8. Start up the VNC server again
    vncserver -geometry 1280×1024 -depth 24
  9. Now you can connect to the server via a VNC client.  On windows, you can use TightVNC
  10. To connect to the server, you want to use IP:1, for example, my server’s IP is, then I need to connect to
  11. Put in the VNC password when requested by TightVNC and it should bring up the server’s desktop




XenServer install one-off CentOS packages (aka install a package from a disabled repo with yum)

This is not specific to XenServer, or to CentOS, but it was what I was trying to do at the time.

XenServer, by default, only has the Citrix repository enabled, so there will be lots of packages you can’t install.

If you want to install a package but don’t want to enable the CentOS repository, you can use the following command:

yum –enablerepo=base install mutt

This installs mutt from the base repo.  If you want a list of available repos, you can do the following:

yum repolist disabled

Example output:

[root@testxenserver yum.repos.d]# yum repolist disabled
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
repo id                         repo name                               status
addons                          CentOS-5 – Addons                       disabled
base                            CentOS-5 – Base                         disabled
c5-media                        CentOS-5 – Media                        disabled
centosplus                      CentOS-5 – Plus                         disabled
contrib                         CentOS-5 – Contrib                      disabled
extras                          CentOS-5 – Extras                       disabled
updates                         CentOS-5 – Updates                      disabled
repolist: 0

And to see enabled repos:

[root@testxenserver yum.repos.d]# yum repolist
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
repo id                   repo name                                   status
citrix                    XenServer 5.6.100 updates                   enabled: 0
repolist: 0

Notes on XenServer and NTP

  • XenServer syncs with the NTP server anytime between once every one minute and once every 15-20 minutes, depending on the amount of “drift” your hardware clock exhibits (ref)
  • XenServer syncs the hardware clock with the system clock automatically – I think when ntp starts (ref: /etc/rc.d/init.d/ntpd – on XenServer 5.6 SP 2)
  • VMs (both with and without XenServer Tools) have their hardware clocks synced with XenServers system clock when they start up – if you try and use a command like “hwclock -r” and get the message that the hardware clock cannot be accessed; do not worry, it still works



Changing XenServer bonding mode back to default (or reducing bond switchover up-delay)

You need to get the pifs again (or ideally, find the lines you used to change the mode in your bash history) – see here for getting pifs.

Then use the following:
xe pif-param-set uuid=bd26334f-f7ff-3206-e5b2-4c017c51907c other-config:bond-mode=balance-slb other-config:bond-updelay=200

This will set your bond back to balance-slb, and reduce the up-delay time (see here for the issue – although I can’t actually confirm right now that the issue still exists in XenServer 5.6, will try and find out and post here if I do*).

*ps, just doing cat /proc/net/bonding/bond0 will tell you.