Ceph + KVM: 4. Orchestrating Ceph RBD backed VMs on KVM Hosts

Posted on by sandeep

Previous: Installing KVM, customised monitoring scripts                                                                                     Next:   Managing KVM/CEPH – CLI commands

Need: Simple shell scripts to orchestrate/manage VMs on KVM hosts

./launch.sh -h <kvmhost> -v <vm-name> -c <no-of-vcpus> -r <ram-in-gb> -p <rbd-pool-name> -s <rbd-block-size>  -i <last-octet-of-ip> -d <distro>

  • -h: KVM Hostname
  • -v: Name of the VM
  • -c: Number of vCPUs to be allocated
  • -r: RAM in GB to be allocated
  • -p: Name of the Ceph Pool from which block storage is to be allocated to be used as a disk for the VM
    • nvmepool
    • ssdpool
  • -s: Size of the disk/block storage (in GB)
  • -i: Last octet of IP.  Each VM will have 3 NICS 10.0.1.x, 10.0.2.x, 10.0.4.x,  ‘x’ gets replaced with the value provided
    • 10.0.1.x – Management Traffic
    • 10.0.2. x – DB Access
    • 10.0.4.x – Ceph Access, this will be required in case of Worker node VMs.
  • -d: distro
    • ‘debian’ in case of Debian 12-based VM
    • ‘ubuntu’ in case of Ubuntu 24.04-based VM
      • Custom images are available
./deletevm.sh -h <kvmhost> -p <rbd-pool-name> -v <vm-name>
  • -h: KVM Hostname where the VM is deployed
    • Delete the domain definition of the VM prepared and copied onto the host
  • -p: Name of the Ceph Pool from which block storage was allocated
    • This can be auto-detected
    • When undefining the VM on the host, the image in the RBD pool should also be deleted.
  • -v: Name of the VM

Deployed a Debian 12-based Orchestrator VM and installed the required packages

apt install virt-manager libvirt-clients qemu-utils libguestfs-tools openssh-client ceph-common

Enable passwordless SSH into the KVM hosts from the Orchestrator VM.

ssh-copy-id root@<kvmhost>

Standard VM templates in /root/kvm/vmtemplates

/root/kvm/vmtemplates/hosts

127.0.0.1 localhost
{{ip1}} {{hostname}}.{{domain}} {{hostname}}
  • ip1: IP (10.0.1.x) – Value for x provided by user
  • hostname:  VM name – provided by user
  • domain: Hard-coded 

/root/kvm/vmtemplates/hosts

[Resolve]
# Some examples of DNS servers which may be used for DNS= and FallbackDNS=:
# Cloudflare: 1.1.1.1#cloudflare-dns.com 1.0.0.1#cloudflare-dns.com 2606:4700:4700::1111#cloudflare-dns.com 2606:4700:4700::1001#cloudflare-dns.com
# Google: 8.8.8.8#dns.google 8.8.4.4#dns.google 2001:4860:4860::8888#dns.google 2001:4860:4860::8844#dns.google
# Quad9: 9.9.9.9#dns.quad9.net 149.112.112.112#dns.quad9.net 2620:fe::fe#dns.quad9.net 2620:fe::9#dns.quad9.net
DNS={{dns}}
FallbackDNS=8.8.8.8
Domains=domain.net
#DNSSEC=no
#DNSOverTLS=no
#MulticastDNS=yes
#LLMNR=yes
#Cache=yes
#CacheFromLocalhost=no
DNSStubListener=no
#DNSStubListenerExtra=
#ReadEtcHosts=yes
#ResolveUnicastSingleLabel=no
  • dns: Value that gets replaced with the local DNS server 10.0.0.1
  • domains:  Hard-coded, domain.net is just representative

/root/kvm/vmtemplates/node.xml – Domain definition: define a VM with placeholders for variables.

<domain type='kvm'>
  <name>{{vmname}}</name>
  <vcpu placement='static'>{{vcpus}}</vcpu>
  <memory unit='GiB'>{{ram}}</memory>
  <os>
    <type arch='x86_64' machine='pc-q35-7.2'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough' check='none'/>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='network' device='disk'>
      <driver name='qemu' type='qcow2' cache='none' io='native'/>
      <auth username='libvirt'>
        <secret type='ceph' uuid='xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'/> 
      </auth>
      <source protocol='rbd' name='{{pool}}/{{vmname}}'>
        <host name='ceph1.domain.net' port='6789'/>
        <host name='ceph2.domain.net' port='6789'/>
        <host name='ceph3.domain.net' port='6789'/>
      </source>
      <target dev='vda' bus='virtio'/>
    </disk>
    <interface type='network'>
      <source network='br1'  />
      <model type='virtio'/>
    </interface>
    <interface type='network'>
      <source network='br2'  />
      <model type='virtio'/>
    </interface>
    <interface type='network'>
      <source network='br4' />
      <model type='virtio'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='virtio'/>
    </video>
    <memballoon model='virtio'/>
  </devices>
</domain>

/root/kvm/vmtemplates/debian/interfaces – Debian specific interface configuration template (Note 10G network – so mtu is 9000)

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
allow-hotplug eth0
iface eth0 inet static
    mtu 9000
    address {{ip1}}/16
    gateway {{gateway}}

# For DB Access
allow-hotplug eth1
iface eth1 inet static
    mtu 9000
    address {{ip2}}/24

# For Ceph Access
allow-hotplug eth2
iface eth2 inet static
    mtu 9000
    address {{ip3}}/24

/root/kvm/vmtemplates/ubuntu/50-cloud-init.yaml – Ubuntu specific network configuration template

network:
    ethernets:
        eth0:
            dhcp4: false
            dhcp6: false
            addresses:
              - {{ip1}}/16
            routes:
              - to: default
                via: {{gateway}}
        eth1:
             dhcp4: false
             dhcp6: false
             addresses:
               - {{ip2}}/24
        eth2:
             dhcp4: false
             dhcp6: false
             addresses:
               - {{ip3}}/24
    version: 2

/root/kvm/deletevm.sh 

#!/bin/bash
host=""
pool=""
vmname=""
while getopts ":h:p:v:" opt; do
  case $opt in
    h) host="$OPTARG" ;;
    p) pool="$OPTARG" ;;
    v) vmname="$OPTARG" ;;
    \?) echo "Invalid option: -$OPTARG" >&2; exit 1 ;;
    :) echo "Option -$OPTARG requires an argument." >&2; exit 1 ;;
  esac
done
if [ -z "$host" ]; then
    echo "KVM host information not provided."
    exit 1
fi
if [ -z "$pool" ]; then
    echo "Pool name not provided."
    exit 1
fi
if [ -z "$vmname" ]; then
    echo "VM name not provided."
    exit 1
fi
remote_uri="qemu+ssh://root@$host/system"
line=$(virsh -c "$remote_uri" list --all | grep "$vmname" | tr -s ' ' | cut -d " " -f4)
if [ -n "$line" ]; then
    if [ "$line" == "running" ] ; then
        echo "Stopping $vmname"
        virsh -c "$remote_uri" shutdown $vmname
    fi
    echo "Undefining..."
    virsh -c "$remote_uri" undefine $vmname
else
    echo "No such vm defined...."
fi
sleep 4
echo "Checking if image present in pool...."
count=`rbd ls --pool $pool | grep $vmname | wc -l`
if [ $count -ne 0 ]; then
    rbd rm $pool/$vmname
fi
echo "Cleaning up ...."
if [ -d /root/kvm/vmtemplates/$vmname ]; then
    rm -rf /root/kvm/vmtemplates/$vmname
    ssh root@$host rm -rf /root/kvm/vmtemplates/$vmname
fi

/root/kvm/preparevm.sh

#!/bin/bash
#$host $vcpu $ram $storage $ip $distro
host=$1
vcpu=$2
ram=$3
storage=$4
ip1="10.0.1.$5"
ip2="10.0.2.$5"
ip3="10.0.4.$5"
distro="debian"
image="debian12"
if [ -n "$6" ]; then
    distro=$6
fi
if [ "$distro" == "ubuntu" ]; then
    image="ubuntu24"
fi
domain=domain.net
dns=10.0.0.1
gateway=10.0.0.1
variant=debian11
base=/root/kvm/vmtemplates
source=$base/$distro
target=$base/$host
mkdir -p $target
rm -f $target/*
echo "Copying base image to instance folder..."
cp $source/$image.qcow2 $target/$host.qcow2
cp $base/resolved.conf $target/resolved.conf
if [ "$distro" == "debian" ]; then
    intfile=$target/interfaces
    cp $source/interfaces $intfile
else
    intfile=$target/50-cloud-init.yaml
    cp $source/50-cloud-init.yaml $intfile
fi
cp $base/hosts $target/hosts
echo "Resizing $target/$1.qcow2 to $storage..."
qemu-img resize $target/$1.qcow2 $storage
echo "Customizing vm...."
resolvefile=$target/resolved.conf
hostfile=$target/hosts
sed -i "s/{{dns}}/$dns/g" $resolvefile
sed -i "s/{{ip1}}/$ip1/g" $intfile
sed -i "s/{{ip2}}/$ip2/g" $intfile
sed -i "s/{{ip3}}/$ip3/g" $intfile
sed -i "s/{{gateway}}/$gateway/g" $intfile
sed -i "s/{{hostname}}/$host/g" $hostfile
sed -i "s/{{domain}}/$domain/g" $hostfile
sed -i "s/{{ip1}}/$ip1/g" $hostfile
if [ "$distro" == "debian" ]; then
    virt-customize -a $target/$host.qcow2 --copy-in $intfile:/etc/network --copy-in $resolvefile:/etc/systemd/ --copy-in $hostfile:/etc/ --hostname $host.$domain --timezone Asia/Kolkata --firstboot-command '/usr/local/bin/resizedisk'
else
    virt-customize -a $target/$host.qcow2 --copy-in $intfile:/etc/netplan --copy-in $resolvefile:/etc/systemd/ --copy-in $hostfile:/etc/ --hostname $host.$domain --timezone Asia/Kolkata --firstboot-command '/usr/local/bin/resizedisk'
fi

/root/kvm/launch.sh

#!/bin/bash
distro="debian"
storage="100G"
vcpu="4"
ram="8"
host="server1"
pool="ssdpool"
ip=""
vmname=""
while getopts ":h:p:c:v:r:s:i:d:" opt; do
    case $opt in
        h) host="$OPTARG" ;;
        p) pool="$OPTARG" ;;
        v) vmname="$OPTARG" ;;
        c) vcpu="$OPTARG" ;;
        r) ram="$OPTARG" ;;
        s) storage="$OPTARG" ;;
        i) ip="$OPTARG" ;;
        d) distro="$OPTARG" ;;
        \?) echo "Invalid option: -$OPTARG" >&2; exit 1 ;;
        :) echo "Option -$OPTARG requires an argument." >&2; exit 1 ;;
    esac
done
if [ -z "$ip" ]; then
    echo "IP octect not provided."
    exit 1
fi
if [ -z "$vmname" ]; then
    echo "VM name not provided."
    exit 1
fi
remote_uri="qemu+ssh://root@$host/system"
echo "Server : $host, Pool $pool, vmname $vmname, vCPU $vcpu, RAM $ram, Disk $storage, IP 10.0.1.$ip, Distro : $distro"
echo "Deleting $host:/$pool/$vmname (if existing)"
./deletevm.sh -h $host -p $pool -v $vmname
base=/root/kvm/vmtemplates
./preparevm.sh $vmname $vcpu $ram $storage $ip $distro
echo "Importing customized image..."
rbd import $base/$vmname/$vmname.qcow2 $pool/$vmname
cp $base/node.xml $base/$vmname/$vmname.xml
sed -i "s/{{vmname}}/$vmname/g" $base/$vmname/$vmname.xml
sed -i "s/{{pool}}/$pool/g" $base/$vmname/$vmname.xml
sed -i "s/{{ram}}/$ram/g" $base/$vmname/$vmname.xml
sed -i "s/{{vcpus}}/$vcpu/g" $base/$vmname/$vmname.xml
echo "Launching vm...."
ssh root@$host "mkdir -p '$base/$vmname'"
scp "$base/$vmname/$vmname.xml" root@$host:"$base/$vmname/$vmname.xml"
virsh -c "$remote_uri" define $base/$vmname/$vmname.xml
virsh -c "$remote_uri" start $vmname
virsh -c "$remote_uri" autostart $vmname
echo "Deleting generated custom image..."
rm -f $base/$vmname/$vmname.qcow2