CloudStack - How to add Ceph 20 (Tentacle) as Primary Storage on KVM

Ceph 20 (“Tentacle”) is the newest Ceph major release, and if you try to add it as RBD primary storage to a CloudStack KVM cluster today, you will very likely hit three separate failures before you get a working setup - none of which is a CloudStack bug. This post walks through all three, and how to get a Tentacle cluster working as CloudStack primary storage today, without downgrading Ceph.

This is a follow-up to Ceph - Deploy Ceph 20 (tentacle) with cephadm-ansible - if you don’t have a Tentacle cluster yet, start there.

Symptom

Adding the RBD pool as primary storage in CloudStack fails, and the KVM agent log on the host shows:

DEBUG [kvm.storage.LibvirtStorageAdaptor] Attempting to create storage pool 5754fb0c-853e-3097-97b1-95a51e1fb080
DEBUG [kvm.storage.LibvirtStorageAdaptor] <pool type='rbd'>
<name>5754fb0c-853e-3097-97b1-95a51e1fb080</name>
<uuid>5754fb0c-853e-3097-97b1-95a51e1fb080</uuid>
<source>
<host name='10.1.33.97'/>
<name>cloudstack</name>
<auth username='cloudstack' type='ceph'>
<secret uuid='5754fb0c-853e-3097-97b1-95a51e1fb080'/>
</auth>
</source>
</pool>

ERROR [kvm.storage.LibvirtStorageAdaptor] Failed to create RBD storage pool: org.libvirt.LibvirtException: failed to connect to the RADOS monitor on: 10.1.33.97,: No such file or directory

or, after fixing the first problem, this one instead:

ERROR [kvm.storage.LibvirtStorageAdaptor] Failed to create RBD storage pool: org.libvirt.LibvirtException: internal error: failed to set RADOS option: auth_supported

And once the pool is up and you actually deploy a VM on it, a third one:

kernel: qemu-kvm[102008]: segfault at 0 ip 0000000000000000 sp 00007ffc94819798 error 14 in qemu-kvm[...] likely on CPU 2 (core 0, socket 4)

with the CloudStack management server reporting Unable to orchestrate the start of VM instance ... and no other detail.

All three are environment/library issues below CloudStack, not something fixed by CloudStack config. Here’s what’s going on and how to fix each one.

Problem 1: the KVM host’s Ceph client is too old for Tentacle

CloudStack’s KVM agent doesn’t talk RBD itself - it hands the mon host, pool name, and cephx secret to libvirt, which opens the connection using whatever librados/librbd is installed on the hypervisor. On stock Rocky Linux 9 / RHEL 9, that’s the AppStream package:

librados2-16.2.4-5.el9.x86_64    # Ceph Pacific (2021)
librbd1-16.2.4-5.el9.x86_64

Ceph Pacific talking to a Tentacle (20.2.4) monitor is a 4-major-release gap, well outside Ceph’s supported client/server compatibility window, and shows up as the generic failed to connect to the RADOS monitor ... No such file or directory - not a helpful version-mismatch error.

Fix: install a current librados2/librbd1 from download.ceph.com on every KVM host. On Rocky/RHEL 9 you also need the CRB repo enabled, for lttng-ust and libpmemobj:

dnf config-manager --set-enabled crb

rpm --import https://download.ceph.com/keys/release.asc
cat <<'EOF' > /etc/yum.repos.d/ceph.repo
[ceph]
name=Ceph packages
baseurl=https://download.ceph.com/rpm-tentacle/el9/x86_64/
gpgcheck=1
gpgkey=https://download.ceph.com/keys/release.asc
EOF

dnf install -y librados2 librbd1
systemctl restart libvirtd

(Reef or Squid client packages also work fine against a Tentacle mon - Ceph supports N-2 - but matching the mon’s release is the safest choice, and it’s what I used here.)

With just this fix applied, virsh pool-create against the RBD pool XML gets further, but now fails with a different error - which is problem 2.

Problem 2: libvirt still sets a config option Ceph removed

Ceph Tentacle (librados ≥ 19.2.6/20.2.4) fully removed the long-deprecated auth_supported config option. libvirt’s RBD storage backend (src/storage/storage_backend_rbd.c, function virStorageBackendRBDOpenRADOSConn) still sets it unconditionally when opening the pool connection, and treats a failed rados_conf_set() as fatal:

if (virStorageBackendRBDRADOSConfSet(ptr->cluster,
                                     "auth_supported", "cephx") < 0)
    goto cleanup;

I checked this against upstream libvirt master and the latest release (v12.7.0, newer than what ships on Rocky 9 today) - both still have this exact code, unfixed. It’s tracked upstream as libvirt#918, open at the time of writing. Proxmox hit the identical wall from their own RBD storage plugin (bugzilla #7961) and worked around it by restoring the option in their own downstream Ceph build - not something available if you’re running upstream Ceph packages.

There’s also an apache/cloudstack#13991 PR that swaps auth_supported for auth_client_required in CloudStack’s own RBD connection string - but that string is only used for qemu-img-driven operations (template seeding, volume/snapshot copies), in KVMPhysicalDisk.RBDStringBuilder(). It never touches LibvirtStorageAdaptor.createRBDStoragePool(), which is the code path that actually fails above - that one goes straight to libvirt’s XML API, and the failing auth_supported call happens entirely inside libvirt’s compiled C code. No string CloudStack builds can reach it. See the discussion on apache/cloudstack#13989 for the full thread - the PR is still worth having (it fixes the qemu-img side), but it doesn’t fix storage pool creation on its own.

Workaround: an LD_PRELOAD shim for libvirtd

Until libvirt#918 lands and gets packaged, the option I’d reach for on the hypervisors themselves is a tiny LD_PRELOAD shim that intercepts the one deprecated rados_conf_set() call libvirt makes and rewrites it to auth_client_required (the same replacement suggested in libvirt#918). It’s scoped to the libvirtd service via a systemd drop-in - but read Problem 3 below before you install this as-is, because a plain Environment= drop-in leaks the preload into every QEMU process libvirtd spawns too, and that will crash your VMs.

rados_auth_shim.c:

#define _GNU_SOURCE
#include <dlfcn.h>
#include <string.h>
#include <stdio.h>

typedef int (*rados_conf_set_fn)(void *cluster, const char *option, const char *value);

int rados_conf_set(void *cluster, const char *option, const char *value)
{
    static rados_conf_set_fn real_rados_conf_set = NULL;
    if (!real_rados_conf_set) {
        real_rados_conf_set = (rados_conf_set_fn)dlsym(RTLD_NEXT, "rados_conf_set");
    }

    if (option != NULL && strcmp(option, "auth_supported") == 0) {
        fprintf(stderr, "[rados_auth_shim] rewriting deprecated option "
                "'auth_supported=%s' -> 'auth_client_required=%s'\n",
                value ? value : "(null)", value ? value : "(null)");
        option = "auth_client_required";
    }

    return real_rados_conf_set(cluster, option, value);
}

Build and install it on each KVM host (needs a C compiler - gcc/glibc-devel):

gcc -Wall -fPIC -shared -o /usr/local/lib/rados_auth_shim.so rados_auth_shim.c -ldl

mkdir -p /etc/systemd/system/libvirtd.service.d
cat > /etc/systemd/system/libvirtd.service.d/rados-auth-shim.conf <<'EOF'
[Service]
Environment=LD_PRELOAD=/usr/local/lib/rados_auth_shim.so
EOF

systemctl daemon-reload
systemctl restart libvirtd

The Environment= line only applies to the libvirtd unit because of the drop-in - but Environment= is inherited by everything libvirtd forks, and libvirtd forks a qemu-kvm process for every VM it starts. That’s Problem 3, next. You can confirm the shim itself is active by watching the journal while a pool is (re)created:

journalctl -u libvirtd -f
# [rados_auth_shim] rewriting deprecated option 'auth_supported=cephx' -> 'auth_client_required=cephx'

With both fixes in place - current librados2/librbd1, and the shim loaded into libvirtd - the RBD storage pool comes up correctly:

$ virsh pool-info 5754fb0c-853e-3097-97b1-95a51e1fb080
Name:           5754fb0c-853e-3097-97b1-95a51e1fb080
UUID:           5754fb0c-853e-3097-97b1-95a51e1fb080
State:          running
Persistent:     no
Autostart:      no
Capacity:       299.99 GiB
Allocation:     0.00 B
Available:      299.91 GiB

…and CloudStack’s own “Add primary storage” retries the same pool creation successfully once the fixes are applied on all KVM hosts. But try to actually deploy a VM onto it and every one will fail - which is Problem 3.

Problem 3: the shim crashes every QEMU process it leaks into

The pool exists, virsh pool-info is happy, and CloudStack’s qemu-img calls into it work fine (that’s the separate, CloudStack-side fix in apache/cloudstack#13991 - see the caveat below on why you still need that PR even with this fix in place). But deploying any VM on the pool fails, and the KVM agent log shows the domain never actually starts:

ERROR [cloud.agent.Agent] internal error: process exited while connecting to monitor

with a matching kernel log line:

kernel: qemu-kvm[102008]: segfault at 0 ip 0000000000000000 sp 00007ffc94819798 error 14 in qemu-kvm[...]

This reproduces with a bare virsh create on any domain with an RBD disk, no CloudStack involved, and it reproduces identically no matter which librados/librbd version is installed (16.2.4, 18.2.8, 20.2.4 - tried all three). The variable that actually matters is whether LD_PRELOAD=/usr/local/lib/rados_auth_shim.so is set in the process’s environment: unset it and the exact same QEMU command line either runs cleanly or fails gracefully; leave it set and QEMU segfaults every time. The shim itself - or more precisely, a QEMU process loading it via preload rather than libvirtd calling into it directly - is the actual crash.

The systemd drop-in from Problem 2 sets LD_PRELOAD as Environment= on the whole libvirtd.service, and systemd’s Environment= is inherited by every child process the unit’s main process forks - including the qemu-kvm binary libvirtd execs for each VM. libvirtd’s own in-process RBD storage-pool code (Problem 2’s fix target) genuinely needs the shim, since that call happens inside libvirtd itself with no exec involved. The spawned VMs don’t need it and must not get it.

Fix: strip LD_PRELOAD before QEMU execs, not before libvirtd starts

Rather than trying to scope Environment= more tightly (systemd has no per-child-process override for this), wrap the qemu-kvm binary itself so it drops LD_PRELOAD before running the real emulator. libvirtd’s own process environment - and so its own librados calls - is untouched; only the exec’d child loses the preload.

mv /usr/libexec/qemu-kvm /usr/libexec/qemu-kvm.real
cat > /usr/libexec/qemu-kvm <<'EOF'
#!/bin/bash
# Strip LD_PRELOAD (rados_auth_shim.so, needed only by libvirtd itself for
# RBD storage-pool management) before exec-ing the real qemu-kvm binary --
# the shim crashes QEMU when it inherits it as a spawned child process.
unset LD_PRELOAD
exec /usr/libexec/qemu-kvm.real "$@"
EOF
chmod 755 /usr/libexec/qemu-kvm
chown --reference=/usr/libexec/qemu-kvm.real /usr/libexec/qemu-kvm

systemctl restart libvirtd

Verify with the same bare virsh create reproduction from above - a domain with a cephx-authenticated RBD <disk> should now start and stay running. With all three fixes in place, actual VM deploys onto the Tentacle-backed pool work end-to-end.

Adding the storage in CloudStack

Once all three host-side fixes are applied, adding the pool is completely ordinary - no CloudStack-side workaround needed:

cmk create storagepool \
  name=ceph-tentacle-primary \
  zoneid=<zone-id> \
  podid=<pod-id> \
  clusterid=<cluster-id> \
  url="rbd://cloudstack:<cephx-key-base64>@10.1.33.97/cloudstack" \
  scope=cluster \
  provider=DefaultPrimary

or via the UI: Infrastructure > Primary Storage > Add Primary Storage, protocol RBD, server = a mon IP, path = the pool name, and the cloudstack cephx user’s key.

Caveats

  • Apply apache/cloudstack#13991 too - the QEMU wrapper doesn’t replace it. qemu-img is a separate binary (/usr/bin/qemu-img, from the qemu-img package) invoked directly by the CloudStack KVM agent, not spawned by libvirtd. It never had the shim’s LD_PRELOAD applied to it in the first place (that env var only ever lived on libvirtd.service), so stripping it from the qemu-kvm wrapper changes nothing for qemu-img. Without #13991, qemu-img convert against the pool still fails with the raw, unpatched invalid conf option auth_supported - template seeding and volume/snapshot copies never get to try the shim at all.
  • The LD_PRELOAD shim (and the QEMU wrapper needed to contain it) is a stopgap, not a real fix. Both should come out once libvirt actually ships a fix for #918 and your distro packages it: rm /etc/systemd/system/libvirtd.service.d/rados-auth-shim.conf /usr/local/lib/rados_auth_shim.so && systemctl daemon-reload && mv /usr/libexec/qemu-kvm.real /usr/libexec/qemu-kvm && systemctl restart libvirtd.
  • All three fixes (client library upgrade, shim, QEMU wrapper) need to be applied on every KVM host in the cluster, not just one.
  • If you rebuild/reimage a KVM host later, remember to reapply all three - none of them are part of the standard CloudStack KVM agent install.
  • If you don’t specifically need Ceph 20 for testing, pinning to Squid (19.2.x) or Reef (18.2.x) avoids all of this entirely - both are well-supported today and don’t have the auth_supported problem.

References

 Date: September 7, 2026
 Tags:  CloudStack KVM Ceph
 Related Posts:
Ceph - Deploy Ceph 20 (tentacle) with cephadm-ansible
CloudStack HowTo - Create Debian 13 Cloud images for VMware and XenServer testing
Wiring CloudStack KVM hosts up to OVN

Previous:
⏪ Ceph - Deploy Ceph 20 (tentacle) with cephadm-ansible