For my setup, I chose to deploy the MariaDB Galera Cluster and RabbitMQ Cluster on dedicated VMs instead of running them within the Kubernetes cluster. A few key considerations drove this decision:
- Simplicity over complexity: The anticipated application load is relatively low, so I avoided over-engineering the architecture.
- Upfront resource overprovisioning: Allocating more resources early on helps ensure stability without complicating scaling efforts.
- Scalability, when needed: Horizontal scaling can be handled easily if required, though its demand would not be frequent or critical at this stage.
- Reliable storage: I use Ceph with RBD-backed volumes, which offer high reliability.
- Snapshot safety net: Ceph’s point-in-time RBD snapshots provide an additional layer of protection.
- Lower network overhead: Running these core services outside of Kubernetes helps reduce unnecessary network complexity within the cluster.
I opted for MariaDB version 11.8 due to its support for Vector DB integration.
Launch 3 Debian 12 VMs (8 vCPU, 32 G RAM).
Install the Mariadb server in all the nodes and stop the service.
sudo apt-get install apt-transport-https curl
sudo mkdir -p /etc/apt/keyrings
sudo curl -o /etc/apt/keyrings/mariadb-keyring.pgp 'https://mariadb.org/mariadb_release_signing_key.pgp'
sudo apt-get update
sudo apt-get install mariadb-server mariadb-backup
systemctl stop mariadb.service
Each VM has 3 x 10G NICs. The first is for VM Management, the second is for applications to access MariaDB, and the third is for the Galera cluster. In my case, the IP addresses associated with the Galera clustering of the three nodes are 10.0.16.3, 10.0.17.3, and 10.0.18.3.
Edit /etc/mysql/mariadb.conf.d/50-server.cnf in all nodes and ensure the bind address is 0.0.0.0. (All nodes)
Create /etc/mysql/conf.d/galera.cnf with the following contents (All nodes)
[mysqld]
binlog_format=ROW
default-storage-engine=innodb
#Innodb specific configurations
# Setting aside 2G of OS
innodb_buffer_pool_size=23G
#Recommended 1/4th or 1/8th of innodb_buffer_pool_size
innodb_log_file_size=4G
#General recommendation 2
innodb_log_files_in_group=2
innodb_flush_method=O_DIRECT
#Automatic=0, recommendation vCPU * 2
innodb_thread_concurrency=16
#galera disables and sets to 0, set to 1
slave_connections_needed_for_purge=2
aria_pagecache_buffer_size=2G
aria_sort_buffer_size=1G
bind-address=0.0.0.0
log-error=/var/log/mysql/mysql.err
#Galera Provider Configuration
wsrep_on=ON
wsrep_provider=/usr/lib/galera/libgalera_smm.so
#Galera Cluster Configuration
wsrep_cluster_name="dccluster"
wsrep_cluster_address="gcomm://10.0.16.3,10.0.17.3,10.0.18.3"
#Galera Synchronization Configuration
wsrep_sst_method=mariabackup
#Galera Node Configuration
wsrep_node_address="10.0.16.3"
wsrep_node_name="db1"
In one of the nodes, initiate the galera_cluster
# galera_new_cluster
In the other two nodes
systemctl start mariadb
Verify cluster status
MariaDB [(none)]> show status like "wsrep_cluster%";
+----------------------------+--------------------------------------+
| Variable_name | Value |
+----------------------------+--------------------------------------+
| wsrep_cluster_weight | 3 |
| wsrep_cluster_capabilities | |
| wsrep_cluster_conf_id | 31 |
| wsrep_cluster_size | 3 |
| wsrep_cluster_state_uuid | 52bc4c69-00dc-11f0-8e21-02fe63fa97a5 |
| wsrep_cluster_status | Primary |
+----------------------------+--------------------------------------+
6 rows in set (0.001 sec)
MariaDB [(none)]>
For the RabbitMQ installation, I followed the steps detailed at https://www.rabbitmq.com/docs/install-debian
sudo apt-get install curl gnupg apt-transport-https -y
sudo curl -1sLf "https://keys.openpgp.org/vks/v1/by-fingerprint/0A9AF2115F4687BD29803A206B73A36E6026DFCA" | sudo gpg --dearmor | sudo tee /usr/share/keyrings/com.rabbitmq.team.gpg > /dev/null
sudo curl -1sLf https://github.com/rabbitmq/signing-keys/releases/download/3.0/cloudsmith.rabbitmq-erlang.E495BB49CC4BBE5B.key | sudo gpg --dearmor | sudo tee /usr/share/keyrings/rabbitmq.E495BB49CC4BBE5B.gpg > /dev/null
sudo curl -1sLf https://github.com/rabbitmq/signing-keys/releases/download/3.0/cloudsmith.rabbitmq-server.9F4587F226208342.key | sudo gpg --dearmor | sudo tee /usr/share/keyrings/rabbitmq.9F4587F226208342.gpg > /dev/null
Create /etc/apt/sources.list.d/rabbitmq.list, so that the modern Erlang/OTP releases are used
## Provides modern Erlang/OTP releases
##
deb [arch=amd64 signed-by=/usr/share/keyrings/rabbitmq.E495BB49CC4BBE5B.gpg] https://ppa1.rabbitmq.com/rabbitmq/rabbitmq-erlang/deb/debian bookworm main
deb-src [signed-by=/usr/share/keyrings/rabbitmq.E495BB49CC4BBE5B.gpg] https://ppa1.rabbitmq.com/rabbitmq/rabbitmq-erlang/deb/debian bookworm main
# another mirror for redundancy
deb [arch=amd64 signed-by=/usr/share/keyrings/rabbitmq.E495BB49CC4BBE5B.gpg] https://ppa2.rabbitmq.com/rabbitmq/rabbitmq-erlang/deb/debian bookworm main
deb-src [signed-by=/usr/share/keyrings/rabbitmq.E495BB49CC4BBE5B.gpg] https://ppa2.rabbitmq.com/rabbitmq/rabbitmq-erlang/deb/debian bookworm main
## Provides RabbitMQ
##
deb [arch=amd64 signed-by=/usr/share/keyrings/rabbitmq.9F4587F226208342.gpg] https://ppa1.rabbitmq.com/rabbitmq/rabbitmq-server/deb/debian bookworm main
deb-src [signed-by=/usr/share/keyrings/rabbitmq.9F4587F226208342.gpg] https://ppa1.rabbitmq.com/rabbitmq/rabbitmq-server/deb/debian bookworm main
# another mirror for redundancy
deb [arch=amd64 signed-by=/usr/share/keyrings/rabbitmq.9F4587F226208342.gpg] https://ppa2.rabbitmq.com/rabbitmq/rabbitmq-server/deb/debian bookworm main
deb-src [signed-by=/usr/share/keyrings/rabbitmq.9F4587F226208342.gpg] https://ppa2.rabbitmq.com/rabbitmq/rabbitmq-server/deb/debian bookworm main
Update and install Erlang packages and rabbitmq-server.
sudo apt-get update -y
sudo apt-get install -y erlang-base erlang-asn1 erlang-crypto erlang-eldap erlang-ftp erlang-inets erlang-mnesia erlang-os-mon erlang-parsetools erlang-public-key \
erlang-runtime-tools erlang-snmp erlang-ssl erlang-syntax-tools erlang-tftp erlang-tools erlang-xmerl
sudo apt-get install rabbitmq-server -y --fix-missing
As the VM is based on Debian 12, On distributions that use systemd, the OS limits are controlled via a configuration file at /etc/systemd/system/rabbitmq-server.service.d/limits.conf.
Create /etc/systemd/system/rabbitmq-server.service.d/limits.conf with following contents
[Service]
LimitNOFILE=64000
Enable the rabbitmq_management plugin, configure an Erlang cookie, ports and restart rabbitmq_server (all nodes)
rabbitmq-plugins enable rabbitmq_management
echo "SOMEALPHANUMERICCOOKIE" | sudo tee /var/lib/rabbitmq/.erlang.cookie
echo "listeners.tcp.1 = 0.0.0.0:5672" | sudo tee -a /etc/rabbitmq/rabbitmq.conf
echo "management.tcp.port = 15672" | sudo tee -a /etc/rabbitmq/rabbitmq.conf
systemctl restart rabbitmq-server
One node 1, create an admin user (dcuser) and delete the default ‘guest’ user
rabbitmqctl add_user dcuser somepassword
rabbitmqctl set_permissions -p / dcuser ".*" ".*" ".*"
rabbitmqctl set_user_tags dcuser administrator
rabbitmqctl delete_user guest
On the other two nodes, stop the application, join the cluster and start the application.
rabbitmqctl stop_app
rabbitmqctl join_cluster rabbit@rmq1
rabbitmqctl start_app
Verify cluster status
root@rmq1:~# rabbitmqctl cluster_status
Cluster status of node rabbit@rmq1 ...
Basics
Cluster name: rabbit@rmq1.xxxxxx.net
Total CPU cores available cluster-wide: 6
Cluster Tags
(none)
Disk Nodes
rabbit@rmq1
rabbit@rmq2
rabbit@rmq3
Running Nodes
rabbit@rmq1
rabbit@rmq2
rabbit@rmq3
Versions
rabbit@rmq1: RabbitMQ 4.0.7 on Erlang 27.3.1
rabbit@rmq2: RabbitMQ 4.0.7 on Erlang 27.3.1
rabbit@rmq3: RabbitMQ 4.0.7 on Erlang 27.3.1