Skip to article frontmatterSkip to article content

HAProxy -aaa- Konfiguration

Active-Active-Active HAProxy Konfiguration mit Openstack

Konzept




Vorgehen & Vorraussetzungen

Software

Openstack Runbook

Octavia Setup

TCP Loadbalancer vor den HAProxys

# Variablen
LB_NAME=loadbalancer-1
VIP_SUBNET_ID="3188133b-d42a-4216-af89-2fe87616e827"
BACKEND_SUBNET_ID="277ac2e6-79ed-4725-98f8-b3e4ad7e6f8d"

LISTENER_80=fe_80
LISTENER_443=fe_443
POOL_80=pool_fe_80
POOL_443=pool_fe_443
HM_80=hm_fe_80
HM_443=hm_fe_443

# 1) Loadbalancer (VIP) anlegen
openstack loadbalancer create --name "$LB_NAME" --vip-subnet-id "$VIP_SUBNET_ID"

# 2) Listener (beide TCP-Frontends)
openstack loadbalancer listener create \
  --name "$LISTENER_80" --protocol TCP --protocol-port 80 "$LB_NAME"

openstack loadbalancer listener create \
  --name "$LISTENER_443" --protocol TCP --protocol-port 443 "$LB_NAME"

# 3) Pools (je Listener einen, TCP/SOURCE_IP)
openstack loadbalancer pool create \
  --name "$POOL_80" --protocol TCP --lb-algorithm SOURCE_IP --listener "$LISTENER_80"

openstack loadbalancer pool create \
  --name "$POOL_443" --protocol TCP --lb-algorithm SOURCE_IP --listener "$LISTENER_443"

# 4) Member in die Pools aufnehmen
# Pool 80
openstack loadbalancer member create "$POOL_80" \
  --subnet-id "$BACKEND_SUBNET_ID" --address 192.168.100.251 --protocol-port 80 --name m80_251
openstack loadbalancer member create "$POOL_80" \
  --subnet-id "$BACKEND_SUBNET_ID" --address 192.168.100.252 --protocol-port 80 --name m80_252
openstack loadbalancer member create "$POOL_80" \
  --subnet-id "$BACKEND_SUBNET_ID" --address 192.168.100.253 --protocol-port 80 --name m80_253

# Pool 443
openstack loadbalancer member create "$POOL_443" \
  --subnet-id "$BACKEND_SUBNET_ID" --address 192.168.100.251 --protocol-port 443 --name m443_251
openstack loadbalancer member create "$POOL_443" \
  --subnet-id "$BACKEND_SUBNET_ID" --address 192.168.100.252 --protocol-port 443 --name m443_252
openstack loadbalancer member create "$POOL_443" \
  --subnet-id "$BACKEND_SUBNET_ID" --address 192.168.100.253 --protocol-port 443 --name m443_253

# 5) Health Checks (nur Ping/ICMP auf die Member-IPs)
openstack loadbalancer healthmonitor create "$POOL_80" \
  --name "$HM_80" --type PING --delay 5 --timeout 3 --max-retries 3

openstack loadbalancer healthmonitor create "$POOL_443" \
  --name "$HM_443" --type PING --delay 5 --timeout 3 --max-retries 3

# 6) Statusübersicht

openstack loadbalancer list
+--------------------------------------+----------------+----------------------------------+-----------------+---------------------+------------------+----------+
| id                                   | name           | project_id                       | vip_address     | provisioning_status | operating_status | provider |
+--------------------------------------+----------------+----------------------------------+-----------------+---------------------+------------------+----------+
| 2e07893c-e0c3-4d15-9aee-fac927f05827 | loadbalancer_1 | 9514d9a60e4847498b7c51a050808c74 | 192.168.100.149 | ACTIVE              | ERROR            | amphora  |
+--------------------------------------+----------------+----------------------------------+-----------------+---------------------+------------------+----------+

openstack loadbalancer show 2e07893c-e0c3-4d15-9aee-fac927f05827
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| admin_state_up      | True                                 |
| availability_zone   | None                                 |
| created_at          | 2025-09-13T14:35:01                  |
| description         |                                      |
| flavor_id           | 406a7541-620d-4bdf-b47f-90efc8f52a69 |
| id                  | 2e07893c-e0c3-4d15-9aee-fac927f05827 |
| listeners           | 44de30bd-ea8e-45a1-a148-1a770b584c42 |
|                     | ca9da044-8a99-4616-867f-99124453dabd |
| name                | loadbalancer_1                       |
| operating_status    | ERROR                                |
| pools               | 0b43985a-9ec1-4b80-8c12-4a341124aa1a |
|                     | ca30ec39-5555-4075-97c8-19e0deafc911 |
| project_id          | 9514d9a60e4847498b7c51a050808c74     |
| provider            | amphora                              |
| provisioning_status | ACTIVE                               |
| updated_at          | 2025-09-13T15:45:18                  |
| vip_address         | 192.168.100.149                      |
| vip_network_id      | 277ac2e6-79ed-4725-98f8-b3e4ad7e6f8d |
| vip_port_id         | aef48293-3955-4179-8ce8-f6632e68f1ae |
| vip_qos_policy_id   | None                                 |
| vip_subnet_id       | f6a5db8f-9fb1-491c-8cc1-68499d030da4 |
| vip_vnic_type       | normal                               |
| vip_sg_ids          |                                      |
| tags                | None                                 |
| additional_vips     |                                      |
+---------------------+--------------------------------------+

openstack floating ip list
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+
| ID                                   | Floating IP Address | Fixed IP Address | Port                                 | Floating Network                     | Project                          |
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+
| 27c02b1c-c33c-4141-a1be-d54e8bf8c928 | 37.228.169.xx       | 192.168.36.223   | 0d14cd20-2322-4fec-b9d1-76f62ffff0b6 | 3188133b-d42a-4216-af89-2fe87616e827 | 9514d9a60e4847498b7c51a050808c74 |
| 87dc29d8-01c1-4a14-80b0-82563f0ac0f3 | 37.228.169.xx       | 192.168.100.149  | aef48293-3955-4179-8ce8-f6632e68f1ae | 3188133b-d42a-4216-af89-2fe87616e827 | 9514d9a60e4847498b7c51a050808c74 |
| d14aa13f-9efd-4671-8ed8-0500d47903f9 | 37.228.168.xx       | None             | None                                 | 3188133b-d42a-4216-af89-2fe87616e827 | 9514d9a60e4847498b7c51a050808c74 |
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+

# 7) adding floating ip
openstack floating ip set --port aef48293-3955-4179-8ce8-f6632e68f1ae 87dc29d8-01c1-4a14-80b0-82563f0ac0f3

VIPs & Allowed Address Pairs

VRRP-IPs Ports erstellen

NETWORK_ID="277ac2e6-79ed-4725-98f8-b3e4ad7e6f8d"  
IMAGE_ID="eab7e278-58af-4922-a6f5-85c272de9738"  
SSH_KEY="ssh-key-opencloud"  
FLAVOR_ID="SCS-2V-4"  
SEC_ID="44c51350-c5cc-4054-9112-d3fc8bebd25c"
ALLOWD_ADDR1="192.168.100.251"
ALLOWD_ADDR2="192.168.100.252"
ALLOWD_ADDR3="192.168.100.253"
SRV1="hapx-1"
SRV2="hapx-2"
SRV3="hapx-3"
openstack port create hapx-1-vrrp --network $NETWORK_ID --fixed-ip ip-address=$ALLOWD_ADDR1
openstack port create hapx-2-vrrp --network $NETWORK_ID --fixed-ip ip-address=$ALLOWD_ADDR2
openstack port create hapx-3-vrrp --network $NETWORK_ID --fixed-ip ip-address=$ALLOWD_ADDR3

VMs erstellen

Einzeiler mit Subshell

openstack server create $SRV1 \
  --availability-zone A \
  --flavor $FLAVOR_ID \
  --nic port-id=$(openstack port create \
    --network $NETWORK_ID \
    --description $SRV1 \
    --dns-name $SRV1 \
    --allowed-address ip-address=$ALLOWD_ADDR1 \
    --allowed-address ip-address=$ALLOWD_ADDR2 \
    --allowed-address ip-address=$ALLOWD_ADDR3 \
    -f value -c id \
    $SRV1 ) \
  --image $IMAGE_ID \
  --boot-from-volume 20 \
  --key-name $SSH_KEY \
  --security-group $SEC_ID 

openstack server create $SRV2 \
  --availability-zone B \
  --flavor $FLAVOR_ID \
  --nic port-id=$(openstack port create \
    --network $NETWORK_ID \
    --description $SRV2 \
    --dns-name $SRV2 \
    --allowed-address ip-address=$ALLOWD_ADDR1 \
    --allowed-address ip-address=$ALLOWD_ADDR2 \
    --allowed-address ip-address=$ALLOWD_ADDR3 \
    -f value -c id \
    $SRV2 ) \
  --image $IMAGE_ID \
  --boot-from-volume 20 \
  --key-name $SSH_KEY \
  --security-group $SEC_ID 

openstack server create $SRV3 \
  --availability-zone C \
  --flavor $FLAVOR_ID \
  --nic port-id=$(openstack port create \
    --network $NETWORK_ID \
    --description $SRV3 \
    --dns-name $SRV3 \
    --allowed-address ip-address=$ALLOWD_ADDR1 \
    --allowed-address ip-address=$ALLOWD_ADDR2 \
    --allowed-address ip-address=$ALLOWD_ADDR3 \
    -f value -c id \
    $SRV3 ) \
  --image $IMAGE_ID \
  --boot-from-volume 20 \
  --key-name $SSH_KEY \
  --security-group $SEC_ID 

Ceph Volume erstellen

openstack volume create --size 10 hapx-1_data
openstack server add volume hapx-1 hapx-1_data

openstack volume create --size 10 hapx-2_data
openstack server add volume hapx-2 hapx-2_data

openstack volume create --size 10 hapx-3_data
openstack server add volume hapx-3 hapx-3_data

Keepalived Konfiguraiton

Microceph

Als gemeinsamer cert-store wird /mnt/ceph/certs verwendet, dass über ein microceph multi-node realisiert wird. Bei Openstack sollte ein weiteres Volume für Ceph hinzugefügt werden.

HAProxy Konfiguration

1) HAProxy Installieren

sudo add-apt-repository ppa:vbernat/haproxy-3.2  
sudo apt update && sudo apt install haproxy keepalived -y
sudo systemctl stop haproxy.service

Server:

VRRP VIPs:

werden in einem weitern Schritt in Openstack angelegt

Damit HAProxy starten kann brauchen wir ein placeholder Zertifkat. Wenn HAProxy damit gestartet wurde, können die Lets-Encrypt Zertifkate angefordert werden

2) Placeholder Zertifkat erstellen

sudo chmod 750 /mnt/ceph/certs
openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes \
  -subj "/CN=placeholder.local" -days 1 \
  -keyout /mnt/ceph/certs/placeholder.key \
  -out    /mnt/ceph/certs/placeholder.pem
cat /mnt/ceph/certs/placeholder.key >> /mnt/ceph/certs/placeholder.pem  

bis unsere Zertifkate ausgestellt sind, müssen auch die Zertifkate für unsere Webserver exisiteren

cp /mnt/ceph/certs/placeholder.pem /mnt/ceph/certs/web1.36px.de.pem
cp /mnt/ceph/certs/placeholder.pem /mnt/ceph/certs/web2.36px.de.pem
cp /mnt/ceph/certs/placeholder.pem /mnt/ceph/certs/web3.36px.de.pem
sudo chgrp haproxy /mnt/ceph/certs/*.pem

3) HAProxy Files

Dateien auf den Hosts verteilen. HAProxy auf den followern nicht starten.

4) Zertifkate ausstellen

erst starten wenn der LB erstellt wurde

ACME Status anzeigen

echo "acme status" | nc -U /var/run/haproxy.sock
echo "show ssl cert" | nc -U /var/run/haproxy.sock

Ausstellung eines Zertifkates triggern

echo "acme renew /mnt/ceph/certs/web1.36px.de.pem" | nc -U /var/run/haproxy.sock
echo "acme renew /mnt/ceph/certs/web2.36px.de.pem" | nc -U /var/run/haproxy.sock
echo "acme renew /mnt/ceph/certs/web3.36px.de.pem" | nc -U /var/run/haproxy.sock

DNS Setup

Security Gruppen anlegen

TBD

empfohlene System-Einstellungen

# HAProxy darf an VIP binden, auch wenn sie gerade auf einem anderen Node aktiv ist
sudo sysctl -w net.ipv4.ip_nonlocal_bind=1

# rp_filter aus, damit Antworten über die VIP nicht gedroppt werden
sudo sysctl -w net.ipv4.conf.all.rp_filter=0
sudo sysctl -w net.ipv4.conf.default.rp_filter=0