⚡ 2 Data Centers High Availability Lab

PostgreSQL Active/Passive Replication with Load Balancer & DNS

Simulate. Automate. Failover.

• Features
• Infrastructure as Code (IaC)
• Safe Playground for Real-World Scenarios
• Architecture
• Network Topology
• Requirements
• Quick Start
• Installation
• Testing the Application
• Advanced Tests (Optional)
• Access from Host Machine
• Project Structure
• Common Issues and Fixes
• Frequently Asked Questions
• Contributing
• License
• Author

• 🏢 2 Data Centers – Simulated with Vagrant (DC1 and DC2)
• 🗄️ PostgreSQL 16 – Active/Passive with Streaming Replication
• 🌐 Flask Application – Deployed on both DCs
• ⚖️ HAProxy – Load balancer with round-robin distribution
• 📡 Bind9 – DNS server for dynamic failover
• 🤖 Fully Automated – Single command deployment (vagrant up)
• 🔄 Replication – Real-time data synchronization
• 🧪 Optional Tests – Switchover and Disaster Recovery scenarios

This project is built on Infrastructure as Code principles:

• Vagrant defines the entire infrastructure (VMs, networks, resources) in a single Vagrantfile.
• Provisioning scripts automate the installation and configuration of all software (PostgreSQL, Flask, HAProxy, Bind9).

With a single command (vagrant up), you can recreate the entire 2-data-center environment from scratch. This is the power of IaC: consistent, repeatable, and automated infrastructure.

💡 Infrastructure as Code is a core practice in DevOps and SRE (Site Reliability Engineering). It enables teams to manage infrastructure with the same rigor as application code.

One of the greatest benefits of this lab is that it provides a safe, isolated environment to test and learn from real-world infrastructure scenarios without any risk to production systems.

⚡ Key advantage: All of these tests can be performed safely and repeatedly in your local environment. Break things, learn from the failures, and rebuild everything with vagrant destroy -f && vagrant up — it's that simple.

This approach helps build the confidence and skills needed to manage real production environments, where mistakes are costly and downtime is unacceptable.

Architecture diagram of the 2 Data Centers High Availability Lab

Component Description 🗄️ PostgreSQL Primary (db1) Handles all write operations 🗄️ PostgreSQL Replica (db2) Real-time replication from db1 🌐 Flask App (app1 & app2) API endpoints connected to local DB ⚖️ HAProxy (lb) Distributes traffic between apps 📡 Bind9 (dns) Resolves db.lab.local and lb.lab.local

✅ Tested on: Windows 11 Pro with VirtualBox 7.2.12 and Vagrant 2.4.9. Compatible with macOS and Linux (may require minor adjustments).

💡 For a detailed step-by-step guide, see Installation below.

# 1. Clone the repository
git clone https://github.com/rafajimenezdev/2dc-ha-lab.git
cd 2dc-ha-lab

# 2. Start the environment (this will create 6 VMs)
vagrant up

# 3. Test the application
curl http://10.10.1.100/items

# 4. Insert data
curl -X POST http://10.10.1.100/items -H "Content-Type: application/json" -d '{"id": 42}'

# 5. Verify replication
curl http://10.10.1.100/items

git clone https://github.com/rafajimenezdev/2dc-ha-lab.git
cd 2dc-ha-lab

# 1. Install VirtualBox: https://www.virtualbox.org/
# 2. Install Vagrant: https://www.vagrantup.com/
# 3. Ensure Hyper-V is disabled (if using VirtualBox)

# Install VirtualBox and Vagrant
sudo apt update
sudo apt install virtualbox vagrant -y

brew install --cask virtualbox
brew install vagrant

vagrant --version
virtualbox --help

curl http://10.10.1.100/

Expected output:

Flask App is running! (DC1)

or

Flask App is running! (DC2)

curl -X POST http://10.10.1.100/items \
  -H "Content-Type: application/json" \
  -d '{"id": 42}'

Expected output:

{"status":"inserted","id":42}

curl http://10.10.1.100/items

Expected output:

[42]

¡ME HA ENCANTADO TU ENTORNO, RAFAEL! 🔥

¡Me alegra un montón que haya funcionado todo a la primera! 🚀

Y sí, claro que podemos integrar esta batería de pruebas en el README. Es exactamente lo que un usuario necesita para validar que su laboratorio está funcionando correctamente. Y que lo hayas disfrutado es la mejor señal de que el proyecto está bien hecho.

Te voy a dar la sección exacta para que la pegues en tu README, dentro de la sección de "Testing the Application". He mantenido el estilo profesional y los emojis que te gustan.

After a successful vagrant up, run these tests to ensure the entire lab is functioning correctly.

vagrant status

All machines should show running (virtualbox).

ping 10.10.1.10   # app1
ping 10.10.1.20   # db1
ping 10.10.2.10   # app2
ping 10.10.2.20   # db2
ping 10.10.1.100  # lb
ping 10.10.1.200  # dns

All should respond successfully.

Invoke-RestMethod -Uri http://10.10.1.10:5000/ -Method GET

Expected output:

Flask App is running! (DC1)

Invoke-RestMethod -Uri http://10.10.2.10:5000/ -Method GET

Expected output:

Flask App is running! (DC2)

Invoke-RestMethod -Uri http://10.10.1.100/ -Method GET

Run this multiple times to see the round-robin distribution:

for ($i=0; $i -lt 5; $i++) { Invoke-RestMethod -Uri http://10.10.1.100/ -Method GET }

Expected output: A mix of DC1 and DC2 responses.

Invoke-RestMethod -Uri http://10.10.1.100/items -Method GET

Expected output: [] (empty array, no data yet).

Invoke-RestMethod -Uri http://10.10.1.100/items -Method POST -Headers @{"Content-Type"="application/json"} -Body '{"id": 42}'

Expected output:

{"status":"inserted","id":42}

Invoke-RestMethod -Uri http://10.10.1.100/items -Method GET

Expected output: [42]

Invoke-RestMethod -Uri http://10.10.2.10:5000/items -Method GET

Expected output: [42] (data is replicated to db2).

This test promotes the replica (db2) to primary and reconfigures db1 as the new replica.

vagrant ssh db1
sudo -u postgres psql -c "SELECT pg_switch_wal();"
sudo systemctl stop postgresql
exit

vagrant ssh db2
sudo pg_ctl promote -D /var/lib/postgresql/16/main
exit

# Reconfigure db1 as replica of db2
vagrant ssh db1
sudo rm -rf /var/lib/postgresql/16/main/*
sudo -u postgres PGPASSWORD='replicator_password' pg_basebackup \
  -h 10.10.2.20 -U replicator \
  -D /var/lib/postgresql/16/main -Fp -Xs -R -v
sudo -u postgres touch /var/lib/postgresql/16/main/standby.signal
sudo systemctl start postgresql

This test simulates the complete loss of DC1 and verifies that the system continues to work using DC2.

# Shut down DC1
vagrant halt db1 app1

# Promote db2 to primary
vagrant ssh db2
sudo pg_ctl promote -D /var/lib/postgresql/16/main
exit

# Verify application still works
curl http://10.10.2.10:5000/items

# To recover DC1, reincorporate it as a replica of db2
vagrant up db1
vagrant ssh db1
sudo systemctl stop postgresql
sudo rm -rf /var/lib/postgresql/16/main/*
sudo -u postgres PGPASSWORD='replicator_password' pg_basebackup \
  -h 10.10.2.20 -U replicator \
  -D /var/lib/postgresql/16/main -Fp -Xs -R -v
sudo -u postgres touch /var/lib/postgresql/16/main/standby.signal
sudo systemctl start postgresql

To access services from your host, add these entries to your hosts file:

Windows: C:\Windows\System32\drivers\etc\hosts

macOS / Linux: /etc/hosts

10.10.1.100 lb.lab.local
10.10.1.20  db.lab.local

After adding the entries:

curl http://lb.lab.local/items

2dc-ha-lab/
├── README.md                 # This file
├── LICENSE                   # MIT License
├── .gitignore                # Git ignore rules
├── Vagrantfile               # VM definitions
├── images/
│   └── arch_whitebg.png      # Architecture diagram (white background)
├── provision/
│   ├── db1.sh                # Primary DB setup
│   ├── db2.sh                # Replica DB setup
│   ├── app.sh                # Flask app setup
│   ├── lb.sh                 # HAProxy setup
│   └── dns.sh                # Bind9 DNS setup
└── app/
    ├── app.py                # Flask application
    └── requirements.txt      # Python dependencies

Problem: Vagrant gets stuck at SSH auth method: private key.

Solution: Add this to your Vagrantfile:

vb.customize ["modifyvm", :id, "--cableconnected1", "on"]

• Workaround during vagrant up: If the VM gets stuck, open the VirtualBox GUI and bring the VM window to the foreground (click on it). This often "wakes up" the boot process and allows Vagrant to continue.

Problem: Error VERR_INTNET_FLT_IF_NOT_FOUND.

Solution:

1. Open VirtualBox → File → Tools → Network Manager.
2. Go to Host-only Networks.
3. Ensure you have two adapters:
   • 10.10.1.1/24 (DC1)
   • 10.10.2.1/24 (DC2)
4. Disable DHCP on both.

Problem: VirtualBox fails to start VMs.

Solution: Disable Hyper-V:

bcdedit /set hypervisorlaunchtype off

Problem: psycopg2.OperationalError: connection refused.

Solution: Ensure PostgreSQL is listening on all interfaces:

sudo sed -i "s/#listen_addresses = 'localhost'/listen_addresses = '*'/" /etc/postgresql/16/main/postgresql.conf
sudo systemctl restart postgresql

The first run downloads the Ubuntu base box (~700 MB) and installs all dependencies. Subsequent runs are much faster.

Yes, but you may need to reduce the memory allocated to each VM in the Vagrantfile. Edit vb.memory values to lower values (e.g., 1024 for databases, 512 for apps).

vagrant halt

To destroy them completely:

vagrant destroy -f

Edit the version in provision/db1.sh and provision/db2.sh. Change postgresql-16 to postgresql-15 or postgresql-14.

You can access it directly via http://10.10.2.10:5000, provided your host has a route to the 10.10.2.0/24 network.

Yes. Vagrant and VirtualBox work on macOS. The scripts are Bash-compatible.

vagrant destroy -f
vagrant up

This project is a personal learning lab and is not currently open to external contributions. However, you are welcome to:

• ⭐ Star the repository if you find it useful.
• 🍴 Fork it and use it for your own learning.
• 🐛 Report issues if you find any bugs or problems.
• 📝 Suggest improvements via GitHub Issues.

💡 Note: This lab is designed to be a reproducible environment for learning high availability concepts. If you have ideas for improvements, feel free to open an issue to discuss them.

This project is licensed under the MIT License - see the LICENSE file for details.

Rafael Jiménez Pérez

• GitHub: @rafajimenezdev
• Email: rafajimenezdev@gmail.com

Built with ❤️ for the infrastructure community