VMware Failure Story: IaC Challenges in On-Premises Environments 😅

Hello everyone! Today I want to share my experience with using VMware and Vagrant for Infrastructure as Code (IaC) in a bare metal environment. Spoiler alert: it was much harder than expected! 🥲

IaC in On-Premises vs. Cloud: Heaven and Hell 📊

For my MoolMeow project, I tried to set up a k3s Kubernetes cluster on a Mac Mini using VMware and Vagrant. What I quickly discovered was that the compatibility between virtualization tools and IaC tools isn’t nearly as seamless as I had hoped.

Everything seems to work fine in the cloud… why is on-premises so difficult? 😭

My Battle with VMware and What I Learned 🔄

The issues I faced with the VMware and Vagrant combination were primarily network-related.

1. VMware’s Unique Network Configuration Approach 🌐

VMware uses MAC addresses to configure internal IP addresses. This was confusing at first, but I started to understand it after spending hours troubleshooting.

# VMware static IP configuration example
v.vmx["ethernet1.present"] = "TRUE"
v.vmx["ethernet1.connectionType"] = "custom"
v.vmx["ethernet1.virtualDev"] = "vmxnet3"
v.vmx["ethernet1.addressType"] = "static"
v.vmx["ethernet1.address"] = "00:50:56:01:00:10"  # MAC address
v.vmx["ethernet1.subnetMask"] = "255.255.255.0"
v.vmx["ethernet1.vnet"] = "vmnet3"

Here, ethernet1.address is a MAC address, which later gets mapped to an IP address by the DHCP server. This approach was very confusing initially.

2. IP Address Conflict Issues 😱

The IP range I configured (192.168.100.x) was already in use on my MacBook. I learned how to handle such situations through this experience.

# Checking network information
netstat -rn

# Result showing the network already in use
192.168.100        link#29            UC              bridge101      !

In such situations, I had to change all configurations to use a different IP range (e.g., 192.168.200.x).

3. Static IP Not Persisting 🔄

The most frustrating issue was that even with perfect VM configurations, after rebooting my Mac, Vagrant would assign random IPs instead of the static IPs I had configured.

➜ vagrant ssh-config
Host k8s-master
  HostName 111.11.11.111  # Should have been 192.168.200.10

When this happens, SSH connections fail and the cluster doesn’t work properly. 😫

ssh: connect to host 192.168.200.10 port 22: No route to host

Understanding VMware Network Configuration 🧠

I invested significant time understanding VMware network configuration, especially learning how to modify the networking file directly.

# VMware networking file example
answer VNET_3_DHCP yes
answer VNET_3_DHCP_CFG_HASH 111111111111111111111111111111111111111
answer VNET_3_HOSTONLY_NETMASK 255.255.255.0
answer VNET_3_HOSTONLY_SUBNET 192.168.200.0
answer VNET_3_HOSTONLY_UUID A111111-1111-1111-1111-111111111111
answer VNET_3_VIRTUAL_ADAPTER yes

I also learned how to restart network services:

sudo /Applications/VMware\ Fusion.app/Contents/Library/vmnet-cli --stop
sudo /Applications/VMware\ Fusion.app/Contents/Library/vmnet-cli --start

Eventually Switching to VirtualBox 🔄

Claude had suggested switching to VirtualBox about 10 times, but I kept ignoring the advice. After two days of struggle, I finally decided to abandon VMware and switch to VirtualBox.

VirtualBox has more intuitive network configuration and works much more reliably:

# Network configuration in VirtualBox - much simpler!
master.vm.network "private_network", ip: "192.168.56.10", netmask: "255.255.255.0"

Although VirtualBox had its own issues like Guest Additions installation failures, at least the IP address problems were nonexistent.

Lessons Learned from This Experience 🧐

Despite the struggles with VMware, my understanding of networking improved significantly:

1. Understanding Virtual Networks: How virtual network interfaces like vmnet and vboxnet work
2. DHCP Server Configuration: How to configure DHCP servers for static IP allocation
3. Network Troubleshooting: How to diagnose and resolve IP conflicts and routing issues
4. MAC and IP Address Relationship: How MAC addresses and IP addresses are mapped in networks

This experience reinforced the fact that “networking is a difficult domain if you don’t understand the basics.”

Conclusion: IaC is Great but Compatibility is Key 🎯

Infrastructure as Code is certainly a valuable concept, but in on-premises environments, compatibility issues can create significant challenges. The VMware and Vagrant combination, in particular, can exhibit unpredictable behavior in network configuration.

For future similar projects, I would more seriously consider:

1. VirtualBox + Vagrant
2. Manual VM configuration

If you’re planning to implement IaC in an on-premises environment, I recommend testing compatibility issues beforehand!

In my next post, I’ll share how I successfully built a k3s cluster using VirtualBox. Stay tuned for more updates! 👋