So here’s a quick post on something that cost me a good bit of time.
I had launched an EC2 instance from a custom AMI, and right after the boot it started failing EC2 instance reachability checks. The instance was running, but I couldn’t SSH into it. So I checked system logs via EC2 console and found this:
network: Bringing up interface ens5: ERROR : [/etc/sysconfig/network-script/ifup-eth] Device ens5 has different MAC address than expected, ignoring
what actually happened?
I had created an AMI from a running instance without cloud-init installed. So the image had the original MAC address hardcoded in /etc/sysconfig/network-scripts/ifcfg-ens5. When I launched a new ec2 instance from this AMI, AWS assigned a new MAC address to the network interface but the OS was still looking at the old one. It was a classic mismatch and therefore network failed to initialise, and so did the reachability check.
How I debugged it?
Since I couldn’t SSH into the instance, here’s what i did:
-
Stopped the instance, detached it’s root volume and attached it another working instance in same availability zone as a secondary volume(e.g., /dev/xvdf).
-
Mounted the volume to a temporary directory:
sudo mkdir /mnt/temp
sudo mount /dev/xvdf1 /mnt/temp -
Edit the network config file and delete the line with HWADDR compeletely.
vi /mnt/temp/etc/sysconfig/ -
Unmounted the volume, detached it and attached back to original instance as root volume and started it.
Hurray! I was able to connect succeessfully
How to prevent this from happening?
Install cloud-init before creating your custom AMI