Another Day in the Trenches

Wow past couple days been having really strange issues between the DataCenter where hosted VoIP server live and customer VPNs.

This morning have a Customer VPN go down (been up and running in operation for over 6 months with no problems) so I do some verification on both sides (I’m able to SSH into customer Firewall)…

DataCenter: 
# show crypto isakmp sa
AM_WAIT_MSG3

Customer: 
# show crypto isakmp sa
AM_WAIT_MSG2

So basically the DataCenter Firewall received the initial DH public key sent to it and responded but the response did not get to the customers Firewall.

So I do some ping tests…

 

DataCenter Firewall is unable to ping the Customer Firewall.

But DataCenter Firewall CAN ping the Customer default gateway (and vice versa).

From a 3rd off-site location (my remote office) I am able to ping ALL IPs!!

So the issue is very unusual and only exists between DataCenter firewall and Customer firewall.

So I can Brighthouse and the second the technician picks up the phone is starts working again.

Nobody can explain why.