Resolving Docker Container Networking Issues while connected to VPN using Cisco AnyConnect

After prepping and polishing a custom legacy Sitecore 8.2 Docker environment for our developers, we ran into a significant blocker that had us questioning whether we needed to backtrack and use locally installed Sitecore instances instead. 

The blocker stemmed from this particular client's VPN: Cisco AnyConnect Mobility Client. 🤮

At a high level, any time we connect to the VPN using Cisco AnyConnect, the running containers would begin to misbehave - and symptoms disappeared after VPN was de-activated.  

Symptoms included:

1. Inability to use custom hostnames to access the site (pings from the host to the CM and Solr hostnames failed with an `Unreachable` or `Request Timeout` code).
2. Inability to use the localhost:portnumber hostname to access the site.
3. Complete loss of internet access from the running containers.

Because a VPN connection is required for several API-based components, it was essential to solve this.  

Some of the troubleshooting attempts included:

• Switched ISOLATION mode from isolation to hyperv.
• Checked and uncheck various options in the Cisco AnyConnect settings (including `Allow local (LAN) access when using VPN`).
• Checked and unchecked various Docker settings under the General tab in Docker Desktop settings (`Expose daemon on tcp://localhost:2375 without TLS`).
• Asserted local firewall settings.
• Applied DNS overrides to the Docker Engine daemon.json file that matched the active DNS configuration for the VPN endpoint.
• Applied various parameters to the docker-compose.yml file (dns, extra_hosts, etc.)
• Fiddled with various `Advanced TCP/IP Settings` under Control Panel\Network and Internet\Network Connections in Windows.
• Created custom Hyper-V and Docker bridge/transparent networks to try to restore internet connectivity. 

The number of tabs I had opened in my browser was unfathomable without much to show for it.  There were undoubtedly many similar issues reported across the web related to Cisco AnyConnect and Docker, but no suggestions remedied the problem.

After hours of troubleshooting, I tried to replicate the behavior with other VPN connections not using the Cisco AnyConnect client and found that none of the symptoms were present.   I couldn't find any evidence that this is an issue with Docker itself but instead caused by how Cisco AnyConnect handled connections and IP routing.  

I then came across a comment in a thread related to drive sharing with Docker when using AnyConnect: https://github.com/docker/for-win/issues/360#issuecomment-442586618 

I ♥ you, jrbercart

Since we don't have any pull over the client's VPN setup and configuration, I decided to try OpenConnect as a substitution for Cisco AnyConnect, which evidently uses the same protocol to establish a VPN connection.  

I connected to the client's VPN endpoint using OpenConnect, and all of the networking issues with the running Docker containers disappeared!  

If you happen to find yourself in a similar situation, go ahead and drop Cisco AnyConnect and give OpenConnect a try to save yourself some troubleshooting hours! ☺