We have been in the process of moving to Kubernetes in our company and one of the things we have done is setup a VPN inside Kubernetes. This allows us to use internal Kubernetes domain names from our laptops. We ran into a few issues getting this up and running.
our first try
Our first port of call was to create an openvpn access server within our cluster. This worked well except that access to the openvpn-as webpage would fail because Kubernetes round-robins connections to pods and it would try to negotiate a token for each new connection. This negotiation caused a redirect, which looked to openvpn as though it was a new connection and so a new negotiation would begin. This was solved by putting an haproxy container in-front of the openvpn access server. The next issue, though, was one that couldn’t be solved easily. The openvpn license ties itself to the specific machine it’s running on and whenever the openvpn container was restarted, the license then became useless.
lets go opensource
So we then looked at using a pure openvpn server. After some time figuring out how to get it working with 2-factor authentication and encrypted certificates, we were up and running. This worked well except that exactly one hour after connecting, the VPN would disconnect. The logs showed there was a soft TLS renegotiation happening every hour. OpenVPN requires the password to be sent whenever the renegotiation happens, but since we use 2-factor authentication, we can’t cache our passwords for this purpose. Looking into the documentation, we found that we could extend the renegotiation time by setting reneg-sec in our setting files. We no longer had disconnections every hour. I then made my changes on a fork and pushed the requests back up.
We continued using the VPN but started to notice that occasionally we’d get disconnected without any pattern. The logs showed that reneg-bytes was being exceeded. This was odd because we don’t set reneg-bytes and by default it should be turned off. Further investigation took us to this page which shows that if openvpn detects it is using a cipher that is vulnerable to sweet32 it will set reneg-bytes to 64MB. The solution in this case was to change to a more secure cipher. Once the new cipher was in place, the VPN no longer disconnected as it was before.
to the future
I’m sure we will continue to improve our VPN over the coming months but so far the experience has been quite good. Using an actively maintained open source project also allowed us to get our changes merged into the upstream project.
If you enjoyed the read, drop us a comment below or share the article, follow us on Twitter or subscribe to our #MetaBeers newsletter. Before you go, grab a PDF of the article, and let us know if it’s time we worked together.