Wednesday, April 26, 2023

containers: Alpine Linux DNS issue

Why I Will Never Use Alpine Linux Ever Again | Martin Heinz | Personal Website & Blog

"To understand what makes Alpine a bad choice in some situations, we first need to talk about musl. musl is an implementation of C standard library. It is more lightweight, faster and simpler than glibc used by other Linux distros, such as Ubuntu. Both of these implementations are interchangeable for the most part, that's why in most cases you can switch from e.g., Ubuntu to Alpine and never notice any difference.

However, the little differences can cause all the grief. Some of it stems from how musl (and therefore also Alpine) handles DNS (it's always DNS), more specifically, musl (by design) doesn't support DNS-over-TCP. Usually, you would not notice this difference, because most of the time a single UDP packet (512 bytes) is enough to resolve hostnames... until it isn't enough and your application (running on Kubernetes) that previously worked completely fine for months suddenly starts throwing "Unknown Host" exceptions for one particular (very critical) hostname. The worst part is that this can manifest randomly, anytime when some external network change causes the resolution of some particular domain to require more than the 512 bytes available in single UDP packet.

By using Alpine, you're getting "free" chaos engineering for you cluster."


workaround: use short names for services in K8s.