Fixing DNS Issues on Hetzner Servers for UMH Installation

Jeremy Theocharis -

Setting up the United Manufacturing Hub on Hetzner servers should be straightforward—but DNS issues can bring your installation to a grinding halt.

Recently, users installing applications like UMH (United Manufacturing Hub) that rely on Kubernetes and CoreDNS have encountered DNS resolution problems on Hetzner virtual machines (VMs). These issues can disrupt the installation process and prevent UMH from functioning correctly. Understanding the root cause and knowing how to fix it is essential for a smooth deployment.

This tutorial will help you understand why these DNS issues occur during UMH installation on Hetzner servers and provide clear steps to resolve them.

By adjusting your DNS settings, you can ensure that UMH installs smoothly and operates without hiccups on your Hetzner server.

Hetzner, a popular hosting provider, offers both recursive and authoritative DNS servers. While there have been no official statements from Hetzner regarding changes to their DNS services, users have reported DNS resolution issues when creating new VMs. Similar problems have been documented by others in the community, indicating that this is a common challenge when deploying services on Hetzner servers.

For instance, in the article "Sometimes the problem is DNS (on Hetzner)", the author describes challenges with cert-manager in a Kubernetes cluster failing due to DNS resolution errors. This underscores that the problem isn't isolated to UMH installations.

We'll walk through three main steps:

  1. Identifying the DNS Issue During UMH Installation
  2. Understanding Why DNS Issues Occur on Hetzner Servers
  3. Applying a Solution to Fix DNS Resolution

Don't let a simple DNS setting prevent you from leveraging the full capabilities of UMH on your Hetzner server.

Ready to understand and fix the DNS issues affecting your UMH installation? Let's dive in.

1. Identifying the DNS Issue During UMH Installation

The first sign of trouble typically appears during the UMH installation process when the system can't resolve certain domain names.

Symptoms of the Problem

  • Installation Hang-ups: The installation script stalls at the point where it needs to reach external resources.
  • Error Messages: You may see logs indicating DNS lookup failures, such as:
  dial tcp: lookup management.umh.app on 10.43.0.10:53: server misbehaving

Checking for DNS Resolution Issues

To confirm if you're facing a DNS problem:

  1. Test DNS Resolution from Your Server:
   dig management.umh.app

If the command times out or fails to retrieve the address, DNS resolution isn't working correctly.

Understanding the Impact

When DNS resolution fails:

  • UMH Components Can't Communicate: Essential services can't reach external APIs or repositories.
  • Installation Scripts Fail: The setup process can't proceed without accessing required resources.

Now that we've identified the DNS issue during installation, let's explore why this problem occurs specifically on Hetzner servers.

2. Understanding Why DNS Issues Occur on Hetzner Servers

The root of the DNS issue lies in the configuration of Hetzner's DNS servers, which may not support certain types of DNS queries needed by CoreDNS in Kubernetes.

Hetzner's DNS Configuration

  • Recursive DNS Servers: Hetzner provides recursive DNS servers for their customers. According to Hetzner's recursive name servers documentation, the official recursive DNS servers are:
  • Non-Recursive Responses: It has been observed that Hetzner's DNS servers may not support recursive queries from clients in certain configurations. This can cause failures when applications within the server attempt to resolve external domain names.

IPv6 Recursive Name Servers:

2a01:4ff:ff00::add:1
2a01:4ff:ff00::add:2

IPv4 Recursive Name Servers:

185.12.64.1
185.12.64.2

How DNS Works in Kubernetes

  • CoreDNS: Kubernetes clusters use CoreDNS for service discovery and DNS resolution within the cluster.
  • Forwarding to Upstream DNS Servers: CoreDNS often forwards queries for external domains to the underlying system's DNS servers.

Why This Causes Problems

  • Lack of Recursive Support: If the system's DNS servers do not support recursive queries or are misconfigured, CoreDNS cannot resolve external domains.
  • Installation Failures: UMH's installation relies on resolving external resources, and DNS failures interrupt this process.
  • Timeouts and Errors: CoreDNS may exhibit timeouts or errors when forwarding queries to DNS servers that do not respond as expected.

Similar Reports from the Community

Other users have faced analogous issues due to DNS configurations on Hetzner servers:

  • Cert-Manager Failures: As reported in "Sometimes the problem is DNS (on Hetzner)", cert-manager in Kubernetes clusters fails HTTP-01 challenges because it cannot resolve domain names internally.
  • Impact on Kubernetes Deployments: The inability to resolve external domain names affects various Kubernetes-based applications beyond UMH.

Understanding the cause equips us to implement a solution. Let's apply the necessary changes to fix the DNS resolution issue.

3. Applying a Solution to Fix DNS Resolution

By modifying the DNS configuration on your Hetzner server and within your Kubernetes cluster, you can resolve the DNS issues affecting UMH installation.

Step-by-Step Guide

A. Modify CoreDNS Configuration

  1. Edit the CoreDNS ConfigMap:
   kubectl -n kube-system edit configmap coredns
  1. Update the forward Directive:

In the Corefile section, locate the line:

   forward . /etc/resolv.conf

Replace it with a public DNS server that supports recursion, such as Cloudflare's 1.1.1.1 or Google's 8.8.8.8:

   forward . 1.1.1.1

Example Updated Corefile:

   .:53 {
       errors
       health
       ready
       kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
       }
       hosts /etc/coredns/NodeHosts {
           ttl 60
           reload 15s
           fallthrough
       }
       prometheus :9153
       forward . 1.1.1.1
       cache 30
       loop
       reload
       loadbalance
       import /etc/coredns/custom/*.override
   }
   import /etc/coredns/custom/*.server
  1. Save and Exit:

Save the changes in your text editor to update the ConfigMap.

  1. Restart CoreDNS Pods:
   kubectl -n kube-system delete pods -l k8s-app=kube-dns

This will force the CoreDNS pods to reload the new configuration.

B. Verify DNS Resolution

  1. Test DNS Resolution Again:
   dig management.umh.app

You should now receive a valid response with the IP address.

  1. Check UMH Installation Logs:

If the installation is still running, it should proceed past the previous failure point.

C. Update System DNS Settings (Optional)

While changing CoreDNS configuration often resolves the issue within Kubernetes, you may also need to adjust the DNS settings on your server to ensure all services can resolve domain names correctly.

  1. Edit /etc/resolv.conf:

Replace the existing nameserver entries with public DNS servers:

   nameserver 1.1.1.1
   nameserver 8.8.8.8

Note: Directly editing /etc/resolv.conf may not be persistent across reboots or network restarts due to DHCP overwriting the file.

  1. Configure Systemd-Resolved (If Applicable):

If your system uses systemd-resolved, adjust its configuration:

  • Prevent DHCP from Overwriting DNS Settings:

Ensure /etc/resolv.conf is Symlinked Correctly:

 ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf

Restart systemd-resolved:

 systemctl restart systemd-resolved

Edit /etc/systemd/resolved.conf:

 [Resolve]
 DNS=1.1.1.1 8.8.8.8

Modify your network configuration to prevent DHCP from overwriting /etc/resolv.conf. This may involve editing network interface configurations or DHCP client settings.

  1. Verify System DNS Resolution:
   dig management.umh.app

The command should return a valid IP address.

Understanding the Solution

  • Why Use Public DNS Servers: Public DNS servers like Cloudflare's 1.1.1.1 or Google's 8.8.8.8 support recursive queries and are reliable.
  • Impact on CoreDNS: By directing CoreDNS to use DNS servers that support recursion, it can successfully resolve external domains needed by UMH.
  • System-wide Changes: Adjusting system DNS settings ensures all services on your server, not just Kubernetes, can resolve domains correctly.

With the DNS issues resolved, your UMH installation should proceed smoothly on your Hetzner server.

Conclusion

By identifying the DNS configuration issue and applying a straightforward fix, we've turned a frustrating installation roadblock into a manageable task.

We've walked through identifying the DNS resolution issues during UMH installation on Hetzner servers, understanding that certain DNS configurations on Hetzner servers can cause problems, and applying a solution by updating CoreDNS and system DNS settings.

Adjusting your DNS configuration ensures that UMH can fully leverage Kubernetes and CoreDNS, providing a stable and functional environment for your manufacturing operations.

Don't let DNS hurdles impede your progress. With this knowledge, you can confidently set up UMH on Hetzner servers and focus on what's important—optimizing your manufacturing processes.

Now it's your turn to implement these steps and get your UMH installation up and running on Hetzner! Follow the guide, and you'll overcome the DNS issues in no time.

Additional Resources:

Note: Always ensure that any changes to DNS configurations comply with your organization's policies and that you understand the implications of directing DNS queries to external servers.

Disclaimer: While this guide provides steps to resolve DNS issues observed on Hetzner servers, the root cause may vary, and there may be other factors at play. As of now, there are no official statements from Hetzner regarding changes to their DNS services that would cause these issues. It's always a good practice to reach out to Hetzner support if problems persist.