Exposing Redpanda/Kafka Externally in United Manufacturing Hub (UMH)

Jeremy Theocharis -

To expose Redpanda (Kafka) externally in the United Manufacturing Hub (UMH), you need to adjust the Helm chart values accordingly. This guide provides step-by-step instructions for setting up external access to Redpanda in a single-node Kubernetes cluster.

Prerequisites

  • Advanced User Access: You should have SSH access to the Kubernetes cluster or use tools like OpenLens to modify Helm chart values.
  • Helm Installed: Ensure that Helm is installed on your local machine.
  • Kubeconfig File: Access to the kubeconfig file (e.g., /etc/rancher/k3s/k3s.yaml).
  • Public IP Address: Replace 10.13.37.200 with your actual public IP address.

Steps to Expose Redpanda Externally

1. Access Your Kubernetes Cluster

Log in to your Kubernetes cluster via SSH:

ssh user@your-cluster-ip

2. Update the UMH Helm Chart

Run the following Helm upgrade command to enable external access for Redpanda:

sudo helm upgrade united-manufacturing-hub united-manufacturing-hub/united-manufacturing-hub \
  --namespace united-manufacturing-hub \
  --kubeconfig /etc/rancher/k3s/k3s.yaml \
  --set redpanda.external.enabled=true \
  --set redpanda.external.service.enabled=true \
  --set redpanda.external.type=LoadBalancer \
  --set redpanda.tls.enabled=false \
  --set "redpanda.external.addresses={YOUR_PUBLIC_IP}" \
  --set redpanda.external.domain="" \
  --wait

Note: Replace YOUR_PUBLIC_IP with your actual public IP address.

Explanation of the Command

  • --set redpanda.external.enabled=true: Enables external access for Redpanda.
  • --set redpanda.external.service.enabled=true: Activates the external service.
  • --set redpanda.external.type=LoadBalancer: Sets the service type to LoadBalancer.
  • --set redpanda.tls.enabled=false: Disables TLS for simplicity (enable TLS in production environments).
  • --set "redpanda.external.addresses={YOUR_PUBLIC_IP}": Specifies the external IP addresses.
  • --set redpanda.external.domain="": Leaves the domain empty; useful if you're not using a custom domain.
  • --wait: Waits until all resources are successfully upgraded.

3. Verify the External Service

Check if the Redpanda service is exposed externally:

kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml get services -n united-manufacturing-hub

You should see a service of type LoadBalancer with an external IP assigned.

4. Restart Redpanda

Restart the Redpanda pod:

kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml delete pod -n united-manufacturing-hub united-manufacturing-hub-kafka-0

4. Connect to Redpanda

You can now connect to Redpanda using the external IP and port 31092:

  • Connection String: YOUR_PUBLIC_IP:31092

Example Use Case: Kafka-to-Kafka Bridge

In scenarios like setting up a Kafka-to-Kafka bridge, you can use the above connection string to link your external Kafka clients to Redpanda.

5. Testing the Connection

Use a Kafka client to test the connection:

kafka-console-producer --broker-list YOUR_PUBLIC_IP:31092 --topic test-topic

Important Considerations

  • Advertised Address: Failing to update the advertised address (redpanda.external.addresses) will result in consumers/producers receiving internal cluster addresses, causing connection failures.
  • Security: Exposing services externally can introduce security risks. Ensure proper firewall rules and security measures are in place.
  • TLS Encryption: TLS is disabled in this guide for simplicity. In a production environment, it is recommended to enable TLS for secure communication.

Additional Resources

Troubleshooting

  • Service Not Accessible: If the service isn't accessible externally, check firewall settings and ensure that the Kubernetes service has assigned the external IP.
  • Connection Timeouts: Verify that the advertised addresses are correctly set and that there are no network policies blocking the traffic.

By following this guide, you should be able to expose Redpanda externally in the UMH and connect your external Kafka clients successfully.