Apache Kafka

Kubernetes

Accessing an internal Strimzi Kafka cluster in Kubernetes without exposing it using port-forward trickery

Debug your remote strimzi kafka cluster without exposing it using port forwad and minor configuration changes.

Photo by Alexander Paul on Unsplash

Yea, this is quite specific. That’s why I decided to write a small article about it.

But, wait, why would I need this? Well, here’s the situation, I have a Kafka cluster running inside a Kubernetes cluster, this cluster works in tandem with other services inside the same kubernetes cluster to manage internal queues and ETL pipelines, here’s the catch, I planned this kafka cluster to never be exposed (because it’s used as an internal queue system and has some sensitive data being processed) However I had a problem…

When I planned this set-up I didn’t foresee any external access to the topics, but suddenly I had the need, for an off-the-grid process that took some of this data out for analysis.

First approach

At first, I thought a simple port forward to the kafka broker would be enough, and I happily generated the service account, role and generated a token for the credentials, I started the port forward, made sure I only could access the desired namespace and run a simple telnet to check and… it went just fine. “That was way too easy” mumbled to myself, and then the madness started, I was able to telnet the broker, but could not connect via scripts, it said.. Timeout?

The second approach

After some documentation digging, I found out the issue. Turns out that when connecting to the broker service, there are some addresses that are advertised back to the consumer/producer, these are the FQDN of the broker in the internal cluster by default, which obviously, the instance running the port forward could not resolve and therefore it ended-up throwing a timeout.

So enough talk, let’s get to work

1. Generate required kube resources (user, role, role binding, the token, and finally the kubeconfig file to be used later on)
2. Edit your strimzi kafka cluster, you will see a listerners key, by default it will be plan, and tls listeners. We need to create a new listener, this one is going to be a bit different to the default ones:

- name: change-me
  port: 9094 # must not collide with other listeners ports
  tls: false #you might want to use true, but we are using TLS port forwarding so we should be good
  type: internal # we are port forwarding so still and internal endpoint
  configuration:
    brokers: # here you will configure the list of brokers you have available
    # and how the brokers are exposed when connected to this listener
    - advertisedHost: localhost
      advertisedPort: 9092 # these can collide with other listeners but not other brokers within this list or the listener itself
      broker: 0
    - advertisedHost: localhost
      advertisedPort: 9093
      broker: 0

Then, on the target instance configure the credentials and kubectl, and then we are going to configure the next port forward:

1. To the listener (broker service) port 9094
2. To each broker pod, (ports 9093, and 9092)

In total 3 port forwards configured.

After launching the script for testing connectivity, I successfully was able to access kafka via port forward.

Summary and tips

So basically to expose kafka correctly via port forward you need to configure a custom listener with advertisedhost / port that exposes a correct hostname and not the internal cluster FQDN of the pod. This is preferred to using the /etc/host file since it will not break if the brokers change names.

Don’t forget to keep minimal permissions on the kubeconfig file generated for the port forward, it could pose a significant security risk.

This is not a production grade way of doing this! But it is a great to get required data in specific moments with minimal arquitectural changes, or as a debug tool for development environments. Be careful when exposing services and stay safe