what does kafka return to clients while connecting

tl;dr

When a client wants to transport or receive a bulletin from Apache Kafka^®, there are two types of connection that must succeed:

1. The initial connection to a broker (the bootstrap). This returns metadata to the client, including a list of all the brokers in the cluster and their connectedness endpoints.
2. The client so connects to one (or more) of the brokers returned in the kickoff step every bit required. If the broker has not been configured correctly, the connections will fail.

What sometimes happens is that people focus on only footstep i to a higher place, and get caught out by stride 2. The banker details returned in step 1 are defined by the advertised.listeners setting of the banker(southward) and must exist resolvable and attainable from the customer machine.

To read more most the protocol, see the docs, besides as this previous commodity that I wrote. If the nuts and bolts of the protocol are the last thing y'all're interested in and you just desire to write applications with Kafka you should check out Confluent Cloud. Information technology'due south a fully managed ​​Apache Kafka service in the deject, with not an advertised.listeners configuration for you to worry about in sight!

Below, I use a customer connecting to Kafka in various permutations of deployment topology. It'southward written using Python with librdkafka (confluent_kafka), simply the principle applies to clients across all languages. You lot tin find the code on GitHub. It's very simple and just serves to illustrate the connection process. Information technology's simplified for clarity, at the expense of expert coding and functionality 🙂

An illustrated example of a Kafka client connecting to a Broker

Let's imagine we have two servers. On ane is our client, and on the other is our Kafka cluster'southward single broker (forget for a moment that Kafka clusters normally take a minimum of three brokers).

1. The customer initiates a connectedness to the bootstrap server(s), which is ane (or more) of the brokers on the cluster.
   
2. The broker returns metadata, which includes the host and port on which all the brokers in the cluster can exist reached.
3. This list is what the customer then uses for all subsequent connections to produce or eat data. The address used in the initial connection is simply for the customer to find a bootstrap server on the cluster of due north brokers, from which the client is then given a current listing of all the brokers. This style, the client doesn't accept to know at all times the list of all the brokers. The reason the client needs the details of all the brokers is that it will connect straight to one or more of the brokers based on which has data for the topic partition with which it wants to interact.
   

What oftentimes goes wrong is that the broker is misconfigured and returns an address (the advertised.listener) on which the customer cannot correctly connect to the broker. In this case, the timeline looks like this:

1. The client initiates a connection to the bootstrap server(due south), which is one (or more) of the brokers on the cluster
   
2. The broker returns an wrong hostname to the client
   
3. The client then tries to connect to this incorrect address, and then fails (since the Kafka broker is not on the client auto, which is what localhost points to)
   

This commodity will walk through some common scenarios and explain how to prepare each one.

Only one broker?

All these examples are using just 1 banker, which is fine for a sandbox but utterly useless for anything approaching a existent environment. In practice, you'd have a minimum of 3 brokers in your cluster. Your client would bootstrap against one (or more) of these, and that broker would return the metadata of each of the brokers in the cluster to the client.

Scenario 0: Client and Kafka running on the same local motorcar

For this example, I'thousand running Confluent Platform on my local car, but you tin can also run this on any other Kafka distribution yous care to.

$ confluent local status kafka … kafka is [UP] zookeeper is [UP]        

My Python client is connecting with a bootstrap server setting of localhost:9092.

This works but fine:

Note: The broker metadata returned is 192.168.10.83, just since that'southward the IP of my local automobile, information technology works just fine.

Scenario 1: Customer and Kafka running on the different machines

Now permit'southward cheque the connection to a Kafka broker running on another motorcar. This could be a auto on your local network, or perhaps running on deject infrastructure such as Amazon Web Services (AWS), Microsoft Azure, or Google Cloud Platform (GCP).

In this example, my client is running on my laptop, connecting to Kafka running on some other auto on my LAN called asgard03:

The initial connection succeeds. But note that the BrokerMetadata nosotros go back shows that at that place is ane broker, with a hostname of localhost. That means that our client is going to be using localhost to endeavor to connect to a banker when producing and consuming letters. That's bad news, considering on our client machine, at that place is no Kafka broker at localhost (or if there happened to be, some actually weird things would probably happen).

And thus it comes to pass:

And then how practise we fix information technology? We go and speak to our lovely Kafka ambassador (who may well be usa) and prepare the server.properties on the banker(s) then that advertised.listeners correctly provides the hostname and port on which the broker tin exist reached from clients. Nosotros saw above that information technology was returning localhost. Allow's get and fix this. In my banker's server.properties, I take this:

advertised.listeners=PLAINTEXT://localhost:9092 listeners=PLAINTEXT://0.0.0.0:9092        

And alter the advertised.listeners configuration thus:

advertised.listeners=PLAINTEXT://asgard03.moffatt.me:9092 listeners=PLAINTEXT://0.0.0.0:9092        

The listener itself remains unchanged (information technology binds to all available NICs, on port 9092). The only divergence is that this listener will tell a client to reach it on asgard03.moffatt.me instead of localhost.

And then after applying these changes to the advertised.listener on each broker and restarting each one of them, the producer and consumer work correctly:

The broker metadata is showing at present with a hostname that correctly resolves from the client.

Scenario ii: Kafka and customer running in Docker

Now nosotros're going to become into the wonderful globe of Docker. Docker networking is a beast in its ain correct and I am not going to embrace it here because Kafka listeners alone are enough to digest in i article. If you think but ane thing, allow it exist this: when y'all run something in Docker, it executes in a container in its own little world. It has what appears to itself as its own hostname, its ain network accost, its ain filesystem. Then, for example, when you ask code in a Docker container to connect to localhost, it volition be connecting to itself and not the host motorcar on which y'all are running it. This catches people out, considering they're used to their laptop being localhost, so it seems puzzling why code running on the laptop cannot connect to localhost. Simply, remember, the code isn't running on your laptop itself. It's running in a container on your laptop.

Nosotros'll start with the simplest permutation here, and run both Kafka and our client inside Docker on the same Docker network. First, create a Dockerfile to include our Python client into a Docker container:

          FROM          python:three          # We'll add together netcat cos it's a really useful # network troubleshooting tool          RUN          apt-get update          RUN          apt-become install -y netcat          # Install the Confluent Kafka python library          RUN          pip install confluent_kafka          # Add together our script          Add          python_kafka_test_client.py /          ENTRYPOINT          [ "python", "/python_kafka_test_client.py"]        

Build the Docker image:

docker build -t python_kafka_test_client .        

And then provision a Kafka banker:

docker network create rmoff_kafka docker run --network=rmoff_kafka --rm --detach --name zookeeper -east ZOOKEEPER_CLIENT_PORT=2181 confluentinc/cp-zookeeper:5.five.0 docker run --network=rmoff_kafka --rm --detach --proper name broker \            -p 9092:9092 \            -e KAFKA_BROKER_ID=1 \            -due east KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 \            -eastward KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 \            -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=one \            confluentinc/cp-kafka:5.five.0        

Confirm that you accept two containers running: one Apache ZooKeeper™ and 1 Kafka broker:

$ docker ps IMAGE                              Status              PORTS                          NAMES confluentinc/cp-kafka:5.v.0        Upwardly 32 seconds       0.0.0.0:9092->9092/tcp         broker confluentinc/cp-zookeeper:five.5.0    Up 33 seconds       2181/tcp, 2888/tcp, 3888/tcp   zookeeper        

Note that nosotros're creating our own Docker network on which to run these containers, so that we tin communicate betwixt them. Even though they're running on Docker on my laptop, so far as each container is concerned, they're on carve up machines and communicating across a network.

Let's spin up the client and see what happens:

$ docker run --network=rmoff_kafka --rm --name python_kafka_test_client \         --tty python_kafka_test_client broker:9092        

You lot can see in the metadata returned that even though we successfully connect to the broker initially, information technology gives us localhost dorsum as the broker host. This means that the producer and consumer fail because they'll be trying to connect to that—and localhost from the customer container is itself, not the broker.

To set it? Tell the broker to annunciate its listener correctly. Since the Kafka broker's name on the network is broker (inherited from its container proper name), we need to set this as its advertised listener and change:

          -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 \        

…to:

          -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://broker:9092 \        

So now our banker looks like this:

docker terminate broker docker run --network=rmoff_kafka --rm --detach --proper name broker \            -p 9092:9092 \            -due east KAFKA_BROKER_ID=1 \            -e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 \            -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://banker:9092 \            -east KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \            confluentinc/cp-kafka:5.five.0        

And the client works but perfectly:

Scenario 3: Kafka in Docker Compose

Mucking about with command line flags for configuration of Docker containers gets kind of gross afterward a brusk amount of time. Much better is to use Docker Compose.
Shut downward the Docker containers from in a higher place first (docker rm -f broker; docker rm -f zookeeper) and so create docker-compose.yml locally using this example.

Making certain you're in the same folder every bit the in a higher place docker-compose.yml run:

docker-compose upwardly        

Yous'll meet ZooKeeper and the Kafka broker start and then the Python test client:

Pretty squeamish, huh 👍

You can find total-diddled Docker Compose files for Apache Kafka and Confluent Platform including multiple brokers in this repository.

Scenario 4: Kafka in Docker container with a client running locally

What if y'all want to run your client locally? Mayhap that's where your IDE resides, or yous just don't want to Docker-ify your client?

Allow'southward take the example we finished up with above, in which Kafka is running in Docker via Docker Etch. If we try to connect our client to it locally, it fails:

$ python python_kafka_test_client.py localhost:9092        

Ah, but above we were using a private Docker network for the containers, and nosotros've non opened up whatsoever port for access from the host car. Let's change that, and expose 9092 to the host. I'm going to exercise this in the Docker Compose YAML—if you want to run it from docker run directly, you tin, but you lot'll need to translate the Docker Compose into CLI directly (which is a faff and not pretty and why you should but use Docker Compose 😉):

…   banker:     image: confluentinc/cp-kafka:five.5.0     container_name: banker     networks:        - rmoff_kafka     ports:       - "9092:9092" …        

You tin run docker-compose upwardly -d and it will restart any containers for which the configuration has inverse (i.e., broker). Note that if y'all just run docker-etch restart broker, it volition restart the container using its existing configuration (and not pick up the ports improver).

Once we've restarted the container, nosotros can bank check that port 9092 is being forwarded:

$ docker ps -a CONTAINER ID        IMAGE                             …  PORTS                          NAMES 37f5df65e9fc        confluentinc/cp-kafka:v.5.0       …  0.0.0.0:9092->9092/tcp         banker …        

Permit's try our local client again. It starts off well—we tin can connect!

Just then things plow sour:

Whilst we tin connect to the bootstrap server, information technology returns broker:9092 in the metadata.

This is exactly what we told it to exercise in the previous section, when we were fixing it to work with clients running within the Docker network. If nosotros change advertised.listener dorsum to localhost now, the Kafka broker won't piece of work except for connections from the host.

Calculation a new listener to the broker

So how do we juggle connections both within and external to Docker? Past creating a new listener. Brokers can accept multiple listeners for exactly this purpose. Network topologies get funky, and when the going gets funky, Kafka rocks out some more than listeners.
The changes look like this:

…     ports:       - "19092:19092"     environment:       KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://banker:9092,CONNECTIONS_FROM_HOST://localhost:19092       KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONNECTIONS_FROM_HOST:PLAINTEXT …        

Nosotros create a new listener called CONNECTIONS_FROM_HOST using port 19092 and the new advertised.listener is on localhost, which is crucial. Because it'southward on a different port, we alter the ports mapping (exposing 19092 instead of 9092).

The existing listener (PLAINTEXT) remains unchanged. We besides demand to specify KAFKA_LISTENER_SECURITY_PROTOCOL_MAP. This previously used a default value for the unmarried listener, but at present that nosotros've added some other, we need to configure it explicitly.

After bouncing the broker to pick up the new config, our local client works perfectly—and then long equally we remember to point it at the new listener port (19092):

Over in Docker Compose, we can see that our Docker-based client still works:

Scenario 5: Kafka running locally with a client in Docker container

What near if we invert this and have Kafka running locally on our laptop merely as we did originally, and instead run the client in Docker? It's not an obvious way to be running things, but ¯\_(ツ)_/¯

First, I shut down the Docker containers from above (docker-compose down) then showtime Kafka running locally (confluent local offset kafka). If we run our client in its Docker container (the image for which we built above), we can see it's not happy:

docker run --tty python_kafka_test_client localhost:9092        

If you remember the Docker/localhost paradox described in a higher place, you'll run across what'south going on here. Inside the customer'south Docker container, localhost is itself —it's not the "localhost" that we think of our laptop, the Docker host, beingness. And of class, on our client'south Docker container at that place is no Kafka broker running at 9092, hence the error.

If y'all don't quite believe me, endeavour running this, which checks from within the Docker container if port 9092 on localhost is open:

$ docker run -it --rm --entrypoint "/bin/nc" \     python_kafka_test_client -vz \     localhost 9092 localhost [127.0.0.one] 9092 (?) : Connection refused        

On the Docker host motorcar, Kafka is up and the port is open:

$ nc -vz localhost 9092 Connectedness to localhost port 9092 [tcp/XmlIpcRegSvc] succeeded!        

So how do we connect our customer to our host? Before we answer that, let'south consider why we might want to exercise this. There are two reasons you'll exist in this country:

1. Yous're at this point because yous're but developing things and trying to get stuff working in any manner you can and volition worry nigh doing information technology "properly" later
2. You're building a client application that volition run on Docker and connect to Kafka running elsewhere

For the latter scenario, you need to refer above to the "client and Kafka on different machines" and make sure that (a) the brokers advertise their right listener details and (b) the container can correctly resolve these host addresses.

For the former (trying to admission Kafka running locally from a customer running in Docker), you have a few options, none of which are particularly pleasant. If you're running Docker on the Mac, in that location'southward a hacky workaround to use host.docker.internal as the accost on which the host automobile tin can be accessed from inside the container:

$ docker run -information technology --rm --entrypoint "/bin/nc" \     python_kafka_test_client -vz \     host.docker.internal 9092  host.docker.internal [192.168.65.2] 9092 (?) open        

And so the container can see the host's 9092 port. What if nosotros endeavour to connect to that from our actual Kafka client?

And so the initial connect actually works, but check out the metadata nosotros get back: localhost:9092. Why? Considering advertised.listeners. So now the producer and consumer won't work, considering they're trying to connect to localhost:9092 within the container, which won't work.

Hack time? OK. Let'south have our poor local Kafka broker and kludge it to expose a listener on host.docker.internal. Because we don't want to break the Kafka broker for other clients that are actually wanting to connect on localhost, we'll create ourselves a new listener. Change the server.properties on the broker from:

listeners=PLAINTEXT://:9092 advertised.listeners=PLAINTEXT://localhost:9092 listener.security.protocol.map=PLAINTEXT:PLAINTEXT        

…to:

listeners=PLAINTEXT://:9092,RMOFF_DOCKER_HACK://:19092 advertised.listeners=PLAINTEXT://localhost:9092,RMOFF_DOCKER_HACK://host.docker.internal:19092 listener.security.protocol.map=PLAINTEXT:PLAINTEXT,RMOFF_DOCKER_HACK:PLAINTEXT        

The original listener remains unchanged. The magic matter we've done hither though is calculation a new listener (RMOFF_DOCKER_HACK), which is on a new port. If you connect to the banker on 9092, yous'll get the advertised.listener defined for the listener on that port (localhost). And if you lot connect to the broker on 19092, you'll get the alternative host and port: host.docker.internal:19092.

Let'south try it out (brand sure yous've restarted the banker commencement to choice up these changes):

It works! It's a Muddied HACK, simply it works 😅. Merely as importantly, we haven't broken Kafka for local (not-Docker) clients as the original 9092 listener still works:

FAQs

Couldn't I simply hack my /etc/hosts file?

• Tin you? Yes
• Should you lot? NO! 🙂

Not unless yous want your client to randomly terminate working each time yous deploy information technology on a machine that you lot forget to hack the hosts file for. This is the whole point of hostnames and DNS resolution—they are how machines know how to talk to each other instead of you hardcoding it into each car individually.

I don't accept advertised.listeners set in my server.properties

By default, it'll take the same value as the listener itself. You can validate the settings in use by checking the broker log file:

[2020-04-27 21:21:00,939] INFO KafkaConfig values:         advertised.host.name = null         advertised.listeners = PLAINTEXT://localhost:9092,RMOFF_DOCKER_HACK://host.docker.internal:19092         advertised.port = null …          listener.security.protocol.map = PLAINTEXT:PLAINTEXT,RMOFF_DOCKER_HACK:PLAINTEXT         listeners = PLAINTEXT://:9092,RMOFF_DOCKER_HACK://:19092 …        

What about advertised.host.proper noun and advertised.port?

They're deprecated. Don't use them.

But…I can telnet to the banker just fine, so surely it should but work?

Yes, yous need to exist able to reach the banker on the host and port you lot provide in your initial bootstrap connection. Every bit explained above, however, it'due south the subsequent connections to the host and port returned in the metadata that must as well be attainable from your customer machine.

Do I have to apply Python to do this?

Nope—any customer library (see this listing and GitHub) should be able to expose the metadata likewise. Hither's an case using kafkacat:

$ kafkacat -b asgard05.moffatt.me:9092 -50 Metadata for all topics (from broker 1: asgard05.moffatt.me:9092/i): 3 brokers: broker ii at asgard05.moffatt.me:19092 broker 3 at asgard05.moffatt.me:29092 broker 1 at asgard05.moffatt.me:9092 (controller)        

Yous can also utilise kafkacat from Docker, but then yous get into some funky networking implications if you lot're trying to troubleshoot something on the local network.

In conclusion…

At that place are 2 types of connection from your client to the Kafka brokers that must succeed:

1. The initial connectedness to a broker (the bootstrap). This returns metadata to the client, including a list of all the brokers in the cluster and their connection endpoints.
2. The client then connects to ane (or more than) of the brokers returned in the first step as required. If the broker has not been configured correctly, the connections will fail.

Resources

• Apache Kafka 101 grooming course
• What is the quickest way to fix advertised.listeners? Don't set information technology! Instead, use ​​Confluent Cloud and make it someone else's trouble 🙂 Sign up today and use code 60DEVADV to get $60 of additional free usage!*

Robin Moffatt is a senior developer advocate at Confluent, equally well as an Oracle Groundbreaker Administrator and ACE Director (alumnus). His career has ever involved data, from the onetime worlds of COBOL and DB2, through the worlds of Oracle and Apache™ Hadoop® and into the current earth with Kafka. His particular interests are analytics, systems compages, performance testing, and optimization.

zamorakerd1985.blogspot.com

Source: https://www.confluent.io/blog/kafka-client-cannot-connect-to-broker-on-aws-on-docker-etc

Share this post