When connecting to a Kafka instance hosted in a public cloud, it is typically configured to allow remote connectively. However, when connecting to a Kafka cluster hosted on a non-public network, you can run into connectivity issues since Kafka publishes connection information for brokers using their local network information. Even if the hosts and ports are open publicly to connect, the remote connection will fail because Kafka will advertise the host as something like "KafkaTest01" not the external host or IP. To get around that, you have to specify an alternate listener used for remote connectivity.
This is a simple example:
listeners=LOCAL://:9092,PUBLIC://:19092
advertised.listeners=LOCAL://:9092,PUBLIC://123.45.6.54:19092
inter.broker.listener.name=LOCAL
listener.security.protocol.map=LOCAL:PLAINTEXT,PUBLIC:SSL
Parameter explanation:
- listeners creates 2 listener ports, assigning one to the listener LOCAL and one to PUBLIC. Which port the application connects to will determine which advertised listener will be returned.
- advertised.listeners identifies the 2 advertised listeners that will be published. LOCAL will return the local hostname and PUBLIC will return the external IP (could be a hostname if configured).
- inter.broker.listener.name sets which of the listeners will be used for broker to broker communication. You typically do not want to use the PUBLIC connections for this.
- listener.security.protocol.map maps the specific security to be used when connecting. As shown, local communication is PLAINTEXT and PUBLIC is SSL.
You would need to configure this consistently on all brokers in the cluster.