In order to set up TLS on Redpanda and rpk, you’ll need a certificate and a key. If you require client authentication, you’ll also need a truststore file containing the client certificates.
Redpanda supports 2 levels of TLS:
Basic TLS: Encrypt all incoming requests using a certificate and a key file.
TLS with client authentication: In addition to encrypting all incoming requests, require that the clients include their certificate, checking that it’s trusted.
These can be set up for both the Kafka®-compatible API, and for the internal RPC API, which is used for communication between all redpanda nodes.
redpanda:kafka_api:- name: defaultaddress: <private IP>port: 9092advertised_kafka_api:- name: defaultaddress: <public IP or domain>port: 9092kafka_api_tls:- name: defaultenabled: truerequire_client_auth: truecert_file: <path to PEM-formatted cert file>key_file: <path to PEM-formatted key file>truststore_file: <path to PEM-formatted truststore file>rpc_server_tls:enabled: truerequire_client_auth: truecert_file: <path to PEM-formatted cert file>key_file: <path to PEM-formatted cert file>truststore_file: <path to PEM-formatted cert file>
There are many important things going on here, so let’s unpack them.
First off, we’re declaring a listener for the broker.
kafka_api:- name: devaultaddress: <private IP>port: 9092
name can be any string you like. The important part is that it matches the
address is any IP that the broker can bind to.
Then, we’re setting the advertised - or “published” - address.
advertised_kafka_api:- name: defaultaddress: <public IP or domain>port: 9092
If you’re trying out this guide locally (i.e. in the same machine),
address can be the same as the one we set for the listener in
redpanda.kafka_api. However, for a production deployment or if you wanna interact with the broker remotely, it can be the machine’s public IP or its domain, if one exists.
After that, there’s the TLS config for the listener we declared:
# A list of TLS configurations for the Kafka API listeners.kafka_api_tls:# The name of the specific listener this TLS to which this config# will be applied. The names must match those in kafka_api.- name: default# Whether to enable TLS for the Kafka API.enabled: true# Require client authenticationrequire_client_auth: false# The path to the server certificate PEM file.cert_file: <path to PEM-formatted cert file># The path to the server key PEM filekey_file: <path to PEM-formatted key file># The path to the truststore PEM file. Only required if client authentication# is enabled.truststore_file: <path to PEM-formatted truststore file>
Notice that even though you can configure both the Kafka API TLS (
redpanda.kafka_api_tls) & RPC API TLS (
redpanda.rpc_server_tls) with the same certificate, it’s highly recommended that you use different ones.
If you set any of the APIs’
true, you’ll need to set the respective
truststore_file, and the PEM truststore file it points to should contain the clients’ certificates to be able to authenticate them.
Besides manually editing the configuration file to configure TLS, you can also use rpk to do it (you may need to run these as root if the config file is in a restricted file):
rpk config set redpanda 'kafka_api_tls:- name: defaultenabled: truerequire_client_auth: truecert_file: <path to PEM-formatted cert file>key_file: <path to PEM-formatted cert file>truststore_file: <path to PEM-formatted cert file>rpc_server_tls:enabled: truerequire_client_auth: truecert_file: <path to PEM-formatted cert file>key_file: <path to PEM-formatted cert file>truststore_file: <path to PEM-formatted cert file>' -- format yaml
After changing the configuration you’ll need to restart the Redpanda broker (e.g. by running
sudo systemctl restart redpanda, if you’re using systemd).
To be able to use
rpk topic with a broker with TLS enabled, you’ll need to update the rpk config in your config file.
rpk:tls:cert_file: <path to PEM-formatted cert file>key_file: <path to PEM-formatted key file>truststore_file: <path to PEM-formatted truststore file>
If you enabled TLS on the broker but set
false, you’ll only need to set
rpk.tls.truststore_file (note that the truststore must be a PEM-formatted file containing the certificates of the brokers to which you’ll make requests). Else you will also need to set
It’s strongly encouraged that the certificate used for
rpk and other clients be different from the ones used in the brokers.
Changing the configuration in the
rpk section doesn’t require restarting Redpanda, so you may test your settings right away with any of the
rpk topic subcommands, such as
rpk topic list --brokers <broker IP>:<port>
The best news is, since Redpanda is 100% API-compatible with Kafka®, you probably won’t need to modify your TLS settings unless you changed the location or format of any of the files involved.
security.protocol=SSLssl.truststore.location=<path to truststore file>ssl.truststore.password=<truststore password>ssl.keystore.location=<path to keystore file>ssl.keystore.password=<keystore password>ssl.key.password=<key password>
Redpanda leverages TLS to ensure encrypted communication, both internally on its RPC API, as well as externally on its Kafka®-compatible API listeners. Additionally, you can require client authentication via mutual TLS to make sure only known clients are able to make requests.
You can configure
rpk and any other clients to be able to make encrypted requests to a remote node’s API.
Don’t forget to follow @VectorizedIO on Twitter, or join our mailing list, and stay tuned for more step-by-step guides from our team!