Prerequisites
In order to set up TLS on Redpanda and rpk, you’ll need a certificate and a key. If you require client authentication, you’ll also need a truststore file containing the client certificates.
1. TLS configuration in Redpanda
Redpanda supports 2 levels of TLS:
Basic TLS: Encrypt all incoming requests using a certificate and a key file.
TLS with client authentication: In addition to encrypting all incoming requests, require that the clients include their certificate, checking that it’s trusted.
These can be set up for both the Kafka®-compatible API, and for the internal RPC API, which is used between all redpanda nodes.
redpanda:
kafka_api_tls:
enabled: true
require_client_auth: true
cert_file: <path to PEM-formatted cert file>
key_file: <path to PEM-formatted key file>
truststore_file: <path to PEM-formatted truststore file>
rpc_server_tls:
enabled: true
require_client_auth: true
cert_file: <path to PEM-formatted cert file>
key_file: <path to PEM-formatted cert file>
truststore_file: <path to PEM-formatted cert file>
Notice that even though you can configure both the Kafka API & RPC API with the same certificate, it’s highly recommended that you use different ones.
If you set any of the APIs’ require_client_auth
to true
, you’ll need to set the respective truststore_file
, and the PEM truststore file it points to should contain the clients’ certificates to be able to authenticate them.
Besides manually editing the configuration file to configure TLS, you can also use rpk to do it (you may need to run these as root if the config file is in a restricted file):
rpk config set redpanda '
kafka_api_tls:
enabled: true
require_client_auth: true
cert_file: <path to PEM-formatted cert file>
key_file: <path to PEM-formatted cert file>
truststore_file: <path to PEM-formatted cert file>
rpc_server_tls:
enabled: true
require_client_auth: true
cert_file: <path to PEM-formatted cert file>
key_file: <path to PEM-formatted cert file>
truststore_file: <path to PEM-formatted cert file>
' -- format yaml
After changing the configuration you’ll need to restart the Redpanda broker (e.g. by running sudo systemctl restart redpanda
, if you’re using systemd).
2. TLS configuration in rpk
If you’re running rpk in the same machine as a broker with TLS enabled, rpk can use the broker’s TLS configuration, meaning you don’t need to change the rpk
section in your configuration file (if the broker’s configuration file is in a non-default path, don’t forget to add --config
to rpk status
, rpk api
, etc).
Otherwise, if you configured and enabled TLS for the Redpanda API (redpanda.kafka_api_tls
), and you’re running rpk commands on a different machine, you’ll need to configure TLS on rpk to be able to interact with the Kafka® API.
rpk:
tls:
cert_file: <path to PEM-formatted cert file>
key_file: <path to PEM-formatted key file>
truststore_file: <path to PEM-formatted truststore file>
If you enabled TLS on the broker but set redpanda.kafka_api_tls.require_client_auth
to false
, you’ll only need to set rpk.tls.truststore_file
(note that the truststore must be a PEM-formatted file containing the certificates of the brokers to which you’ll make requests). Else you will also need to set rpk.tls.cert_file
and rpk.tls.key_file
.
It’s strongly encouraged that the certificate used for rpk and other clients be different from the ones used in the brokers.
Changing the configuration in the rpk
section doesn’t require restarting Redpanda, so you may test your settings right away with any of the rpk api
subcommands, such as
rpk api topic describe <topic name> --brokers <broker IP>:<port>
3. Configuring your current Kafka® client
The best news is, since Redpanda is 100% API-compatible with Kafka®, you probably won’t need to modify your TLS settings unless you changed the location or format of any of the files involved.
security.protocol=SSL
ssl.truststore.location=<path to truststore file>
ssl.truststore.password=<truststore password>
ssl.keystore.location=<path to keystore file>
ssl.keystore.password=<keystore password>
ssl.key.password=<key password>
Summary
Redpanda leverages TLS to ensure encrypted communication, both internally on its RPC API, as well as externally on its Kafka®-compatible API. Additionally, you can require client authentication via 2-way TLS to make sure only known clients are able to make requests.
You can use the local Redpanda broker’s configuration to interact with the cluster without additional steps. Alternatively, you can edit the rpk section on your configuration file to be able to make requests to a remote node’s API.
Don’t forget to follow @VectorizedIO on Twitter, or join our mailing list, and stay tuned for more step-by-step guides from our team!