TLS configuration for Redpanda and rpk

Prerequisites

In order to set up TLS on Redpanda and rpk, you’ll need a certificate and a key. If you require client authentication, you’ll also need a truststore file containing the client certificates.

1. TLS configuration in Redpanda

Redpanda supports 2 levels of TLS:

Basic TLS: Encrypt all incoming requests using a certificate and a key file.

TLS with client authentication: In addition to encrypting all incoming requests, require that the clients include their certificate, checking that it’s trusted.

These can be set up for both the Kafka®-compatible API, and for the internal RPC API, which is used between all redpanda nodes.

redpanda:

  kafka_api_tls:
    enabled: true
    require_client_auth: true
    cert_file: <path to PEM-formatted cert file>
    key_file: <path to PEM-formatted key file>
    truststore_file: <path to PEM-formatted truststore file>

  rpc_server_tls:
    enabled: true
    require_client_auth: true
    cert_file: <path to PEM-formatted cert file>
    key_file: <path to PEM-formatted cert file>
    truststore_file: <path to PEM-formatted cert file>

Notice that even though you can configure both the Kafka API & RPC API with the same certificate, it’s highly recommended that you use different ones.

If you set any of the APIs’ require_client_auth to true, you’ll need to set the respective truststore_file, and the PEM truststore file it points to should contain the clients’ certificates to be able to authenticate them.

Besides manually editing the configuration file to configure TLS, you can also use rpk to do it (you may need to run these as root if the config file is in a restricted file):

rpk config set redpanda '
kafka_api_tls:
  enabled: true
  require_client_auth: true
  cert_file: <path to PEM-formatted cert file>
  key_file: <path to PEM-formatted cert file>
  truststore_file: <path to PEM-formatted cert file>

rpc_server_tls:
  enabled: true
  require_client_auth: true
  cert_file: <path to PEM-formatted cert file>
  key_file: <path to PEM-formatted cert file>
  truststore_file: <path to PEM-formatted cert file>

' -- format yaml

After changing the configuration you’ll need to restart the Redpanda broker (e.g. by running sudo systemctl restart redpanda, if you’re using systemd).

2. TLS configuration in rpk

If you’re running rpk in the same machine as a broker with TLS enabled, rpk can use the broker’s TLS configuration, meaning you don’t need to change the rpk section in your configuration file (if the broker’s configuration file is in a non-default path, don’t forget to add --config to rpk status, rpk api, etc).

Otherwise, if you configured and enabled TLS for the Redpanda API (redpanda.kafka_api_tls), and you’re running rpk commands on a different machine, you’ll need to configure TLS on rpk to be able to interact with the Kafka® API.

rpk:

  tls:
    cert_file: <path to PEM-formatted cert file>
    key_file: <path to PEM-formatted key file>
    truststore_file: <path to PEM-formatted truststore file>

If you enabled TLS on the broker but set redpanda.kafka_api_tls.require_client_auth to false, you’ll only need to set rpk.tls.truststore_file (note that the truststore must be a PEM-formatted file containing the certificates of the brokers to which you’ll make requests). Else you will also need to set rpk.tls.cert_file and rpk.tls.key_file.

It’s strongly encouraged that the certificate used for rpk and other clients be different from the ones used in the brokers.

Changing the configuration in the rpk section doesn’t require restarting Redpanda, so you may test your settings right away with any of the rpk api subcommands, such as

rpk api topic describe <topic name> --brokers <broker IP>:<port>

3. Configuring your current Kafka® client

The best news is, since Redpanda is 100% API-compatible with Kafka®, you probably won’t need to modify your TLS settings unless you changed the location or format of any of the files involved.

security.protocol=SSL
ssl.truststore.location=<path to truststore file>
ssl.truststore.password=<truststore password>
ssl.keystore.location=<path to keystore file>
ssl.keystore.password=<keystore password>
ssl.key.password=<key password>

Summary

Redpanda leverages TLS to ensure encrypted communication, both internally on its RPC API, as well as externally on its Kafka®-compatible API. Additionally, you can require client authentication via 2-way TLS to make sure only known clients are able to make requests.

You can use the local Redpanda broker’s configuration to interact with the cluster without additional steps. Alternatively, you can edit the rpk section on your configuration file to be able to make requests to a remote node’s API.

Don’t forget to follow @VectorizedIO on Twitter, or join our mailing list, and stay tuned for more step-by-step guides from our team!