TLS configuration for Redpanda and rpk

Prerequisites

In order to set up TLS on Redpanda and rpk, you’ll need a certificate and a key. If you require client authentication, you’ll also need a truststore file containing the client certificates.

1. TLS configuration in Redpanda

Redpanda supports 2 levels of TLS:

Basic TLS: Encrypt all incoming requests using a certificate and a key file.

TLS with client authentication: In addition to encrypting all incoming requests, require that the clients include their certificate, checking that it’s trusted.

These can be set up for both the Kafka®-compatible API, and for the internal RPC API, which is used for communication between all redpanda nodes.

redpanda:

  kafka_api:
  - name: default
    address: <private IP>
    port: 9092

  advertised_kafka_api:
  - name: default
    address: <public IP or domain>
    port: 9092

  kafka_api_tls:
  - name: default
    enabled: true
    require_client_auth: true
    cert_file: <path to PEM-formatted cert file>
    key_file: <path to PEM-formatted key file>
    truststore_file: <path to PEM-formatted truststore file>

  rpc_server_tls:
    enabled: true
    require_client_auth: true
    cert_file: <path to PEM-formatted cert file>
    key_file: <path to PEM-formatted cert file>
    truststore_file: <path to PEM-formatted cert file>

There are many important things going on here, so let’s unpack them.

First off, we’re declaring a listener for the broker.

kafka_api:
  - name: devault
    address: <private IP>
    port: 9092

name can be any string you like. The important part is that it matches the name in redpanda.advertised_kafka_api and redpanda.kafka_api_tls.

address is any IP that the broker can bind to.

Then, we’re setting the advertised - or “published” - address.

advertised_kafka_api:
  - name: default
    address: <public IP or domain>
    port: 9092

If you’re trying out this guide locally (i.e. in the same machine), address can be the same as the one we set for the listener in redpanda.kafka_api. However, for a production deployment or if you wanna interact with the broker remotely, it can be the machine’s public IP or its domain, if one exists.

After that, there’s the TLS config for the listener we declared:

# A list of TLS configurations for the Kafka API listeners.
  kafka_api_tls:
    # The name of the specific listener this TLS to which this config
    # will be applied. The names must match those in kafka_api.
  - name: default
    # Whether to enable TLS for the Kafka API.
    enabled: true
    # Require client authentication
    require_client_auth: false
    # The path to the server certificate PEM file.
    cert_file: <path to PEM-formatted cert file>
    # The path to the server key PEM file
    key_file: <path to PEM-formatted key file>
    # The path to the truststore PEM file. Only required if client authentication
    # is enabled.
    truststore_file: <path to PEM-formatted truststore file>

Notice that even though you can configure both the Kafka API TLS (redpanda.kafka_api_tls) & RPC API TLS (redpanda.rpc_server_tls) with the same certificate, it’s highly recommended that you use different ones.

If you set any of the APIs’ require_client_auth to true, you’ll need to set the respective truststore_file, and the PEM truststore file it points to should contain the clients’ certificates to be able to authenticate them.

Besides manually editing the configuration file to configure TLS, you can also use rpk to do it (you may need to run these as root if the config file is in a restricted file):

rpk config set redpanda '
kafka_api_tls:
- name: default
  enabled: true
  require_client_auth: true
  cert_file: <path to PEM-formatted cert file>
  key_file: <path to PEM-formatted cert file>
  truststore_file: <path to PEM-formatted cert file>

rpc_server_tls:
  enabled: true
  require_client_auth: true
  cert_file: <path to PEM-formatted cert file>
  key_file: <path to PEM-formatted cert file>
  truststore_file: <path to PEM-formatted cert file>

' -- format yaml

After changing the configuration you’ll need to restart the Redpanda broker (e.g. by running sudo systemctl restart redpanda, if you’re using systemd).

2. TLS configuration in rpk

To be able to use rpk topic with a broker with TLS enabled, you’ll need to update the rpk config in your config file.

rpk:

  tls:
    cert_file: <path to PEM-formatted cert file>
    key_file: <path to PEM-formatted key file>
    truststore_file: <path to PEM-formatted truststore file>

If you enabled TLS on the broker but set redpanda.kafka_api_tls.require_client_auth to false, you’ll only need to set rpk.tls.truststore_file (note that the truststore must be a PEM-formatted file containing the certificates of the brokers to which you’ll make requests). Else you will also need to set rpk.tls.cert_file and rpk.tls.key_file.

It’s strongly encouraged that the certificate used for rpk and other clients be different from the ones used in the brokers.

Changing the configuration in the rpk section doesn’t require restarting Redpanda, so you may test your settings right away with any of the rpk topic subcommands, such as

rpk topic list --brokers <broker IP>:<port>

3. Configuring your current Kafka® client

The best news is, since Redpanda is 100% API-compatible with Kafka®, you probably won’t need to modify your TLS settings unless you changed the location or format of any of the files involved.

security.protocol=SSL
ssl.truststore.location=<path to truststore file>
ssl.truststore.password=<truststore password>
ssl.keystore.location=<path to keystore file>
ssl.keystore.password=<keystore password>
ssl.key.password=<key password>

Summary

Redpanda leverages TLS to ensure encrypted communication, both internally on its RPC API, as well as externally on its Kafka®-compatible API listeners. Additionally, you can require client authentication via mutual TLS to make sure only known clients are able to make requests.

You can configure rpk and any other clients to be able to make encrypted requests to a remote node’s API.

Don’t forget to follow @VectorizedIO on Twitter, or join our mailing list, and stay tuned for more step-by-step guides from our team!