Security should be at the heart of the design of any software project, and Redpanda is not an exception. Today, I wanna introduce the different built-in mechanisms by which you will be able to make your Redpanda cluster more secure. By following this guide, you will familiarize yourself with the available authorization and authentication methods that Redpanda support, both at a conceptual level and at a technical level through hands-on examples.
A note on scope: Let’s emphasize the word built-in in the previous paragraph. Security is a goal achieved through many different aspects, such as network configuration, organization-wide roles management, among many others. However, they are external to Redpanda so this guide will not cover them.
All the concepts described here are compatible with the current version of Kafka® and its client libraries and CLIs.
- A running redpanda node.
If you haven’t checked them out, you can follow our Getting Started guides, where you’ll find how to get started with Redpanda.
- TLS certificates.
If you have your own certificates, either self-signed or issued by a trusted Certificate Authority, you can use them for this guide. Otherwise, feel free to generate them using this script.
Mutual TLS, or 2-way TLS, is an authentication method in which the server keeps a set of trusted certificates in the form of a “truststore” file, and all clients attempting to establish a connection must present their certificate.
To enable it, set the
require_client_auth field to
true in the required listener’s configuration. For example, to enable mTLS for the “external” API listener:
redpanda:kafka_api:# The listener declaration. `name` can have any value.- name: internaladdress: <private IP>port: 9092advertised_kafka_api:# The advertised listeners. `name` should match the name of a declared listener.- name: internaladdress: localhostport: 9092kafka_api_tls:# The listener's TLS config. `name` must match the corresponding listener's name.- name: internalenabled: truerequire_client_auth: true # <- This needs to be enabled!cert_file: <path to PEM-formatted cert file>key_file: <path to PEM-formatted key file>truststore_file: <path to PEM-formatted truststore file>
rpk’s side, you can use the
--tls-truststore flags to have it authenticate and establish a TLS connection:
$ rpk topic create test-topic \--tls-key <path to PEM-formatted key file> \--tls-cert <path to PEM-formatted cert file> \--tls-truststore <path to PEM-formatted truststore file>Created topic 'test-topic'.You may check its config withrpk topic describe 'test-topic'
rpkdefaults to connecting to
localhost:9092. If you’re connecting to a remote broker, you will need to pass
--brokers <node IP>:<kafka API port>
For an in-depth guide about enabling TLS and mutual TLS in redpanda check out our guide on the subject.
Redpanda also supports client authentication via SASL/SCRAM (that is, the Simple Authentication and Security Layer protocol, using the Salted Challenge Response Authentication Mechanism), which is based on usernames & passwords.
To enable it, set
true in the configuration file, as well as list at least one “superuser” which after created will have permissions for all operations on the clusters.
redpandaenable_sasl: truesuperusers:- admin# The rest of the config...
Then, set a password for the user by running the following command. Replace
<password> for a password of your choice.
$ rpk acl user create \--new-username admin \--new-password <password> \--api-urls localhost:9644Created user 'admin'
Refer to the next section, Managing users, for more details on
rpk acl user.
You’re now able to use the created identity to interact with the Kafka API. For example:
Note: If you still have the TLS config from the previous section, you’ll also need to pass the TLS flags.
$ rpk topic describe test-topic \--user admin \--password <password> \--sasl-mechanism SCRAM-SHA-256 \--brokers localhost:9092Name test-topicInternal falseCleanup policy deleteConfig:Name Value Read-only Sensitivepartition_count 1 false falsereplication_factor 1 false falsecleanup.policy delete false falsePartitions 1 - 1 out of 1Partition Leader Replicas In-Sync Replicas High Watermark0 1   0
While having a superuser is a must to get started, it’s not really a good idea to either share the superuser’s credentials everywhere or to make everyone a superuser.
You can create, delete and list users with
rpk acl user.
Creating a user
$ rpk acl user create \--new-username Jack \--new-password <password> \--api-urls <comma-separated URLs of the nodes' admin APIs>Created user 'Jack'
Deleting a user
$ rpk acl user delete \--delete-username Jack \--api-urls <comma-separated URLs of the nodes' admin APIs>Deleted user 'Jack'
$ rpk acl user list \--api-urls <comma-separated URLs of the nodes' admin APIs>USERNAMEMichaelJimPamDwightKelly
While authentication is about making sure that whatever client connects to your cluster is trusted — or, to put it in other words, who they say they are —, authorization is about making sure that each client has access to exactly the data it should.
ACLs is the main mechanism supported by Redpanda to manage user permissions.
At a high level, ACLs specify what users can or cannot do. They can be managed with rpk, which supports creating, deleting or listing ACLs through
rpk acl create,
rpk acl delete and
rpk acl list, respectively.
In ACL terminology, the entities accessing the resources are called ”principals”. “User” and “Group” (as in “consumer group”) are 2 different types of principals.
Principals are allowed or denied access to resources, and you can also specify from which hosts they will be allowed or denied access.
This “access” is represented as ”operations”, such as “read”, “write”, or “describe”, and said operations can be performed on resources, such as a topic. You can further filter the resources by name,
Here’s a reference of all the concepts and their supported values.
rpk acl quick reference
Please note that you may have to add the
--tls-truststore flags, as well as
--sasl-mechanism if mTLS and SASL/ SCRAM are enabled (recommended). They are ommitted here for brevity.
- Create an ACL allowing a user to perform all operations from all hosts to a topic named “pings”:
$ rpk acl create \--allow-principal 'User:Charlie' \--allow-hosts '*' \--operation all \--resource topic \--resource-name pingsCreated ACL for principal 'User:Charlie' with host '*', operation 'All' and permission 'Allow'
- Create an ACL to deny all users to alter the “cluster” resource from 2 hosts:
$ rpk acl create \--deny-principal 'User:*' \--deny-host 192.168.98.74,10.235.78.12 \--operation alter \--resource clusterCreated ACL for principal 'User:*' with host '10.235.78.12', operation 'Alter' and permission 'Deny'Created ACL for principal 'User:*' with host '192.168.98.74', operation 'Alter' and permission 'Deny'
As you can see, a single command may result in multiple ACLs being created. You an also create ACLs denying and allowing principals access:
$ rpk acl create \--resource cluster \--allow-host 22.214.171.124 \--deny-host 126.96.36.199 \--deny-principal 'User:david' \--allow-principal 'User:Alex' \--allow-principal 'User:Ben'Created ACL for principal 'User:david' with host '188.8.131.52', operation 'All' and permission 'Deny'Created ACL for principal 'User:Alex' with host '184.108.40.206', operation 'All' and permission 'Allow'Created ACL for principal 'User:Ben' with host '220.127.116.11', operation 'All' and permission 'Allow'
rpk acl list allows you to list and filter existing ACLs.
- List all ACLs:
$ rpk acl listPRINCIPAL HOST OPERATION PERMISSION TYPE RESOURCE TYPE RESOURCE NAMEUser:Charlie * All Allow Topic pingsUser:* 192.168.98.74 Alter Deny Cluster kafka-clusterUser:david 18.104.22.168 All Deny Cluster kafka-clusterUser:Ben 22.214.171.124 All Allow Cluster kafka-clusterUser:* 10.235.78.12 Alter Deny Cluster kafka-clusterUser:Alex 126.96.36.199 All Allow Cluster kafka-cluster
- List all ACLs for a specific principal:
$ rpk acl list --principal 'User:Ben'PRINCIPAL HOST OPERATION PERMISSION TYPE RESOURCE TYPE RESOURCE NAMEUser:Ben 188.8.131.52 All Allow Cluster kafka-cluster
- List all ACLs that deny principals to alter a resource:
$ rpk acl list \--permission deny \--operation alterPRINCIPAL HOST OPERATION PERMISSION TYPE RESOURCE TYPE RESOURCE NAMEUser:* 192.168.98.74 Alter Deny Cluster kafka-clusterUser:* 10.235.78.12 Alter Deny Cluster kafka-cluster
rpk acl delete allows you to delete ACLs. It’s important to note, however, that wildcard values such as
any for operations and permissions, as well as
* for hosts and resources may result in the deletion filters matching more than one ACL.
- Delete all ACLs for a specific user targeting a specific resource:
$ rpk acl delete --deny-principal 'User:david' --resource clusterDELETED PRINCIPAL HOST OPERATION PERMISSION TYPE RESOURCE TYPE RESOURCE NAME ERROR MESSAGEyes User:david 184.108.40.206 All Deny Cluster kafka-cluster None
- Delete all ACLs granting a principal permissions to all operations from a host:
$ rpk acl delete \--allow-principal 'User:Ben' \--allow-host 220.127.116.11 \--operation all