We are building a real-time streaming engine for modern applications - from the enterprise to the solo dev prototyping a react application on her laptop. We go beyond the Kafka protocol, into the future of streaming with inline WASM transforms and geo-replicated hierarchical storage. A new platform that scales with you from the smallest projects to petabytes of data distributed across the globe.
As easy to run as
nginx. No dependencies. Ability to flush to disk with
Leverage a huge and active ecosystem. It must be fast, really fast. With this wishlist
in mind, I wrote the first line of code of what eventually became
redpanda. It was January 7th,
2019 and I was still living in Miami before relocating to San Francisco. I hadn’t had as much
fun hacking on anything since the initial prototype of my previous project & company concord.io
and… it was equally all-consuming.
Here we are today, 22 months later. A team one dreams to be part of and a product we feel proud to share with you. Ready to be put through the paces in even more ways that we could have anticipated, whether embedding Redpanda in a security appliance, or using it as part of your new NodeJS application because it’s so simple to use. Whoever you are, welcome! We are excited to have you in our community.
The project is released under the Source Available License - BSL - similar to what our friends at CockroachDB have done. We try to make this clear in the license, but worth reiterating here. Our intention is to deter cloud providers from offering our work as a service. For 99.999% of you, restrictions will not apply - welcome to our community!
There will be enterprise, pay-only features that will be obvious, since to turn them on you have to
enterprise section of the configuration.
The simplest thing you can do is run in Docker. Follow the tutorial here. But for the truly impatient, here is the executive summary:
$ rpk container start -n 3 NODE ID ADDRESS CONFIG 0 172.24.1.2:9092 /home/david/.rpk/cluster/node-0/conf/redpanda.yaml 1 172.24.1.4:9092 /home/david/.rpk/cluster/node-1/conf/redpanda.yaml 2 172.24.1.3:9092 /home/david/.rpk/cluster/node-2/conf/redpanda.yaml Cluster started! You may use 'rpk api' to interact with the cluster. E.g: rpk api status
It says we can check our cluster with
rpk api status Let’s try that!
$ rpk api status Redpanda Cluster Status 0 (172.24.1.2:9092) (No partitions) 1 (172.24.1.3:9092) (No partitions) 2 (172.24.1.4:9092) (No partitions)
All of the
rpk api subcommands will detect the local cluster and use its addresses, so you don’t have to configure anything or keep track of IPs and ports.
For example, you can run
rpk api topic create and it will work!
$ rpk api topic create -p 6 -r 3 new-topic Created topic 'new-topic'. Partitions: 6, replicas: 3, cleanup policy: 'delete'
This decision comes after almost a year of thinking and mentorship with a very large group of experts, OSS enthusiasts, lawyers and quiet time thinking. Special thanks to Peter Mattis from CockroachDB for sharing his experience with BSL which ultimately made us feel comfortable with our decision to also choose BSL. Adam Jacob for sharing his experiences with Chef, his never-ending expertise around licensing and for taking the time to walk me through business models with me. Thanks to Ajay Kulkarni from TimescaleDB for sharing his wealth of knowledge and experience building a community. Thanks to Megan Gill at MongoDB for helping me better understand OSS in general, and to Gaurav Gupta now at LSVP for helping me understand Elastic a bit better and how the OSS+Source Available has matured in the last decade. We are better because of your advice, I am forever thankful.