Jump Trading Drives Faster Insights at Scale with Redpanda (2024)

Jump Trading is a proprietary trading firm with a focus on algorithmic and high-frequency trading strategies. To be successful, we tap into real-time insights, so streaming data infrastructure is critical to our business.

Over the past 20 years, Jump Trading iterated through several messaging and streaming data platforms. Our primary use cases are telemetry and log files. We have evaluated and deployed most of the best-of-breed on-premises and cloud messaging systems. In doing so, we found all of them lacking due to some combination of:

• Consistency of performance (tail latency, tolerance of slow consumers, etc.)
• Access to low-level metrics to debug performance issues (especially in cloud products)
• Reliability of message delivery
• Cost-effectiveness
• Compatibility with other products

As Jump Trading grew, our workloads required us to process millions of messages per second, and it became clear that we needed a new streaming data platform that could support this mission-critical aspect of our infrastructure.

In Search of a Streaming Data Platform

When looking for a new solution, our starting point was Kafka. We really like the Kafka protocol because it is an industry standard — lots of data tools speak Kafka, making it much easier to engineer pipelines.

What we did not like was the idea of a Java application in such a mission-critical role. We did not want memory allocation controlled by a Java Virtual Machine (JVM), we wanted memory tailored to the application itself, and we found that Redpanda satisfied that requirement by its implementation in C++. Plus, Jump Trading is more focused on C++ software architecture, so the Redpanda codebase was in line with our core software competencies.

With Redpanda, we saw we could get Kafka compatibility along with stability, ease of maintenance, and platform simplicity. Plus, there were no ZooKeepers to manage. By eliminating the Java dependencies in Kafka, Redpanda gave us less complexity and superior performance on like-for-like hardware.

The simple architecture of Redpanda as a single binary and no JVM dependencies created a better experience for our engineers. And all this plus no data loss due to the Raft-native design. The choice became a no brainer.

Today we have implementations of Redpanda on bare metal, in containerized environments like Podman and Kubernetes, and the Redpanda fully managed cloud service. The communication and interactions with Redpanda’s support and engineering team was a key factor, particularly in the early days of product adoption, that kept us on board. We’d push the envelope in earlier iterations of the product, trigger bugs, and the Redpanda engineers would fix them seemingly overnight.

It’s worth spending a little more time delving into the three key benefits that have made Redpanda so popular within Jump Trading: simplicity, performance and open source.

Simplicity

We really appreciate Redpanda’s single binary installation, upgrading with no downtime, and the maintenance cycle. The lean architecture is less complex than Apache Kafka to deploy and scale because we do not need to install, monitor and maintain ZooKeeper, nor do we need to deal with JVM compatibility issues or JVM upgrades.

We also like the built-in Prometheus exporter that gives us visibility into performance internals, helping us steer operations and system load configuration. And, from a getting started perspective, we appreciate the documentation and blog posts available on the website, as well as the courses at Redpanda University.

It’s essential for our engineers to have solutions with open versus proprietary standards, because it translates to less time learning new skills. That’s why we were drawn to Redpanda’s Kafka-compatible API. It gives Jump Trading access to the entire existing open source Kafka ecosystem. When companies build on open source standards, they are leveraging the foundational body of content publicly available, accelerating engineering time to productivity.

And because we are not locked into proprietary tech, we have the flexibility to work across different environments and platforms without fear of vendor lock-in. In theory, we could swap Redpanda for any other compatible Kafka platform if our needs changed. In the same manner, we can also use Redpanda with any Kafka-compatible tools like the Amazon Kinesis Client Library.

In addition to Kafka compliance, it’s great to see Redpanda’s S3-compatible storage feature for cost-effective data retention. This is a huge win because there is an entire ecosystem of tooling out there already for S3.

Performance

One of Jump Trading’s largest workloads is our mission-critical telemetry pipeline. It is complex, data-intensive and very producer heavy. We have tens of thousands of nodes publishing data such as system measurements, market information and other varied bits of data. Additionally, we have high-performance nodes consuming and processing this data in real time. Our many-to-few data pipeline is fully supported and highly efficient with Redpanda.

Because of Redpanda’s performance-engineered architecture, we see little jitter in terms of our p95 and p99 latencies. We have very few fat or angry tails. Redpanda’s C++ codebase means we can load it up to line rate data loads and know that all the packets and messages will get delivered within an optimal distribution of latencies. That is huge, especially for a financial company that is latency sensitive.

In an industry where you tend to have bursty traffic, it’s invaluable to have a message bus that doesn’t have fat tails. And with Redpanda, we are seeing really low latencies: 50 milliseconds for our p95 and 150 milliseconds for p99.

Open Source

Redpanda is a source-available platform and this is in line with the values of Jump Trading. We believe in open source software development and actively support many open source projects. Having the codebase available for inspection allows us to understand it in depth, assist in troubleshooting when possible and be a partner in the growth of the product.

What’s Next?

We started with a self-hosted deployment of Redpanda on bare metal, and then added containerized deployments. Now we are expanding into fully managed, dedicated cloud instances. We have been running critical workloads on Redpanda for two years, reliably shipping billions of messages daily.

Jump Trading is a leader in global financial markets, and our streaming data infrastructure can make or break our business. With Redpanda, it’s not just meeting our requirements, but taking us to the next level.