On the Impact of Network Transport Protocols on Leader-Based Consensus Communication

State Machine Replication (SMR) allows implementation of fault-tolerant systems and secure critical infrastructure. The advent of cryptocurrencies has increased research toward more efficient and performant consensus and SMR systems. Still, current performance is not satisfactory for all envisioned use cases. We focus on heterogeneous SMR deployments, that cannot benefit from e.g., Remote Direct Memory Access (RDMA) in pure data center setups, but offer more predictable network conditions than a set of globally distributed virtual machines. In this space, tuning of network transport protocols allows for optimization. In this paper, we analyze secure channel and network stack interdependencies in context of leader-based consensus. We experimentally quantify the impact of four transport protocols and two secure channel implementations on a HotStuff deployment. Our results show, that delays of a single processing layer often impact all layers above, and typical optimizations such as command batching or pipelining act as amplifiers. Except for edge cases, TCP performs best but offers further optimization potential through configuration of retransmission behavior in lossy scenarios. For large loss probabilities above 2%, transport protocol configuration is not sufficient to confine significant replication performance penalties. We demonstrate that tuning of the transport protocol building block opens a novel optimization space to a class of leader-based consensus algorithm deployments.