Towards Designing Future-Proof Data Processing Systems

Data processing systems find themselves crushed between two moving tectonic plates: the usage plate driven by the system’s users and their requirements; and the environment plate driven by various technological changes. We argue that the existing status quo of constantly adapting and thus bloating the system’s implementation is simply unsustainable in the long run. We further argue that now is the right time to take a step back and establish the foundations of future-proof data processing systems that can easily adapt to different workloads and input formats, and that can run efficiently in any type of environment, today and in the future. With our paper, we analyze and learn from prior attempts, identify key design principles, and present our vision on how to design such systems.