Thriving in the no man’s land between compilers and databases

When developing new data-intensive applications, one faces a build- or-buy decision: use an existing off-the-shelf data management system (DMS) or implement a custom solution. While off-the-shelf systems offer quick results, they lack the flexibility to accommodate the changing requirements of long-term projects. Building a solution from scratch in a general-purpose programming language, however, comes with long-term development costs that may not be justified. What is lacking is a middle ground or, more precisely, a clear migration path from off-the-shelf Data Management Systems to customized applications in general-purpose programming languages. There is, in effect, a no man’s land that neither compiler nor database researchers have claimed. We believe that this problem is an opportunity for the database community to claim a stake. We need to invest effort to transfer the outcomes of data management research into fields of programming languages and compilers. The common complaint that other fields are re-inventing database techniques bears witness to the need for that knowledge transfer. In this paper, we motivate the necessity for data management techniques in general-purpose programming languages and outline a number of specific opportunities for knowledge transfer. This effort will not only cover the no man’s land but also broaden the impact of data management research.