IndiLog: Bridging Scalability and Performance in Stateful Serverless Computing with Shared Logs

State management has long been a challenge for serverless applications. Owing to their failure resilience and consistency guarantees, distributed shared logs have been recently proposed as a promising storage substrate enabling stateful serverless applications. We show that, unfortunately, state-of-the-art sacrifices compute tier scalability for log access performance, a particularly undesirable exchange for the dynamic serverless environment. The culprit is the log indexing architecture, namely relying on complete local indexes colocated with serverless functions. This design prevents efficient scaling and even risks out-of-memory errors. IndiLog is a novel distributed indexing architecture enabling stateful serverless applications to efficiently access a distributed shared log for state management without impeding compute tier scalability. IndiLog uses a combination of local, size-bounded indexes designed to capture the expected locality patterns alongside a sharded and balanced index tier which tackles the challenges of supporting log sub-streams and bounded reads. IndiLog bests or matches Boki, a state-of-the-art distributed shared log, over various index hit rates, workload concurrency and compute tier scaling sizes.