Improving distributed memory applications testing by message perturbation

We present initial work on perturbation techniques that cause the manifestation of timing-related bugs in distributed memory Message Passing Interface (MPI)-based applications. These techniques improve the coverage of possible message orderings in MPI applications that rely on nondeterministic point-to-point communication and work with small processor counts to alleviate the need to test at larger scales. Using carefully designed model problems, we show that these techniques aid testing for problems that are often not easily reproduced when running on small fractions of the machine. Our perturbation layer, JITTERBUG, builds on P^N MPI, an extension of the MPI profiling interface that supports multiple layers of profiling libraries. We discuss how JITTERBUG complements existing MPI checking tools through the P^N MPI framework. We present opportunities to build additional tools that statically analyze and directly transform the source code to support testing and debugging MPI applications at reduced scale.