Crowdsourced Qualitative Evaluation of Pedestrian Movement Models

In driving simulator studies, it is crucial to provide a highly realistic virtual environment in order to allow for the study participants to immerse themselves more fully, which in turn provides for more realistic driver behavior. In studies involving interaction with pedestrians, this means providing realistic pedestrian behavior, which can be achieved in one of three ways: by recording pedestrian movements in the real world and playing these back in the simulation environment, by manually animating the behavior of each pedestrian, or by using a pedestrian movement model to simulate behavior. Each of these approaches has its own advantages and drawbacks, but it is clear that in order to simulate larger amounts of pedestrians in scenarios for which there is no physical analog, realistic pedestrian movement models are crucial. However, most evaluations of pedestrian models focus on macroscopic effects, such as flow rates through a bottleneck, or on their ability to reproduce certain empirically observed emergent phenomena. What is often ignored in such evaluations is the perceived realism imparted by the model at the level of an individual pedestrian, and whether humans are able to differentiate between said models and real pedestrian behavior in a driving-simulator-like environment. In order to address this issue, we propose a highly automatable methodology which leverages the power of crowdsourcing to quickly and cheaply evaluate the perceived realism of various pedestrian models, as well as the impact of their various parameters on the achieved realism. The methodology consists of generating short video clips from a 3D rendered scene, using Unity as the visualization engine, in which the movements of various pedestrians are controlled by either a model with a given set of parameters or played back from a real-world recorded trajectory. In an exercise analogous to a classical Turing Test, survey participants are shown the videos and asked to rate the realism of the displayed pedestrian behavior. The participants’ ratings of the respective real and simulated videos are then compared in order to arrive at a qualitative measure of how convincing the pedestrian behavior generated by the model is. We then perform a small proof-of-concept study applying the described methodology, present some preliminary results, and list a number of areas for future research and improvement.