Head-Related Transfer Functions (HRTFs) are used for adding spatial information in 3D audio synthesis or for binaural robotic sound localization. Both tasks work best when using a custom HRTF database that fits the physiology of each person or robot. Usually, measuring HRTFs is a time consuming and complex procedure that is performed with expensive equipment in an anechoic chamber. In this paper we present a method that enables HRTF measurement in everyday environments by passively recording the surroundings without the need to actively emit special excitation signals. Experiments show that our method captures the HRTF's spatial cues and enables accurate sound localization.