S 2 P 3: Self-Supervised Polarimetric Pose Prediction

Patrick Ruhkamp, Daoyi Gao, Nassir Navab, Benjamin Busam

Research output: Contribution to journalArticlepeer-review


This paper proposes the first self-supervised 6D object pose prediction from multimodal RGB + polarimetric images. The novel training paradigm comprises (1) a physical model to extract geometric information of polarized light, (2) a teacher–student knowledge distillation scheme and (3) a self-supervised loss formulation through differentiable rendering and an invertible physical constraint. Both networks leverage the physical properties of polarized light to learn robust geometric representations by encoding shape priors and polarization characteristics derived from our physical model. Geometric pseudo-labels from the teacher support the student network without the need for annotated real data. Dense appearance and geometric information of objects are obtained through a differentiable renderer with the predicted pose for self-supervised direct coupling. The student network additionally features our proposed invertible formulation of the physical shape priors that enables end-to-end self-supervised training through physical constraints of derived polarization characteristics compared against polarimetric input images. We specifically focus on photometrically challenging objects with texture-less or reflective surfaces and transparent materials for which the most prominent performance gain is reported.

Original languageEnglish
Pages (from-to)2177-2194
Number of pages18
JournalInternational Journal of Computer Vision
Issue number6
StatePublished - Jun 2024


  • Differentiable rendering
  • Multi-modalities
  • Pose estimation
  • Self-supervision


Dive into the research topics of 'S 2 P 3: Self-Supervised Polarimetric Pose Prediction'. Together they form a unique fingerprint.

Cite this