Actions of noncompact groups and algorithm design: A case study

Numerical matrix computations involving actions of non-compact transformation groups are known to produce numerical problems since the elements of the pertaining matrix representations are inherently unbounded. In this case study we analyze numerical problems occurring in a class of algorithms that is based on actions of the pseudo-orthogonal group O_n,m - a group that is noncompact (hyperbolic geometry) and well established in signal processing (Schur methods). As a major result, it is shown how to exploit the additional degrees of freedom in defining coordinate frames in a Grassmannian setting in order to impose an a priori bound on the norm of the transformation matrices. This way, numerically disastrous situations can be circumvented systematically. Hence, it becomes possible to develop modified algorithms which exhibit superior numerical performance for a large class of problems based on e.g. hyperbolic transformations.