The holographic principle originates from the observation that black hole entropy is proportional to the horizon area and not, as expected from a quantum field theory perspective, to the volume. This principle has found a concrete realization in the Anti-de Sitter/Conformal Field Theory (AdS/CFT) correspondence. It is interesting to ponder whether the key insights about holography so far are specific to AdS/CFT or if they are general lessons for quantum gravity and (non)relativistic field theories. Relativistic and nonrelativistic geometries play a fundamental role in advances of holography beyond AdS spacetimes, e.g., for strongly coupled systems in condensed matter physics. Holography for higher spin theories is comparably well understood and they are therefore good candidates to gain further insights. In three spacetime dimensions they are distinguished by technical simplicity, the possibility to write the theory in ChernSimons form and the option to consistently truncate the infinite higher spin fields to any integer spin greater than two. Here we will show progress that has been made to construct relativistic and nonrelativistic theories in spin-three gravity. These theories describe a coupled spin two and three field and are based on ChernSimons theories with kinematical gauge algebras of which the Poincaré, Galilei and Carroll algebra are prominent examples. To have a spin-three theory where all fields are dynamical it is sometimes necessary, as will be shown, to extend the gauge algebras accordingly. We will also discuss concepts which are useful in these constructions. Guidance is provided by combining Lie algebra contractions and, a procedure that will be reviewed extensively, double extensions.