Visuo-spatial perspective taking (VSPT) concerns the ability to understand something about the visual relationship between an Agent or observation point on the one hand, and a target or scene on the other. Despite its importance to a wide variety of other abilities, from communication to navigation, and decades of research, there is as yet no theory of VSPT. Indeed, the heterogeneity of results from different (and sometimes the same) VSPT tasks point to a complex picture suggestive of multiple VSPT strategies, individual differences in performance, and context-specific factors that together have a bearing on both the efficiency and accuracy of outcomes. In this paper, we review the evidence in search of patterns in the data. We find a number of predictors of VSPT performance but also a number of gaps in our understanding that suggest useful pathways for future research and, possibly, a theory (or theories) of VSPT. Overall, this review makes the case for understanding VSPT by better understanding the perspective taker rather than the target agent or their perception.