Least squares (LS) estimation is one of the most important tools in geodetic data analysis. However, its prevailing use is not often complemented by an objective view of its rudiments. Within the standard formalism of LS estimation theory there are actually several paradoxical and curious issues which are seldom explicitly formulated. The aim of this expository paper is ro present some of these issues and ro discuss their implications for geodetic data analysis and parameter estimation problems. In the first part of the paper, an alternative view of the statistical principles that are traditionally linked to LS estimation is given. Particularly, we show that the property of unbiasedness for the ordinary LS estimators can be replaced with a different, yet equivalent, constraint which implies that the numerical range of the unknown parameters is boundless. In the second part of the paper, the shortcomings of the LS method are exposed from a purely algebraic perspective, without employing any concepts from the probabilistic/statistical framework of estimation theory. In particular, it is explained that what is 'least' in least squares is certainly not the errors in the estimated model parameters, and that in every LS-based inversion of a linear model there exists a critical trade-off between the Euclidean norms of the parameter estimation errors and the adjusted residuals.