Three-dimensional transfer functions (3D TFs) are generally assumed to fully describe the transfer behavior of optical topography measuring instruments such as coherence scanning interferometers in the spatial frequency domain. Therefore, 3D TFs are supposed to be independent of the surface under investigation resulting in a clear separation of surface properties and transfer characteristics. In this paper we show that the 3D TF of an interference microscope differs depending on whether the object is specularly reflecting or consists of point scatterers. In addition to the 3D TF of a point scatterer, we will derive an analytical expression for the 3D TF corresponding to specular surfaces and demonstrate this as being most relevant in practical applications of coherence scanning interferometry (CSI). We additionally study the effects of temporal coherence and disclose, that in conventional CSI temporal coherence effects dominate. However, narrowband light sources are advantageous if high spatial frequency components of weak phase objects are to be resolved, whereas for low-frequency phase objects of higher amplitude the temporal coherence is less affecting. Finally, we present an approach that explains the different transfer characteristics of coherence peak and phase detection in CSI signal analysis.