The cellular stress response corresponds to the molecular changes that cell undergoes in response to various environmental stimuli. It induces drastic changes in the regulation of gene expression, at transcriptional and post-transcriptional levels. Actually, translation is strongly affected with a blockade of the classical cap-dependent mechanism, whereas alternative mechanisms are activated to support translation of specific mRNAs. One of the major mechanisms involved in stress-activated translation is the internal ribosome entry site (IRES)-driven initiation. IRESs, first discovered in viral mRNAs, are present in cellular mRNAs coding for master regulators of cell responses, whose expression must be tightly controlled. IRESs allow translation of these mRNAs in response to different stresses, including DNA damage, amino-acid starvation, hypoxia or endoplasmic reticulum stress, as well as to physiological stimuli such as cell differentiation or synapse network formation. Importantly, cellular mRNA IRESs are regulated by IRES trans-acting factor (ITAFs), exerting their action by at least nine different mechanisms. This review presents an update of the reported ITAFs regulating cellular mRNA translation and of the different mechanisms allowing them to control translation initiation in specific conditions. The impact of ITAFs on coordinated expression of mRNA families and consequences in cell physiology and diseases are also highlighted.