Thermal shape memory polymers (SMPs) find increasing applications in biology and medicine due to their promising potential for improved biocompatibility. In such applications, the inevitable penetration of small molecules from the ambient fluid into the polymer influences the shape recovery process and often leads to a reduction of the temperature, at which shape recovery takes place. We show here via molecular dynamics simulations that the size of additive molecules plays a key role for this process. While the effect of concentration on the recovery rate is monotonic in the investigated range, a non-monotonic dependence on the size of additive molecules emerges at temperatures close to the glass transition. This work thus identifies the additives’ size to be a qualitatively novel parameter for switching the recovery process in polymer-based shape memory materials.