Developmental modularity has long been viewed as a hierarchical organization that facilitates evolution over macro-evolutionary time through modification or co-option of preexisting modules. More recently, developmental modularity has been proposed as a micro-evolutionary mechanism capable of driving rapid evolution of novel color pattern phenotypes between closely related taxa. In this scenario, swapping allelic variants of modular cis-regulatory elements (CREs) via recombination generates novel phenotypes by shuffling preexisting color pattern modules into new arrangements. Recent evidence from Drosophila and butterflies, however, provides a series of examples in which pleiotropic CREs function in multiple developmental contexts. The potential prevalence of pleiotropy in CRE function is a major barrier to the proposed evolutionary role of CRE modules and encourages us to reconsider the relative importance of modularity for microevolutionary change. Here we first review the case for the apparent frequent exchange of modular color pattern phenotypes as a mechanism facilitating diversification. We then contrast this with recent evidence of CRE pleiotropy and argue that exchange of CRE modules should not be the default assumption, even when phenotypes look modular. Finally, we review experimental data on Heliconius butterfly wing patterns—which appear modular—and introduce the concept of evolutionary modularity as an alternative to developmental modularity. Evolutionary modularity reconciles the appearance of modularity in comparative genomic studies of Heliconius color patterns with experimental data supporting a non-modular architecture. We propose that evolutionary modularity provides a potentially important pathway for exchange of phenotypic elements between hybridizing taxa independent of the underlying developmental architecture.