It is familiar that the Casimir self-energy of a homogeneous dielectric ball is divergent, although a finite self-energy can be extracted through second order in the deviation of the permittivity from the vacuum value. The exception occurs when the speed of light inside the spherical boundary is the same as that outside, so the self-energy of a perfectly conducting spherical shell is finite, as is the energy of a dielectric-diamagnetic sphere with $\varepsilon\mu=1$, a so-called isorefractive or diaphanous ball. Here we re-examine that example, and attempt to extend it to an electromagnetic $\delta$-function sphere, where the electric and magnetic couplings are equal and opposite. Unfortunately, although the energy expression is superficially ultraviolet finite, additional divergences appear that render it difficult to extract a meaningful result in general, but some limited results are presented.