Smartphone use has changed patterns of online and offline interaction. Phubbing (i.e., looking at one’s phone instead of paying attention to others) is an increasingly recognized phenomenon in offline interaction. We examined whether people who phub are more likely to have lower social intelligence, whether phubbing is considered more annoying than being ignored due to reading a magazine, and if people describe smartphones and magazines differently as sources of social distraction. We collected two survey samples (N = 112, N = 108) for a cartoon-based role-playing experiment (the Bystander Inaccessibility Experiment) in which a smartphone user and a person reading a magazine ignored the respondents’ conversational initiatives. Annoyance in each scenario was measured, and written accounts were collected on why the respondents rated the scenarios the way they did. Other measures used included the Generic Scale of Phubbing, Generic Scale of Being Phubbed, and Tromsø Social Intelligence Scale. The results showed that participants in both samples were more annoyed by phubbing than by being ignored due to reading a magazine. Linear regression analyses showed that phubbing was associated with lower social intelligence, even after adjusting for confounding factors. The annoyingness of phubbing was explained with negative attitudes toward smartphones, which were assumed to be used for useless endeavors, while magazines were more appreciated and seen as more cultivating. The role of bystanders’ epistemic access to the smartphone user’s activities is discussed.