Commercial sleep devices and mobile-phone applications for scoring sleep are gaining ground. In order to provide reliable information about the quantity and/or quality of sleep, their performance needs to be assessed against the current gold-standard, i.e. polysomnography (PSG; measuring brain, eye and muscle activity). We here assessed some commercially available sleep trackers, namely; a commercial activity tracker: Mi band (Xiaomi, BJ, CHN), a scientific actigraph: Motionwatch 8 (CamNTech, CB, UK), and a much used sleep application: Sleep Cycle (Northcube, GOT, SE). We recorded 27 nights in healthy sleepers using PSG and these devices. Surprisingly, all devices had very poor agreement with the gold standard. Sleep parameter comparisons revealed that specifically the Mi band and the sleep cycle application had difficulties in detecting wake periods which negatively affected the total sleep time and sleep efficiency estimations. However, all 3 devices were good in detecting the most basic parameter, the actual time in bed. In summary, our results suggest that, to-date; available sleep trackers do not provide meaningful sleep analysis but may be interesting for simply tracking times in bed. A much closer interaction with the scientific field seems necessary if reliable information shall be derived from such devices in the future.