Lung ultrasound (LUS) allows the detection of a series of manifestations of COVID-19 such as B lines and consolidations. The objective of this work was to study the inter-rater reliability (IRR) when detecting signs associated with COVID-19 in the LUS, as well as the impact of performing the test in the longitudinal or transverse orientation. 33 physicians with advanced experience in LUS, independently evaluated ultrasound videos previously acquired with the ULTRACOV system of 20 patients with confirmed COVID-19. In each patient, 24 videos of 3 seconds were acquired (using 12 positions with the probe in longitudinal and transverse orientations). Physicians had no information about the patients or other previous evaluations. The score assigned to each acquisition followed the convention applied in previous studies. A substantial IRR was found in the cases of normal LUS (κ = 0.74), only a fair IRR for the presence of individual B lines (κ = 0.36) and for confluent B lines occupying <50% (κ = 0.26), and a moderate IRR in consolidations and B-lines >50% (κ = 0.50). No statistically significant differences between the longitudinal and transverse scans were found. The IRR in LUS of COVID-19 patients may benefit from more standardization of the clinical protocols.