(1) Background: It is unclear whether the reported presenting clinical features of coronavirus disease 2019 (COVID-19) are useful in identifying high-risk patients for early testing and isolation in the emergency department (ED). We aimed to compare the exposure history, clinical, laboratory, and radiographic features of ED patients who tested positive and negative for COVID-19; (2) Methods: We conducted a case-control study in seven EDs during the first five weeks of the COVID-19 outbreak in Hong Kong. Thirty-seven laboratory-confirmed COVID-19 patients were compared with 111 age- and gender-matched controls; (3) Results: There were no significant differences in patient characteristics and reported symptoms between the groups, except patient-reported fever. A positive travel history or contact history was the most significant predictor for COVID-19 infection. After adjustment for age and presumed location of acquiring the infection in Wuhan/Hubei, patient-reported fever (OR 2.6, 95% CI 1.1 to 6.3), delayed presentation (OR 5.0, 95% CI 2.0 to 12.5), having medical consultation before ED presentation (OR 7.4, 95% 2.9 to19.1), thrombocytopenia (OR 4.0, 95% CI 1.6 to 9.7), raised lactate dehydrogenase (OR 5.9, 95% CI 1.9 to 18.5), haziness, consolidation or ground-glass opacity on chest radiography (OR 5.6, 95% CI 2.0 to 16.0), and bilateral changes on chest radiography (OR 13.2, 95% CI 4.7 to 37.4) were associated with a higher odds of COVID-19 separately while neutrophilia was associated with a lower odds (OR 0.3, 95% CI 0.1-0.8); and (4) Conclusions: This study highlights several features that may be useful in identifying high-risk patients for early testing and isolation while waiting for test result. Further studies are warranted to verify the findings.