It is widely recognized that public health interventions benefit from community engagement and leadership, yet there are challenges to evaluating complex, community-led interventions assuming hierarchies of evidence derived from laboratory experimentation and clinical trials. Particular challenges include, first, inconsistency of the intervention across sites; and second, absence of researcher control over the sampling frame and methodology. This report highlights these challenges as they played out in the evaluation of a community-organized health project in South London. The project aimed to benefit maternal mental health, health literacy and social capital, and especially to engage local populations known to have reduced contact with statutory services. We evaluated the project using two studies with different designs, sampling frames and methodologies. In one the sampling frame and methodology were under community control, permitting comparison of change in outcomes from before to after participation in the project. In the other, the sampling frame and methodology were under researcher control, permitting a case-control design. The two evaluations led to different results however: participants in the community-controlled study showed benefits, while participants in the researcher-controlled study did not. The principal conclusions are that while there are severe challenges to evaluating a community-led health intervention using a controlled design, measurement of pre-/post-participation changes in well-defined health outcomes should typically be a minimum evaluation requirement, and confidence in attributing causation of any positive changes to participation can be increased by use of interventions in the project and in the engagement process itself that have a credible theoretical and empirical basis.