Artificial Intelligence (AI) has the disruptive potential to transform patients’ lives via innovations in pharmaceutical sciences, drug development, clinical trials, and manufacturing. However, it presents significant challenges, ethical concerns, and risks across sectors and societies. AI’s rapid advancement has revealed regulatory gaps as existing public policies struggle to keep pace with the challenges posed by these emerging technologies. The term AI itself has become commonplace to argue that greater “human oversight” for “machine intelligence” is needed to harness the power of this revolutionary technology for both potential and risk management, and hence to call for more practical regulatory guidelines, harmonized frameworks, and effective policies to ensure safety, scalability, data privacy, and governance, transparency, and equitable treatment. In this review paper, we employ a holistic multidisciplinary lens to overview the current regulatory landscape with a synopsis of the FDA workshop perspectives on the use of AI in drug and biological product development. We discuss the promises of responsible data-driven AI, challenges and related practices adopted to overcome limitations, and our practical reflections on regulatory oversight. Finally, the paper outlines a path forward and future opportunities for lawful ethical AI. This review highlights the importance of risk-based regulatory oversight, including diverging views in the field, in reaching a consensus.