Artificial intelligence (AI) is a tool that financial intermediaries and insurance companies use in most cases or are willing to use it in almost all their activities. AI can have a positive impact on almost all aspects of the insurance value chain.: pricing, underwriting, marketing, claims management, after-sales services. While it is very important and useful, AI is not free of risks, including its robustness against cyber-attacks and so-called adversarial attacks. Adversarial attacks are conducted by external entities to misguide and defraud the AI algorithms. The paper is designed to provide a review of adversarial AI and discuss its implications for the insurance sector. The study starts with a taxonomy of adversarial attacks and presents a fully-fledged example of claims falsification in health insurance. Some remedies, consistent with the current regulatory framework, are presented.