China is facing huge pressure on CO2 emissions reduction. The heavy industry accounts for over 60% of China’s total energy consumption, and thus lead to a large number of energy-related carbon emissions. This paper adopts the Log Mean Divisia Index (LMDI) method based on the extended Kaya identity to explore the influencing factors of CO2 emissions from China’s heavy industry; we calculate the trend of decoupling by presenting a theoretical framework for decoupling. The results show that labor productivity, energy intensity, and industry scale are the main factors affecting CO2 emissions in the heavy industry. The improvement of labor productivity is the main cause of the increase in CO2 emissions, while the decline in energy intensity leads to CO2 emissions reduction, and the industry scale has different effects in different periods. Results from the decoupling analysis show that efforts made on carbon emission reduction, to a certain extent, achieved the desired outcome but still need to be strengthened.