A reinforcement learning control method for a solid attitude and divert propulsion system is proposed. The system in this research includes 4 divert thrust nozzles, 6 attitude thrust nozzles, and a common combustion chamber. To achieve the required thrust, the pressure in the combustion chamber is first adjusted by controlling the total opening of the nozzles to generate the gas source. Next, by controlling the opening of nozzles at different positions, the required thrust is produced in five-axis directions. Finally, the motor speed is regulated to drive the valve core to the specified position, completing the closed-loop control of the nozzle opening. The control algorithm used is the Proximal Policy Optimization (PPO) reinforcement learning algorithm. Through system identification and numerical modeling, the training environment for the intelligent agent is created. To accommodate different training objectives, multiple reward functions are implemented. Ultimately, through training, a multi-layer intelligent agent architecture for pressure, thrust, and nozzle opening is established, achieving effective system pressure and thrust control.