1. Introduction

2. Problem Formulation

Table 1. Notations of mathematical model.

3. Encoding and Decoding Design

3.1. Encoding

In this study, the solution to the problem is represented using an operation-based encoding method. The operations of a job are indicated by number of the job, and the number of times that the job number appears represents which operation it is for that job. Due to the design order requirements of the products considered in this paper, and the possibility of identical jobs existing among different products, product information needs to be read sequentially to determine the required jobs for the product, and each job is assigned a unique code for identification. The pseudocode for the encoding is shown in Algorithm 1.

Algorithm 1 Encoding Method

To better illustrate the encoding process, we use an example where the demand for product 1 is 2 and the demand for product 2 is 1. Product 1 consists of components 1 and 2, while product 2 consists of components 3 and 4. Component 1 is composed of job 1, component 2 is composed of jobs 2 and 3, component 3 is composed of jobs 2 and 4, and component 4 is composed of jobs 5, 6 and 7. Each job is processed through corresponding operations. The specific encoding method for this example is shown in Figure 2.

4. IICA-QL for EEMsMlAJSP

4.2. Assimilation Based on Q-Learning Adaptive Adjustment of Operator Parameter

Assimilation is the process by which colonies within an empire are moved towards the imperialist, with population evolution achieved through the assimilation of colonies by the imperialist. To ensure convergence speed while maintaining population diversity, a dynamic adaptive adjustment strategy for operator parameter based on Q-learning is designed. First, two different operator co-evolution operations are employed. Then, a Q-learning-based adaptive adjustment strategy for the UX operator parameter is implemented to enhance the convergence and diversity of the algorithm. Finally, an elite retention strategy is used to update the individuals of each empire, thereby improving the convergence speed. The pseudocode is shown in Algorithm 2.

Algorithm 2 Assimilation based on Q-learning adaptive adjustment of operator parameter

4.2.1. Adaptive Adjustment of UX Operator Parameter Based on Q-Learning

During the assimilation process, the uniform crossover (UX) operator [21] and the two-point Crossover (TPX) operator [22] are used, respectively applied to the movement of colonies towards the imperialist and the external archive set. Among these, the UX operator is used as the main evolutionary operator to guide the evolutionary process of population and can also be applied for local search [21], playing an important role in the quality of population evolution. The parameter of the UX operator determines the number of high-quality genes passed from parents to offspring. In the early stages of evolution, a larger parameter is helpful for the algorithm converge, while in the later stages of evolution, a larger parameter can result in poor population diversity, causing the algorithm to fall into local optima. Therefore, a Q-learning-based adaptive adjustment strategy for the UX operator parameter is devised to dynamically adjust the evolutionary process of population, enabling the algorithm to converge more efficiently to the Pareto front.

State: The state of the entire population is determined by evaluating the convergence metric $\Delta GD$ of the non-dominated solution set compared to the reference set and the diversity metric $\Delta Ratio$, which represents the coverage ratio of the solution space. Here, $\Delta GD$ represents the generational distance difference of the non-dominated solutions obtained compared to the Pareto front $P{F}^{*}$ between generation t and t+1. $P{F}^{*}$ is a preset reference point, and $\Delta Ratio$ represents the proportion of subregions occupied by the non-dominated solutions in the current generation in the objective space. The objective space is divided into several non-overlapping regions by weight vectors [23]. $\Delta GD$ indicates the degree of convergence of generation t+1 compared to generation t, the smaller the value of $\Delta GD$, the better the convergence. $\Delta Ratio$ represents the change in non-dominated set diversity, with $\Delta Ratio<0$ indicating a decrease in diversity and $\Delta Ratio>0$ indicating an increase in diversity. The calculations for $\Delta GD$ and $\Delta Ratio$ are given by Equation (24) and Equation (25) respectively, where $\left|P{F}_t\right|$ and $\left|{\psi }_t\right|$ represent the number of non-dominated solutions in generation t and the proportion of the objective space they occupy. During the evolutionary, there are four possible states: (1) $\Delta GD>=0,\Delta Ratio>0$;(2)$\Delta GD>=0,\Delta Ratio<=0$;(3)$\Delta GD<0,\Delta Ratio>0$;(4) $\Delta GD<0,\Delta Ratio<=0$.

$$\Delta GD=(\sqrt{{\displaystyle \sum _{x\in P{F}_{t+1}}\underset{y\in P{F}^{*}}{\min }d{(x,y)}^2}}/\left|P{F}_{t+1}\right|)-(\sqrt{{\displaystyle \sum _{x\in P{F}_t}\underset{y\in P{F}^{*}}{\min }d{(x,y)}^2}}/\left|P{F}_t\right|)$$

(24)

$$\Delta Ratio=\left|{\psi }_{t+1}\right|-\left|{\psi }_t\right|$$

(25)

Action: The action set $\mathbb{A}$ is composed of ten actions, representing ten levels of the UX operator parameter. By determining the current state of the population, the action with the highest Q-value in the Q-table is selected based on the ε-greedy strategy.

ε-greedy: In the initial iterations, Q-values usually lack prior knowledge, making it easy for the agent to fall into local optima during learning. Therefore, a ε-greedy strategy based on sigmod is used to guide the learning of agent. The mathematical expression for ε-greedy is as follows:

$$\epsilon =0.1+（\epsilon -0.1）\times {\displaystyle \frac{1}{1+e^{(-0.2\times (g-50))}}}$$

(26)

where g represents the number of iterations.

Reward: By calculating the values of $\Delta GD$ and $\Delta Ratio$ after executing an action, the agent receives a corresponding reward and updates the Q-value of the action in the Q-table for the corresponding state according to Equation (27). If $\Delta GD\ge 0,\Delta Ratio>0$, then reward = 1; if $\Delta GD\ge 0,\Delta Ratio<=0$, then reward = -2; if $\Delta GD<0,\Delta Ratio>0$, then reward = 4; if $\Delta GD<0,\Delta Ratio\le 0$, then reward = 2.

$$Q（s_t,a_t)=Q（s_t,a_t)+\alpha [rewar{d}_{t+1}+\gamma \underset{a\in \mathbb{A}}{\max }Q(s_{t+1},a)-Q（s_t,a_t)]$$

(27)

where $\alpha$ and $\gamma$ are the learning rate and discount factor respectively, and $\mathbb{A}$ is the set of available actions.

Figure 4 provides an example. In generation t, the population is in state S₃ and the action A₂ is chosen. After this action is executed, the population is transitioned to state S₁ in generation t+1. In this state, the action A₁₀ is chosen, and in generation t+2, the population is transitioned to state S₄.

4.2.2. Update Strategy

To achieve rapid convergence of the population, an elite retention strategy is employed to select the colonies that need to be retained after the assimilation process. The specific operations are as follows: individuals generated during the assimilation process of imperialist $Im{p}_k$ are placed into the colony offspring population $Offsprin{g}_k$. After the assimilation operation within the empires is completed, the original colony population is merged with the colony offspring population to form a new population $SubPo{p}_{\mathrm{k}}$. Then, the individuals in $SubPo{p}_{\mathrm{k}}$ are non-dominated sorted and their crowding distances are calculated. A number of $N{C}_{\mathrm{k}}$ colonies are selected as the next generation of colonies. Additionally, for new solutions generated by the assimilation of the imperialist, if they are not dominated by the old solutions, the original imperialist is replaced by the new solutions; otherwise, no replacement is made.

4.3. Revolution Based on Hyper-Heuristic Variable Neighborhood Search

The colonial revolution is a process of further exploring and exploiting the neighborhood of superior colonies after completing the assimilation operation. In this paper, various neighborhood search (VNS) operations are designed, and high-level information constructed from a two-dimensional probability matrix is used to guide the selection of corresponding low-level variable neighborhood search operation sets. The selection of variable neighborhood search operators is dynamically adjusted by the high-level strategy based on the feedback information received from the environment after each application of the low-level variable neighborhood search operation set, thereby improving the search performance of the algorithm.

4.3.1. Neighborhood Search (NS) Operations

To implement an effective perturbation strategy for individuals, two single-operation-based neighborhood operations and five block-based neighborhood operations are designed as neighborhood search operators.

NS1: Single-operation insertion operation for individual encoding. A random operation is selected from the individual encoding and inserted into another position

NS2: Single-operation exchange operation for individual encoding. Two different operations are randomly selected and their positions are swapped.

NS3: Reversal operation for individual encoding. Two positions are randomly selected, and the segment of operations between these positions is reversed.

NS4: Segment exchange operation for individual encoding. Two segments of operations are randomly selected and their positions are exchanged.

NS5: Segment insertion operation for individual encoding. Two segments of operations are randomly selected and one segment is inserted into another position within the encoding.

NS6: Segment shuffling operation for individual encoding. A segment of operations is randomly selected, and operations within segment are randomly shuffled.

NS7: Segment gene dispersal operation for individual encoding. The individual encoding is divided into several segments, and the consecutive repeated genes within each segment are dispersed to different positions.

Figure 5. Schematic Diagram of Neighborhood Structure.

Figure 5. Schematic Diagram of Neighborhood Structure.

4.3.2. Two-Dimensional Probability Model

Define $NS=\{N{S}^{G,k}(1),N{S}^{G,k}(2),\cdots ,N{S}^{G,k}(l)\}$ as the neighborhood operations sequence executed by the k-th individual in generation G, where l represents the length of variable neighborhood search. The two-dimensional probability model $P^G(x,y)$, shown in Equation (28), is used as the high-level strategy to guide the selection of the low-level the variable neighborhood search operations set.

$$P^G(x,y)=\left[\begin{array}{l}{P}_{l\times l}^G(1,1)\cdots P_{l\times l}^G(1,l)\\ ⋮\ddots ⋮\\ P_{l\times l}^G(l,1)\cdots P_{l\times l}^G(l,l)\end{array}\right]$$

(28)

The initialization of the two-dimensional probability model and its update in the G-th generation are shown in Equation (29) and Equation (30) respectively.

$$P^0(x,y)=1/l,x,y=1,2,\cdots ,l$$

(29)

$$P^G(x,y)=（1+reward)\times P^{G-1}(x,y),x=1,2,\cdots ,l,y=NS(1),NS(2),\cdots ,NS(l)$$

(30)

$$reward=\{\begin{array}{l}0.1,\mathrm{new}\mathrm{solution}\mathrm{is}\mathrm{not}\mathrm{dominated}\mathrm{by}\mathrm{old}\mathrm{solution}\\ -0.1,\mathrm{otherwise}\end{array}$$

(31)

After completing the revolution operations, the cost of each country within the empire is calculated, and the country with the highest cost is selected as the new imperialist, while the original imperialist becomes a colony. $P^G(x,y)$ is updated according to Equation (31).

Algorithm 3 Revolution based on hyper-heuristic variable neighborhood search

4.4. Joint Imperialist Invasion

Dividing the population into multiple subpopulations is helpful for individuals to extensively search different areas, and balances the exploration and exploitation capabilities of the algorithm through co-evolution [24]. To address the issues of poor diversity and premature convergence in the basic Imperialist Competitive Algorithm, a joint imperialist competition mechanism is designed to replace the standard imperialist competition operations. Each empire is regarded as a population that collaboratively searches different regions of the solution space. This mechanism ensures that the collaborative evolutionary search is maintained within each empire and that information sharing and interaction are enhanced between empires, thereby improving diversity and the search efficiency of the algorithm.

The specific operations of the joint imperialist invasion are as follows: The total cost of each imperial group is calculated and ranked according to Equation (32), where $Em{p}_k$ represents the set of colonies owned by $Im{p}_k$, and $\xi$ is the proportional coefficient of the total cost of the empire accounted for by the colonies, which is set to 0.1. The imperialists of the top $N_{im}-1$ empires perform POX crossover [4] with random solutions, and the corresponding number of the worst colonies in the worst-ranked empire are replaced by the newly generated solutions. Through joint imperialist competition, the collaborative evolution within each empire can be maintained, while information sharing and interaction between empires can be enhanced, thus improving diversity and increasing the search efficiency of the algorithm.

$$T{C}_k^g=c_i^g+\xi {\displaystyle \sum _{\sigma \in Em{p}_k}{c}_{\sigma }^g}/N{C}_k$$

(32)

Algorithm 4 Joint imperialist invasion

5. Experimental Results and Analysis

6. Conclusions

References

1. Wang J, Lei D, Cai J. An adaptive artificial bee colony with reinforcement learning for distributed three-stage assembly scheduling with maintenance. Appl Soft Comput. 2022;117:108371. [CrossRef]
2. Cheng L, Tang Q, Liu S, Zhang L. Mathematical model and augmented simulated annealing algorithm for mixed-model assembly job shop scheduling problem with batch transfer. Knowl-Based Syst. 2023;279:110968. [CrossRef]
3. Cheng L, Tang Q, Zhang L. Production costs and total completion time minimization for three-stage mixed-model assembly job shop scheduling with lot streaming and batch transfer. Eng Appl Artif Intell. 2024;130:107729. [CrossRef]
4. Gong G, Chiong R, Deng Q, et al. A two-stage memetic algorithm for energy-efficient flexible job shop scheduling by means of decreasing the total number of machine restarts. Swarm Evol Comput. 2022;75:101131. [CrossRef]
5. Wang JJ, Wang L. A Cooperative Memetic Algorithm With Learning-Based Agent for Energy-Aware Distributed Hybrid Flow-Shop Scheduling. IEEE Trans Evol Comput. 2022;26(3):461-475. [CrossRef]
6. Cao S, Li R, Gong W, Lu C. Inverse model and adaptive neighborhood search based cooperative optimizer for energy-efficient distributed flexible job shop scheduling. Swarm Evol Comput. 2023;83:101419. [CrossRef]
7. Wang J jing, Wang L. A cooperative memetic algorithm with feedback for the energy-aware distributed flow-shops with flexible assembly scheduling. Comput Ind Eng. 2022;168:108126. [CrossRef]
8. Li R, Gong W, Wang L, Lu C, Zhuang X. Surprisingly Popular-Based Adaptive Memetic Algorithm for Energy-Efficient Distributed Flexible Job Shop Scheduling. IEEE Trans Cybern. Published online 2023:1-11. [CrossRef]
9. Meng L, Zhang C, Zhang B, Gao K, Ren Y, Sang H. MILP modeling and optimization of multi-objective flexible job shop scheduling problem with controllable processing times. Swarm Evol Comput. 2023;82:101374. [CrossRef]
10. Lei D, Li M, Wang L. A Two-Phase Meta-Heuristic for Multiobjective Flexible Job Shop Scheduling Problem With Total Energy Consumption Threshold. IEEE Trans Cybern. 2019;49(3):1097-1109. [CrossRef]
11. Zhou R, Lei D, Zhou X. Multi-Objective Energy-Efficient Interval Scheduling in Hybrid Flow Shop Using Imperialist Competitive Algorithm. IEEE Access. 2019;7:85029-85041. [CrossRef]
12. Lei D, Yuan Y, Cai J, Bai D. An imperialist competitive algorithm with memory for distributed unrelated parallel machines scheduling. Int J Prod Res. 2020;58(2):597-614. [CrossRef]
13. Li Y, Yang Z, Wang L, Tang H, Sun L, Guo S. A hybrid imperialist competitive algorithm for energy-efficient flexible job shop scheduling problem with variable-size sublots. Comput Ind Eng. 2022;172:108641. [CrossRef]
14. Chen Y, Zhong J, Mumtaz J, Zhou S, Zhu L. An improved spider monkey optimization algorithm for multi-objective planning and scheduling problems of PCB assembly line. Expert Syst Appl. 2023;229:120600. [CrossRef]
15. Chen R, Yang B, Li S, Wang S. A self-learning genetic algorithm based on reinforcement learning for flexible job-shop scheduling problem. Comput Ind Eng. 2020;149:106778. [CrossRef]
16. Li R, Gong W, Lu C, Wang L. A Learning-Based Memetic Algorithm for Energy-Efficient Flexible Job-Shop Scheduling With Type-2 Fuzzy Processing Time. IEEE Trans Evol Comput. 2023;27(3):610-620. [CrossRef]
17. Pan Z, Wang L, Wang J, Yu Y. Distributed Energy-Efficient Flexible Manufacturing With Assembly and Transportation: A Knowledge-Based Bi-Hierarchical Optimization Approach. IEEE Trans Autom Sci Eng. Published online 2024:1-17. [CrossRef]
18. Zou P, Rajora M, Liang SY. A new algorithm based on evolutionary computation for hierarchically coupled constraint optimization: methodology and application to assembly job-shop scheduling. J Sched. 2018;21(5):545-563. [CrossRef]
19. Zhu Z, Zhou X. An efficient evolutionary grey wolf optimizer for multi-objective flexible job shop scheduling problem with hierarchical job precedence constraints. Comput Ind Eng. 2020;140:106280. [CrossRef]
20. Li M, Lei D, Cai J. Two-level imperialist competitive algorithm for energy-efficient hybrid flow shop scheduling problem with relative importance of objectives. Swarm Evol Comput. 2019;49:34-43. [CrossRef]
21. Fontes DBMM, Homayouni SM, Fernandes JC. Energy-efficient job shop scheduling problem with transport resources considering speed adjustable resources. Int J Prod Res. 2023;0(0):1-24. [CrossRef]
22. Li J, Han Y, Gao K, Xiao X, Duan P. Bi-Population Balancing Multi-Objective Algorithm for Fuzzy Flexible Job Shop With Energy and Transportation. IEEE Trans Autom Sci Eng. Published online 2023:1-17. [CrossRef]
23. Ming M, Trivedi A, Wang R, Srinivasan D, Zhang T. A Dual-Population-Based Evolutionary Algorithm for Constrained Multiobjective Optimization. IEEE Trans Evol Comput. 2021;25(4):739-753. [CrossRef]
24. Cheng L, Tang Q, Zhang L, Meng K. Mathematical model and enhanced cooperative co-evolutionary algorithm for scheduling energy-efficient manufacturing cell. J Clean Prod. 2021;326:129248. [CrossRef]
25. Wang J jing, Wang L, Xiu X. A cooperative memetic algorithm for energy-aware distributed welding shop scheduling problem. Eng Appl Artif Intell. 2023;120:105877. [CrossRef]
26. Ren W, Wen J, Yan Y, Hu Y, Guan Y, Li J. Multi-objective optimisation for energy-aware flexible job-shop scheduling problem with assembly operations. Int J Prod Res. 2021;59(23):7216-7231. [CrossRef]