1. Introduction

1.2. Trilemma of Generative Learning

Xiao et al. describe the Trilemma of Generative Learning as the inability of any single deep generative modelling framework to solve the following requirements for wide adoption and application of image synthesis: (i) high-quality sampling, (ii) mode coverage and sample diversity, and (iii) fast and computationally inexpensive sampling. [7]. Current research primarily focuses on high-quality image generation and ignores the real-world sampling constraints and the need for high diversity and mode coverage. Fast sampling allows for the generative models to be utilized in greater fast-learning applications, which require quick image synthesis, e.g. interactive image editing [7]. Diversity and mode coverage ensure generated images are not direct copies of, but are also not significantly skewed from, the training data.

Figure 1. Generative learning trilemma [7]. Labels show frameworks that tackle two of the three requirements well.

Figure 1. Generative learning trilemma [7]. Labels show frameworks that tackle two of the three requirements well.

This paper reviews research that aims to tackle this trilemma with the D-Wave quantum annealer and attempts to determine the efficacy of modelling on the three axes of the trilemma. In doing so the success of the quantum annealer will be tested against other classical generative modelling methodologies. Success in showing the quantum annealer’s ability to produce (i) high-quality images, (ii) mode coverage and diversity, and (iii) fast sampling will demonstrate the supremacy of quantum annealers over classical methods for the balanced task of image synthesis.

2. Background

2.1. Classical Image Synthesis

2.1.1. Restricted Boltzmann Machine

Boltzmann machines are a class of energy-based generative learning models. A Restricted Boltzmann Machine (RBM), a subset of Boltzmann Machines, is a fully connected bipartite graph that is segmented into visible and hidden neurons.

Figure 2. Restricted Boltzmann Machine Architecture [4].

Figure 2. Restricted Boltzmann Machine Architecture [4].

RBMs are generative models that embed the latent feature space in the weights between the visible and hidden layers. RBMs were first introduced in 1986 by Smolensky and were further developed by Freund and D. Haussler in 1991 [8,9]. The energy function to minimize when training an RBM is the following [10]:

$$E(v,h)=-a^{T}v-b^{T}h-v^{T}Wh$$

(1)

Training is the process of tuning the weights matrix W and bias vectors a & b on the visible v and hidden h layers respectively. v represents the visible units, i.e. the observed values, a training sample. The network assigns a probability to every possible pair of a visible and a hidden vector via this energy function [11]:

$$p(v,h)=\frac{1}{Z}{e}^{-E(v,h)}$$

(2)

Z is the partition function given by summing over all possible pairs of v & h[11]. Thus the probability of a given v is:

$$p\left(v\right)=\frac{1}{Z}\sum _h{e}^{-E(v,h)}$$

(3)

$$Z=\sum _{v,h}{e}^{-E(v,h)}$$

(4)

The difficulty in evaluating the partition function Z introduces the need to use Gibbs sampling with Contrastive Divergence Learning, introduced by Hinton et al. in 2005 [12]. By utilizing such methods one can train the RBM quickly via gradient descent, similar to other neural networks. By adding more hidden layers a deeper embedding can be contained in the system; such a system is called a Deep Belief Network (DBN).

Restricted Boltzmann Machines, while of little note in the modern landscape of machine learning research due to their limited performance and relatively slow training times, are of particular note to this research as they have direct parallels with both the architecture of the D-Wave 2000Q quantum processor and the method by which they reduce the total energy of their respective systems. RBMs also have limited application in computer vision but were an important advancement in the field of generative modelling as a whole.

2.1.2. Variational Autoencoder

A Variational Autoencoder (VAE) is a generative machine learning model developed in 2013 composed of a neural network that is able to generate novel high-fidelity images, text, sound, etc. [13].

Autoencoders seek to compress an input space into a compressed latent representation from which the original input space can be recovered [13]. Variational autoencoders improve upon traditional Autoencoders by recognizing the input space has an underlying distribution and seeks to learn the parameters of that distribution [13]. Once trained VAEs can be used to generate novel data, similar to the input space, by removing the encoding layers and exploring the latent space [13]. Exploring the latent space is simply treating the latent compression layer as an input layer and observing the output of the VAE for various inputs. VAEs marked the first reliable way to generate somewhat high-fidelity images using machine learning [14].

Figure 3. Variational Autoencoder Architecture [14].

Figure 3. Variational Autoencoder Architecture [14].

2.1.3. Generative Adversarial Network

The most significant development in high-fidelity generative image synthesis was in 2014 with the introduction of Generative Adversarial Networks (GANs) by Ian Goodfellow et al. [15]. Goodfellow et al . propose a two-player minimax game composed of a generator model (G) and a discriminator model (D). As the game progresses both the generator and discriminator models improve.

GANs are trained via an adversarial contest between the generator model (G) and discriminator model (D) [15]. x contains samples from both the training set and $p_g$, the images generated by G. $D(x;{\theta }_d)$ outputs the probability that x originates from the training dataset as opposed to $p_g$. Meanwhile $G(z,{\theta }_g)$ outputs $p_g$ given noise z. G’s goal is to fool D while D aims to reliably differentiate real training data from data generated by G. The loss function for G is $log(1-D(G\left(z\right)\left)\right)$. Thus the value/loss function, error, of a GAN is represented as:

$$\underset{\mathrm{G}}{\min }\underset{\mathrm{D}}{\max }V(D,G)=E_{x\sim p_{data}\left(x\right)}\left[logD\left(x\right)\right]+E_{z\sim p_z\left(z\right)}\left[log(1-D\left(G\left(z\right)\right))\right]$$

(5)

Figure 4. GANs Architecture [16].

Figure 4. GANs Architecture [16].

Both G and D are trained simultaneously. This algorithm allows for lock-step improvements to both G and D. Towards the conclusion of training, G becomes a powerful image generator which closely replicates the input space, i.e. training data.

GANs have a number of shortcomings that make them difficult to train. Due to the adversarial nature of GAN training the model can face the issue of Vanishing Gradients when the discriminator develops more quickly than the generator consequently correctly predicting every x and leaving no error to train on for the generator [17]. Another common issue is Mode Collapse when the generator learns to generate a particularly successful x such that the discriminator is consistently fooled and the generator continues to only produce that singular x and have no variability in image generation [17]. Both Vanishing Gradients and Mode Collapse are consequences of one of the adversarial models improving faster than the other.

2.1.4. Denoising Diffusion Probabilistic Model

Denoising Diffusion Probabilistic Models (DDPMs) are a recent development proposed by Jonathan Ho et al. (2020) inspired by nonequilibrium thermodynamics that produces high-fidelity image synthesis using a parameterized Markov chain [2]. Beginning with the training sample, each step of the Markov chain adds a single layer of Gaussian noise. A neural network is trained on parameterizing these additional Gaussian noise layers in order to reverse the process from random noise to a high-fidelity image.

Figure 5. DDPM Markov Chain [2].

Figure 5. DDPM Markov Chain [2].

$q_{\theta }\left(x_t\right|x_{t-1})$ represents the forward process, adding Gaussian noise, and $p_{\theta }\left(x_{t-1}\right|x_t)$ represents the reverse process, denoising. The reverse process is captured by training.

$$p_{\theta }\left(x_0\right):=\int p_{\theta }\left(x_{0:T}\right)d{x}_{1:T}$$

(6)

where

$$p_{\theta }\left(x_{0:T}\right):=p\left(x_T\right)\prod _{t=1}^T{p}_{\theta }\left(x_{t-1}\right|x)$$

(7)

and

$$p_{\theta }\left(x_{t-1}\right|x):=\mathcal{N}(x_{t-1};{\mu }_{\theta }(x_t,t);{\mathsf{\Sigma }}_{\theta }(x_t,t))$$

(8)

For clarity, we remind the reader that $\mathcal{N}(x_{t-1};{\mu }_{\theta }(x_t,t);{\mathsf{\Sigma }}_{\theta }(x_t,t))$ is the normal distribution with mean ${\mu }_{\theta }(x_t,t)$ and covariance matrix ${\mathsf{\Sigma }}_{\theta }(x_t,t)$. The loss function for a DDPM is as follows:

$$L:={\mathbf{E}}_q[-\log p\left(x_T\right)-\sum _{t\ge 1}\log \frac{p_{\theta }\left(x_{t-1}\right|x)}{q_{\theta }\left(x\right|x_{t-1})}]$$

(9)

Using a U-Net, a CNN with upsampling, with stochastic gradient descent and $T=1000$ Ho et al. were able to generate samples with an impressive, but not state-of-the-art, FID score of 0.317 on the CIFAR10 dataset. On CelebA-HQ 256 x 256 the team was able to generate the novel images in Figure 6.

In 2021, Dhariwal et al at OpenAI made improvements upon the original DDPM parameters and achieved state-of-the-art FID scores of 2.97 on ImageNet 128×128, 4.59 on ImageNet 256×256, and 7.72 on ImageNet 512×512 [3].

The first improvement is to not set ${\mathsf{\Sigma }}_{\theta }(x_t,t)$ as a constant but rather as the following:

$${\mathsf{\Sigma }}_{\theta }(x_t,t)=\exp (v\log {\beta }_t+(1-v)\log \widetilde{{\beta }_t})$$

(10)

Where ${\beta }_t$ and $\widetilde{{\beta }_t}$ correspond to the upper and lower bounds of the Gaussian variance.

Dhariwal et al also explore the following architectural changes; note: attention heads refer to embedding blocks in the U-Net [3]:

• Increasing depth versus width, holding model size relatively constant.
• Increasing the number of attention heads.
• Using attention at 32×32, 16×16, and 8×8 resolutions rather than only at 16×16.
• Using the BigGAN residual block for upsampling and downsampling the activations, following
• Rescaling residual connections with $\frac{1}{\sqrt{2}}$

With these changes, Dhariwal et al. were able to demonstrate their DDPM beating GANs in every single class by FID score and establishing DDPMs as the new state-of-the-art for image synthesis [3].

3. Methods

3.2. Data

We utilize a standardized dataset,CIFAR-10, for experiments. The CIFAR-10 dataset consists of sixty thousand 32 by 32 three-channel (color) images in ten uniform classes [21]. The data was initially collected in 2009 by Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton and has become the standard for machine learning research relating to computer vision [22]. One of the primary reasons CIFAR-10 is so popular is because the small image sizes allow for quick training and testing of new models [23]. In addition, the ubiquity of testing models on CIFAR-10 allows researchers to quickly benchmark their model performance against prior research [23].

Figure 9. 10 random images from each class of CIFAR-10 with respective class labels [21].

Figure 9. 10 random images from each class of CIFAR-10 with respective class labels [21].

The images in CIFAR-10 are exclusive to photographs of discrete distinct objects on a generally neutral background. The dataset contains photographs, which are 2-dimensional projections of 3-dimensional objects, from various angles.

3.6. Metrics

3.6.1. Inception Score

Inception score measures two primary attributes of the generated images: (i) the fidelity of the images, i.e. the image distinctly belongs to a particular class and (ii) the diversity of the generated images [24]. The Inception classifier is a convolutional neural network (CNN) built by Google and trained on the ImageNet dataset consisting of 14 million images and 1000 classes [25].

(i) Fidelity is captured by the probability distribution produced as classification output by the Inception classifier on a generated image [24]. Note, that a highly skewed distribution with a single peak indicates that the Inception classifier is able to identify the image as belonging to a specific class with high confidence. Therefore the image is high fidelity.

(ii) Diversity is captured by summing all the probability distributions produced for individually generated classes. The uniform nature of the resultant sum of distributions is indicative of the diversity of generated images. E.g. a model trained on CIFAR-10 that only manages to produce high-fidelity images of dogs would severely fail to capture diversity.

The average of the K-L divergences between the produced probability distribution and the summed distribution is the final Inception score, capturing both diversity and fidelity. Rigorously, each generated image $x_i$ is classified using the Inception Classifier to obtain the probability distribution $p\left(y\right|x_i)$ over classes y[26]. Calculate the marginal distribution is provided by:

$$p\left(y\right)=\frac{1}{N}\sum _{i=1}^{N}p\left(y\right|x_i)$$

(15)

Compute the KL Divergence:

$$D_{\mathrm{KL}}(p\left(y\right|x_i)\left|\right|p\left(y\right))=\sum _{y}p\left(y\right|x_i)log\left(\frac{p\left(y\right|x_i)}{p\left(y\right)}\right)\mathrm{[26]}$$

(16)

Take the expected value of these KL Divergences values over all N generated images:

$${\mathbb{E}}_x[D_{\mathrm{KL}}\left(p\left(y\right|x)\right|\left|p\left(y\right)\right)]=\frac{1}{N}\sum _{i=1}^N{D}_{\mathrm{KL}}(p\left(y\right|x_i)\left|\right|p\left(y\right))\mathrm{[26]}$$

(17)

Finally we exponentiate the above value to evaluate an Inception score:

$$\mathrm{IS}\left(G\right)=exp(E_{x\sim pg}{\mathcal{D}}_{KL}\left(p\left(y\right|x)\right|\left|p\left(y\right)\right)\mathrm{[26]}$$

(18)

3.6.2. Fréchet Inception Distance (FID)

Fréchet Inception Distance improves upon the inception score by capturing the relationship between the generated images against the training images, whereas the inception score only captures the characteristics of the generated images against each other and their classifications. The Inception classifier, used to determine the Inception score, also embeds a feature vector. I.e. the architecture of the Inception Classifier captures the salient features of the images it is trained on.

The FID score is determined by taking the Wasserstein metric between the two multivariate Gaussian distributions of the feature vectors for the training and generated images on the Inception model [27]. Simply, the dissimilarity between the features found in the training and generated data. This is an improvement upon inception score since it captures the higher level features that would be more human identifiable when comparing model performance. The Gaussian distributions of the feature vector for the generated images and the training images are $N(\mu ,\mathsf{\Sigma })$ and $N({\mu }_w,{\mathsf{\Sigma }}_w)$ respectively [28]. The Wasserstein metric, resulting in the FID score is as follows:

$$\mathrm{FID}=\left|\right|\mu -{\mu }_w{\left|\right|}_2^2+tr(\mathsf{\Sigma }+{\mathsf{\Sigma }}_w-2{\left({\mathsf{\Sigma }}^{1/2}{\mathsf{\Sigma }}_w{\mathsf{\Sigma }}^{1/2}\right)}^{1/2})\mathrm{[28]}$$

(19)

3.6.3. Kernel Inception Distance (KID)

Kernel Inception Distance measures the maximum mean discrepancy of the distributions of training and generated images by randomly sampling from them both [29]. KID does not specifically account for differences in high-level features and rather compares the raw distributions more directly.

Specifically, for generator X with probability measure $\mathbb{P}$ and random variable Y with probability measure $\mathbb{Q}$ we have:

$${\mathcal{D}}_F(\mathbb{P},\mathbb{Q})=\underset{f\in F}{sup}{\mathbb{E}}_{\mathbb{P}}f\left(X\right)-{\mathbb{E}}_{\mathbb{Q}}f\left(Y\right)\mathrm{[29]}$$

(20)

3.6.4. Quantitative Metrics

The following table summarizes the different metrics we used to evaluate our models:

Table 2. Summary of quantitative metrics for generative image synthesis evaluation [26,28,29].

Table 2. Summary of quantitative metrics for generative image synthesis evaluation [26,28,29].

Metric	Description	Performance
Inception	KL-Divergence between conditional and marginal label distributions over generated data	Higher is better
FID	Wasserstein-2 distance between multivariate Gaussians fitted to data embedded into a feature space	Lower is better
KID	Measures the dissimilarity between two probability distributions $P_r$ and $P_g$ using samples drawn independently from each distribution	Lower is better

3.6.5. Qualitative Metrics

Our qualitative evaluation was performed by analyzing the visual discernment of generated images in relation to their respective classes in a less stringent manner. This approach aims to foster a broader discussion about the applicability of such models and their effectiveness.

4. Results

4.4. Denoising Diffusion Probabilistic Model (DDPM)

The quality of the results for the DDPM is limited by the computational power available to run the experiment. DDPMs have been shown to be state-of-the-art for image generation when scored on fidelity, but require several hours of training on a Tensor Processing Unit (TPU). A TPU can perform one to two orders of magnitude greater operations than an equivalent GPU [30]. Without access to these Google-exclusive TPUs, we were unable to replicate state-of-the-art generation results.

Figure 14. DDPM generated image synthesis output from random noise inputs.

Figure 14. DDPM generated image synthesis output from random noise inputs.

5. Analysis

6. Conclusion & Future Work

References

1. Weng, L. What are diffusion models? lilianweng.github.io/lil-log.
2. Ho, J.; Jain, A.; Abbeel, P. 2020; arXiv:cs.LG/2006.11239].
3. Dhariwal, P.; Nichol, A. 2021; arXiv:cs.LG/2105.05233].
4. Jain, S.; Ziauddin, J.; Leonchyk, P.; Yenkanchi, S.; Geraci, J. Quantum and classical machine learning for the classification of non-small-cell lung cancer patients. SN Applied Sciences 2020, 2. [Google Scholar] [CrossRef]
5. Thulasidasan, S. Generative Modeling for Machine Learning on the D-Wave. Technical report, 2016. [CrossRef]
6. Amin, M.H.; Andriyash, E.; Rolfe, J.; Kulchytskyy, B.; Melko, R. Quantum Boltzmann Machine. Physical Review X 2018, 8. [Google Scholar] [CrossRef]
7. Xiao, Z.; Kreis, K.; Vahdat, A. Tackling the Generative Learning Trilemma with Denoising Diffusion GANs. In Proceedings of the International Conference on Learning Representations (ICLR); 2022. [Google Scholar]
8. Smolensky, P. Information processing in dynamical systems: foundations of harmony theory. 1986.
9. Freund, Y.; Haussler, D. Unsupervised learning of distributions on binary vectors using two layer networks. In Proceedings of the Advances in Neural Information Processing Systems; Moody, J.; Hanson, S.; Lippmann, R., Eds. Morgan-Kaufmann, Vol. 4. 1991. [Google Scholar]
10. Hopfield, J.J. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences 1982, 79, 2554–2558. [Google Scholar] [CrossRef] [PubMed]
11. Hinton, G.E. A Practical Guide to Training Restricted Boltzmann Machines. In Lecture Notes in Computer Science; Springer Berlin Heidelberg, 2012; pp. 599–619. [CrossRef]
12. Carreira-Perpiñán, M.Á.; Hinton, G.E. On Contrastive Divergence Learning. In Proceedings of the AISTATS; 2005. [Google Scholar]
13. Kingma, D.P.; Welling, M. 2014; arXiv:stat.ML/1312.6114].
14. Rocca, J. Understanding variational autoencoders (VAES), 2021.
15. Goodfellow, I.J.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; Bengio, Y. 2014; arXiv:stat.ML/1406.2661].
16. A beginner’s guide to generative adversarial networks (gans).
17. Common problems | generative adversarial networks | google developers.
18. Hinton, G.E. Training Products of Experts by Minimizing Contrastive Divergence. Neural Comput. 2002, 14, 1771–1800. [Google Scholar] [CrossRef] [PubMed]
19. What is quantum annealing?
20. Sleeman, J.; Dorband, J.; Halem, M. A Hybrid Quantum enabled RBM Advantage: Convolutional Autoencoders For Quantum Image Compression and Generative Learning 2020.
21. Krizhevsky, A.; Nair, V.; Hinton, G. CIFAR-10 (Canadian Institute for Advanced Research).
22. Krizhevsky, A. Learning multiple layers of features from tiny images. Technical report, 2009.
23. Peter Eckersley, Y.N.e.a. EFF AI Progress Measurement Project, 2017.
24. Mack, D. A simple explanation of the inception score, 2019.
25. Szegedy, C.; Vanhoucke, V.; Ioffe, S.; Shlens, J.; Wojna, Z. Rethinking the Inception Architecture for Computer Vision, 2015. [CrossRef]
26. Salimans, T.; Goodfellow, I.; Zaremba, W.; Cheung, V.; Radford, A.; Chen, X. Improved Techniques for Training GANs, 2016. [CrossRef]
27. Heusel, M.; Ramsauer, H.; Unterthiner, T.; Nessler, B.; Hochreiter, S. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In Proceedings of the Advances in Neural Information Processing Systems; Guyon, I.; Luxburg, U.V.; Bengio, S.; Wallach, H.; Fergus, R.; Vishwanathan, S.; Garnett, R., Eds. Curran Associates, Inc., Vol. 30. 2017. [Google Scholar]
28. Heusel, M.; Ramsauer, H.; Unterthiner, T.; Nessler, B.; Hochreiter, S. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium 2017. [CrossRef]
29. Bińkowski, M.; Sutherland, D.J.; Arbel, M.; Gretton, A. Demystifying MMD GANs, 2018. [CrossRef]
30. Cloud tensor processing units (tpus) | google cloud.
31. Dhillon, P.S.; Foster, D.; Ungar, L. Transfer Learning Using Feature Selection, 2009. [CrossRef]