Cycle Generative Adversarial Network (CycleGAN)

Cycle Generative Adversarial Network (CycleGAN)

CycleGAN is a GAN design for image-to-image translation that does not need paired training examples. Two generators and two discriminators map images between domains, and cycle consistency loss encourages reconstructing the original image after a round trip.

Paired vs Unpaired images

• Performs image translation without paired images
• Uses two generators and two discriminators
• Learns reversible mappings between image domains
• Uses cycle consistency loss for reconstruction
• Applied in style transfer and image transformation tasks

Architecture of CycleGAN

CycleGAN relies on two generators and two discriminators to translate between domains without paired data.

1. Generators

Produce new images in the target style.

CycleGAN defines two generators, G and F:

• G maps images from domain X (for example, photos) into domain Y (for example, artwork).
• F maps images from domain Y back into domain X.

The generator mapping functions are:

\[ \begin{array}{l} G : X \rightarrow Y \\ F : Y \rightarrow X \end{array} \]

where X is the input image distribution and Y is the desired output distribution, such as Van Gogh styles.

2. Discriminators

Judge whether images are real (from the dataset) or fake (generated). The two discriminators are Dₓ and Dᵧ.

• Dₓ separates real images from X from images produced by F(y).
• Dᵧ separates real images from Y from images produced by G(x).

Cycle Consistency Loss

Beyond adversarial loss, CycleGAN adds two further losses to regularize the mappings.

• Forward Cycle Consistency Loss: Ensures that applying G and then F returns approximately the original image

For example: \(x \rightarrow G(x) \rightarrow F(G(x)) \approx x\)

• Backward Cycle Consistency Loss: Ensures that applying F and then G returns approximately the original image.

For example: \(y \xrightarrow{F} F(y) \xrightarrow{G} G(F(y)) \approx y\)

Generator Architecture

Each CycleGAN generator uses an encoder, a transformer, and a decoder for image translation.

• Encoder: The input passes through three convolution layers that extract features and compress spatial size while increasing channel count. For example, a 256×256×3 image becomes 64×64×256 after this stage.
• Transformer: The encoded representation goes through 6 or 9 residual blocks (depending on input size) so important image details are preserved.
• Decoder: Two deconvolution layers upsample the transformed features and restore the original spatial size.

Generator Structure:

c7s1-64 → d128 → d256 → R256 (×6 or 9) → u128 → u64 → c7s1-3

• c7s1-k: 7×7 convolution layer with k filters.
• dk: 3×3 convolution with stride 2 (down-sampling).
• Rk: Residual block with two 3×3 convolutions.
• uk: Fractional-stride deconvolution (up-sampling).

Discriminator Architecture (PatchGAN)

CycleGAN’s discriminator is a PatchGAN rather than a standard whole-image GAN discriminator.

• A conventional GAN discriminator inspects the full image (for example, 256×256 pixels) and emits a single real/fake score.
• PatchGAN divides the image into smaller patches (for example, 70×70). It outputs a grid of scores in which each value rates whether the matching patch looks real or fake.

Discriminator Structure

C64 → C128 → C256 → C512 → Final Convolution

• Ck: 4×4 convolution with k filters, InstanceNorm, and LeakyReLU, except in the first layer.
• The final layer yields a 1×1 output that marks real versus fake patches.

Cost Function in CycleGAN

Training uses a combined objective covering adversarial loss and cycle consistency loss.

1. Adversarial Loss

Adversarial loss pushes generators to create images realistic enough to fool the discriminators.

\[ Loss_{advers}\left ( G, D_y, X, Y \right ) =\frac{1}{m}\sum \left ( 1 - D_y\left ( G\left ( x \right ) \right ) \right )^{2} \]

\[ Loss_{advers}\left ( F, D_x, Y, X \right ) =\frac{1}{m}\sum \left ( 1 - D_x\left ( F\left ( y \right ) \right ) \right )^{2} \]

• Encourages generators to create realistic images
• Helps discriminators distinguish real and fake images

2. Cycle Consistency Loss

Cycle consistency loss requires that mapping an image to another domain and back reconstructs the original.

• Preserves important image content during translation
• Ensures meaningful reversible mappings between domains

\[ Loss_{cyc}\left ( G, F, X, Y \right ) =\frac{1}{m}\left [ \left ( F\left ( G\left ( x_i \right ) \right )-x_i \right ) +\left ( G\left ( F\left ( y_i \right ) \right )-y_i \right ) \right ] \]

The overall cost is the sum of adversarial loss and cycle consistency loss:

\[ L\left ( G, F, D_x, D_y \right ) = L_{advers}\left (G, D_y, X, Y \right ) + L_{advers}\left (F, D_x, Y, X \right ) + \lambda L_{cycl}\left ( G, F, X, Y \right ) \]

and the training objective is:

\[ arg \underset{G, F}{min}\underset{D_x, D_y}{max}L\left ( G, F, D_x, D_y \right ) \]

Applications

• Collection Style Transfer: CycleGAN can absorb styles from full artwork collections such as Van Gogh, Monet, and Cezanne, then generate images in those artistic styles.

• Object Transformation: CycleGAN can convert objects across classes—apples to oranges or zebras to horses—which is useful for image editing and content generation.

• Seasonal Transfer: CycleGAN can shift scenes across seasons, for example turning winter landscapes into summer ones and the reverse.

• Photo Generation from Paintings: CycleGAN can turn paintings into realistic photos and photos into artistic paintings, supporting artistic and editing workflows. An identity loss used in this setting is:

\[ L_{identity}\left ( G, F \right ) =\mathbb{E}_{y \sim p\left ( y \right )}\left [ \left \| G(y)-y \right \|_1 \right ] + \mathbb{E}_{x \sim p\left ( x \right )}\left [ \left \| F(x)-x \right \|_1 \right ] \]

• Photo Enhancement: CycleGAN can upgrade smartphone photos toward DSLR-like quality by improving visual fidelity and depth cues.

Evaluating CycleGAN’s Performance

Performance is assessed with both human judgment and quantitative metrics.

• AMT Perceptual Studies: Real users compare generated images with authentic ones to judge visual realism
• FCN Scores: Measure image understanding accuracy with metrics such as pixel accuracy and Intersection over Union (IoU)

Limitations

CycleGAN excels at texture and style changes but is weaker when large structural edits are required.

• Works better for changing textures and colors than object shapes
• Struggles with large structural modifications
• Generated images may sometimes contain distortions or unrealistic details
• Can produce unpredictable results in complex transformations