Super-Resolution with GANs

The problem: upscaling loses information, by definition

Taking a low-resolution image and producing a larger, sharper version — super-resolution — sounds like it should just be interpolation. Resize a 64x64 image to 256x256 with bicubic interpolation and you do get a bigger image, but it's blurry: interpolation can only smoothly guess pixel values between known ones, it cannot invent texture, sharp edges, or fine detail that was never captured in the first place. The information is genuinely gone. The question super-resolution models answer isn't 'what were the missing pixels,' which is unanswerable, but 'what plausible high-resolution image would have downsampled to this low-resolution input.'

Why plain CNNs produce blurry results

Early deep learning approaches to super-resolution trained a CNN to directly minimize pixel-wise mean squared error (MSE) between its output and the real high-resolution image. This works, but it has a structural flaw: when a region's true high-resolution appearance is genuinely ambiguous (a stretch of grass could plausibly have many different blade textures), the MSE-optimal prediction is the *average* of all plausible outcomes — which looks smooth and blurry rather than textured and sharp. Minimizing average pixel error systematically produces images that look safe rather than realistic.

SRGAN: adversarial training as the fix

SRGAN (Super-Resolution GAN) reframes the problem with an adversarial setup. A generator network upscales the low-resolution input. A discriminator network is trained simultaneously to distinguish the generator's outputs from genuine high-resolution photographs. The generator is trained to fool the discriminator, which pushes it toward producing images with realistic high-frequency detail and texture — even if that detail is a plausible hallucination rather than the exact original content — because blurry, averaged-out images are trivially easy for the discriminator to spot as fake.

Perceptual loss: comparing feature maps, not raw pixels

SRGAN doesn't rely on the adversarial loss alone — it combines it with a perceptual loss. Instead of comparing generated and real images pixel-by-pixel, perceptual loss passes both images through a pretrained CNN (classically VGG) and compares their activations at a chosen intermediate layer. Because deep CNN features represent texture and structure rather than exact pixel values, matching features produces results that look right to a human even when individual pixels differ substantially from the ground truth. This loss anchors the generator to the actual input content, while the adversarial loss pushes it toward realistic texture.

Pixel-loss CNN vs. GAN-based super-resolution

The PSNR paradox

A counterintuitive result in super-resolution research: GAN-based models frequently score *worse* on Peak Signal-to-Noise Ratio (PSNR), the classic pixel-accuracy metric, than blurrier MSE-trained models — yet human evaluators consistently rate the GAN outputs as more realistic. This is a direct consequence of the blur-equals-safe-average problem: PSNR rewards being close to the true pixels on average, which rewards blur, not sharpness. It's a clear example of a benchmark metric diverging from the quality humans actually perceive, which is why super-resolution papers increasingly report human preference studies and perceptual metrics (like LPIPS) alongside or instead of PSNR.

What's next

SRGAN's adversarial and perceptual loss ideas generalize well beyond upscaling — the same generator-vs-discriminator framework underlies neural style transfer's cousins and many other image-to-image translation tasks. Continue to neural style transfer to see a related but distinct way of manipulating image content using CNN feature representations.