High resolution imaging is a learning problem

I built ImAGeD (Imaging As Gradient Descent), a system that reconstructs a high-resolution microscope image from a series of low-resolution captures, without giving up field of view. Each capture shows the same sample lit from a different angle, and each angle reveals detail the others miss. No single image resolves more than the lens allows, but together they contain enough information to reconstruct one with higher resolution.

The reconstruction treats the sample as the weights of a model and the microscope as its architecture. A differentiable simulation of the optics turns those weights into predicted images; we compare them against what the sensor actually recorded, and gradient descent updates the sample until the two agree.

This technique is called Fourier ptychography, and it isn't new.⁠^[1][1]1Zheng, Horstmeyer and Yang. “Wide-field, high-resolution Fourier ptychographic microscopy.” Nature Photonics, 2013. The classic reconstruction algorithms, like Gerchberg–Saxton⁠^[2][2]2Gerchberg and Saxton. “A practical algorithm for the determination of phase from image and diffraction plane pictures.” Optik, 1972. and ePIE,⁠^[3][3]3Maiden and Rodenburg. “An improved ptychographical phase retrieval algorithm for diffractive imaging.” Ultramicroscopy, 2009. are hand-derived update rules: elegant, but change the model and you're back to deriving equations. Machine learning has turned iterative updates to a pile of numbers into a commodity, so I wondered what would happen if I treated this as an ML problem instead.

Autograd made the model cheap to change, and that exposed the real problem. The textbook physics is slightly wrong (stray light, imperfect light sources, an imperfect lens), and with so little data, the optimizer exploits every mismatch. The fix turned out to be the opposite of what I expected: not more constraint, but carefully placed freedom. Letting the model disagree with the physics in a few controlled places is what made the reconstructions work. When data is this scarce, the architecture is where you add information.

To understand why that works, we need a little optics.

The problem with lenses

(Almost) every image in the world is captured with the same basic setup: light from a scene enters a lens, which focuses it onto a digital sensor that measures the amount of light reaching each pixel. In a microscope, light usually shines through the sample rather than bouncing off it. By adjusting the distances between the sample and the lens, and the lens and the sensor, we can adjust the zoom of the image; however, the amount of information in the image, the number of useful pixels (which in optics is called the space–bandwidth product), stays constant. If you zoom in, you get finer detail, but the field of view shrinks.

When light hits the sample, it diffracts. Finer features diffract the light at sharper angles, while coarse features leave the light mostly on track. As a result, each direction of light corresponds to a different spatial frequency: the propagating light effectively computes a Fourier transform for free. Higher diffraction angles mean higher spatial frequencies. But only a limited cone of light passes through the lens (the size of which is defined by its numerical aperture, which is a function of its geometry); it acts as a low-pass filter. When the filtered light forms an image, the result is a blur. The camera only measures the intensity of the light, but the light itself is a complex value, with both an amplitude and a phase. Intensity keeps the amplitude and throws the phase away.

Writing it out, with capital letters representing the Fourier transform of the corresponding lowercase quantity:

$$U(\mathbf{k}) = M(\mathbf{k}) \odot S(\mathbf{k}) \qquad\ \hat{y} = |u|^2$$

Here $s = \mathcal{F}^{-1}\{S\}$ is the sample (the weights of our model), $u = \mathcal{F}^{-1}\{U\}$ is the light that reaches the sensor after passing through the lens, and $\hat{y}$ is the predicted image. In an ideal lens, $M$ is a disc-shaped binary mask, 1 inside and 0 out. Its radius is determined by the numerical aperture of the lens.

This means that an ever-larger lens is needed to capture higher resolution images with more useful pixels. But what if we don't have a larger lens?

The trick

If we light the sample from an angle instead of straight from below, all of the diffracted light tilts with it. A different cone of angles now lands in the lens, so in Fourier space the spectrum slides relative to the fixed aperture. Some high frequencies now get through, and some low frequencies get cut. The result is a different image, capturing different information about the sample.

$$U_{j}(\mathbf{k}) = M(\mathbf{k}) \odot S(\mathbf{k-k_{j}}) \qquad\ \hat{y}_{j} = |u_{j}|^2$$

Here, $\mathbf{k_j}$ is the frequency shift that the $j$-th illumination angle yields. We do this physically by using a grid of LEDs that we alternately turn on beneath the sample. In my setup I used 145 LEDs arranged in a polar grid of 7 concentric rings, with the widest angle being about 18º. In most of the captures where $\mathbf{k_j}$ is suitably large, direct light doesn't even pass through the lens. Transparent regions that look white in a normal image turn black, and only edges and fine detail light up.

Neighboring angles shift the spectrum by less than the width of the aperture, so the discs overlap, and every frequency is seen in several captures. That redundancy matters. Intensity depends on the relative phases of the frequencies inside the disc, so changing which frequencies share a capture lets the full set pin down phase information that no single image records. By capturing images under each of these angles, we build a dataset that covers a much wider synthetic aperture than any single capture ever can.

Gradient descent

The classic algorithms, like Gerchberg–Saxton and ePIE, alternate between two steps: force each predicted image to match its measured amplitude, solve for the portion of the sample spectrum that produced it, and then stitch together into a complete sample. This is elegant, but each new variant of the physics needs its own hand-derived update. Autograd removes that step entirely. Write the forward model, and the gradients come for free.

I'm not the first to notice this. Fourier ptychography has been written in TensorFlow,⁠^[4][4]4Jiang, Guo, Liao and Zheng. “Solving Fourier ptychographic imaging problems via neural network modeling and TensorFlow.” Biomedical Optics Express, 2018. and mathematically, the classic algorithms are effectively gradient descent in disguise.⁠^[5][5]5Zuo, Sun and Chen. “Adaptive step-size strategy for noise-robust Fourier ptychographic microscopy.” Optics Express, 2016. What I wanted was a setup where changing the model cost nothing, to see where that led.

Training means finding the sample that makes this virtual microscope reproduce the real one. I compute the loss on the square root of the images rather than the images themselves, using a small offset $\epsilon$ so that pixels near zero don't get near-infinite weight:

$$L \sim \sum_{j}{\lVert\sqrt{\hat{y}_{j} + \epsilon} - \sqrt{y_{j} + \epsilon}\rVert^2}$$

This trains much faster, and likely for a simple reason: the far-angle captures where DC is filtered out are much darker than low-angle captures, so a loss on raw intensity is dominated by the few bright, low-angle captures, which don't even carry the high-frequency data we're after. The square root evens that out; it's also the same quantity the classic algorithms enforce.

Once the model is trained, we swap the mask. Replacing $M$ with $M'$, a disc the size of the synthetic aperture, simulates a lens we never had:

$$U'(\mathbf{k}) = M'(\mathbf{k}) \odot S(\mathbf{k}) \qquad\ \hat{y}' = |u'|^2$$

And the result is that much higher resolution image we were after! We combine the information available across the dataset into a single image that simulates one captured with a higher-numerical-aperture lens.

At least, that's the theory.

Why it fails

When I used this system on simulated data, everything worked beautifully. But with real-world data, something was very clearly wrong. Loss dropped as expected, but the reconstruction was covered in a blotchy, repeated cheetah-print texture. This artifact appeared consistently, and contributed only a very specific series of frequencies to the result. In frequency space, it showed up as concentric bands that matched the frequency shifts imposed by the concentric rings of LEDs.

Initially, I tried to add regularization that suppressed those frequencies in the output. I thought the issue might have been noise in the input images, and it turns out I was partly correct. There was noise, but the noise itself wasn't the issue. The issue was that the textbook forward model was a misspecified simulator of a very messy real world. We were making a hidden assumption that all light entering the camera came from the sample, through an ideal lens, from ideal light sources. The real microscope differs from the ideal physics description in subtle, but critical ways:

• Stray light from the environment, and scattered light reflecting off the interior of the microscope.
• Uneven LED brightnesses.
• Defects and optical aberrations in the cheap lens.

Naively I thought at first this could have been solved with more data, but the problem is much more fundamental. The architecture of our model imposes a very strong inductive bias on the structure of the solution, and in the real world, that bias is wrong. So the optimizer answers a slightly different question than the one we asked. Not "what is the sample?" but "what sample would make a perfect microscope produce these images?" That sample doesn't exist, so it invents one.

Fixing the architecture

The standard answer I found in the literature is more constraint. One popular version models the sample as the output of a CNN and learns the network's weights instead of the pixel grid directly.⁠^[6][6]6Chen, Huang and Chen. “Fourier ptychographic microscopy with untrained deep neural network priors.” Optics Express, 2022. The network's structure acts as a prior, steering the optimizer toward natural-looking images and away from degenerate ones. I implemented this myself, and it did remove the artifacts. It also blurred away the resolution I was after. In hindsight, that makes sense: it constrained the sample, the one part of the model that was already right. The error lived in the microscope.

So instead of adding more constraints, what if we added more freedom? Let the model of the microscope be as imperfect as the real one.

Not unlimited freedom, though. With only about four data points per parameter, every new parameter has to earn its place. What I discovered is that we can give each known source of error its own parameters, shaped by what we know about how that error behaves. I added three.

The lens: The first isn't new. Learning the mask $M$ that actually models the real lens has been done before,⁠^[7][7]7Ou, Zheng and Yang. “Embedded pupil function recovery for Fourier ptychographic microscopy.” Optics Express, 2014. and can be implemented pretty easily: just set the mask tensor to be learnable along with the sample. A real lens isn't a clean disc. Imperfections delay light unevenly across the aperture, and a delay is a phase, so the mask needs to be complex-valued, not just on or off. The naive approach learns it as a free pixel grid like the sample, but that doubles the parameter count for something we already know a lot about. This area of optics is extremely well studied, and lens distortions tend to occur in a series of smooth, well-characterized shapes described by a sequence of Zernike polynomials.⁠^[8][8]8Mahajan. “Zernike Circle Polynomials and Optical Aberrations of Systems with Circular Pupils.” Applied Optics, 1994. If we decompose the mask tensor into a vector of Zernike coefficients, we can define it almost perfectly with only a handful of parameters. I also let the model learn the radius of the mask, which turned out to be crucial. The manufacturer's spec for the numerical aperture was off by about 15%.

LED brightness: A scalar multiple ($g_j$) defines the brightness of each LED. The LEDs I used weren't perfectly matched, so without this, a brighter capture would have wrongly told the optimizer that the sample was more transparent at that angle. This was a pretty trivial modification, but it helped.

Stray light: Not all the light reaching the sensor comes through the sample and lens; some leaks in or bounces around inside the microscope. That light doesn't interfere with the image-forming light; it simply adds to the intensity. I model it with one learned number per capture: a uniform glow added to every pixel ($b_j$). A constant can't draw any structure, so it can only soak up light that has none, and it can't steal signal from the sample. This is what fixed the cheetah-print texture: switch it off, and the texture comes back. For each predicted image $\hat{y}_j$, we model:

$$\hat{y}_j = g_j \left| u_j \right|^2 + b_j$$

With the real-world error now having a place to go, the sample is free to learn just the sample. The result is a reconstruction that increases the resolution of our captured data without sacrificing field-of-view. The critical lever I found in this process was the careful selection of how free our added parameterization was. If I added too much model capacity, it starts to memorize the dataset as the error terms absorb signal from the sample. Too little, and it can't learn the error sources we are trying to model. By identifying the sources of uncertainty in my simulator, I was able to give the model freedom to learn its own limitations. Where I was confident there wasn't uncertainty, I took it away to maximize the amount of model capacity directed at our problem. For example, I also tried to learn the positions of each LED, but this proved unnecessary because my measurements were accurate enough, so I turned it off.

Evaluation

How can we prove this process actually recovers real high-resolution signal from the data, and isn't simply hallucinating something that looks like it's correct? There were two main metrics I used, and both rely on capturing a dataset from a sample we know exactly.

MTF: The modulation transfer function, or MTF, of an imaging system describes how much of the sample's contrast survives at each spatial frequency.⁠^[9][9]9Coltman. “The Specification of Imaging Properties by Response to a Sine Wave Input.” Journal of the Optical Society of America, 1954. A value of 1 means that a signal at that frequency is perfectly recorded in the image, and 0 means it's completely lost. Normally this function takes on a decreasing curve with high values near DC, declining to 0 at the resolution limit. If the reconstruction works, we should see this function raised, with its $x$-intercept pushed into higher frequencies. This shows how well information in the sample gets preserved in the reconstruction. It served as my performance metric.

NPS: The noise power spectrum, or NPS, of an imaging system describes how much noise arises in an imaging system.⁠^[10][10]10Dobbins III, Samei, Ranger and Chen. “Intercomparison of methods for image quality characterization. II. Noise power spectrum.” Medical Physics, 2006. It's also a function of frequency: take an image of something we know to be constant, and see what frequencies we measure. Since we know the input is pure DC, every other component must come from noise. Ideally, it's as low and as flat as possible, with any elevations showing data in the final image that didn't come from the sample. It shows how well a lack of information is preserved in the reconstruction. It served as my anti-hallucination metric.

I used MTF to measure how well my system enhanced the output resolution. When I added the CNN regularizer, the blur effect was numerically obvious here as the MTF curve fell at high frequency. But too much benchmaxxing leads to another issue: hallucination. It's very easy to get a high value if the model makes up signal where it shouldn't. By also tracking NPS, I was able to guard against this, because hallucinated signal shows up in constant regions. I needed to monitor both metrics while iterating on the architecture.

Finally, it was crucial to visually inspect the results across a series of different datasets: both several captures of the same sample, and several different samples. I used several different line patterns, plus images of blood and other cells which have a very different frequency spectrum, letting me evaluate across a heterogeneous set of inputs. The same hyperparameters worked across all samples, and weren't sensitive to the microscope setup, so the architecture seems to generalize and gave me better results than using any of the existing solutions.

What still doesn't work

There are a few things that don't work as well as I'd like.

The sample is assumed to be thin: The forward model treats the sample as two-dimensional: a single flat layer that the light passes through once. That's fine for the samples I had access to, but a thicker sample scatters light more than once on the way through, and a single layer can't describe that. The approach I'd use is to model the sample as a three-dimensional voxel grid, with a function to model light propagation between slices, but I'm sure there are many additional complications to consider.

Memory demand: With 145 separate captures, this process needs about 1,400 bytes of GPU-resident memory per output pixel. Computing the entire 8-megapixel frame at once would be about 180 GB, and a frustrating ~170 GB of that is overhead from FFTs and the backward pass. I haven't found a good way around it yet, so I split the frame into smaller patches, which works but leaves visible seams where neighboring patches disagree.

Single-channel only: The image sensor captures color through a pattern of red, green and blue pixels, but my system is limited to processing a single color channel only. To do a full-color reconstruction, we'd need three separate reconstructions, one per channel, because the way light diffracts depends on its wavelength.

Data quality: The peak resolution we can achieve is largely dependent on the high-angle captures that don't receive much light from the LED. Currently, the maximum signal in those images is only slightly above what I estimate the actual sensor noise to be, which limits the fidelity of the information it can give us. To increase that ratio, I would re-capture the datasets using a longer exposure time for the images taken at higher angles: signal scales with exposure faster than noise does. I'm quite confident that doing so would enable a much higher resolution enhancement than the ~20% I was able to achieve.

All code is available at https://github.com/rspcunningham/imaged.

Notes

1. Zheng, Horstmeyer and Yang. “Wide-field, high-resolution Fourier ptychographic microscopy.” Nature Photonics, 2013.

2. Gerchberg and Saxton. “A practical algorithm for the determination of phase from image and diffraction plane pictures.” Optik, 1972.

3. Maiden and Rodenburg. “An improved ptychographical phase retrieval algorithm for diffractive imaging.” Ultramicroscopy, 2009.

4. Jiang, Guo, Liao and Zheng. “Solving Fourier ptychographic imaging problems via neural network modeling and TensorFlow.” Biomedical Optics Express, 2018.

5. Zuo, Sun and Chen. “Adaptive step-size strategy for noise-robust Fourier ptychographic microscopy.” Optics Express, 2016.

6. Chen, Huang and Chen. “Fourier ptychographic microscopy with untrained deep neural network priors.” Optics Express, 2022.

7. Ou, Zheng and Yang. “Embedded pupil function recovery for Fourier ptychographic microscopy.” Optics Express, 2014.

8. Mahajan. “Zernike Circle Polynomials and Optical Aberrations of Systems with Circular Pupils.” Applied Optics, 1994.

9. Coltman. “The Specification of Imaging Properties by Response to a Sine Wave Input.” Journal of the Optical Society of America, 1954.

10. Dobbins III, Samei, Ranger and Chen. “Intercomparison of methods for image quality characterization. II. Noise power spectrum.” Medical Physics, 2006.

@misc{cunningham2026imagingislearning,
  author = {Robin Cunningham},
  title = {High resolution imaging is a learning problem},
  year = {2026},
  howpublished = {www.rspcunningham.com},
  url = {https://www.rspcunningham.com/imaging-is-learning},
}