What the Detector Could Not Record

A diffraction detector counts photons, so it measures intensity, which is the squared modulus of the object's Fourier transform. The phase is not recorded, and at these wavelengths no device could record it. Oppenheim and Lim showed in 1981 that the phase is the half that carries what a person recognises. So the measurement keeps the half that does not look like the thing. This page throws the phase away in front of you, computes the picture back out of what is left, measures the point at which that stops being possible, and then does the whole thing to a real pattern recorded in 2006 from a sample that the measuring pulse destroyed.

Everything below runs in your browser. The transforms, the reconstructions, the error curves and the resolution estimate are all computed here, now, from files you can read. Numbers taken from published papers are marked and cited where they appear. Nothing is uploaded.

I

Which half is the picture

Take a photograph and Fourier transform it. Every frequency comes back as a complex number, which has a magnitude and a phase. The magnitude says how much of that ripple is present. The phase says where its crests sit.

It is not obvious that either half should matter more. Oppenheim and Lim settled it by an experiment anyone can repeat: take two pictures, give each one the other's magnitude, and see which one you recognise.

It is shown that the Fourier transform phase, in many cases, captures the intelligibility of a signal better than the magnitude. A. V. Oppenheim and J. S. Lim, "The importance of phase in signals", Proceedings of the IEEE 69(5), 529-541 (1981)

Here are two photographs in the public domain, reduced to 256 by 256 grey. The page transforms both, swaps the halves, and transforms back. Nothing is chosen for you: press the button and watch the arithmetic happen.

The number, and why the obvious number is useless

It is tempting to put a correlation coefficient on this. We did, and it failed in an instructive way. Correlate each hybrid against each donor and the two swaps come back with the same score, to about twelve decimal places, and the page prints the actual difference rather than asking you to take a digit count on trust. That agreement is not a coincidence and it is not a bug. Writing A and B for the two images and using Parseval's theorem, the correlation of the hybrid with its phase donor reduces algebraically to

corr( |A| with phase of B , B )  =  Σ|A_k||B_k| ⁄ √( Σ|A_k|² Σ|B_k|² )

which is symmetric in A and B. The plain correlation cannot tell the two swaps apart, because it is really measuring how similar the two magnitude spectra are, and natural images all have roughly the same falling spectrum. An instrument that returns the same number for two different questions is not measuring either of them.

So the page reports a second number that is not degenerate: the correlation of the gradient magnitude fields, which is dominated by edges, and edges are where phase lives. Both numbers are computed below, and the identity above is checked to machine precision, because a claim like that is worth nothing unless it is tested.

II

And a detector keeps the other one

Put an object in a coherent beam and let the light travel far enough, and what arrives at a distant screen is the Fourier transform of the object. This is not a metaphor or an approximation you have to be told to accept: it is what the far field is.

The screen is a detector, and a detector counts photons. What it can record is how many arrived, which is proportional to the squared modulus. The phase is a property of a field oscillating at around 10¹⁶ times a second, and nothing you can build responds that fast. The half of the transform that carries the picture is not thrown away by carelessness. It is not recordable.

III

Counting what you have against what you want

Half the transform is gone, so it looks as though half the information is gone, and the problem looks hopeless. The counting argument says otherwise, and it is worth doing slowly because everything below rests on it.

Suppose the object lives on an N × N grid but is known to be zero outside a smaller region, its support, of S pixels. Then the unknowns are the S values inside the support, not all N² of them. The measurements are the N² recorded moduli. Define

σ = N² / S

and you have σ measurements for every unknown. For a complex object each pixel hides two real numbers, so the equations only outnumber the unknowns when σ is greater than about two. That is the published threshold:

The phase information is in principle encoded in the diffraction pattern when the ratio of the number of independent equations to the number of unknown variables is larger than one, which corresponds to an oversampling ratio σ > 2 in two dimensions. after J. Miao, D. Sayre and H. N. Chapman, Journal of the Optical Society of America A 15, 1662-1669 (1998)

There is a second and stricter statement in the literature that is easy to confuse with the first, and the two are not in conflict. Sampling the diffraction intensity finely enough that the autocorrelation of the object does not wrap around itself requires a factor of two on each axis, which in two dimensions is σ ≥ 4. One statement counts equations, the other prevents aliasing. They give different numbers because they are different requirements.

We are not going to take either of them on trust. Section V measures where recovery actually starts, with the same solver, on the same object, under a protocol fixed before any result was seen.

IV

The instrument

There is no formula that inverts a modulus. What there is instead is a pair of facts you can alternate between until they agree.

Fact one: whatever the object is, the modulus of its transform is the thing you measured. Fact two: whatever the object is, it is zero outside the support. Neither fact on its own picks out an answer. Together, applied over and over, they usually do.

Each fact is a projection. Given any current guess, the Fourier projection transforms it, replaces every modulus with the measured one while keeping the phase it currently has, and transforms back. The support projection sets everything outside the support to zero. Applying them alternately is the error-reduction algorithm, and it is the obvious thing to do. It is also, on its own, poor: it is a local search and it stalls.

Fienup's hybrid input-output algorithm is the fix that made the field work, and its trick is counterintuitive. Outside the support, instead of setting the value to zero it steps away from the violation by a controlled amount, feeding the error back rather than deleting it. That single change turns a method that stalls into one that escapes.

Run one. The starting phases are random, so this is a genuine search and not a replay: change the seed and you get a different path to the same place, or sometimes to a different place, which is the subject of section VI.

V

Where recovery actually starts

Section III quoted two thresholds from the literature. This section tests them, and the protocol below was written down before any result was looked at, because a threshold you go looking for is a threshold you will find.

The second arm is the interesting one. The counting argument in section III doubled the unknowns because a complex pixel hides two real numbers. If the object is known to be real and non-negative, that factor of two goes away, and the threshold should move. Nothing in the literature we quoted tells us where it moves to. This is the page finding out.

This curve is not computed while you wait, because it is hundreds of independent reconstructions and that is many minutes of your machine rather than a few seconds. It is computed by the verifier from the same retrieve.js this page loads, with the seeds recorded, so it is reproducible rather than merely reported. The protocol panel above shows the seed count of the file that actually loaded, so it cannot claim a number the data does not have.

The same protocol was also run by a second, independent implementation, in Python and numpy, sharing no line of code with ours and using a different Fourier transform underneath. It is committed beside the verifier as sweep-numpy-crosscheck.py with its output, at thirty seeds per point. Two implementations agreeing matters more here than one implementation running longer, because the thing most likely to be wrong in a result like this is not the sample size, it is the code.

You can also check any single point yourself, live, with the button below, and if your browser disagrees with our table we want to know.

VI

The answer that is not unique, and costs nothing

Three transformations of an object leave the modulus of its transform exactly unchanged. You can multiply the whole object by a constant phase. You can slide it cyclically. And you can replace it by the complex conjugate of its own point reflection, which is a picture rotated by half a turn.

These are not approximate symmetries or numerical accidents. They are exact, and no amount of data or iteration will break them, because the data is identical for both. A reconstruction that comes back upside down is not a failure. It is the other correct answer.

VII

A pattern from an object that no longer exists

Everything so far has been done to pictures we made the patterns from, which means we always had the answer in our pocket. This section does not.

In 2006 a group at the FLASH free-electron laser in Hamburg put a 20 nanometre thick silicon nitride membrane in the beam. A picture had been cut clean through the membrane with a focused ion beam, inside a 20 micrometre square window. They fired a single pulse: 25 femtoseconds long, around 10¹² photons at 32 nanometres, focused hard enough that the sample reached about 60,000 kelvin and turned into plasma.

The pattern below is what came back from that one pulse. The sample it came from was destroyed by the pulse that made it. There is no going back to measure it again, and the reconstruction is the only image of it that can now exist.

An intense 25 fs, 4 × 10¹³ W/cm² pulse, containing 10¹² photons at 32 nm wavelength, produced a coherent diffraction pattern from a nano-structured non-periodic object, before destroying it at 60,000 °K. H. N. Chapman et al., "Femtosecond diffractive imaging with a soft-X-ray free-electron laser", Nature Physics 2, 839-843 (2006)

The data is deposited in the Coherent X-ray Imaging Data Bank as entry 3, under a CC0 public domain waiver, and this page ships it: 1300 by 1340 sixteen-bit counts, repacked losslessly from the depositor's file. The SHA-256 of both is printed at the foot of this page so you can check that we did not touch it.

Two things the file does not tell you

Where the beam was. The deposited file records an image centre, and it is the geometric centre of the array, which is a fact about the array rather than a measurement of the experiment. So the page finds the beam itself, using a property the pattern must have: for a real object the intensity is symmetric under a half turn about the zero frequency, which is Friedel's law again. Scan the assumed centre, correlate the pattern against its own half turn, and the true centre is where the correlation peaks. It is a sharp peak, and the page shows you the whole landscape rather than just the answer.

Which pixels are real. The mirror had a hole in it so the direct beam could pass through and not destroy the camera, so the lowest spatial frequencies were not recorded. Worse, the paper says the mirror's reflectivity "diminishes smoothly to zero close to the edge of the central hole", so there is a ring just outside the hole where the counts are real photons attenuated by an unknown factor. Feeding those to the algorithm as if they were the truth is a way to poison the low frequencies while every error metric stays green. The page marks that whole region as unmeasured and lets the algorithm put whatever it likes there, which is what Chapman and colleagues did too.

The support, and the number that fooled us

Section III said the support is a claim about the object rather than a measurement of it. Here is what that costs when you get it wrong, and it is not a hypothetical: it is what the first working version of this page did.

It started the reconstruction with the support set to the whole array, intending shrinkwrap to tighten it. That puts σ at exactly 1. At σ = 1 the object has more free parameters than there are recorded moduli, so any magnitude can be matched exactly just by choosing phases, and there is nothing left for the data to determine. The run drove its Fourier error down to about 3.5 × 10^-16, which is machine precision, on a real experimental diffraction pattern. It looked like the best reconstruction ever performed. It was noise.

The honest starting support comes from the deposited geometry and nothing else. The field of view is λD/p = 88.0 micrometres and does not depend on the array size, and the specimen is a 20 micrometre square window, so the object occupies 22.7 per cent of the array width. The page starts a little wider than that and lets shrinkwrap tighten it, prints σ live while the run happens, and refuses its own number if shrinkwrap ever walks the support below the point section V measured as recoverable.

One more constraint we do not impose, because it would be false. It is tempting to tell the solver the object is real and non-negative, since that is a strong constraint and the counting argument likes it. But the specimen is a transmission function: the membrane passes about 44 per cent of the light with a phase advance of roughly twenty degrees, and the holes milled through it pass all of it. That object is complex, and Chapman and colleagues say in their methods that they did not constrain it to be real or positive either. Imposing it here costs about a factor of three in the final Fourier error, which is what a false constraint looks like from the inside: not an error message, just a worse answer.

How good is it, in nanometres

The honest way to answer is the phase retrieval transfer function. Run the reconstruction many times from different random starts, align them, and ask at each spatial frequency whether the independent runs agreed on the phase there. Where they did, the vectors add and the ratio is one. Where they did not, they cancel and it falls to zero. The convention is to call the resolution the point where it drops to 1/e.

VIII

Where it stops working

An instrument that only ever succeeds is not an instrument, it is a demonstration. Here are the three ways this one fails, each of which you can drive yourself.

Stagnation, and the object that will not come back

HIO does not converge in the sense a numerical analyst would want. It wanders, and sometimes it wanders into a place it cannot leave, with a Fourier error that has stopped improving and a picture that is wrong. This is why the success fraction over independent starts is the number that matters, and why every reconstruction on this page reports it.

The sharper version of that, and the thing this page did not expect to find: at the same oversampling ratio, with the same solver and the same iteration budget, one object comes back to fifteen decimal places and another does not come back at all. Nine scattered rectangles at σ = 8.09 reach an aligned residual around 10^-15. The object called window, which is a near-uniform square frame with a small figure inside it and is deliberately shaped like the real specimen in section VII, sits at an aligned residual between 0.30 and 0.49 on every one of twenty-four independent starts, while its Fourier error falls to about 10^-2 and looks respectable. Press the button and watch it happen.

Nothing in the counting argument of section III sees this. Both objects have the same number of unknowns and the same number of equations. What differs is whether the search can get there, and that is a property of the object, not of the data. It is also, we think, most of the reason section VII is as hard as it turns out to be: the FLASH specimen is a mostly-uniform window, and the mostly-uniform window is the case that resists.

The hole in the middle

Low spatial frequencies carry the overall shape. Take enough of them away and the reconstruction keeps the fine detail and loses the form, which looks like a picture with the lighting removed. Drag the unmeasured radius in section VII and watch it happen.

Photons

A diffraction pattern's dynamic range is enormous, and the outer parts of it are made of very few counts. Add Poisson noise at a stated photon budget and the high frequencies stop being informative before they stop being present, and the PRTF says so before the picture does.

IX

Your own picture, with its phase thrown away

Drop in any image. The page will transform it, keep only the modulus, discard the phase in front of you, and then try to get your picture back from what is left. It will run several independent starts and tell you how many of them worked, and whether the one it shows you came back inverted.

Nothing leaves your machine. The file is read with a FileReader and drawn to a canvas, and the pixels never go anywhere.

X

What was checked

Every claim on this page is recomputed in Node by research/what-the-detector-could-not-record/verify.mjs, from the same module files your browser just loaded, and it fails on any disagreement.

Sources, and what we changed