Journal papers
Ruiming Cao; Nikita S. Divekar; James K. Nuñez; Srigokul Upadhyayula; Laura Waller
Neural space–time model for dynamic multi-shot imaging Journal Article
In: Nature Methods, pp. 1–6, 2024, ISSN: 1548-7105, (Publisher: Nature Publishing Group).
Abstract | Links | BibTeX | Tags: Imaging, Phase-contrast microscopy, Super-resolution microscopy
@article{cao_neural_2024,
title = {Neural space–time model for dynamic multi-shot imaging},
author = {Ruiming Cao and Nikita S. Divekar and James K. Nuñez and Srigokul Upadhyayula and Laura Waller},
url = {https://www.nature.com/articles/s41592-024-02417-0},
doi = {10.1038/s41592-024-02417-0},
issn = {1548-7105},
year = {2024},
date = {2024-09-24},
urldate = {2024-09-01},
journal = {Nature Methods},
pages = {1--6},
abstract = {Computational imaging reconstructions from multiple measurements that are captured sequentially often suffer from motion artifacts if the scene is dynamic. We propose a neural space–time model (NSTM) that jointly estimates the scene and its motion dynamics, without data priors or pre-training. Hence, we can both remove motion artifacts and resolve sample dynamics from the same set of raw measurements used for the conventional reconstruction. We demonstrate NSTM in three computational imaging systems: differential phase-contrast microscopy, three-dimensional structured illumination microscopy and rolling-shutter DiffuserCam. We show that NSTM can recover subcellular motion dynamics and thus reduce the misinterpretation of living systems caused by motion artifacts.},
note = {Publisher: Nature Publishing Group},
keywords = {Imaging, Phase-contrast microscopy, Super-resolution microscopy},
pubstate = {published},
tppubtype = {article}
}
Computational imaging reconstructions from multiple measurements that are captured sequentially often suffer from motion artifacts if the scene is dynamic. We propose a neural space–time model (NSTM) that jointly estimates the scene and its motion dynamics, without data priors or pre-training. Hence, we can both remove motion artifacts and resolve sample dynamics from the same set of raw measurements used for the conventional reconstruction. We demonstrate NSTM in three computational imaging systems: differential phase-contrast microscopy, three-dimensional structured illumination microscopy and rolling-shutter DiffuserCam. We show that NSTM can recover subcellular motion dynamics and thus reduce the misinterpretation of living systems caused by motion artifacts.
Tiffany Chien; Ruiming Cao; Fanglin Linda Liu; Leyla A. Kabuli; Laura Waller
Space-time reconstruction for lensless imaging using implicit neural representations Journal Article
In: Opt. Express, vol. 32, no. 20, pp. 35725–35732, 2024.
Abstract | Links | BibTeX | Tags: Computational imaging; Imaging systems; Inverse design; Machine learning; Machine vision; Neural networks
@article{Chien:24,
title = {Space-time reconstruction for lensless imaging using implicit neural representations},
author = {Tiffany Chien and Ruiming Cao and Fanglin Linda Liu and Leyla A. Kabuli and Laura Waller},
url = {https://opg.optica.org/oe/abstract.cfm?URI=oe-32-20-35725},
doi = {10.1364/OE.530480},
year = {2024},
date = {2024-09-01},
journal = {Opt. Express},
volume = {32},
number = {20},
pages = {35725--35732},
publisher = {Optica Publishing Group},
abstract = {Many computational imaging inverse problems are challenged by noise, model mismatch, and other imperfections that decrease reconstruction quality. For data taken sequentially in time, instead of reconstructing each frame independently, space-time algorithms simultaneously reconstruct multiple frames, thereby taking advantage of temporal redundancy through space-time priors. This helps with denoising and provides improved reconstruction quality, but often requires significant computational and memory resources. Designing effective but flexible temporal priors is also challenging. Here, we propose using an implicit neural representation to model dynamics and act as a computationally tractable and flexible space-time prior. We demonstrate this approach on video captured with a lensless imager, DiffuserCam, and show improved reconstruction results and robustness to noise compared to frame-by-frame methods.},
keywords = {Computational imaging; Imaging systems; Inverse design; Machine learning; Machine vision; Neural networks},
pubstate = {published},
tppubtype = {article}
}
Many computational imaging inverse problems are challenged by noise, model mismatch, and other imperfections that decrease reconstruction quality. For data taken sequentially in time, instead of reconstructing each frame independently, space-time algorithms simultaneously reconstruct multiple frames, thereby taking advantage of temporal redundancy through space-time priors. This helps with denoising and provides improved reconstruction quality, but often requires significant computational and memory resources. Designing effective but flexible temporal priors is also challenging. Here, we propose using an implicit neural representation to model dynamics and act as a computationally tractable and flexible space-time prior. We demonstrate this approach on video captured with a lensless imager, DiffuserCam, and show improved reconstruction results and robustness to noise compared to frame-by-frame methods.
Guanghan Meng; Dekel Galor; Laura Waller; Martin S. Banks
BiPMAP: a toolbox for predicting perceived motion artifacts on modern displays Journal Article
In: Opt. Express, vol. 32, no. 7, pp. 12181–12199, 2024.
Abstract | Links | BibTeX | Tags:
@article{Meng:24,
title = {BiPMAP: a toolbox for predicting perceived motion artifacts on modern displays},
author = {Guanghan Meng and Dekel Galor and Laura Waller and Martin S. Banks},
url = {https://opg.optica.org/oe/abstract.cfm?URI=oe-32-7-12181},
doi = {10.1364/OE.510985},
year = {2024},
date = {2024-03-01},
urldate = {2024-03-01},
journal = {Opt. Express},
volume = {32},
number = {7},
pages = {12181--12199},
publisher = {Optica Publishing Group},
abstract = {Viewers of digital displays often experience motion artifacts (e.g., flicker, judder, edge banding, motion blur, color breakup, depth distortion) when presented with dynamic scenes. We developed an interactive software tool for display designers that predicts how a viewer perceives motion artifacts for a variety of stimulus, display, and viewing parameters: the Binocular Perceived Motion Artifact Predictor (BiPMAP). The tool enables the user to specify numerous stimulus, display, and viewing parameters. It implements a model of human spatiotemporal contrast sensitivity in order to determine which artifacts will be seen by a viewer and which will not. The tool visualizes the perceptual effects of discrete space-time sampling on the display by presenting side by side the expected perception when the stimulus is continuous compared to when the same stimulus is presented with the spatial and temporal parameters of a prototype display.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Viewers of digital displays often experience motion artifacts (e.g., flicker, judder, edge banding, motion blur, color breakup, depth distortion) when presented with dynamic scenes. We developed an interactive software tool for display designers that predicts how a viewer perceives motion artifacts for a variety of stimulus, display, and viewing parameters: the Binocular Perceived Motion Artifact Predictor (BiPMAP). The tool enables the user to specify numerous stimulus, display, and viewing parameters. It implements a model of human spatiotemporal contrast sensitivity in order to determine which artifacts will be seen by a viewer and which will not. The tool visualizes the perceptual effects of discrete space-time sampling on the display by presenting side by side the expected perception when the stimulus is continuous compared to when the same stimulus is presented with the spatial and temporal parameters of a prototype display.
Nathan Tessema Ersaro; Cem Yalcin; Liz Murray; Leyla Kabuli; Laura Waller; Rikky Muller
Fast non-iterative algorithm for 3D point-cloud holography Journal Article
In: Opt. Express, vol. 31, no. 22, pp. 36468–36485, 2023.
Abstract | Links | BibTeX | Tags: Diode pumped lasers; Fast Fourier transforms; Image quality; Phase retrieval; Spatial light modulators; Three dimensional imaging
@article{Ersaro:23,
title = {Fast non-iterative algorithm for 3D point-cloud holography},
author = {Nathan Tessema Ersaro and Cem Yalcin and Liz Murray and Leyla Kabuli and Laura Waller and Rikky Muller},
url = {https://opg.optica.org/oe/abstract.cfm?URI=oe-31-22-36468},
doi = {10.1364/OE.498302},
year = {2023},
date = {2023-10-01},
urldate = {2023-10-01},
journal = {Opt. Express},
volume = {31},
number = {22},
pages = {36468--36485},
publisher = {Optica Publishing Group},
abstract = {Recently developed iterative and deep learning-based approaches to computer-generated holography (CGH) have been shown to achieve high-quality photorealistic 3D images with spatial light modulators. However, such approaches remain overly cumbersome for patterning sparse collections of target points across a photoresponsive volume in applications including biological microscopy and material processing. Specifically, in addition to requiring heavy computation that cannot accommodate real-time operation in mobile or hardware-light settings, existing sampling-dependent 3D CGH methods preclude the ability to place target points with arbitrary precision, limiting accessible depths to a handful of planes. Accordingly, we present a non-iterative point cloud holography algorithm that employs fast deterministic calculations in order to efficiently allocate patches of SLM pixels to different target points in the 3D volume and spread the patterning of all points across multiple time frames. Compared to a matched-performance implementation of the iterative Gerchberg-Saxton algorithm, our algorithm’s relative computation speed advantage was found to increase with SLM pixel count, reaching >100,000x at 512 × 512 array format.},
keywords = {Diode pumped lasers; Fast Fourier transforms; Image quality; Phase retrieval; Spatial light modulators; Three dimensional imaging},
pubstate = {published},
tppubtype = {article}
}
Recently developed iterative and deep learning-based approaches to computer-generated holography (CGH) have been shown to achieve high-quality photorealistic 3D images with spatial light modulators. However, such approaches remain overly cumbersome for patterning sparse collections of target points across a photoresponsive volume in applications including biological microscopy and material processing. Specifically, in addition to requiring heavy computation that cannot accommodate real-time operation in mobile or hardware-light settings, existing sampling-dependent 3D CGH methods preclude the ability to place target points with arbitrary precision, limiting accessible depths to a handful of planes. Accordingly, we present a non-iterative point cloud holography algorithm that employs fast deterministic calculations in order to efficiently allocate patches of SLM pixels to different target points in the 3D volume and spread the patterning of all points across multiple time frames. Compared to a matched-performance implementation of the iterative Gerchberg-Saxton algorithm, our algorithm’s relative computation speed advantage was found to increase with SLM pixel count, reaching >100,000x at 512 × 512 array format.
Gautam Gunjala; Antoine Wojdyla; Kenneth A. Goldberg; Zhi Qiao; Xianbo Shi; Lahsen Assoufid; Laura Waller
Data-driven modeling and control of an X-ray bimorph adaptive mirror Journal Article
In: Journal of Synchrotron Radiation, vol. 30, no. 1, 2023.
Abstract | Links | BibTeX | Tags: adaptive optics, beamline optics, x ray imaging
@article{Gunjala:tv5041,
title = {Data-driven modeling and control of an X-ray bimorph adaptive mirror},
author = {Gautam Gunjala and Antoine Wojdyla and Kenneth A. Goldberg and Zhi Qiao and Xianbo Shi and Lahsen Assoufid and Laura Waller},
url = {https://doi.org/10.1107/S1600577522011080},
doi = {10.1107/S1600577522011080},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {Journal of Synchrotron Radiation},
volume = {30},
number = {1},
abstract = {Adaptive X-ray mirrors are being adopted on high-coherent-flux synchrotron and X-ray free-electron laser beamlines where dynamic phase control and aberration compensation are necessary to preserve wavefront quality from source to sample, yet challenging to achieve. Additional difficulties arise from the inability to continuously probe the wavefront in this context, which demands methods of control that require little to no feedback. In this work, a data-driven approach to the control of adaptive X-ray optics with piezo-bimorph actuators is demonstrated. This approach approximates the non-linear system dynamics with a discrete-time model using random mirror shapes and interferometric measurements as training data. For mirrors of this type, prior states and voltage inputs affect the shape-change trajectory, and therefore must be included in the model. Without the need for assumed physical models of the mirror's behavior, the generality of the neural network structure accommodates drift, creep and hysteresis, and enables a control algorithm that achieves shape control and stability below 2nm RMS. Using a prototype mirror and it ex situ metrology, it is shown that the accuracy of our trained model enables open-loop shape control across a diverse set of states and that the control algorithm achieves shape error magnitudes that fall within diffraction-limited performance.},
keywords = {adaptive optics, beamline optics, x ray imaging},
pubstate = {published},
tppubtype = {article}
}
Adaptive X-ray mirrors are being adopted on high-coherent-flux synchrotron and X-ray free-electron laser beamlines where dynamic phase control and aberration compensation are necessary to preserve wavefront quality from source to sample, yet challenging to achieve. Additional difficulties arise from the inability to continuously probe the wavefront in this context, which demands methods of control that require little to no feedback. In this work, a data-driven approach to the control of adaptive X-ray optics with piezo-bimorph actuators is demonstrated. This approach approximates the non-linear system dynamics with a discrete-time model using random mirror shapes and interferometric measurements as training data. For mirrors of this type, prior states and voltage inputs affect the shape-change trajectory, and therefore must be included in the model. Without the need for assumed physical models of the mirror's behavior, the generality of the neural network structure accommodates drift, creep and hysteresis, and enables a control algorithm that achieves shape control and stability below 2nm RMS. Using a prototype mirror and it ex situ metrology, it is shown that the accuracy of our trained model enables open-loop shape control across a diverse set of states and that the control algorithm achieves shape error magnitudes that fall within diffraction-limited performance.
Eric Li; Stuart Sherwin; Gautam Gunjala; Laura Waller
Exceeding the limits of algorithmic self-calibrated aberration recovery in Fourier ptychography Journal Article
In: Opt. Continuum, vol. 2, no. 1, pp. 119–130, 2023.
Abstract | Links | BibTeX | Tags: Computational imaging; Image quality; Imaging systems; Optical aberrations; Phase imaging; Reconstruction algorithms
@article{Li:23,
title = {Exceeding the limits of algorithmic self-calibrated aberration recovery in Fourier ptychography},
author = {Eric Li and Stuart Sherwin and Gautam Gunjala and Laura Waller},
url = {https://opg.optica.org/optcon/abstract.cfm?URI=optcon-2-1-119},
doi = {10.1364/OPTCON.475990},
year = {2023},
date = {2023-01-01},
journal = {Opt. Continuum},
volume = {2},
number = {1},
pages = {119--130},
publisher = {Optica Publishing Group},
abstract = {Fourier ptychographic microscopy is a computational imaging technique that provides quantitative phase information and high resolution over a large field-of-view. Although the technique presents numerous advantages over conventional microscopy, model mismatch due to unknown optical aberrations can significantly limit reconstruction quality. A practical way of correcting for aberrations without additional data capture is through algorithmic self-calibration, in which a pupil recovery step is embedded into the reconstruction algorithm. However, software-only aberration correction is limited in accuracy. Here, we evaluate the merits of implementing a simple, dedicated calibration procedure for applications requiring high accuracy. In simulations, we find that for a target sample reconstruction error, we can image without any aberration corrections only up to a maximum aberration magnitude of $łambda$/40. When we use algorithmic self-calibration, we can tolerate an aberration magnitude up to $łambda$/10 and with our proposed diffuser calibration technique, this working range is extended further to $łambda$/3. Hence, one can trade off complexity for accuracy by using a separate calibration process, which is particularly useful for larger aberrations.},
keywords = {Computational imaging; Image quality; Imaging systems; Optical aberrations; Phase imaging; Reconstruction algorithms},
pubstate = {published},
tppubtype = {article}
}
Fourier ptychographic microscopy is a computational imaging technique that provides quantitative phase information and high resolution over a large field-of-view. Although the technique presents numerous advantages over conventional microscopy, model mismatch due to unknown optical aberrations can significantly limit reconstruction quality. A practical way of correcting for aberrations without additional data capture is through algorithmic self-calibration, in which a pupil recovery step is embedded into the reconstruction algorithm. However, software-only aberration correction is limited in accuracy. Here, we evaluate the merits of implementing a simple, dedicated calibration procedure for applications requiring high accuracy. In simulations, we find that for a target sample reconstruction error, we can image without any aberration corrections only up to a maximum aberration magnitude of $łambda$/40. When we use algorithmic self-calibration, we can tolerate an aberration magnitude up to $łambda$/10 and with our proposed diffuser calibration technique, this working range is extended further to $łambda$/3. Hence, one can trade off complexity for accuracy by using a separate calibration process, which is particularly useful for larger aberrations.
Joseph D. Malone; Neerja Aggarwal; Laura Waller; Audrey K. Bowden
DiffuserSpec: spectroscopy with Scotch tape Journal Article
In: Opt. Lett., vol. 48, no. 2, pp. 323–326, 2023.
Abstract | Links | BibTeX | Tags: Near infrared radiation; Optical components; Reconstruction algorithms; Speckle patterns; Spectrometers; Spectroscopy
@article{Malone:23,
title = {DiffuserSpec: spectroscopy with Scotch tape},
author = {Joseph D. Malone and Neerja Aggarwal and Laura Waller and Audrey K. Bowden},
url = {https://opg.optica.org/ol/abstract.cfm?URI=ol-48-2-323},
doi = {10.1364/OL.476472},
year = {2023},
date = {2023-01-01},
journal = {Opt. Lett.},
volume = {48},
number = {2},
pages = {323--326},
publisher = {Optica Publishing Group},
abstract = {Computational spectroscopy breaks the inherent one-to-one spatial-to-spectral pixel mapping of traditional spectrometers by multiplexing spectral data over a given sensor region. Most computational spectrometers require components that are complex to design, fabricate, or both. DiffuserSpec is a simple computational spectrometer that uses the inherent spectral dispersion of commercially available diffusers to generate speckle patterns that are unique to each wavelength. Using Scotch tape as a diffuser, we demonstrate narrowband and broadband spectral reconstructions with 2-nm spectral resolution over an 85-nm bandwidth in the near-infrared, limited only by the bandwidth of the calibration dataset. We also investigate the effect of spatial sub-sampling of the 2D speckle pattern on resolution performance.},
keywords = {Near infrared radiation; Optical components; Reconstruction algorithms; Speckle patterns; Spectrometers; Spectroscopy},
pubstate = {published},
tppubtype = {article}
}
Computational spectroscopy breaks the inherent one-to-one spatial-to-spectral pixel mapping of traditional spectrometers by multiplexing spectral data over a given sensor region. Most computational spectrometers require components that are complex to design, fabricate, or both. DiffuserSpec is a simple computational spectrometer that uses the inherent spectral dispersion of commercially available diffusers to generate speckle patterns that are unique to each wavelength. Using Scotch tape as a diffuser, we demonstrate narrowband and broadband spectral reconstructions with 2-nm spectral resolution over an 85-nm bandwidth in the near-infrared, limited only by the bandwidth of the calibration dataset. We also investigate the effect of spatial sub-sampling of the 2D speckle pattern on resolution performance.
Yi Xue; David Ren; Laura Waller
Three-dimensional bi-functional refractive index and fluorescence microscopy (BRIEF) Journal Article
In: Biomed. Opt. Express, vol. 13, no. 11, pp. 5900–5908, 2022.
Abstract | Links | BibTeX | Tags: Digital imaging; Fluorescence microscopy; Image quality; Imaging techniques; Optical imaging; Three dimensional imaging
@article{Xue:22,
title = {Three-dimensional bi-functional refractive index and fluorescence microscopy (BRIEF)},
author = {Yi Xue and David Ren and Laura Waller},
url = {https://opg.optica.org/boe/abstract.cfm?URI=boe-13-11-5900},
doi = {10.1364/BOE.456621},
year = {2022},
date = {2022-11-01},
journal = {Biomed. Opt. Express},
volume = {13},
number = {11},
pages = {5900--5908},
publisher = {Optica Publishing Group},
abstract = {Fluorescence microscopy is a powerful tool for imaging biological samples with molecular specificity. In contrast, phase microscopy provides label-free measurement of the sample’s refractive index (RI), which is an intrinsic optical property that quantitatively relates to cell morphology, mass, and stiffness. Conventional imaging techniques measure either the labeled fluorescence (functional) information or the label-free RI (structural) information, though it may be valuable to have both. For example, biological tissues have heterogeneous RI distributions, causing sample-induced scattering that degrades the fluorescence image quality. When both fluorescence and 3D RI are measured, one can use the RI information to digitally correct multiple-scattering effects in the fluorescence image. Here, we develop a new computational multi-modal imaging method based on epi-mode microscopy that reconstructs both 3D fluorescence and 3D RI from a single dataset. We acquire dozens of fluorescence images, each ‘illuminated’ by a single fluorophore, then solve an inverse problem with a multiple-scattering forward model. We experimentally demonstrate our method for epi-mode 3D RI imaging and digital correction of multiple-scattering effects in fluorescence images.},
keywords = {Digital imaging; Fluorescence microscopy; Image quality; Imaging techniques; Optical imaging; Three dimensional imaging},
pubstate = {published},
tppubtype = {article}
}
Fluorescence microscopy is a powerful tool for imaging biological samples with molecular specificity. In contrast, phase microscopy provides label-free measurement of the sample’s refractive index (RI), which is an intrinsic optical property that quantitatively relates to cell morphology, mass, and stiffness. Conventional imaging techniques measure either the labeled fluorescence (functional) information or the label-free RI (structural) information, though it may be valuable to have both. For example, biological tissues have heterogeneous RI distributions, causing sample-induced scattering that degrades the fluorescence image quality. When both fluorescence and 3D RI are measured, one can use the RI information to digitally correct multiple-scattering effects in the fluorescence image. Here, we develop a new computational multi-modal imaging method based on epi-mode microscopy that reconstructs both 3D fluorescence and 3D RI from a single dataset. We acquire dozens of fluorescence images, each ‘illuminated’ by a single fluorophore, then solve an inverse problem with a multiple-scattering forward model. We experimentally demonstrate our method for epi-mode 3D RI imaging and digital correction of multiple-scattering effects in fluorescence images.
Henry Pinkard; Laura Waller
Microscopes are coming for your job Journal Article
In: Nature Methods, pp. 1–2, 2022.
@article{pinkard2022microscopes,
title = {Microscopes are coming for your job},
author = {Henry Pinkard and Laura Waller},
url = {https://www.nature.com/articles/s41592-022-01566-4},
year = {2022},
date = {2022-09-08},
urldate = {2022-01-01},
journal = {Nature Methods},
pages = {1--2},
publisher = {Nature Publishing Group},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Michael L. Whittaker; David Ren; Colin Ophus; Yugang Zhang; Laura Waller; Benjamin Gilbert; Jillian F. Banfield
Ion complexation waves emerge at the curved interfaces of layered minerals Journal Article
In: Nature Communications volume , vol. 13, iss. 1, 2022.
Abstract | Links | BibTeX | Tags: tomography
@article{Ion2022,
title = {Ion complexation waves emerge at the curved interfaces of layered minerals},
author = {Michael L. Whittaker and David Ren and Colin Ophus and Yugang Zhang and Laura Waller and Benjamin Gilbert and Jillian F. Banfield },
doi = {https://doi.org/10.1038/s41467-022-31004-0},
year = {2022},
date = {2022-06-13},
urldate = {2022-06-13},
journal = {Nature Communications volume },
volume = {13},
issue = {1},
abstract = {Visualizing hydrated interfaces is of widespread interest across the physical sciences and is a particularly acute need for layered minerals, whose properties are governed by the structure of the electric double layer (EDL) where mineral and solution meet. Here, we show that cryo electron microscopy and tomography enable direct imaging of the EDL at montmorillonite interfaces in monovalent electrolytes with ångstrom resolution over micron length scales. A learning-based multiple-scattering reconstruction method for cryo electron tomography reveals ions bound asymmetrically on opposite sides of curved, exfoliated layers. We observe conserved ion-density asymmetry across stacks of interacting layers in cryo electron microscopy that is associated with configurations of inner- and outer-sphere ion-water-mineral complexes that we term complexation waves. Coherent X-ray scattering confirms that complexation waves propagate at room-temperature via a competition between ion dehydration and charge interactions that are coupled across opposing sides of a layer, driving dynamic transitions between stacked and aggregated states via layer exfoliation.},
keywords = {tomography},
pubstate = {published},
tppubtype = {article}
}
Visualizing hydrated interfaces is of widespread interest across the physical sciences and is a particularly acute need for layered minerals, whose properties are governed by the structure of the electric double layer (EDL) where mineral and solution meet. Here, we show that cryo electron microscopy and tomography enable direct imaging of the EDL at montmorillonite interfaces in monovalent electrolytes with ångstrom resolution over micron length scales. A learning-based multiple-scattering reconstruction method for cryo electron tomography reveals ions bound asymmetrically on opposite sides of curved, exfoliated layers. We observe conserved ion-density asymmetry across stacks of interacting layers in cryo electron microscopy that is associated with configurations of inner- and outer-sphere ion-water-mineral complexes that we term complexation waves. Coherent X-ray scattering confirms that complexation waves propagate at room-temperature via a competition between ion dehydration and charge interactions that are coupled across opposing sides of a layer, driving dynamic transitions between stacked and aggregated states via layer exfoliation.
Ruiming Cao; Michael Kellman; David Ren; Regina Eckert; Laura Waller
Self-calibrated 3D differential phase contrast microscopy with optimized illumination Journal Article
In: Biomed. Opt. Express, vol. 13, no. 3, pp. 1671–1684, 2022.
Abstract | Links | BibTeX | Tags: Discrete Fourier transforms; Illumination design; Inverse problems; LED lighting; Phase contrast; Three dimensional imaging
@article{Cao:22,
title = {Self-calibrated 3D differential phase contrast microscopy with optimized illumination},
author = {Ruiming Cao and Michael Kellman and David Ren and Regina Eckert and Laura Waller},
url = {http://opg.optica.org/boe/abstract.cfm?URI=boe-13-3-1671},
doi = {10.1364/BOE.450838},
year = {2022},
date = {2022-03-01},
urldate = {2022-03-01},
journal = {Biomed. Opt. Express},
volume = {13},
number = {3},
pages = {1671--1684},
publisher = {OSA},
abstract = {3D phase imaging recovers an object’s volumetric refractive index from intensity and/or holographic measurements. Partially coherent methods, such as illumination-based differential phase contrast (DPC), are particularly simple to implement in a commercial brightfield microscope. 3D DPC acquires images at multiple focus positions and with different illumination source patterns in order to reconstruct 3D refractive index. Here, we present a practical extension of the 3D DPC method that does not require a precise motion stage for scanning the focus and uses optimized illumination patterns for improved performance. The user scans the focus by hand, using the microscope’s focus knob, and the algorithm self-calibrates the axial position to solve for the 3D refractive index of the sample through a computational inverse problem. We further show that the illumination patterns can be optimized by an end-to-end learning procedure. Combining these two, we demonstrate improved 3D DPC with a commercial microscope whose only hardware modification is LED array illumination.},
keywords = {Discrete Fourier transforms; Illumination design; Inverse problems; LED lighting; Phase contrast; Three dimensional imaging},
pubstate = {published},
tppubtype = {article}
}
3D phase imaging recovers an object’s volumetric refractive index from intensity and/or holographic measurements. Partially coherent methods, such as illumination-based differential phase contrast (DPC), are particularly simple to implement in a commercial brightfield microscope. 3D DPC acquires images at multiple focus positions and with different illumination source patterns in order to reconstruct 3D refractive index. Here, we present a practical extension of the 3D DPC method that does not require a precise motion stage for scanning the focus and uses optimized illumination patterns for improved performance. The user scans the focus by hand, using the microscope’s focus knob, and the algorithm self-calibrates the axial position to solve for the 3D refractive index of the sample through a computational inverse problem. We further show that the illumination patterns can be optimized by an end-to-end learning procedure. Combining these two, we demonstrate improved 3D DPC with a commercial microscope whose only hardware modification is LED array illumination.
Yi Xue; Laura Waller; Hillel Adesnik; Nicolas Pégard
Three-dimensional multi-site random access photostimulation (3D-MAP) Journal Article
In: eLife, vol. 2022, no. 11, pp. e73266, 2022.
Links | BibTeX | Tags: 3D imaging
@article{Xue:2022,
title = {Three-dimensional multi-site random access photostimulation (3D-MAP)},
author = {Yi Xue and Laura Waller and Hillel Adesnik and Nicolas Pégard},
doi = {10.7554/eLife.73266},
year = {2022},
date = {2022-02-14},
journal = {eLife},
volume = { 2022},
number = {11},
pages = {e73266},
keywords = {3D imaging},
pubstate = {published},
tppubtype = {article}
}
Vivek Boominathan; Jacob T Robinson; Laura Waller; Ashok Veeraraghavan
Recent advances in lensless imaging Journal Article
In: Optica, vol. 9, no. 1, pp. 1–16, 2022.
Abstract | Links | BibTeX | Tags: Charge coupled devices; Image processing; Imaging systems; Three dimensional imaging; Vertical cavity surface emitting lasers; X ray imaging, lensless imaging
@article{Boominathan:22,
title = {Recent advances in lensless imaging},
author = {Vivek Boominathan and Jacob T Robinson and Laura Waller and Ashok Veeraraghavan},
url = {http://www.osapublishing.org/optica/abstract.cfm?URI=optica-9-1-1},
doi = {10.1364/OPTICA.431361},
year = {2022},
date = {2022-01-01},
journal = {Optica},
volume = {9},
number = {1},
pages = {1--16},
publisher = {OSA},
abstract = {Lensless imaging provides opportunities to design imaging systems free from the constraints imposed by traditional camera architectures. Due to advances in imaging hardware, fabrication techniques, and new algorithms, researchers have recently developed lensless imaging systems that are extremely compact and lightweight or able to image higher-dimensional quantities. Here we review these recent advances and describe the design principles and their effects that one should consider when developing and using lensless imaging systems.},
keywords = {Charge coupled devices; Image processing; Imaging systems; Three dimensional imaging; Vertical cavity surface emitting lasers; X ray imaging, lensless imaging},
pubstate = {published},
tppubtype = {article}
}
Lensless imaging provides opportunities to design imaging systems free from the constraints imposed by traditional camera architectures. Due to advances in imaging hardware, fabrication techniques, and new algorithms, researchers have recently developed lensless imaging systems that are extremely compact and lightweight or able to image higher-dimensional quantities. Here we review these recent advances and describe the design principles and their effects that one should consider when developing and using lensless imaging systems.
Kyrollos Yanny; Kristina Monakhova; Richard W Shuai; Laura Waller
Deep learning for fast spatially varying deconvolution Journal Article
In: Optica, vol. 9, no. 1, pp. 96–99, 2022.
Abstract | Links | BibTeX | Tags: 3D imaging, Hyperspectral imaging; Image quality; Imaging systems; Neural networks; Reconstruction algorithms; Three dimensional reconstruction, spatially-varying
@article{Yanny:22,
title = {Deep learning for fast spatially varying deconvolution},
author = {Kyrollos Yanny and Kristina Monakhova and Richard W Shuai and Laura Waller},
url = {http://www.osapublishing.org/optica/abstract.cfm?URI=optica-9-1-96},
doi = {10.1364/OPTICA.442438},
year = {2022},
date = {2022-01-01},
journal = {Optica},
volume = {9},
number = {1},
pages = {96--99},
publisher = {OSA},
abstract = {Deconvolution can be used to obtain sharp images or volumes from blurry or encoded measurements in imaging systems. Given knowledge of the system's point spread function (PSF) over the field of view, a reconstruction algorithm can be used to recover a clear image or volume. Most deconvolution algorithms assume shift-invariance; however, in realistic systems, the PSF varies laterally and axially across the field of view due to aberrations or design. Shift-varying models can be used, but are often slow and computationally intensive. In this work, we propose a deep-learning-based approach that leverages knowledge about the system's spatially varying PSFs for fast 2D and 3D reconstructions. Our approach, termed MultiWienerNet, uses multiple differentiable Wiener filters paired with a convolutional neural network to incorporate spatial variance. Trained using simulated data and tested on experimental data, our approach offers a 625textminus1600texttimes increase in speed compared to iterative methods with a spatially varying model, and outperforms existing deep-learning-based methods that assume shift invariance.},
keywords = {3D imaging, Hyperspectral imaging; Image quality; Imaging systems; Neural networks; Reconstruction algorithms; Three dimensional reconstruction, spatially-varying},
pubstate = {published},
tppubtype = {article}
}
Deconvolution can be used to obtain sharp images or volumes from blurry or encoded measurements in imaging systems. Given knowledge of the system's point spread function (PSF) over the field of view, a reconstruction algorithm can be used to recover a clear image or volume. Most deconvolution algorithms assume shift-invariance; however, in realistic systems, the PSF varies laterally and axially across the field of view due to aberrations or design. Shift-varying models can be used, but are often slow and computationally intensive. In this work, we propose a deep-learning-based approach that leverages knowledge about the system's spatially varying PSFs for fast 2D and 3D reconstructions. Our approach, termed MultiWienerNet, uses multiple differentiable Wiener filters paired with a convolutional neural network to incorporate spatial variance. Trained using simulated data and tested on experimental data, our approach offers a 625textminus1600texttimes increase in speed compared to iterative methods with a spatially varying model, and outperforms existing deep-learning-based methods that assume shift invariance.
Sylvain Gigan; Ori Katz; Hilton Barbosa de Aguiar; Esben Andresen; Alexandre Aubry; Jacopo Bertolotti; Emmanuel Bossy; Dorian Bouchet; Josh Brake; Sophie Brasselet; Yaron Bromberg; Hui Cao; Thomas Chaigne; Zhongtao Cheng; Won-Shik Choi; Tomas Cizmar; Meng Cui; Vincent Curtis; Hugo Defienne; Matthias Hofer; Ryoichi Horisaki; Roarke Horstmeyer; Na Ji; Aaron LaViolette; Jerome Mertz; Christophe Moser; Allard P. Mosk; Nicolas Pégard; Rafael Piestun; Sébastien Popoff; Dave Phillips; D Psaltis; Babak Rahmani; Herve Rigneault; Stefan Rotter; Lei Tian; Ivo M Vellekoop; Laura Waller; Lihong V Wang; Timothy Weber; Sheng Xiao; Chris Xu; Alexey Yamilov; Changhuei Yang; Hasan Yılmaz
Roadmap on Wavefront Shaping and deep imaging in complex media Journal Article
In: Journal of Physics: Photonics, 2022.
Abstract | Links | BibTeX | Tags:
@article{10.1088/2515-7647/ac76f9,
title = {Roadmap on Wavefront Shaping and deep imaging in complex media},
author = {Sylvain Gigan and Ori Katz and Hilton Barbosa de Aguiar and Esben Andresen and Alexandre Aubry and Jacopo Bertolotti and Emmanuel Bossy and Dorian Bouchet and Josh Brake and Sophie Brasselet and Yaron Bromberg and Hui Cao and Thomas Chaigne and Zhongtao Cheng and Won-Shik Choi and Tomas Cizmar and Meng Cui and Vincent Curtis and Hugo Defienne and Matthias Hofer and Ryoichi Horisaki and Roarke Horstmeyer and Na Ji and Aaron LaViolette and Jerome Mertz and Christophe Moser and Allard P. Mosk and Nicolas Pégard and Rafael Piestun and Sébastien Popoff and Dave Phillips and D Psaltis and Babak Rahmani and Herve Rigneault and Stefan Rotter and Lei Tian and Ivo M Vellekoop and Laura Waller and Lihong V Wang and Timothy Weber and Sheng Xiao and Chris Xu and Alexey Yamilov and Changhuei Yang and Hasan Yılmaz},
url = {http://iopscience.iop.org/article/10.1088/2515-7647/ac76f9},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Journal of Physics: Photonics},
abstract = {The last decade has seen the development of a wide set of tools, such as wavefront shaping, computational or fundamental methods, that allow to understand and control light propagation in a complex medium, such as biological tissues or multimode fibers. A vibrant and diverse community is now working on this field, that has revolutionized the prospect of diffraction-limited imaging at depth in tissues. This roadmap highlights several key aspects of this fast developing field, and some of the challenges and opportunities ahead.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
The last decade has seen the development of a wide set of tools, such as wavefront shaping, computational or fundamental methods, that allow to understand and control light propagation in a complex medium, such as biological tissues or multimode fibers. A vibrant and diverse community is now working on this field, that has revolutionized the prospect of diffraction-limited imaging at depth in tissues. This roadmap highlights several key aspects of this fast developing field, and some of the challenges and opportunities ahead.
Kristina Monakhova; Vi Tran; Grace Kuo; Laura Waller
Untrained networks for compressive lensless photography Journal Article
In: Opt. Express, vol. 29, no. 13, pp. 20913–20929, 2021.
Links | BibTeX | Tags: compressive imaging, compressive photography, high speed video, hyperspectral imaging, Hyperspectral imaging; Image quality; Imaging systems; Optical imaging; Phase imaging; Three dimensional imaging, learning-based
@article{Monakhova:21,
title = {Untrained networks for compressive lensless photography},
author = {Kristina Monakhova and Vi Tran and Grace Kuo and Laura Waller},
url = {http://www.opticsexpress.org/abstract.cfm?URI=oe-29-13-20913},
doi = {10.1364/OE.424075},
year = {2021},
date = {2021-06-01},
journal = {Opt. Express},
volume = {29},
number = {13},
pages = {20913--20929},
publisher = {OSA},
keywords = {compressive imaging, compressive photography, high speed video, hyperspectral imaging, Hyperspectral imaging; Image quality; Imaging systems; Optical imaging; Phase imaging; Three dimensional imaging, learning-based},
pubstate = {published},
tppubtype = {article}
}
Henry Pinkard; Hratch Baghdassarian; Adriana Mujal; Ed Roberts; Kenneth H Hu; Daniel Haim Friedman; Ivana Malenica; Taylor Shagam; Adam Fries; Kaitlin Corbin; others
Learned adaptive multiphoton illumination microscopy for large-scale immune response imaging Journal Article
In: Nature communications, vol. 12, no. 1, pp. 1–14, 2021.
Links | BibTeX | Tags: learning-based, microscopy, multiphoton
@article{pinkard2021learned,
title = {Learned adaptive multiphoton illumination microscopy for large-scale immune response imaging},
author = {Henry Pinkard and Hratch Baghdassarian and Adriana Mujal and Ed Roberts and Kenneth H Hu and Daniel Haim Friedman and Ivana Malenica and Taylor Shagam and Adam Fries and Kaitlin Corbin and others},
doi = {https://doi.org/10.1038/s41467-021-22246-5},
year = {2021},
date = {2021-01-01},
journal = {Nature communications},
volume = {12},
number = {1},
pages = {1--14},
publisher = {Nature Publishing Group},
keywords = {learning-based, microscopy, multiphoton},
pubstate = {published},
tppubtype = {article}
}
Henry Pinkard; Nico Stuurman; Ivan E Ivanov; Nicholas M Anthony; Wei Ouyang; Bin Li; Bin Yang; Mark A Tsuchida; Bryant Chhun; Grace Zhang; others
Pycro-Manager: open-source software for customized and reproducible microscope control Journal Article
In: Nature Methods, vol. 18, no. 3, pp. 226–228, 2021.
Links | BibTeX | Tags: microsopy, open-source software
@article{pinkard2021pycro,
title = {Pycro-Manager: open-source software for customized and reproducible microscope control},
author = {Henry Pinkard and Nico Stuurman and Ivan E Ivanov and Nicholas M Anthony and Wei Ouyang and Bin Li and Bin Yang and Mark A Tsuchida and Bryant Chhun and Grace Zhang and others},
doi = {https://doi.org/10.1038/s41592-021-01087-6},
year = {2021},
date = {2021-01-01},
journal = {Nature Methods},
volume = {18},
number = {3},
pages = {226--228},
publisher = {Nature Publishing Group},
keywords = {microsopy, open-source software},
pubstate = {published},
tppubtype = {article}
}
Stuart Sherwin; Isvar Cordova; Ryan Miyakawa; Markus Benk; Laura Waller; Andrew Neureuther; Patrick Naulleau
Picometer sensitivity metrology for EUV absorber phase Journal Article
In: Journal of Micro/Nanopatterning, Materials, and Metrology, vol. 20, no. 3, pp. 1 – 19, 2021.
Links | BibTeX | Tags: attenuated phase-shift mask, Carbon, Contamination, Etching, EUV absorber, extreme ultraviolet, mask three-dimensional, Metrology, Modulation, Multilayers, phase measurement, Phase shift keying, photomask contamination, Reflection, Reflectivity, Reflectometry
@article{10.1117/1.JMM.20.3.031011,
title = {Picometer sensitivity metrology for EUV absorber phase},
author = {Stuart Sherwin and Isvar Cordova and Ryan Miyakawa and Markus Benk and Laura Waller and Andrew Neureuther and Patrick Naulleau},
url = {https://doi.org/10.1117/1.JMM.20.3.031011},
doi = {10.1117/1.JMM.20.3.031011},
year = {2021},
date = {2021-01-01},
journal = {Journal of Micro/Nanopatterning, Materials, and Metrology},
volume = {20},
number = {3},
pages = {1 -- 19},
publisher = {SPIE},
keywords = {attenuated phase-shift mask, Carbon, Contamination, Etching, EUV absorber, extreme ultraviolet, mask three-dimensional, Metrology, Modulation, Multilayers, phase measurement, Phase shift keying, photomask contamination, Reflection, Reflectivity, Reflectometry},
pubstate = {published},
tppubtype = {article}
}
Stuart Sherwin; Isvar A Cordova; Ryan H Miyakawa; Markus P Benk; Laura Waller; Andrew R Neureuther; Patrick Naulleau
Picometer sensitivity metrology for EUV absorber phase Journal Article
In: Journal of Micro/Nanopatterning, Materials, and Metrology, vol. 20, no. 3, pp. 1 – 19, 2021.
Links | BibTeX | Tags: attenuated phase-shift mask, Carbon, Contamination, Etching, EUV absorber, extreme ultraviolet, mask three-dimensional, Metrology, Modulation, Multilayers, phase measurement, Phase shift keying, photomask contamination, Reflection, Reflectivity, Reflectometry
@article{10.1117/1.JMM.20.3.031011b,
title = {Picometer sensitivity metrology for EUV absorber phase},
author = {Stuart Sherwin and Isvar A Cordova and Ryan H Miyakawa and Markus P Benk and Laura Waller and Andrew R Neureuther and Patrick Naulleau},
url = {https://doi.org/10.1117/1.JMM.20.3.031011},
doi = {10.1117/1.JMM.20.3.031011},
year = {2021},
date = {2021-01-01},
journal = {Journal of Micro/Nanopatterning, Materials, and Metrology},
volume = {20},
number = {3},
pages = {1 -- 19},
publisher = {SPIE},
keywords = {attenuated phase-shift mask, Carbon, Contamination, Etching, EUV absorber, extreme ultraviolet, mask three-dimensional, Metrology, Modulation, Multilayers, phase measurement, Phase shift keying, photomask contamination, Reflection, Reflectivity, Reflectometry},
pubstate = {published},
tppubtype = {article}
}
Michael Kellman; Kevin Zhang; Eric Markley; Jon Tamir; Emrah Bostan; Michael Lustig; Laura Waller
Memory-efficient Learning for Large-Scale Computational Imaging Journal Article
In: IEEE Transactions on Computational Imaging, vol. 6, pp. 1403-1414, 2020.
Links | BibTeX | Tags: algorithms, computational imaging, experimental design, learning-based, LED array, memory efficient, memory-efficient, physics-based
@article{kellman2020memory,
title = {Memory-efficient Learning for Large-Scale Computational Imaging},
author = { Michael Kellman and Kevin Zhang and Eric Markley and Jon Tamir and Emrah Bostan and Michael Lustig and Laura Waller},
url = {https://ieeexplore.ieee.org/document/9204455},
doi = {10.1109/TCI.2020.3025735},
year = {2020},
date = {2020-10-14},
journal = {IEEE Transactions on Computational Imaging},
volume = {6},
pages = {1403-1414},
keywords = {algorithms, computational imaging, experimental design, learning-based, LED array, memory efficient, memory-efficient, physics-based},
pubstate = {published},
tppubtype = {article}
}
Kyrollos Yanny; Nick Antipa; William Liberti; Sam Dehaeck; Kristina Monakhova; Fanglin Linda Liu; Konlin Shen; Ren Ng; Laura Waller
Miniscope3D: optimized single-shot miniature 3D fluorescence microscopy Journal Article
In: Light: Science & Applications, vol. 9, no. 171, 2020.
Abstract | Links | BibTeX | Tags: 3D imaging, algorithms, diffuser, fluorescence imaging
@article{yanny2020,
title = {Miniscope3D: optimized single-shot miniature 3D fluorescence microscopy},
author = {Kyrollos Yanny and Nick Antipa and William Liberti and Sam Dehaeck and Kristina Monakhova and Fanglin Linda Liu and Konlin Shen and Ren Ng and Laura Waller},
url = {https://www.nature.com/articles/s41377-020-00403-7},
doi = {https://doi.org/10.1038/s41377-020-00403-7},
year = {2020},
date = {2020-10-02},
journal = {Light: Science & Applications},
volume = {9},
number = {171},
abstract = {Miniature fluorescence microscopes are a standard tool in systems biology. However, widefield miniature microscopes capture only 2D information, and modifications that enable 3D capabilities increase the size and weight and have poor resolution outside a narrow depth range. Here, we achieve the 3D capability by replacing the tube lens of a conventional 2D Miniscope with an optimized multifocal phase mask at the objective’s aperture stop. Placing the phase mask at the aperture stop significantly reduces the size of the device, and varying the focal lengths enables a uniform resolution across a wide depth range. The phase mask encodes the 3D fluorescence intensity into a single 2D measurement, and the 3D volume is recovered by solving a sparsity-constrained inverse problem. We provide methods for designing and fabricating the phase mask and an efficient forward model that accounts for the field-varying aberrations in miniature objectives. We demonstrate a prototype that is 17 mm tall and weighs 2.5 grams, achieving 2.76 μm lateral, and 15 μm axial resolution across most of the 900 × 700 × 390 μm3 volume at 40 volumes per second. The performance is validated experimentally on resolution targets, dynamic biological samples, and mouse brain tissue. Compared with existing miniature single-shot volume-capture implementations, our system is smaller and lighter and achieves a more than 2× better lateral and axial resolution throughout a 10× larger usable depth range. Our microscope design provides single-shot 3D imaging for applications where a compact platform matters, such as volumetric neural imaging in freely moving animals and 3D motion studies of dynamic samples in incubators and lab-on-a-chip devices.},
keywords = {3D imaging, algorithms, diffuser, fluorescence imaging},
pubstate = {published},
tppubtype = {article}
}
Miniature fluorescence microscopes are a standard tool in systems biology. However, widefield miniature microscopes capture only 2D information, and modifications that enable 3D capabilities increase the size and weight and have poor resolution outside a narrow depth range. Here, we achieve the 3D capability by replacing the tube lens of a conventional 2D Miniscope with an optimized multifocal phase mask at the objective’s aperture stop. Placing the phase mask at the aperture stop significantly reduces the size of the device, and varying the focal lengths enables a uniform resolution across a wide depth range. The phase mask encodes the 3D fluorescence intensity into a single 2D measurement, and the 3D volume is recovered by solving a sparsity-constrained inverse problem. We provide methods for designing and fabricating the phase mask and an efficient forward model that accounts for the field-varying aberrations in miniature objectives. We demonstrate a prototype that is 17 mm tall and weighs 2.5 grams, achieving 2.76 μm lateral, and 15 μm axial resolution across most of the 900 × 700 × 390 μm3 volume at 40 volumes per second. The performance is validated experimentally on resolution targets, dynamic biological samples, and mouse brain tissue. Compared with existing miniature single-shot volume-capture implementations, our system is smaller and lighter and achieves a more than 2× better lateral and axial resolution throughout a 10× larger usable depth range. Our microscope design provides single-shot 3D imaging for applications where a compact platform matters, such as volumetric neural imaging in freely moving animals and 3D motion studies of dynamic samples in incubators and lab-on-a-chip devices.
Kristina Monakhova; Kyrollos Yanny; Neerja Aggarwal; Laura Waller
Spectral DiffuserCam: lensless snapshot hyperspectral imaging with a spectral filter array Journal Article
In: Optica, vol. 7, no. 10, pp. 1298–1307, 2020.
Abstract | Links | BibTeX | Tags: algorithm, compressed sensing, diffuser, Hyperspectral imaging; Imaging systems; Optical components; Optical design; Spectral imaging; Systems design, Image reconstruction, Image sensors, Inverse problems, Sensors
@article{Monakhova:20b,
title = {Spectral DiffuserCam: lensless snapshot hyperspectral imaging with a spectral filter array},
author = {Kristina Monakhova and Kyrollos Yanny and Neerja Aggarwal and Laura Waller},
url = {http://www.osapublishing.org/optica/abstract.cfm?URI=optica-7-10-1298},
doi = {10.1364/OPTICA.397214},
year = {2020},
date = {2020-10-01},
journal = {Optica},
volume = {7},
number = {10},
pages = {1298--1307},
publisher = {OSA},
abstract = {Hyperspectral imaging is useful for applications ranging from medical diagnostics to agricultural crop monitoring; however, traditional scanning hyperspectral imagers are prohibitively slow and expensive for widespread adoption. Snapshot techniques exist but are often confined to bulky benchtop setups or have low spatio-spectral resolution. In this paper, we propose a novel, compact, and inexpensive computational camera for snapshot hyperspectral imaging. Our system consists of a tiled spectral filter array placed directly on the image sensor and a diffuser placed close to the sensor. Each point in the world maps to a unique pseudorandom pattern on the spectral filter array, which encodes multiplexed spatio-spectral information. By solving a sparsity-constrained inverse problem, we recover the hyperspectral volume with sub-super-pixel resolution. Our hyperspectral imaging framework is flexible and can be designed with contiguous or non-contiguous spectral filters that can be chosen for a given application. We provide theory for system design, demonstrate a prototype device, and present experimental results with high spatio-spectral resolution.},
keywords = {algorithm, compressed sensing, diffuser, Hyperspectral imaging; Imaging systems; Optical components; Optical design; Spectral imaging; Systems design, Image reconstruction, Image sensors, Inverse problems, Sensors},
pubstate = {published},
tppubtype = {article}
}
Hyperspectral imaging is useful for applications ranging from medical diagnostics to agricultural crop monitoring; however, traditional scanning hyperspectral imagers are prohibitively slow and expensive for widespread adoption. Snapshot techniques exist but are often confined to bulky benchtop setups or have low spatio-spectral resolution. In this paper, we propose a novel, compact, and inexpensive computational camera for snapshot hyperspectral imaging. Our system consists of a tiled spectral filter array placed directly on the image sensor and a diffuser placed close to the sensor. Each point in the world maps to a unique pseudorandom pattern on the spectral filter array, which encodes multiplexed spatio-spectral information. By solving a sparsity-constrained inverse problem, we recover the hyperspectral volume with sub-super-pixel resolution. Our hyperspectral imaging framework is flexible and can be designed with contiguous or non-contiguous spectral filters that can be chosen for a given application. We provide theory for system design, demonstrate a prototype device, and present experimental results with high spatio-spectral resolution.
Bahram Javidi; Artur Carnicer; Jun Arai; Toshiaki Fujii; Hong Hua; Hongen Liao; Manuel Martínez-Corral; Filiberto Pla; Adrian Stern; Laura Waller; Qiong-Hua Wang; Gordon Wetzstein; Masahiro Yamaguchi; Hirotsugu Yamamoto
Roadmap on 3D integral imaging: sensing, processing, and display Journal Article
In: Opt. Express, vol. 28, no. 22, pp. 32266–32293, 2020.
Abstract | Links | BibTeX | Tags: Holographic displays; Image processing; Image quality; Integral photography; Low light levels; Spatial light modulators
@article{Javidi:20,
title = {Roadmap on 3D integral imaging: sensing, processing, and display},
author = {Bahram Javidi and Artur Carnicer and Jun Arai and Toshiaki Fujii and Hong Hua and Hongen Liao and Manuel Martínez-Corral and Filiberto Pla and Adrian Stern and Laura Waller and Qiong-Hua Wang and Gordon Wetzstein and Masahiro Yamaguchi and Hirotsugu Yamamoto},
url = {http://www.opticsexpress.org/abstract.cfm?URI=oe-28-22-32266},
doi = {10.1364/OE.402193},
year = {2020},
date = {2020-10-01},
journal = {Opt. Express},
volume = {28},
number = {22},
pages = {32266--32293},
publisher = {OSA},
abstract = {This Roadmap article on three-dimensional integral imaging provides an overview of some of the research activities in the field of integral imaging. The article discusses various aspects of the field including sensing of 3D scenes, processing of captured information, and 3D display and visualization of information. The paper consists of a series of 15 sections from the experts presenting various aspects of the field on sensing, processing, displays, augmented reality, microscopy, object recognition, and other applications. Each section represents the vision of its author to describe the progress, potential, vision, and challenging issues in this field.},
keywords = {Holographic displays; Image processing; Image quality; Integral photography; Low light levels; Spatial light modulators},
pubstate = {published},
tppubtype = {article}
}
This Roadmap article on three-dimensional integral imaging provides an overview of some of the research activities in the field of integral imaging. The article discusses various aspects of the field including sensing of 3D scenes, processing of captured information, and 3D display and visualization of information. The paper consists of a series of 15 sections from the experts presenting various aspects of the field on sensing, processing, displays, augmented reality, microscopy, object recognition, and other applications. Each section represents the vision of its author to describe the progress, potential, vision, and challenging issues in this field.
Fanglin Linda Liu; Grace Kuo; Nick Antipa; Kyrollos Yanny; Laura Waller
Fourier DiffuserScope: single-shot 3D Fourier light field microscopy with a diffuser Journal Article
In: Opt. Express, vol. 28, no. 20, pp. 28969–28986, 2020.
Abstract | Links | BibTeX | Tags: Light fields; Numerical simulation; Plenoptic imaging; Three dimensional imaging; Two photon polymerization; Wavefront encoding
@article{LindaLiu:20,
title = {Fourier DiffuserScope: single-shot 3D Fourier light field microscopy with a diffuser},
author = {Fanglin Linda Liu and Grace Kuo and Nick Antipa and Kyrollos Yanny and Laura Waller},
url = {http://www.opticsexpress.org/abstract.cfm?URI=oe-28-20-28969},
doi = {10.1364/OE.400876},
year = {2020},
date = {2020-09-01},
journal = {Opt. Express},
volume = {28},
number = {20},
pages = {28969--28986},
publisher = {OSA},
abstract = {Light field microscopy (LFM) uses a microlens array (MLA) near the sensor plane of a microscope to achieve single-shot 3D imaging of a sample without any moving parts. Unfortunately, the 3D capability of LFM comes with a significant loss of lateral resolution at the focal plane. Placing the MLA near the pupil plane of the microscope, instead of the image plane, can mitigate the artifacts and provide an efficient forward model, at the expense of field-of-view (FOV). Here, we demonstrate improved resolution across a large volume with Fourier DiffuserScope, which uses a diffuser in the pupil plane to encode 3D information, then computationally reconstructs the volume by solving a sparsity-constrained inverse problem. Our diffuser consists of randomly placed microlenses with varying focal lengths; the random positions provide a larger FOV compared to a conventional MLA, and the diverse focal lengths improve the axial depth range. To predict system performance based on diffuser parameters, we, for the first time, establish a theoretical framework and design guidelines, which are verified by numerical simulations, and then build an experimental system that achieves < 3 µm lateral and 4 µm axial resolution over a 1000 × 1000 × 280 µm3 volume. Our diffuser design outperforms the MLA used in LFM, providing more uniform resolution over a larger volume, both laterally and axially.},
keywords = {Light fields; Numerical simulation; Plenoptic imaging; Three dimensional imaging; Two photon polymerization; Wavefront encoding},
pubstate = {published},
tppubtype = {article}
}
Light field microscopy (LFM) uses a microlens array (MLA) near the sensor plane of a microscope to achieve single-shot 3D imaging of a sample without any moving parts. Unfortunately, the 3D capability of LFM comes with a significant loss of lateral resolution at the focal plane. Placing the MLA near the pupil plane of the microscope, instead of the image plane, can mitigate the artifacts and provide an efficient forward model, at the expense of field-of-view (FOV). Here, we demonstrate improved resolution across a large volume with Fourier DiffuserScope, which uses a diffuser in the pupil plane to encode 3D information, then computationally reconstructs the volume by solving a sparsity-constrained inverse problem. Our diffuser consists of randomly placed microlenses with varying focal lengths; the random positions provide a larger FOV compared to a conventional MLA, and the diverse focal lengths improve the axial depth range. To predict system performance based on diffuser parameters, we, for the first time, establish a theoretical framework and design guidelines, which are verified by numerical simulations, and then build an experimental system that achieves < 3 µm lateral and 4 µm axial resolution over a 1000 × 1000 × 280 µm3 volume. Our diffuser design outperforms the MLA used in LFM, providing more uniform resolution over a larger volume, both laterally and axially.