Research ArticleAPPLIED OPTICS

Adaptive foveated single-pixel imaging with dynamic supersampling

See allHide authors and affiliations

Science Advances  21 Apr 2017:
Vol. 3, no. 4, e1601782
DOI: 10.1126/sciadv.1601782
  • Fig. 1 Experimental setup.

    The scene is flood-illuminated and imaged onto a DMD, which operates as a dynamic mask: Light from a subset of the micromirrors is reflected to an avalanche photodiode (APD), which records the total intensity transmitted by each binary masking pattern. More details are given in Materials and Methods.

  • Fig. 2 Single-pixel imaging with spatially variant resolution.

    (A to C) Uniform resolution. (A) Uniform 32 × 32–pixel grid with N = 1024 pixels. (B) Examples of a complete 1024 Hadamard pattern set (negatives not shown) reformatted onto the 2D uniform grid shown in (A). (C) Image of a cat recorded experimentally in ~0.125 s, reconstructed from the level of correlation with each of the 1024 masks shown in (B). (D to F) Spatially variant resolution. (D) Spatially variant pixel grid, also containing N = 1024 pixels of varying area. Within the fovea, the pixels follow a Cartesian grid, chosen to avoid aliasing with the underlying Cartesian grid of the DMD at high resolutions. Surrounding the fovea is a peripheral cylindrical polar system of pixels. (E) Examples of the 1024 Hadamard patterns reformatted onto the spatially variant grid shown in (A). (F) Image of the identical scene to that shown in (C), reconstructed here from correlations with the 1024 spatially variant masks shown in (E). In the central region of (F), the linear resolution is twice that of the uniform image (C).

  • Fig. 3 Reconstructing images with a spatially variant effective exposure time using digital supersampling.

    All images are reconstructed from experimental data. (A) Four subframes, each with the foveal cells shifted by half a cell in x and/or y with respect to one another (45). The number of cells in each subframe is N = 1024. The purple insets show the underlying cell grid in each case. Movie S1 shows the changing footprints of the subframes in real time (see section S7 for full description of movie S1). (B) Composite images reconstructed from increasing numbers of subframes using the weighted averaging method. (C) Composite images reconstructed from increasing numbers of subframes using the linear constraint method. The number of hr-pixels in the high-resolution composite images is M = 128 × 128 = 16,384, although not all of these may be independently recovered, depending on the number of sub-images combined and the configuration of each sub-image’s cells. Insets bridging (B) and (C) color-code the local time taken to perform the measurements used to reconstruct each region within the field of view, that is, the spatially variant effective exposure time of the images. The central region only uses data from the most recent four subframes (taking 0.5 s), whereas the reconstruction of the periphery uses data from subframes going progressively further back in time. Movie S2 shows a movie of the progressive linear constraint reconstruction (see section S7 for full description of movie S2). (D) Reconstructions of a uniform grid of points from 36 subframes to compare the PSF of the two reconstruction methods.

  • Fig. 4 Fovea guidance by motion tracking.

    (A) Low-resolution blip-frames (i to ii), recorded after every fourth subframe. The difference between consecutive blip-frames reveals regions that have changed (iii). A binary difference map (iv) is then constructed from (iii) (see Materials and Methods for details). This analysis is performed in real time, enabling the fovea relocation to a region of the scene that has changed in the following subframes. (B) Frame excerpts from movie S3 showing examples of subframes (each recorded in 0.125 s) guided using blip-frame analysis to detect motion (fovea location updated at 2 Hz). The purple insets show the space-variant cell grid of each subframe. (C) Frame excerpts from movie S4 showing the reconstructed (using linear constraints) video stream of the scene also capture the static parts of the scene at higher resolution. Here, difference map stacks (shown as insets) have been used to estimate how recently different regions of the scene have changed, guiding how many subframes can contribute data to different parts of the reconstruction. This represents an effective exposure time that varies across the field of view. Here, the maximum exposure time has been set to 4 s (that is, all data in the reconstruction are refreshed at most after 4 s), and the effective exposure time has also been color-coded into the red plane of the reconstructed images. (D) Conventional uniform-resolution computational images of a similar scene for comparison (also shown in movie S4). These use the same measurement resource as (B) and (C). Section S7 gives a detailed description of movies S3 and S4.

  • Fig. 5 Detail estimation and infrared dual fovea reconstruction.

    (A to C) Fovea guidance by wavelet transform. (A) The fovea trajectory is determined by first measuring a blip-frame. A single-tier Haar wavelet transform is then performed on (A) to produce an edge contrast map (B) from which the fovea trajectory is then determined (for details, see Materials and Methods). (C) Map of the fovea trajectory within the field of view. Brighter regions indicate areas that the fovea visits earlier. Arrows show the trajectory of the fovea. (D to G) Image reconstructions after sampling the scene with various numbers of subframes and fovea positions. In this example, the fovea trajectory determined by the wavelet transform samples most of the detail in the scene after eight subframes. This is 50% of the time required to sample the entire field of view at the same resolution as the center has been sampled here. (H and I) Dual fovea infrared image. (H) Weighted average of four subframes (1368 cells per subframe; frame rate, 6 Hz), each having two fovea. (I) Cell grid of one of the subframes.

Supplementary Materials

  • Supplementary material for this article is available at http://advances.sciencemag.org/cgi/content/full/3/4/e1601782/DC1

    section S1. Hadamard correlation measurements

    section S2. Foveated subframe reconstruction

    section S3. Signal-to-noise ratio

    section S4. Weighted averaging image fusion

    section S5. Linear constraint image fusion

    section S6. Reconstructions with additional assumptions

    section S7. Supplementary movie file descriptions

    fig. S1. Reconstruction comparison.

    fig. S2. Movie S1 snapshot.

    fig. S3. Movie S2 snapshot.

    fig. S4. Movie S3 snapshot.

    fig. S5. Movie S4 snapshot.

    movie S1. Real-time subframe display.

    movie S2. Postprocessed linear constraint reconstruction.

    movie S3. Real-time motion tracking and fovea guidance.

    movie S4. Real-time weighted averaging and postprocessed linear constraint reconstruction of a dynamic scene.

    References (6163)

  • Supplementary Materials

    This PDF file includes:

    • section S1. Hadamard correlation measurements
    • section S2. Foveated subframe reconstruction
    • section S3. Signal-to-noise ratio
    • section S4. Weighted averaging image fusion
    • section S5. Linear constraint image fusion
    • section S6. Reconstructions with additional assumptions
    • section S7. Supplementary movie file descriptions
    • fig. S1. Reconstruction comparison.
    • fig. S2. Movie S1 snapshot.
    • fig. S3. Movie S2 snapshot.
    • fig. S4. Movie S3 snapshot.
    • fig. S5. Movie S4 snapshot.
    • Legends for movies S1 to S4
    • References (61–63)

    Download PDF

    Other Supplementary Material for this manuscript includes the following:

    • movie S1 (.mov format). Real-time subframe display.
    • movie S2 (.mov format). Postprocessed linear constraint reconstruction.
    • movie S3 (.mov format). Real-time motion tracking and fovea guidance.
    • movie S4 (.mov format). Real-time weighted averaging and postprocessed linear constraint reconstruction of a dynamic scene.

    Files in this Data Supplement:

Stay Connected to Science Advances

Navigate This Article