Research ArticleOPTICS

De-scattering with Excitation Patterning enables rapid wide-field imaging through scattering media

See allHide authors and affiliations

Science Advances  07 Jul 2021:
Vol. 7, no. 28, eaay5496
DOI: 10.1126/sciadv.aay5496

Abstract

Nonlinear optical microscopy has enabled in vivo deep tissue imaging on the millimeter scale. A key unmet challenge is its limited throughput especially compared to rapid wide-field modalities that are used ubiquitously in thin specimens. Wide-field imaging methods in tissue specimens have found successes in optically cleared tissues and at shallower depths, but the scattering of emission photons in thick turbid samples severely degrades image quality at the camera. To address this challenge, we introduce a novel technique called De-scattering with Excitation Patterning or “DEEP,” which uses patterned nonlinear excitation followed by computational imaging–assisted wide-field detection. Multiphoton temporal focusing allows high-resolution excitation patterns to be projected deep inside specimen at multiple scattering lengths due to the use of long wavelength light. Computational reconstruction allows high-resolution structural features to be reconstructed from tens to hundreds of DEEP images instead of millions of point-scanning measurements.

INTRODUCTION

Point-scanning multiphoton microscopy (PSMPM), based on two- or three-photon excitation, is used routinely for in vivo, volumetric biological imaging, especially in deep tissues (15). Imaging of cortical vasculature in mouse brain has been demonstrated down to 1.6 mm (3). The near-infrared (NIR) or short-wave infrared femtosecond laser pulses, used in PSMPM, penetrate deep in tissue due to the strong inverse relationship between light scattering and wavelength. The excitation light of PSMPM is focused at the diffraction limit enabling efficient nonlinear excitation. Emission photons from the focal spot, scattered or not, are then collected by a point detector, such as a photomultiplier tube, and assigned to a single pixel of the image. Despite the excellent penetration depth, a conventional PSMPM is slow due to the sequential acquisition process, and imaging time scales linearly as volume increases, limiting studies of fast biological dynamics.

An attractive alternative to PSMPM is wide-field multiphoton microscopy, typically called temporal focusing microscopy (TFM), that achieves optical sectioning by modulating laser pulse width while maintaining wide-field illumination (68). In TFM, wide-field multiphoton excitation is enabled by femtosecond laser pulses with high pulse energy (~μJ − mJ). By placing a grating on a conjugate image plane at the excitation path, optical dispersion is controlled so that laser pulse width rapidly broadens away from the focal plane, resulting in reduced multiphoton excitation efficiency (6). Benefiting from long excitation wavelength, TFM provides excellent penetration of multiphoton excitation light through scattering media. Penetration depths up to over 7 scattering lengths (based on emission wavelength) have been demonstrated with two-photon excitation (9). With three-photon excitation, penetration up to 700 μm through scattering brain tissue has been shown (10). However, in TFM, the emitted photons, with their relatively short wavelengths compared to excitation photons, are strongly scattered by tissues. Since typical wide-field microscopy registers spatial information by relaying the light emitted in the specimen plane to the camera, the signal-to-noise ratio and spatial resolution of the final image is highly susceptible to the scattering of emitted photons and aberration of the tissue specimen. Consequently, at shallower image planes, TFM images show a background haze (Fig. 1B); as the imaging depth is increased, TFM images lose their high-resolution information. It is important to note that this susceptibility to emission light scattering and aberration is common to all wide-field tissue imaging modalities such as various light sheet approaches (11).

Fig. 1 Comparison of two-photon imaging scenarios of a mouse neuron (256×256×156 voxels) in a live mouse.

(A) PSMPM image (experimental), (B) TFM image (simulation; see text S1 for details), and (C) DEEP-TFM reconstruction (simulation). Shown on the top row are the top XY views of the image stacks; shown on the bottom row are the side XZ views. (D) The number of measurements used at each z-plane for PSMPM, TFM, and DEEP-TFM. (We only demonstrate up to 2.25 scattering lengths at the emission wavelength, which used 128 patterns here. Please refer to fig. S7 for a detailed analysis for 6 scattering lengths).

Here, we demonstrate a powerful approach to “de-scatter” wide-field TFM images. Many computational imaging approaches have previously been proposed (1215); while they improve image resolution and contrast at shallower depths (<1 scattering length), most of them have not been applied for deep imaging. Notably, Escobet-Montalbán and co-workers (9) demonstrated a method called “TempoRAl Focusing microscopy with single-pIXel detection (TRAFIX)”; they used a set of two-dimensional illumination patterns along with single pixel detection (16) to image as deep as 7 scattering lengths through a scattering phantom (9). However, this approach requires the same number of illumination patterns as the number of pixels in the imaged field-of-view with no evident speed-up over PSMPM. Alemohammad et al. also used a similar technique called compressive temporal focusing two photon (CS-TFTP) (14), demonstrating 10 times speed-up over PSMPM, but the scattering length is not indicated. Both TRAFIX and CS-TFTP, however, have field-of-view (FOV)–dependent acquisition times because they use point (or single pixel) detection. Theoretical speed-up over PSMPM, therefore, is strictly limited to compressibility of the image. To the best of our knowledge, no computational TFM with wide-field detection has been demonstrated.

RESULTS

DEEP-TFM microscope principle

In this paper, we introduce a novel computational de-scattering technique called De-scattering with Excitation Patterning in TFM (DEEP-TFM). Similar to previous approaches, we use wide-field temporal focusing two-photon patterned excitation; however, the signal is measured with a wide-field detector. Briefly, we built a modified temporal focusing microscope that projects arbitrary excitation patterns onto the focal plane using a digital micromirror device (DMD). Emission light from the modulated excitation is then detected by a camera (see Fig. 2A and Materials and Methods for a detailed description of the microscope). Because of the NIR wavelengths, excitation patterns maintain their fidelity despite traveling through scattering media (10). However, emission photons are scattered by tissues, and scattering strength is strongly depth-dependent in most biological tissues (10, 17). In practice, TFM images are minimally affected by scattering at or near the surface; as the imaging depth increases, scattering gradually degrades mostly high-frequency information in the images. Nevertheless, low spatial frequency information in the images is retained for most depths with wide-field detection. Single-pixel detection approaches discard this low-frequency information, hence requiring a large number of excitation patterns to reconstruct a de-scattered image. In contrast, DEEP-TFM combines the information about the excitation patterns with the acquired images, to computationally reconstruct a de-scattered image (see Fig. 2B). Experimentally, the number of images needed to de-scatter a single FOV depends on the loss of high-frequency information due to scattering and hence depends only on the imaging depth. (see Materials and Methods and text S2 for a detailed description of the mathematical model and image reconstruction).

Fig. 2 Experimental setup and imaging strategy of DEEP-TFM.

(A) Optical schematic of the imaging system: L1, L2, L3, L4—lenses; DIO, dichroic mirror; OBJ, microscope objective; M, Mirror. (B) Proposed computational imaging strategy. First, a set of patterns are projected on a calibration specimen (homogeneous thin fluorescent layer) to record the calibration image set in the absence of any scattering. Then, the same patterns are projected to record the encoded images through a scattering medium. The de-scattered images are then reconstructed.

Simulation verification with TFM and PSMPM

The efficacy of DEEP-TFM was first verified through comparison with TFM and PSMPM in a simulation. In Fig. 1A, a stack of mouse neuron images (256 by 256 by 156 voxels), acquired experimentally with a point-scanning two-photon microscope, was used as the ground truth. For this, PSMPM requires over 10 million measurements (one for each voxel). Then, for each depth plane, the ground truth data were convolved with the scattering point spread function (sPSF) of the corresponding depth to generate the simulated TFM images in Fig. 1B. A conventional TFM requires only 156 measurements (one for each depth), but the image quality degrades as the imaging depth is increased. As expected, a huge haze in the background could be observed, especially near the brighter regions.

To obtain DEEP-TFM images, a set of binary random illumination patterns of the same image size were generated. Random patterns were chosen since they contain all frequencies, thus enabling a spread-frequency spectrum modulation. They were then convolved with the excitation PSF, multiplied pixel-wise with the ground truth data at each plane, and convolved with the sPSF of the corresponding depth. Last, these simulated DEEP-TFM data were reconstructed with the algorithm presented in Materials and Methods. The number of illumination patterns used at different depths increases as we go deeper (Fig. 1D). In total, DEEP-TFM required 8488 measurements, which is much less than the 10.2 million measurements need by PSMPM. As shown in Fig. 1C, DEEP-TFM is able to reject the scattering background and maintain similar image quality as PSMPM.

Experimental demonstration on fluorescent beads through a scattering medium

Figure 3(A and B) shows representative examples of DEEP-TFM imaging of fluorescent beads. We first imaged a mixture of 4- and 10-μm beads through a scattering lipid solution of 1.7 mm (0.15%) corresponding to approximately 2 scattering lengths at the emission wavelength of 532 nm (all scattering lengths in Results are based on emission light wavelength) (18). Figure 3, A1 shows a conventional TFM image; Fig. 3, A2 shows the final DEEP-TFM image reconstructed with Nt = 128 measurements. Since the light from all the beads goes through the same thickness of scattering medium, one would expect all 4-μm beads to show similar scattering behavior in Fig. 3, A1. They do; some beads only appear to show more scattering because they are out of the imaging focal plane (also see fig. S3A) but are excited due to the thickness of the TFM excitation plane (which is around 15 μm; see fig. S3E). In the DEEP-TFM image, the out of focus beads are not visible. Thus, in addition to de-scattering, it is interesting to note that DEEP-TFM also improves on the axial resolution over TFM (also see fig. S3C). Pushing further to demonstrate imaging at 7 scattering lengths (similar to TRAFIX), bead mixture of 1 and 4 μm were imaged successfully through a layer of high concentration (0.51%) lipid solution; the out-of-focus beads shown in Fig. 3B1 again disappeared in Fig. 3B2 due to the improved axial resolution of DEEP-TFM. For robust specimens, like bead samples, it is clear that DEEP-TFM has not reached its depth limit even at 7 scattering lengths; however, 128 versus 256 patterns were required for reconstruction at 2 scattering lengths versus 7 scattering lengths, respectively. Thus, DEEP-TFM imaging speed will decrease necessarily as expected from our simulation. Nevertheless, even at 7 scattering lengths, for the 128 × 128 (px2) image shown in Fig. 3B2, DEEP is 64 times faster than single-pixel geometries like TRAFIX at the same conditions. These results clearly demonstrate that some spatial information is available even at large scattering lengths (such as at 7 scattering lengths).

Fig. 3 DEEP-TFM results of fluorescent beads and ex vivo tissue.

(A1) Wide-field temporal focusing two-photon microscopy (TFM) image of a mixture of 4- and 10-μm beads imaged through a scattering medium of 2 scattering lengths. (A2) DEEP-TFM image of the FOV in (A1) reconstructed with 128 measurements (Nt = 128). (B1 and B2) Respectively the TFM and DEEP-TFM (with Nt = 256) images of a mixture of 1- and 4-μm beads imaged through a scattering medium of 7 scattering lengths. (C1 and C2) Respectively the TFM and DEEP-TFM (with Nt = 128) images of a mouse kidney specimen. Shown in blue, green, and red channels are respectively nucleus, Alexa Fluor 488–conjugated WGA, and F-actin. (D1 and D2) Respectively the TFM and DEEP-TFM (with Nt = 128) images of a mouse muscle specimen at a 190-μm-deep imaging plane. The blue and red channels are respectively nucleus (stained with Hoechst 33342) and F-actin (stained with Alexa Fluor 568 Phalloidin). (E) A representative image of F-actin (the red channel) of the same sample in (D) at a 170-μm-deep imaging plane comparing the TFM versus DEEP-TFM (with Nt = 128) images. Scale bars in (A) to (C) and (D) and (E) are 30 and 20 μm, respectively, in length.

Experimental demonstration on ex vivo biological samples

Next, we imaged a 16-μm-thick ex vivo mouse kidney histological section through the same 1.7 mm of scattering lipid solution of 0.15% corresponding to 2 scattering lengths (Fig. 3, C1 and C2). Figure 3C2 illustrates the immediate improvement in image contrast and signal-to-background ratio achieved by DEEP-TFM (see fig. S4 for additional results). We also imaged a 200-μm-thick section of ex vivo muscle tissue stained for nuclei (blue channel, Hoechst 33342) and F-actin (red channel, Alexa Fluor 568 Phalloidin). The FOV was nearly 150 μm × 150 μm with 256 × 256 pixels. All DEEP-TFM reconstructions were performed with Nt = 128 measurements. Representative images comparing TFM and DEEP-TFM at a 190 μm deep plane are shown in Fig. 3 (D1 and D2). Last, in Fig. 3E, we show a direct comparison of TFM and DEEP-TFM at a 170-μm-deep plane in the same F-actin image. As seen, in deep imaging, TFM loses a substantial amount of high-frequency information, with almost no high-resolution detail visible. In contrast, with DEEP-TFM, most fine details are reconstructed.

Experimental demonstration on in vivo mouse brain vasculature

The use of computational approaches to overcome tissue scattering often works well in fixed tissue samples but not in complicated in vivo experiments. As computational approaches are being developed for brain imaging experiments in neuroscience to improve imaging depth, speed, and resolution, it is critical to evaluate their performance for imaging in vivo biomedical specimens. As a first demonstration of this kind, DEEP-TFM in vivo imaging of mouse brain vasculature was undertaken. Mouse blood vessels were labeled with a rhodamine + dextran dye (70 kDa; D1841, Thermo Fisher Scientific). During the experiment, surface of the brain was found as the first plane with clear blood vessels while the animal was gradually brought toward the objective by a stage; then different depths in the brain were located with a motorized actuator. Results corresponding to imaging depths of 0 μm (0 scattering length, i.e., the surface), 100 μm (2 to 2.4 scattering lengths), 200 μm (4 to 4.8 scattering lengths), and 300 μm (6 to 7 scattering lengths) are shown in Fig. 4(A to D), respectively. The scattering length of brain tissue at the emission wavelength of 630 nm has previously been reported to be 42 μm (18). However, here, we make a conservative assumption of 42 to 50 μm for the scattering length calculations. The first column is the conventional TFM image, which was already blurry and noisy at surface due to the multiple scattering of forward emitted fluorescence captured by the camera. At 300 μm, the structure of the blood vessel was nearly overwhelmed by the scattered photon background losing nearly all high-frequency information. The second and third columns show the reconstruction of DEEP-TFM without a prior and with a wavelet sparsity regularization prior. While the reconstruction without regularization at surface is still satisfactory, the noise in the background increases quickly as we go deeper to 300 μm. With the sparse regularization, we are able to resolve the high-resolution structures with good signal-to-noise ratio. Although contaminated by noise, the ability to reconstruct without a prior demonstrates the strength of DEEP compared to other single-pixel–like geometries. At similar conditions, single-pixel geometries must rely on compressive sensing for reconstruction in all cases, including the surface, presented here.

Fig. 4 DEEP-TFM results of in vivo mouse brain vasculature.

(A1) Wide-field temporal focusing two-photon microscopy (TFM) image of cortical vasculature in a mouse brain at the surface. (A2) DEEP-TFM image of the FOV in (A1) reconstructed without using wavelet sparsity regularization. (A3) DEEP-TFM image of the FOV in (A1) reconstructed with wavelet sparsity regularization. (B1 to B3) The TFM image and DEEP-TFM reconstructions of cortical vasculature at 100 μm below the surface. (B2) is without regularization, and (B3) is with regularization. (C1 to C3) The TFM image and DEEP-TFM reconstructions of cortical vasculature in a mouse brain at 200 μm below the surface. (C2) is without regularization, and (C3) is with regularization. (D1 to D3) the TFM image and DEEP-TFM reconstructions of cortical vasculature in a mouse brain at 300 μm below the surface. (D2) is without regularization, and (D3) is with regularization. All images were reconstructed from 256 patterns. Scale bars, 20 μm.

DISCUSSION

A novel wide-field multiphoton microscopy approach was established to enable fast and deep biological imaging. Both simulation (Fig. 1) and experimental (Figs. 3 and 4) results demonstrate the potential of DEEP-TFM for biological tissue and in vivo mouse brain imaging. DEEP-TFM can resolve deep tissue biological images with similar quality to PSMPM at high resolution. Lateral resolution of DEEP-TFM is determined by the maximum resolution of the projected patterns and hence matches PSMPM. The results in Fig. 3(A1 and A2) also suggests an axial resolution improvement compared to TFM. Our work is consistent with prior works where patterned excitation has been observed to improve axial resolution of TFM (12, 13). However, a thorough theoretical underpinning of this effect remains to be developed and will represent an important future work. Beyond that, we have shown that DEEP-TFM can push imaging depth in scattering media down to at least 7 scattering lengths, by making use of the pattern information. To the best of our knowledge, DEEP-TFM is currently the only computational wide-field multiphoton imaging method with a FOV-independent frame rate. Theoretically, millimeter-large FOVs at diffraction limited resolution may be achieved with no sacrifice of speed. Further, DEEP-TFM uniquely provides flexible, depth-dependent imaging speeds: Shallow imaging is almost single-shot while imaging speed can be customized for each depth depending on the magnitude of emission photon scattering. Thus, in theory, an increase in speed of more than three orders of magnitude over PSMPM may be achieved for a volume of 256 pixels by 256 pixels by 156 pixels, with the assumption of the same acquisition time per measurement (see Fig. 1). Last, DEEP-TFM satisfies all the requirements in modern compressive sensing theory (16). With suitable image priors, an additional 10 times increase in speed may be achieved (see fig. S5) (14). Moreover, one might be able to learn specimen-specific image priors through learning approaches, which would outperform conventional nonspecific image priors such as the wavelets sparsity.

While we have achieved reasonable successes with DEEP-TFM, practical applications of TFM for biomedical applications still have to overcome several limitations. We have demonstrated imaging through 7 scattering lengths in ex vivo samples while only 6 to 7 scattering lengths in the mouse brain (18). The issue is that the maximum laser power that we can use in the mouse brain is much lower (~100 mW) due to the occurrence of vascular damage likely caused by hemoglobin absorption. This limitation may be overcome by shifting excitation power to longer a wavelength such as 1040 nm, where up to 1 W excitation had been used (19, 20). Nonetheless, it is clear that the ultimate limitation of this approach is that increasing power is required to further increase FOV and imaging speed. Future work in optimizing this class of approaches would need to include thermal modeling to determine maximum applicable power as a function of laser wavelength, laser repetition rate, image FOV, and specimen absorbance. Moreover, one may also alleviate this thermal power limit by reducing the laser pulse width from 120 to 10 to 20 fs since excitation efficiency improves linearly and quadratically for two- and three-photon cases, respectively. Last, patterns used here were of 50% fill factor; one may also investigate patterns with lower fill factors to reduce the total power on the sample while maintaining the same power per diffraction limited spot.

Besides fundamental limits due to photophysics of specimen damage, instrument and algorithm improvements can still drastically improve DEEP-TFM’s performance. First, the current implementation speed is limited by the frame rate of the EMCCD camera used. DEEP-TFM, and other similar approaches like TRAFIX, requires combining information from tens to hundreds of images. Therefore, an important factor that can quickly degrade the final image’s signal-to-noise ratio is the read noise of the camera system. We choose a low read noise EMCCD camera for this reason, but its ~50 to 100 Hz frame rate limits the final imaging speed. This limitation can be overcome by future implementations using either intensified fast complementary metal-oxide semiconductor cameras or avalanche photodiode arrays that have seen substantial advances recently. Second, since the speed of DEEP-TFM is independent of its FOV, its speed scales linearly with FOV. We currently use only ~150 μm × 150 μm FOV, but commercial high quality, high numerical aperture objectives have nearly 1000 μm × 1000 μm FOV while customized lenses have reached millimeter scale (21, 22). Third, same as TRAFIX, one major factor that limits the ultimate imaging depth is the maintenance of the illumination patterns at increasing depth. High-fidelity patterns can probably be formed in homogeneous scattering beyond 1 to 2 mm based on three-photon excitation. However, previous literature (12) shows that patterns are only able to penetrate several hundred microns in actual tissues under temporal focusing. The main issue is that the patterns are distorted because of aberration resulting from ever present sample inhomogeneity in biomedical specimens. Distorted patterns result in artifacts in the reconstructed images. In the future, specimen aberrations at different depths can be minimized by implementing different adaptive optics compensation schemes (2326). Fourth, as discussed previously, many aspects of the algorithm can be optimized. We have demonstrated that a wavelet prior can significantly improve reconstruction results. However, the optimal prior will probably depend on the type of specimens being imaged. A more modern approach is to use a deep convolutional neural network (dCNN) trained on a specific class of specimens; this approach would learn an optimal reconstruction algorithm for the said class of specimens. Alternatively, one could train a deep generative prior, again for a specific class of specimen, to replace the conventional prior in the current optimization algorithm. We believe that these advanced algorithms would extend the depth range for the same number of patterns presented here or improve the speed by being able to reconstruct with a lower number of patterns. One may also note that the excitation pattern (random, Hadamard, and others) and its duty cycle may also be optimized for a specific class of specimen. Using deep learning tools, one could model the forward image problem with patterns as trainable parameters. This model can then be cascaded with a dCNN reconstructor, for end-to-end optimization. Such an approach would simultaneously learn the optimal patterns as well as the optimal reconstruction algorithm for a class of specimen. Nevertheless, despite that many aspects of DEEP-TFM can be further optimized, it is promising that it has demonstrated imaging over a larger FOV at 80 times faster speed over TRAFIX (the speed here is calculated with 256 patterns at exposure time of 50 ms per frame).

In summary, here, we present DEEP-TFM, a novel computational wide-field technology for deep tissue multiphoton microscopy. Our results suggest that DEEP-TFM can resolve images with similar quality to point scanning two-photon microscopy but in a wide-field design. The acquisition time, therefore, is FOV independent. We believe that, with optimized instrumentation and algorithms, DEEP-TFM can accelerate volumetric multiphoton imaging by orders of magnitude.

MATERIALS AND METHODS

Pattern-illuminated temporally focused wide-field two-photon microscope

Figure 2A shows the schematic diagram of a temporal focusing microscope that enables arbitrary patterned illumination. First, an ultrafast pulsed laser beam [800-nm center wavelength, 120-fs pulse width, 10-kHz repetition rate, and ~8-mm beam diameter (1/e2)] from a regenerative amplifier (Legend Elite, Coherent, Santa Clara, CA, USA) was magnified to ~32 mm and directed to a DMD (DLP LightCrafter 9000 EVM, Texas Instruments, TX, USA). The DMD was used as a diffractive element and pattern generator, simultaneously. The beam was diffracted from the DMD with an effective grating period of ~190 lines/mm with an incident angle of 26.4°. Arbitrary patterns could be uploaded onto the DMD using a control program (DLP LCR 9000 GUI) provided by Texas Instruments. The DMD was followed by a 4f-lens system consisting of two planoconvex lenses (L1; f = 250 mm; LA1461, Thorlabs and L2; f = 125 mm; AC254-125-B-ML, Thorlabs, Newton, NJ, USA), which served to project and magnify the image of the DMD. Then, the images formed by L1 and L2 were relayed onto the sample, S, through another tube lens (L3; f = 300 mm; AC508-300-B-ML, Thorlabs), and an objective lens (OBJ) (water immersion 20×/1.0, Zeiss, Jena, Germany). The system magnification is about 73× according to the focal lengths of tube lenses and the effective focal length of the objective lens. The geometric dispersion of the system ensured that the pulse width was broad enough to minimize multiphoton excitation outside the sample plane. The objective location was controlled with an objective piezo positioner (MIPOS-500, Piezosystem Jena, Jena, Germany).

The two-photon excited fluorescence from the sample, S, was collected by the same objective lens, OBJ, in an epi detection geometry and reflected by a dichroic filter, DIO (Di03-R635-t3, Semrock, Rochester, NY, USA) to a camera. Then, fluorescence signals were imaged by another tube lens, L4 (f = 200 mm; PAC064, Newport, Irvine, CA, USA) onto an EMCCD camera (iXon+, Andor, Belfast, Northern Ireland). Another EMCCD camera (HNü 512, Nuvu Cameras, Montreal, Canada) was used for in vivo mouse brain imaging. For multicolor detection, three combinations of filter sets were used: blue channel centered at 460 nm (Semrock FF01-460/60-25 and Chroma E530SP-SPC), green channel centered at 535 nm (Chroma ET535/70 M and Chroma ET680SP-2P8), and red channel centered at 605 nm (Chroma ET605/70 M and Chroma E700SP-2P). An achromatic doublet lens pair (1:2, MAP1050100-A, Thorlabs) was used to expand the image size onto the camera when DMD pixels of 1024 × 1024 were used for pattern generation. For patterns of larger pixel size (1600 × 1600), a 1:1 achromatic doublet lens pair (MAP107575-A, Thorlabs) was used to ensure that the image fit the detector size. Data from the camera were transferred using either a control program (Andor Solis) provided by Andor or a control program (NuPixel) provided by Nuvu or a control software implemented using LabVIEW 2015 (National Instruments, Austin, TX, USA).

Preparation of the calibration and the scattering samples

A thin quantum-dot layer was used for the calibration of patterns for the green (535 nm) and red channels (605 nm). A thin, fluorescent layer of green quantum dots (supplied by QD Vision, Lexington, MA, USA) dispersed in hexane (10 μl) was dropped onto a coverslip (thickness, 170 μm) and allowed to dry. The coverslip was affixed to a glass slide and sealed by transparent nail varnish. A thin 4′,6-diamidino-2-phenylindole (DAPI) solution layer was used for the calibration of patterns for the blue channel (460 nm). Saturated DAPI solution in 1:1 mixture of deionized (DI) water and dimethyl sulfoxide were dropped in a preholed spacer (120 μm thick, Secure-Seal imaging spacers, Grace Bio-Labs, OR, USA) onto a glass slide, and a coverslip was placed on top of the spacer. The coverslip was sealed using clear nail varnish.

A mixture of 4- and 10-μm-sized yellow-green fluorescent beads (FluoSpheres Sulfate Microspheres, 4.0 μm; and FluoSpheres Polystyrene Microspheres, 10 μm; Thermo Fisher Scientific, MA, USA) were used to demonstrate the approach. A mixture of 4-μm-sized and 10-μm-sized yellow-green fluorescent beads was dropped in warm 1% agarose gel solution and stirred thoroughly. Then, 25 μl of the mixture was dropped in a preholed spacer (120 μm thick) onto a glass slide, and a coverslip was placed on top of the spacer. The coverslip was sealed using clear nail varnish. The slide was left to cool down before the experiment to solidify.

Preparation of mouse tissues

We used a prepared slide of sectioned mouse kidney (F24630, Invitrogen, Carlsbad, CA, USA) to demonstrate the utility of the pattern-illuminated TF. The slide contains a 16-μm cryostat section of mouse kidney stained with Alexa Fluor 488 wheat germ agglutinin (WGA), Alexa Fluor 568 phalloidin, and DAPI. While DI water was used as immersion medium for nonscattering case, and 0.15% lipid solution was used as immersion medium to mimic scattering environment since the sectioned mouse kidney is only 16 μm thick.

Animal protocol and labeling procedure for ex vivo mouse muscle tissue imaging

The animal procedure (transcardial perfusion) was approved by the Massachusetts Institute of Technology (MIT) Committee on Animal Care and meets the National Institutes of Health (NIH) guidelines for the care and use of vertebrate animals. Mice were deeply anesthetized with 1.25% avertin solution (350 mg/kg intraperitoneal) and transcardially perfused with phosphate-buffered saline (PBS) containing 4% paraformaldehyde. After perfusion, thigh muscle was excised and postfixed in 4% paraformaldehyde overnight. Muscle tissue was cryoprotected in 30% sucrose for 48 hours, embedded in optimal cutting temperature formulation (Tissue-Tek), frozen at −20°C, and sliced at a thickness of 200 μm on a cryostat. Frozen sections were immersed in PBS for staining. Solutions of Alexa Fluor 568 Phalloidin (Invitrogen, catalog number A12380) and Hoechst 33342 (Invitrogen, catalog number H21492) were prepared as follows: The entire contents of the 300-U vial of Alexa Fluor 568 Phalloidin was dissolved in methanol (1.5 ml) to produce a 40x stock solution, which was diluted 1:40 in PBS for staining. A Hoechst stock solution was prepared by dissolving Hoechst in water at a concentration of 10 mg/ml. The Hoechst stock solution was diluted 1:2000 in PBS for a final concentration of 5 μg/ml for staining. Muscle slices were permeabilized in a solution of 1% Triton X-100 in PBS for 20 min at room temperature with gentle shaking. Slices were then incubated in a working solution of the dyes (dissolved in PBS as described above) for 20 min at room temperature with gentle shaking. Excess dye was removed by washing slices in PBS three times (6 min per wash, with gentle shaking at room temperature). Slices were then mounted on slides using Fluoromount-G or Vectashield as mounting media. Slides were coverslipped, and slides containing Vectashield as the mounting medium were sealed along the edges of the coverslip with clear nail polish. Slides were allowed to dry for at least 48 hours before imaging.

Surgical procedure and in vivo mouse neural vascular imaging preparation

Experiments were carried out under protocols approved by MIT’s Animal Care and Use Committee and conformed to NIH guidelines. All data in this study were collected from adult (>8 weeks old) mice of either sex. The mice were wild type and acquired from the Jackson laboratory (#000664). Mice were initially anesthetized with 4% isoflurane in oxygen and maintained on 1.5 to 2% isoflurane throughout the surgery. Buprenorphine (1 mg/kg, subcutaneous) and/or meloxicam (1 mg/kg, subcutaneous) was administered preoperatively and every 24 hours for 3 days to reduce inflammation. Ophthalmic ointment was used to protect the animal’s eyes during the surgery. Body temperature was maintained at 37.5°C with a heating pad. The scalp overlying the dorsal skull was sanitized and removed. The periosteum was removed with a scalpel and a craniotomy (5 mm) was made over the primary visual cortex (V1, 4.2 mm posterior, 3.0 mm lateral to Bregma) on either the left or right hemisphere, leaving the dura intact. For calcium imaging, a circular cover glass (3 mm; Warner Instruments) was implanted over the craniotomy as a cranial window and sealed with dental acrylic (C&B-Metabond, Parkell) mixed with black ink to reduce light transmission. Last, a custom-designed stainless steel head plate (eMachineShop.com) was affixed to the skull using dental acrylic. Experiments were performed at least 5 days after head plate implantation to allow animals to recover. For labeling blood vessels with a fluorescent dye, a rhodamine + dextran dye (70 kDa; D1841, Thermo Fisher Scientific) mixed with saline solution at 5% (w/v) concentration was applied retroorbitally with 100-μl volume (27). During the retroorbital injection, the animal was anesthetized with 2% isoflurane in oxygen. During multiphoton imaging, the animal was anesthesized with ketamine + xylazine mixture with 0.1-ml volume, and this mixture was applied as needed after checking the reflexes. The imaging sessions lasted for a maximum of 2 hours.

Pattern design and projection

A random or a randomized block Hadamard–patterned image [similar to (28)] set was generated using MATLAB. To get the randomized block Hadamard patterns, a Hadamard matrix of size Nt × Nt was first created (here Nt is the number of patterns). Each row of the matrix was then used to generate a replicating block (see fig. S8). To generate the full pattern, the blocks were tessellated to maximize the distance between repeating time series–patterned pixels (see fig. S8B). Then, the resulting full patterns were multiplied with the same fully random mask (of +1s and −1s) to get the randomized block Hadamard–patterned image set (see a representative pattern in fig. S8C). In beads and mouse kidney experiments, patterns of 1024 × 1024 pixels resized by a factor of 8 were used for excitation patterns. This combination defines the unit block of 8 × 8 pixels at the DMD, which corresponds to 60.8 μm for the length of one side. The corresponding size of the unit block at the sample plane is 0.83 μm, which is close to the effective diffraction limit of the system [λ/(2NA2)]. The total number of patterns for each imaging session was 256 for a complete basis set. For mouse muscle imaging, patterns of 1600 × 1600 pixels at the DMD were used to enlarge the FOV of the system with the modification of a magnifying lens compound in front of the EMCCD camera. The exposure time of the camera was adjusted in the range of 100 to 500 ms per pattern depending on the signal intensity of the specimen. The electron multiplying (EM) gain of the camera was set to be 3 to 100 depending on the signal intensity of the specimen as well.

DEEP-TFM mathematical model and image reconstruction technique

Mathematically, the DEEP-TFM imaging process can be modeled by the following equationYt(x,y)=sPSF(x,y)*{([exPSF(x,y)*H˜t(x,y)])X(x,y)}(1)

Here, x and y are spatial coordinates; t is modulation pattern’s (and acquired image’s) index; exPSF(x, y) is the excitation point spread function (assumed known); sPSF(x, y) is the scattering point spread function; and Ht(x,y) is the tth modulation pattern projected by the DMD. X(x, y) is the object being imaged, and Yt(x, y) is the tth image acquired on the camera. The operators * and ∘ represent spatial convolution and pixel wise multiplication (see section S1 for more details). Writing 1 in the spatial Fourier domain we getYt(kx,ky)=sPSF(kx,ky){[FexPSF(kx,ky)FH˜t(kx,ky)]*FX(kx,ky)}(2)

Assuming N pixels (both in the image, Yt, and the object, X), the above equation has 2N unknowns (N in ℱX and N in ℱsPSF). As written, each measurement (i.e., an image taken at time point t) appears to provide N equations. However, ℱsPSF acts as a low-pass filter and for out-band frequencies (out of the frequency support of ℱsPSF) the right-hand side of the 2 is zero. Now, assume that ℱsPSF’s frequency support has M pixels. Then, each measurement provides M independent equations. Thus, for the above system of equations to be solvable, we need Nt > 2N/M measurements. For deep tissue imaging applications, the frequency support of ℱsPSF changes with imaging depth. As there is negligible scattering at the surface, at the surface M~N; we only need Nt = O(1) measurements [here “O(. )” represents the asymptotic “big-O” notation] at very deep frames where there is no spatial information on the recorded images, M~1. We hence need Nt = O(N) measurements. Please also note that because of the frequency domain convolution between Ht and ℱX (2) out-band frequencies (of the frequency support of ℱsPSF) in X are still sampled on to Yt as long as H captures all possible frequencies of X. It can be shown that a random pattern of H satisfies this criterion (see fig. S2 and section S2). Thus, an ensemble of O(2N/M) random patterns {Ht}, can be used to fully measure X in DEEP-TFM (see section S2 for a detailed description). Upon such measurement, we record an ensemble of {Yt} images corresponding to {Ht}; X can be reconstructed using, {Yt} and {Ht}, by solving Eq. 1 or its corresponding frequency domain representation, i.e., 2.

The calibration experiment gives the ensemble of patterns, {Ht}, used to modulate spatial features. The imaging experiment gives the ensemble of measurement images, {Yt}. Therefore, the reconstruction of the de-scattered image, X, is possible from solving the set of Eq. 1 equations (or the set of 2 equations). However, the constituting set of equations in Eq. 1 (and its corresponding frequency domain form in 2) are not linear but quadratic with respect to the unknowns (X and sPSF). To solve this system, one could first assume a form for sPSF, and then the Eq. 1 becomes a linear system that can be solved for X with commonly available linear optimization methods. When a solution for X is found that can be substituted in 2 which makes a similar linear system that can be solved for ℱsPSF (and hence for sPSF). Thus, a proper solution for X can be iteratively found. For the results shown in this paper, we only performed one iteration assuming a canonical form for sPSF, which resulted in visually accurate reconstructions. We used two-step iterative shrinkage/thresholding algorithm (TwIST) (29) or CVX (30, 31) (a package for specifying and solving convex programs) to solve the above linear equations. Please refer to Supplementary text for a detailed description of the problem formulation.

SUPPLEMENTARY MATERIALS

Supplementary material for this article is available at http://advances.sciencemag.org/cgi/content/full/7/28/eaay5496/DC1

https://creativecommons.org/licenses/by/4.0/

This is an open-access article distributed under the terms of the Creative Commons Attribution license, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

REFERENCES AND NOTES

Acknowledgments: We thank E. Nedivi for comments on the manuscript and providing laboratory supplies and tissue slides. We also thank K. Burnell for technical assistance with tissue sample preparation. Funding: This work was supported by 5-P41-EB015871, 5R21NS091982-02, 1-U01-NS090438-01, 1R21NS105070-01, K99EB027706, and F32 MH115441; Hamamatsu Corporation; Samsung Advanced Institute of Technology; Singapore-Massachusetts Institute of Technology Alliance for Research and Technology (SMART) Center, BioSystems, and Micromechanics (BioSyM), and Critical Analytics for Manufacturing Personalized Medicine (CAMP); Weill Neurohub Fellowship; Center for Advanced Imaging at Harvard University; and the John Harvard Distinguished Science Fellowship Program within the FAS Division of Science of Harvard University. Author contributions: D.N.W. and P.T.C.S. conceived the idea. J.K.P. and Y.X. designed the initial optical setup. J.K.P. and C.Z. developed the experimental setup with P.T.C.S.’s guidance. D.N.W. developed the initial algorithms and performed the numerical simulations. J.K.P. performed the initial experiments with low scattering lengths. J.R.B. prepared the ex vivo mouse tissue samples. C.Z. performed the large scattering lengths experiments on beads and in vivo cortical vasculature. M.Y. performed the mouse surgery for in vivo experiments under M.S.’s guidance and helped C.Z. with in vivo experiments. C.Z. performed computational reconstructions under D.N.W.’s guidance. All authors contributed to writing the manuscript. P.T.C.S. and D.N.W supervised the project. Competing interests: J.K.P., Y.X., P.T.C.S., and D.N.W. are inventors on a patent related to this work filed by MIT (no. US20200342205A1, filed 9 April 2020, published 29 October 2020). The authors declare that they have other no competing interests. Data and materials availability: All data needed to evaluate the conclusions in the paper are present in the paper and/or the Supplementary Materials. Codes used in the reconstruction can be downloaded at https://zenodo.org/badge/latestdoi/368268534 along with an example dataset at https://zenodo.org/record/4769097 for testing. Additional data related to this paper may be requested from the authors.

Stay Connected to Science Advances

Navigate This Article