Light Field Microscope

Introduction

Background and Motivation

Traditional light microscopy has been used to illuminate miniature biological specimens since the mid-1600s. Since, researchers have come to realize and (attempt to) circumvent the inherent limitations of the technique. One such limitation is that of superimposed features on the captured image. With no depth information from the subject, it is difficult to extract data specific to the focal plane from the rest of the image.

Confocal microscopy excludes light from above or below the focal plane of the image using a pinhole placed where the light from the focal plane comes to a waist. It also provides a method of visualizing 3-dimensional images and has better spatial resolution than light field microscopy, but images require scanning the entire sample at every depth of interest. Light field has the advantage of capturing relative depth information in a single photo. This additional visual information comes at a cost, however. Diffraction places an upper limit on the lateral and axial resolution of the images. We thus sacrifice spatial resolution to obtain angular resolution.

The light field microscope allows angular resolution information to be recorded in a single image. The light field is a function that represents the amount of light traveling in every direction through every point in space. By recording a sample’s light field, one can produce perspective and parallax views of the specimen. This is accomplished by inserting a microlens array in a conventional bright field microscope, and analyzing the images in silico. A microlens array is an optical component containing a spread of thousands of lenses with diameters in the micron scale. Applying 3D deconvolution to the focal stacks from the array produces a series of cross sections which allow for a 3D reconstruction of the sample.

The objective of this project is to create a light field microscope from a Lytro™ camera and the Thorlabs materials available in the 20.345 classroom, record associated documentation and code, and describe recommended experiments for use in a teaching undergraduate laboratory.

Prior Work

The only prior work we have to build upon are papers by Ren Ng's original paper on the light field microscope^[1], his further work^[2], his thesis^[3], and some work from a postdoc at Harvard^[4]. This project has not been done previously in the 20.345 lab. Ruben actually seems to have taken some shots of neurons firing, (using Calcium green - Life Technologies) and watched them in the 3d rendering.

Design

Optical setup

Evolution of the optical setup

The microscope design went through a few iterations and continues to evolve. We originally tried to use a 2" lens telescope to magnify the image. At first the telescope was before the tube lens, but since that contributes to aberration it was placed after the tube lens. While doing this, we realized that the image should be in focus just before the Lytro™ optics. We tried to put a different tube lens in the system to magnify the image with a single lens, however the led to a considerable amount of spherical aberration.

Image Processing

Reverse engineering the Lytro™ images

Getting images from Lytro™

1. Pictures > Lytro™ (Show Package Contents) > images.
2. You will see images organized by upload, in .lfp and -stk.lfp. The .lfp are the larger raw light field files. The -stk.lfp have already determined the most important focal depths in the image and keep only those depths.

Obtaining lfptools

1. Download the lfptools from Nirav Patel on Github, or find it in Departmental > BioEng > BioEngLab > 20.309 > Students > Projects > Light Field Microscopy.
2. Unzip.
   1. In Macs. Open Terminal. cd to folder. make.
   2. If you get the error "make: command not found" then download Xcode, go to Preferences>Downloads, and Install Command Line Tools.
   3. Move files you created to the folders HD > usr > local > bin. This folders might be invisible.

Creating .jpg image stacks from -stk.lfp

1. In terminal: lfpsplitter ~/Desktop/IMG_0010-stk.lfp
   1. Or wherever you keep your LFP files.
2. It will save in the same folder: a .json, a .txt, and .jpgs corresponding to those depths that the algorithm deemed important.

Creating .raw from .lfp

1. In terminal: lfpsplitter ~/Desktop/IMG_0010.lfp
   1. Or wherever you keep your LFP files.
2. It will save in the same folder: a .raw, a metadata .json, a private metadata .json, and a table .json.

Creating .tif from .raw

1. Download raw2tiff from here
   1. In Macs. Open Terminal. cd to folder. make.
   2. If you get the error "make: command not found" then download Xcode, go to Preferences>Downloads, and Install Command Line Tools.
   3. Move files you created to the folders HD > usr > local > bin. This folders might be invisible.
2. In terminal: raw2tiff -w 3280 -l 3280 -d short ~/Desktop/IMG_0010_imageRef0.raw ~/Desktop/Out10.tif

Dealing with the hexagonal microarray

Visualizing the sample in LFDisplay

1. Download LFDisplay
2. Run. Data > Open Input > Single Image
3. If a square array (microarray was a regular grid of lenses)
   1. Move the center lenslet position until the red circle covers the center lenslet.
   2. Use dx in "Lenslet to the right" to give the program the correct distance between lenses.
   3. Do the same for dy in "One lenslet down."
4. Move the radio button at the top to 3D to see the rendering.

List of parts

Methodology

Disassembling the Lytro

Aligning the beam path

Making sample slides of fluorescent beads

Acknowledgments

1. Nirav Patel, for reverse engineering the Lytro image and lfpsplitter.
2. Frank Warmerdam, Andrey Kiselev, Bob Friesenhahn, Joris Van Damme and Lee Howard for raw2tiff tools.
3. Other references: ^[5]

^{[6] [7] [8] [9] [10] [11]}

References

1. ↑ [1]
2. ↑ [2]
3. ↑ [3]
4. ↑ [4]
5. ↑ G. LIPPMAN, Epreuves réversibles donnant la sensation du relief, J. Phys. 7 (4) (1908) 821–825.
6. ↑ WEINSTEIN, R.S., DESCOUR, M.R., et al. 2004. An array microscope for ultrarapid virtual slide processing and telepathology. Design, fabrication, and validation study. Human Pathology 35, 11, 1303-1314.
7. ↑ NG, R., LEVOY, M., BREDIF, M., DUVAL, G., HOROWITZ, M., HANRAHAN, P. 2005. Light Field Photography with a Hand-Held Plenoptic Camera. Stanford Tech Report CTSR 2005-02.
8. ↑ Levoy, Marc, Ren Ng, Andrew Adams, Matthew Footer, and Mark Horowitz. "Light Field Microscopy." ACM Transactions on Graphics 25.3 (2006): 924. Print.
9. ↑ Levoy, M. Zhang, Z. McDowall, I. Recording and controlling the 4D light field in a microscope. Journal of Microscopy, Volume 235, Part 2, 2009, pp. 144-162.
10. ↑ Light field microscopy - Stanford 2006 (pdf)
11. ↑ ZHANG, ZHENGYUN. "A Practical Introduction to Light Field Microscopy." (2010). A Practical Introduction to Light Field Microscopy. Web. 10 Mar. 2012.