Light Field Microscope

Introduction

Background and Motivation

Traditional light microscopy has been used to illuminate miniature biological specimens since the mid-1600s. Since, researchers have come to realize and (attempt to) circumvent the inherent limitations of the technique. One such limitation is that of superimposed features on the captured image. With no depth information from the subject, it is difficult to extract data specific to the focal plane from the rest of the image.

Confocal microscopy excludes light from above or below the focal plane of the image using a pinhole placed where the light from the focal plane comes to a waist. It also provides a method of visualizing 3-dimensional images and has better spatial resolution than light field microscopy, but images require scanning the entire sample at every depth of interest. Light field has the advantage of capturing relative depth information in a single photo. This additional visual information comes at a cost, however. Diffraction places an upper limit on the lateral and axial resolution of the images. We thus sacrifice spatial resolution to obtain angular resolution.

The light field microscope allows angular resolution information to be recorded in a single image. The light field is a function that represents the amount of light traveling in every direction through every point in space. By recording a sample’s light field, one can produce perspective and parallax views of the specimen. This is accomplished by inserting a microlens array in a conventional bright field microscope, and analyzing the images in silico. A microlens array is an optical component containing a spread of thousands of lenses with diameters in the micron scale. Applying 3D deconvolution to the focal stacks from the array produces a series of cross sections which allow for a 3D reconstruction of the sample.

The objective of this project is to create a light field microscope from a Lytro™ camera and the Thorlabs materials available in the 20.345 classroom, record associated documentation and code, and describe recommended experiments for use in a teaching undergraduate laboratory.

Prior Work

The only prior work we have to build upon are papers by Ren Ng's original paper on the light field microscope^[1], his further work^[2], his thesis^[3], and some work from a postdoc at Harvard^[4]. This project has not been done previously in the 20.345 lab. Ruben actually seems to have taken some shots of neurons firing, (using Calcium green - Life Technologies) and watched them in the 3d rendering.

Design

Optical setup

Evolution of the optical setup

The microscope design went through a few iterations and continues to evolve. We originally tried to capture images with the Lytro with the provided optics. This actually ended up being more work than it was worth. Our method was basically to have the image plane lie just in front of the Lytro optics and then force the Lytro to focus on that plane using a lens tissue. This method did not always work, and in fact usually the Lytro would not like focusing on the light from the image instead of the surrounding lab, and switch accordingly.

We originally tried to use a 2" lens telescope to magnify the image. At first the telescope was before the tube lens, but since that contributes to aberration it was then placed after the tube lens. We tried a number of different setups to magnify the sample image such that it filled the field of view of the Lytro (which was fairly large considering its nice optics). We tried to put a different tube lens in the system to magnify the image with a single lens, however the led to a considerable amount of spherical aberration. We tried using a little pre-made telescope, but we found using its built in aperture that our NA was probably limited by the objective.

We noticed that once we inserted the images into LFDisplay for 3d rendering, we could see a very bright light spot that moved as we rotated our image. To counteract this, we used a medium-NA halogen light with a diffuser for more uniform illumination.

After much suffering, Vincent finally took the reins and tore the Lytro apart. He removed the front end containing the optics, and put the microlens array/sensor setup at the image plane. We took much better images this way.

Image Processing

Reverse engineering the Lytro™ images

Getting images from Lytro™

1. Pictures > Lytro™ (Show Package Contents) > images.
2. You will see images organized by upload, in .lfp and -stk.lfp. The .lfp are the larger raw light field files. The -stk.lfp have already determined the most important focal depths in the image and keep only those depths.

Obtaining lfptools

1. Download the lfptools from Nirav Patel on Github, or find it in Departmental > BioEng > BioEngLab > 20.309 > Students > Projects > Light Field Microscopy.
2. Unzip.
   1. In Macs. Open Terminal. cd to folder. make.
   2. If you get the error "make: command not found" then download Xcode, go to Preferences>Downloads, and Install Command Line Tools.
   3. Move files you created to the folders HD > usr > local > bin. This folders might be invisible.

Creating .jpg image stacks from -stk.lfp

1. In terminal: lfpsplitter ~/Desktop/IMG_0010-stk.lfp
   1. Or wherever you keep your LFP files.
2. It will save in the same folder: a .json, a .txt, and .jpgs corresponding to those depths that the algorithm deemed important.

Creating .raw from .lfp

1. In terminal: lfpsplitter ~/Desktop/IMG_0010.lfp
   1. Or wherever you keep your LFP files.
2. It will save in the same folder: a .raw, a metadata .json, a private metadata .json, and a table .json.

Creating .tif from .raw

1. Download raw2tiff from here
   1. In Macs. Open Terminal. cd to folder. make.
   2. If you get the error "make: command not found" then download Xcode, go to Preferences>Downloads, and Install Command Line Tools.
   3. Move files you created to the folders HD > usr > local > bin. This folders might be invisible.
2. In terminal: raw2tiff -w 3280 -l 3280 -d short ~/Desktop/IMG_0010_imageRef0.raw ~/Desktop/Out10.tif

Dealing with the hexagonal microarray

Visualizing the sample in LFDisplay

1. Download LFDisplay
2. Run. Data > Open Input > Single Image
3. If a square array (microarray was a regular grid of lenses)
   1. Move the center lenslet position until the red circle covers the center lenslet.
   2. Use dx in "Lenslet to the right" to give the program the correct distance between lenses.
   3. Do the same for dy in "One lenslet down."
4. Move the radio button at the top to 3D to see the rendering.

List of parts

Methodology

Disassembling the Lytro

Aligning the beam path

Making sample slides of fluorescent beads

Results

Images

Issues

• noise
• light

Acknowledgments

1. Nirav Patel, for reverse engineering the Lytro image and lfpsplitter.
2. Frank Warmerdam, Andrey Kiselev, Bob Friesenhahn, Joris Van Damme and Lee Howard for raw2tiff tools.
3. Other references: ^{[5] [6] [7] [8] [9] [10] [11]}

References

1. ↑ [1]
2. ↑ [2]
3. ↑ [3]
4. ↑ [4]
5. ↑ G. LIPPMAN, Epreuves réversibles donnant la sensation du relief, J. Phys. 7 (4) (1908) 821–825.
6. ↑ WEINSTEIN, R.S., DESCOUR, M.R., et al. 2004. An array microscope for ultrarapid virtual slide processing and telepathology. Design, fabrication, and validation study. Human Pathology 35, 11, 1303-1314.
7. ↑ NG, R., LEVOY, M., BREDIF, M., DUVAL, G., HOROWITZ, M., HANRAHAN, P. 2005. Light Field Photography with a Hand-Held Plenoptic Camera. Stanford Tech Report CTSR 2005-02.
8. ↑ Levoy, Marc, Ren Ng, Andrew Adams, Matthew Footer, and Mark Horowitz. "Light Field Microscopy." ACM Transactions on Graphics 25.3 (2006): 924. Print.
9. ↑ Levoy, M. Zhang, Z. McDowall, I. Recording and controlling the 4D light field in a microscope. Journal of Microscopy, Volume 235, Part 2, 2009, pp. 144-162.
10. ↑ Light field microscopy - Stanford 2006 (pdf)
11. ↑ ZHANG, ZHENGYUN. "A Practical Introduction to Light Field Microscopy." (2010). A Practical Introduction to Light Field Microscopy. Web. 10 Mar. 2012.