Light Field Microscope

Introduction

Background and Motivation

Traditional light microscopy has been used to illuminate miniature biological specimens since the mid-1600s. Since, researchers have come to realize and (attempt to) circumvent the inherent limitations of the technique. One such limitation is that of superimposed features on the captured image. With no depth information from the subject, it is difficult to extract data specific to the focal plane from the rest of the image.

Confocal microscopy excludes light from above or below the focal plane of the image using a pinhole placed where the light from the focal plane comes to a waist. It also provides a method of visualizing 3-dimensional images and has better spatial resolution than light field microscopy, but images require scanning the entire sample at every depth of interest. Light field has the advantage of capturing relative depth information in a single photo. This additional visual information comes at a cost, however. Diffraction places an upper limit on the lateral and axial resolution of the images. We thus sacrifice spatial resolution to obtain angular resolution.

The light field microscope allows angular resolution information to be recorded in a single image. The light field is a function that represents the amount of light traveling in every direction through every point in space. By recording a sample’s light field, one can produce perspective and parallax views of the specimen. This is accomplished by inserting a microlens array in a conventional bright field microscope, and analyzing the images in silico. A microlens array is an optical component containing a spread of thousands of lenses with diameters in the micron scale. Applying 3D deconvolution to the focal stacks from the array produces a series of cross sections which allow for a 3D reconstruction of the sample.

The objective of this project is to create a light field microscope from a Lytro™ camera and the Thorlabs materials available in the 20.345 classroom, record associated documentation and code, and describe recommended experiments for use in a teaching undergraduate laboratory.

Prior Work

The only prior work we have to build upon are papers by Ren Ng's original paper on the light field microscope^[1], his further work^[2], his thesis^[3], and some work from a postdoc at Harvard^[4]. This project has not been done previously in the 20.345 lab. Ruben actually seems to have taken some shots of neurons firing, (using Calcium green - Life Technologies) and watched them in the 3d rendering.

Design Choices

In designing the setup for our project, we essentially had two choices.

1. Buy the microlens array and a better CCD camera (more pixels, larger sensor)
2. Buy the commercial light field camera, the Lytro

We decided to purchase the Lytro for two reasons:

1. Cost. The Lytro is selling at $399 per camera now. If we were to have purchased the components separately, the microlens array would have been >$400 on its own.
2. Alignment. The Lytro's lens array and sensor have the pixel size, lens size, and lens pitch all perfectly suited to work in unison. In purchasing these components separately, it would have been very unlikely that we would have found products such that their sizes and alignments corresponded.

Lytro Camera

The Lytro camera teardown can be found here. It is essentially divided in half, with dense, complex set of optics in the front and the electronic components in the back. The microarray and the 8.1 megapixel sensor are located at the interface of these two sections.

Not much is known about the lenses, just that together they exhibit a high NA and are designed to allow minimal aberration. The camera has a optical zoom controlled by the electronics on the far end of the camera. A touch screen allows the consumer to refocus the image before capturing a light field, to review all past photos still on the camera, and to refocus after capturing the image.

The camera is charged and connected to a computer through a USB cable. The application currently only works on Mac OSX. Once plugged in, the Lytro prompts you to install its software, then uploads the new photos it took in the last session. In this process, it transfers two types of files to the Lytro images folder: IMG_001.lfp and IMG_001-stk.lfp. The .lfp is the full information set, the -stk.lfp represents a stack of focal depths that the software deemed significant.

Design

Optical setup

Our final setup (after many iterations) was simply the 20.309 bright field and epifluorescent microscope setup with a beam splitter and a Lytro lens array/camera in the image plane. We used the beam splitter so that we could include a focusing CCD camera in our setup, which was especially helpful when we were trying to focus the Lytro with the optics included. It turns out that it is useful anyways, since it seems that we get some extra mystery noise in the Lytro images, and therefore the CCD helps us quickly find a region of interest in the sample.

The fluorescent portion of our laser is as of yet, not built. It is very possible to add it, however, as shown in the diagram.

Evolution of the optical setup

The microscope design went through a few iterations and continues to evolve. We originally tried to capture images with the Lytro with the provided optics. This actually ended up being more work than it was worth. Our method was basically to have the image plane lie just in front of the Lytro optics and then force the Lytro to focus on that plane using a lens tissue. This method did not always work, and in fact usually the Lytro would not like focusing on the light from the image instead of the surrounding lab, and switch accordingly.

We originally tried to use a 2" lens telescope to magnify the image. At first the telescope was before the tube lens, but since that contributes to aberration it was then placed after the tube lens. We tried a number of different setups to magnify the sample image such that it filled the field of view of the Lytro (which was fairly large considering its nice optics). We tried to put a different tube lens in the system to magnify the image with a single lens, however the led to a considerable amount of spherical aberration. We tried using a little pre-made telescope, but we found using its built in aperture that our NA was probably limited by the objective.

We noticed that once we inserted the images into LFDisplay for 3d rendering, we could see a very bright light spot that moved as we rotated our image. To counteract this, we used a medium-NA halogen light with a diffuser for more uniform illumination.

After much suffering, Vincent finally took the reins and tore the Lytro apart. He removed the front end containing the optics, and put the microlens array/sensor setup at the image plane. We took much better images this way.

Image Processing

Extracting useful information from the Lytro™ .lfp images

Getting images from Lytro™

1. Pictures > Lytro™ (Show Package Contents) > images.
2. You will see images organized by upload, in .lfp and -stk.lfp. The .lfp are the larger raw light field files. The -stk.lfp have already determined the most important focal depths in the image and keep only those depths.

Obtaining lfptools

1. Download the lfptools from Nirav Patel on Github, or find it in Departmental > BioEng > BioEngLab > 20.309 > Students > Projects > Light Field Microscopy.
2. Unzip.
   1. In Macs. Open Terminal. cd to folder. make.
   2. If you get the error "make: command not found" then download Xcode, go to Preferences>Downloads, and Install Command Line Tools.
   3. Move files you created to the folders HD > usr > local > bin. This folders might be invisible.

Creating .jpg image stacks from -stk.lfp

1. In terminal: lfpsplitter ~/Desktop/IMG_0010-stk.lfp
   1. Or wherever you keep your LFP files.
2. It will save in the same folder: a .json, a .txt, and .jpgs corresponding to those depths that the algorithm deemed important.

Creating .raw from .lfp

1. In terminal: lfpsplitter ~/Desktop/IMG_0010.lfp
   1. Or wherever you keep your LFP files.
2. It will save in the same folder: a .raw, a metadata .json, a private metadata .json, and a table .json.

Creating .tif from .raw

1. Download raw2tiff from here
   1. In Macs. Open Terminal. cd to folder. make.
   2. If you get the error "make: command not found" then download Xcode, go to Preferences>Downloads, and Install Command Line Tools.
   3. Move files you created to the folders HD > usr > local > bin. This folders might be invisible.
2. In terminal: raw2tiff -w 3280 -l 3280 -d short ~/Desktop/IMG_0010_imageRef0.raw ~/Desktop/Out10.tif

Dealing with the hexagonal microarray

Vincent wrote code (see below) to

Visualizing the sample in LFDisplay

1. Download LFDisplay
2. Run. Data > Open Input > Single Image
3. If a square array (microarray was a regular grid of lenses)
   1. Move the center lenslet position until the red circle covers the center lenslet.
   2. Use dx in "Lenslet to the right" to give the program the correct distance between lenses.
   3. Do the same for dy in "One lenslet down."
4. Move the radio button at the top to 3D to see the rendering.

List of parts

Methodology

Disassembling the Lytro

Aligning the beam path

Making sample slides of fluorescent beads

Results

Images

We were able to capture a few images of beads, beads in agarose, Air Force Rulings, hair, and onion cells. Our depth of focus achieved in post-processing is unfortunately low, but we believe that this can be solved with a slightly different approach to our microscope design.

Issues

• noise
• light

Code

lf_fourier_slice.m

Departmental > BioEng > BioEngLab > 20.309 > Students > Projects > Light Field Microscopy > Code

• Reads .tif from raw2tiff
• Pads image with ~5% of the image size
• Establishes lenslet stats. AKA:
  • How far to the right is the closest lenslet in the same row?
    • dx: 9.930846473 dy: 0
  • How far diagonally is the closest lenslet one row above?
    • dx: 4.965423236 dy: 8.600911765
• Gets a .png mask for the lenslet
• Calls lenslet function to make new lenslet image in a new coordinate
• Constructs a rectangular equivalent of the lightfield

lenslet.m

Departmental > BioEng > BioEngLab > 20.309 > Students > Projects > Light Field Microscopy > Code

• Takes indices, location, cell distance, half cell, and mask
• Calculates new coordinates
• Establishes new image around new coordinates
• Masks with mask.png
• Returns new image and coordinates

Other

Acknowledgments

1. Nirav Patel, for reverse engineering the Lytro image and lfpsplitter.
2. Frank Warmerdam, Andrey Kiselev, Bob Friesenhahn, Joris Van Damme and Lee Howard for raw2tiff tools.
3. Other references: ^{[5] [6] [7] [8] [9] [10] [11]}

References

1. ↑ [1]
2. ↑ [2]
3. ↑ [3]
4. ↑ [4]
5. ↑ G. LIPPMAN, Epreuves réversibles donnant la sensation du relief, J. Phys. 7 (4) (1908) 821–825.
6. ↑ WEINSTEIN, R.S., DESCOUR, M.R., et al. 2004. An array microscope for ultrarapid virtual slide processing and telepathology. Design, fabrication, and validation study. Human Pathology 35, 11, 1303-1314.
7. ↑ NG, R., LEVOY, M., BREDIF, M., DUVAL, G., HOROWITZ, M., HANRAHAN, P. 2005. Light Field Photography with a Hand-Held Plenoptic Camera. Stanford Tech Report CTSR 2005-02.
8. ↑ Levoy, Marc, Ren Ng, Andrew Adams, Matthew Footer, and Mark Horowitz. "Light Field Microscopy." ACM Transactions on Graphics 25.3 (2006): 924. Print.
9. ↑ Levoy, M. Zhang, Z. McDowall, I. Recording and controlling the 4D light field in a microscope. Journal of Microscopy, Volume 235, Part 2, 2009, pp. 144-162.
10. ↑ Light field microscopy - Stanford 2006 (pdf)
11. ↑ ZHANG, ZHENGYUN. "A Practical Introduction to Light Field Microscopy." (2010). A Practical Introduction to Light Field Microscopy. Web. 10 Mar. 2012.