AI in Microscope Image Processing

Overview

Cellpose is an advanced, deep-learning–based segmentation tool designed for microscopy images. As a generalist algorithm, it accurately segments diverse cell types (nuclei, cytoplasm, etc.) across different imaging modalities without requiring model retraining. With human-in-the-loop capabilities, researchers can refine results, adapt the model to their data, and apply the system to both 2D and 3D imaging workflows.

Key Features

Download or Access

Technical Background

Cellpose was introduced in a seminal study by Stringer, Wang, Michaelos, and Pachitariu, trained on a large and highly varied dataset containing over 70,000 segmented objects. This diversity enables the model to generalize across cell shapes, sizes, and microscopy settings, significantly reducing the need for custom training in most use cases. For 3D data, Cellpose cleverly reuses its 2D model in a "2.5D" fashion, avoiding the need for fully 3D-annotated training data while still delivering volumetric segmentation. Cellpose 2.0 introduced human-in-the-loop retraining, allowing users to manually correct predictions and retrain on their own images for improved performance on specific datasets.

Installation & Setup

conda create -n cellpose python=3.10

# For GUI support
pip install cellpose[gui]

# For minimal setup (API/CLI only)
pip install cellpose

Getting Started

GUI Mode

1. Launch the GUI by running: python -m cellpose
2. Drag and drop image files (.tif, .png, etc.) into the interface
3. Select model type (e.g., "cyto" for cytoplasm or "nuclei" for nuclei)
4. Set estimated cell diameter or let Cellpose auto-calibrate
5. Click to start segmentation and view results

Python API Mode

from cellpose import models

# Load model
model = models.Cellpose(model_type='cyto')

# Segment images
masks, flows = model.eval(images, diameter=30)

Refine & Retrain

1. After generating masks, correct segmentation in the GUI by merging or deleting masks manually
2. Use built-in training functions to retrain on corrected examples
3. Improved model performance on your specific dataset

Process 3D Data

1. Load a multi-Z TIFF or volumetric stack
2. Use the --Zstack flag in GUI or API to process as 3D
3. Optionally refine 3D flows via smoothing or specialized parameters for better segmentation

Limitations & Considerations

• Model Generality Trade-off: While the generalist model works broadly, highly unusual cell shapes or imaging conditions may require retraining.
• Annotation Effort: Human-in-the-loop training requires manual corrections, which can be time-consuming for large datasets.
• Installation Complexity: GUI installation may require command-line use, conda environments, and managing Python dependencies — not always straightforward for non-programmers.
• Desktop Only: Cellpose is designed for desktop use; no native Android or iOS applications available.

Frequently Asked Questions

Overview

StarDist is a deep-learning tool for instance segmentation in microscopy images. It represents each object (such as cell nuclei) as a star-convex polygon in 2D or polyhedron in 3D, enabling accurate detection and separation of densely packed or overlapping objects. With its robust architecture, StarDist is widely used for automated cell and nucleus segmentation in fluorescence microscopy, histopathology, and other bioimage analysis applications.

Key Features

Technical Background

Originally introduced in a MICCAI 2018 paper, StarDist's core innovation is the prediction of radial distances along fixed rays combined with object probability for each pixel, enabling accurate reconstruction of star-convex shapes. This approach reliably segments closely touching objects that are difficult to separate using traditional pixel-based or bounding-box methods.

Recent developments have expanded StarDist to histopathology images, enabling not only nucleus segmentation but also multi-class classification of detected objects. The method achieved top performance in challenges such as the CoNIC (Colon Nuclei Identification and Counting) challenge.

Download or Access

Installation & Setup

pip install stardist

pip install stardist-napari

Running Segmentation

from stardist.models import StarDist2D
model = StarDist2D.from_pretrained('2D_versatile_fluo')
labels, details = model.predict_instances(image)

Training & Fine-Tuning

Post-Processing Options

• Apply non-maximum suppression (NMS) to eliminate redundant candidate shapes
• Use StarDist OPP (Object Post-Processing) to merge masks for non-star-convex shapes

Limitations & Considerations

• Star-convex assumption may not model highly non-convex or very irregular shapes perfectly
• Installation complexity: custom installs require a compatible C++ compiler for building extensions
• GPU acceleration depends on compatible TensorFlow, CUDA, and cuDNN versions
• Some users report issues running the ImageJ plugin due to Java configuration

Frequently Asked Questions

Application Information

General Overview

SAM (Segment Anything Model) is a powerful AI foundation model created by Meta that enables interactive and automatic segmentation of virtually any object in images. Using prompts such as points, bounding boxes, or rough masks, SAM generates segmentation masks without requiring task-specific retraining. In microscopy research, SAM's flexibility has been adapted for cell segmentation, organelle detection, and histopathology analysis, offering a scalable solution for researchers needing a promptable, general-purpose segmentation tool.

Detailed Introduction

Originally trained by Meta on over 1 billion masks across 11 million images, SAM was designed as a promptable foundation model for segmentation with "zero-shot" performance on novel domains. In medical imaging research, SAM has been evaluated for whole-slide pathology segmentation, tumor detection, and cell nuclei identification. However, its performance on densely packed instances—such as cell nuclei—is mixed: even with extensive prompts (e.g., 20 clicks or boxes), zero-shot segmentation can struggle in complex microscopy images.

To address this limitation, domain-specific adaptations have emerged:

• SAMCell — Fine-tuned on large microscopy datasets for strong zero-shot segmentation across diverse cell types without per-experiment retraining
• μSAM — Retrained on over 17,000 manually annotated microscopy images to improve accuracy on small cellular structures

Key Features

Download or Access

User Guide

Limitations & Considerations

• Zero-shot performance inconsistent on dense or overlapping structures without domain tuning
• Segmentation quality depends heavily on prompt design and strategy
• GPU strongly recommended; CPU inference is very slow
• Struggles with very high-resolution whole-slide images and multi-scale tissue structures
• Fine-tuning or adapting SAM for microscopy may require machine learning proficiency

Frequently Asked Questions

Overview

AxonDeepSeg is an AI-powered tool for automatic segmentation of axons and myelin in microscopy images. Using convolutional neural networks, it delivers accurate three-class segmentation (axon, myelin, background) across multiple imaging modalities including TEM, SEM, and bright-field microscopy. By automating morphometric measurements such as axon diameter, g-ratio, and myelin thickness, AxonDeepSeg streamlines quantitative analysis in neuroscience research, significantly reducing manual annotation time and improving reproducibility.

Key Features

Technical Details

Developed by the NeuroPoly Lab, AxonDeepSeg leverages deep learning to deliver high-precision segmentation for neuroscientific applications. Pre-trained models are available for different microscopy modalities, ensuring versatility across imaging techniques. The tool integrates with Napari, allowing interactive corrections of segmentation masks, which enhances accuracy on challenging datasets. AxonDeepSeg computes key morphometric metrics, supporting high-throughput studies of neural tissue structure and pathology. Its Python-based framework enables integration into custom pipelines for large-scale analysis of axon and myelin morphology.

Download or Access

Installation & Setup

pip install axondeepseg napari

Important Considerations

• Performance may decrease on novel or untrained imaging modalities
• Manual corrections may be required for challenging or complex regions
• GPU is recommended for faster processing of large datasets; CPU processing is also supported

Frequently Asked Questions

Overview

Ilastik is a powerful, AI-driven tool for interactive image segmentation, classification, and analysis of microscopy data. Using machine learning techniques like Random Forest classifiers, it enables researchers to segment pixels, classify objects, track cells over time, and perform density counting in both 2D and 3D datasets. With its intuitive interface and real-time feedback, Ilastik is accessible to scientists without programming expertise and is widely adopted in cell biology, neuroscience, and biomedical imaging.

Key Features

Download

Getting Started Guide

Limitations & Considerations

• Interactive labeling can be time-consuming for very large datasets
• Accuracy depends on the quality and representativeness of user annotations
• Memory requirements — very high-resolution or multi-gigabyte datasets may require significant RAM
• Complex data — Random Forest classifiers may underperform compared to deep neural networks on highly variable or complex imaging data

Frequently Asked Questions