Image Segmentation

Basic concepts

Segmentation of images (also known as contouring or annotation) is a procedure to delinate regions in the image, typically corresponding to anatomical structures, lesions, and various other object space. It is a very common procedure in medical image computing, as it is required for visualization of certain structures, quantification (measuring volume, surface, shape properties), 3D printing, and masking (restricting processing or analysis to a specific region), etc.

Segmentation may be performed manually, for example by iterating through all the slices of an image and drawing a contour at the boundary; but often semi-automatic or fully automatic methods are used. Segment Editor module offers a wide range of segmentation methods.

Segmentation and segment

Result of a segmentation is stored in segmentation node in 3D Slicer. A segmentation node consists of multiple segments.

A segment specifies region for a single structure. Each segment has a number of properties, such as name, preferred display color, content description (capable of storing standard DICOM coded entries), and custom properties. Segments may overlap each other in space.


A region can be represented in different ways, for example as a binary labelmap (value of each voxel specifies if that voxel is inside or outside the region) or a closed surface (surface mesh defines the boundary of the region). There is no one single representation that works well for everything: each representation has its own advantages and disadvantages and used accordingly.

Binary labelmap Closed surface Fractional labelmap Planar contours, ribbons
easy 2D viewing and editing,
always valid (even if
transformed or edited)
easy 3D visualization quite easy 2D viewing
and editing,
always valid,
quite accurate
accurate 2D viewing and editing
inaccurate (finite resolution)
requires lots of memory
if overlap is allowed
difficult to edit,
can be invalid
(e.g., self-intersecting),
especially after non-linear
requires lots of memory ambiguous in 3D,
poor quality
3D visualization

Each segment stored in multiple representations. One representation is designated as the source representation (marked with a “gold star” on the user interface). The source representation is the only editable representation, it is the only one that is stored when saving to file, and all other representations are computed from it automatically.

Binary labelmap representation

Binary labelmap representation is probably the most commonly used representation because this representation is the easiest to edit. Most software that use this representation, store all segments in a single 3D array, therefore each voxel can belong to a single segment: segments cannot overlap. In 3D Slicer, overlapping between segments is allowed. To store overlapping segments in binary labelmaps, segments are organized into layers. Each layer is stored internally as a separate 3D volume, and one volume may be shared between many non-overlapping segments to conserve memory.

In a segmentation with its source representation set to binary labelmap, each layer is allowed to have different geometry (origin, spacing, axis directions, extents) temporarily - to allow moving segments between segmentations without unnecessary quality loss (each resampling of a binary labelmap can lead to slight changes). All layers are forced to have the same geometry during certain editing operations and when the segmentation is saved to file.

Segmentation modules

There are many modules in 3D Slicer for manipulating segmentations. Overview of the most important is provided below.

Segmentation modules in Slicer core:

  • Segmentations: Adjust display options, manage segment representations and layers, copy/move segments between segmentations, convert between segmentation and models and labelmap volumes, export to files.

  • Segment Editor: Create and edit segmentations from images using manual (paint, draw, …), semi-automatic (thresholding, region growing, interpolation, …) and automatic tools. A number of editor effects are built into the Segment Editor module and many more are provided by extensions (in “Segmentations” category in the Extensions Manager).

  • Segment statistics: Computes intensity and geometric properties for each segment, such as volume, surface, minimum/maximum/mean intensity, oriented bounding box, sphericity, etc. See more information in.

Extensions for creating/editing segmentations:

  • SegmentEditorExtraEffects: Adds 8 more effects to Segment Editor.

  • SurfaceWrapSolidify: fill in internal holes in a segmented image region or retrieve the largest cavity inside a segmentation.

  • MONAILabel: AI-based segmentation of various organs using MONAILabel.

  • TotalSegmentator: AI-based fully automatic segmentation of 104 structures in whole-body CT images.

  • DensityLungSegmentation: AI-based fully automatic lung segmentation.

  • HDBrainExtraction: AI-based fully automatic skull stripping in brain MRI images.

  • NVIDIA-AIAA: AI-based fully automatic segmentation of several organs. Segmentation is performed on a remote server.

  • RVesselX: Semi-automatic liver parenchyma and vessels segmentation.

Extensions for analyzing and processing segmentations:

  • Segment comparison: Compute similarity between two segments based on metrics such as Hausdorff distance and Dice coefficient.

  • Segment registration (provided by SegmentRegistration extension): Compute rigid or deformable transform that aligns two selected segments.

  • SegmentMesher: Creating volumetric (tetrahedral) meshes from segmentations.

  • OpenAnatomy: Export segmentations or model hierarchies for external viewers in glTF or OBJ format.

  • Sandbox: provides importer for Osirix ROI and SliceOmatic segmentation files.

For more extensions related to segmentations, open the “Segmentations” category in the Extensions Catalog.


To get started, check out these pages: