Welcome to RNA2seg’s documentation!

RNA2seg is a deep learning-based segmentation model designed to improve cell segmentation in Imaging-based Spatial Transcriptomics (IST). Traditional IST methods rely on nuclear and membrane staining to define cell boundaries, but segmentation can be challenging due to the variable quality of membrane markers.

RNA2seg addresses this issue by integrating an arbitrary number of staining channels along with RNA spatial distributions to enhance segmentation accuracy, particularly in regions with low-quality membrane staining. It is built on SpatialData, enabling seamless processing and analysis of spatial transcriptomics data.

Check out the Installation Guide section for further information about how to install the package.

released version and code

the package code is at https://github.com/fish-quant/rna2seg

12/03/25 RNA2seg 0.0.7 :

• fix RNA embbeding bug

• add pretrained model for brain data

Contents

• Installation Guide
• User guide
  • Apply RNA2seg on spatial data
    • Step 1: Create patches from Zarr files
    • Step 2: Initialize a RNA2segDataset
    • Step 3: Initialize a model RNA2seg
    • Step 4: Run segmentation
      • Add segmented cell in the spatialdata as shape
    • Step 5: Visualize segmentation
  • Training RNA2seg on Zarr-Saved SpatialData
    • Import
      • Set your own path
    • Step 1: Create training patches from Zarr files
    • Step 2: Create a Consistent Target for Training RNA2seg
    • Step 3: Training RNA2seg
      • Initialize a RNA2segDataset
      • Train / Validataion split
      • Initialize Dataloaders
      • Initilize RNA2seg Model
      • Training RNA2seg
• Pretrained Model
• Modules
  • RNA2segDataset
    • RNA2segDataset.__getitem__()
    • RNA2segDataset.__init__()
    • RNA2segDataset.get_rna_img()
    • RNA2segDataset.get_segmentation_img()
    • RNA2segDataset.get_staining_img()
  • RNA2seg
    • RNA2seg.__init__()
    • RNA2seg.encode()
    • RNA2seg.forward()
    • RNA2seg.load_model()
    • RNA2seg.run()
    • RNA2seg.save_model()
  • create_patch_rna2seg()

Support

If you have any questions relative to the package, please open an issue on GitHub.

Citation

If you use this library, please be sure to cite:

@article {Defard2025.03.03.641259,
     author = {Defard, Thomas and Blondel, Alice and Coleon, Anthony and Dias de Melo, Guilherme and Walter, Thomas and Mueller, Florian},
     title = {RNA2seg: a generalist model for cell segmentation in image-based spatial transcriptomics},
     elocation-id = {2025.03.03.641259},
     year = {2025},
     doi = {10.1101/2025.03.03.641259},
     publisher = {Cold Spring Harbor Laboratory},
     URL = {https://www.biorxiv.org/content/early/2025/03/11/2025.03.03.641259},
     eprint = {https://www.biorxiv.org/content/early/2025/03/11/2025.03.03.641259.full.pdf},
     journal = {bioRxiv}

}

Indices and tables

• Index

• Module Index

• Search Page