{ "cells": [ { "cell_type": "markdown", "id": "520e30ae", "metadata": {}, "source": [ "## Quantify structure reconstruction accuracy" ] }, { "cell_type": "markdown", "id": "3f8e1934", "metadata": {}, "source": [ "The datasets used for this tutorial is the reconstruction of human metastatic lymph node and can be downloaded at:\n", "\n", "https://drive.google.com/drive/folders/1SwzdBRVat83-pmN3RTKLBuivWv5l7lfs?usp=sharing" ] }, { "cell_type": "markdown", "id": "c23a1ae8", "metadata": {}, "source": [ "This module provides comprehensive metrics for evaluating structural accuracy of interpolation results, supporting both 2D slice-level and 3D volume-level evaluation with boundary-based and voxel-based metrics." ] }, { "cell_type": "markdown", "id": "9e495248", "metadata": {}, "source": [ "#### Features\n", "\n", "- **2D and 3D Support**: Evaluate both individual slices and entire volumes\n", "- **Boundary-based Metrics**: Hausdorff Distance, Average Surface Distance, Chamfer Distance, Boundary IoU\n", "- **Voxel-based Metrics**: Dice Coefficient, Jaccard Index (IoU), FPR, FNR\n", "- **Batch Evaluation**: Process multiple slice pairs with automatic file matching" ] }, { "cell_type": "markdown", "id": "428ce031", "metadata": {}, "source": [ "#### Metrics Overview\n", "\n", "##### Boundary-based Metrics\n", "\n", "- **Boundary IoU**: Intersection over Union on boundary bands\n", "- **Hausdorff Distance (HD)**: Maximum distance between boundaries\n", "- **HD95**: 95th percentile Hausdorff Distance (more robust to outliers)\n", "- **Average Surface Distance (ASD)**: Symmetric mean surface distance\n", "- **Chamfer Distance**: Average distance from each boundary point to the nearest point on the other boundary\n", "\n", "##### Voxel-based Metrics\n", "\n", "- **Dice Coefficient**: F1 score for overlap\n", "- **Jaccard Index (IoU)**: Intersection over Union\n", "- **Hausdorff Distance (Voxel)**: HD computed on all voxels\n", "- **HD95 (Voxel)**: 95th percentile HD on all voxels\n", "- **Average Surface Distance (Voxel)**: ASD computed on all voxels\n", "- **Chamfer Distance (Voxel)**: Chamfer distance computed on all voxels\n", "- **False Positive Rate (FPR)**: Rate of false positives\n", "- **False Negative Rate (FNR)**: Rate of false negatives" ] }, { "cell_type": "code", "execution_count": 1, "id": "afdd91a2", "metadata": {}, "outputs": [], "source": [ "import unist" ] }, { "cell_type": "code", "execution_count": 2, "id": "02bc75d2", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: imagecodecs in /opt/anaconda3/envs/unist/lib/python3.10/site-packages (2025.3.30)\n", "Requirement already satisfied: numpy in /opt/anaconda3/envs/unist/lib/python3.10/site-packages (from imagecodecs) (1.26.4)\n" ] } ], "source": [ "! pip install imagecodecs" ] }, { "cell_type": "markdown", "id": "e09c2564", "metadata": {}, "source": [ "### 2D" ] }, { "cell_type": "code", "execution_count": 3, "id": "5c6b5a31", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Found 15 matched pairs for evaluation.\n", "--------------------------------------------------\n", "Processing pair 1: occupancy_z57.97_bin1_gray.tif vs occupancy1.tif\n", " ✓ Computed 13 metrics\n", "Processing pair 2: occupancy_z86.96_bin2_gray.tif vs occupancy2.tif\n", " ✓ Computed 13 metrics\n", "Processing pair 3: occupancy_z115.94_bin3_gray.tif vs occupancy3.tif\n", " ✓ Computed 13 metrics\n", "Processing pair 4: occupancy_z144.93_bin4_gray.tif vs occupancy4.tif\n", " ✓ Computed 13 metrics\n", "Processing pair 8: occupancy_z260.87_bin8_gray.tif vs occupancy8.tif\n", " ✓ Computed 13 metrics\n", "Processing pair 9: occupancy_z318.84_bin9_gray.tif vs occupancy9.tif\n", " ✓ Computed 13 metrics\n", "Processing pair 15: occupancy_z492.75_bin15_gray.tif vs occupancy15.tif\n", " ✓ Computed 13 metrics\n", "Processing pair 16: occupancy_z521.74_bin16_gray.tif vs occupancy16.tif\n", " ✓ Computed 13 metrics\n", "Processing pair 17: occupancy_z550.72_bin17_gray.tif vs occupancy17.tif\n", " ✓ Computed 13 metrics\n", "Processing pair 21: occupancy_z666.67_bin21_gray.tif vs occupancy21.tif\n", " ✓ Computed 13 metrics\n", "Processing pair 22: occupancy_z695.65_bin22_gray.tif vs occupancy22.tif\n", " ✓ Computed 13 metrics\n", "Processing pair 24: occupancy_z753.62_bin24_gray.tif vs occupancy24.tif\n", " ✓ Computed 13 metrics\n", "Processing pair 26: occupancy_z811.59_bin26_gray.tif vs occupancy26.tif\n", " ✓ Computed 13 metrics\n", "Processing pair 31: occupancy_z956.52_bin31_gray.tif vs occupancy31.tif\n", " ✓ Computed 13 metrics\n", "Processing pair 32: occupancy_z985.51_bin32_gray.tif vs occupancy32.tif\n", " ✓ Computed 13 metrics\n", "\n", "✓ Results saved to: /Users/shuilan/Documents/GitHub/UniST/evaluation_results.csv\n", "\n", "Summary Statistics:\n", " Dice IoU HD_voxel HD95_voxel ASD_voxel ChamferDistance_voxel FPR FNR BoundaryIoU HD HD95 ASD ChamferDistance\n", "mean 0.7501 0.7144 22.0124 1.1132 0.3955 0.7910 0.0293 0.2605 0.8271 22.0124 1.1132 0.3987 0.7975\n", "std 0.3663 0.4182 44.9466 1.6347 0.5811 1.1622 0.0455 0.3819 0.2544 44.9466 1.6347 0.5858 1.1715\n" ] } ], "source": [ "from unist.metrics import evaluate_slices\n", "\n", "results_df = evaluate_slices(\n", " true_dir=\"/Users/shuilan/Documents/GitHub/UniST/tumor_occupancy_tifs_gray\", # path to ground truth images (.tif)\n", " pred_dir=\"/Users/shuilan/Documents/GitHub/UniST/tumor_occupancy_tifs_rgb_10slices_avg_thresh\", # path to predicted images (.tif)\n", " true_pattern=r'_bin(\\d+)_gray\\.tif$',\n", " pred_pattern=r'^occupancy(\\d+)\\.tif$',\n", " output_csv=\"/Users/shuilan/Documents/GitHub/UniST/evaluation_results.csv\",\n", " compute_boundary=True,\n", " compute_voxel=True,\n", " boundary_thickness=1,\n", " boundary_tolerance=1,\n", " verbose=True\n", ")" ] }, { "cell_type": "markdown", "id": "f573754a", "metadata": {}, "source": [ "### 3D" ] }, { "cell_type": "markdown", "id": "782af617", "metadata": {}, "source": [ "This step is computationally intensive; we recommend running it on a machine with sufficient memory.\n", "\n", "Here I test on a subset of the volume which contains 4 slices." ] }, { "cell_type": "code", "execution_count": 3, "id": "ff593d6e", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Stacked 4 slices into volume shape (4, 266, 200)\n", "3D Dice: 0.6160\n", "3D HD95: 1.7321\n" ] } ], "source": [ "from unist.metrics import evaluate_volume_from_dirs\n", "\n", "results = evaluate_volume_from_dirs(\n", " true_dir=\"/Users/shuilan/Documents/GitHub/UniST/tumor_occupancy_tifs_gray_sub\",\n", " pred_dir=\"/Users/shuilan/Documents/GitHub/UniST/tumor_occupancy_tifs_rgb_10slices_avg_thresh_sub\",\n", " true_pattern=r'_bin(\\d+)_gray\\.tif$',\n", " pred_pattern=r'^occupancy(\\d+)\\.tif$',\n", " compute_boundary=True,\n", " compute_voxel=True,\n", " verbose=True,\n", ")\n", "print(f\"3D Dice: {results['Dice']:.4f}\")\n", "print(f\"3D HD95: {results['HD95']:.4f}\")" ] }, { "cell_type": "code", "execution_count": 4, "id": "27e05a7c", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'Dice': 0.6160497682219381,\n", " 'IoU': 0.44513867196205564,\n", " 'HD_voxel': 27.910571473905726,\n", " 'HD95_voxel': 1.7320508075688772,\n", " 'ASD_voxel': 0.46173223021893384,\n", " 'ChamferDistance_voxel': 0.9234644604378677,\n", " 'FPR': 0.029736855982703158,\n", " 'FNR': 0.44652567975830815,\n", " 'BoundaryIoU': 0.44484987489574646,\n", " 'HD': 27.910571473905726,\n", " 'HD95': 1.7320508075688772,\n", " 'ASD': 0.4620237222467427,\n", " 'ChamferDistance': 0.9240474444934854}" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "results" ] } ], "metadata": { "kernelspec": { "display_name": "unist", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.19" } }, "nbformat": 4, "nbformat_minor": 5 }