M3D Viewer: 3D side panel and viewport integration for STL series (visibility, active/lock wireframe, color)

1. Overview

Provide a 3D sidebar panel that allows users to modify the presentation state of the Three.js (M3D) viewport, using the same Segmentation Table components used elsewhere in the Sonador Viewer. From the panel a user can:

  • Toggle the display of a model on/off.
  • Toggle a model between active (regular rendering) and locked (wireframe rendering).
  • Change a model's color via the color picker dialog.

STL series only. GLB series load a complete scene comprised of multiple objects; introspection/interaction with GLB scenes is out of scope (see §9). If the series is a GLB series, the side-panel should not be visible.

Design elaborated from the discussion on imaging-development-env#95 (notes of 2026-07-11).

2. Background and current state

  • extensions/viewer3d/src/components/panels/M3DViewerPanel.jsx is currently a stub: it attaches attachCoreSegmentationTableEvents and renders a bare SegmentationTable titled "Models" with no data source, no header content, and no action callbacks. This issue replaces the stub with a functional panel.
  • extensions/viewer3d/src/ohifComponents/OHIFDicomM3DViewport.js acquires models through the reference-counted M3D geometry cache (m3dCache/m3dCacheService.js, cache-first + single-flight over the Cornerstone3D geometry cache) and hydrates per-viewport Three.js instances (m3dCache/hydrateM3DInstance.js). For STL, each hydrated instance is a fresh Mesh over the shared BufferGeometry with its own MeshStandardMaterial — per-viewport color/wireframe changes are already safe by construction.
  • Per-instance display color is resolved from series metadata by OHIFDicomM3DViewport.getInstanceColor() (RecommendedDisplayCIELabValue → hex via dcmjs.data.Colors.dicomlab2RGB), available without fetching the encapsulated document.
  • The wiki pages 3D Model Viewer (Three.js / M3D) and VTK.js <-> Cornerstone3D Integration document the cache layer and the volume/MPR patterns this feature mirrors; the Cornerstone3D Integration Reference documents the conceptual model.

3. Functional requirements

  • FR-1 — Model list. For an STL series, the side panel lists every model (DICOM instance) in the series using the @ohif/ui-next SegmentationTable components.
  • FR-2 — Header. The M3D/STL series title is displayed in the Segmentation Header (SonadorSegmentationHeader, extensions/cornerstone/src/components/SonadorSegmentationHeader.jsx), following the layout used by SegmentationEditorPanel.jsx.
  • FR-3 — Row identity. Each row displays: number taken from the instance's Instance Number (0020,0013); title/description loaded from the DICOM Content Description (0070,0081) (fallback: Content Label (0070,0080), then Model <n>); color dot taken from the model color (§2, getInstanceColor; fallback defaultGeometryColor).
  • FR-4 — Visibility toggle. Per-model show/hide via the row's eye toggle, plus the whole-series toggle on the table header. Hidden models set mesh.visible = false in every M3D viewport displaying the series.
  • FR-5 — Active/lock toggle. A model is either active (regular material rendering) or locked (rendered in wireframe: material.wireframe = true). Lock is toggled from the row actions menu.
  • FR-6 — Color change. Row action opens the color picker dialog (callColorPickerDialog from @ohif/extension-cornerstone cextUtils, as used by editSegmentColor in extensions/vtk/src/commandsModule.js); the chosen color is applied to the model material in every M3D viewport.
  • FR-7 — Action visibility. Only Change Color and Lock/Unlock appear in the row actions menu. Rename and Delete must not be visible, and there is no Add Segment row. Requires the component refactor in §6.
  • FR-8 — Multi-viewport consistency. All M3D viewports displaying the same series render the same presentation state (visibility, lock/wireframe, color). When more than one viewport is visible, only the first initializes the segmentation state (labels, colors); subsequent viewports consume the existing state.
  • FR-9 — STL gating. The interactive features are enabled only when the display set's model type is STL (MIMETYPE_STL from extensions/viewer3d/src/sopClassHandlers/OHIFDicom3DSopClassHandler.js). For GLB series the panel renders the series header only, with no rows or actions.

4. Architectural requirements

These follow the architectural notes on imaging-development-env#95, expanded against the source:

  • AR-1 — State of record: Cornerstone3D segmentation metadata. Model presentation state is stored in the Cornerstone3D segmentation state (segmentation.state from @cornerstonejs/tools) as a segmentation whose segments correspond to the STL instances. State replication between the panel and viewports is driven by Sonador SegmentationService events (extensions/cornerstone/src/services/SegmentationService/SegmentationService.ts) — the same service the Segmentation Editor and volume viewer use.
  • AR-2 — No viewport or representation registration. The Three.js viewer is not part of Cornerstone3D's rendering system. M3D viewports must not register a Cornerstone3D viewport ID and must not call addSegmentationRepresentations. (The existing _viewportId GUID in OHIFDicomM3DViewport is a cache-refcount identity only and stays out of Cornerstone3D state.)
  • AR-3 — Same loading patterns as volume/MPR. Segmentation metadata loading follows the patterns in the volume viewport and MPR views: register segmentation state with c3dSegmentations.state.addSegmentations([...]) during load (compare Cornerstone3DLabelmapBaseView.loadSegImageVolume, extensions/vtk/src/components/Cornerstone3DLabelmapBaseView.js); publish the segmentation ID on the display set (displaySet.segmentationId via DisplaySetApi) so panels bootstrap through attachCoreSegmentationTableEvents / attachSegmentationAddTableEvents (extensions/viewer3d-volume/src/utils/cornerstone3dSegmentations.js) exactly as SegmentationEditorPanel.jsx and VolumeViewerPanel.jsx do.
  • AR-4 — First-viewport initialization. Before registering, check c3dSegmentations.state.getSegmentation(segmentationId); if present, skip registration and consume existing state (FR-8).
  • AR-5 — Event-driven viewport updates. The M3D viewport subscribes to SegmentationService events and updates local Three.js display state by pulling current state from the Cornerstone3D metadata cache (c3dSegmentations.state.getSegmentation, segmentLocking.isSegmentIndexLocked, segmentIndex.getActiveSegmentIndex) — events signal that something changed; the metadata is the source of truth for what the state is. This mirrors how SegmentationEditorLayout.js (extensions/seg-editor) consumes SEGMENT_ACTIVE / SEGMENT_LOCK / SEGMENT_COLOR_MODIFIED events.

5. State model

5.1 Segmentation registration (new: extensions/viewer3d/src/m3dCache/m3dSegmentationState.js)

A helper module alongside the existing m3dCache files:

  • getM3DSegmentationId(seriesInstanceUID) → m3dseg:<SeriesInstanceUID> — parallel to the m3d:<SOPInstanceUID> scheme in m3dGeometryId.js.
  • registerM3DSegmentation({ displaySet, models }) — called by OHIFDicomM3DViewport once all STL models are acquired (end of fetchModel). Guards per AR-4, then:
    • c3dSegmentations.state.addSegmentations([{ segmentationId, representation: { type: SegmentationRepresentations.Labelmap, data: {} }, config: { label: <SeriesDescription>, segments } }]) — Labelmap type is nominal; no representation is ever attached to a viewport (AR-2).
    • segments keyed by InstanceNumber: { segmentIndex, label: <ContentDescription>, active: true, locked: false, visible: true, color: <model hex color>, sopInstanceUID, geometryId }. Instance metadata is read via series.getInstanceByUID(sop).getData().metadata (same access pattern as getInstanceColor). sopInstanceUID/geometryId provide the segmentIndex <-> Three.js model mapping.
    • Publishes displaySet.segmentationId through DisplaySetApi.Instance.displaySetService.addDisplaySets([...]) (AR-3).
  • releaseM3DSegmentation(seriesInstanceUID) — removes the segmentation from Cornerstone3D state when the last M3D viewport for the series unmounts. Reference counting piggybacks on m3dCacheService.getReferenceCount() for the series' geometry IDs, so segmentation lifetime matches cached-geometry lifetime.

5.2 Viewport-independent state fields

State Storage Read API Notes
Active segment Cornerstone3D segmentIndex.getActiveSegmentIndex(segId) Already viewport-free
Locked Cornerstone3D segmentLocking.isSegmentIndexLocked(segId, idx) Already viewport-free; drives wireframe
Label Segmentation segments[idx].label state.getSegmentation(segId) Set at registration from Content Description
Visibility Segment metadata segments[idx].visible state.getSegmentation(segId) Per-viewport config.visibility API is unavailable without registered viewports (AR-2)
Color Segment metadata segments[idx].color state.getSegmentation(segId) Per-viewport config.color API is unavailable without registered viewports (AR-2)

Mutations to the metadata-backed fields go through segmentationService.addOrUpdateSegmentation({ segmentationId, segments }) so SEGMENTATION_MODIFIED fires and every subscriber re-pulls state (AR-5). Lock and active use the service's viewport-free methods (toggleSegmentLocked, setActiveSegment) which emit SEGMENT_LOCK / SEGMENT_ACTIVE.

5.3 Table-data mapping in viewport-free mode

c3dSeg2SegmentationTableData() and syncTableSegRepData() (extensions/viewer3d-volume/src/utils/cornerstone3dSegmentations.js) currently throw when getViewportIdsWithSegmentation() returns no viewports, and read color/visibility from the per-viewport config APIs. Add a source: 'viewport' | 'metadata' option (default 'viewport', preserving current behavior for all existing panels):

  • metadata mode skips the active-viewports check and reads color and visible from the segment metadata fields (§5.2); active and locked continue to use the viewport-free APIs they already use.
  • attachCoreSegmentationTableEvents accepts and forwards the option so the M3D panel's SEGMENTATION_MODIFIED handler regenerates table data in metadata mode.

6. Component refactor: action items render only when handlers are defined

platform/ui-next/src/components/DataRow/DataRow.tsx currently renders Rename, Delete, and Lock menu items unconditionally (only Copy and Color are conditional on their handlers), and platform/ui-next/src/components/SegmentationTable/SegmentationSegments.tsx always passes defined wrapper functions for onRename/onDelete even when the table-context callbacks (onSegmentEdit, onSegmentDelete) were never provided. Required changes:

  • DataRow.tsx — make onRename, onDelete, onToggleLocked optional in DataRowProps, and render each DropdownMenuItem (Rename / Duplicate / Delete / Change Color / Lock-Unlock) only when its handler is defined — the pattern already used for onCopy and onColor. When no action handlers are defined at all, render the placeholder spacer (<div className="h-6 w-6" />) instead of the dropdown trigger, matching the existing disableEditing branch.
  • SegmentationSegments.tsx — forward undefined rather than an always-defined wrapper when the corresponding context callback is absent, e.g. onRename={onSegmentEdit ? () => onSegmentEdit(segmentationId, segmentIndex) : undefined} (same treatment for onDelete, onToggleLocked, onColor, onToggleVisibility, onSelect).
  • Compatibility. These components were ported from OHIF v3 in the current development branch; keep the change additive (optional props, presence checks) so panels that pass every handler — SegmentationEditorPanel.jsx, VolumeViewerPanel.jsx, SonadorMprSegmentationPanel.jsx — render exactly as today.

7. Implementation plan by file

platform/ui-next

  1. components/DataRow/DataRow.tsx — conditional action items (§6).
  2. components/SegmentationTable/SegmentationSegments.tsx — conditional handler forwarding (§6).

extensions/viewer3d

  1. src/m3dCache/m3dSegmentationState.js (new) — id scheme, registerM3DSegmentation, releaseM3DSegmentation (§5.1); export from m3dCache/index.js.
  2. src/ohifComponents/OHIFDicomM3DViewport.js —
    • After STL fetchModel completes: registerM3DSegmentation (guarded per AR-4); publish displaySet.segmentationId.
    • Subscribe to segmentationService SEGMENTATION_MODIFIED, SEGMENT_LOCK, SEGMENT_ACTIVE; on each event pull segmentation metadata and reconcile every model: mesh.visible from visible, material.wireframe from locked, material.color from color (mapping segmentIndex -> model via sopInstanceUID/geometryId). Unsubscribe in componentWillUnmount and on display-set change, following the subscribe/unsubscribe discipline in SegmentationEditorLayout.js.
    • releaseM3DSegmentation alongside releaseAcquiredModels().
    • Needs servicesManager (or segmentationService) passed down — follow how ConnectedOHIFDicomM3DViewport wires props today.
  3. src/threejs/M3DModelView.js — extend the api object handed to onCreated with per-model presentation setters: setModelVisibility(geometryId, visible), setModelWireframe(geometryId, wireframe), setModelColor(geometryId, hexColor), plus getModelPresentation(geometryId). STL instances own their MeshStandardMaterial (hydrateM3DInstance), so material mutations are per-viewport-safe; a shared-state reconcile from segmentation metadata keeps viewports consistent (FR-8).
  4. src/components/panels/M3DViewerPanel.jsx — replace the stub (§8).

extensions/viewer3d-volume

  1. src/utils/cornerstone3dSegmentations.js — source: 'metadata' mode (§5.3).

8. Panel specification (M3DViewerPanel.jsx)

Follow the structure of SegmentationEditorPanel.jsx (state refs, attachCoreSegmentationTableEvents + attachSegmentationAddTableEvents with { source: 'metadata' }, portal container, SonadorSegmentationPanelTheme.module.scss):

  • SegmentationTable with title={t('Models')}, mode='expanded', data={m3dSegmentations}.
  • SegmentationTable.Expanded containing SonadorSegmentationHeader (series title; no Rename dropdown item — pass no dropdownMenuContent or an empty one) and SegmentationTable.Segments. No SegmentationTable.AddSegmentRow, no SegmentationTable.Config (labelmap fill/outline styles do not apply to meshes).
  • Callbacks supplied: onToggleSegmentationRepresentationVisibility (all models), onToggleSegmentVisibility, onToggleSegmentLock -> segmentationService.toggleSegmentLocked, onSegmentClick -> segmentationService.setActiveSegment, onSegmentColorClick -> callColorPickerDialog seeded from segment metadata color, writing back via addOrUpdateSegmentation. Callbacks not supplied (and therefore not rendered, per §6): onSegmentEdit, onSegmentDelete, onSegmentAdd, onSegmentCopy.
  • STL gating per FR-9, keyed off the display set's model type.

9. Out of scope

  • GLB series: scene introspection, per-node presentation control, and any GLB entries in the panel beyond the series header.
  • Undo/redo of presentation changes; persistence of presentation state beyond the session.
  • Writing presentation state back to DICOM.
  • OpenCascade/CAD interactions with the retained source bytes.

10. Acceptance criteria

  • Opening an STL series shows the Models panel with one row per instance: number = Instance Number, title = Content Description, color dot = model color.
  • Eye toggle hides/shows the model in the 3D viewport; header toggle hides/shows all models.
  • Lock action renders the model in wireframe; unlock restores regular rendering.
  • Change Color opens the color picker and applies the color to the model.
  • Row actions menu contains only Change Color and Lock/Unlock; Rename/Delete/Add Segment are absent. Panels that pass full handler sets are visually unchanged.
  • With two viewports on the same STL series, presentation changes made in the panel appear in both; the second viewport does not re-initialize labels/colors.
  • No Cornerstone3D viewport IDs or segmentation representations are registered by M3D viewports (verify via c3dSegmentations.state.getViewportIdsWithSegmentation returning empty for the M3D segmentation).
  • GLB series: panel shows the series header only; no interactive rows.
  • Unmounting the last M3D viewport for a series removes its segmentation from Cornerstone3D state; no leaked subscriptions (see the cleanup discipline in OHIFDicomM3DViewport.releaseAcquiredModels / componentWillUnmount).

11. References

Edited by Rob Oakes