Add support for Agentic AI to Sonador: RAG in Orthanc, MCP server, inference runtime, diagnosis of CXR conditions and report generation

NANDA: blueprint/architecture for agentic systems.

Architecture and changes to Sonador:

  • Implement RAG/embeddings endpoints within Sonador
    • Add python-pgvector to Orthanc Cloud plugin
    • Create embeddings models (similar to comments) for patient, study, and series
    • Create REST endpoints (three total) for embeddings: patient, study, and series
  • Update the Sonador environment
    • Change the Orthanc database from PostgreSQL image to PgVector image
    • Update Kafka and drop etcd
  • Sonador maintenance
    • Update Django to 5.2 from 4.2 (beyond EoL)
  • MCP server PoC: expose embeddings capabilities to agents

PoC: demonstrate the feasibility of end-to-end case planning for total knees. Needed actions/tasks: receive medical images, segment structures of interest (femur and tibia), approve segmentations (or refine), determine anatomic landmarks, determine plan parameters (cut-plan, implant, sizing, ... )

  • Inference server
    • NVIDIA Triton
      • Able to serve any type of self-hosted model
      • Supports VLLM
    • VLLM
      • Limited to LLMs only
    • Cloud-native / provider tooling
  • Runtime environment (DevOps): cloud versus edge (bare-metal K8s)
    • MLflow: supports NVIDIA triton as a backend
Edited by Rob Oakes