Add support for Agentic AI to Sonador: RAG in Orthanc, MCP server, inference runtime, diagnosis of CXR conditions and report generation
NANDA: blueprint/architecture for agentic systems.
Architecture and changes to Sonador:
- Implement RAG/embeddings endpoints within Sonador
- Add
python-pgvectorto Orthanc Cloud plugin - Create embeddings models (similar to comments) for patient, study, and series
- Create REST endpoints (three total) for embeddings: patient, study, and series
- Add
- Update the Sonador environment
- Change the Orthanc database from PostgreSQL image to PgVector image
- Update Kafka and drop
etcd
- Sonador maintenance
- Update Django to 5.2 from 4.2 (beyond EoL)
- MCP server PoC: expose embeddings capabilities to agents
PoC: demonstrate the feasibility of end-to-end case planning for total knees. Needed actions/tasks: receive medical images, segment structures of interest (femur and tibia), approve segmentations (or refine), determine anatomic landmarks, determine plan parameters (cut-plan, implant, sizing, ... )
- Inference server
-
NVIDIA Triton
- Able to serve any type of self-hosted model
- Supports VLLM
-
VLLM
- Limited to LLMs only
- Cloud-native / provider tooling
-
NVIDIA Triton
- Runtime environment (DevOps): cloud versus edge (bare-metal K8s)
- MLflow: supports NVIDIA triton as a backend
Edited by Rob Oakes