Orthanc S3 issueshttps://code.oak-tree.tech/oak-tree/medical-imaging/orthanc-s3/-/issues2023-02-13T12:59:40Zhttps://code.oak-tree.tech/oak-tree/medical-imaging/orthanc-s3/-/issues/3Refactor Osimis AWS S3 object storage plugin to include a transfer manager2023-02-13T12:59:40ZRob OakesRefactor Osimis AWS S3 object storage plugin to include a transfer manager### Background
For the past two years, Oak-Tree has used Orthanc (version 1.9) alongside an S3 plugin (https://code.oak-tree.tech/oak-tree/medical-imaging/orthanc-s3) and the PostgreSQL plugin for the imaging database within our Sonador ...### Background
For the past two years, Oak-Tree has used Orthanc (version 1.9) alongside an S3 plugin (https://code.oak-tree.tech/oak-tree/medical-imaging/orthanc-s3) and the PostgreSQL plugin for the imaging database within our Sonador project. This configuration has worked very well and allows us to store several terabytes of studies within a single Orthanc instance. (You can find the exact Orthanc config here: https://code.oak-tree.tech/oak-tree/medical-imaging/imaging-development-env/-/blob/master/config/orthanc/config.web-auth.json.)
There are some downsides to this setup, however:
* Because the S3 plugin (originally developed by Radpoint) is not compatible with newer version of Orthanc (>1.10), we're currently stuck on 1.9. This prevents us from taking advantage of some of the new (and cool!) features that are beginning to emerge in Orthanc, like the extra database tags.
* Though it's worked well to date, some of our databases are now getting large enough that the Radpoint S3 plugin does hit a performance wall. This happened pretty dramatically last month with one of our instances and seems to happen when there are several thousand large studies with many series in each one.
In the past few weeks, the performance issues have caused significant enough pain that we need to do something. To that end, I've look at a number of options such as upgrading to the Radpoint plugin to the new storage API or adopting the Orthanc object storage plugins. Unfortunately, there isn't an ideal upgrade path that BOTH gets us to Orthanc 1.11 and keeps the same general performance profile we've seen with the Radpoint plugin. The performance difference between the two plugins (Radpoint and Orthanc object storage) is pretty stark. The Radpoint plugin is often anywhere from twenty to one-hundred times faster.
To that end, I decided to try and refactor the Orthanc plugin so that it provided many of the same options as the Radpoint plugin. The most important change would be a transfer agent capable of async/multi-threaded upload and download.
Unfortunately, while I was able to get something working, it doesn't provide the speed up I was hoping for. The transfer agent doesn't work correctly and sometimes corrupts file metadata, so even though I've got an implementation, I've just turned it off. Worse, in terms of raw throughput, the transfer manager is very similar to the "direct" implementation which transfers data synchronously. This makes me think that there might need to be changes elsewhere in the code to enable asynchronous/multi-threaded transfer and get us closer to the performance of the Radpoint plugin. (I'm not an expert on the internals of Orthanc, but it looks like there was a big change in the internal API of the storage plugins in version 1.10.)
### Goal
Refactor the object storage plugins of Orthanc to allow for the same level of async performance as seen in the Radpoint (legacy) plugin.Industrial Strength Imaging Platform Part 2: Monitoring, More Scalability Work, Additional Cloud Tuning