Savu - DLS Large Data Processing Pipeline

28 Sept 2017, 10:00 → 29 Sept 2017, 15:30 Europe/Stockholm

MAX rooms Ia/b, II (Max IV Lab)

Zdenek Matej

Nicola Wadeson and Mark Basham, from the Data Analysis Group at the Diamond Light Source, will present a reconstruction and processing pipeline for large volume datasets coming from parallel-beam tomography synchrotron experiments.

"Savu" originates in a tomography community. However it is designed to allow greater flexibility and be adopted for other types of experiments as Powder diffraction, SAXS or Ptychography.

Savu Home on GitHub

In short:

Thursday morning (MAX IV room Ia-Ib):
10:30 - 11:00 - General introduction to Savu
11:15 - 11:35 - Savu demo
11:35 - 11:55 - Viktor Nikitin (MAX IV) - lprecon

Thursday afternoon (MAX IV room II):
13:30 - 14:30 - Hands on Savu: Example data

Friday (MAX IV room II):
10:30 - 11:30 - Hands on writing a Savu plugin
13:30 - 14:30 - Hands on Savu: Building a process list
14:50 - 15:30 - Summary and posible collaborations

more information savu-maxiv-info.pdf

Thursday 28 September

10:00 → 12:00 Savu: General introduction

10:00 Coffee

10:20 Introduction

10:30 Savu - Tomography Reconstruction and Processing Pipeline

General introduction to Savu data processing pipeline

Speakers: Dr Mark Basham (Diamond Light Source), Dr Nicola Wadeson (DLS)

Slides Savu Tomography pipeline - General introduction by Nicola Wadeson and Mark Basham

11:00 discussion

11:15 Savu - short demo

Speakers: Dr Mark Basham (Diamond Light Source), Dr Nicola Wadeson (DLS)

Slides Short Savu demo by Nicola Wadeson

Video Creating Savu processing list (screen recorded video) Running Savu: screen recorded video

11:35 lprecon

Log-polar reconstruction and its GPU implementation

Speaker: Viktor Nikitin

11:55 discussion

12:00 → 13:30 Lunch break

13:30 → 14:30 Hands on Savu: Example data

Hands on session on how to write processing list. (for interested users)

Conveners: Dr Mark Basham (Diamond Light Source), Dr Nicola Wadeson (DLS)

14:30 → 14:50 coffe break

14:50 → 15:20 Discussion on usefull and required features

15:20 → 17:30 Savu in MAX IV: (closed working session)

Optimization of Savu installation in MAX IV compute environment.

Conveners: Artur Barczyk, Dr Mark Basham (Diamond Light Source), Nicola Wadeson (DLS), Dr Rajmund Mokso (Max IV Laboratory, Lund University), Viktor Nikitin, Zdenek Matej

From Rajmund:

Discussion about various features (new and existing):

alignment:
        what is cSAX using? Is there any  software for this. Fiducial markers still work best.
        See if NINA could share some tools to be implemented in SAVU
       
GUI:
        • what type of details we want to expose to users
        • Is the best that the BL scientist always sets up the pipeline and then the users will not have to change anything.
        • buttons such as make a preview / run full reconstruction
        • Dawn could be one way if it is possible to trim back to only topography:
       
Streaming:
        • processing without saving any partial results  - some of these has been tried when SAVU works in the sinogram space and goes all the way
        • We maybe can work on this together: apply for grants MAX IV - DIAMOND
       
               
reliability:
        • has to be robust
        • SAVU2 has solved some issues already
        • the bottle neck in tenability/optimization of SAVU is in parallel Mpi HDF5. One would need a software which is portable and can be optimized at different sites.


Friday:

Bee Dataset:
padding was fixed
reconstruction with CGLS-CUDA in Astra gives also nice results

Viktor’s plugin:
fixed!

Friday 29 September

09:00 → 09:30 coffe break

09:30 → 10:30 Savu in MAX IV: (MAX IV cluster optimization - what has not been finished the previous day)

Optimization of Savu installation in MAX IV compute environment.

Conveners: Artur Barczyk, Dr Mark Basham (Diamond Light Source), Nicola Wadeson (DLS), Dr Rajmund Mokso (Max IV Laboratory, Lund University), Viktor Nikitin, Zdenek Matej

From Rajmund:

Discussion about various features (new and existing):

alignment:
        what is cSAX using? Is there any  software for this. Fiducial markers still work best.
        See if NINA could share some tools to be implemented in SAVU
       
GUI:
        • what type of details we want to expose to users
        • Is the best that the BL scientist always sets up the pipeline and then the users will not have to change anything.
        • buttons such as make a preview / run full reconstruction
        • Dawn could be one way if it is possible to trim back to only topography:
       
Streaming:
        • processing without saving any partial results  - some of these has been tried when SAVU works in the sinogram space and goes all the way
        • We maybe can work on this together: apply for grants MAX IV - DIAMOND
       
               
reliability:
        • has to be robust
        • SAVU2 has solved some issues already
        • the bottle neck in tenability/optimization of SAVU is in parallel Mpi HDF5. One would need a software which is portable and can be optimized at different sites.


Friday:

Bee Dataset:
padding was fixed
reconstruction with CGLS-CUDA in Astra gives also nice results

Viktor’s plugin:
fixed!

10:30 → 11:30 Hands on writing a Savu plugin

Introduction into writing Savu extensions (plugins), e.g. for data loading, filtering, alignment, integration, reconstruction etc. (for developers)

Speakers: Dr Mark Basham (Diamond Light Source), Dr Nicola Wadeson (DLS)

Slides Writing Savu plugins by Nicola Wadeson

11:30 → 12:00 disscussion and cofee break

12:00 → 13:30 lunch break

13:30 → 14:30 Hands on Savu: Building a process list

Hands on session on how to write processing list. (for interested users)

Conveners: Dr Mark Basham (Diamond Light Source), Dr Nicola Wadeson (DLS)

14:30 → 14:50 cofee break

14:50 → 15:30 Savu in MAX IV: Summary and posible collaborations

Optimization of Savu installation in MAX IV compute environment.

Conveners: Artur Barczyk, Mr Darren Spruce (MAX IV Laboratory), Dr Mark Basham (Diamond Light Source), Dr Nicola Wadeson (DLS), Dr Rajmund Mokso (Max IV Laboratory, Lund University), Viktor Nikitin, Zdenek Matej

From Rajmund:

Discussion about various features (new and existing):

alignment:
        what is cSAX using? Is there any  software for this. Fiducial markers still work best.
        See if NINA could share some tools to be implemented in SAVU
       
GUI:
        • what type of details we want to expose to users
        • Is the best that the BL scientist always sets up the pipeline and then the users will not have to change anything.
        • buttons such as make a preview / run full reconstruction
        • Dawn could be one way if it is possible to trim back to only topography:
       
Streaming:
        • processing without saving any partial results  - some of these has been tried when SAVU works in the sinogram space and goes all the way
        • We maybe can work on this together: apply for grants MAX IV - DIAMOND
       
               
reliability:
        • has to be robust
        • SAVU2 has solved some issues already
        • the bottle neck in tenability/optimization of SAVU is in parallel Mpi HDF5. One would need a software which is portable and can be optimized at different sites.


Friday:

Bee Dataset:
padding was fixed
reconstruction with CGLS-CUDA in Astra gives also nice results

Viktor’s plugin:
fixed!