[Python] hdf5 reader/writer (with optional dependencies)

[ritchie46]

We can reduce friction by figuring out how to load data most efficiently to polars memory.

[jorgecarleitao]

If there is a backend for this in Rust, I think we could work it out in arrow2. It is a quite important format imo.

[ritchie46]

That's a good idea too!

[ghuls]

HDF5 has a very big specification: https://docs.hdfgroup.org/hdf5/develop/_f_m_t3.html
and as HDF5 is very similar to a filesystem, data stored in HDF5 can be stored in quite a lot of different ways.

Rust bindings to libhdf5 can be found at: https://github.com/aldanor/hdf5-rust

[ritchie46]

I think we should explore both. Rust backed under a feature flag, and python as optional dependency. I can imagine that it increases binary size quite a bit.