If I want to read a lot mdf and concat them together ,is there easy way to save memory/calculate out of memory?

[shangrilaer]

Hello , I want to read a lot of mdf files during road test from several collages. but met a out-of-memory problem.
I want to make a big dataframe to do this statistics.so I can make some histograms or kind of things as calculating total distance /time this kind of things.

so I met some problems:
1, I need to resample different ,and asammdf helps and can do fast enough.
2, I want to join them together to a big dataframe, but met a out-of-memory problem. maybe the file is big or not , during reading will cost memory. during concat will cost momory . too many mdf files to join to a big dataframe will cost a lot . Is anyone have good ideas for this? I can use tall array in Matlab , but I dont know how to do in Python.

[epfi]

The last time I had to glue hundreds of measurements together, I first filtered out the relevant signals before calling concatenate(). Of course this only helps, in case you need just a small subset of the measured values for your analysis.

Also it might be a good idea to check, if you are using a 64 bit version of Python.

[shangrilaer]

Thank you very much for replying. I now was using this method to treat road fleet test record .

Now I am thinking two things , if we are upscaling to hundreds of measurements :
1, how to save time I can see only one CPU-core is working during extracting data from mdf files
2, I'm trying some bigdata tools such as Matlab:tall() , Python:Dask /pandas ; and find if we conert to parquet files will help to use this power tools to statistic .

[danielhrisca]

1, how to save time I can see only one CPU-core is working during extracting data from mdf files

Unfortunately this is how Python works at the moment. Even if we load the files in different processes it will be very costly(and probably impossible) to bring the loaded files in a common process for the concatenation

[danielhrisca]

You should definitely use the channel filtering at load time to speed up the process (see https://asammdf.readthedocs.io/en/latest/tips.html)