Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CityJSON creation silently fails on big datasets #91

Open
vvmruder opened this issue Nov 4, 2019 · 2 comments
Open

CityJSON creation silently fails on big datasets #91

vvmruder opened this issue Nov 4, 2019 · 2 comments

Comments

@vvmruder
Copy link
Contributor

vvmruder commented Nov 4, 2019

Trying out 3dfier on a dataset of approximately 6000 polygon features and LAS data around 400 million points end up in silent exit of 3dfier with simple std::bad_alloc output. Tracing this down leads to static packaged JSON library https://github.com/tudelft3d/3dfier/blob/master/thirdparty/nlohmann-json/json.hpp
All process until file creation was going correctly as far as I interpret the output.

I tried this on a machine with 16gb of RAM. Clearly this is the bottleneck. But I think there could be some improvement. Maybe writing to the file not in one dump but in steps? Or at least catch this a bit better to give some hint why this fails.

Running the same on a machine with 48gb of RAM works through all process and results in proper output file.

@tcommandeur
Copy link
Contributor

I haven't found this issue yet. I did create datasets with more then 6000 polygons into CityJSON output before but not a lot. What was the output size of the CityJSON file on the 48gb RAM machine?
Large file output with CityGML is fine since we implemented it ourselfs by writing all to a file stream. Not sure how the NLohmann json package does the trick. I imagine it keeps the complete object in memory and writes to disk once at the end.

@vvmruder
Copy link
Contributor Author

vvmruder commented Nov 6, 2019

Resulting file was around 1.9gb.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants