File handling capacity

[ravigupta131]

hi,
What is the maximum limit of files to be compared.
I found it working maximally with 22 files

[bmesuere]

We have tested it with classes of over 500 students, so this should work. Do you get a specific error when you're trying to run this?

[ravigupta131]

Yes i am getting error ‘maximum call stack size exceeded’

On Tue, 6 Dec 2022 at 1:27 AM, Bart Mesuere ***@***.***> wrote: We have tested it with classes of over 500 students, so this should work. Do you get a specific error when you're trying to run this? — Reply to this email directly, view it on GitHub <#968 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AP4WE5EP7GSITS4U7ZZFNGLWLZCJLANCNFSM6AAAAAASUVEIMA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- Regards, Ravi Kant Gupta Research Scholar EE Department IIT Bombay

[rien]

You may be analyzing very large and/or very similar files, Dolos can indeed start using a lot of memory in such situations.

You can decrease memory usage by using larger kgram length and window size by using the commandline arguments:

dolos -k 100 -w 100 ...

However, this decreases the ability to detect small copied fragments.

You can also increase the memory allocated to Node with the following environment variable:

NODE_OPTIONS=--max_old_space_size=4096

The size is in megabytes, so above command allocates 4GB for the node vm.

[ravigupta131]

Some time FAtal error : newspace :: revalance allocation failed - javascript heap out of memory When we run with around 40 ipynb files

On Tue, 6 Dec 2022 at 1:29 AM, Ravi Gupta ***@***.***> wrote: Yes i am getting error ‘maximum call stack size exceeded’ On Tue, 6 Dec 2022 at 1:27 AM, Bart Mesuere ***@***.***> wrote: > We have tested it with classes of over 500 students, so this should work. > Do you get a specific error when you're trying to run this? > > — > Reply to this email directly, view it on GitHub > <#968 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AP4WE5EP7GSITS4U7ZZFNGLWLZCJLANCNFSM6AAAAAASUVEIMA> > . > You are receiving this because you authored the thread.Message ID: > ***@***.***> > -- Regards, Ravi Kant Gupta Research Scholar EE Department IIT Bombay

-- Regards, Ravi Kant Gupta Research Scholar EE Department IIT Bombay

[rien]

You can also increase the memory allocated to Node with the following environment variable:

NODE_OPTIONS=--max_old_space_size=4096

The size is in megabytes, so above command allocates 4GB for the node vm.

[tsieger]

Hi,
thanks for such a neat tool. For me, the NODE_OPTIONS=--max_old_space_size=4096 does not seem to work. I process some 100 files. It goes smoothly with the default -w 17 parameter, also with -w 8 and with -w 4. However, when I use -w 2, I get:

[error] Maximum call stack size exceeded

and setting NODE_OPTIONS=--max_old_space_size=4096 (even to much larger values) does not help. Is there any way how to check the effective value of max_old_space_size? Or any other way how to increase the call stack size?
My node version is 16.14.0.
thanks

[rien]

You can check the effective value of max_old_space_size by looking at the process memory usage with any top-like tool (I personally prefer htop). This value effectively sets the heap size of the Node VM.

Decreasing the w (window) parameter should actually increase memory usage. Higher values of both k (kgram size) and w will decrease memory usage and analysis time, at the cost of decreasing Dolos' sensitivity for small similar fragments.

Can you check which version of Dolos you are running (with dolos --version), the latest version (v2.0.1) should have fixes for the call stack size and console color output.

[tsieger]

I'm running

Dolos v2.0.1
Node v16.14.0
Tree-sitter v0.20.1

Thanks for explaining the meaning of the w parameter. I was aware of that. And I know it was weird that I was trying to increase w when running into memory problems :-). In fact, I was testing how low can I go with w and when I ran into problems I was trying to resolve it by increasing max_old_space_size.

When I run dolos run -l cpp -w 4 *.cpp on some 100 source files (mean size 10k), Node takes ~2GB of memory.
When I run dolos run -l cpp -w 2 *.cpp on the same file set, dolos crashes with [error] Maximum call stack size exceeded, consuming (according to top) some 700M of memory. Increasing max_old_space_size does not help (which seems to be reasonable as the top mem usage is far below 4GB) .

I'm curious why the error message states Maximum call stack size exceeded. It seems to relate not to the heap memory, but the call stack. Perhaps it insinuates some problem with deeply nested data structures or recursive calls?

[rien]

Yes, that error is probably caused by recursive calls and some excessive use of the JavaScript spread operator, which puts a lot of items on the stack. We've done some improvements already, but we will focus on improving performance (runtime and memory wise) in our future releases (#974).

There is probably a node setting for increasing the stack size as well, but I don't know it by hearth.

[tsieger]

#974: great to know!

Unfortunately, I can't find node setting for stack size in https://nodejs.org/api/cli.html.
https://stackoverflow.com/questions/11332422/how-can-i-increase-the-maximum-call-stack-size-in-node-js recommends --max-stack-size=val, but

node: --stack_size= is not allowed in NODE_OPTIONS

[rien]

With Dolos v2.1.0, we should be able to handle a lot more files while using less memory. You should no longer encounter any Maximum call stack size exceeded errors.

[tsieger]

Thanks. Indeed, with Dolos v2.1.0, Node v16.18.1, Tree-sitter v0.20.1, comparing two C++ sources of 8MB (100k lines!, but real student's work :-) ) and 4k has not crashed, but also has not finished within 2 hours. Is that expectable for such a large input?

[rien]

Good to hear it does not crash, but 2 hours is very long for only two files.

However, it is not unexpected as we now use an algorithm that will scale O(n²) with the length of both files to find the longest common fragment. I think Dolos will also have issues in displaying the results, Dolos is not really made with submissions of 100k lines in mind, but we will think about adding some improvements.