Performance drop in OpenMP build from Version 6.0.7.1 to current development version 7

[dpzimmer]

Hi all,

I have noticed a significant increase in run time (2.7X) for the attached case (ST4, RECT grid) from V6.0.7.1 to V7 develop. Some model logs are attached. These models were compiled with gfortran on a recent Ubuntu distro. Has anyone else noticed this behavior, or can explain it?

Perhaps related, I have noticed that the OMPX switch and related OpenMP pragmas were removed between versions... could this account for it?

Thanks in advance,
Don.
grid_V6.log
grid_V7.log
switch_v6.txt
switch_v7.txt
wave_V6.log
wave_V7.log

[JessicaMeixner-NOAA]

We do know that there have been a few performance hits moving along. A few have been addressed and updated in develop, although I'm sure there's plenty of others that can also be addressed. The OMPX switch wasn't really used (although this is based on memory, so perhaps could be fact checked), so that shouldn't be the issue. I don't remember it being 2.7 but I haven't run clean tests myself in a long time.

[JessicaMeixner-NOAA]

The commit you reference was simply to add variables as private so that we obtain the same answers no matter how many threads were used. Anything associated with OMPX's removal was entirely separate from that.

[dpzimmer]

Hi Jessica,

I do see that OMPX was removed at a later date, however its effectiveness was clearly removed in commit #160 as shown below with the newly commented out lines.

So, I am trying to figure out if this was inadvertent or intentional, and why. In either case, the change decimates OpenMP performance.

[ukmo-ccbunney]

That's a good point @dpzimmer
I can't immediately remember why those particular OMPX lines were removed, although I do vaguely remember a discussion around them being redundant - even though in this case it was removing a sensible looking OMP directive...

I see you have tried reinserting those particular OMP directives with significant performance gains. Are your results still bit-4-bit after doing this? If so, perhaps we should consider putting the directives back in.

One thing to note, however, is that some of the subroutines called in that loop have their own OMP threading, e.g. the W3PSMC subroutine. We would need to check how this interacts with threading in the outer loop...

[JessicaMeixner-NOAA]

I've gone back and looked at some old information I had about the changes to make things b4b. There's no mention of OMPX in these threads. Basically the requirement is to have everything be bit-for-bit no matter how many threads there are and the suspicion is that as code was updated, the commands for OMP were not, so the variables in the private section were updated so that we again retained bit-for-bit answers. I agree with @ukmo-ccbunney - if we can add these back in and maintain b4b answers then, let's do this. If we can do it with either OMPG or OMPH instead of adding the OMPX back in that might be desirable. I odn't remember exactly what the OMPX was for, and if it's needed, we can always add it back in, assuming answers are b4b.

[dpzimmer]

Thanks for the replies. I will run my fork with and without OMPX, compare the output and report back here in coming days.