interop: don't run interop measurements if the Transfer testcase failed

[WesleyRosenblum]

Issue #, if available:

Description of changes: This change attempts to speed up the interop tests by skipping the measurements when the Transfer testcase fails, as it is unlikely Crosstraffic and Goodput will succeed anyway.

Rev 2:

• Removed Quant and Quicly as they have consistently failed and don't show short term signs of improvement
• Disabled ZeroRtt, Http3, and Resumption

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[dougch]

Benchmark report
No change in performance detected.

[camshaft]

Wouldn't this be a failure instead of unsupported? Isn't the idea that if transfer failed these would also fail?

[WesleyRosenblum]

I went with unsupported since if its Failure it would show up as a link to the logs in the report, but there would be no logs if it isn't run.

[camshaft]

Hmm... I guess it's not a big deal either way. My concern is we just end up having a less accurate sense of what's passing.

[WesleyRosenblum]

I think it would be better to add Goodput and Crosstraffic to the required check then

[camshaft]

While you're here speeding up interop... We should probably disable quant and quicly since they don't pass at all and show no sign of changing in the short to mid future.

Also we should just comment out the tests in the interop lists that we know we don't support. The current behavior still spins up the whole stack just to find the 127 exit status code.

[dougch]

Benchmark report
No change in performance detected.

[WesleyRosenblum]

We should probably disable quant and quicly since they don't pass at all and show no sign of changing in the short to mid future.

Yeah Quant isn't looking too active: NTAP/quant#77 (comment)

Also we should just comment out the tests in the interop lists that we know we don't support. The current behavior still spins up the whole stack just to find the 127 exit status code.

Is that just TestCaseIPv6? I believe all the others are in the enum: https://github.com/awslabs/s2n-quic/blob/663f86947f9996728723cce135c026faa22c9eef/quic/s2n-quic-qns/src/interop.rs#L84-L96 even if we don't actually support them yet. Or I could remove ZeroRtt, Http3, ConnectionMigration, Resumption, KeyUpdate too

[WesleyRosenblum]

Or I could remove ZeroRtt, Http3, ConnectionMigration, Resumption, KeyUpdate too

actually is that enum just used for running individual tests, since we do run ConnectionMigration, KeyUpdate and Ipv6 in the CI? I'll just remove ZeroRtt, Http3, and Resumption

[dougch]

Benchmark report
No change in performance detected.

[camshaft]

That is exactly what I was thinking

[dougch]

Benchmark report
No change in performance detected.