Mirrored stream stops when an invalid source Nats-Expected-Last-Subject-Sequence appears [v2.10.20]

[VeryStrongFingers]

Observed behavior

Creating a Mirrored stream using a source that contains messages with Nats-Expected-Last-Subject-Sequence headers still performs sequence checks as per the source stream once performed.

Expected behavior

Mirrored streams to ignore Nats-Expected-Last-Subject-Sequence headers (also possibly other 'Expect' headers?).

Existence of the message in the source alone means the original assertion was valid.
There is no need for a mirror to have the same expectations and checks as the original source stream.

Alternatively an option for mirrored streams to remove source headers would work too for a resolution. (#5409)

Server and client version

nats-cli: 0.1.5
nats-server: v2.10.20

Host environment

Tested in both Linux/amd64 & Darwin/arm64

Steps to reproduce

# Create stream
nats stream create cool_stream \
	--subjects="cool.*" --storage=file \
	--replicas=1 --retention=limits \
	--discard=old --max-msgs=-1 --max-msgs-per-subject=-1 \
	--max-bytes=-1 --ack --max-age=-1 --max-msg-size=-1 \
	--dupe-window=2m0s --allow-rollup \
	--no-deny-delete --no-deny-purge

# Publish a bunch of messages
nats pub "cool.aaa" --header "Nats-Expected-Last-Subject-Sequence:0"  "a"
nats pub "cool.aaa" --header "Nats-Expected-Last-Subject-Sequence:1"  "b"
nats pub "cool.aaa" --header "Nats-Expected-Last-Subject-Sequence:2"  "c"
nats pub "cool.aaa" --header "Nats-Expected-Last-Subject-Sequence:3"  "d"
nats pub "cool.aaa" --header "Nats-Expected-Last-Subject-Sequence:4"  "e"
nats pub "cool.aaa" --header "Nats-Expected-Last-Subject-Sequence:5"  "f"
nats pub "cool.aaa" --header "Nats-Expected-Last-Subject-Sequence:6"  "g"
nats pub "cool.aaa" --header "Nats-Expected-Last-Subject-Sequence:7"  "h"

# Remove sequence number 5 from stream (message body: e)
nats stream rmm cool_stream 5 -f

# Create mirrored stream (after messages are already published)
nats stream create cool_stream_mirrored \
	--storage=file --replicas=1 \
	--retention=limits --discard=old \
	--max-msgs=-1 --max-msgs-per-subject=-1 \
	--max-bytes=-1 --ack --max-age=-1 \
	--max-msg-size=-1 --dupe-window=2m0s \
	--allow-rollup --no-deny-delete \
	--no-deny-purge --mirror=cool_stream

# Publish some more messages

nats pub "cool.aaa" --header "Nats-Expected-Last-Subject-Sequence:8"  "i"
nats pub "cool.aaa" --header "Nats-Expected-Last-Subject-Sequence:9"  "j"
nats pub "cool.aaa" --header "Nats-Expected-Last-Subject-Sequence:10"  "k"

# View streams
nats stream ls

╭────────────────────────────────────────────────────────────────────────────────────────────╮
│                                           Streams                                          │
├──────────────────────┬─────────────┬─────────────────────┬──────────┬───────┬──────────────┤
│ Name                 │ Description │ Created             │ Messages │ Size  │ Last Message │
├──────────────────────┼─────────────┼─────────────────────┼──────────┼───────┼──────────────┤
│ cool_stream_mirrored │             │ 2024-09-08 21:35:20 │ 4        │ 376 B │ 15m12s       │
│ cool_stream          │             │ 2024-09-08 19:37:21 │ 10       │ 943 B │ 2.25s        │
╰──────────────────────┴─────────────┴─────────────────────┴──────────┴───────┴──────────────╯

cool_stream_mirrored stream will no longer mirror any new messages because of the invalid Nats-Expected-Last-Subject-Sequence header

[36649] 2024/09/08 21:35:20.258473 [WRN] Error processing inbound mirror message for '$G' > 'cool_stream_mirrored': last sequence by subject mismatch: 5 vs 4

Relevant code seems to be

• nats-server/server/stream.go

  Lines 2468 to 2474 in 7f92c34

  	if err != errLastSeqMismatch {
  	mset.mu.RLock()
  	accName, sname := mset.acc.Name, mset.cfg.Name
  	mset.mu.RUnlock()
  	s.RateLimitWarnf("Error processing inbound mirror message for '%s' > '%s': %v",
  	accName, sname, err)
  	} else {

• nats-server/server/stream.go

  Lines 4742 to 4752 in 7f92c34

  	if err != nil || fseq != seq {
  	mset.mu.Unlock()
  	bumpCLFS()
  	if canRespond {
  	resp.PubAck = &PubAck{Stream: name}
  	resp.Error = NewJSStreamWrongLastSequenceError(fseq)
  	b, _ := json.Marshal(resp)
  	outq.sendMsg(reply, b)
  	}
  	return fmt.Errorf("last sequence by subject mismatch: %d vs %d", seq, fseq)
  	}

[derekcollison]

Yes this is a known issue we plan on improving in 2.11.