Consumer Update Delivered Last Sequence Behavior [v2.10.17] #5844
Comments
|
I set two the leaf nodes servers to trace mode and analyzed the logs. What I observed is this: (I tried to clean up the logs. I've left some lines that I think aren't important, but I'm not sure. I deleted lines that I believe are not relevant (I hope I didn't delete any important ones))
I don't have the expertise to fully analyze this, but maybe it helps. What seems most suspicious to me is this message: This ack message is received by both servers, but I understand that only the server where the consumer receives the message should receive it. However, I could be wrong... |
|
I made a test using different consumer names on each leaf node. I also had to change the way we consume messages from our Python program to a CLI command. Now, when I send a message to the stream with the highest sequence, the logs are:
Another important point is that we are mostly sure this behavior appeared when we changed the consumers from push to pull. |
|
What's happening here appears to be 2 domains with identical stream and consumer names. The ack messages like Because we don't track what are valid outstanding ACKs and just accept any ACK this causes actual ACKs to happen. |
|
Best option today is to make unique stream names in each domain. |
|
We do have a domain aware ACK format, but at least in my setup and yours that's not used - asking around to find out how that works. |
Make unique consumers is valid too? or is better to make unique streams? |
|
Unique consumer would also avoid it yeah |
wallyqs
changed the title
Observed behavior
Consumers increase the delivered last sequence, even though the stream they are connected to hasn't received any messages
Expected behavior
Each consumer only increases its own delivered last sequence when it reads a message from its stream.
Server and client version
In the cloud:
In the leafnodes:
Host environment
Cloud Environment:
IoT Environment:
Stream and Consumer Setup:
Each IoT device has the following NATS resources:
Specific Example (for IoT device “Alice”):
Steps to reproduce
Set up an environment as described, and send a message to down_alice with a client connected directly to the cloud. Then, check the delivered last sequence in all consumers.
Side effects
If we have Alice and Bob IoT devices, and we send 10 messages to Alice, both Bob's and Alice's consumers will have their delivered last sequence set to 10. If I now send a message to Bob, the message is lost. Both streams, 'down_bob' and 'down' (the local one), increase their own last sequence, but Bob's consumer does not change anything.
If I continue sending messages to Bob, the behavior remains the same until the sequence number of a message is 11. At that point, because it is greater than the consumer's delivered last sequence, the consumer reads it. However, Alice's consumer also increases its own last delivered sequence.
I suspect that the messages lost is because the 'down' queue is a workqueue, and since the consumer's last delivered sequence is higher than the sequence number of the message, the consumer assumes the message has already been read.
The stream with the highest sequence number will work (the messages will be read by the consumer), while the other consumers will discard all messages until their stream has the highest sequence.
If you need more information, feel free to ask.
Thanks!
The text was updated successfully, but these errors were encountered: