Bug #74
Asyn Write/Subscribe
0%
Description
If I write async to log, router log is filled with such messages:
Incomplete PDU fd 8 pduLen 439 bufferDataLen 103
Incomplete PDU fd 8 pduLen 441 bufferDataLen 157
Incomplete PDU fd 8 pduLen 440 bufferDataLen 210
Incomplete PDU fd 8 have 54 need 80 bytes
....
....
Also the watch is sending data after 10s, with gdp-reader subscribed model, it seems I can read only the first burst of data, after that it seems to be just waiting. I can't read data there on, if restarted I can see that records have been written.
Everything seems to work with Sync write, however slower. Please advise.
gdplogd-log.txt 
(71.5 KB)
Related issues
History
#1 Updated by Syed Hasnain almost 7 years ago
The log also contains records like the following for async writing
recno -1, len 0, ts 0
#2 Updated by Eric Allman almost 7 years ago
Incomplete PDU fd 8 pduLen 439 bufferDataLen 103
Incomplete PDU fd 8 pduLen 441 bufferDataLen 157
Incomplete PDU fd 8 pduLen 440 bufferDataLen 210
Incomplete PDU fd 8 have 54 need 80 bytes
FYI, these are really warnings rather than errors. It just means that the router (gdp_router_click) wasn't able to read an entire PDU in one read(2) syscall. It keeps reading until it has an entire This can happen based on network MTUs, congestion, or several other reasons. However, it is odd that these are so small; normally a PDU that fits into one ethernet frame (about 1500 octets) will appear at the same time. There may be a clue there.
#3 Updated by Nitesh Mor almost 7 years ago
- Related to Bug #75: Subscription times out added
#4 Updated by Nitesh Mor almost 7 years ago
I believe this is an issue related to subscriptions, rather than mode of append (synchronous vs asynchronous). More specifically, a subscriber sends a subscription-renewal request to the server if it hasn't received any data in a while. If a subscriber keeps receiving data periodically (with a short enough period), no subscription renewals are needed.
There was a bug (see #75) which caused the subscription renewals to not work as expected. I hope that commit:7bf6e9b2 fixes this, but it hasn't been deployed to the production infrastructure yet. I am wondering if the append workload triggered this subscription bug.
Could you please see if this fixes the issue? After a git pull, a sudo make install in the top directory for the source should install the latest log-server code appropriately.
#5 Updated by Nitesh Mor almost 7 years ago
- Status changed from New to Feedback
#6 Updated by Nitesh Mor almost 7 years ago
Ali, can you please report back whether this fix for stuck subscriptions works for you?
#7 Updated by Syed Hasnain almost 7 years ago
- File gdplogd-log.txt added
Nitesh, As I mentioned earlier, our watches our sending out packets after different sleep times. For Instance for the sensors on the watch, we store data for 10s locally before transmitting it to gateway and then to GDP. For sync mode I see a delay before we can even read data at gateway, while writing to GDP. With Asyn mode writing works without delay. However, Previously I was only able to read records from the first burst after subscription.
After the fix, I can read more records, but randomly the subscription hangs i.e. I can see data is being written by writer, but I don't receive any data. It then reads the data (for one burst) and exits with this message (the reader client was run with -D *=10 flag):
End of Subscription
STATUS: WARNING: end of file [EPLIB:generic:3]
<<< gdp_gcl_close(107OxcL32wGOxpz7Q_NVoJ2kQMIC93pP-Is584lzODE): OK
exiting after 1179 records with status WARNING: end of file [EPLIB:generic:3]
I have attached the log for a test where I was writing to logs, and was subscribed to it as well.
#8 Updated by Eric Allman over 4 years ago
- Status changed from Feedback to Closed