Bug #112
_gdp_gcl_cache_get(): assertion fails: gdp_gcl_cache.c:227: _ep_thr_mutex_lock: mutex 0x39963e0 (gcl) already self-locked
0%
Description
When running the TestGDPToWebSocket.xml model locally under Darwin, the GDPToWebSocket.js process on terra crashes.
- Become the sbuser on terra and restart the GDPToWebSocket.js process. Eric, I added your ssh pub keys to ~sbuser/.ssh/authorized_keys2 file, so you should be able to use ssh -l sbuser terra.eecs.berkeley.edu
[sbuser@terra .ssh]$ ~/cg/node_modules/.bin/pm2 status ┌─────────────────────┬────┬──────┬──────┬────────┬─────────┬────────┬─────┬───────────┬──────────┐ │ App name │ id │ mode │ pid │ status │ restart │ uptime │ cpu │ mem │ watching │ ├─────────────────────┼────┼──────┼──────┼────────┼─────────┼────────┼─────┼───────────┼──────────┤ │ GDPTestGenerator.js │ 0 │ fork │ 8032 │ online │ 30666 │ 2s │ 0% │ 29.0 MB │ disabled │ │ GDPToWebSocket.js │ 8 │ fork │ 1604 │ online │ 34 │ 7m │ 0% │ 28.7 MB │ disabled │ └─────────────────────┴────┴──────┴──────┴────────┴─────────┴────────┴─────┴───────────┴──────────┘ Use `pm2 show <id|name>` to get more details about an app [sbuser@terra .ssh]$ ~/cg/node_modules/.bin/pm2 restart 8 Use --update-env to update environment variables [PM2] Applying action restartProcessId on app [8](ids: 8) [PM2] [GDPToWebSocket.js](8) ✓ ┌─────────────────────┬────┬──────┬──────┬────────┬─────────┬────────┬─────┬───────────┬──────────┐ │ App name │ id │ mode │ pid │ status │ restart │ uptime │ cpu │ mem │ watching │ ├─────────────────────┼────┼──────┼──────┼────────┼─────────┼────────┼─────┼───────────┼──────────┤ │ GDPTestGenerator.js │ 0 │ fork │ 8032 │ online │ 30666 │ 6s │ 0% │ 29.3 MB │ disabled │ │ GDPToWebSocket.js │ 8 │ fork │ 8069 │ online │ 35 │ 0s │ 0% │ 7.9 MB │ disabled │ └─────────────────────┴────┴──────┴──────┴────────┴─────────┴────────┴─────┴───────────┴──────────┘ Use `pm2 show <id|name>` to get more details about an app [sbuser@terra .ssh]$
- Run the model
cd $PTII svn update $PTII/bin/capecode $PTII/ptolemy/actor/lib/jjs/modules/gdp/demo/GDPToWebSocket/TestGDPToWebSocket.xml
- Double click on the WebSocketClient actor and change the server to
terra.eecs.berkeley.edu
Then click commit
Run the model, data should appear in a Display actor
Stop the model
Start the model again. No data appears
On terra as sbuser, look in ~sbuser/cg/GDPToWebSocket/pm2.accessor-8.log
*** Processing ack/nak 133=ACK_DATA_CONTENT from socket 17
gdp_pdu_proc_resp(0x3977580 ACK_DATA_CONTENT) gcl 0x39963e0
gdp_pdu_proc_resp: no req for incoming response
PDU@0x3977580:
v=3, ttl=15, rsvd1=0, cmd=133=ACK_DATA_CONTENT
dst=9cTqOvxpw1IYDDIUjhv4cg-w50dPJ09yS8pC_aGRgQU
src=T2bu1EV7VRFDM3Ao5ybbtWS8uxPrCOyJngCv0JMcils
rid=2, olen=24, chan=0x3997cc0, seqno=0
flags=0xa<HAS_RECNO,HAS_TS>
datum=0x7f82e0000e30, recno=318502, dbuf=0x7f82e0000f70, dlen=178
ts=2017-06-22 16:41:33.465243000Z
sigmdalg=0x0, siglen=0, sig=(nil)
total header=128
GCL@0x39963e0: T2bu1EV7VRFDM3Ao5ybbtWS8uxPrCOyJngCv0JMcils
iomode = 1, refcnt = 2, reqs = (nil), nrecs = 318501
flags = 0xe<INCACHE,ISLOCKED,INUSE>
freefunc = (nil), gclmd = (nil), digest = (nil)
utime = 2017-06-22 16:41:23, x = (nil)
4f66eed4457b551143337028e726dbb564bcbb13eb8ec899e0afd0931c8a5bReading GCL T2bu1EV7VRFDM3Ao5ybbtWS8uxPrCOyJngCv0JMcils
>>> gdp_gcl_open(T2bu1EV7VRFDM3Ao5ybbtWS8uxPrCOyJngCv0JMcils)
_gdp_pdu_in(ACK_DATA_CONTENT) => OK
*** Processing ack/nak 133=ACK_DATA_CONTENT from socket 17
Assertion failed at node /home/sbuser/cg/GDPToWebSocket/invoke.js:gdp_gcl_cache.c:227: _ep_thr_mutex_lock: mutex 0x39963e0 (gcl) already self-locked
I'm not sure why I'm not getting the libgdp version line? I'm running with debugging set to *=20, which is the same as what I run with in CapeCode, where I get the libgdp version line. One difference is that here we are running the node host, not CapeCode.
This error started appearing after I updated the gdp. I believe the last time I updated the gdp was June 5.
History
#1 Updated by Eric Allman over 6 years ago
- Status changed from New to Feedback
Sorry, but I can't reproduce this at all. I can start and stop the model many times and it keeps working.
#2 Updated by Nitesh Mor over 6 years ago
I can confirm that I have started running into this same issue starting sometime yesterday/today as well using Python bindings, although it seems to be non-deterministic. I will update once I get a way to reproduce this with some certainty.
#3 Updated by Nitesh Mor over 6 years ago
Eric, I tried a few times, but I can't reproduce the bug with the latest version (commit:cf05aa39)
Christopher, a potential reason for not getting the version line could be the output redirection not handling STDERR. The version line is sent to STDERR whereas the rest of the debug output is sent to STDOUT.
#4 Updated by Eric Allman over 6 years ago
Christopher, since neither Nitesh nor I can seem to be able to reproduce this under the latest GDP code, could you please update and give it another try? I did make a change since June 5 affecting GCL locking, although I wouldn't have thought it would affect this — but still, it's possible.
#5 Updated by Anonymous over 6 years ago
- Status changed from Feedback to Closed
Thanks for the tip about stderr and the gdp version string.
It seems like there is a problem with the GDP/WebSocket interface that uses Node. After an update of the gdp, I can see that the test log server is appending to the log because I can read from the log and publish to a WebSocket under Darwin. However, it does not work under RHEL as the sbuser until a delay of a minute or two.
I'm not sure why.
BTW - I did see one run have:
****** received from GDP: {"device":"TEST","pressure_pascals":72800,"humidity_percent":6,"temperature_celcius":94,"light_lux":60,"battery_percent":94,\
"sequence_number":6,"time":"2017-06-26T02:49:29.949Z"}
Assertion failed at node /home/sbuser/cg/GDPToWebSocket/invoke.js:gdp_event.c:265: gev->type != _GDP_EVENT_FREE
I'm not sure how to reproduce the above.
Thanks for the analysis, I'm closing this.