Project

General

Profile

Bug #112

_gdp_gcl_cache_get(): assertion fails: gdp_gcl_cache.c:227: _ep_thr_mutex_lock: mutex 0x39963e0 (gcl) already self-locked

Added by Anonymous over 2 years ago. Updated over 2 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
gdplogd
Start date:
06/22/2017
Due date:
% Done:

0%


Description

When running the TestGDPToWebSocket.xml model locally under Darwin, the GDPToWebSocket.js process on terra crashes.

  • Become the sbuser on terra and restart the GDPToWebSocket.js process. Eric, I added your ssh pub keys to ~sbuser/.ssh/authorized_keys2 file, so you should be able to use ssh -l sbuser terra.eecs.berkeley.edu
[sbuser@terra .ssh]$ ~/cg/node_modules/.bin/pm2 status
┌─────────────────────┬────┬──────┬──────┬────────┬─────────┬────────┬─────┬───────────┬──────────┐
│ App name            │ id │ mode │ pid  │ status │ restart │ uptime │ cpu │ mem       │ watching │
├─────────────────────┼────┼──────┼──────┼────────┼─────────┼────────┼─────┼───────────┼──────────┤
│ GDPTestGenerator.js │ 0  │ fork │ 8032 │ online │ 30666   │ 2s     │ 0%  │ 29.0 MB   │ disabled │
│ GDPToWebSocket.js   │ 8  │ fork │ 1604 │ online │ 34      │ 7m     │ 0%  │ 28.7 MB   │ disabled │
└─────────────────────┴────┴──────┴──────┴────────┴─────────┴────────┴─────┴───────────┴──────────┘
 Use `pm2 show <id|name>` to get more details about an app
[sbuser@terra .ssh]$ ~/cg/node_modules/.bin/pm2 restart 8
Use --update-env to update environment variables
[PM2] Applying action restartProcessId on app [8](ids: 8)
[PM2] [GDPToWebSocket.js](8) ✓
┌─────────────────────┬────┬──────┬──────┬────────┬─────────┬────────┬─────┬───────────┬──────────┐
│ App name            │ id │ mode │ pid  │ status │ restart │ uptime │ cpu │ mem       │ watching │
├─────────────────────┼────┼──────┼──────┼────────┼─────────┼────────┼─────┼───────────┼──────────┤
│ GDPTestGenerator.js │ 0  │ fork │ 8032 │ online │ 30666   │ 6s     │ 0%  │ 29.3 MB   │ disabled │
│ GDPToWebSocket.js   │ 8  │ fork │ 8069 │ online │ 35      │ 0s     │ 0%  │ 7.9 MB    │ disabled │
└─────────────────────┴────┴──────┴──────┴────────┴─────────┴────────┴─────┴───────────┴──────────┘
 Use `pm2 show <id|name>` to get more details about an app
[sbuser@terra .ssh]$
  • Run the model
cd $PTII
svn update
$PTII/bin/capecode $PTII/ptolemy/actor/lib/jjs/modules/gdp/demo/GDPToWebSocket/TestGDPToWebSocket.xml
  • Double click on the WebSocketClient actor and change the server to

terra.eecs.berkeley.edu

Then click commit

  • Run the model, data should appear in a Display actor

  • Stop the model

  • Start the model again. No data appears

  • On terra as sbuser, look in ~sbuser/cg/GDPToWebSocket/pm2.accessor-8.log

*** Processing ack/nak 133=ACK_DATA_CONTENT from socket 17
gdp_pdu_proc_resp(0x3977580 ACK_DATA_CONTENT) gcl 0x39963e0
gdp_pdu_proc_resp: no req for incoming response
PDU@0x3977580:
        v=3, ttl=15, rsvd1=0, cmd=133=ACK_DATA_CONTENT
        dst=9cTqOvxpw1IYDDIUjhv4cg-w50dPJ09yS8pC_aGRgQU
        src=T2bu1EV7VRFDM3Ao5ybbtWS8uxPrCOyJngCv0JMcils
        rid=2, olen=24, chan=0x3997cc0, seqno=0
        flags=0xa<HAS_RECNO,HAS_TS>
        datum=0x7f82e0000e30, recno=318502, dbuf=0x7f82e0000f70, dlen=178
                ts=2017-06-22 16:41:33.465243000Z
        sigmdalg=0x0, siglen=0, sig=(nil)
        total header=128
GCL@0x39963e0: T2bu1EV7VRFDM3Ao5ybbtWS8uxPrCOyJngCv0JMcils
        iomode = 1, refcnt = 2, reqs = (nil), nrecs = 318501
        flags = 0xe<INCACHE,ISLOCKED,INUSE>
        freefunc = (nil), gclmd = (nil), digest = (nil)
        utime = 2017-06-22 16:41:23, x = (nil)
4f66eed4457b551143337028e726dbb564bcbb13eb8ec899e0afd0931c8a5bReading GCL T2bu1EV7VRFDM3Ao5ybbtWS8uxPrCOyJngCv0JMcils

>>> gdp_gcl_open(T2bu1EV7VRFDM3Ao5ybbtWS8uxPrCOyJngCv0JMcils)
_gdp_pdu_in(ACK_DATA_CONTENT) => OK

*** Processing ack/nak 133=ACK_DATA_CONTENT from socket 17
Assertion failed at node /home/sbuser/cg/GDPToWebSocket/invoke.js:gdp_gcl_cache.c:227: _ep_thr_mutex_lock: mutex 0x39963e0 (gcl) already self-locked

I'm not sure why I'm not getting the libgdp version line? I'm running with debugging set to *=20, which is the same as what I run with in CapeCode, where I get the libgdp version line. One difference is that here we are running the node host, not CapeCode.

This error started appearing after I updated the gdp. I believe the last time I updated the gdp was June 5.

History

#1 Updated by Eric Allman over 2 years ago

  • Status changed from New to Feedback

Sorry, but I can't reproduce this at all. I can start and stop the model many times and it keeps working.

#2 Updated by Nitesh Mor over 2 years ago

I can confirm that I have started running into this same issue starting sometime yesterday/today as well using Python bindings, although it seems to be non-deterministic. I will update once I get a way to reproduce this with some certainty.

#3 Updated by Nitesh Mor over 2 years ago

Eric, I tried a few times, but I can't reproduce the bug with the latest version (commit:cf05aa39)

Christopher, a potential reason for not getting the version line could be the output redirection not handling STDERR. The version line is sent to STDERR whereas the rest of the debug output is sent to STDOUT.

#4 Updated by Eric Allman over 2 years ago

Christopher, since neither Nitesh nor I can seem to be able to reproduce this under the latest GDP code, could you please update and give it another try? I did make a change since June 5 affecting GCL locking, although I wouldn't have thought it would affect this — but still, it's possible.

#5 Updated by Anonymous over 2 years ago

  • Status changed from Feedback to Closed

Thanks for the tip about stderr and the gdp version string.

It seems like there is a problem with the GDP/WebSocket interface that uses Node. After an update of the gdp, I can see that the test log server is appending to the log because I can read from the log and publish to a WebSocket under Darwin. However, it does not work under RHEL as the sbuser until a delay of a minute or two.

I'm not sure why.

BTW - I did see one run have:

****** received from GDP: {"device":"TEST","pressure_pascals":72800,"humidity_percent":6,"temperature_celcius":94,"light_lux":60,"battery_percent":94,\
"sequence_number":6,"time":"2017-06-26T02:49:29.949Z"}
Assertion failed at node /home/sbuser/cg/GDPToWebSocket/invoke.js:gdp_event.c:265: gev->type != _GDP_EVENT_FREE

I'm not sure how to reproduce the above.

Thanks for the analysis, I'm closing this.

Also available in: Atom PDF