GDP

I'm currently pulling data from a log that I'm feeding the data to. So I think the data is being logged in there, it just gets stuck after a few data points collected through subscription.
I added the timeout, and now I'm using this line to get data from the log that I'm generating (new data every 4-5 second).
t = {'tv_sec':0, 'tv_nsec':10*(10**9), 'tv_accuracy':0.0}

newRecord = gdp.GDP_GCL.get_next_event(t)

The problem is that the first data is collected OK, but in the next one "None" is returned. (I assume it has timed out, I tried different values of timeout too)

Previously I was calling get_next_event(None), which would work for around 10-15 data points being collected through subscribe, and then it would get stuck on that line.

By the way, I just tried the same code on SwarmBox, and it isn't getting stuck there.

As I mentioned in my other bug report, I cannot run gdp-setup.bash on Raspberry Pi, because it is not supported. I run make, which compiles some of the c files, but fails on db.h.
So I'm not sure if the latest changes are applied to the RPi version.

Also, the issue does not exist on SwarmBox (which runs Ubuntu). I was able to run the setup bash script on SwarmBox though, which may or may not be the reason behind it working there.

Another point is that the sample python function for subscribe in gdp/lang/python/apps runs fine as far as I can see on both RPi and SwarmBox.

Nima, commit:4770b39a should have fixed the libdb compile problem (sorry, I don't think I remembered to tell you that I had fixed it). If that doesn't work, you can use sudo apt-get install libdb-dev.

I tried it again and apparently now it was able to install using the bash script. I still had the same freezing issue though when I tried the code right afterwards.
(Although now I'm receiving Error 600, which I think is a separate problem. I submitted a new bug for this)
Thanks.

Nima, can you clarify if the problem you are seeing is that your program hangs or gets a 600 error? The 600 error is probably a router problem (which has been acting up again). The hang could be because there are simply no events to read, which is not an error. If there is nothing to read and no timeout, get_next_event is supposed to hang. In particular, the only way to exit reader_multi_subscribe.py is to interrupt the process (probably using ^C), so that's not a bug.

On the other hand, if there is more data being fed to the logs to which reader_multi_subscribe.py has subscribed, but with no output, then there is definitely a bug.

The hanging issue was happening prior to the error 600. So they are separate issues.

There is data being added to the log, as I am writing to a log with one machine, and reading from it on a separate machine which hangs.

Hello,
I just tried again to reproduce the bug that I reported before (#55) and it is still there. I think it may have a different nature from the bug that has been resolved in #75, as I'm constantly writing to the log (every 5 seconds ish), yet still after a while the subscription gets stuck..
Is there any hope to resolve this?

ioClass.py(238): if self.IOtype == 'SER':
ioClass.py(256): newRecord = gdp.GDP_GCL.get_next_event(None)
--- modulename: GDP_GCL, funcname: get_next_event
GDP_GCL.py(628): return cls._helper_get_next_event(None, timeout)
--- modulename: GDP_GCL, funcname: _helper_get_next_event
GDP_GCL.py(542): __func1 = gdp.gdp_event_next
GDP_GCL.py(545): if __gcl_handle == None:
GDP_GCL.py(546): __func1_arg1_type = c_void_p
GDP_GCL.py(551): if timeout == None:
GDP_GCL.py(552): __timeout = None
GDP_GCL.py(553): __func1_arg2_type = c_void_p
GDP_GCL.py(563): __func1.argtypes = [__func1_arg1_type, __func1_arg2_type]
GDP_GCL.py(564): __func1.restype = POINTER(cls.gdp_event_t)
GDP_GCL.py(566): event_ptr = __func1(__gcl_handle, __timeout)


I tried another approach with 1 second timeout, and ignoring the result if the returned value was None. The code still hangs in the same line:


ioClass.py(256): t = {'tv_sec':0, 'tv_nsec':10*(10**9), 'tv_accuracy':0.0}
ioClass.py(257): newRecord = gdp.GDP_GCL.get_next_event(t)
--- modulename: GDP_GCL, funcname: get_next_event
GDP_GCL.py(628): return cls._helper_get_next_event(None, timeout)
--- modulename: GDP_GCL, funcname: _helper_get_next_event
GDP_GCL.py(542): __func1 = gdp.gdp_event_next
GDP_GCL.py(545): if __gcl_handle == None:
GDP_GCL.py(546): __func1_arg1_type = c_void_p
GDP_GCL.py(551): if timeout == None:
GDP_GCL.py(555): __timeout = GDP_DATUM.EP_TIME_SPEC()
GDP_GCL.py(556): __timeout.tv_sec = c_int64(timeout['tv_sec'])
GDP_GCL.py(557): __timeout.tv_nsec = c_uint32(timeout['tv_nsec'])
GDP_GCL.py(558): __timeout.tv_accuracy = c_float(timeout['tv_accuracy'])
GDP_GCL.py(559): __func1_arg2_type = POINTER(GDP_DATUM.EP_TIME_SPEC)
GDP_GCL.py(563): __func1.argtypes = [__func1_arg1_type, __func1_arg2_type]
GDP_GCL.py(564): __func1.restype = POINTER(cls.gdp_event_t)
GDP_GCL.py(566): event_ptr = __func1(__gcl_handle, __timeout)


I just deployed a more recent version of the code on gdp-0[2,3,4].eecs.berkeley.edu. (still need to upgrade gdp-01.eecs.berkeley.edu). The servers had not been updated in a while.

Could you verify that you still run into this issue with the most recent code? I would ask you to create a new log on a server other than gdp-01. You would do this by using the -s edu.berkeley.eecs.gdp-02.gdplogd flag for gcl-create.

If the issue persists, then could you please share the following:

• The version of GDP code that you are running. You can find it out by using strings /usr/lib/libgdp.so.0.7 | grep libgdp (or something similar, based on your specific environment/configuration). If you are not running somewhat recent version of the code, I'd ask you to upgrade and try to reproduce the bug.
• More debugging output. You can include gdp.dbg_set("*=50") in your Python code right after you call gdp.gdp_init().

Ok, this is certainly progress.

Could you please share the entire debug output (rather than a filtered version) for cases where you are missing events? I can see that incoming subscription data is discarded, because the event processing routine fails to find a corresponding subscription request. However, the filtered version does not tell me what happened to that missing request.

The output is sent to STDERR, so it should be fairly easy to redirect it to a file selectively.

Okay, it appears that you are using a multiread and a subscription at the same time.

Currently, multiple subscriptions and/or multireads to the same log in the same process don't work. We are aware of this bug and it requires some major changes in the way various components are structured. So, however annoying it might be, it is certainly not going to be fixed in the next few days.

Just a quick update, gdp-01 has been upgraded to the same state as the other three servers. So, you should not have to avoid gdp-01.

Well that explains it then.
I'm now just using subscribe and it seems to be working.
You may want to close this bug and I'll let you know if there were other issues.

Thanks.

I suggest creating a test case using python.

See gdp/lang/python/test/writer_subscriber2_test.py

If you add a file there and the name has _test in it, then it will be run
automatically.

_Christopher

On Sun, Oct 23, 2016 at 1:05 PM, gdp@berkeley.edu wrote:

Issue #55 has been updated by Nima Mousavi.

I tried another approach with 1 second timeout, and ignoring the result if
the returned value was None. The code still hangs in the same line:

ioClass.py(256): t = {'tv_sec':0, 'tv_nsec':10*(10**9), 'tv_accuracy':0.0}
ioClass.py(257): newRecord = gdp.GDP_GCL.get_next_event(t)
--- modulename: GDP_GCL, funcname: get_next_event
GDP_GCL.py(628): return cls.helper_get_next_event(None, timeout)
--- modulename: GDP_GCL, funcname: helper_get_next_event
GDP_GCL.py(542): _func1 = gdp.gdp_event_next
GDP_GCL.py(545): if __gcl_handle == None:
GDPGCL.py(546): _func1_arg1_type = cvoidp
GDP_GCL.py(551): if timeout == None:
GDP_GCL.py(555): _timeout = GDP_DATUM.EP_TIME_SPEC()
GDP_GCL.py(556): _timeout.tv_sec = c_int64(timeout['tvsec'])
GDP_GCL.py(557): timeout.tv_nsec = c_uint32(timeout['tv_nsec'])
GDP_GCL.py(558): _timeout.tv_accuracy = c_float(timeout['tv_accuracy'])
GDP_GCL.py(559): _func1arg2type = POINTER(GDP_DATUM.EP_TIME_SPEC)
GDP_GCL.py(563): __func1.argtypes = [func1_arg1_type, _func1_arg2_type]
GDP_GCL.py(564): __func1.restype = POINTER(cls.gdpevent_t)

GDP_GCL.py(566): event_ptr = func1(gcl_handle, __timeout)¶

Bug #55: GDP log subscribe hangs
https://gdp.cs.berkeley.edu/redmine/issues/55#change-286

• Author: Nima Mousavi
• Status: Feedback
• Priority: Normal
• Assignee: Eric Allman
• Category: libgdp
• Target version:

I've been using the function calls that have been used in
gdp/lang/python/reader_multi_subscribe.py, and I'm facing this issue.
After getting data for a while, the running just freezes. I traces the
runtime and found that is freezes in __func1 call. (Trace attached)

I didn't face a similar issue when running reader_multi_subscribe.py (at
least not while I was watching.). Do you know what may be the source of the
problem?
I've tried two different logs so far:
edu.berkeley.eecs.swarmlab.device.c098e5300003
edu.berkeley.eecs.bwrc.device.c098e530005d

Thanks.
Nima
Files gdp.JPG
https://gdp.cs.berkeley.edu/redmine/attachments/download/49/gdp.JPG

(163 KB)¶

You have received this notification because you have either subscribed to
it, or are involved in it.
To change your notification preferences, please click here:
https://gdp.cs.berkeley.edu/redmine/my/account

--
Christopher Brooks, PMP
Academic Program Manager
CHESS/Ptolemy/TerraSwarm
707.332.0670, cxh@eecs.berkeley.edu, https://ptolemy.eecs.berkeley.edu/~cxh