Add some scaffolding to allow multiple advertisements in a single PDU.
Update license date on all files. No functional changes.
Return an error if retransmissions fail (this caused a SEGV).
Second cut at sorting asynchronous data records. Currently all done with
simulation (-Dtest.gdp.chan.seqno=102 randomizes the incoming sequence
numbers), and so missing or duplicated incoming records have not been
First pass at ordering results of asynchronous reads. Works if the
reads come in pre-sorted, but so far untested for out-of-order delivery.
Better logic about command retries; makes signature checking work. Now
Add optional signature (proof) checking when reading records. Lightly
tested. Seems to do the right thing if the signature does verify, but
fails incorrectly if it does not. Does not do any hash chain checking
Additional debugging for events and PDU processing for debugging out of
order PDU problems.
Tweaks to hopefully do a better job of handling out-of-order results for
Tweak debugging output to print "end of results" more preferentially
Produce more debug output in _gdp_req_dispatch.
First cut at fixing bug that sometimes caused an "End of Results" PDU to
be delivered before some of the results themselves. This involves
passing a "number of results sent" field in the "End of Results" PDU and
saving that PDU until the indicated number of results are processed....
Don't throw an assertion failure if an unimplemented command comes in.
Get signatures on APPEND working again. Still incomplete, but handles
the single datum in a request case.
Compilation and comment glitches.
Snapshot: change physical interface to make more efficient use of SQLite
when retrieving sets. Use this for asynchronous reads.
Checkpoint, mostly locking issues.
The grand renaming: GCL is (mostly) gone in favor of GOB and GIN.
Checkpoint: basic create, write, read all work. Subscriptions, async
commands, and multireads untested.
Snapshot: compiles cleanly with updated protocol and crypto
support, but completely untested.
Other fixes necessary to compile with a C++ compiler:
* Subtlety different semantics for extern keyword.
* Inability to do a forward declaration of a static variable.
* Structs must be initialized in the same order as declared.
* Cannot initialize a fixed size character array with a quoted string....
Changes for signed/unsigned conflicts and other type nits. C++ is much
more picky about these than C, but this is probably justified. Sadly,
some of the changes are pretty ugly.
First cut at flattening out protobuf definition. Some changes to data
structure printing (e.g., _gdp_pdu_dump) to get indenting more rational
so as to make reading the output easier.
First cut at running Layer 5 network protocol version 4 over old
(version 3) Layer 4 stack. Substantial parts are unimplemented,
including signature generation and checking, read or subscribe by
timestamp or hash, and others. No asynchronous operations including...
Snapshot. Not tested.
First cut at new net protocol. Compiles, but not yet tested (not even
once --- do not try to run this!).
First cut at "delete log" command (lightly tested).
First cut at separating GCL instances (GINs) from GDP Objects (GOBs).
This is intended to fix some semantic problems with subscriptions. End
users always use GINs (aliased to gdp_gcl_t for now for compatibility),
and there is one GIN for each call to gdp_gcl_open (i.e., no returning...
Generalize ack/nak to EP_STAT mapping to be more flexible and better
allow for improved error messages. Still more to be done, but the
basics are in.
Changes to handle certain error conditions with log creation. This
still isn't right: see the comments in gdp/gdp_main.c.
Snapshot on reliability work.
Delete some debugging cruft.
SNAPSHOT: still with some locking problems.
SNAPSHOT: fewer race conditions; still need more testing.
Initial implementation of unsubscribe.
Print better debugging info for command tracing.
Tweak debugging to make command traces easier.
Code cleanup for GDP command retry.
Make sure _gdp_invoke returns an error status if a command times out.
Cleanup (and hopefully simplification) of assertion subsystem. Lots of
changes, but nothing particularly obtuse.
Don't reset gcl->nrecs on reads unless the datum shows more records than
the GCL structure; this avoids problems with subscriptions in the
presence of asynchronous appends that would move the number of records
Attempt to avoid assertion error if the server fails on a command.
Implement a way to deliver "record missing" back to client.
Significant changes to locking to avoid problems with threaded clients.
Change EP_ASSERT_MUTEX_IS(UN)?LOCKED to EP_THR_MUTEX_ASSERT_IS(UN)LOCKED
for consistency with other routine names.
Make sure number of records is updated when getting subscription
Fairly extensive locking changes to avoid races/deadlocks.
Add special debug flag to do command tracing.
Update licenses to 2017.
Split req->pdu into two: cpdu holds the PDU for the command and rpdu
holds the PDU for the response. This makes it easier to make restart
work and the code to understand. However, it also means more copying of
data and a general lack of elegance. It also means that the datum in...
Extensive work on reliability, notably sharing a datum between threads.
Increases reliability, but still not fully tested. Basic things seem to
work though. Note: this may disrupt the replication code.
Avoid interspersed debugging messages when threaded.
First cut at using kinder, gentler assertions (mostly in gdp library).
Default swarm.gdplogd.sequencing.allowdups to true. Improved debugging.
Make sure that the ASYNCIO flag is off whenever we are waiting in
_gdp_invoke to avoid misinterpretation of responses. This makes the
req->state = WAITING very similar to !ASYNCIO; perhaps the flag should
Fix problem that might occur if an initial response PDU to a command
such as MULTIREAD is delayed; follow-on data was treated as a response to
the initial command rather than as asynchronous events.
Fix signature buffer reuse and ensure that datum->siglen ==
Fix problem raised by issue #16 (threaded applications failed in
unpredicatable ways). The problem was that calls to gdp_gcl_open need to
be single threaded so that there will be a unique name => handle mapping
for GCLs. This puts a mutex around the entire open call; not ideal...
Fix a problem associating "NOROUTE" responses with the correct command.
When swapping PDUs on a response, be sure signatures are properly
Don't keep retrying command if response is a server NAK.
Fix Assertion Failure after timeout (related to GCLF_INUSE).
Make sure that gdp_gcl_append returns the same dbuf that it was passed,
since the application conceivably could have kept a pointer to it.
More tweaking on re-synchronizing recno conflicts.
Add swarm.gdp.sequencing.allow(dups|gaps) options. Fix
resynchronization mechanism (hopefully).
Add some debugging hooks in GDP library.
Next take on read-by-timestamp. This version works for read, multiread,
and subscribe. There are still some glitches (e.g., you can't have a
starting timestamp in the future) but fixing these would require major
code restructuring, and besides, you can't specify a starting record...
First cut at read-by-timestamp. Adds Berkeley DB as a dependency (two
versions supported). The interface still isn't quite right; in
particular, subscribe and multiread don't take timestamps. This should
be remedied before this code is released.
Fix likely race condition in GCL req list handling; improved debugging.
Rename "extents" to be "segments" because of different potential
semantics and enable them by default.
Cleanup for convenience macros for ep_time_nanosleep.
Add debugging to force parallelism in gdplogd.
Fix bug that would cause clients to get out of sync if both appending
and subscribing to a single log.
Be more tolerant of missing records and duplicated records (preparation
Tweak some debugging levels to control verbosity.
First cut on implementing FWD_APPEND command. Still not working for all
More cleanup of reference counting for GCLs that have error conditions.
Additional debugging for request locking. Change _gdp_req_free API so
it can NULL-out the pointer to the request.
Fix a reference counting bug. Improve reference counting and locking
Add status code/NAK command corresponding to 410 Gone (for expired
Implement ability to create new extents. Currently there is no attempt
made to include authorization.
Fix a minor glitch in debugging output.
Update license. No code changes.
Adjust comment on Nitesh's patch. No code changes.
Bug-fix for number of records in GCL
On receiving data from gdplogd after a read command, check before copying
record number to gcl->nrecs. Otherwise reading any old data sets gcl->nrecs
to a smaller value, which then makes any subsequent appends fail.
Code cleanup: pull out dead code (#ifdefed out) and old SVN version
Include copyright in all files. There are scripts to do bulk
replacement of the copyright if desired.
Reliability fixes: reinitialize subscriptions periodically and retry
commands on router or server errors (allows gdplogd restarts with open
connections). Subscriptions will catch up and commands such as APPEND
will complete when the daemon returns. Adds a new thread in clients to...
First cut at recovering from gdplogd failures (which will result in
NAK_R_NOROUTE errors). Still doesn't work, but no worse than before.
Implement subscription renewals. This causes clients who have open
subscriptions to re-issue the subscription every
swarm.gdp.subscr.pokeintvl seconds. It's not complete yet: gdplogd
still needs to expire subscriptions after a timeout.
Work-in-Progress: automatic subscription renewals. Incomplete.
Implement read and append filtering so people can add hooks to do (for
example) encryption and decryption. Adds two new APIs:
EP_STAT (*filter)(gdp_datum_t *datum, void *closure)
gdp_gcl_set_read_filter(gcl, filter, closure)
gdp_gcl_set_append_filter(gcl, filter, closure)
Sample fix for Nitesh (Readable and Appendable logs)
Signature checking working. No action taken on bad signature yet.
Still considerable cleanup left to be done.
Implement asynchronous writes. Adds new interfaces:
* gdp_gcl_append_async -- initiate an asynchronous write.
* gdp_event_getstat -- get the status code from an event.
Asynchronous reads can be done using multiread.
Note that asynchronous writes do not attempt retransmissions, unlike...
Cleanup for code walkthrough:
* some function rearrangement
* make data structure detail printing more consistent
Cleanup prior to code review: chiefly comments, a bit of code reordering
for better flow.
Eliminate troublesome GDP_REQ_LOCKED flag; it was really only for
Significant changes to locking and subscriptions:
* Locks on gdp_req_t are now more extensive to avoid data races.
* Care is given to prevent subscription data being processed before
the subscription itself is in place.
* Move channel pointer into gdp_pdu_t purely as a convenience when...
Back out broad request locks since that ended up passing locks between
threads; the request state does this now.