2.3.0: add -b flag to gdp-reader to allow binary output.
Version 2.2.3: Make failure to read a secret key for a writable log an
error instead of waiting for the first append to fail (this can be
turned off using gdp_open_info_set_no_skey_nonfatal, which causes
gdp_gin_open to return a warning instead of an error). Add a -S flag to...
Add GDP_INIT_OPT_HONGDS and swarm.gdp.hongd.optional to not give an
error if HONGD is unavailable (i.e., the default has been changed to be
more restrictive); do stricter checking during initialization (and give
better error messages); bump version to 2.1.19.
Do primitive caching of default creation service name.
Pass flags to GDP library initialization routines. Add flags to disable
Zeroconf and suppress initialization of HONGDS, which is unused in some
Update license date on all files. No functional changes.
Fix a problem with locating secret keys for logs when using a search
path. Do some prep for making this more flexible. It turns out the
semantics of -K in some applications has changed, but the current
behavior still matches the documentation.
There may be security implications with allowing the ability to name...
Cleanup on human-oriented name handling during creation, including
documentation. Big changes: don't allow a "human name" that looks like
a base-64-encoded name; use GDP_NAME_ROOT when creating a new log using
gdp-create; treat a zero-length GDP_NAME_ROOT as though the variable...
Mega-change: new object creation API with updated metadata; internal
name is hash of metadata, not hash of human name; query and maintain
human-oriented name to GDPname directory (HONGD).
Begin implementation of signature checking by readers.
First cut at external => internal name mapping using MySQL. It is
currently not an error if the mapping doesn't exist --- it just falls
back to the old SHA256 behavior --- and there is no automated addition
of the mapping to the database when you create a log. To make this work...
Avoid problems with the name of gdplogd during upgrades.
Improvements to event and metadata printing.
Pull old back-compat defines out of gdp.h. They are now in
Get signatures on APPEND working again. Still incomplete, but handles
the single datum in a request case.
Cleanup signature and hash type handling. There's no point in storing
the MD algorithm with each datum since it is fixed per GOB, nor in
passing it over the wire since it is just a blob unless you have either
the secret or public key, either of which contain that information...
Better take on gdp_gin_read_by_hash_async for multiple records.
Comment typo fixes.
Change GDP_EVENT_EOS to GDP_EVENT_DONE, since it isn't specific to
subscriptions. Also documentation updates reflecting the datum changes
in the previous commit.
It turns out that including a GOB pointer in the datum causes too many
locking problems. Just pass the gob in as needed.
The grand renaming: GCL is (mostly) gone in favor of GOB and GIN.
Simplify the datum interface back to what it was before. Basic
subscriptions and subscription timeout seems to work now.
Snapshot: compiles cleanly with updated protocol and crypto
support, but completely untested.
Snapshot in on-disk format conversion from ad hoc files to SQLite
databases. Compiles but is completely untested.
Add UUID to new log metadata to guarantee unique names once we switch to
the new naming scheme (name == HASH(metadata)).
First step at adding QoS information to subscriptions.
First cut at flattening out protobuf definition. Some changes to data
structure printing (e.g., _gdp_pdu_dump) to get indenting more rational
so as to make reading the output easier.
First cut at running Layer 5 network protocol version 4 over old
(version 3) Layer 4 stack. Substantial parts are unimplemented,
including signature generation and checking, read or subscribe by
timestamp or hash, and others. No asynchronous operations including...
First cut at new net protocol. Compiles, but not yet tested (not even
once --- do not try to run this!).
First cut at "delete log" command (lightly tested).
First cut at separating GCL instances (GINs) from GDP Objects (GOBs).
This is intended to fix some semantic problems with subscriptions. End
users always use GINs (aliased to gdp_gcl_t for now for compatibility),
and there is one GIN for each call to gdp_gcl_open (i.e., no returning...
Minor tweaks for _GDP_MODE_PEEK (it's a bit mask, not an absolute
More work on making helgrind happy. This seems to be pretty close,
maybe worth a try in production.
Initial implementation of unsubscribe.
Define log metadata field for expiration time.
Add gdp_gcl_open_info_set_signkey_cb to set a callback to get the secret
key. This is to avoid asking for a decryption password if it isn't
going to be used (e.g., if you are opening a GCL that is already in the
cache). This is particularly important for gdp-rest.
Attempt to allow GCLs to be cached in applications. This is
specifically for gdp-rest, but there may be other cases where it is
Implement a way to deliver "record missing" back to client.
Add new internal "I/O mode" _GDP_MODE_PEEK to signal _gdp_gcl_cache_get
to not update the use time; this is used for statistics gathering.
Add some new log metadata names. Still needs proper documentation.
Update licenses to 2017.
Snapshot: get basic reading working again with new lock changes.
Add gdp_datum_copy routine to copy datum contents.
Yet more debug tweaking.
Implement gdp_gcl_read_async. This also fixes some obscure problems
with other asynchronous operations, and tweaks debugging flags to make
it easier to pick out the operations you want to see.
Make many parameters "const" where possible.
Next take on read-by-timestamp. This version works for read, multiread,
and subscribe. There are still some glitches (e.g., you can't have a
starting timestamp in the future) but fixing these would require major
code restructuring, and besides, you can't specify a starting record...
First cut at read-by-timestamp. Adds Berkeley DB as a dependency (two
versions supported). The interface still isn't quite right; in
particular, subscribe and multiread don't take timestamps. This should
be remedied before this code is released.
Rename "extents" to be "segments" because of different potential
semantics and enable them by default.
First cut on implementing FWD_APPEND command. Still not working for all
Changes to make it possible to include gdp.h and gdp_priv.h in C++
Implement ability to create new extents. Currently there is no attempt
made to include authorization.
Add -q flag to gdp-reader to suppress all metadata output; this allows
it to be piped into another program to read the data.
Update license. No code changes.
Include copyright in all files. There are scripts to do bulk
replacement of the copyright if desired.
Implement subscription renewals. This causes clients who have open
subscriptions to re-issue the subscription every
swarm.gdp.subscr.pokeintvl seconds. It's not complete yet: gdplogd
still needs to expire subscriptions after a timeout.
Add new gdp_gcl_open_info interface to allow tweaking of GCL parameters
on open. The only thing included now is the setting of the signing key.
Use the third parameter to gdp_gcl_open for general open information,
not just QoS.
Change on-disk log to store signature.
Return that signature on READ commands.
Update reader-test to add a -v flag to print signature info.
Document new interfaces.
Implement read and append filtering so people can add hooks to do (for
example) encryption and decryption. Adds two new APIs:
EP_STAT (*filter)(gdp_datum_t *datum, void *closure)
gdp_gcl_set_read_filter(gcl, filter, closure)
gdp_gcl_set_append_filter(gcl, filter, closure)
Add log-mirror command for trivial replication. Required also adding
gdp_gcl_getnrecs to API.
Allow logs to opened for both read and append (GDP_MODE_RA).
Check GCL mode when doing I/O (to avoid writing to a read-only log or
reading an append-only log).
Fix some initialization bugs that caused mutex failures.
Add creation time to logs. This also changes the ep_time_format and
Another signing snapshot: signatures are created and sent but not
checked. Metadata is returned when a GCL is opened. Still have a
problem with reference counting of GCL handles.
Snapshot adding signing. Signatures are computed but not sent or
Implement asynchronous writes. Adds new interfaces:
* gdp_gcl_append_async -- initiate an asynchronous write.
* gdp_event_getstat -- get the status code from an event.
Asynchronous reads can be done using multiread.
Note that asynchronous writes do not attempt retransmissions, unlike...
Cleanup for code walkthrough:
* some function rearrangement
* make data structure detail printing more consistent
Add -t option to reader-test to print plain text (instead of a hexdump).
Also changes gdp_datum_print to take a flags argument.
Clean up gdp_datum_print (it's not just for debugging any more!).
Change gdp_event_next to allow selection of events for a specific GCL.
Run subscription callbacks in dedicated thread. Untested.
Add placeholder qos requirements parameter on open.
Document gdp_gcl_getstat as taking a name, not a handle.
Rename PUBLISH => APPEND (API change).
Eliminate gdp_datum_setrecno and gdp_datum_setts (didn't make sense).
Fix example code in protocol documentation.
Implement gdplogd shutdown actions: withdraw advertisements, explicitly
terminate all subscriptions.
Change gdp_event_next to take an int64_t timeout (in nanoseconds);
also, convert it to use the new ep_thr_cond_wait "absolute time"
Clean up GCL creation: better diagnostics and feedback.
Allow for multiple log daemons; in particular, GCL creation is
now a separate program which requires the name of a log server.
Begin change to new on-wire protocol. Involves lots of renaming.
Amazingly it still seems to run.
Change gdp_gcl_name_is_zero => gdp_gcl_name_is_valid (and invert the
sense) so that we can easily switch to a non-zero value to represent the
"no GCL name" case if necessary in the future.
Incorporate changes from eric-multidaemon branch that might be useful
someday: bug fixes, clarifications, extensions (e.g., supporting IPv6
literal addresses), and so forth.
Change gclh => gcl throughout for consistency
Merge metadata work into callback work so that API changes will happen
at the same time.
Merge commit '57efec14997ab65a1bba56b8865a83acc8c041d7'
Change ep_thr_cond_wait and gdp_event_next to take timeout parameters
(NULL = forever). THESE CHANGE THE API!!
Implement subscription callbacks. Note that these almost never do what
you actually want, so the event interface is preferred.
New thread pool implementation as prep for subscription callbacks.
Add key reading, creation, and writing to writer-test. This is another
Add symbolic GCL names to metadata on create.
Implement metadata reading.
More work on metadata: allow specification of metadata when a GCL is
First step at implementing metadata. Internals are there, but there's
no way to use it yet.
Finish making gdp_datum_t opaque. Unfortunately rw-bench now needs to
see inside, but that should be fixable in the future.
Change gdp_init to take a parameter with the host:port info for
contacting the GDP daemon. Set most apps to take a -G flag to set this.
Separate out subscribe and multiread (read multiple values in one call)
First cut at returning multiple results. This extends "subscribe" to
have a starting record number and a record count.
Eliminate datum->dlen; it's redundant now.
Hide data structure internals by wrapping them in functions. This also
updates the documentation, but it hasn't been significantly reviewed
Portability fixes for Linux and FreeBSD.
Allow users to use non-base64 (i.e., human-grokable) names on the
command line; these are hashed using sha256 to an internal name. Still
need to include that human-readable name as metadata so you have some
chance of knowing what you have available.