| Branch: | Tag: | Revision:

gdp / gdp / gdp.h @ master

History | View | Annotate | Download (22.8 KB)

# Date Author Comment
c3369218 07/06/2020 06:42 PM Eric Allman

2.3.0: add -b flag to gdp-reader to allow binary output.

49d76189 01/14/2020 01:06 PM Eric Allman

Version 2.2.3: Make failure to read a secret key for a writable log an
error instead of waiting for the first append to fail (this can be
turned off using gdp_open_info_set_no_skey_nonfatal, which causes
gdp_gin_open to return a warning instead of an error). Add a -S flag to...

9f1349ae 06/04/2019 04:40 PM Eric Allman

Add GDP_INIT_OPT_HONGDS and swarm.gdp.hongd.optional to not give an
error if HONGD is unavailable (i.e., the default has been changed to be
more restrictive); do stricter checking during initialization (and give
better error messages); bump version to 2.1.19.

c6ccd9ef 01/29/2019 01:42 PM Eric Allman

Do primitive caching of default creation service name.

866c2ca2 01/23/2019 01:17 PM Eric Allman

Pass flags to GDP library initialization routines. Add flags to disable
Zeroconf and suppress initialization of HONGDS, which is unused in some

c87dd166 01/23/2019 07:56 AM Eric Allman

Update license date on all files. No functional changes.

32dc12ee 01/22/2019 05:06 PM Eric Allman

Fix a problem with locating secret keys for logs when using a search
path. Do some prep for making this more flexible. It turns out the
semantics of -K in some applications has changed, but the current
behavior still matches the documentation.

There may be security implications with allowing the ability to name...

c39a16b1 12/11/2018 11:59 AM Eric Allman

Cleanup on human-oriented name handling during creation, including
documentation. Big changes: don't allow a "human name" that looks like
a base-64-encoded name; use GDP_NAME_ROOT when creating a new log using
gdp-create; treat a zero-length GDP_NAME_ROOT as though the variable...

80da996e 10/24/2018 12:53 PM Eric Allman

Mega-change: new object creation API with updated metadata; internal
name is hash of metadata, not hash of human name; query and maintain
human-oriented name to GDPname directory (HONGD).

c9f4745b 07/27/2018 10:52 AM Eric Allman

Begin implementation of signature checking by readers.

59c6c7e0 07/26/2018 10:50 AM Eric Allman

First cut at external => internal name mapping using MySQL. It is
currently not an error if the mapping doesn't exist --- it just falls
back to the old SHA256 behavior --- and there is no automated addition
of the mapping to the database when you create a log. To make this work...

1a1a88c7 06/25/2018 01:55 PM Eric Allman

Avoid problems with the name of gdplogd during upgrades.

ada5f328 06/20/2018 07:15 AM Eric Allman

Improvements to event and metadata printing.

ee59736e 06/19/2018 03:43 PM Eric Allman

Pull old back-compat defines out of gdp.h. They are now in

2da682ef 06/18/2018 02:51 PM Eric Allman

Get signatures on APPEND working again. Still incomplete, but handles
the single datum in a request case.

9f6be413 06/15/2018 03:26 PM Eric Allman

Cleanup signature and hash type handling. There's no point in storing
the MD algorithm with each datum since it is fixed per GOB, nor in
passing it over the wire since it is just a blob unless you have either
the secret or public key, either of which contain that information...

4217d271 06/14/2018 05:28 PM Eric Allman

Better take on gdp_gin_read_by_hash_async for multiple records.

c5884671 06/14/2018 05:27 PM Eric Allman

Comment typo fixes.

5ced6d34 06/09/2018 01:37 PM Eric Allman

Change GDP_EVENT_EOS to GDP_EVENT_DONE, since it isn't specific to
subscriptions. Also documentation updates reflecting the datum changes
in the previous commit.

8cfe4117 05/25/2018 01:22 PM Eric Allman

It turns out that including a GOB pointer in the datum causes too many
locking problems. Just pass the gob in as needed.

eae1d3ec 05/25/2018 01:22 PM Eric Allman

The grand renaming: GCL is (mostly) gone in favor of GOB and GIN.

2e65953f 05/25/2018 01:22 PM Eric Allman

Simplify the datum interface back to what it was before. Basic
subscriptions and subscription timeout seems to work now.

fa7cfdb9 05/25/2018 01:22 PM Eric Allman

Snapshot: compiles cleanly with updated protocol and crypto
support, but completely untested.

92243bf4 05/25/2018 01:22 PM Eric Allman

Snapshot in on-disk format conversion from ad hoc files to SQLite
databases. Compiles but is completely untested.

05c663a2 03/01/2018 02:00 PM Eric Allman

Add UUID to new log metadata to guarantee unique names once we switch to
the new naming scheme (name == HASH(metadata)).

10125206 02/27/2018 06:02 PM Eric Allman

First step at adding QoS information to subscriptions.

4a89fdda 01/17/2018 11:34 AM Eric Allman

First cut at flattening out protobuf definition. Some changes to data
structure printing (e.g., _gdp_pdu_dump) to get indenting more rational
so as to make reading the output easier.

d65c01e2 01/09/2018 11:18 AM Eric Allman

First cut at running Layer 5 network protocol version 4 over old
(version 3) Layer 4 stack. Substantial parts are unimplemented,
including signature generation and checking, read or subscribe by
timestamp or hash, and others. No asynchronous operations including...

5a73c6ad 11/08/2017 12:36 PM Eric Allman

First cut at new net protocol. Compiles, but not yet tested (not even
once --- do not try to run this!).

4a6db795 10/27/2017 07:35 PM Eric Allman

First cut at "delete log" command (lightly tested).

fec93aac 10/24/2017 02:38 PM Eric Allman

First cut at separating GCL instances (GINs) from GDP Objects (GOBs).
This is intended to fix some semantic problems with subscriptions. End
users always use GINs (aliased to gdp_gcl_t for now for compatibility),
and there is one GIN for each call to gdp_gcl_open (i.e., no returning...

dc01a9c8 09/15/2017 10:43 AM Eric Allman

Minor tweaks for _GDP_MODE_PEEK (it's a bit mask, not an absolute

1ca7751e 09/01/2017 03:45 PM Eric Allman

More work on making helgrind happy. This seems to be pretty close,
maybe worth a try in production.

9c515e64 07/26/2017 06:21 PM Eric Allman

Initial implementation of unsubscribe.

a9b676b1 05/20/2017 05:08 AM Eric Allman

Define log metadata field for expiration time.

284a8bac 05/11/2017 01:50 PM Eric Allman

Add gdp_gcl_open_info_set_signkey_cb to set a callback to get the secret
key. This is to avoid asking for a decryption password if it isn't
going to be used (e.g., if you are opening a GCL that is already in the
cache). This is particularly important for gdp-rest.

ee6b5440 05/10/2017 01:17 PM Eric Allman

Attempt to allow GCLs to be cached in applications. This is
specifically for gdp-rest, but there may be other cases where it is

6f52b2b7 04/14/2017 03:16 PM Eric Allman

Implement a way to deliver "record missing" back to client.

94d9f479 02/17/2017 03:10 PM Eric Allman

Add new internal "I/O mode" _GDP_MODE_PEEK to signal _gdp_gcl_cache_get
to not update the use time; this is used for statistics gathering.

bb4511c7 02/16/2017 03:13 PM Eric Allman

Add some new log metadata names. Still needs proper documentation.

897accec 02/06/2017 10:41 AM Eric Allman

Update licenses to 2017.

7bc0e2cf 12/11/2016 08:51 AM Eric Allman

Snapshot: get basic reading working again with new lock changes.

39fae5ac 12/10/2016 01:13 PM Eric Allman

Add gdp_datum_copy routine to copy datum contents.

56f047d8 10/08/2016 11:59 AM Eric Allman

Yet more debug tweaking.

8204d4b2 09/28/2016 10:45 AM Eric Allman

Implement gdp_gcl_read_async. This also fixes some obscure problems
with other asynchronous operations, and tweaks debugging flags to make
it easier to pick out the operations you want to see.

0d8c13e8 08/27/2016 08:36 AM Eric Allman

Make many parameters "const" where possible.

d2647d1f 07/26/2016 07:34 PM Eric Allman

Next take on read-by-timestamp. This version works for read, multiread,
and subscribe. There are still some glitches (e.g., you can't have a
starting timestamp in the future) but fixing these would require major
code restructuring, and besides, you can't specify a starting record...

65309098 07/26/2016 08:36 AM Eric Allman

First cut at read-by-timestamp. Adds Berkeley DB as a dependency (two
versions supported). The interface still isn't quite right; in
particular, subscribe and multiread don't take timestamps. This should
be remedied before this code is released.

0fbf9115 07/13/2016 12:55 PM Eric Allman

Rename "extents" to be "segments" because of different potential
semantics and enable them by default.

8cc28925 05/26/2016 05:10 PM Eric Allman

First cut on implementing FWD_APPEND command. Still not working for all

fa419c23 04/12/2016 12:06 PM Eric Allman

Changes to make it possible to include gdp.h and gdp_priv.h in C++
programs. Ugh.

f9b45862 01/28/2016 01:08 PM Eric Allman

Implement ability to create new extents. Currently there is no attempt
made to include authorization.

87a0bdac 12/17/2015 04:01 PM Eric Allman

Add -q flag to gdp-reader to suppress all metadata output; this allows
it to be piped into another program to read the data.

6bd5476b 12/01/2015 01:22 PM Eric Allman

Update license. No code changes.

055d3009 11/21/2015 01:20 PM Eric Allman

Include copyright in all files. There are scripts to do bulk
replacement of the copyright if desired.

02de7463 10/27/2015 04:53 PM Eric Allman

Implement subscription renewals. This causes clients who have open
subscriptions to re-issue the subscription every
swarm.gdp.subscr.pokeintvl seconds. It's not complete yet: gdplogd
still needs to expire subscriptions after a timeout.

7d83fa37 10/05/2015 06:38 PM Eric Allman

Add new gdp_gcl_open_info interface to allow tweaking of GCL parameters
on open. The only thing included now is the setting of the signing key.

8ce5b4db 10/05/2015 11:50 AM Eric Allman

Use the third parameter to gdp_gcl_open for general open information,
not just QoS.

c8e97446 10/01/2015 11:58 AM Eric Allman

Change on-disk log to store signature.
Return that signature on READ commands.
Update reader-test to add a -v flag to print signature info.
Document new interfaces.

34612550 09/24/2015 01:53 PM Eric Allman

Implement read and append filtering so people can add hooks to do (for
example) encryption and decryption. Adds two new APIs:

EP_STAT (*filter)(gdp_datum_t *datum, void *closure)
gdp_gcl_set_read_filter(gcl, filter, closure)
gdp_gcl_set_append_filter(gcl, filter, closure)
9c59e638 09/18/2015 10:54 AM Eric Allman

Add log-mirror command for trivial replication. Required also adding
gdp_gcl_getnrecs to API.

f43e276d 08/20/2015 03:58 PM Eric Allman

Allow logs to opened for both read and append (GDP_MODE_RA).

d64e4288 08/06/2015 03:57 PM Eric Allman

Check GCL mode when doing I/O (to avoid writing to a read-only log or
reading an append-only log).

b8591e8b 08/06/2015 01:59 PM Eric Allman

Fix some initialization bugs that caused mutex failures.

034d97fd 07/30/2015 12:06 PM Eric Allman

Add creation time to logs. This also changes the ep_time_format and
ep_time_print APIs.

9a47c6c7 07/24/2015 09:38 AM Eric Allman

Another signing snapshot: signatures are created and sent but not
checked. Metadata is returned when a GCL is opened. Still have a
problem with reference counting of GCL handles.

f80ae9c0 07/22/2015 09:54 AM Eric Allman

Snapshot adding signing. Signatures are computed but not sent or

e2682ba9 05/28/2015 10:36 AM Eric Allman

Implement asynchronous writes. Adds new interfaces:
* gdp_gcl_append_async -- initiate an asynchronous write.
* gdp_event_getstat -- get the status code from an event.
Asynchronous reads can be done using multiread.
Note that asynchronous writes do not attempt retransmissions, unlike...

4102ad3a 03/05/2015 05:59 PM Eric Allman

Cleanup for code walkthrough:
* comments
* some function rearrangement
* make data structure detail printing more consistent

b42c4583 02/13/2015 02:48 PM Eric Allman

Add -t option to reader-test to print plain text (instead of a hexdump).

Also changes gdp_datum_print to take a flags argument.

153ffd95 02/10/2015 05:40 PM Eric Allman

Clean up gdp_datum_print (it's not just for debugging any more!).

55b8c1ad 02/02/2015 08:17 AM Eric Allman

Change gdp_event_next to allow selection of events for a specific GCL.

be7cbe23 01/31/2015 05:11 PM Eric Allman

Run subscription callbacks in dedicated thread. Untested.

6fcf54ad 01/29/2015 04:56 PM Eric Allman

Add placeholder qos requirements parameter on open.
Document gdp_gcl_getstat as taking a name, not a handle.

05e1cb19 01/29/2015 04:40 PM Eric Allman

Rename PUBLISH => APPEND (API change).
Eliminate gdp_datum_setrecno and gdp_datum_setts (didn't make sense).
Fix example code in protocol documentation.

cf71c5ea 01/29/2015 11:39 AM Eric Allman

Implement gdplogd shutdown actions: withdraw advertisements, explicitly
terminate all subscriptions.

d3cbb6dc 01/22/2015 01:16 PM Eric Allman

Change gdp_event_next to take an int64_t timeout (in nanoseconds);
also, convert it to use the new ep_thr_cond_wait "absolute time"

3e54310e 01/10/2015 04:23 PM Eric Allman

Clean up GCL creation: better diagnostics and feedback.

c6293003 01/10/2015 03:28 PM Eric Allman

Allow for multiple log daemons; in particular, GCL creation is
now a separate program which requires the name of a log server.

549b367d 12/26/2014 07:55 AM Eric Allman

Begin change to new on-wire protocol. Involves lots of renaming.
Amazingly it still seems to run.

3b5938ca 12/18/2014 10:07 AM Eric Allman

Change gdp_gcl_name_is_zero => gdp_gcl_name_is_valid (and invert the
sense) so that we can easily switch to a non-zero value to represent the
"no GCL name" case if necessary in the future.

a7ab7f37 12/05/2014 10:08 AM Eric Allman

Incorporate changes from eric-multidaemon branch that might be useful
someday: bug fixes, clarifications, extensions (e.g., supporting IPv6
literal addresses), and so forth.

e70f42ee 12/04/2014 11:36 AM Eric Allman

Change gclh => gcl throughout for consistency

d0a68561 11/25/2014 02:31 PM Eric Allman

Merge metadata work into callback work so that API changes will happen
at the same time.
Merge commit '57efec14997ab65a1bba56b8865a83acc8c041d7'

686e4579 11/25/2014 01:30 PM Eric Allman

Change ep_thr_cond_wait and gdp_event_next to take timeout parameters

fe034bdb 11/25/2014 01:04 PM Eric Allman

Implement subscription callbacks. Note that these almost never do what
you actually want, so the event interface is preferred.

50d98aaa 11/25/2014 08:33 AM Eric Allman

New thread pool implementation as prep for subscription callbacks.

e2d81e0f 11/12/2014 11:42 AM Eric Allman

Add key reading, creation, and writing to writer-test. This is another
intermediate step.

5bd0848f 11/11/2014 12:14 PM Eric Allman

Add symbolic GCL names to metadata on create.

158a49ba 11/06/2014 06:24 PM Eric Allman

Implement metadata reading.

3ddd2ed9 11/06/2014 10:37 AM Eric Allman

More work on metadata: allow specification of metadata when a GCL is

5c6e2b40 11/05/2014 11:11 AM Eric Allman

First step at implementing metadata. Internals are there, but there's
no way to use it yet.

e5d36455 09/23/2014 07:35 AM Eric Allman

Finish making gdp_datum_t opaque. Unfortunately rw-bench now needs to
see inside, but that should be fixable in the future.

5cd53c1a 09/16/2014 04:06 PM Eric Allman

Change gdp_init to take a parameter with the host:port info for
contacting the GDP daemon. Set most apps to take a -G flag to set this.

328d0948 09/12/2014 06:51 PM Eric Allman

Separate out subscribe and multiread (read multiple values in one call)

2e4a5d7d 09/12/2014 12:52 PM Eric Allman

First cut at returning multiple results. This extends "subscribe" to
have a starting record number and a record count.

7fc131d1 09/09/2014 11:33 AM Eric Allman

Eliminate datum->dlen; it's redundant now.

22442bd4 09/08/2014 03:37 PM Eric Allman

Hide data structure internals by wrapping them in functions. This also
updates the documentation, but it hasn't been significantly reviewed

cd4cc511 08/30/2014 03:52 PM Eric Allman

Portability fixes for Linux and FreeBSD.

ecbc8f12 08/27/2014 10:54 AM Eric Allman

Allow users to use non-base64 (i.e., human-grokable) names on the
command line; these are hashed using sha256 to an internal name. Still
need to include that human-readable name as metadata so you have some
chance of knowing what you have available.