Project

General

Profile

gdp-programmatic-api.html

Historic - Eric Allman, 03/28/2018 12:45 PM

Download (62.7 KB)

 
1
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
2
<html>
3
  <head>
4
    <meta http-equiv="content-type" content="text/html; charset=windows-1252">
5
    <title>Global Data Plane Programmatic API</title>
6
    <style type="text/css">
7
.warning {
8
  font-weight: bold;
9
  font-style: italic;
10
  color: red;
11
  background-color: #ffff99;
12
}
13
.command {
14
  font-family: monospace;
15
}
16
.variable {
17
  font-style: oblique;
18
}
19
.filename {
20
  font-family: monospace;
21
}
22
.admin-param {
23
  font-family: monospace;
24
}
25
.function {
26
  font-family: monospace;
27
}
28
.manifest {
29
  font-family: monospace;
30
}
31
.datatype {
32
  font-family: monospace;
33
}
34

    
35
</style></head>
36
  <body>
37
    <h1>Global Data Plane Programmatic API </h1>
38
    <h4>Editor: Eric Allman, U.C. Berkeley Swarm Lab, eric@cs.berkeley.edu<br>
39
      Version 0.8.0, 2017-07-27</h4>
40
    <p>This document describes the procedural programmatic interface to the
41
      Global Data Plane.&nbsp; The native code is written in C for maximum
42
      flexibility and performance, but it is expected that most applications
43
      will be written in higher level languages, and hence there will be
44
      multiple language bindings for the GDP library.&nbsp; There is also a REST
45
      interface that is not described in this document. </p>
46
    <p>The GDP library uses the EP portability library, and applications are
47
      free to use that library as well; in particular, since the GDP libraries
48
      makes extensive use of the EP library some efficiencies may result from
49
      using them both.&nbsp; However, this document does not attempt to define
50
      the EP library and describes it only as necessary.&nbsp; However, one EP
51
      concept that appears commonly is the <code>EP_STAT</code> data type,
52
      which represents a status code that can be returned by a function to
53
      indicate completion status that includes a "severity" (e.g. OK, ERROR,
54
      SEVERE), a "registry" (in our case always UC Berkeley), a "module" (e.g.,
55
      GDP or the EP library itself), and detail information.&nbsp; An OK status
56
      can return a positive integer as extra information. </p>
57
    <p>The code distribution includes an "apps" directory with two example
58
      programs: gdp-writer.c and gdp-reader.c, that show simple cases of how to
59
      append to a GCL and read from a GCL (including subscriptions). </p>
60
    <h2>1&nbsp; Terminology</h2>
61
    <dl>
62
      <dt>Datum</dt>
63
      <dd>A unit of data in the GDP; essentially a record.&nbsp; Each datum has
64
        associated with it a record number (numbered sequentially from one), a
65
        commit timestamp (the time the record was committed into the GDP, as
66
        distinct from the time that the data originated), an associated blob
67
        containing the data itself, which we expect to be encrypted, and in most
68
        cases a signature. </dd>
69
      <dt>GDP Channel/Log</dt>
70
      <dd>Global Data Plane Channel/Log.&nbsp; This represents an addressable
71
        entity in the Global Data Plane, which may be a log or a service.</dd>
72
    </dl>
73
    <h2> 2&nbsp; Theory of Operation, Data Types, and Initialization </h2>
74
    <p> GDP-based applications rely on three pieces: an in-process GDP library,
75
      a GDP Log Daemon, and the Routing Layer.&nbsp;&nbsp; This document
76
      describes the GDP library API. </p>
77
    <p> The primary point of the GDP library is to speak the network protocol
78
      between the application and the GDP Daemon. The library is threaded, with
79
      (at the moment) two threads: one to process events (data arriving from the
80
      daemon, although others can be added), and the other to run the
81
      application itself. This allows the application to pretend it is a
82
      sequential program while still allowing asynchronous input from the GDP
83
      Daemon (e.g., processing results from subscription requests).&nbsp;
84
      Applications are free to create other threads as desired.&nbsp; The code
85
      has been written to do the locking as necessary, but stress tests have not
86
      been run, so you may find unhappy results. </p>
87
    <p> The primary abstraction is the GDP Channel-Log (GCL). A GCL represents
88
      the rendezvous point for communication in the data plane. It is not
89
      directly tied to either a file or a network connection. On creation, a GCL
90
      is assigned a 256-bit opaque name. A GCL is append-only to writers.&nbsp;
91
      For the moment you can access the dataplane in one of two modes:
92
      synchronous mode (using <code>gdp_gcl_read</code> for reading) an
93
      asynchronous mode (using <code>gdp_gcl_subscribe</code> for reading). To
94
      use it in asynchronous mode you must subscribe to any GCLs of interest and
95
      then call <code>gdp_event_next</code> repeatedly to read the
96
      results.&nbsp; These are described in more detail below. </p>
97
    <p>All GCLs are named with an opaque, location independent, 256-bit number
98
      from a flat namespace.&nbsp; When printed it is shown as a base64-encoded
99
      value that is tweaked to be valid in a URI (that is, "+" and "/" are
100
      replaced with "&ndash;" and "_").&nbsp; Applications may choose to overlay
101
      these unsightly names with some sort of directory service. </p>
102
    <p>Applications using the GDP library should <code>#include
103
        &lt;gdp/gdp.h&gt;</code> for all the essential definitions.</p>
104
    <hr>
105
    <h4>Name</h4>
106
    <p>GDP data types and basic utilities</p>
107
    <h4>Synopsis</h4>
108
    <pre>#include &lt;gdp/gdp.h&gt;<br><br>gdp_name_t        InternalGdpName;        // 256-bit number<br>gdp_pname_t        PrintableGdpName;        // base-64 encoded string<br>bool                GDP_NAME_SAME(gdp_name_t a, gdp_name_t b);<br>bool                gdp_name_is_valid(gdp_name_t gname);<br>char                *gdp_printable_name(const gdp_name_t Internal, gdp_pname_t Printable);<br>EP_STAT                gdp_internal_name(const gdp_pname_t Printable, gdp_name_t Internal);<br>EP_STAT                gdp_parse_name(const char *external, gdp_name_t Internal);<br><br>gdp_gcl_t        *GdpChannelLog;<br><br>gdp_datum_t        *GdpDatum;<br>gdp_recno_t        RecordNumber;<br>EP_TIME_SPEC        TimeStampSpec;</pre>
109
    <h4>Notes</h4>
110
    <ul>
111
      <li>Most of these are described in more detail below.</li>
112
    </ul>
113
    <h3> </h3>
114
    <hr width="100%" size="2"><br>
115
    <h4> Name</h4>
116
    gdp_init &mdash; Initialize the GDP library
117
    <h4> Synopsis</h4>
118
    <pre>#include &lt;gdp/gdp.h&gt;<br><br>EP_STAT gdp_init(const char *gdpd_addr)
119
</pre>
120
    <h4> Notes</h4>
121
    <ul>
122
      <li> Initializes the GDP library.&nbsp; <em><strong>Must</strong></em> be
123
        called before any other GDP functions are invoked.</li>
124
      <li>The <code>gdpd_addr</code> parameter is the address to use to contact
125
        the GDP routing layer in the format "host:port".&nbsp; If <code>NULL</code>
126
        a system default is used. </li>
127
      <li>If the status is not <code>EP_STAT_OK</code> then the library failed
128
        to initialize (for example, by being unable to acquire resources.&nbsp;
129
        Failure to check this status may result in mysterious failures later.</li>
130
      <li>All source files using the GDP must include <code>&lt;gdp/gdp.h&gt;</code>.
131
      </li>
132
    </ul>
133
    <hr width="100%" size="2">
134
    <h4>Name</h4>
135
    <p>GDP_LIB_VERSION &mdash; GDP library version</p>
136
    <h4>Synopsis</h4>
137
    <pre>#include &lt;gdp/gdp_version.h&gt;</pre>
138
    <h4>Notes</h4>
139
    <ul>
140
      <li>The <code>gdp_version.h</code> file defines the integer constant <code>GDP_LIB_VERSION</code>
141
        as the major, minor, and patch level of this version of the GDP library,
142
        for example, 0x010203 for version 1.2.3.&nbsp; It can be used during
143
        compilation.&nbsp; There is also a string <code>GdpVersion</code> that
144
        is suitable for printing.</li>
145
    </ul>
146
    <hr>
147
    <h2>3&nbsp; GCL Operations</h2>
148
    <h3>3.1&nbsp; GCL Synchronous Operations</h3>
149
    <p>Synchronous operations block until the operation is complete.&nbsp; They
150
      are the easiest interface for simple programs. </p>
151
    <hr width="100%" size="2">
152
    <h4> Name</h4>
153
    gdp_gcl_create &mdash; Create an append-only GCL on a specified log daemon
154
    node &nbsp; <span class="warning">[TEMPORARY INTERFACE]</span>
155
    <div> </div>
156
    <h4> Synopsis</h4>
157
    <pre>EP_STAT gdp_gcl_create(gdp_name_t gcl_name,
158
                gdp_name_t logdname,<br>                gdp_gclmd_t *gmd,<br>                gdp_gcl_t **gclp)
159
</pre>
160
    <h4> Notes</h4>
161
    <ul>
162
      <li> Creates a GCL with the given name on the indicated log server, both
163
        of which must be 256-bit values.</li>
164
      <li>Metadata can be passed in that will be stored with the GCL.</li>
165
      <li> Returns a GCL handle (indirectly through <code>*gclp</code>).</li>
166
      <li> The returned GCL handle is an append-only object.&nbsp; <i>(Actually,
167

    
168

    
169

    
170

    
171

    
172

    
173

    
174
          this is not enforced at this time).</i></li>
175
      <li><span class="warning">This interface <strong><em>will</em></strong>
176
          be replaced or updated in the future.</span> </li>
177
      <li> We probably need payment information, some way of specifying resource
178
        requirements, and so forth.</li>
179
      <li> Palmer did not describe anything that explicitly creates a GCL. I
180
        think explicit creation is important (because of Quality of Service),
181
        but the semantics could be automatic creation on first reference with
182
        default QoS.</li>
183
      <li><span class="warning">Ultimately this will be a call to a service that
184
          will acquire the resources, determine physical location, etc., and
185
          then tell the appropriate log daemon(s) to do the physical creation.</span></li>
186
    </ul>
187
    <br>
188
    <hr width="100%" size="2">
189
    <h4> Name</h4>
190
    gdp_gcl_open &mdash; Open an existing GCL
191
    <h4> Synopsis</h4>
192
    <pre>EP_STAT gdp_gcl_open(gdp_name_t name,
193
                gcl_iomode_t rw,
194
                gdp_gcl_open_info_t *info,
195
                gdp_gcl_t **gclp)
196
</pre>
197
    <h4> Notes</h4>
198
    <ul>
199
      <li> Opens the GCL with the indicated name for the mode indicated by <code>rw</code>,
200
        which may be <code>GDP_MODE_RO</code> (read only), <code>GDP_MODE_AO</code>
201
        (append only), or <code>GDP_MODE_RA</code> (read and append).</li>
202
      <li>Open information is to pass in detailed information needed for the
203
        open as described below.&nbsp; In most cases it can be passed as
204
        NULL.&nbsp; Eventually it will be used to convey signing keys, quality
205
        of service requests, payment information, authorization tokens, etc.</li>
206
      <li>The handle itself is returned indirectly through <code>*gclp</code>.</li>
207
      <li>If a signing key associated with the GCL is found, all writes to this
208
        GCL will automatically be signed.&nbsp; See the section on Signing and
209
        Encryption below for details.</li>
210
    </ul>
211
    <hr>
212
    <h4>Name</h4>
213
    gdp_gcl_open_info_new &mdash; Create a new open information data structure<br>
214
    <h4>Synopsis</h4>
215
    <pre>gdp_gcl_open_info_t *gdp_gcl_open_info_new(void)</pre>
216
    <h4>Notes</h4>
217
    <ul>
218
      <li>Creates a new data structure to be used in the following routines.</li>
219
    </ul>
220
    <hr><br>
221
    <h4>Name</h4>
222
    gdp_gcl_open_info_free &mdash; Free the open information structure<br>
223
    <h4>Synopsis</h4>
224
    <pre>void gdp_gcl_open_info_free(gdp_gcl_open_info_t *info)</pre>
225
    <h4>Notes</h4>
226
    <ul>
227
      <li>Frees the data structure pointed to by <code>info</code>.</li>
228
      <li>This can be freed as soon as it has been used in <code>gdp_gcl_open</code>.</li>
229
    </ul>
230
    <hr>
231
    <h4>Name</h4>
232
    gdp_gcl_open_info_set_signing_key &mdash; Set signing key for an open GCL<br>
233
    <h4>Synopsis</h4>
234
    <pre>EP_STAT gdp_gcl_open_info_set_signing_key(<br>&nbsp;&nbsp;&nbsp;             gdp_gcl_open_info_t *info,<br>&nbsp;&nbsp;&nbsp;             EP_CRYPTO_KEY *skey)</pre>
235
    <h4>Notes</h4>
236
    <ul>
237
      <li>Sets the signing key to be used when appending data to a log.</li>
238
    </ul>
239
    <hr>
240
    <h4>Name</h4>
241
    <p>gdp_gcl_open_info_set_signkey_cb &mdash; Set a callback function to read
242
      a signing key</p>
243
    <h4>Synopsis</h4>
244
    <pre>EP_STAT gdp_gcl_open_info_set_signkey_cb(<br>                gdp_gcl_open_info_t *info,<br>                EP_STAT (*signkey_cb)(<br>                        gdp_name_t gname,<br>                        void *signkey_udata,<br>                        EP_CRYPTO_KEY **skey),<br>                void *signkey_udata)</pre>
245
    <h4>Notes</h4>
246
    <ul>
247
      <li>If the <code>gdp_gcl_open</code> call requires a secret key, that
248
        that key was not passed in using <code>gdp_gcl_open_info_set_signing_key</code>,
249
        the callback function <code>signkey_cb</code> is invoked to get a
250
        key.&nbsp; It will only be invoked if the key is required (notably
251
        because it isn't already cached).</li>
252
      <li>The arbitrary data pointer <code>signkey_udata</code> is passed
253
        through to <code>signkey_cb</code> if it is invoked.</li>
254
    </ul>
255
    <hr>
256
    <h4>Name</h4>
257
    gdp_gcl_open_info_set_caching &mdash; set caching behavior<br>
258
    <h4>Synopsis</h4>
259
    <pre>EP_STAT gdp_gcl_open_info_set_caching(<br>&nbsp;&nbsp;            &nbsp; gdp_gcl_open_info_t *info,<br>&nbsp;&nbsp;            &nbsp; bool keep_in_cache)</pre>
260
    <h4>Notes</h4>
261
    <ul>
262
      <li>Sets whether this GCL should be kept in the cache after <code>gdp_gcl_close</code>
263
        is called.</li>
264
      <li>If set, cached information will be reclaimed based on the last usage
265
        time of the GCL.</li>
266
      <li>Use of this call with <code>keep_in_cache</code> set to <code>TRUE</code>
267
        may cause a cleanup thread to be spawned.</li>
268
    </ul>
269
    <hr width="100%" size="2">
270
    <h4> Name</h4>
271
    gdp_gcl_close &mdash; Close a GCL and release resources
272
    <h4> Synopsis</h4>
273
    <pre>EP_STAT gdp_gcl_close(gdp_gcl_t *gcl)
274
</pre>
275
    <h4> Notes</h4>
276
    <ul>
277
      <li>Sends a hint to the log daemon that the associated resources are no
278
        longer being used.</li>
279
      <li>Releases the client-side resources (that is, memory) for this GCL
280
        handle.</li>
281
      <li><i>Should this interface say whether to preserve or drop the GCL
282
          (i.e., is it persistent, or for how long)?</i></li>
283
    </ul>
284
    <br>
285
    <hr width="100%" size="2">
286
    <h4> Name</h4>
287
    gdp_gcl_getname &mdash; Return the name of a GCL
288
    <h4> Synopsis</h4>
289
    <pre>EP_STAT gdp_gcl_getname(gdp_gcl_t *gcl,
290
                gdp_name_t namebuf)
291
</pre>
292
    <h4> Notes</h4>
293
    <ul>
294
      <li> Returns the name of the GCL referenced by <code>gcl</code> into <code>
295
          namebuf</code>. </li>
296
      <li> May not be necessary if the creator provides the name; really only
297
        intended after <code>gdp_gcl_create</code> so that the name can be
298
        shared to other nodes that want to <code>gdp_gcl_open</code> it.</li>
299
    </ul>
300
    <br>
301
    <hr width="100%" size="2">
302
    <h4> Name</h4>
303
    gdp_gcl_getstat &mdash; Return information about a GCL&nbsp; <span class="warning">[NOT
304
      YET IMPLEMENTED] </span>
305
    <h4> Synopsis</h4>
306
    <pre>EP_STAT gdp_gcl_getstat(gdp_name_t gclname,
307
                gcl_stat_t *statbuf)
308
</pre>
309
    <h4> Notes</h4>
310
    <ul>
311
      <li> <i>What status is included? Size (or number of records/messages),
312
          last access, &hellip;</i></li>
313
    </ul>
314
    <br>
315
    <hr width="100%" size="2">
316
    <h4> Name</h4>
317
    gdp_gcl_print &mdash; print a GCL handle (for debugging)
318
    <h4>Synopsis</h4>
319
    <pre>void gdp_gcl_print(const gdp_gcl_t *gclh, FILE *fp)</pre>
320
    <h4>Notes</h4>
321
    <ul>
322
      <li>Prints the GCL on the indicated file.</li>
323
      <li>The output will contain internal information; it is not intended to
324
        display information to end users. </li>
325
    </ul>
326
    <hr width="100%" size="2">
327
    <h4> Name</h4>
328
    gdp_gcl_append &mdash; Append a record to a writable GCL
329
    <h4> Synopsis</h4>
330
    <pre>EP_STAT gdp_gcl_append(gdp_gcl_t *gcl,
331
                gdp_datum_t *datum)
332
</pre>
333
    <h4> Notes</h4>
334
    <ul>
335
      <li> Appends the indicated datum to the GCL.</li>
336
      <li>Any subscribers get immediate updates about the new datum.</li>
337
      <li>If a secret key is available for this GCL, the appends will be signed.</li>
338
    </ul>
339
    <br>
340
    <hr width="100%" size="2">
341
    <h4> Name</h4>
342
    gdp_gcl_read_by_recno, gdp_gcl_read_by_ts &mdash; Read from a readable GCL
343
    <h4> Synopsis</h4>
344
    <pre>EP_STAT gdp_gcl_read_by_recno(gdp_gcl_t *gcl,
345
                gdp_recno_t recno,
346
                gdp_datum_t *datum)<br>EP_STAT gdp_gcl_read_by_ts(gdp_gcl_t *gcl,
347
                EP_TIME_SPEC *ts,
348
                gdp_datum_t *datum)<br> </pre>
349
    <h4> Notes</h4>
350
    <ul>
351
      <li> <code>gdp_gcl_read_by_recno</code> reads the specified record number
352
        and returns it in the user-supplied datum (see below).</li>
353
      <li><code>gdp_gcl_read_by_ts</code> reads the record dated on or
354
        immediately after the indicated timestamp.</li>
355
      <li>Presents a message-oriented interface.</li>
356
      <li> An OK stat includes the number of octets actually read. Probably
357
        passed back in datum, so unneeded here.</li>
358
      <li> The offset indicates a message number (sequential starting from one).
359
        The value &ndash;1 indicates any new messages.</li>
360
    </ul>
361
    <hr width="100%" size="2">
362
    <h4>Name</h4>
363
    gdp_parse_name &mdash; parse an external representation to internal
364
    <h4>Synopsis</h4>
365
    <pre>EP_STAT gdp_parse_name(const char *ext, gdp_name_t gcl_name)</pre>
366
    <h4>Notes</h4>
367
    <ul>
368
      <li>Converts an external string representation of a GCL name to an
369
        internal 256-bit encoding.</li>
370
      <li>If ext is a URI-base64-encoded string of the correct length, it is
371
        converted directly to 256 bits.</li>
372
      <li>Otherwise, it is hashed using sha256 to create the internal name.</li>
373
      <li><span class="warning">[This functionality will disappear in the
374
          future, after a Directory Service is defined and implemented.&nbsp; In
375
          particular, it will not be possible for users to select the name of a
376
          GCL.]</span> </li>
377
    </ul>
378
    <ul>
379
    </ul>
380
    <hr width="100%" size="2">
381
    <h3>3.2&nbsp; GCL Asynchronous Operations (Asynchronous I/O, Subscriptions,
382
      and Events) </h3>
383
    <p>Asynchronous operations allow an application to subscribe to one or more
384
      GCLs and receive events as those GCLs see activity.&nbsp; The event
385
      mechanism is intended to be extensible for possible future expansion. </p>
386
    <p>Every event has a type, a pointer to the GCL handle, and a pointer to a
387
      datum.&nbsp; Applications could in principle define their own event types,
388
      but at the moment this functionality is not exposed.</p>
389
    <p>All asynchronous operations return status and/or data via either a
390
      callback function or the event interface.&nbsp; Callback functions may not
391
      be called in the same thread as the operation initiation.&nbsp; If no
392
      callback function is given then the event interface is used; this has the
393
      effect of serializing the event stream.&nbsp; In either case, it is the
394
      responsibility of the caller to free the event after use using <code>gdp_event_free</code>.</p>
395
    <p>Note that asynchronous calls do not do retransmissions.</p>
396
    <br>
397
    <hr>
398
    <h4>Name</h4>
399
    <p>gdp_event_t &mdash; event structure</p>
400
    <h4>Synopsis</h4>
401
    <p><code>typedef struct _gdp_event&nbsp;&nbsp;&nbsp; gdp_event_t;</code></p>
402
    <h4>Notes</h4>
403
    <ul>
404
      <li>This is an opaque type.</li>
405
    </ul>
406
    <hr>
407
    <h4>Name</h4>
408
    <p>gdp_event_cbfunc_t &mdash; event callback function type</p>
409
    <h4>Synopsis</h4>
410
    <p><code>typedef void (*gdp_event_cbfunc_t)(gdp_event_t *gev);</code></p>
411
    <h4>Notes</h4>
412
    <ul>
413
      <li>This is the type of callback function as passed into the asynchronous
414
        interfaces.</li>
415
      <li>All the interesting data is encoded into the event.&nbsp; This is
416
        exactly the same data structure as returned by <code>gdp_event_next</code>.</li>
417
      <li>In all cases, if the callback function is not specified the
418
        information will be returned through the event interface.&nbsp; The two
419
        interfaces are related but mutually exclusive.</li>
420
    </ul>
421
    <p><br>
422
    </p>
423
    <ul>
424
    </ul>
425
    <hr>
426
    <h4>Name</h4>
427
    <p>gdp_gcl_read_async &mdash; Asynchronously read data from a GCL</p>
428
    <h4>Synopsis</h4>
429
    <pre>EP_STAT gdp_gcl_read_async(<br>                gdp_gcl_t *gcl,<br>                gdp_recno_t recno,<br>                gdp_event_cbfunc_t *cbfunc,<br>                void *udata)</pre>
430
    <h4><span style="font-family: monospace;"><span style="font-family: serif;">Notes</span></span></h4>
431
    <ul>
432
      <li>Initializes a read of the given record number.</li>
433
      <li>This function returns before any result is read and thus does not
434
        include the final status information.</li>
435
      <li>The return status will be <code>GDP_STAT_OK</code> if the read
436
        command is successfully sent, and a later callback or event will give
437
        the actual status; otherwise no callback or event will occur.</li>
438
      <li>Status is returned through the event interface (if <code>cbfunc</code>
439
        is <code>NULL</code>) or through <code>cbfunc</code>.</li>
440
      <li>Exactly one event will be generated for each successful return.&nbsp;
441
        Those events will be <code>GDP_EVENT_DATA</code> if the read succeeded
442
        or an error event if the read failed.</li>
443
    </ul>
444
    <hr width="100%" size="2">
445
    <h4> Name</h4>
446
    <p>gdp_gcl_append_async &mdash; Asynchronously append to a writable GCL</p>
447
    <h4>Synopsis</h4>
448
    <pre>EP_STAT gdp_gcl_append_async(<br>                gdp_gcl_t *gcl,<br>                gdp_datum_t *datum,<br>                gdp_event_cbfunc_t *cbfunc,<br>                void *udata)</pre>
449
    <h4>Notes</h4>
450
    <ul>
451
      <li>Appends the indicated <code>datum</code> to the GCL.</li>
452
      <li>This function returns before any result is read and thus does not
453
        include the final status information.</li>
454
      <li>The return status will be <code>GDP_STAT_OK</code> if the append
455
        command is successfully sent, and a later callback or event will give
456
        the actual status; otherwise no callback or event will occur.</li>
457
      <li><em>Should a warning status be returned to make it clear that a status
458
          will be returned later?</em></li>
459
      <li>Status is returned through the event interface (if <code>cbfunc</code>
460
        is <code>NULL</code>) or through <code>cbfunc</code> with an event
461
        type of <code>GDP_EVENT_ASTAT</code>.</li>
462
    </ul>
463
    <hr>
464
    <h4>Name</h4>
465
    gdp_gcl_subscribe_by_* &mdash; Subscribe to a readable GCL
466
    <h4> Synopsis</h4>
467
    <pre>EP_STAT gdp_gcl_subscribe_by_recno(<br>                gdp_gcl_t *gcl,
468
                gdp_recno_t start,<br>                int32_t numrecs,<br>                gdp_sub_qos_t *qos;<br>                gdp_event_cbfunc_t *cbfunc,<br>                void *udata)<br>EP_STAT gdp_gcl_subscribe_by_ts(<br>                gdp_gcl_t *gcl,
469
                EP_TIME_SPEC *start,<br>                int32_t numrecs,<br>                gdp_sub_qos_t *qos;<br>                gdp_event_cbfunc_t *cbfunc,<br>                void *udata) </pre>
470
    <h4> Notes</h4>
471
    <ul>
472
      <li>If a <code>cbfunc</code> is specified, arranges to call callback when
473
        a message is generated on the <code>gcl</code>.</li>
474
      <li> The callback is not necessarily invoked instantly, and may or may not
475
        be called in a separate thread.</li>
476
      <li>It is the responsibility of the callback function to call <code>gdp_event_free(gev)</code>.
477
      </li>
478
      <li>If qos is specified, it contains quality of service information about
479
        the subscription &mdash; for example, whether subscriptions should be
480
        reliable or best effort (default).</li>
481
      <li>If no <code>cbfunc</code> is specified, subscription information is
482
        available through the <code>gdp_gcl_event</code> interface (see below).</li>
483
      <li> The <code>udata</code> is passed through untouched in generated
484
        events.&nbsp; See below for the definition of <code>gdp_event_t</code>.</li>
485
      <li>At most <code>numrecs</code> records are returned, after which the
486
        subscription is terminated.&nbsp; If <code>numrecs</code> is 0 it waits
487
        for data forever. </li>
488
      <li> The <code>start</code> parameter tells when to start the
489
        subscription (that is, the starting record number for <code>gdp_gcl_subscribe_by_recno</code>
490
        or the earliest time of interest for <code>gdp_gcl_subscribe_by_ts</code>).</li>
491
      <li>In <code>gdp_gcl_subscribe_by_recno</code>, if <code>start</code> is
492
        negative, returns the most recent &ndash;<code>start</code>
493
        records.&nbsp; If a negative <code>start</code> indicates going back
494
        more records than are available, it starts from the first record. </li>
495
      <li>If <code>start</code> specifies an existing record but there are
496
        fewer than <code>numrecs</code> records available, this returns the
497
        available records and then waits for the additional data to appear as it
498
        is published.</li>
499
      <li>In <code>gdp_gcl_subscribe_by_recno</code>, if <code>start</code> is
500
        zero, or in <code>gdp_gcl_subscribe_by_ts</code>, it points past the
501
        last record already in the log, no current records are returned (i.e.,
502
        it returns new records as they are published).</li>
503
      <li><em>Callbacks make binding to languages like Java particularly
504
          difficult, but fit more naturally in with languages such as
505
          Javascript.&nbsp; Note also that callbacks will generally run in the
506
          GDP I/O thread (so no further GDP operations will run until the
507
          callback completes) or in a separate thread (in which case several
508
          instances of the callback may be run at once). </em></li>
509
      <li>Example:&nbsp; suppose there are 20 records already in a GCL.&nbsp;
510
        Then:</li>
511
    </ul>
512
    <table align="center" width="80%" cellspacing="2" cellpadding="2" border="1">
513
      <tbody>
514
        <tr>
515
          <td valign="top">start </td>
516
          <td valign="top">numrecs </td>
517
          <td valign="top">Behavior </td>
518
        </tr>
519
        <tr>
520
          <td valign="top">1 </td>
521
          <td valign="top">10 </td>
522
          <td valign="top">Returns records 1&ndash;10 immediately and terminates
523
            the subscription. </td>
524
        </tr>
525
        <tr>
526
          <td valign="top">&ndash;10 </td>
527
          <td valign="top">10 </td>
528
          <td valign="top">Returns records 11&ndash;20 immediately and
529
            terminates the subscription. </td>
530
        </tr>
531
        <tr>
532
          <td valign="top">0 </td>
533
          <td valign="top">0 </td>
534
          <td valign="top">Starts returning data when record 21 is published and
535
            continues forever. </td>
536
        </tr>
537
        <tr>
538
          <td valign="top">&ndash;10 </td>
539
          <td valign="top">20 </td>
540
          <td valign="top">Returns records 11&ndash;20 immediately, then returns
541
            records 21&ndash;30 as they are published.&nbsp; The subscription
542
            then terminates. </td>
543
        </tr>
544
        <tr>
545
          <td valign="top">1 </td>
546
          <td valign="top">0</td>
547
          <td valign="top">Returns records 1&ndash;20 immediately, then returns
548
            any new records published in perpetuity. </td>
549
        </tr>
550
        <tr>
551
          <td valign="top">&ndash;30 </td>
552
          <td valign="top">30 </td>
553
          <td valign="top">Returns records 1&ndash;20 immediately, then returns
554
            records 21&ndash;30 as they are published. </td>
555
        </tr>
556
        <tr>
557
          <td valign="top">30 </td>
558
          <td valign="top">10 </td>
559
          <td valign="top"><i>Currently undefined.&nbsp; Should probably wait
560
              until 10 more records are added before starting to return the
561
              data.</i> </td>
562
        </tr>
563
        <tr>
564
          <td valign="top">any </td>
565
          <td valign="top">&ndash;1 </td>
566
          <td valign="top">Returns "4.02 bad option" failure. </td>
567
        </tr>
568
      </tbody>
569
    </table>
570
    <br>
571
    <hr width="100%" size="2">
572
    <h4> Name</h4>
573
    <p>gdp_sub_qos_new, gdp_sub_qos_free &mdash; allocate/free subscription
574
      quality of service information</p>
575
    <h4>Synopsis</h4>
576
    <pre>gdp_sub_qos_t *gdp_sub_qos_new(void)</pre>
577
    <pre>void *gdp_sub_qos_free(<br>                gdp_sub_qos_t *qos)</pre>
578
    <h4>Notes</h4>
579
    <hr>
580
    <h4>Name</h4>
581
    <p>gdp_sub_qos_set_xyzzy &mdash; set xyzzy qos</p>
582
    <h4>Synopsis</h4>
583
    <pre>gdp_sub_qos_set_xyzzy(<br>                gdp_sub_qos_t *qos,<br>                xxx yyy)</pre>
584
    <h4>Notes</h4>
585
    <hr>
586
    <p><br>
587
    </p>
588
    <h4>Name</h4>
589
    gdp_gcl_multiread&mdash; Read multiple records from a readable GCL
590
    <h4> Synopsis</h4>
591
    <pre>EP_STAT gdp_gcl_multiread(<br>                gdp_gcl_t *gcl,
592
                gdp_recno_t start,<br>                int32_t numrecs,<br>                gdp_event_cbfunc_t cbfunc,<br>                void *udata)<br>EP_STAT gdp_gcl_multiread_ts(<br>                gdp_gcl_t *gcl,
593
                EP_TIME_SPEC *start,<br>                int32_t numrecs,<br>                gdp_event_cbfunc_t cbfunc,<br>                void *udata) void (*cbfunc)(gdp_event_t *gev)
594
</pre>
595
    <h4> Notes</h4>
596
    <ul>
597
      <li>Similar to a subscription, except data in the future is never read
598
        (i.e., this is only for reading historic data).&nbsp; This is the
599
        interface to use for asynchronous reads.</li>
600
      <li>If a <code>cbfunc</code> is specified, arranges to call callback when
601
        a message is generated on the <code>gcl</code>.&nbsp; See below for the
602
        definition of gdp_event_t. </li>
603
      <li> The callback is not necessarily invoked instantly, and may or may not
604
        be called in a separate thread.</li>
605
      <li>It is the responsibility of the callback function to call <code>gdp_event_free(gev)</code>.<br>
606
      </li>
607
      <li>If no <code>cbfunc</code> is specified, subscription information is
608
        available through the <code>gdp_gcl_event</code> interface (see below).</li>
609
      <li> The <code>udata</code> is passed through untouched in generated
610
        events.&nbsp; See below for the definition of <code>gdp_event_t</code>.</li>
611
      <li>At most <code>numrecs</code> records are returned, after which the
612
        subscription is terminated.&nbsp; If <code>numrecs</code> is 0 it reads
613
        to the end of the existing data. </li>
614
      <li> The <code>start</code> parameter tells when to start the
615
        subscription (that is, the starting record number).</li>
616
      <li>If <code>start</code> is negative, returns the most recent &ndash;<code>start</code>
617
        records.&nbsp; If a negative <code>start</code> indicates going back
618
        more records than are available, it starts from the first record. </li>
619
      <li>If <code>start</code> specifies an existing record but there are
620
        fewer than <code>numrecs</code> records available, only the existing
621
        records are returned.</li>
622
      <li>If <code>start</code> is zero, an error is returned.</li>
623
      <li><em>Callbacks make binding to languages like Java particularly
624
          difficult</em><em>, but fit more naturally in with languages such as
625
          Javascript.</em><em>&nbsp; Note also that callbacks will generally run
626
          in the GDP I/O thread (so no further GDP operations will run until the
627
          callback completes) or in a separate thread (in which case several
628
          instances of the callback may be run at once).</em></li>
629
      <li>Example:&nbsp; suppose there are 20 records already in a GCL.&nbsp;
630
        Then:</li>
631
    </ul>
632
    <table align="center" width="80%" cellspacing="2" cellpadding="2" border="1">
633
      <tbody>
634
        <tr>
635
          <td valign="top">start </td>
636
          <td valign="top">numrecs </td>
637
          <td valign="top">Behavior </td>
638
        </tr>
639
        <tr>
640
          <td valign="top">1 </td>
641
          <td valign="top">10 </td>
642
          <td valign="top">Returns records 1&ndash;10 immediately and terminates
643
            the read. </td>
644
        </tr>
645
        <tr>
646
          <td valign="top">&ndash;10 </td>
647
          <td valign="top">10 </td>
648
          <td valign="top">Returns records 11&ndash;20 immediately and
649
            terminates the read. </td>
650
        </tr>
651
        <tr>
652
          <td valign="top">0 </td>
653
          <td valign="top">any </td>
654
          <td valign="top">Returns "4.02 bad option" failure. </td>
655
        </tr>
656
        <tr>
657
          <td valign="top">&ndash;10 </td>
658
          <td valign="top">20 </td>
659
          <td valign="top">Returns records 11&ndash;20. </td>
660
        </tr>
661
        <tr>
662
          <td valign="top">1 </td>
663
          <td valign="top">0</td>
664
          <td valign="top">Returns records 1&ndash;20 immediately. </td>
665
        </tr>
666
        <tr>
667
          <td valign="top">&ndash;30 </td>
668
          <td valign="top">30 </td>
669
          <td valign="top">Returns records 1&ndash;20 immediately. </td>
670
        </tr>
671
        <tr>
672
          <td valign="top">30 </td>
673
          <td valign="top">10 </td>
674
          <td valign="top"><i>Currently undefined.&nbsp; Should probably wait
675
              until 10 more records are added before starting to return the
676
              data.</i> </td>
677
        </tr>
678
        <tr>
679
          <td valign="top">any </td>
680
          <td valign="top">&ndash;1 </td>
681
          <td valign="top">Returns "4.02 bad option" failure. </td>
682
        </tr>
683
      </tbody>
684
    </table>
685
    <br>
686
    <hr width="100%" size="2">
687
    <h4> Name</h4>
688
    gdp_gcl_unsubscribe &mdash; Unsubscribe from a GCL<span class="warning"></span>
689
    <h4> Synopsis</h4>
690
    <pre>EP_STAT gdp_gcl_unsubscribe(gdp_gcl_t *gcl,
691
                gdp_event_cbfunc_t *cbfunc,
692
                void *udata)
693
</pre>
694
    <h4> Notes</h4>
695
    <ul>
696
      <li> Deletes all subscriptions matching the given {gcl, cbfunc, udata}
697
        tuple.</li>
698
      <li>If <code>cbfunc</code> and or <code>udata</code> is <code>NULL</code>
699
        they are treated as wildcards.&nbsp; For example, "<code>gdp_gcl_unsubscribe(gcl,
700
          NULL, NULL)</code>" deletes all subscriptions for the given GCL.</li>
701
      <li>If there are multiple subscriptions matching this tuple, they will all
702
        be terminated.&nbsp; For example, this might happen in a multi-threaded
703
        application.</li>
704
      <li><em>(Should subscribe/multiread return a handle to be passed to this
705
          function rather than this interface?)</em></li>
706
    </ul>
707
    <br>
708
    <hr width="100%" size="2">
709
    <h4> Name</h4>
710
    gdp_event_next &mdash; get next asynchronous event
711
    <h4> Synopsis</h4>
712
    <pre>gdp_event_t *gdp_event_next(<br>                gdp_gcl_t *gcl,<br>                EP_TIME_SPEC *timeout)</pre>
713
    <h4> Notes</h4>
714
    <ul>
715
      <li> Returns the next asynchronous event on the specified GCL.&nbsp; If
716
        gcl is NULL, returns the next event from any GCL.</li>
717
      <li>Returns <code>NULL</code> if the <code>timeout</code> expires.&nbsp;
718
        If <code>timeout</code> is <code>NULL</code>, it waits forever. </li>
719
      <li>Currently the only asynchronous events are data arriving as the result
720
        of a subscription.</li>
721
      <li><em>(Should timeout be an absolute time or a delta on the current
722
          time?&nbsp; Currently it is implemented as a delta.)</em></li>
723
    </ul>
724
    <hr width="100%" size="2">
725
    <h4>Name</h4>
726
    gdp_event_gettype &mdash; extract the type from the event
727
    <h4>Synopsis</h4>
728
    <pre>int gdp_event_gettype(gdp_event_t *gev)</pre>
729
    <h4>Notes</h4>
730
    <ul>
731
      <li>The event types are as follows:</li>
732
    </ul>
733
    <table align="center" width="80%" cellspacing="2" cellpadding="2" border="1">
734
      <tbody>
735
        <tr>
736
          <td style="width: 162.2px;" valign="top">Event Name </td>
737
          <td style="width: 486.6px;" valign="top">Meaning </td>
738
        </tr>
739
        <tr>
740
          <td style="text-align: left; vertical-align: top;" valign="top"><code>GDP_EVENT_DATA</code>
741
          </td>
742
          <td style="text-align: left; vertical-align: top;" valign="top">Data
743
            is returned in the event from a previous subscription or
744
            asynchronous read. </td>
745
        </tr>
746
        <tr>
747
          <td style="text-align: left; vertical-align: top;" valign="top"><code>GDP_EVENT_EOS</code>
748
          </td>
749
          <td style="text-align: left; vertical-align: top;" valign="top">Subscription
750
            is terminated. </td>
751
        </tr>
752
        <tr>
753
          <td style="text-align: left; vertical-align: top;" valign="top"><tt>GDP_EVENT_SHUTDOWN</tt>
754
          </td>
755
          <td style="text-align: left; vertical-align: top;" valign="top">Subscription
756
            is terminated because the log daemon has shut down. </td>
757
        </tr>
758
        <tr>
759
          <td style="text-align: left; vertical-align: top;"><code>GDP_EVENT_CREATED</code></td>
760
          <td style="text-align: left; vertical-align: top;">Status is returned
761
            from an asynchronous append, create, or other similar operation.</td>
762
        </tr>
763
        <tr>
764
          <td style="vertical-align: top; background-color: white;"><code>GDP_EVENT_MISSING</code></td>
765
          <td style="vertical-align: top; background-color: white;">The
766
            requested data was not available at this time, but more data may be
767
            available.</td>
768
        </tr>
769
        <tr>
770
          <td style="text-align: left; vertical-align: top;"><span style="font-family: monospace;">GDP_EVENT_SUCCESS<br>
771
            </span></td>
772
          <td style="text-align: left; vertical-align: top;">Generic
773
            asynchronous success status. See the detailed status using <span style="font-family: monospace;">gdp_event_getstat</span>.</td>
774
        </tr>
775
        <tr>
776
          <td style="text-align: left; vertical-align: top;"><span style="font-family: monospace;">GDP_EVENT_FAILURE</span></td>
777
          <td style="text-align: left; vertical-align: top;">Generic
778
            asynchronous failure status.&nbsp; See the detailed status using <span
779
              style="font-family: monospace;">gdp_event_getstat</span>.</td>
780
        </tr>
781
      </tbody>
782
    </table>
783
    <ul>
784
      <li>Unrecognized events should be ignored. </li>
785
    </ul>
786
    <ul>
787
    </ul>
788
    <hr width="100%" size="2">
789
    <h4>Name</h4>
790
    <p>gdp_event_getstat &mdash; extract the detailed result status from the
791
      event</p>
792
    <h4>Synopsis</h4>
793
    <p><code>EP_STAT gdp_event_getstat(gdp_event_t *gev)</code></p>
794
    <h4>Notes</h4>
795
    <ul>
796
      <li>Returns the status code from the event.&nbsp; In most cases this will
797
        be <code>EP_STAT_OK</code>, but may be otherwise in some event types
798
        such as <code>GDP_EVENT_ASTAT</code>.</li>
799
    </ul>
800
    <hr>
801
    <h4>Name</h4>
802
    gdp_event_getgcl &mdash; extract the GCL handle from the event
803
    <h4>Synopsis</h4>
804
    <pre>gdp_gcl_t *gdp_event_getgcl(gdp_event_t *gev)</pre>
805
    <h4>Notes</h4>
806
    <ul>
807
      <li>Returns the GCL handle that triggered this event.</li>
808
    </ul>
809
    <hr width="100%" size="2">
810
    <h4>Name</h4>
811
    gdp_event_getdatum &mdash; get the datum associated with this event
812
    <h4>Synopsis</h4>
813
    <pre>gdp_datum_t *gdp_event_getdatum(gdp_event_t *gev)</pre>
814
    <h4>Notes</h4>
815
    <ul>
816
      <li>Returns the data associated with the event.</li>
817
    </ul>
818
    <hr width="100%" size="2">
819
    <h4>Name</h4>
820
    <p>gdp_event_getudata &mdash; get user data associated with this event </p>
821
    <h4>Synopsis</h4>
822
    <pre>void *gdp_event_getudata(gdp_event_t *gev)
823
    </pre>
824
    <h4>Notes</h4>
825
    <ul>
826
      <li>Returns user data associated with the event.&nbsp; The user data is
827
        from the call that initiated the asynchronous operation. </li>
828
    </ul>
829
    <hr width="100%" size="2">
830
    <h3>3.3&nbsp; GCL Open Information</h3>
831
    <p>There is potentially a huge amount of information that might be provided
832
      when a GCL is opened.&nbsp; Because this set is open-ended, it is
833
      abstracted out into a separate API.&nbsp; The <code class="datatype">gdp_gcl_open_info_t</code>
834
      datastructure encapsulates this information and can be passed into <code
835
        class="function">gdp_gcl_open</code>.</p>
836
    <hr>
837
    <h4>Name</h4>
838
    <p>gdp_gcl_open_info_new &mdash; create new open information datastructure</p>
839
    <h4>Synopsis</h4>
840
    <pre><span>gdp_gcl_open_info_t *gdp_gcl_open_info_new(void)</span></pre>
841
    <h4>Notes</h4>
842
    <ul>
843
      <li>Allocates a new <code class="datatype">gdp_gcl_open_info_t</code>
844
        datastructure.</li>
845
    </ul>
846
    <hr>
847
    <h4>Name</h4>
848
    <p>gdp_gcl_open_info_free &mdash; free an existing open information
849
      datastructure</p>
850
    <h4>Synopsis</h4>
851
    <pre>void gdp_gcl_open_info_free(gdp_gcl_open_info_t *info)</pre>
852
    <h4>Notes</h4>
853
    <ul>
854
      <li>Frees an existing <code class="datatype">gdp_gcl_open_info_t</code>
855
        datastructure.</li>
856
    </ul>
857
    <hr>
858
    <h4>Name</h4>
859
    <p>gdp_gcl_open_info_set_signing_key &mdash; set the signing key in an open
860
      information datastructure</p>
861
    <h4>Synopsis</h4>
862
    <pre>EP_STAT gdp_gcl_open_info_set_signing_key(
863
                gdp_gcl_open_info_t *info,
864
                EP_CRYPTO_KEY *skey)</pre>
865
    <h4>Notes</h4>
866
    <ul>
867
      <li>Uses the indicated secret signing key <code class="variable">skey</code>
868
        to do the signing for any GCL opened using this datastructure.</li>
869
      <li>When <code class="variable">info</code> is freed, the signing key
870
        will be freed as well.</li>
871
    </ul>
872
    <hr><br>
873
    <p><br>
874
    </p>
875
    <h2>4&nbsp; Datums (Records)</h2>
876
    <p>GCLs are represented as a series of records of type <code>gdp_datum_t</code>.&nbsp;
877
      Each record has a record number, a commit timestamp, associated data, and
878
      possible signature information if the record was signed.&nbsp; Record
879
      numbers are of type <code>gdp_recno_t</code> and count up by one as
880
      records are added (i.e., record numbers are unique within a GCL and
881
      dense).&nbsp; Data is represented in dynamic buffers, as described below.
882
    </p>
883
    <h3>4.1&nbsp; Datum Headers</h3>
884
    <hr width="100%" size="2">
885
    <h4>Name</h4>
886
    <p>gdp_datum_new, gdp_datum_free, gdp_datum_print &mdash;
887
      allocate/free/print a datum structure </p>
888
    <h4>Synopsis</h4>
889
    <pre>gdp_datum_t *gdp_datum_new(void)
890
void gdp_datum_free(gdp_datum_t *datum)
891
void gdp_datum_print(const gdp_datum_t *datum,
892
                FILE *fp,
893
                uint32_t flags)</pre>
894
    <h4>Notes</h4>
895
    <ul>
896
      <li><code>gdp_datum_new</code> allocates a new empty datum.</li>
897
      <li><code>gdp_datum_free</code> frees a datum.</li>
898
      <li><code>gdp_datum_print</code> writes a description of the datum
899
        (including the data contents) to the given file.&nbsp; If flags includes
900
        the <span class="manifest">GDP_DATUM_PRTEXT</span> bit, it shows the
901
        datum as plain text (the default shows it as a hex dump).&nbsp; It is up
902
        to the caller to determine that the datum is printable.&nbsp; If the <span
903
          class="manifest">GDP_DATUM_PRSIG</span> bit is set, signature
904
        information is included.&nbsp; If the <span class="manifest">GDP_DATUM_PRDEBUG</span>
905
        flag is set, additional information about the datum is printed.</li>
906
    </ul>
907
    <hr width="100%" size="2">
908
    <h4>Name</h4>
909
    gdp_datum_getrecno &mdash; get the record number from a datum
910
    <h4>Synopsis</h4>
911
    <pre>    <tt>gdp_recno_t gdp_datum_getrecno(const gdp_datum_t *datum)
912
</tt></pre>
913
    <h4>Notes </h4>
914
    <hr width="100%" size="2">
915
    <h4>Name</h4>
916
    gdp_datum_getts &mdash; get the timestamp from a datum
917
    <h4>Synopsis</h4>
918
    <pre>    void gdp_datum_getts(const gdp_datum_t *datum, EP_TIME_SPEC *ts)
919
</pre>
920
    <h4>Notes</h4>
921
    <hr width="100%" size="2">
922
    <h4>Name</h4>
923
    gdp_datum_getdlen &mdash; get the data length from a datum
924
    <h4>Synopsis</h4>
925
    <pre>    size_t gdp_datum_getdlen(const gdp_datum_t *datum)</pre>
926
    <h4>Notes</h4>
927
    <hr width="100%" size="2">
928
    <h4>Name</h4>
929
    gdp_datum_getdbuf &mdash; get the data buffer from a datum
930
    <h4>Synopsis</h4>
931
    <pre>    gdp_buf_t *gdp_datum_getdbuf(const gdp_datum_t *datum)</pre>
932
    <h4>Notes</h4>
933
    <hr width="100%" size="2">
934
    <h4>Name</h4>
935
    gdp_datum_getsigmdalg &mdash; get the signature message digest algorithm
936
    from a datum
937
    <h4>Synopsis</h4>
938
    <pre>    int gdp_datum_getmdalg(const gdp_datum_t *datum)</pre>
939
    <h4>Notes</h4>
940
    <ul>
941
      <li>Returns the type of the signature digest algorithm (if a signature
942
        exists).</li>
943
      <li>The values are defined by the libep.&nbsp; They consist of the major
944
        SHA algorithms in various sizes, e.g., <span class="manifest">EP_CRYPTO_MD_SHA1</span>,
945
        <span class="manifest">EP_CRYPTO_MD_SHA224</span>, etc. through <span class="manifest">SHA512</span>.</li>
946
    </ul>
947
    <hr width="100%" size="2">
948
    <h4>Name</h4>
949
    gdp_datum_getsig &mdash; get the signature from a datum
950
    <h4>Synopsis</h4>
951
    <pre>    gdp_buf_t *gdp_datum_getsig(const gdp_datum_t *datum)</pre>
952
    <h4>Notes</h4>
953
    <ul>
954
      <li>Can return <span class="manifest">NULL</span> or an empty buffer if
955
        there is no signature.</li>
956
    </ul>
957
    <hr width="100%" size="2">
958
    <h3>4.2&nbsp; Data Buffers</h3>
959
    <p>Data buffers grow dynamically as needed. </p>
960
    <hr width="100%" size="2">
961
    <h4>Name</h4>
962
    gdp_buf_new, gdp_buf_reset, gdp_buf_free &mdash; allocate, reset, or free a
963
    buffer
964
    <h4>Synopsis</h4>
965
    <pre>gdp_buf_t *gdp_buf_new(void)<br>void gdp_buf_reset(gdp_buf_t *b)<br>void gdp_buf_free(gdp_buf_t *b)</pre>
966
    <h4>Notes</h4>
967
    <ul>
968
      <li><code>gdp_buf_new</code> creates a new, empty buffer.</li>
969
      <li><code>gdp_buf_reset</code> clears the buffer, leaving it in the same
970
        condition as when it was first created.</li>
971
      <li><code>gdp_buf_free</code> frees the buffer.&nbsp; It must not be used
972
        again after being freed.</li>
973
    </ul>
974
    <hr width="100%" size="2">
975
    <h4>Name</h4>
976
    gdp_buf_getlength &mdash; return the length of the data in the buffer
977
    <h4>Synopsis</h4>
978
    <pre>size_t gdp_buf_getlength(gdp_buf_t *b)</pre>
979
    <h4>Notes</h4>
980
    <ul>
981
      <li>Returns the number of bytes of data currently in the buffer.</li>
982
    </ul>
983
    <hr width="100%" size="2">
984
    <h4>Name</h4>
985
    gdp_buf_read, gdp_buf_peek, gdp_buf_drain &mdash; remove or peek at data in
986
    a buffer
987
    <h4>Synopsis</h4>
988
    <pre>size_t gdp_buf_read(gdp_buf_t *b, void *out, size_t sz)<br>size_t gdp_buf_peek(gdp_buf_t *b, void *out, size_t sz)<br>int gdp_buf_drain(gdp_buf_t *b, size_t sz)</pre>
989
    <h4>Notes</h4>
990
    <ul>
991
      <li>Data can be consumed from the buffer by calling <code>gdp_buf_read</code>;
992
        data is copied from the buffer into a memory area.</li>
993
      <li>Applications can "peek" at the buffer using <code> gdp_buf_peek</code>.&nbsp;
994
        This is identical to <code>gdp_buf_read</code> except that the data
995
        remains in the buffer.</li>
996
      <li>Applications can discard data from the buffer using <code>
997
          gdp_buf_drain</code>.</li>
998
      <li>In all cases, <code>sz</code> is the number of bytes to copy out
999
        and/or discard.</li>
1000
    </ul>
1001
    <hr width="100%" size="2">
1002
    <h4>Name</h4>
1003
    gdp_buf_write, gdp_buf_printf &mdash; copy data into a buffer
1004
    <h4>Synopsis</h4>
1005
    <pre>int gdp_buf_write(gdp_buf_t *b, void *in, size_t sz)<br>int gdp_buf_printf(gdp_buf_t *b, const char *fmt, ...)</pre>
1006
    <h4>Notes</h4>
1007
    <ul>
1008
      <li>These routines insert bytes into the named buffer.</li>
1009
      <li><code>gdp_buf_write</code> copies <code>sz</code> bytes into the
1010
        buffer from the memory area in and returns 0 on success or &ndash;1 on
1011
        failure. </li>
1012
      <li><code>gdp_buf_printf</code> essentially does a "printf" into the
1013
        buffer and returns the number of bytes appended. </li>
1014
    </ul>
1015
    <hr width="100%" size="2">
1016
    <h4>Name</h4>
1017
    gdp_buf_move &mdash; move data from one buffer into another
1018
    <h4>Synopsis</h4>
1019
    <pre>int gdp_buf_move(gdp_buf_t *ob, gdp_buf_t *ib, size_t sz)</pre>
1020
    <h4>Notes</h4>
1021
    <ul>
1022
      <li>Appends the first <code>sz</code> bytes of <code>ib</code> to the
1023
        end of <code>ob</code>.</li>
1024
      <li>This is more efficient than using <code>gdp_buf_read</code> and <code>gdp_buf_write</code>.</li>
1025
    </ul>
1026
    <hr width="100%" size="2">
1027
    <h4>Name</h4>
1028
    gdp_buf_dump &mdash; print the contents of the buffer for debugging
1029
    <h4>Synopsis</h4>
1030
    <pre>void gdp_buf_dump(gdp_buf_t *b, FILE *fp)</pre>
1031
    <h4>Notes</h4>
1032
    <ul>
1033
      <li>Prints the contents of buffer b to the file fp.</li>
1034
      <li>This is not intended for end user presentation.</li>
1035
    </ul>
1036
    <hr width="100%" size="2">
1037
    <h3>4.3&nbsp; Timestamps</h3>
1038
    <p>The time abstraction is imported directly from the ep library.&nbsp;
1039
      Times are represented as follows: </p>
1040
    <blockquote>
1041
      <pre><code>#pragma pack(push, 1)
1042
typedef struct
1043
{
1044
     int64_t        tv_sec;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp; // seconds since January 1, 1970
1045
    &nbsp;uint32_t&nbsp;&nbsp; tv_nsec;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; // nanoseconds<br>     float&nbsp;&nbsp;&nbsp;&nbsp;  tv_accuracy;&nbsp;&nbsp;&nbsp; // accuracy in seconds<br>} EP_TIME_SPEC;<br>#pragma pack(pop)</code></pre>
1046
    </blockquote>
1047
    Note that the host system <code>struct timespec</code> may not match this
1048
    structure; some systems still represent the time with only four bytes for <code>tv_sec</code>,
1049
    which expires in 2038.&nbsp; The <code>tv_accuracy</code> field indicates
1050
    an estimate for how accurate the clock is; for example, if you are running
1051
    NTP this value is likely to be on the order of a few tens to a few hundreds
1052
    of milliseconds, but if you set your clock manually it is likely to be
1053
    several seconds or worse.
1054
    <h2>5&nbsp; Signing and Encryption</h2>
1055
    <h3> 5.1&nbsp; Signing</h3>
1056
    <p>Each log should have a public key in the metadata which is used to verify
1057
      writes to the log.&nbsp; The library hides most of the details of this,
1058
      but some still appear.</p>
1059
    <p> The <span class="command">gcl-create</span> command automatically
1060
      creates a public/secret keypair unless otherwise specified.&nbsp; See the
1061
      man page for details.&nbsp; The public part of the key is inserted into
1062
      the log metadata and stored with the log.&nbsp; The secret part is stored
1063
      somewhere on your local filesystem, typically <span class="filename">KEYS/<span
1064
          class="variable">gcl-id</span>.pem</span>.&nbsp; Normally <span class="command">gcl-create</span>
1065
      will encrypt the secret key with another key entered from the command
1066
      line, although this can also be turned off.</p>
1067
    <p>When a GDP application attempts to open a log using <span class="function">gdp_gcl_open</span>,
1068
      the library will attempt to find a secret key by searching the directories
1069
      named in the <span class="admin-param">swarm.gdp.crypto.key.path</span>
1070
      administrative parameter for a file having the same name as the log (with
1071
      a <span class="filename">.pem</span> file suffix).&nbsp; If that secret
1072
      key is encrypted, the library will prompt the (human) user for the secret
1073
      key password.&nbsp; The default path is "<span class="filename">.</span>",
1074
      "<span class="filename">KEYS</span>", "<span class="filename">~/.swarm/gdp/keys</span>",
1075
      "<span class="filename">/usr/local/etc/swarm/gdp/keys</span>", and "<span
1076
        class="filename">/etc/swarm/gdp/keys</span>".</p>
1077
    <p>Once the secret key has been located and decrypted, all further append
1078
      requests will be signed using the secret key and verified by the log
1079
      daemon against the public key in the log metadata.</p>
1080
    <h3>5.2&nbsp; Encryption</h3>
1081
    <p>Encryption is explicitly not part of the GDP.&nbsp; Ideally the GDP will
1082
      never see unencrypted data.&nbsp; However, read and write filters (see the
1083
      next section) can be used to set encryption and decryption hooks for
1084
      externally implemented encryption.</p>
1085
    <h2>6&nbsp; Miscellaneous and Utilities</h2>
1086
    <hr>
1087
    <h4>Name</h4>
1088
    <p>gdp_gcl_getnrecs &mdash; return the number of records in an existing GCL</p>
1089
    <h4>Synopsis</h4>
1090
    <p><code>gdp_recno_t gdp_gcl_getnrecs(gdp_gcl_t *gcl)</code></p>
1091
    <h4>Notes</h4>
1092
    <ul>
1093
      <li>Returns the number of records in a GCL.</li>
1094
      <li>This does not check with the log server, i.e., the return value is
1095
        cached in the local process (and is updated each time the GCL is read or
1096
        written).&nbsp; This shouldn't be a problem for writers (since there
1097
        should only be one writer), but may be for readers.&nbsp; Readers
1098
        wanting to get the latest number of records can read the last record in
1099
        the GCL (using gdp_gcl_read) and check the record number in the returned
1100
        datum.</li>
1101
    </ul>
1102
    <hr>
1103
    <h4>Name</h4>
1104
    <p>gdp_gcl_set_append_filter &mdash; filter appended data</p>
1105
    <h4>Synopsis</h4>
1106
    <code>void gdp_gcl_set_append_filter(gdp_gcl_t *gcl,<br>
1107
      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
1108
      EP_STAT (*filter(gdp_datum_t *, void *),<br>
1109
      &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp; void *filterdata)</code><br>
1110
    <h4>Notes</h4>
1111
    <ul>
1112
      <li>All data appended to the named <span class="variable">gcl</span> are
1113
        first run through the <span class="variable">filter</span>.&nbsp; The
1114
        filter may modify the <span class="variable">datum</span> before it is
1115
        sent.</li>
1116
      <li>The <span class="variable">filterdata</span> parameter is passed as
1117
        the second argument to <span class="variable">filter</span>.</li>
1118
    </ul>
1119
    <hr><br>
1120
    <br>
1121
    <h4>Name</h4>
1122
    <p>gdp_gcl_set_read_filter &mdash; filter read data</p>
1123
    <h4>Synopsis</h4>
1124
    <p><code>void gdp_gcl_set_read_filter(gdp_gcl_t *gcl,<br>
1125
        &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
1126
        EP_STAT (*filter(gdp_datum_t *, void *),<br>
1127
        &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp; void *filterdata)</code></p>
1128
    <h4>Notes</h4>
1129
    <ul>
1130
      <li>All data read from the named <span class="variable">gcl</span> are
1131
        passed through this filter before being returned.&nbsp; The filter may
1132
        modify the <span class="variable">datum</span>.</li>
1133
      <li>The <span class="variable">filterdata</span> parameter is passed as
1134
        the second argument to <span class="variable">filter</span>.</li>
1135
    </ul>
1136
    <hr>
1137
    <h3>To be done</h3>
1138
    <p>Header files<br>
1139
      Version info<br>
1140
      PRIgdp_recno macro<br>
1141
    </p>
1142
    <h2>Appendix A:&nbsp; Examples</h2>
1143
    <p>The following pseudo-code example excerpts from <tt>apps/gdp-writer.c</tt>.
1144
    </p>
1145
    <pre>#include &lt;gdp/gdp.h&gt;<br>#include &lt;stdio.h&gt;<br>#include &lt;string.h&gt;<br><br>int main(int argc, char **argv)<br>{<br>        gdp_tcl_t *gcl;<br>        EP_STAT estat;<br>        gdp_name_t gcliname;        // internal name of GCL<br>        gdp_datum_t *d;<br><br>        // general startup and initialization<br>        if (argc &lt; 2)<br>                usage_error();<br>        estat = gdp_init();<br>        if (!EP_STAT_ISOK(estat))<br>                initialization_error(estat);<br>        d = gdp_datum_new();
1146
<br><br>        // parse command line name to internal format<br>        estat = gdp_gcl_parse_name(argv[1], gcliname);<br>        if (!EP_STAT_ISOK(estat))<br>                name_syntax_error();<br><br>        // attempt to create that name<br>        estat = gdp_gcl_create(gcliname, &amp;gcl);<br>        if (!EP_STAT_ISOK(estat))<br>                creation_error(estat);<br><br>        // read lines from standard input<br>        while (fgets(buf, sizeof buf, stdin) != NULL)<br>        {<br>                char *p = strchr(buf, '\n');<br>                if (p != NULL)<br>                        *p = '\0';<br><br>                // write them to the dataplane<br>                if (gdp_buf_write(gdp_datum_getbuf(d), buf, strlen(buf)) &lt; 0)<br>                        estat = GDP_STAT_BUFFER_FAILURE;<br>                else<br>                        estat = gdp_gcl_append(gcl, d);<br>                EP_STAT_CHECK(estat, break);<br>        }<br><br>        // cleanup and exit<br>        gdp_gcl_close(gcl);<br>        exit(!EP_STAT_ISOK(estat));<br>}</pre>
1147
    <hr width="100%" size="2">
1148
    <p>This example is a similar excerpt from apps/gdp-reader.c (without using
1149
      subscriptions):</p>
1150
    <pre>#include &lt;gdp/gdp.h&gt;<br><br>int main(int argc, char **argv)<br>{<br>        gdp_gcl_t *gcl;<br>        EP_STAT estat;<br>        gdp_name_t gcliname;        // internal name of GCL<br>        gdp_datum_t *d;<br>        gdp_recno_t recno;<br><br>        // general startup and initialization<br>        if (argc &lt; 2)<br>                usage_error();<br>        estat = gdp_init();<br>        if (!EP_STAT_ISOK(estat))<br>                initialization_error(estat);<br>        d = gdp_datum_new();
1151
<br><br>        // parse command line name to internal format<br>        estat = gdp_gcl_parse_name(argv[1], gcliname);<br>        if (!EP_STAT_ISOK(estat))<br>                name_syntax_error();
1152
<br>        // attempt to open the GCL<br>        estat = gdp_gcl_open(gcliname, GDP_MODE_RO, &amp;gcl);<br>        if (!EP_STAT_ISOK(estat))<br>                open_error(estat, argv[1]);<br>
1153
&nbsp;        recno = 1;<br>        for (;;)<br>        {<br>                estat = gdp_gcl_read_by_recno(gcl, recno++, d);<br>                EP_STAT_CHECK(estat, break);<br>                gdp_datum_print(d, stdout);<br>        }<br>        exit(0);<br>}</pre>
1154
    <br>
1155
    <hr width="100%" size="2">
1156
    <p>If you want to use subscriptions, the recno variable can be removed and
1157
      the for loop replaced with: </p>
1158
    <pre>        // enable the subscription<br>        estat = gdp_gcl_subscribe_by_recno(gcl, 1, -1, NULL, NULL);<br>        if (!EP_STAT_ISOK(estat))<br>                subscribe_error(estat, argv[1]);<br><br>        for (;;)<br>        {<br>                gdp_event_t *gev = gdp_event_next(true);<br>                if (gdp_event_gettype(gev) != GDP_EVENT_DATA)<br>                        continue;<br>                gdp_datum_print(gdp_event_getdatum(gev), stdout);<br>                gdp_event_free(gev);<br>        }</pre>
1159
    <br>
1160
    <hr width="100%" size="2">
1161
    <h2>Appendix B:&nbsp; Compiling and Linking</h2>
1162
    <p>The GDP library uses a reduced version of libep and also uses the
1163
      libevent library version 2.1. These will need to be included both during
1164
      compilation and linking.</p>
1165
    <p> At compile time you must use:</p>
1166
    <p><code>-I</code><code><i>libevent_includes_parent</i></code><code> -I</code><code><i>libep_includes_parent</i></code></p>
1167
    <p> Note that these take the parent of the directory containing the include
1168
      files. For example, if the include files for libevent are in <tt>/usr/local/include/event2</tt>
1169
      and the include files for libep are in <tt>/usr/local/include/ep</tt> you
1170
      only need to specify the one flag "<code>-I/usr/local/include</code>".</p>
1171
    For linking you must use:<br>
1172
    <pre>-L<i>libevent_libraries</i> -levent -levent_pthreads -L<i>libep_libraries</i> -lep</pre>
1173
    As before, if the libraries for libevent and libep are in the same directory
1174
    you only need a single <tt>-L</tt> flag.<br>
1175
    Libep is a library that I produced several years ago intended for use in
1176
    sendmail. This uses a stripped down version of that library that excludes
1177
    several things that would not be helpful here. For more details of the
1178
    original (full) library, see <a href="http://www.neophilic.com/blogs/eric.php/2014/05/12/libep-portable-c-runtime">http://www.neophilic.com/blogs/eric.php/2014/05/12/libep-portable-c-runtime</a>.
1179
    <p>For additional information, see the <tt>README</tt> file in the
1180
      distribution directory. </p>
1181
    <h2> Appendix C: Open Questions</h2>
1182
    <p>This section is really an addendum to the document &mdash; a "scratch
1183
      area" to keep track of issues that we still need to consider.&nbsp; It may
1184
      not be up to date. </p>
1185
    <h3>C.1 Access Control</h3>
1186
    <p> Do this using Access Control Lists (so each user/app has a keypair) or
1187
      by passing public/secret keys around (so each GCL has a secret keypair).
1188
      The latter makes revocation impossible (even for write access), so I
1189
      prefer the ACL approach. Third way?</p>
1190
    <p> Revocation? Deep vs. Shallow. Deep = take away permissions that have
1191
      already been given. Shallow = you can only prevent an accessor from
1192
      getting to new versions. Argument: deep revocation is hard to do from a
1193
      technical perspective and ultimately futile (someone might have taken a
1194
      photo of a screen while they still had access), but is still what people
1195
      are used to (Unix and SQL permissions work this way). Shallow is all that
1196
      can really be guaranteed. Also, anything involving Certificate Revocation
1197
      Lists (CRLs) is doomed to failure. This implies that ACLs are the correct
1198
      direction.</p>
1199
    <p> ACLs get us into the question of identity. Pretending that a keypair
1200
      represents an identity doesn't work in the real world where bad players
1201
      simply create new "identities" (keypairs) when an old identity has become
1202
      untrusted. See the extensive work in email sender reputation. However,
1203
      when a bad player creates a new identity/keypair they do not get access to
1204
      any previous grants, so this may be sufficient.</p>
1205
    <h3> C.2 Naming</h3>
1206
    <p> If each GCL has a secret keypair, then the public key is sufficient to
1207
      name the entity. If not, then assigning a GCL a GUID on creation seems
1208
      like the best approach. Having the user assign a name seems like a
1209
      non-starter, if only because of the possibility of conflicts.</p>
1210
    <p> There will probably be some need for external naming, e.g., some overlay
1211
      directory structure. That might be a different gcl_type.</p>
1212
    <p> This seems like an open research topic.</p>
1213
    <h3> C.3 Orphans, Expiration, Charging, and Accounting</h3>
1214
    <p> If a GCL isn't linked into a directory structure and everyone forgets
1215
      its name then it will live forever (or until it expires). This could be
1216
      quite common if a GCL is temporary, that is, not a candidate for long-term
1217
      archival.</p>
1218
    <p> Expiration could be an issue without some sort of charging, which
1219
      implies accounting.</p>
1220
    <p> Charging and accounting will affect the API. It seems like on GCL
1221
      creation the creator needs to offer payment for both carrying and storing
1222
      the data. This payment would presumably accrue to the actors providing the
1223
      actual service. Payment for storage might be limited time or indefinite
1224
      time (i.e., it would be an endowment).</p>
1225
    <p> The creator could also specify a cost for any potential consumer in
1226
      order to access the GCL. Such payments would accrue to the creator of the
1227
      GCL, and might be used to fund continued access, i.e. it could be rolled
1228
      back into the endowment. This would lean toward making less-used data
1229
      disappear: appealing in some ways, but anathema to librarians and
1230
      historians.</p>
1231
    <p> As for API effects, it seems that GCL creation needs to include a
1232
      payment for initial service, a cost for access, and an account into which
1233
      to deposit any consumer payments. Accessing a GCL only requires an offered
1234
      payment (which is probably best viewed as a bid rather than a payment,
1235
      thus allowing multiple providers to compete for access).</p>
1236
    <p> Note that none of this is dependent on the form of payment. It does
1237
      however assume that there is a mutually agreed upon form of payment, i.e.,
1238
      a universal currency.</p>
1239
    <h3> C.4 Quality of Service</h3>
1240
    <p> Is Quality of Service specified on a particular GCL, a particular open
1241
      instance of a GCL, or between a pair of endpoints?</p>
1242
    <p> What does QoS actually mean? For example, in a live media stream it
1243
      probably means the resolution of the data stream (which determines
1244
      real-time bandwidth), latency, and possibly jitter, but after that stream
1245
      is stored the QoS will be resolution (as before), delivery bandwidth (how
1246
      quickly you can download the video, for example), and possibly jitter of
1247
      the network connection (that is, how even the data flow will be). Delivery
1248
      bandwidth depends on the entire path between the data source and the data
1249
      sync, and may be higher or lower than the bandwidth required to send a
1250
      real-time version of the stream &mdash; for example, over a slow network
1251
      link.</p>
1252
    <h2> Appendix D: References</h2>
1253
    <table width="100%" cellspacing="2" cellpadding="2" border="1">
1254
      <tbody>
1255
        <tr>
1256
          <td valign="top">[Dab13a] </td>
1257
          <td valign="top">Palmer Dabbelt, Swarm OS Universal Dataplane, August
1258
            22, 2013</td>
1259
        </tr>
1260
        <tr>
1261
          <td valign="top">[Dab13b]</td>
1262
          <td valign="top">Palmer Dabbelt, What is the Universal Dataplane,
1263
            Anyway?, September 17, 2013</td>
1264
        </tr>
1265
      </tbody>
1266
    </table>
1267
  </body>
1268
</html>