Project

General

Profile

Statistics
| Branch: | Tag: | Revision:

gdp / doc / developer / gdp-programmatic-api.html @ master

History | View | Annotate | Download (76.3 KB)

1
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
2
<html>
3
  <head>
4
    <meta http-equiv="content-type" content="text/html; charset=windows-1252">
5
    <title>Global Data Plane Programmatic API</title>
6
    <style type="text/css">
7
.warning {
8
  font-weight: bold;
9
  font-style: italic;
10
  color: red;
11
  background-color: #ffff99;
12
}
13
.meta {
14
  font-style:italic;
15
  color: blue;
16
}
17
.command {
18
  font-family: monospace;
19
}
20
.variable {
21
  font-style: oblique;
22
}
23
.filename {
24
  font-family: monospace;
25
}
26
.admin-param {
27
  font-family: monospace;
28
}
29
.function {
30
  font-family: monospace;
31
}
32
.manifest {
33
  font-family: monospace;
34
}
35
.datatype {
36
  font-family: monospace;
37
}
38

    
39
</style></head>
40
  <body>
41
    <h1>Global Data Plane Programmatic API </h1>
42
    <h4>Editor: Eric Allman, U.C. Berkeley Swarm Lab, eric@cs.berkeley.edu<br>
43
      Version 2.1.0, 2018-10-20</h4>
44
    <p>This document describes the procedural programmatic interface to the
45
      Global Data Plane.&nbsp; The native code is written in C for maximum
46
      flexibility and performance, but it is expected that most applications
47
      will be written in higher level languages, and hence there will be
48
      multiple language bindings for the GDP library.&nbsp; There is also a REST
49
      interface that is not described in this document. </p>
50
    <p>The GDP library uses the EP portability library, and applications are
51
      free to use that library as well; in particular, since the GDP libraries
52
      makes extensive use of the EP library some efficiencies may result from
53
      using them both.&nbsp; However, this document does not attempt to define
54
      the EP library and describes it only as necessary.&nbsp; However, one EP
55
      concept that appears commonly is the <code>EP_STAT</code> data type,
56
      which represents a status code that can be returned by a function to
57
      indicate completion status that includes a "severity" (e.g. OK, ERROR,
58
      SEVERE), a "registry" (in our case always UC Berkeley), a "module" (e.g.,
59
      GDP or the EP library itself), and detail information.&nbsp; An OK status
60
      can return a positive integer as extra information. </p>
61
    <p>The code distribution includes an "apps" directory with two example
62
      programs: <code>gdp-writer.c</code> and <code>gdp-reader.c</code>, that
63
      show simple cases of how to append to a GOB and read from a GOB (including
64
      subscriptions). </p>
65
    <p>This document corresponds to version 2.1 of the GDP library and
66
      associated applications.&nbsp; Note that this interface is massively
67
      incompatible with version 0.9.</p>
68
    <h2 class="meta">0&nbsp; To Be Done</h2>
69
    <ul>
70
      <li>Document metadata interfaces, e.g., <code>gdp_gin_getmetadata</code>,
71
        <code>gdp_md_get</code>.</li>
72
      <li>Convert this documentation into man pages.</li>
73
    </ul>
74
    <h2>1&nbsp; Terminology</h2>
75
    <dl>
76
      <dt>Datum</dt>
77
      <dd>A unit of data in the GDP; essentially a record.&nbsp; Each datum has
78
        associated with it a record number (numbered sequentially from one), a
79
        commit timestamp (the time the record was committed into the GDP, as
80
        distinct from the time that the data originated), an associated blob
81
        containing the data itself, which we expect to be encrypted, and in most
82
        cases a signature.&nbsp; Beware: because of our unique single-writer
83
        commit protocol, record numbers are not guaranteed to be unique nor
84
        continuous; for example, there can be no record 3 but two record 4s in
85
        some circumstances.&nbsp; This should be rare.</dd>
86
      <dt>GDP Object (GOB)</dt>
87
      <dd>This represents an addressable entity in the Global Data Plane, which
88
        may be a log or a service.&nbsp; It is always addressed with a
89
        location-independent 256-bit name (a "GDPName").&nbsp; Applications
90
        interact with GOBs via GDP Instances (GINs).</dd>
91
      <dt>GOB Instance (GIN)</dt>
92
      <dd>An instance of a Global Data Plane log.&nbsp; An application can have
93
        multiple instances of a given GDP Object.&nbsp; It is the equivalent of
94
        a Unix File Descriptor (as distinct from a Unix file) that is created by
95
        <code>gdp_gin_open</code>.</dd>
96
      <dt>GDP Name</dt>
97
      <dd>The name of a Global Data Plane Object.&nbsp; This is a 256-bit,
98
        opaque, globally unique number created as the SHA-256 hash of the
99
        metadata of the object.&nbsp; The usual representation for printing is a
100
        43 character base64 encoded string.</dd>
101
      <dt>Human-Oriented Name</dt>
102
      <dd>A name that human beings like to use, such as a conventional file
103
        name.&nbsp; These are unrelated to the internal name.&nbsp; A
104
        Human-Oriented Name to GDPname Directory Service (HONGDS) maintains a
105
        database mapping from one to the other.</dd>
106
    </dl>
107
    <p> In earlier versions of the GDP, both GINs and GOBs were referred to as
108
      GCLs (GDP Channel-Logs). These names are now broken apart for clarity. For
109
      historic reasons, some of the interfaces and much of the documentation
110
      still refer to GCLs, but that name is officially deprecated.</p>
111
    <p><i>A Note on Naming:</i> Function and type names are mostly intended to
112
      map cleanly to class and method names, which occasionally leads to
113
      somewhat tortured names.&nbsp; For example, function names beginning with
114
      <code>gdp_gin_</code> operate on objects of type <code>gdp_gin_t</code>
115
      (with a few exceptions such as <code>gdp_gin_new</code>) and will take a
116
      pointer to a <code>gdp_gin_t</code> as the first ("self") argument.&nbsp;
117
      In most cases, everything that has a type (identified by a name ending <code>_t</code>)
118
      is probably a class.&nbsp; More concretely, <code>gdp_gin_read_by_recno(gin,
119
        ...)</code> where <code>gin</code> is type <code>gdp_gin_t</code> is
120
      intended to map to a method "<code>gin.read_by_recno(...)</code>" (i.e,
121
      the operation <code>read_by_recno</code> on a variable of class <code>gdp_gin</code>),
122
      <code>gdp_buf_getlength(buf, ...)</code> maps to <code><em>buf</em>.getlength(...)</code>,
123
      etc.&nbsp; In some cases these also apply to what would be class methods;
124
      for example <code>gdp_gin_create(...)</code> would map to a class method
125
      <code>gdp_gin.create(...)</code>.</p>
126
    <br>
127
    <h2> 2&nbsp; Theory of Operation </h2>
128
    <p> GDP-based applications rely on three major pieces: an in-process GDP
129
      library, a GDP Log Daemon, and the Routing Layer.&nbsp; In the future
130
      there will also be a service layer, but to date that is ad hoc at
131
      best.&nbsp; This document describes the GDP library API. </p>
132
    <p> The primary point of the GDP library is to speak the network protocol
133
      between the application and the GDP Daemon. The library is threaded, with
134
      (at the moment) at least two threads: one to process events (data arriving
135
      from the daemon, although others can be added), and the other to run the
136
      application itself. This allows the application to pretend it is a
137
      sequential program while still allowing asynchronous input from the GDP
138
      Daemon (e.g., processing results from subscription requests).&nbsp;
139
      Applications are free to create other threads as desired.&nbsp; The code
140
      has been written to do the locking as necessary, but stress tests have not
141
      been run, so you may find unhappy results. </p>
142
    <p> The primary abstraction is the GDP Object (GOB). A GOB represents the
143
      rendezvous point for communication in the data plane. It is not directly
144
      tied to either a file or a network connection. On creation, a GOB is
145
      assigned a 256-bit opaque name. A GOB is append-only to writers.&nbsp; For
146
      the moment you can access the dataplane in one of two modes: synchronous
147
      mode (e.g., using <code>gdp_gin_read_by_*</code> for reading) or an
148
      asynchronous mode (e.g., using <code>gdp_gin_read_by_*_async</code> or <code>gdp_gin_subscribe</code>
149
      for reading). In asynchronous mode the original call return status applies
150
      to the sending of the command, while final results (including any read
151
      data) will be returned using callbacks or an event interface.&nbsp; These
152
      are described in more detail below. </p>
153
    <p>All GOBs are named with an opaque, location independent, 256-bit number
154
      from a flat namespace.&nbsp; When printed it is shown as a base64-encoded
155
      value that is tweaked to be valid in a URI (that is, "+" and "/" are
156
      replaced with "&ndash;" and "_").&nbsp; Applications may choose to overlay
157
      these unsightly names with some sort of directory service.&nbsp; <span class="meta">Such
158
        a directory service is planned but not yet implemented.</span></p>
159
    <p>Applications using the GDP library should <code>#include
160
        &lt;gdp/gdp.h&gt;</code> for all the essential definitions.</p>
161
    <br>
162
    <br>
163
    <h2>3&nbsp; GDP Operations</h2>
164
    <h3>3.1 Data Types and Initialization</h3>
165
    <hr>
166
    <h4>Name</h4>
167
    <p>GDP data types and basic utilities</p>
168
    <h4>Synopsis</h4>
169
    <pre>#include &lt;gdp/gdp.h&gt;<br><br>// names: internal and printable<br>gdp_name_t        InternalGdpName;        // 256-bit number<br>gdp_pname_t        PrintableGdpName;        // base-64 encoded string<br>bool                GDP_NAME_SAME(gdp_name_t a, gdp_name_t b);<br>bool                gdp_name_is_valid(gdp_name_t gdpname);<br><br>// convert between printable and internal names<br>char                *gdp_printable_name(const gdp_name_t InternalFormat,<br>                        gdp_pname_t PrintableFormat);<br>EP_STAT                gdp_internal_name(const gdp_pname_t PrintableFormat,<br>                        gdp_name_t Internal);<br><br>// this will do human-friendly name lookup (may be expensive)<br>EP_STAT                gdp_parse_name(const char *external,<br>                        gdp_name_t Internal);<br><br>gdp_gin_t        *GdpLogInstance;<br><br>gdp_datum_t        *GdpDatum;                // data storage unit<br>gdp_hash_t      *GdpHash;                // hash function over a gdp_datum_t<br>gdp_sig_t        *GdpSignature;                // signature (multiple usage)<br>gdp_recno_t        RecordNumber;                // datum record number<br>EP_TIME_SPEC        TimeStampSpec;                // timestamp: see libep</pre>
170
    <h4>Notes</h4>
171
    <ul>
172
      <li>Most of these are described in more detail below.</li>
173
    </ul>
174
    <br>
175
    <hr width="100%" size="2">
176
    <h4> Name</h4>
177
    gdp_init &mdash; Initialize the GDP library
178
    <h4> Synopsis</h4>
179
    <pre>#include &lt;gdp/gdp.h&gt;<br><br>EP_STAT gdp_init(const char *gdpd_addr)
180
</pre>
181
    <h4> Notes</h4>
182
    <ul>
183
      <li> Initializes the GDP library.&nbsp; <em><strong>Must</strong></em> be
184
        called before any other GDP functions are invoked.</li>
185
      <li>The <code>gdpd_addr</code> parameter is the address to use to contact
186
        the GDP routing layer in the format "host:port".&nbsp; If <code>NULL</code>
187
        a system default is used. </li>
188
      <li>If the status is not <code>EP_STAT_OK</code> then the library failed
189
        to initialize (for example, by being unable to acquire resources.&nbsp;
190
        Failure to check this status (generally using the <code class="function">EP_STAT_ISOK</code>
191
        predicate) may result in mysterious failures later.</li>
192
      <li>All source files using the GDP must include <code>&lt;gdp/gdp.h&gt;</code>.
193
      </li>
194
    </ul>
195
    <hr width="100%" size="2">
196
    <h4>Name</h4>
197
    <p>GDP_LIB_VERSION &mdash; GDP library version</p>
198
    <h4>Synopsis</h4>
199
    <pre>#include &lt;gdp/gdp_version.h&gt;</pre>
200
    <h4>Notes</h4>
201
    <ul>
202
      <li>The <code>gdp_version.h</code> file defines the integer constant <code>GDP_LIB_VERSION</code>
203
        as the major, minor, and patch level of this version of the GDP library,
204
        for example, 0x010203 for version 1.2.3.&nbsp; It can be used during
205
        compilation.&nbsp; There is also a string <code>GdpVersion</code> that
206
        is suitable for printing.</li>
207
    </ul>
208
    <hr>
209
    <h3>3.2&nbsp; GOB Management</h3>
210
    <p>This section covers operations on GOBs as a whole, notably creating,
211
      opening, and closing.</p>
212
    <br>
213
    <hr width="100%" size="2">
214
    <h4> Name</h4>
215
    gdp_gin_create &mdash; Create an append-only GOB, returning a GIN<span class="warning"></span>
216
    <h4> Synopsis</h4>
217
    <pre>EP_STAT gdp_gin_create(gdp_create_info_t *gci,<br>                const char *human_name,<br>                gdp_gin_t **ginp)
218
</pre>
219
    <h4> Notes</h4>
220
    <ul>
221
      <li> Creates a GOB with the given human-oriented name.&nbsp; The
222
        human-oriented name is a text string which is stored in the metadata and
223
        inserted into the Human-Oriented Name to GDPname Directory by the
224
        creation service.</li>
225
      <li>Metadata and other details are provided by the indicated creation
226
        information, which may be <code>NULL</code>.</li>
227
      <li>Arguably should be called <code>gdp_gob_create</code> since it
228
        creates a GOB, but it returns a GIN.</li>
229
      <li>Unless otherwise arranged via the <code>gci</code> parameter, a
230
        keypair will be created to be associated with this log.</li>
231
      <li> Returns a GOB instance (indirectly through <code>*ginp</code>).</li>
232
      <li> The returned GIN handle is an append-only object.&nbsp; <i>(Actually,
233

    
234

    
235

    
236

    
237

    
238

    
239

    
240
          this is not enforced at this time).</i></li>
241
    </ul>
242
    <br>
243
    <hr>
244
    <h4>Name</h4>
245
    <p>gdp_create_info_new &mdash; allocate a new creation information data
246
      structure</p>
247
    <h4>Synopsis</h4>
248
    <pre>gdp_create_info_t *gdp_create_info_new(void)</pre>
249
    <h4>Notes</h4>
250
    <ul>
251
      <li>Returns a new creation information data structure to be used in the
252
        following routines.</li>
253
    </ul>
254
    <hr>
255
    <h4>Name</h4>
256
    gdp_create_info_free &mdash; Free the creation information structure<br>
257
    <h4>Synopsis</h4>
258
    <pre>void gdp_create_info_free(gdp_create_info_t *info)</pre>
259
    <h4>Notes</h4>
260
    <ul>
261
      <li>Frees the data structure pointed to by <code>info</code>.</li>
262
      <li>This can be freed as soon as it has been used in <code>gdp_create</code>
263
        or re-used to create multiple logs.</li>
264
    </ul>
265
    <hr>
266
    <h4>Name</h4>
267
    gdp_create_info_set_owner_key, gdp_create_info_set_writer_key &mdash; Set
268
    owner/writer key for a new GOB<br>
269
    <h4>Synopsis</h4>
270
    <pre>EP_STAT gdp_create_info_set_owner_key(<br>                gdp_create_info_t *info,
271
                EP_CRYPTO_KEY *skey,
272
                const char *dig_alg_name)
273

    
274

    
275
EP_STAT gdp_create_info_set_writer_key(
276
                gdp_create_info_t *info,
277
                EP_CRYPTO_KEY *skey,
278
                const char *dig_alg_name)</pre>
279
    <h4>Notes</h4>
280
    <ul>
281
      <li>Sets the owner or writer key to be used for a new GOB.</li>
282
      <li>The <code>dig_alg_name</code> parameter specifies the message digest
283
        algorithm to be used when signing.</li>
284
    </ul>
285
    <hr>
286
    <p></p>
287
    <ul>
288
    </ul>
289
    <h4>Name</h4>
290
    gdp_create_info_new_owner_key, gdp_create_info_new_writer_key &mdash; Create
291
    new owner/writer key for a new GOB<br>
292
    <h4>Synopsis</h4>
293
    <pre>EP_STAT gdp_create_info_new_owner_key(<br>&nbsp;&nbsp;&nbsp;             gdp_create_info_t *info,<br>&nbsp;&nbsp;&nbsp;             const char *dig_alg_name,<br>                const char *key_alg_name,<br>                int key_bits,<br>                const char *curve_name,<br>                const char *key_enc_alg_name)
294

    
295

    
296
EP_STAT gdp_create_info_new_writer_key(
297
&nbsp;&nbsp;&nbsp;             gdp_create_info_t *info,<br>&nbsp;&nbsp;&nbsp;             const char *dig_alg_name,<br>                const char *key_alg_name,<br>                int key_bits,<br>                const char *curve_name,<br>                const char *key_enc_alg_name)
298
</pre>
299
    <h4>Notes</h4>
300
    <ul>
301
      <li>Creates a new owner or writer key to be used for a new GOB.</li>
302
      <li>The <code>dig_alg_name</code> parameter specifies the message digest
303
        algorithm to be used when signing.</li>
304
      <li>The <code>key_alg_name</code>, <code>key_bits</code>, and <code>curve_name</code>
305
        parameters specify the signing algorithm.</li>
306
      <li>The <code>key_enc_alg_name</code> parameter specifies the symmetric
307
        encryption algorithm to be used when the key is written to disk.</li>
308
      <li>If any of the parameters are <code>NULL</code> (or zero, in the case
309
        of <code>key_bits</code>), a default is used; these defaults can be set
310
        using administrative parameters.</li>
311
    </ul>
312
    <hr>
313
    <h4>Name</h4>
314
    <p>gdp_create_info_set_creator &mdash; set name and domain of entity
315
      creating this GOB</p>
316
    <h4>Synopsis</h4>
317
    <pre>EP_STAT gdp_create_info_set_creator(
318
&nbsp;&nbsp;&nbsp;             gdp_create_info_t *info,<br>&nbsp;&nbsp;&nbsp;             const char *user,<br>                const char *domain)</pre>
319
    <p></p>
320
    <h4>Notes</h4>
321
    <ul>
322
      <li>Sets the name and/or domain of the creator of this GOB.</li>
323
      <li>The full creator name is set to <code>user</code>@<code>domain</code>.</li>
324
      <li>If only the <code>user</code> is specified, the <code>domain</code>
325
        is set to the DNS domain name of the current host.</li>
326
      <li>If only the <code>domain</code> is specified, the <code>user</code>
327
        is set to the currently logged in user.</li>
328
      <li>If user contains an "<code>@</code>" sign and no <code>domain</code>
329
        is specified, the <code>user</code> is used directly as the creator.</li>
330
      <li>The creator name is included in the log metadata for informational
331
        purposes; it has no direct impact on the semantics of the GOB.</li>
332
    </ul>
333
    <hr>
334
    <h4>Name</h4>
335
    <p>gdp_create_info_set_creation_service &mdash; set the name of the GDP
336
      creation service</p>
337
    <h4>Synopsis</h4>
338
    <pre>EP_STAT gdp_create_info_set_creation_service(
339
&nbsp;&nbsp;&nbsp;             gdp_create_info_t *info,<br>&nbsp;&nbsp;&nbsp;             const char *creation_service_name)</pre>
340
    <p></p>
341
    <h4>Notes</h4>
342
    <ul>
343
      <li>Sets the name of the creation service.&nbsp; Use of this should be
344
        rare.</li>
345
      <li>Currently defaults to a system-wide default.</li>
346
      <li>In the future it will probably try with Zeroconf.</li>
347
    </ul>
348
    <p></p>
349
    <ul>
350
    </ul>
351
    <hr>
352
    <h4>Name</h4>
353
    <p>gdp_create_info_set_expiration &mdash; set GOB expiration parameters</p>
354
    <h4>Synopsis</h4>
355
    <pre>EP_STAT gdp_create_info_add_expiration(<br>&nbsp;&nbsp;&nbsp;             gdp_create_info_t *info,<br>                &lt;to be determined&gt;)</pre>
356
    <p></p>
357
    <h4>Notes</h4>
358
    <ul>
359
      <li class="warning">It still isn't clear how this is going to work.&nbsp;
360
        For the time being it should not be used.&nbsp; The parameters will
361
        probably change.</li>
362
    </ul>
363
    <ul>
364
    </ul>
365
    <hr>
366
    <h4>Name</h4>
367
    <p>gdp_create_info_add_metadata &mdash; add user-defined metadata to the GOB
368
      metadata</p>
369
    <h4>Synopsis</h4>
370
    <pre>EP_STAT gdp_create_info_add_metadata(
371
&nbsp;&nbsp;&nbsp;             gdp_create_info_t *info,<br>&nbsp;&nbsp;&nbsp;             uint32_t md_name,<br>                size_t md_len,<br>                const char *md_val)</pre>
372
    <p></p>
373
    <h4>Notes</h4>
374
    <ul>
375
      <li>Adds the indicated value to the log metadata.</li>
376
      <li>Metadata names are 32-bit unsigned numbers.&nbsp; Values where the
377
        high-order byte is zero are reserved for system-defined use.</li>
378
      <li>Conventionally metadata names are actually ASCII characters encoded
379
        into a four-byte integer.&nbsp; For example, the name <code>0x44454d4f</code>
380
        could also be read as "<code>DEMO</code>".</li>
381
      <li>Metadata values are arbitrary binary strings up to 255 bytes in
382
        length.</li>
383
    </ul>
384
    <ul>
385
    </ul>
386
    <hr width="100%" size="2">
387
    <h4> Name</h4>
388
    gdp_gin_open &mdash; Open an existing GOB, returning a GIN<br>
389
    <h4> Synopsis</h4>
390
    <pre>EP_STAT gdp_gin_open(gdp_name_t name,
391
                gdp_iomode_t rw,
392
                gdp_open_info_t *info,
393
                gdp_gin_t **ginp)
394
</pre>
395
    <h4> Notes</h4>
396
    <ul>
397
      <li> Opens the GOB with the indicated name for the mode indicated by <code>rw</code>,
398
        which may be <code>GDP_MODE_RO</code> (read only), <code>GDP_MODE_AO</code>
399
        (append only), or <code>GDP_MODE_RA</code> (read and append).&nbsp; It
400
        returns an instance of that object.</li>
401
      <li>Open information is to pass in detailed information needed for the
402
        open as described below.&nbsp; In most cases it can be passed as
403
        NULL.&nbsp; Eventually it will be used to convey signing keys, quality
404
        of service requests, payment information, authorization tokens, etc.</li>
405
      <li>The handle itself is returned indirectly through <code>*ginp</code>.</li>
406
      <li>If a signing key associated with the GOB is found, all writes to this
407
        GOB will automatically be signed.&nbsp; See the section on Signing and
408
        Encryption below for details.</li>
409
    </ul>
410
    <hr>
411
    <h4>Name</h4>
412
    gdp_open_info_new &mdash; Create a new open information data structure<br>
413
    <h4>Synopsis</h4>
414
    <pre>gdp_open_info_t *gdp_open_info_new(void)</pre>
415
    <h4>Notes</h4>
416
    <ul>
417
      <li>Creates a new data structure to be used in the following routines.</li>
418
    </ul>
419
    <hr>
420
    <h4>Name</h4>
421
    gdp_open_info_free &mdash; Free the open information structure<br>
422
    <h4>Synopsis</h4>
423
    <pre>void gdp_open_info_free(gdp_open_info_t *info)</pre>
424
    <h4>Notes</h4>
425
    <ul>
426
      <li>Frees the data structure pointed to by <code>info</code>.</li>
427
      <li>This can be freed as soon as it has been used in <code>gdp_open</code>.</li>
428
    </ul>
429
    <hr>
430
    <h4>Name</h4>
431
    gdp_open_info_set_signing_key &mdash; Set signing key for an open GOB<br>
432
    <h4>Synopsis</h4>
433
    <pre>EP_STAT gdp_open_info_set_signing_key(<br>&nbsp;&nbsp;&nbsp;             gdp_open_info_t *info,<br>&nbsp;&nbsp;&nbsp;             EP_CRYPTO_KEY *skey)</pre>
434
    <h4>Notes</h4>
435
    <ul>
436
      <li>Sets the signing key to be used when appending data to a log.</li>
437
      <li>If no signing key is specified, the library attempts to find the key
438
        using a search path.</li>
439
      <li>If the open <code>info</code> passed to <code>gdp_open</code> for a
440
        GOB that has already been opened using a different key (that is, on a
441
        different instance), the result is undefined.</li>
442
    </ul>
443
    <hr>
444
    <h4>Name</h4>
445
    <p>gdp_open_info_set_signkey_cb &mdash; Set a callback function to read a
446
      signing key</p>
447
    <h4>Synopsis</h4>
448
    <pre>EP_STAT gdp_open_info_set_signkey_cb(<br>                gdp_open_info_t *info,<br>                EP_STAT (*signkey_cb)(<br>                        gdp_name_t gname,<br>                        void *signkey_udata,<br>                        EP_CRYPTO_KEY **skey),<br>                void *signkey_udata)</pre>
449
    <h4>Notes</h4>
450
    <ul>
451
      <li>If the <code>gdp_open</code> call requires a secret key, that that
452
        key was not passed in using <code>gdp_open_info_set_signing_key</code>,
453
        the callback function <code>signkey_cb</code> is invoked to get a
454
        key.&nbsp; It will only be invoked if the key is required (notably
455
        because it isn't already cached).</li>
456
      <li>The arbitrary data pointer <code>signkey_udata</code> is passed
457
        through to <code>signkey_cb</code> if it is invoked.</li>
458
    </ul>
459
    <hr>
460
    <h4>Name</h4>
461
    gdp_open_info_set_caching &mdash; set caching behavior<br>
462
    <h4>Synopsis</h4>
463
    <pre>EP_STAT gdp_open_info_set_caching(<br>&nbsp;&nbsp;            &nbsp; gdp_open_info_t *info,<br>&nbsp;&nbsp;            &nbsp; bool keep_in_cache)</pre>
464
    <h4>Notes</h4>
465
    <ul>
466
      <li>Sets whether this GOB should be kept in the cache after <code>gdp_gin_close</code>
467
        is called.</li>
468
      <li>If set, cached information will be reclaimed based on the last usage
469
        time of the GOB.</li>
470
      <li>Use of this call with <code>keep_in_cache</code> set to <code>TRUE</code>
471
        may cause a cleanup thread to be spawned.</li>
472
    </ul>
473
    <hr>
474
    <h4>Name</h4>
475
    <p>gdp_open_info_set_vrfy &mdash; set log verification behavior</p>
476
    <h4>Synopsis</h4>
477
    <pre>EP_STAT gdp_open_info_set_vrfy(<br>&nbsp;&nbsp;            &nbsp; gdp_open_info_t *info,<br>&nbsp;&nbsp;            &nbsp; bool do_verification)</pre>
478
    <p></p>
479
    <h4>Notes</h4>
480
    <ul>
481
      <li>Sets whether data retrieved from a log server should be verified.</li>
482
      <li>The computational cost of verifying additional signatures may
483
        negatively affect performance, but it gives you important security
484
        protection.</li>
485
      <li>Defaults to <code>false</code>.</li>
486
    </ul>
487
    <ul>
488
    </ul>
489
    <hr width="100%" size="2">
490
    <h4> Name</h4>
491
    gdp_gin_close &mdash; Close a GDP Instance (GIN) and release resources
492
    <h4> Synopsis</h4>
493
    <pre>EP_STAT gdp_gin_close(gdp_gin_t *gin)
494
</pre>
495
    <h4> Notes</h4>
496
    <ul>
497
      <li>Closes an open GDP Instance.</li>
498
      <li>If this is the last open instance for a GOB in this process, sends a
499
        hint to the log daemon that the associated resources are no longer
500
        referenced.</li>
501
      <li>Releases the client-side resources (that is, memory) for this GIN.</li>
502
      <li><i>Should this interface say whether to preserve or drop the GIN
503
          (i.e., is it persistent, or for how long)?</i></li>
504
    </ul>
505
    <hr width="100%" size="2">
506
    <h3>3.3&nbsp; Synchronous Operations</h3>
507
    <p>Synchronous operations block until the operation is complete.&nbsp; They
508
      are the easiest interface for simple programs, but may not perform as well
509
      as the asynchronous versions.&nbsp; The synchronous calls only read or
510
      write single records at a time; to operate on many records in one call,
511
      use the asynchronous versions.</p>
512
    <p>If synchronous operations do not receive an acknowledgement, they will
513
      attempt to re-send the request after a timeout.</p>
514
    <hr>
515
    <h4>Name</h4>
516
    gdp_gin_append &mdash; Append a record to a GOB
517
    <h4> Synopsis</h4>
518
    <pre>EP_STAT gdp_gin_append(gdp_gin_t *gin,
519
                gdp_datum_t *datum,<br>                gdp_hash_t *prevhash)
520
</pre>
521
    <h4> Notes</h4>
522
    <ul>
523
      <li> Appends the indicated datum to the GOB.</li>
524
      <li>Any subscribers get immediate updates about the new datum.</li>
525
      <li>If a secret key is available for this GOB, the appends will be signed.</li>
526
      <li>The <code>prevhash</code> parameter should be the hash of the
527
        previous record.&nbsp; It is required to provide GDP security
528
        guarantees.&nbsp; If <code>prevhash</code> is NULL, the GDP library
529
        will attempt to fill in the appropriate value.&nbsp; <span class="meta">Someday.</span></li>
530
      <li>Note that when this returns the datum will still be filled in
531
        (including the signature).&nbsp; To re-use the same datum, the
532
        application <strong>must</strong> use <code>gdp_datum_reset</code> to
533
        clear the old information before adding new data.</li>
534
    </ul>
535
    <br>
536
    <hr width="100%" size="2">
537
    <h4> Name</h4>
538
    gdp_gin_read_by_recno, gdp_gin_read_by_ts, gdp_gin_read_by_hash &mdash; Read
539
    from a readable GIN
540
    <h4> Synopsis</h4>
541
    <pre>EP_STAT gdp_gin_read_by_recno(gdp_gin_t *gin,
542
                gdp_recno_t recno,
543
                gdp_datum_t *datum)<br>EP_STAT gdp_gin_read_by_ts(gdp_gin_t *gin,
544
                EP_TIME_SPEC *ts,
545
                gdp_datum_t *datum)<br>EP_STAT gdp_gin_read_by_hash(gdp_gin_t *gin,<br>                gdp_hash_t *hash,<br>                gdp_datum_t *datum)<br> </pre>
546
    <h4> Notes</h4>
547
    <ul>
548
      <li> These present a message-oriented interface.</li>
549
      <li>The user provides the space in which to store the result (see section
550
        4).</li>
551
      <li> An OK stat includes the number of octets actually read. (This is
552
        passed back in datum, so unneeded here.)</li>
553
      <li><code>gdp_gin_read_by_recno</code> reads the specified record number
554
        (sequential, starting from 1).&nbsp; Negative <code>recno</code>s are
555
        interpreted relative to the end of the log (so the value &ndash;1
556
        indicates the last message in the log).</li>
557
      <li><code>gdp_gin_read_by_ts</code> reads the record dated on or
558
        immediately after the indicated timestamp.</li>
559
      <li><code>gdp_gin_read_by_hash</code> returns the record with the
560
        indicated hash.</li>
561
    </ul>
562
    <ul>
563
    </ul>
564
    <hr width="100%" size="2">
565
    <h3>3.4&nbsp; Asynchronous Operations (Asynchronous I/O, Subscriptions, and
566
      Events) </h3>
567
    <p>Asynchronous operations allow an application to subscribe to one or more
568
      GOBs and receive events as those GOBs see activity.&nbsp; The event
569
      mechanism is intended to be extensible for possible future expansion. </p>
570
    <p>Every event has a type, a pointer to the GIN handle, and a pointer to a
571
      datum.&nbsp; Applications could in principle define their own event types,
572
      but at the moment this functionality is not exposed.</p>
573
    <p>All asynchronous operations return status and/or data via either a
574
      callback function or the event interface.&nbsp; Callback functions may not
575
      be called in the same thread as the operation initiation.&nbsp; If no
576
      callback function is given then the event interface is used; this has the
577
      effect of serializing the event stream.&nbsp; In either case, it is the
578
      responsibility of the caller to free the event after use using <code>gdp_event_free</code>.</p>
579
    <p>Note that asynchronous calls do not do retransmissions.</p>
580
    <br>
581
    <hr>
582
    <h4>Name</h4>
583
    <p>gdp_event_t &mdash; event structure</p>
584
    <h4>Synopsis</h4>
585
    <p><code>typedef struct _gdp_event&nbsp;&nbsp;&nbsp; gdp_event_t;</code></p>
586
    <h4>Notes</h4>
587
    <ul>
588
      <li>This is an opaque type.</li>
589
    </ul>
590
    <hr>
591
    <h4>Name</h4>
592
    <p>gdp_event_cbfunc_t &mdash; event callback function type</p>
593
    <h4>Synopsis</h4>
594
    <p><code>typedef void (*gdp_event_cbfunc_t)(gdp_event_t *gev);</code></p>
595
    <h4>Notes</h4>
596
    <ul>
597
      <li>This is the type of callback function as passed into the asynchronous
598
        interfaces.</li>
599
      <li>All the interesting data is encoded into the event.&nbsp; This is
600
        exactly the same data structure as returned by <code>gdp_event_next</code>.</li>
601
      <li>In all cases, if the callback function is not specified the
602
        information will be returned through the event interface.&nbsp; The two
603
        interfaces are related but mutually exclusive.</li>
604
    </ul>
605
    <br>
606
    <hr>
607
    <h4>Name</h4>
608
    gdp_gin_read_by_recno_async, gdp_gin_read_by_ts_async,
609
    gdp_gin_read_by_hash_async &mdash; Asynchronously read records from a
610
    readable GOB
611
    <h4> Synopsis</h4>
612
    <pre>typedef void (*gdp_event_cbfunc_t)(gdp_event_t *gev)
613
<br>EP_STAT gdp_gin_read_by_recno_async(<br>                gdp_gin_t *gin,
614
                gdp_recno_t start,<br>                int32_t numrecs,<br>                gdp_event_cbfunc_t cbfunc,<br>                void *udata)<br>EP_STAT gdp_gin_read_by_ts_async(<br>                gdp_gin_t *gin,
615
                EP_TIME_SPEC *start,<br>                int32_t numrecs,<br>                gdp_event_cbfunc_t cbfunc,<br>                void *udata)<br>EP_STAT gdp_gin_read_by_hash_async(<br>                gdp_gin_t *gin,<br>                int32_t n_hashes,<br>                gdp_hash_t **hashes,<br>                gdp_event_cbfunc_t cbfunc,<br>                void *udata)
616
</pre>
617
    <h4> Notes</h4>
618
    <ul>
619
      <li>Initializes an asynchronous read of data.</li>
620
      <li>These functions return before any result is read and thus do not
621
        include the final status information.</li>
622
      <li>The return status will be <code>GDP_STAT_OK</code> if the read
623
        command is successfully sent, and a later callback or event will give
624
        the actual status; otherwise no callback or event will occur.</li>
625
      <li>Status is returned through the event interface (if <code>cbfunc</code>
626
        is <code>NULL</code>) or through <code>cbfunc</code>.</li>
627
      <li>Similar to a subscription, except data in the future is never read
628
        (i.e., this is only for reading historic data).&nbsp; This is the
629
        interface to use for asynchronous reads.</li>
630
      <li>If a <code>cbfunc</code> is specified, arranges to call callback when
631
        a message is generated on the <code>gin</code>.&nbsp; See below for the
632
        definition of <code>gdp_event_t</code>. </li>
633
      <li> The callback is not necessarily invoked instantly, and may or may not
634
        be called in a separate thread.</li>
635
      <li>It is the responsibility of the callback function to call <code>gdp_event_free(gev)</code>.<br>
636
      </li>
637
      <li>If no <code>cbfunc</code> is specified, subscription information is
638
        available through the <code>gdp_event</code> interface (see below).</li>
639
      <li> The <code>udata</code> is passed through untouched in generated
640
        events.&nbsp; See below for the definition of <code>gdp_event_t</code>.</li>
641
      <li>At most <code>numrecs</code> records are returned, after which the
642
        read is terminated.&nbsp; If <code>numrecs</code> is 0 it reads to the
643
        end of the data.&nbsp; Since hash values must be unique and are
644
        unordered, <code>gdp_gin_read_by_hash_async</code> always returns at
645
        most one record.</li>
646
      <li> The <code>start</code> parameter tells when to start the
647
        subscription (that is, the starting record number or timestamp).</li>
648
      <li>For <code>gdp_gin_read_by_recno_async</code>, if <code>start</code>
649
        is negative, returns the most recent &ndash;<code>start</code>
650
        records.&nbsp; If a negative <code>start</code> indicates going back
651
        more records than are available, it starts from the first record.&nbsp;
652
        If <code>start</code> is zero, an error is returned. </li>
653
      <li>If <code>start</code> specifies an existing record but there are
654
        fewer than <code>numrecs</code> records available, only the existing
655
        records are returned.</li>
656
      <li>If there are multiple records matching a <code>recno</code> or <code>timestamp</code>,
657
        all of them are returned.</li>
658
      <li><em>Callbacks make binding to languages like Java particularly
659
          difficult</em><em>, but fit more naturally in with languages such as
660
          Javascript.</em><em>&nbsp; Note also that callbacks will generally run
661
          in the GDP I/O thread (so no further GDP operations will run until the
662
          callback completes) or in a separate thread (in which case several
663
          instances of the callback may be run at once).</em></li>
664
      <li>Example for <code>gdp_gin_read_by_recno_async</code>:&nbsp; suppose
665
        there are 20 records already in a GOB.&nbsp; Then:</li>
666
    </ul>
667
    <table cellspacing="2" cellpadding="2" border="1" align="center" width="80%">
668
      <tbody>
669
        <tr>
670
          <td valign="top">start </td>
671
          <td valign="top">numrecs </td>
672
          <td valign="top">Behavior </td>
673
        </tr>
674
        <tr>
675
          <td valign="top">1 </td>
676
          <td valign="top">10 </td>
677
          <td valign="top">Returns records 1&ndash;10 immediately and terminates
678
            the read. </td>
679
        </tr>
680
        <tr>
681
          <td valign="top">&ndash;10 </td>
682
          <td valign="top">10 </td>
683
          <td valign="top">Returns records 11&ndash;20 immediately and
684
            terminates the read. </td>
685
        </tr>
686
        <tr>
687
          <td valign="top">0 </td>
688
          <td valign="top">any </td>
689
          <td valign="top">Returns "4.02 bad option" failure. </td>
690
        </tr>
691
        <tr>
692
          <td valign="top">&ndash;10 </td>
693
          <td valign="top">20 </td>
694
          <td valign="top">Returns records 11&ndash;20. </td>
695
        </tr>
696
        <tr>
697
          <td valign="top">1 </td>
698
          <td valign="top">0</td>
699
          <td valign="top">Returns records 1&ndash;20 immediately. </td>
700
        </tr>
701
        <tr>
702
          <td valign="top">&ndash;30 </td>
703
          <td valign="top">30 </td>
704
          <td valign="top">Returns records 1&ndash;20 immediately. </td>
705
        </tr>
706
        <tr>
707
          <td valign="top">30 </td>
708
          <td valign="top">10 </td>
709
          <td valign="top"><i>Currently undefined.&nbsp; Should probably wait
710
              until 10 more records are added before starting to return the
711
              data.</i> </td>
712
        </tr>
713
        <tr>
714
          <td valign="top">any </td>
715
          <td valign="top">&ndash;1 </td>
716
          <td valign="top">Returns "4.02 bad option" failure. </td>
717
        </tr>
718
      </tbody>
719
    </table>
720
    <br>
721
    <br>
722
    <br>
723
    <hr width="100%" size="2">
724
    <h4> Name</h4>
725
    <p>gdp_gin_append_async &mdash; Asynchronously append one or more records to
726
      a writable GOB</p>
727
    <h4>Synopsis</h4>
728
    <pre>EP_STAT gdp_gin_append_async(<br>                gdp_gin_t *gin,<br>                int32_t n_datums<br>                gdp_datum_t **datums,<br>                gdp_hash_t *prevhash,<br>                gdp_event_cbfunc_t *cbfunc,<br>                void *udata)</pre>
729
    <h4>Notes</h4>
730
    <ul>
731
      <li>Appends the indicated <code>datum</code> to the GOB.</li>
732
      <li>This function returns before any result is read and thus does not
733
        include the final status information.</li>
734
      <li>The return status will be <code>GDP_STAT_OK</code> if the append
735
        command is successfully sent, and a later callback or event will give
736
        the actual status; otherwise no callback or event will occur.</li>
737
      <li><em>Should a warning status be returned to make it clear that a status
738
          will be returned later?</em></li>
739
      <li>Status is returned through the event interface (if <code>cbfunc</code>
740
        is <code>NULL</code>) or through <code>cbfunc</code> with an event
741
        type of <code>GDP_EVENT_ASTAT</code>.</li>
742
    </ul>
743
    <hr>
744
    <h4>Name</h4>
745
    gdp_gin_subscribe_by_* &mdash; Subscribe to a readable GOB
746
    <h4> Synopsis</h4>
747
    <pre>EP_STAT gdp_gin_subscribe_by_recno(<br>                gdp_gin_t *gin,
748
                gdp_recno_t start,<br>                int32_t numrecs,<br>                gdp_sub_qos_t *qos;<br>                gdp_event_cbfunc_t *cbfunc,<br>                void *udata)<br>EP_STAT gdp_gin_subscribe_by_ts(<br>                gdp_gin_t *gin,
749
                EP_TIME_SPEC *start,<br>                int32_t numrecs,<br>                gdp_sub_qos_t *qos;<br>                gdp_event_cbfunc_t *cbfunc,<br>                void *udata) </pre>
750
    <h4> Notes</h4>
751
    <ul>
752
      <li>If a <code>cbfunc</code> is specified, arranges to call callback when
753
        a message is generated on the <code>gin</code>.</li>
754
      <li> The callback is not necessarily invoked instantly, and may or may not
755
        be called in a separate thread.</li>
756
      <li>It is the responsibility of the callback function to call <code>gdp_event_free(gev)</code>.
757
      </li>
758
      <li>If qos is specified, it contains quality of service information about
759
        the subscription &mdash; for example, whether subscriptions should be
760
        reliable or best effort (default).</li>
761
      <li>If no <code>cbfunc</code> is specified, subscription information is
762
        available through the <code>gdp_event</code> interface (see below).</li>
763
      <li> The <code>udata</code> is passed through untouched in generated
764
        events.&nbsp; See below for the definition of <code>gdp_event_t</code>.</li>
765
      <li>At most <code>numrecs</code> records are returned, after which the
766
        subscription is terminated.&nbsp; If <code>numrecs</code> is 0 it waits
767
        for data forever. </li>
768
      <li> The <code>start</code> parameter tells when to start the
769
        subscription (that is, the starting record number for <code>gdp_gin_subscribe_by_recno</code>
770
        or the earliest time of interest for <code>gdp_gin_subscribe_by_ts</code>).</li>
771
      <li>In <code>gdp_gin_subscribe_by_recno</code>, if <code>start</code> is
772
        negative, returns the most recent &ndash;<code>start</code>
773
        records.&nbsp; If a negative <code>start</code> indicates going back
774
        more records than are available, it starts from the first record. </li>
775
      <li>If <code>start</code> specifies an existing record but there are
776
        fewer than <code>numrecs</code> records available, this returns the
777
        available records and then waits for the additional data to appear as it
778
        is published.</li>
779
      <li>In <code>gdp_gin_subscribe_by_recno</code>, if <code>start</code> is
780
        zero, or in <code>gdp_gin_subscribe_by_ts</code>, it points past the
781
        last record already in the log, no current records are returned (i.e.,
782
        it returns new records as they are published).</li>
783
      <li><em>Callbacks make binding to languages like Java particularly
784
          difficult, but fit more naturally in with languages such as
785
          Javascript.&nbsp; Note also that callbacks will generally run in the
786
          GDP I/O thread (so no further GDP operations will run until the
787
          callback completes) or in a separate thread (in which case several
788
          instances of the callback may be run at once). </em></li>
789
      <li>Example:&nbsp; suppose there are 20 records already in a GOB.&nbsp;
790
        Then:</li>
791
    </ul>
792
    <table cellspacing="2" cellpadding="2" border="1" align="center" width="80%">
793
      <tbody>
794
        <tr>
795
          <td valign="top">start </td>
796
          <td valign="top">numrecs </td>
797
          <td valign="top">Behavior </td>
798
        </tr>
799
        <tr>
800
          <td valign="top">1 </td>
801
          <td valign="top">10 </td>
802
          <td valign="top">Returns records 1&ndash;10 immediately and terminates
803
            the subscription. </td>
804
        </tr>
805
        <tr>
806
          <td valign="top">&ndash;10 </td>
807
          <td valign="top">10 </td>
808
          <td valign="top">Returns records 11&ndash;20 immediately and
809
            terminates the subscription. </td>
810
        </tr>
811
        <tr>
812
          <td valign="top">0 </td>
813
          <td valign="top">0 </td>
814
          <td valign="top">Starts returning data when record 21 is published and
815
            continues forever. </td>
816
        </tr>
817
        <tr>
818
          <td valign="top">&ndash;10 </td>
819
          <td valign="top">20 </td>
820
          <td valign="top">Returns records 11&ndash;20 immediately, then returns
821
            records 21&ndash;30 as they are published.&nbsp; The subscription
822
            then terminates. </td>
823
        </tr>
824
        <tr>
825
          <td valign="top">1 </td>
826
          <td valign="top">0</td>
827
          <td valign="top">Returns records 1&ndash;20 immediately, then returns
828
            any new records published in perpetuity. </td>
829
        </tr>
830
        <tr>
831
          <td valign="top">&ndash;30 </td>
832
          <td valign="top">30 </td>
833
          <td valign="top">Returns records 1&ndash;20 immediately, then returns
834
            records 21&ndash;30 as they are published. </td>
835
        </tr>
836
        <tr>
837
          <td valign="top">30 </td>
838
          <td valign="top">10 </td>
839
          <td valign="top"><i>Currently undefined.&nbsp; Should probably wait
840
              until 10 more records are added before starting to return the
841
              data.</i> </td>
842
        </tr>
843
        <tr>
844
          <td valign="top">any </td>
845
          <td valign="top">&ndash;1 </td>
846
          <td valign="top">Returns "4.02 bad option" failure. </td>
847
        </tr>
848
      </tbody>
849
    </table>
850
    <br>
851
    <hr width="100%" size="2">
852
    <h4> Name</h4>
853
    <p>gdp_sub_qos_new, gdp_sub_qos_free &mdash; allocate/free subscription
854
      quality of service information</p>
855
    <h4>Synopsis</h4>
856
    <pre>gdp_sub_qos_t *gdp_sub_qos_new(void)</pre>
857
    <pre>void *gdp_sub_qos_free(<br>                gdp_sub_qos_t *qos)</pre>
858
    <h4>Notes</h4>
859
    <ul>
860
      <li class="warning">Tentative &mdash; do not use.</li>
861
    </ul>
862
    <hr>
863
    <h4>Name</h4>
864
    <p>gdp_sub_qos_set_<i>xyzzy</i> &mdash; set <i>xyzzy</i> qos</p>
865
    <h4>Synopsis</h4>
866
    <pre>gdp_sub_qos_set_<code>xyzzy</code>(<br>                gdp_sub_qos_t *qos,<br>                xxx yyy)</pre>
867
    <h4>Notes</h4>
868
    <ul>
869
      <li class="warning">Tentative &mdash; do not use.<br>
870
      </li>
871
    </ul>
872
    <p></p>
873
    <hr>Name gdp_gin_unsubscribe &mdash; Unsubscribe GIN from an associated GOB<span
874
      class="warning"></span>
875
    <h4> Synopsis</h4>
876
    <pre>EP_STAT gdp_gin_unsubscribe(gdp_gin_t *gin,
877
                gdp_event_cbfunc_t *cbfunc,
878
                void *udata)
879
</pre>
880
    <h4> Notes</h4>
881
    <ul>
882
      <li> Deletes all subscriptions matching the given {gin, cbfunc, udata}
883
        tuple.</li>
884
      <li>If <code>cbfunc</code> and or <code>udata</code> is <code>NULL</code>
885
        they are treated as wildcards.&nbsp; For example, "<code>gdp_gin_unsubscribe(gin,
886
          NULL, NULL)</code>" deletes all subscriptions for the given GIN.</li>
887
      <li>If there are multiple subscriptions matching this tuple, they will all
888
        be terminated.&nbsp; For example, this might happen in a multi-threaded
889
        application.</li>
890
      <li><em>(Should subscribe/multiread return a handle to be passed to this
891
          function rather than this interface?)</em></li>
892
    </ul>
893
    <br>
894
    <hr width="100%" size="2">
895
    <h4> Name</h4>
896
    gdp_event_next &mdash; get next asynchronous event
897
    <h4> Synopsis</h4>
898
    <pre>gdp_event_t *gdp_event_next(<br>                gdp_gin_t *gin,<br>                EP_TIME_SPEC *timeout)</pre>
899
    <h4> Notes</h4>
900
    <ul>
901
      <li> Returns the next asynchronous event on the specified GIN.&nbsp; If
902
        gin is NULL, returns the next event from any GIN.</li>
903
      <li>Returns <code>NULL</code> if the <code>timeout</code> expires.&nbsp;
904
        If <code>timeout</code> is <code>NULL</code>, it waits forever. </li>
905
      <li>Currently the only asynchronous events are data arriving as the result
906
        of a subscription.</li>
907
      <li><em>(Should timeout be an absolute time or a delta on the current
908
          time?&nbsp; Currently it is implemented as a delta.)</em></li>
909
    </ul>
910
    <hr width="100%" size="2">
911
    <h4>Name</h4>
912
    gdp_event_gettype &mdash; extract the type from the event
913
    <h4>Synopsis</h4>
914
    <pre>int gdp_event_gettype(gdp_event_t *gev)</pre>
915
    <h4>Notes</h4>
916
    <ul>
917
      <li>The event types are as follows:</li>
918
    </ul>
919
    <table cellspacing="2" cellpadding="2" border="1" align="center" width="80%">
920
      <tbody>
921
        <tr>
922
          <td style="width: 162.2px;" valign="top">Event Name </td>
923
          <td style="width: 486.6px;" valign="top">Meaning </td>
924
        </tr>
925
        <tr>
926
          <td style="text-align: left; vertical-align: top;" valign="top"><code>GDP_EVENT_DATA</code>
927
          </td>
928
          <td style="text-align: left; vertical-align: top;" valign="top">Data
929
            is returned in the event from a previous subscription or
930
            asynchronous read. </td>
931
        </tr>
932
        <tr>
933
          <td style="text-align: left; vertical-align: top;" valign="top"><code>GDP_EVENT_DONE</code>
934
          </td>
935
          <td style="text-align: left; vertical-align: top;" valign="top">Indicates
936
            the end of a subscription or asynchronous read. </td>
937
        </tr>
938
        <tr>
939
          <td style="text-align: left; vertical-align: top;" valign="top"><tt>GDP_EVENT_SHUTDOWN</tt>
940
          </td>
941
          <td style="text-align: left; vertical-align: top;" valign="top">Subscription
942
            is terminated because the log daemon has shut down. </td>
943
        </tr>
944
        <tr>
945
          <td style="text-align: left; vertical-align: top;"><code>GDP_EVENT_CREATED</code></td>
946
          <td style="text-align: left; vertical-align: top;">Status is returned
947
            from an asynchronous append, create, or other similar operation.</td>
948
        </tr>
949
        <tr>
950
          <td style="vertical-align: top; background-color: white;"><code>GDP_EVENT_MISSING</code></td>
951
          <td style="vertical-align: top; background-color: white;">The
952
            requested data was not available at this time, but more data may be
953
            available.</td>
954
        </tr>
955
        <tr>
956
          <td style="text-align: left; vertical-align: top;"><span style="font-family: monospace;">GDP_EVENT_SUCCESS<br>
957
            </span></td>
958
          <td style="text-align: left; vertical-align: top;">Generic
959
            asynchronous success status. See the detailed status using <span style="font-family: monospace;">gdp_event_getstat</span>.</td>
960
        </tr>
961
        <tr>
962
          <td style="text-align: left; vertical-align: top;"><span style="font-family: monospace;">GDP_EVENT_FAILURE</span></td>
963
          <td style="text-align: left; vertical-align: top;">Generic
964
            asynchronous failure status.&nbsp; See the detailed status using <span
965
              style="font-family: monospace;">gdp_event_getstat</span>.</td>
966
        </tr>
967
      </tbody>
968
    </table>
969
    <ul>
970
      <li>Unrecognized events should be ignored. </li>
971
    </ul>
972
    <hr width="100%" size="2">
973
    <h4>Name</h4>
974
    <p>gdp_event_getstat &mdash; extract the detailed result status from the
975
      event</p>
976
    <h4>Synopsis</h4>
977
    <p><code>EP_STAT gdp_event_getstat(gdp_event_t *gev)</code></p>
978
    <h4>Notes</h4>
979
    <ul>
980
      <li>Returns the status code from the event.&nbsp; In most cases this will
981
        be <code>EP_STAT_OK</code>, but may be otherwise in some event types
982
        such as <code>GDP_EVENT_ASTAT</code>.</li>
983
    </ul>
984
    <hr>
985
    <h4>Name</h4>
986
    gdp_event_getgin &mdash; extract the GIN handle from the event
987
    <h4>Synopsis</h4>
988
    <pre>gdp_gin_t *gdp_event_getgin(gdp_event_t *gev)</pre>
989
    <h4>Notes</h4>
990
    <ul>
991
      <li>Returns the GIN handle that triggered this event.</li>
992
    </ul>
993
    <hr width="100%" size="2">
994
    <h4>Name</h4>
995
    gdp_event_getdatum &mdash; get the datum associated with this event
996
    <h4>Synopsis</h4>
997
    <pre>gdp_datum_t *gdp_event_getdatum(gdp_event_t *gev)</pre>
998
    <h4>Notes</h4>
999
    <ul>
1000
      <li>Returns the data associated with the event.</li>
1001
    </ul>
1002
    <hr width="100%" size="2">
1003
    <h4>Name</h4>
1004
    <p>gdp_event_getudata &mdash; get user data associated with this event </p>
1005
    <h4>Synopsis</h4>
1006
    <pre>void *gdp_event_getudata(gdp_event_t *gev)
1007
    </pre>
1008
    <h4>Notes</h4>
1009
    <ul>
1010
      <li>Returns user data associated with the event.&nbsp; The user data is
1011
        from the call that initiated the asynchronous operation. </li>
1012
    </ul>
1013
    <hr>
1014
    <h3> 3.6&nbsp; Utilities </h3>
1015
    <hr width="100%" size="2">
1016
    <h4>Name</h4>
1017
    gdp_name_parse, gdp_printable_name &mdash; parse an external representation
1018
    to internal, create printable version of name<br>
1019
    <h4>Synopsis</h4>
1020
    <pre>EP_STAT gdp_name_parse(<br>                const char *external_name,<br>               &nbsp;gdp_name_t gob_name,<br>                char **extended_name)<br><br>char *gdp_printable_name(const gdp_name_t gob_name,<br>                gdp_pname_t printable)</pre>
1021
    <h4>Notes</h4>
1022
    <ul>
1023
      <li>The <code>gdp_parse_name</code> function converts an external string
1024
        representation of a GOB name to an internal 256-bit encoding.</li>
1025
      <li>If <code>external_name</code> is a URI-base64-encoded string of the
1026
        correct length, it is converted directly to 256 bits.</li>
1027
      <li>Otherwise, it is looked up in the Human-Oriented Name to GDPname
1028
        Directory.</li>
1029
      <li>If the <code>GDP_NAME_ROOT</code> environment variable is set, it can
1030
        be used when looking up the name.&nbsp; If the input name has no dots
1031
        then the root name will be prepended to the name before lookup.&nbsp;
1032
        For example, if <code>GDP_NAME_ROOT=edu.berkeley.eecs.eric</code> and
1033
        the <code>external_name</code> is "<code>test</code>" then <code>gdp_name_parse</code>
1034
        will search for "<code>edu.berkeley.eecs.eric.test</code>" in the
1035
        directory.</li>
1036
      <li>If <code>extended_name</code> is non-<code>NULL</code>, <code>*extended_name</code>
1037
        will be used to store a pointer to the name actually used.&nbsp; This
1038
        will be the same as <code>external_name</code> unless name extension is
1039
        performed.&nbsp; If the status is not <code>EP_STAT_OK</code>, <code>*extended_name</code>
1040
        will be set to <code>NULL</code>.&nbsp; This memory is dynamically
1041
        allocated; the application is responsible for freeing the memory when it
1042
        is done with it.</li>
1043
      <li>The <code>gdp_printable_name</code> function converts an internal
1044
        name to a URI-base64-encoded string &mdash; note, not the original
1045
        human-oriented name.</li>
1046
    </ul>
1047
    <hr>
1048
    <h4>Name</h4>
1049
    <p>gdp_name_root_set, gdp_name_root_get &mdash; set/get the root name used
1050
      by gdp_name_parse</p>
1051
    <h4>Synopsis</h4>
1052
    <pre>EP_STAT gdp_name_root_set(<br>                const char *root_name)<br>const char *gdp_name_root_get(void)</pre>
1053
    <h4>Notes</h4>
1054
    <ul>
1055
      <li>The root name used by <code>gdp_name_parse</code> can be set or
1056
        queried using these routines.</li>
1057
      <li class="warning">It's possible that the name root may be converted to a
1058
        colon-separated path in the future.&nbsp; Caution is advised.</li>
1059
    </ul>
1060
    <ul>
1061
    </ul>
1062
    <hr>
1063
    <h4>Name</h4>
1064
    gdp_gin_getname &mdash; Return the name of a GOB from a GIN
1065
    <h4> Synopsis</h4>
1066
    <pre>EP_STAT gdp_gin_getname(gdp_gin_t *gin,
1067
                gdp_name_t namebuf)
1068
</pre>
1069
    <h4> Notes</h4>
1070
    <ul>
1071
      <li> Returns the internal name of the GOB referenced by <code>gin</code>
1072
        into <code> namebuf</code>. </li>
1073
      <li> May not be necessary if the creator provides the name; really only
1074
        intended after <code>gdp_gin_create</code> so that the name can be
1075
        shared to other nodes that want to <code>gdp_gin_open</code> it.</li>
1076
    </ul>
1077
    <br>
1078
    <hr width="100%" size="2">
1079
    <h4> Name</h4>
1080
    gdp_getstat &mdash; Return information about a GIN&nbsp; <span class="warning">[NOT
1081
      YET IMPLEMENTED] </span>
1082
    <h4> Synopsis</h4>
1083
    <pre>EP_STAT gdp_getstat(gdp_name_t gobname,
1084
                gob_stat_t *statbuf)
1085
</pre>
1086
    <h4> Notes</h4>
1087
    <ul>
1088
      <li> <i>What status is included? Size (or number of records/messages),
1089
          last access, &hellip;</i></li>
1090
    </ul>
1091
    <br>
1092
    <hr>
1093
    <h4>Name</h4>
1094
    <p>gdp_gin_getnrecs &mdash; return the number of records in an existing GOB</p>
1095
    <h4>Synopsis</h4>
1096
    <p><code>gdp_recno_t gdp_gin_getnrecs(gdp_gin_t *gin)</code></p>
1097
    <h4>Notes</h4>
1098
    <ul>
1099
      <li>Returns the number of records in a GOB.</li>
1100
      <li>This does not check with the log server, i.e., the return value is
1101
        cached in the local process (and is updated each time the GOB is read or
1102
        written).&nbsp; This shouldn't be a problem for writers (since there
1103
        should only be one writer), but may be for readers.&nbsp; Readers
1104
        wanting to get the latest number of records can read the last record in
1105
        the GOB (using <code>gdp_gin_read_by_recno</code>) and check the record
1106
        number in the returned datum.</li>
1107
    </ul>
1108
    <hr>
1109
    <h4>Name</h4>
1110
    gdp_gin_gethashalg &mdash; get the hash algorithm used by a GOB
1111
    <h4>Synopsis</h4>
1112
    <pre>    int gdp_gin_gethashalg(const gdp_gin_t *gin)</pre>
1113
    <h4>Notes</h4>
1114
    <ul>
1115
      <li>Returns the type of the hash (a.k.a. message digest) algorithm.</li>
1116
      <li>The values are defined by the libep.&nbsp; They consist of the major
1117
        SHA algorithms in various sizes, e.g., <span class="manifest">EP_CRYPTO_MD_SHA1</span>,
1118
        <span class="manifest">EP_CRYPTO_MD_SHA224</span>, etc. through <span class="manifest">SHA512</span>.</li>
1119
    </ul>
1120
    <hr>
1121
    <h4>Name</h4>
1122
    gdp_gin_getsigalg &mdash; get the signature algorithm used by a GOB
1123
    <h4>Synopsis</h4>
1124
    <pre>    int gdp_gin_getsigalg(const gdp_gin_t *gin)</pre>
1125
    <h4>Notes</h4>
1126
    <ul>
1127
      <li>Returns the type of the signature algorithm.</li>
1128
      <li>The values are defined by the libep.</li>
1129
    </ul>
1130
    <hr>
1131
    <h4>Name</h4>
1132
    <p>gdp_gin_set_append_filter &mdash; filter appended data</p>
1133
    <h4>Synopsis</h4>
1134
    <code>void gdp_gin_set_append_filter(gdp_gin_t *gin,<br>
1135
      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
1136
      EP_STAT (*filter(gdp_datum_t *, void *),<br>
1137
      &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp; void *filterdata)</code><br>
1138
    <h4>Notes</h4>
1139
    <ul>
1140
      <li>All data appended to the named <span class="variable">gin</span> are
1141
        first run through the <span class="variable">filter</span>.&nbsp; The
1142
        filter may modify the <span class="variable">datum</span> before it is
1143
        sent.</li>
1144
      <li>The <span class="variable">filterdata</span> parameter is passed as
1145
        the second argument to <span class="variable">filter</span>.</li>
1146
    </ul>
1147
    <hr>
1148
    <h4>Name</h4>
1149
    <p>gdp_gin_set_read_filter &mdash; filter read data</p>
1150
    <h4>Synopsis</h4>
1151
    <p><code>void gdp_gin_set_read_filter(gdp_gin_t *gin,<br>
1152
        &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
1153
        EP_STAT (*filter(gdp_datum_t *, void *),<br>
1154
        &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp; void *filterdata)</code></p>
1155
    <h4>Notes</h4>
1156
    <ul>
1157
      <li>All data read from the named <span class="variable">gin</span> are
1158
        passed through this filter before being returned.&nbsp; The filter may
1159
        modify the <span class="variable">datum</span>.</li>
1160
      <li>The <span class="variable">filterdata</span> parameter is passed as
1161
        the second argument to <span class="variable">filter</span>.</li>
1162
    </ul>
1163
    <br>
1164
    <hr width="100%" size="2">
1165
    <h4> Name</h4>
1166
    gdp_gin_print &mdash; print a GIN and associated GOB (for debugging)
1167
    <h4>Synopsis</h4>
1168
    <pre>void gdp_gin_print(const gdp_gin_t *gin, FILE *fp)</pre>
1169
    <h4>Notes</h4>
1170
    <ul>
1171
      <li>Prints the GIN and GOB on the indicated file.</li>
1172
      <li>The output will contain internal information; it is not intended to
1173
        display information to end users.</li>
1174
    </ul>
1175
    <hr>
1176
    <h2>4&nbsp; Datums (Records)</h2>
1177
    <p>GOBs are represented as a series of records of type <code>gdp_datum_t</code>.&nbsp;
1178
      Each record has a record number, a commit timestamp, associated data, and
1179
      possible signature information if the record was signed.&nbsp; Record
1180
      numbers are of type <code>gdp_recno_t</code> and count up by one as
1181
      records are added (i.e., record numbers are unique within a GOB and
1182
      dense).&nbsp; Data is represented in dynamic buffers, as described below.
1183
    </p>
1184
    <h3>4.1&nbsp; Datum Headers</h3>
1185
    <hr width="100%" size="2">
1186
    <h4>Name</h4>
1187
    <p>gdp_datum_new, gdp_datum_reset, gdp_datum_free, gdp_datum_print &mdash;
1188
      allocate/free/print a datum structure </p>
1189
    <h4>Synopsis</h4>
1190
    <pre>gdp_datum_t *gdp_datum_new(void)<br>void gdp_datum_reset(gdp_datum_t *datum)<br>void gdp_datum_free(gdp_datum_t *datum)
1191
void gdp_datum_print(const gdp_datum_t *datum,
1192
                FILE *fp,
1193
                uint32_t flags)</pre>
1194
    <h4>Notes</h4>
1195
    <ul>
1196
      <li><code>gdp_datum_new</code> allocates a new empty datum.</li>
1197
      <li><code>gdp_datum_reset</code> resets a datum to the same state as a
1198
        newly created datum.</li>
1199
      <li><code>gdp_datum_free</code> frees a datum.</li>
1200
      <li><code>gdp_datum_print</code> writes a description of the datum
1201
        (including the data contents) to the given file.&nbsp; If flags includes
1202
        the <span class="manifest">GDP_DATUM_PRTEXT</span> bit, it shows the
1203
        datum as plain text (the default shows it as a hex dump).&nbsp; It is up
1204
        to the caller to determine that the datum is printable.&nbsp; If the <span
1205
          class="manifest">GDP_DATUM_PRSIG</span> bit is set, signature
1206
        information is included.&nbsp; If the <span class="manifest">GDP_DATUM_PRDEBUG</span>
1207
        flag is set, additional information about the datum is printed.</li>
1208
    </ul>
1209
    <hr width="100%" size="2">
1210
    <h4>Name</h4>
1211
    gdp_datum_getrecno &mdash; get the record number from a datum
1212
    <h4>Synopsis</h4>
1213
    <pre>    <tt>gdp_recno_t gdp_datum_getrecno(const gdp_datum_t *datum)
1214
</tt></pre>
1215
    <h4>Notes </h4>
1216
    <hr width="100%" size="2">
1217
    <h4>Name</h4>
1218
    gdp_datum_getts &mdash; get the timestamp from a datum
1219
    <h4>Synopsis</h4>
1220
    <pre>    void gdp_datum_getts(const gdp_datum_t *datum, EP_TIME_SPEC *ts)
1221
</pre>
1222
    <h4>Notes</h4>
1223
    <hr width="100%" size="2">
1224
    <h4>Name</h4>
1225
    gdp_datum_getdlen &mdash; get the data length from a datum
1226
    <h4>Synopsis</h4>
1227
    <pre>    size_t gdp_datum_getdlen(const gdp_datum_t *datum)</pre>
1228
    <h4>Notes</h4>
1229
    <hr width="100%" size="2">
1230
    <h4>Name</h4>
1231
    gdp_datum_getdbuf &mdash; get the data buffer from a datum
1232
    <h4>Synopsis</h4>
1233
    <pre>    gdp_buf_t *gdp_datum_getdbuf(const gdp_datum_t *datum)</pre>
1234
    <h4>Notes</h4>
1235
    <hr width="100%" size="2">
1236
    <h4>Name</h4>
1237
    gdp_datum_getsig &mdash; get the signature from a datum
1238
    <h4>Synopsis</h4>
1239
    <pre>    gdp_sig_t *gdp_datum_getsig(const gdp_datum_t *datum)</pre>
1240
    <h4>Notes</h4>
1241
    <ul>
1242
      <li>Can return <span class="manifest">NULL</span> or an empty buffer if
1243
        there is no signature.</li>
1244
    </ul>
1245
    <hr width="100%" size="2">
1246
    <h3>4.2&nbsp; Data Buffers</h3>
1247
    <p>Data buffers grow dynamically as needed. </p>
1248
    <hr width="100%" size="2">
1249
    <h4>Name</h4>
1250
    gdp_buf_new, gdp_buf_reset, gdp_buf_free &mdash; allocate, reset, or free a
1251
    buffer
1252
    <h4>Synopsis</h4>
1253
    <pre>gdp_buf_t *gdp_buf_new(void)<br>void gdp_buf_reset(gdp_buf_t *b)<br>void gdp_buf_free(gdp_buf_t *b)</pre>
1254
    <h4>Notes</h4>
1255
    <ul>
1256
      <li><code>gdp_buf_new</code> creates a new, empty buffer.</li>
1257
      <li><code>gdp_buf_reset</code> clears the buffer, leaving it in the same
1258
        condition as when it was first created.</li>
1259
      <li><code>gdp_buf_free</code> frees the buffer.&nbsp; It must not be used
1260
        again after being freed.</li>
1261
    </ul>
1262
    <hr width="100%" size="2">
1263
    <h4>Name</h4>
1264
    gdp_buf_getlength &mdash; return the length of the data in the buffer
1265
    <h4>Synopsis</h4>
1266
    <pre>size_t gdp_buf_getlength(gdp_buf_t *b)</pre>
1267
    <h4>Notes</h4>
1268
    <ul>
1269
      <li>Returns the number of bytes of data currently in the buffer.</li>
1270
    </ul>
1271
    <hr width="100%" size="2">
1272
    <h4>Name</h4>
1273
    gdp_buf_read, gdp_buf_peek, gdp_buf_drain &mdash; remove or peek at data in
1274
    a buffer
1275
    <h4>Synopsis</h4>
1276
    <pre>size_t gdp_buf_read(gdp_buf_t *b, void *out, size_t sz)<br>size_t gdp_buf_peek(gdp_buf_t *b, void *out, size_t sz)<br>int gdp_buf_drain(gdp_buf_t *b, size_t sz)</pre>
1277
    <h4>Notes</h4>
1278
    <ul>
1279
      <li>Data can be consumed from the buffer by calling <code>gdp_buf_read</code>;
1280
        data is copied into a memory area and removed from the buffer.</li>
1281
      <li>Applications can "peek" at the buffer using <code> gdp_buf_peek</code>.&nbsp;
1282
        This is identical to <code>gdp_buf_read</code> except that the data
1283
        remains in the buffer.</li>
1284
      <li>Applications can discard data from the buffer using <code>
1285
          gdp_buf_drain</code>.</li>
1286
      <li>In all cases, <code>sz</code> is the number of bytes to copy out
1287
        and/or discard.</li>
1288
    </ul>
1289
    <hr width="100%" size="2">
1290
    <h4>Name</h4>
1291
    gdp_buf_write, gdp_buf_printf &mdash; copy data into a buffer
1292
    <h4>Synopsis</h4>
1293
    <pre>int gdp_buf_write(gdp_buf_t *b, void *in, size_t sz)<br>int gdp_buf_printf(gdp_buf_t *b, const char *fmt, ...)</pre>
1294
    <h4>Notes</h4>
1295
    <ul>
1296
      <li>These routines append bytes into the named buffer.</li>
1297
      <li><code>gdp_buf_write</code> copies <code>sz</code> bytes into the
1298
        buffer from the memory area in and returns 0 on success or &ndash;1 on
1299
        failure. </li>
1300
      <li><code>gdp_buf_printf</code> essentially does a "printf" into the
1301
        buffer and returns the number of bytes appended. </li>
1302
    </ul>
1303
    <hr width="100%" size="2">
1304
    <h4>Name</h4>
1305
    gdp_buf_move &mdash; move data from one buffer into another
1306
    <h4>Synopsis</h4>
1307
    <pre>int gdp_buf_move(gdp_buf_t *ob, gdp_buf_t *ib, size_t sz)</pre>
1308
    <h4>Notes</h4>
1309
    <ul>
1310
      <li>Appends the first <code>sz</code> bytes of <code>ib</code> to the
1311
        end of <code>ob</code>.</li>
1312
      <li>This is more efficient than using <code>gdp_buf_read</code> and <code>gdp_buf_write</code>.</li>
1313
    </ul>
1314
    <hr width="100%" size="2">
1315
    <h4>Name</h4>
1316
    gdp_buf_dump &mdash; print the contents of the buffer for debugging
1317
    <h4>Synopsis</h4>
1318
    <pre>void gdp_buf_dump(gdp_buf_t *b, FILE *fp)</pre>
1319
    <h4>Notes</h4>
1320
    <ul>
1321
      <li>Prints the contents of buffer b to the file fp.</li>
1322
      <li>This is not intended for end user presentation.</li>
1323
    </ul>
1324
    <hr width="100%" size="2">
1325
    <h3>4.3&nbsp; Timestamps</h3>
1326
    <p>The time abstraction is imported directly from the ep library.&nbsp;
1327
      Times are represented as follows: </p>
1328
    <blockquote>
1329
      <pre>#pragma pack(push, 1)
1330
typedef struct
1331
{
1332
     int64_t        tv_sec;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp; // seconds since January 1, 1970
1333
    &nbsp;uint32_t&nbsp;&nbsp; tv_nsec;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; // nanoseconds<br>     float&nbsp;&nbsp;&nbsp;&nbsp;  tv_accuracy;&nbsp;&nbsp;&nbsp; // accuracy in seconds<br>} EP_TIME_SPEC;<br>#pragma pack(pop)</pre>
1334
    </blockquote>
1335
    Note that the host system <code>struct timespec</code> may not match this
1336
    structure; some systems still represent the time with only four bytes for <code>tv_sec</code>,
1337
    which expires in 2038.&nbsp; The <code>tv_accuracy</code> field indicates
1338
    an estimate for how accurate the clock is; for example, if you are running
1339
    NTP this value is likely to be on the order of a few tens to a few hundreds
1340
    of milliseconds, but if you set your clock manually it is likely to be
1341
    several seconds or worse.
1342
    <h2>5&nbsp; Signing and Encryption</h2>
1343
    <h3> 5.1&nbsp; Signing</h3>
1344
    <p>Each log should have a public key in the metadata which is used to verify
1345
      writes to the log.&nbsp; The library hides most of the details of this,
1346
      but some still appear.</p>
1347
    <p> The <span class="command">gdp-create</span> command automatically
1348
      creates a public/secret keypair unless otherwise specified.&nbsp; See the
1349
      man page for details.&nbsp; The public part of the key is inserted into
1350
      the log metadata and stored with the log.&nbsp; The secret part is stored
1351
      somewhere on your local filesystem, typically <span class="filename">KEYS/<span
1352
          class="variable">gob-id</span>.pem</span>.&nbsp; Normally <span class="command">gdp-create</span>
1353
      will encrypt the secret key with another key entered from the command
1354
      line, although this can also be turned off.</p>
1355
    <p>When a GDP application attempts to open a log using <span class="function">gdp_gin_open</span>,
1356
      the library will attempt to find a secret key by searching the directories
1357
      named in the <span class="admin-param">swarm.gdp.crypto.key.path</span>
1358
      administrative parameter for a file having the same name as the log (with
1359
      a <span class="filename">.pem</span> file suffix).&nbsp; If that secret
1360
      key is encrypted, the library will prompt the (human) user for the secret
1361
      key password.&nbsp; The default path is "<span class="filename">.</span>",
1362
      "<span class="filename">KEYS</span>", "<span class="filename">~/.swarm/gdp/keys</span>",
1363
      "<span class="filename">/usr/local/etc/swarm/gdp/keys</span>", and "<span
1364
        class="filename">/etc/swarm/gdp/keys</span>".</p>
1365
    <p>Once the secret key has been located and decrypted, all further append
1366
      requests will be signed using the secret key and verified by the log
1367
      daemon against the public key in the log metadata.</p>
1368
    <h3>5.2&nbsp; Encryption</h3>
1369
    <p>Encryption is explicitly not part of the GDP.&nbsp; Ideally the GDP will
1370
      never see unencrypted data.&nbsp; However, read and write filters (see <code>gdp_gin_set_append_filter</code>
1371
      and <code>gdp_gin_set_append_filter</code> for details) can be used to
1372
      set encryption and decryption hooks for externally implemented
1373
      encryption.&nbsp; <span class="meta">We need to make this easier.</span></p>
1374
    <br>
1375
    <hr>
1376
    <h3>To be done</h3>
1377
    <p>Header files<br>
1378
      Version info<br>
1379
      PRIgdp_recno macro<br>
1380
    </p>
1381
    <h2>Appendix A:&nbsp; Examples</h2>
1382
    <p class="meta">The following examples have not been validated with the v2
1383
      API, so they may be inaccurate.</p>
1384
    <p>The following pseudo-code example excerpts from <tt>apps/gdp-writer.c</tt>.
1385
    </p>
1386
    <pre>#include &lt;gdp/gdp.h&gt;<br>#include &lt;stdio.h&gt;<br>#include &lt;string.h&gt;<br><br>int main(int argc, char **argv)<br>{<br>        gdp_gin_t *gin;<br>        EP_STAT estat;<br>        gdp_name_t gobiname;        // internal name of GOB<br>        gdp_datum_t *d;<br><br>        // general startup and initialization<br>        if (argc &lt; 2)<br>                usage_error();<br>        estat = gdp_init();<br>        if (!EP_STAT_ISOK(estat))<br>                initialization_error(estat);<br>        d = gdp_datum_new();
1387
<br><br>        // parse command line name to internal format<br>        estat = gdp_parse_name(argv[1], gobiname);<br>        if (!EP_STAT_ISOK(estat))<br>                name_syntax_error();<br><br>        // attempt to create that name<br>        estat = gdp_gin_create(gobiname, &amp;gin);<br>        if (!EP_STAT_ISOK(estat))<br>                creation_error(estat);<br><br>        // read lines from standard input<br>        while (fgets(buf, sizeof buf, stdin) != NULL)<br>        {<br>                char *p = strchr(buf, '\n');<br>                if (p != NULL)<br>                        *p = '\0';<br><br>                // write them to the dataplane<br>                if (gdp_buf_write(gdp_datum_getbuf(d), buf, strlen(buf)) &lt; 0)<br>                        estat = GDP_STAT_BUFFER_FAILURE;<br>                else<br>                        estat = gdp_gin_append(gin, d);<br>                EP_STAT_CHECK(estat, break);<br>        }<br><br>        // cleanup and exit<br>        gdp_gin_close(gin);<br>        exit(!EP_STAT_ISOK(estat));<br>}</pre>
1388
    <hr width="100%" size="2">
1389
    <p>This example is a similar excerpt from apps/gdp-reader.c (without using
1390
      subscriptions):</p>
1391
    <pre>#include &lt;gdp/gdp.h&gt;<br><br>int main(int argc, char **argv)<br>{<br>        gdp_gin_t *gin;<br>        EP_STAT estat;<br>        gdp_name_t gobiname;        // internal name of GOB<br>        gdp_datum_t *d;<br>        gdp_recno_t recno;<br><br>        // general startup and initialization<br>        if (argc &lt; 2)<br>                usage_error();<br>        estat = gdp_init();<br>        if (!EP_STAT_ISOK(estat))<br>                initialization_error(estat);<br>        d = gdp_datum_new();
1392
<br><br>        // parse command line name to internal format<br>        estat = gdp_parse_name(argv[1], gobiname);<br>        if (!EP_STAT_ISOK(estat))<br>                name_syntax_error();
1393
<br>        // attempt to open the GOB<br>        estat = gdp_gin_open(gobiname, GDP_MODE_RO, &amp;gin);<br>        if (!EP_STAT_ISOK(estat))<br>                open_error(estat, argv[1]);<br>
1394
&nbsp;        recno = 1;<br>        for (;;)<br>        {<br>                estat = gdp_gin_read_by_recno(gin, recno++, d);<br>                EP_STAT_CHECK(estat, break);<br>                gdp_datum_print(d, stdout);<br>        }<br>        exit(0);<br>}</pre>
1395
    <br>
1396
    <hr width="100%" size="2">
1397
    <p>If you want to use subscriptions, the recno variable can be removed and
1398
      the for loop replaced with: </p>
1399
    <pre>        // enable the subscription<br>        estat = gdp_gin_subscribe_by_recno(gin, 1, -1, NULL, NULL);<br>        if (!EP_STAT_ISOK(estat))<br>                subscribe_error(estat, argv[1]);<br><br>        for (;;)<br>        {<br>                gdp_event_t *gev = gdp_event_next(true);<br>                if (gdp_event_gettype(gev) != GDP_EVENT_DATA)<br>                        continue;<br>                gdp_datum_print(gdp_event_getdatum(gev), stdout);<br>                gdp_event_free(gev);<br>        }</pre>
1400
    <br>
1401
    <hr width="100%" size="2">
1402
    <h2>Appendix B:&nbsp; Compiling and Linking</h2>
1403
    <p>The GDP library uses a reduced version of libep and also uses the
1404
      libevent library version 2.1. These will need to be included both during
1405
      compilation and linking.</p>
1406
    <p> At compile time you must use:</p>
1407
    <p><code>-I</code><code><i>libevent_includes_parent</i></code><code> -I</code><code><i>libep_includes_parent</i></code></p>
1408
    <p> Note that these take the parent of the directory containing the include
1409
      files. For example, if the include files for libevent are in <tt>/usr/local/include/event2</tt>
1410
      and the include files for libep are in <tt>/usr/local/include/ep</tt> you
1411
      only need to specify the one flag "<code>-I/usr/local/include</code>".</p>
1412
    For linking you must use:<br>
1413
    <pre>-L<i>libevent_libraries</i> -levent -levent_pthreads -L<i>libep_libraries</i> -lep</pre>
1414
    As before, if the libraries for libevent and libep are in the same directory
1415
    you only need a single <tt>-L</tt> flag.<br>
1416
    Libep is a library that I produced several years ago intended for use in
1417
    sendmail. This uses a stripped down version of that library that excludes
1418
    several things that would not be helpful here. For more details of the
1419
    original (full) library, see <a href="http://www.neophilic.com/blogs/eric.php/2014/05/12/libep-portable-c-runtime">http://www.neophilic.com/blogs/eric.php/2014/05/12/libep-portable-c-runtime</a>.
1420
    <p>For additional information, see the <tt>README</tt> file in the
1421
      distribution directory. </p>
1422
    <h2> Appendix C: Open Questions</h2>
1423
    <p class="meta">This section is really an addendum to the document &mdash; a
1424
      "scratch area" to keep track of issues that we still need to
1425
      consider.&nbsp; It may not be up to date. </p>
1426
    <h3>C.1 Access Control</h3>
1427
    <p> Do this using Access Control Lists (so each user/app has a keypair) or
1428
      by passing public/secret keys around (so each GOB has a secret keypair).
1429
      The latter makes revocation impossible (even for write access), so I
1430
      prefer the ACL approach. Third way?</p>
1431
    <p> Revocation? Deep vs. Shallow. Deep = take away permissions that have
1432
      already been given. Shallow = you can only prevent an accessor from
1433
      getting to new versions. Argument: deep revocation is hard to do from a
1434
      technical perspective and ultimately futile (someone might have taken a
1435
      photo of a screen while they still had access), but is still what people
1436
      are used to (Unix and SQL permissions work this way). Shallow is all that
1437
      can really be guaranteed. Also, anything involving Certificate Revocation
1438
      Lists (CRLs) is doomed to failure. This implies that ACLs are the correct
1439
      direction.</p>
1440
    <p> ACLs get us into the question of identity. Pretending that a keypair
1441
      represents an identity doesn't work in the real world where bad players
1442
      simply create new "identities" (keypairs) when an old identity has become
1443
      untrusted. See the extensive work in email sender reputation. However,
1444
      when a bad player creates a new identity/keypair they do not get access to
1445
      any previous grants, so this may be sufficient.</p>
1446
    <h3> C.2 Naming</h3>
1447
    <p> If each GOB has a secret keypair, then the public key is sufficient to
1448
      name the entity. If not, then assigning a GOB a GUID on creation seems
1449
      like the best approach. Having the user assign a name seems like a
1450
      non-starter, if only because of the possibility of conflicts.</p>
1451
    <p> There will probably be some need for external naming, e.g., some overlay
1452
      directory structure. That might be a different gob_type.</p>
1453
    <p> This seems like an open research topic.</p>
1454
    <h3> C.3 Orphans, Expiration, Charging, and Accounting</h3>
1455
    <p> If a GOB isn't linked into a directory structure and everyone forgets
1456
      its name then it will live forever (or until it expires). This could be
1457
      quite common if a GOB is temporary, that is, not a candidate for long-term
1458
      archival.</p>
1459
    <p> Expiration could be an issue without some sort of charging, which
1460
      implies accounting.</p>
1461
    <p> Charging and accounting will affect the API. It seems like on GOB
1462
      creation the creator needs to offer payment for both carrying and storing
1463
      the data. This payment would presumably accrue to the actors providing the
1464
      actual service. Payment for storage might be limited time or indefinite
1465
      time (i.e., it would be an endowment).</p>
1466
    <p> The creator could also specify a cost for any potential consumer in
1467
      order to access the GOB. Such payments would accrue to the creator of the
1468
      GOB, and might be used to fund continued access, i.e. it could be rolled
1469
      back into the endowment. This would lean toward making less-used data
1470
      disappear: appealing in some ways, but anathema to librarians and
1471
      historians.</p>
1472
    <p> As for API effects, it seems that GOB creation needs to include a
1473
      payment for initial service, a cost for access, and an account into which
1474
      to deposit any consumer payments. Accessing a GOB only requires an offered
1475
      payment (which is probably best viewed as a bid rather than a payment,
1476
      thus allowing multiple providers to compete for access).</p>
1477
    <p> Note that none of this is dependent on the form of payment. It does
1478
      however assume that there is a mutually agreed upon form of payment, i.e.,
1479
      a universal currency.</p>
1480
    <h3> C.4 Quality of Service</h3>
1481
    <p> Is Quality of Service specified on a particular GOB, a particular open
1482
      instance of a GOB, or between a pair of endpoints?</p>
1483
    <p> What does QoS actually mean? For example, in a live media stream it
1484
      probably means the resolution of the data stream (which determines
1485
      real-time bandwidth), latency, and possibly jitter, but after that stream
1486
      is stored the QoS will be resolution (as before), delivery bandwidth (how
1487
      quickly you can download the video, for example), and possibly jitter of
1488
      the network connection (that is, how even the data flow will be). Delivery
1489
      bandwidth depends on the entire path between the data source and the data
1490
      sync, and may be higher or lower than the bandwidth required to send a
1491
      real-time version of the stream &mdash; for example, over a slow network
1492
      link.</p>
1493
    <h2> Appendix D: References</h2>
1494
    <table cellspacing="2" cellpadding="2" border="1" width="100%">
1495
      <tbody>
1496
        <tr>
1497
          <td valign="top">[Dab13a] </td>
1498
          <td valign="top">Palmer Dabbelt, Swarm OS Universal Dataplane, August
1499
            22, 2013</td>
1500
        </tr>
1501
        <tr>
1502
          <td valign="top">[Dab13b]</td>
1503
          <td valign="top">Palmer Dabbelt, What is the Universal Dataplane,
1504
            Anyway?, September 17, 2013</td>
1505
        </tr>
1506
      </tbody>
1507
    </table>
1508
  </body>
1509
</html>