GDP

<?xml version="1.0" encoding="US-ASCII"?>

<!DOCTYPE article PUBLIC "-//OASIS//DTD DocBook XML V4.3//EN"

"http://www.oasis-open.org/docbook/xml/4.3/docbookx.dtd">

<article class="techreport">

  <articleinfo>

    <title>Notes on "EP" Library</title>

    <author>

      <firstname>Eric</firstname>

      <surname>Allman</surname>

    </author>

    <copyright>

      <year>2008, 2014&ndash;2018 Eric P. Allman. All rights reserved.</year>

    </copyright>

    <date>2018-08-15</date>

  </articleinfo>

  <section>

    <title>INTRODUCTION</title>

    <para>This document describes the Enhanced Portability library. This is a

    reduced version of Eric's Portability library (designed for sendmail),

    removing many things that didn't work out or proved unnecessary, e.g., the

    entire I/O subsystem. It was originally intended to be used in sendmail,

    so some of the terminology is geared toward the email world; none the

    less, it should be generally useful.</para>

    <para>Programs using <symbol>libep</symbol> should be able to minimize

    <symbol>#ifdef</symbol> since often non-portable functionality is wrapped

    in portability routines; for example, BSD-derived systems and

    Linux-derived systems differ in how you get the name of the currently

    running program. These differences are hidden if the application calls

    <function>ep_app_getprogname</function>.</para>

    <section>

      <title>Design Goals</title>

      <itemizedlist>

        <listitem>

          <para>Portable, to the extent possible. Where not possible, there

          needs to be a clearly codified way to represent the externally

          visible semantic differences.</para>

        </listitem>

        <listitem>

          <para>Efficient.</para>

        </listitem>

        <listitem>

          <para>Customizable -- try to implement mechanism, not policy.</para>

        </listitem>

        <listitem>

          <para>TBD: It should be entirely UTF-8 internally. Any translations

          to other character sets should be done on input or output, and then

          only as strictly necessary.</para>

        </listitem>

      </itemizedlist>

    </section>

    <section>

      <title>Assumptions</title>

      <para>Use of this library requires that you have a C compiler that is

      compliant with ANSI C as defined by ANSI/ISO 9899-1999. Also requires an

      environment that is at least Posix based on Posix.1-2008.</para>

    </section>

    <section>

      <title>Conventions</title>

      <para><itemizedlist>

          <listitem>

            <para>All externally visible names (i.e., those not declared

            "static" in a file) shall be named ep_* (for routine names) or Ep*

            (for variable names). In a few cases, the names may begin with

            __ep or __Ep; such names would be in the global namespace, but

            would be intended for use internal to the library only. There are

            a few cases where "standard" names (such as

            <function>strlcpy</function>) are defined if they are not included

            in the standard library.</para>

          </listitem>

        </itemizedlist></para>

    </section>

    <section>

      <title>Terminology</title>

      <section>

        <title>Warning, Error, Severe, Abort</title>

        <para>These words get used fairly loosely, so they are worth defining.

        In the context of libep:</para>

        <itemizedlist>

          <listitem>

            <para>Warning means a condition that is expected in normal

            operations, but is not the usual case. Reading an end-of-file on a

            file might be a warning. Applications need to be aware of these,

            but are expected to either ignore them or recover easily. This can

            also be used for temporary errors which are likely to recover

            after a delay. For example, the inability to open a connection to

            a remote server might recover automatically if that server is

            re-started. However, this sort of warnings that persist should

            turn into permanent errors in a fashion appropriate for the

            application.</para>

          </listitem>

          <listitem>

            <para>Error means a situation that should not occur, but isn't

            terribly unusual. For example, an attempt to open a file that

            isn't accessible would be an error. Applications must be aware of

            such conditions and handle them gracefully.</para>

          </listitem>

          <listitem>

            <para>Severe means a situation that should not occur iand requires

            exceptional handling. Severe errors are drastic conditions, but

            are not so severe that the application can't take some reasonable

            backout action.</para>

          </listitem>

          <listitem>

            <para>Abort means a situation so drastic that an application

            cannot be expected to make any reasonable recovery. These might

            include assertion errors and memory allocation failures during a

            critical step (e.g., something where backing out a single thread

            won't solve the problem). About the only thing an application can

            reasonably do is log an abort error and exit. In particular, an

            abort is appropriate when an attempt for an application to recover

            is likely to do additional damage. These should be extremely

            rare.</para>

          </listitem>

        </itemizedlist>

      </section>

    </section>

  </section>

  <section>

    <title>GENERAL ISSUES</title>

    <para>All files using this library must use "<code>#include

    &lt;ep/ep.h&gt;</code>".</para>

  </section>

  <section>

    <title>STATUS CODES</title>

    <para>Almost all functions return an <type>EP_STAT</type> value. This is a

    short (integer-encoded) status value that gives you a brief idea of how

    severe the problem was and some idea of what it was, but not much else.

    Think of it as an errno equivalent. Functions returning any status other

    than OK are expected to provide some other way of returning detailed

    data.</para>

    <para><type>EP_STAT</type>s are also used as message identifiers for

    logging (below).</para>

    <para>Status codes are defined in

    <filename>&lt;ep/ep_stat.h&gt;</filename>.</para>

    <section>

      <title>Severities</title>

      <para>Severities are:</para>

      <variablelist termlength="20">

        <varlistentry>

          <term><errortype>EP_STAT_SEV_OK</errortype></term>

          <listitem>

            <para>Everything is fine. Detail may contain info. For messages,

            can be used for debugging.</para>

          </listitem>

        </varlistentry>

        <varlistentry>

          <term><errortype>EP_STAT_SEV_WARN</errortype></term>

          <listitem>

            <para>The function partially succeeded, but there is something

            that the application should be aware of, e.g., an end of file or a

            short data read. Alternatively, the functionfailed, but it might

            work again on a later try.</para>

          </listitem>

        </varlistentry>

        <varlistentry>

          <term><errortype>EP_STAT_SEV_ERROR</errortype></term>

          <listitem>

            <para>A normal error status. The call failed.</para>

          </listitem>

        </varlistentry>

        <varlistentry>

          <term><errortype>EP_STAT_SEV_SEVERE</errortype></term>

          <listitem>

            <para>A severe error status. The call failed, and the caller

            should try to back out.</para>

          </listitem>

        </varlistentry>

        <varlistentry>

          <term><errortype>EP_STAT_SEV_ABORT</errortype></term>

          <listitem>

            <para>A critical error occured &mdash; you should clean up and

            exit as soon as possible; the program cannot be expected to

            operate correctly.</para>

          </listitem>

        </varlistentry>

      </variablelist>

      <para>Some functions for testing values:</para>

      <variablelist>

        <varlistentry>

          <term><function>EP_STAT_SEV_ISOK</function>(st)</term>

          <listitem>

            <para>Returns true if this is an

            <errortype>EP_STAT_SEV_OK</errortype> status code.</para>

          </listitem>

        </varlistentry>

        <varlistentry>

          <term><function>EP_STAT_SEV_WARN</function>(st)</term>

          <listitem>

            <para>Returns true if this is an "warning" severity status code:

            <errortype>EP_STAT_SEV_WARN</errortype>.</para>

          </listitem>

        </varlistentry>

        <varlistentry>

          <term><function>EP_STAT_SEV_ISERROR</function>(st)</term>

          <listitem>

            <para>Returns true if this is an "error" severity status code:

            <errortype>EP_STAT_SEV_ERROR</errortype></para>

          </listitem>

        </varlistentry>

        <varlistentry>

          <term><function>EP_STAT_SEV_ISFAIL</function>(st)</term>

          <listitem>

            <para>Returns true if this message is a "failure" severity status

            code: <errortype>EP_STAT_SEV_ERROR</errortype> or higher.</para>

          </listitem>

        </varlistentry>

        <varlistentry>

          <term><function>EP_STAT_SEV_ISSEVERE</function>(st)</term>

          <listitem>

            <para>Returns true if this is an "severe" severity status code:

            <errortype>EP_STAT_SEV_SEVERE</errortype></para>

          </listitem>

        </varlistentry>

        <varlistentry>

          <term><function>EP_STAT_SEV_ISSFAIL</function>(st)</term>

          <listitem>

            <para>Returns true if this message is a "major" severity status

            code: <errortype>EP_STAT_SEV_SEVERE</errortype> or higher</para>

          </listitem>

        </varlistentry>

        <varlistentry>

          <term><function>EP_STAT_SEV_ISABORT</function>(st)</term>

          <listitem>

            <para>Returns true if this is an "abort" severity status code:

            <errortype>EP_STAT_SEV_ABORT</errortype></para>

          </listitem>

        </varlistentry>

      </variablelist>

    </section>

    <section>

      <title>Status Code Representation</title>

      <para>Status codes are represented as four-part values: severity,

      registry, module, and detail. The severities are described above.

      Registries are globally registered by neophilic.com and are defined in

      ep_registry.h. There are some registries for general use; in particular,

      registry numbers between 0x001 and 0x1FF are available for local

      (non-global) registry at the corporate or local level. Modules are

      defined by registries, and detail is defined by module. It is

      <emphasis>never</emphasis> acceptable to look at detail unless you

      recognize the module. (OK, you can print it out for debugging.) Severity

      = 3 bits, registry = 13 bits, module = 6 bits, detail = 10 bits.</para>

      <para>Any severity where the top bit is zero is considered "OK", and the

      rest of the word is available to encode a non-negative integer.</para>

      <para>Status codes are represented as structures to ensure type safety.

      Occassionally you might want to convert a status to or from a long

      int:</para>

      <programlisting>int      EP_STAT_TO_INT(EP_STAT stat)          // convert status to unsigned int

EP_STAT  EP_STAT_FROM_INT(unsigned int istat)  // convert unsigned integer to status</programlisting>

      <para>The constituent parts of the status code can also be

      extracted:</para>

      <variablelist>

        <varlistentry>

          <term><function>EP_STAT_SEV</function>(st)</term>

          <listitem>

            <para>Returns the severity part of the status code.</para>

          </listitem>

        </varlistentry>

        <varlistentry>

          <term><function>EP_STAT_REGISTRY</function>(st)</term>

          <listitem>

            <para>Returns the registry part of the status code.</para>

          </listitem>

        </varlistentry>

        <varlistentry>

          <term><function>EP_STAT_MODULE</function>(st)</term>

          <listitem>

            <para>Returns the module part of the status code.</para>

          </listitem>

        </varlistentry>

        <varlistentry>

          <term><function>EP_STAT_DETAIL</function>(st)</term>

          <listitem>

            <para>Returns the detail part of the status code.</para>

          </listitem>

        </varlistentry>

      </variablelist>

      <para>To compare two statuses for equality, use

      <function>EP_STAT_IS_SAME</function>(a, b).</para>

      <para>As a special case, if the severity is

      <errortype>EP_STAT_SEV_OK</errortype> the rest of the word is ignored;

      this can be used to pass small integers (no more than 31 bits) of

      information.</para>

    </section>

    <section>

      <title>Predefined Status Codes</title>

      <para>All status codes from this library are in the

      <constant>EP_REGISTRY_EPLIB</constant> registery. There are several

      predefined status codes for generic use, all using module

      <constant>EP_STAT_MOD_GENERIC</constant>:</para>

      <informaltable border="1">

        <col width="4*"/>

        <col width="6*"/>

        <tr>

          <td>EP_STAT_OK</td>

          <td>No error (also integer 0)</td>

        </tr>

        <tr>

          <td>EP_STAT_WARN</td>

          <td>Generic warning status</td>

        </tr>

        <tr>

          <td>EP_STAT_ERROR</td>

          <td>Generic error status</td>

        </tr>

        <tr>

          <td>EP_STAT_SEVERE</td>

          <td>Generic severe error status</td>

        </tr>

        <tr>

          <td>EP_STAT_ABORT</td>

          <td>Generic abortive status</td>

        </tr>

        <tr>

          <td>EP_STAT_OUT_OF_MEMORY</td>

          <td>Out of memory</td>

        </tr>

        <tr>

          <td>EP_STAT_ARG_OUT_OF_RANGE</td>

          <td>An argument was out of range</td>

        </tr>

        <tr>

          <td>EP_STAT_END_OF_FILE</td>

          <td>End of input</td>

        </tr>

        <tr>

          <td>EP_STAT_TIME_BADFORMAT</td>

          <td>Couldn't parse a date/time string</td>

        </tr>

        <tr>

          <td>EP_STAT_BUF_OVERFLOW</td>

          <td>Buffer overflow averted</td>

        </tr>

        <tr>

          <td>EP_STAT_ASSERT_ABORT</td>

          <td>Assertion failiure: backout now!</td>

        </tr>

      </informaltable>

      <para>There is also a special module

      <constant>EP_STAT_MOD_ERRNO</constant> that encodes Posix-style errnos

      (i.e., use <function>EP_STAT_DETAIL</function> on codes returned by that

      module to get the Posix errno code).</para>

    </section>

    <section>

      <title>Manipulating Status Codes</title>

      <para>There are several routines to print error codes or create them on

      the fly. Note that <function>ep_stat_tostr</function> returns the buffer

      itself.</para>

      <programlisting>// create status code from UNIX errno

extern EP_STAT  ep_stat_from_errno(

                        int uerrno);

// return string representation of status

char            *ep_stat_tostr(

                        EP_STAT estat,

                        char *buf,

                        size_t bsize);

// return string representation of severity (in natural language)

const char      *ep_stat_sev_tostr(

                        int sev);

// print a status code and abort (never returns)

void            ep_stat_abort(

                        EP_STAT estat);</programlisting>

    </section>

    <section>

      <title>Creating New Status Codes</title>

      <para>Libraries and applications can create their own specific error

      codes. There are four steps to do this:</para>

      <orderedlist>

        <listitem>

          <para>Determine the registry. The registry name space is divided as

          follows:</para>

          <informaltable border="1">

            <col width="4*"/>

            <col width="6*"/>

            <tr>

              <td>0x000 (<constant>EP_REGISTRY_GENERIC</constant>)</td>

              <td>reserved for generic status codes</td>

            </tr>

            <tr>

              <td>0x001 (<constant>EP_REGISTRY_USER</constant>)</td>

              <td>available for internal use to an application</td>

            </tr>

            <tr>

              <td>0x002&ndash;0x07F</td>

              <td>available for local, unregistered use, such as separate

              applications within an application suite</td>

            </tr>

            <tr>

              <td>0x080&ndash;0x0FF</td>

              <td>available for internal corporate registry, but not

              registered globally; conflicts may occur between organizations

              but not within an organization</td>

            </tr>

            <tr>

              <td>0x100 (<constant>EP_REGISTRY_EPLIB</constant>)</td>

              <td>reserved for libep</td>

            </tr>

            <tr>

              <td>0x101&ndash;0x6FF</td>

              <td>available for centrally managed global registry &mdash;

              contact the libep maintainers for an allocation</td>

            </tr>

            <tr>

              <td>0x700&ndash;0x7FF</td>

              <td>reserved</td>

            </tr>

          </informaltable>

        </listitem>

        <listitem>

          <para>Determine the module(s) in which the error code should exist.

          These must be unique within a registry and in the range

          0x00&ndash;0xFF.</para>

        </listitem>

        <listitem>

          <para>Define the error codes you want to use using

          <function>EP_STAT_NEW</function>, e.g.,</para>

          <programlisting>#define FOO_STAT_ELEPHANT    EP_STAT_NEW(ERROR, EP_REGISTRY_FOO, MOD_BAR, 1)

#define FOO_STAT_GIRAFFE     EP_STAT_NEW(SEVERE, EP_REGISTRY_FOO, MOD_BAR, 2)</programlisting>

        </listitem>

        <listitem>

          <para>(Optional step) Define the strings associated with the error

          codes when they are printed. These are done by populating a table

          and then calling <function>ep_stat_register_strings</function>. For

          example:</para>

          <programlisting>struct ep_stat_reg_strings  FooStatusCodes[] =

{

    FOO_STAT_ELEPHANT,    "elephant in the room",        },

    FOO_STAT_GIRAFFE,     "too tall",                    },

    EP_STAT_OK,           NULL,                          }

}

ep_stat_reg_strings(FooStatusCodes);</programlisting>

        </listitem>

      </orderedlist>

    </section>

  </section>

  <section>

    <title>INITIALIZATION</title>

    <para>Although libep will generally work without initialization, in some

    cases you may need to give it information about your usage. To do this

    call <function>ep_lib_init</function>:</para>

    <programlisting>#include &lt;ep/ep.h&gt;

EP_STAT

ep_lib_init(uint32_t flags)</programlisting>

    <para>Flags can be:</para>

    <informaltable>

      <tgroup cols="2">

        <tbody>

          <row>

            <entry>EP_LIB_USEPTHREADS</entry>

            <entry>Initialize the thread support</entry>

          </row>

        </tbody>

      </tgroup>

    </informaltable>

  </section>

  <section>

    <title>MEMORY ALLOCATION AND RESOURCE POOLS</title>

    <section>

      <title>Memory</title>

      <para>Memory support is much like malloc/free, but with some additional

      functionality. One crucial difference is that most of these routines do

      not return if memory is exhausted; instead they can call a cleanup

      routine that might (for example) eliminate some old cache entries, or

      pick a "victim" thread to kill and reclaim its memory. If successful

      they can continue, otherwise the process is aborted.</para>

      <programlisting>    #include &lt;ep/ep_mem.h&gt;

    void *

    ep_mem_malloc(size_t nbytes)       // allocate uninitialized memory

    void *

    ep_mem_zalloc(size_t nbytes)       // allocate zeroed memory

    void *

    ep_mem_ralloc(size_t nbytes)       // allocate randomized memory

    void *

    ep_mem_ealloc(size_t nbytes)       // allocate memory, failure OK

    void *

    ep_mem_realloc(size_t nbytes,      // reallocate (extend) memory

                void *curmem)

    void *

    ep_mem_falloc(size_t nbytes,       // allocate memory (see flags)

                uint32_t flags)

    void

    ep_mem_mfree(void *mem)            // free indicated memory

    struct ep_malloc_functions

    {

        void    *(*m_malloc)(size_t);

        void    *(*m_realloc)(void*, size_t);

        void    *(*m_valloc)(size_t);

        void    (*m_free)(void*);

    };

    void

    ep_mem_set_malloc_functions(       // set underlying malloc functions

                struct ep_malloc_functions *funcs)</programlisting>

      <para>The <function>ep_mem_malloc</function>,

      <function>ep_mem_zalloc</function>, <function>ep_mem_ralloc</function>,

      and <function>ep_mem_realloc</function> are all implemented in terms of

      <function>ep_mem_falloc</function>, which uses flags to tune the

      behavior (see below). The primary interface is

      <function>ep_mem_malloc</function>, which returns uninitialized data;

      <function>ep_mem_zalloc</function> returns zeroed memory, and

      <function>ep_mem_ralloc</function> returns memory that is initialized to

      random or some other nonsensical data. The last would probably be used

      only for debugging, and can be turned on at runtime using a debug flag

      <remark>XXX TBD</remark>.</para>

      <para>In all allocation schemes, the function returns a pointer to the

      allocated data — they cannot normally return

      <constant>NULL</constant> (but see below). If they cannot allocate the

      memory, they <remark>do error recovery (XXX describe)</remark>. If

      recovery fails, the allocation system will abort the process. However,

      <function>ep_mem_ealloc</function> can return <constant>NULL</constant>

      on memory allocation failure, as can <function>ep_mem_falloc</function>

      if the <constant>EP_MEM_F_FAILOK</constant> flag bit is set (see

      below).</para>

      <para>Flag bits are as follows:</para>

      <variablelist>

        <varlistentry>

          <term><constant>EP_MEM_F_FAILOK</constant></term>

          <listitem>

            <para>Permits the routine to return <constant>NULL</constant> on

            failure. This modifies the behavior described above. Note that if

            this is set every call to the <function>ep_*malloc</function>

            routines may potentially fail.</para>

          </listitem>

        </varlistentry>

        <varlistentry>

          <term><constant>EP_MEM_F_ZERO</constant></term>

          <listitem>

            <para>Zero any returned memory.</para>

          </listitem>

        </varlistentry>

        <varlistentry>

          <term><constant>EP_MEM_F_TRASH</constant></term>

          <listitem>

            <para>Randomize any returned memory.</para>

          </listitem>

        </varlistentry>

        <varlistentry>

          <term><constant>EP_MEM_F_ALIGN</constant></term>

          <listitem>

            <para>The application would prefer that the allocation is

            page-aligned. This is not available on all architectures, and

            other architectures do it automatically if the allocation is at

            least as large as a page.</para>

          </listitem>

        </varlistentry>

        <varlistentry>

          <term><constant>EP_MEM_F_WAIT</constant></term>

          <listitem>

            <para>If memory is unavailable, try to wait for it to become

            available (e.g., because another thread has released memory).

            <remark>This is not yet implemented.</remark></para>

          </listitem>

        </varlistentry>

      </variablelist>

      <para>Specifying <constant>EP_MEM_F_ZERO</constant> and

      <constant>EP_MEM_F_TRASH</constant> at the same time is

      undefined.</para>

      <para>Since <function>ep_mem_[mzr]alloc</function> are implemented as

      macros, they can't be used as pointers to functions (e.g., for

      specifying a memory allocator callback to a third party app). For this

      reason, there are also <function>ep_mem_[mzr]alloc_f</function> "real"

      functions to be used in this context.</para>

      <para>Generally, unthreaded code and most application code will probably

      be happy with the defaults. Threaded server code (which cannot be

      permitted to die) is expected to catch the out of memory condition, do

      some recovery operation such as terminating a task, and return

      <errorcode>EP_MEM_STAT_TRYAGAIN</errorcode> so the memory allocation can

      retry.</para>

      <para>[[[XXX Document ep_set_malloc_functions XXX]]]</para>

    </section>

    <section>

      <title>Resource Pools</title>

      <para>Resources are allocatable global entities such as memory, file

      descriptors, etc. Resources can be collected together into pools and

      then freed in one call. Memory is specially handled to allow fast

      allocation from a pool --- specifically, a chunk of memory can be

      allocated from the heap to a pool and then sub-allocated as needed.

      Allocating memory from resource pools is particularly fast for small

      allocations. Also, pool allocations that are of a size that is a

      multiple of the page size are guaranteed to return a page-aligned

      pointer. This is particularly useful to allow the I/O level to implement

      zero-copy I/O.</para>

      <para>The heap used is the one that is current when

      <function>ep_rpool_new</function> is invoked.</para>

      <programlisting>    #include &lt;ep/ep_mem.h&gt;

    EP_RPOOL *

    ep_rpool_new(const char *name,              // for debugging

                size_t qsize)                   // min memory allocation quantum

    EP_STAT

    ep_rpool_free(EP_RPOOL *rp)                 // free pool and all resources

    void *

    ep_rpool_malloc(EP_RPOOL *rp,               // the pool to allocate from

                size_t nbytes)                  // number of bytes

    void *

    ep_rpool_zalloc(EP_RPOOL *rp,               // the pool to allocate from

                size_t nbytes)                  // number of bytes

    void *

    ep_rpool_xalloc(EP_RPOOL *rp,               // the pool to allocate from

                size_t nbytes,                  // number of bytes

                const char *filename,           // file name (for debugging)

                int lineno,                     // line number (for debugging)

                uint32_t flags)                 // flag bits (see below)

    void *

    ep_rpool_strdup(EP_RPOOL *rp,               // the pool to allocate from

                char *str)                      // the string to save

    void *

    ep_rpool_realloc(EP_RPOOL *rp,              // pool to allocate from

                void *old mem,                  // old memory pointer

                size_t oldsize,                 // old allocation size

                size_t newsize)                 // new allocation size

    void

    ep_rpool_mfree(EP_RPOOL *rp,                // the pool to release to

                void *p);                       // the memory

    void

    ep_rpool_mfreeto(EP_RPOOL *rp,              // the pool to release to

                void *p);                       // restore up the pool to here

    EP_STAT

    ep_rpool_attach(EP_RPOOL *rp,               // the resource pool

                void freefunc(void *arg),       // a function to call on free

                void *arg)                      // argument to pass to it</programlisting>

      <para>The <function>ep_rpool_mfreeto</function>() routine lets you treat

      rpool memory like a stack; this call releases everything allocated back

      to (and including) the pointer given. If <computeroutput>p ==

      NULL</computeroutput>, the entire memory contents of the rpool are

      freed, but the rpool itself is still active. Deep care needs to be taken

      here: if a subordinate routine is called that allocates memory from the

      rpool, you may end up deallocating memory that is still in use.

      <remark>Not implemented at this time.</remark></para>

      <para>The <function>ep_rpool_attach</function>() routine is used to

      associate other resources (such as files) with a pool. The corresponding

      free functions will be invoked when the pool is freed.</para>

      <para>In most cases, passing in <parameter>rp</parameter> ==

      <constant>NULL</constant> treats the call like the corresponding heap

      allocation. In this case the caller is responsible for freeing the

      memory. For example,

      <function>ep_rpool_malloc</function>(<constant>NULL</constant>,

      <varname>nbytes</varname>) is equivalent to

      <function>ep_mem_malloc</function>(<varname>nbytes</varname>).</para>

      <para>The distinction between multiple heaps and resource pools are that

      heaps are not intended for application use other than for doing recovery

      for out-of-memory conditions. Pools are intended for general use. Pools

      are fast at allocation time (since they just grab space from the end of

      the pool) and fast at free time (since the entire pool can be

      deallocated at once); heaps are comparatively slow.</para>

      <para>When any memory collections (heaps or pools) are freed, all

      objects allocated from that collection are freed (i.e., their

      destructors are automatically invoked).</para>

    </section>

    <section>

      <title>Opening Memory as a File</title>

      <programlisting>    FILE *

    ep_fopen_smem(void *buf,          // block of memory to open

        size_t bsize,                 // size of that memory

        const char *mode)             // fopen(3) mode string</programlisting>

    </section>

  </section>

  <section>

    <title>TIME</title>

    <para>The ep library has a separate time abstraction. This is for two

    reasons: first, it guarantees that the number of seconds since January 1,

    1970 will be sufficiently long to last past 2038 (this varies from system

    to system), and it includes a "<structfield>tv_accuracy</structfield>"

    (type float) to indicate the approximate accuracy of the clock relative to

    absolute time. For example, a clock synchronized from a GPS clock might be

    accurate within perhaps 100nsec, whereas a standard crystal clock

    synchronized once a day might only have an accuracy of a few

    seconds.</para>

    <programlisting>#include &lt;ep/ep_time.h&gt;

typedef struct

{

        int64_t     tv_sec;          // seconds since Jan 1, 1970

        int32_t     tv_nsec;         // nanoseconds

        float       tv_accuracy;     // clock accuracy in seconds

} EP_TIME_SPEC;

#define EP_TIME_NOTIME      (-INT64_MAX)

#define EP_TIME_MAXTIME     (INT64_MAX)

EP_STAT

ep_time_now(                         // return current time

        EP_TIME_SPEC *tv);

EP_STAT

ep_time_deltanow(                    // return time in the future (or past)

        uint64_t delta_nanoseconds,

        EP_TIME_SPEC *tv);

void

ep_time_add_delta(                   // add a delta to a time (delta may be negative)

        EP_TIME_SPEC *delta,

        EP_TIME_SPEC *tv);

bool

ep_time_before(                      // determine if A occurred before B

        EP_TIME_SPEC *a,

        EP_TIME_SPEC *b);

void

ep_time_from_nsec(                   // create a time from a scalar number of nanoseconds

        int64_t nsec,

        EP_TIME_SPEC *tv);

void

ep_time_from_sec(                    // create a time from a scalar number of seconds

        int64_t sec,

        EP_TIME_SPEC *tv);

float

ep_time_accuracy(void);              // return putative clock accuracy

void

ep_time_setaccuracy(                 // set the clock accuracy (may not be available)

        float accuracy);

void

ep_time_format(                      // format a time string into a buffer

        EP_TIME_SPEC *tv,

        char *buf,

        size_t bufsize,

        uint32_t flags);

void

ep_time_print(                       // format a time string to a file

        EP_TIME_SPEC *tv,

        FILE *fp,

        uint32_t flags);

// values for ep_time_format and ep_time_print flags

#define EP_TIME_FMT_DEFAULT     0               // pseudo-flag

#define EP_TIME_FMT_HUMAN       0x00000001      // format for humans

#define EP_TIME_FMT_NOFUZZ      0x00000002      // suppress accuracy printing

EP_STAT

ep_time_parse(                       // parse a time string

        const char *timestr,

        EP_TIME_SPEC *tv,

        uint32_t flags);

// values for ep_time_parse flags

#define EP_TIME_USE_UTC         0x00000000      // assume UTC (default)

#define EP_TIME_USE_LOCALTIME   0x00000001      // assume times in local zone

EP_STAT

ep_time_nanosleep(                   // sleep for the indicated number of nanoseconds

        int64_t nanoseconds);

bool

EP_TIME_IS_VALID(                    // test to see if a timestamp is valid

        EP_TIME_SPEC *tv);

void

EP_TIME_INVALIDATE(                  // invalidate a timestamp

        EP_TIME_SPEC *tv);</programlisting>

    <para>"Human" formatted times are intended to be human readable, and may

    use non-ASCII characters. Otherwise the format is intended to be machine

    readable, e.g., using <function>ep_time_parse</function>.</para>

  </section>

  <section>

    <title>DATA STRUCTURES</title>

    <section>

      <title>Property Lists</title>

      <para><remark>Not implemented at this time.</remark> A series of

      key=value pairs. Used for many things, including configuration files.

      For example, looking in the "configuration" property list for

      "<varname>mailer.local.timeout.connect</varname>" would return the

      connect timeout for the local mailer. <remark>[[How does this deal with

      nested defaults — e.g., looking for timeout.connect if the full

      path cannot be found?]]</remark></para>

      <programlisting>    EP_PLIST *

    ep_plist_new(

                const char *name)              // for printing

    EP_STAT

    ep_plist_load(

                EP_PLIST *plp,                 // the list to read into

                FILE *sp,                      // the stream to load from

                const char *prefix)            // prefix added to all properties

    EP_STAT

    ep_plist_set(

                EP_PLIST *plp,                 // the plist in which to set

                const char *keyname,           // the name of the key to set

                const char *value)             // the value to set (will be copied)

    const char *

    ep_plist_get(

                EP_PLIST *plp,                 // the plist to search

                const char *keyname)           // the name of the key to get

    void

    ep_plist_dump(

                EP_PLIST *plp,                 // plist to print

                FILE *sp)                      // stream to print to

    void

    ep_plist_free(

                EP_PLIST *plp)                 // plist to free</programlisting>

      <para>A property list can be loaded from an external stream using

      <function>ep_plist_load</function>. The syntax of the file is a simple

      text file with "key=value" pairs on separate lines, with blank lines and

      those with # at the beginning of the line ignored. The values are

      strictly strings. <remark>[[Does it make sense to type

      them?]]</remark></para>

      <para><remark>[[Note the overlap between plists and the

      <function>ep_adm</function> interface. Does this make

      sense?]]</remark></para>

      <para>Property lists can be printed using

      <function>ep_plist_dump</function>. The output format will be readable

      by <function>ep_plist_load</function>. For the time being,

      <varname>flags</varname> should always be <constant>0</constant>.</para>

      <warning>

        <para>The property list is not guaranteed to be dumped in the same

        order items are inserted.</para>

      </warning>

    </section>

    <section>

      <title>Hashes</title>

      <programlisting>    #include &lt;ep/ep_hash.h&gt;

    EP_HASH *

    ep_hash_new(

                const char *name,               // for printing

                EP_HASH_HASH_FUNCP *hfunc,      // alternate hash function

                int tabsize)                    // hash table function size

    void

    ep_hash_free(

                EP_HASH *hp)                    // hash to free

    void *

    ep_hash_search(

                const EP_HASH *hp,              // hash to search

                size_t keylen,                  // length of key

                const void *key)                // pointer to key

    void *

    ep_hash_insert(                     // returns old value for key

                EP_HASH *hp,                    // hash to modify

                size_t keylen,                  // length of key

                const void *key,                // pointer to key

                void *val)                      // value to insert

    ep_hash_forall(EP_HASH *hp,                 // hash to walk

                void (func)(                    // function to call

                        int keylen,                // key length

                        const void *key,           // key value

                        void *val,                 // value

                        void *closure),            // from caller

                void *closure)                  // passed to func

    ep_hash_dump(EP_TREE *tree,                 // tree to dump

                FILE *sp)                       // stream to print on</programlisting>

      <para><remark>[[Should <function>ep_hash_dump</function> take the same

      parameters as the usual object print routine? For that matter, should

      there be a separate <function>ep_hash_dump</function> routine, or should

      it just be a generic <function>ep_obj_dump</function>? Note that

      <function>ep_hash_dump</function> is not implemented at this time, but

      an internal (object-based) dump is.]]</remark></para>

    </section>

    <section>

      <title>Function Lists</title>

      <programlisting>    #include &lt;ep/ep_funclist.h&gt;

    EP_FUNCLIST *

    ep_funclist_new(

                const char *name)       // name for printing/debugging

    void

    ep_funclist_free(EP_FUNCLIST *fp)   // list to free

    void

    ep_funclist_push(EP_FUNCLIST *fp,   // list to push to

                void (*func)(           // the function to invoke

                        void *closure,  // from the ep_funclist_push call

                        void *arg),     // from the ep_funclist_invoke call

                void *closure)          // the closure arg to pass to it

    void

    ep_funclist_pop(EP_FUNCLIST *fp)    // list to pop from, value discarded

    void

    ep_funclist_clear(EP_FUNCLIST *fp)  // list to clear

    void

    ep_funclist_invoke(EP_FUNCLIST *fp, // invoke all functions in list

                void *arg)              // second func arg</programlisting>

    </section>

  </section>

  <section>

    <title>CRYPTOGRAPHIC SUPPORT</title>

    <para>The current implementation wraps the OpenSSL library, but it could

    be retargeted.</para>

    <para>Before any cryptographic functions can be used, the library must be

    initialized:</para>

    <programlisting>void         ep_crypto_init(uint32_t flags)</programlisting>

    <para>At the moment flags is unused (just pass zero). There are also

    several general purpose definitions, useful for declaring buffers without

    memory allocation:</para>

    <informaltable border="1">

      <col width="4*"/>

      <col width="6*"/>

      <tr>

        <td><constant>EP_CRYPTO_MAX_PUB_KEY</constant></td>

        <td>Maximum length of a public key</td>

      </tr>

      <tr>

        <td><constant>EP_CRYPTO_MAX_SEC_KEY</constant></td>

        <td>Maximum length of a secret key</td>

      </tr>

      <tr>

        <td><constant>EP_CRYPTO_MAX_DIGEST</constant></td>

        <td>Maximum length of a message digest</td>

      </tr>

      <tr>

        <td><constant>EP_CRYPTO_MAX_DER</constant></td>

        <td>Maximum length of a DER-encoded key</td>

      </tr>

    </informaltable>

    <section>

      <title>Key Management</title>

      <para>Internally all keys are represented as <type>EP_CRYPTO_KEY</type>

      variables, defined in <filename>ep_crypto.h</filename>. External

      representations for keys may be either PEM (Privacy Enhanced Mail,

      represented as text) or DER (Distinguished Encoding Rules, represented

      in binary). PEM self identifies the type of key, but DER does not, so in

      some cases the key type needs to be pre-arranged.</para>

      <programlisting>// on-disk key formats

# define EP_CRYPTO_KEYFORM_UNKNOWN      0       // error

# define EP_CRYPTO_KEYFORM_PEM          1       // PEM (ASCII-encoded text)

# define EP_CRYPTO_KEYFORM_DER          2       // DER (binary ASN.1)</programlisting>

      <para>Internally, algorithms (e.g., for keys and hash/digest functions)

      are represented by a scalar value. Keys also have to be identified as

      public or secret.</para>

      <remark>[[Note: DH is not supported at this time.]]</remark>

      <programlisting>// key types

# define EP_CRYPTO_KEYTYPE_UNKNOWN      0       // error

# define EP_CRYPTO_KEYTYPE_RSA          1       // RSA

# define EP_CRYPTO_KEYTYPE_DSA          2       // DSA

# define EP_CRYPTO_KEYTYPE_EC           3       // Elliptic curve

# define EP_CRYPTO_KEYTYPE_DH           4       // Diffie-Hellman

// flag bits

# define EP_CRYPTO_F_PUBLIC             0x0000  // public key (no flags set)

# define EP_CRYPTO_F_SECRET             0x0001  // secret key</programlisting>

      <para>Keys are represented as an <type>EP_CRYPTO_KEY</type>. New keys

      can be created by giving the type of the key desired, the length of the

      key in bits, and two other values that are interpreted by the key type.

      The first is primarily for RSA and gives the exponent, and the second is

      primarily for EC and gives the curve name.</para>

      <programlisting>EP_CRYPTO_KEY           *ep_crypto_key_create(

                                int keytype,

                                int keylen,

                                int keyexp,

                                const char *curve);</programlisting>

      <para>Keys can be read from or written to named files, open files, or

      memory. All the read routines create and return a new key data

      structure. If the <varname>keyform</varname> is

      <constant>EP_CRYPTO_KEYFORM_PEM</constant> then the

      <varname>keytype</varname> need not be specified.</para>

      <programlisting>EP_CRYPTO_KEY           *ep_crypto_key_read_file(

                                const char *filename,

                                int keyform,

                                uint32_t flags);

EP_CRYPTO_KEY           *ep_crypto_key_read_fp(

                                FILE *fp,

                                const char *filename,

                                int keyform,

                                uint32_t flags);

EP_CRYPTO_KEY           *ep_crypto_key_read_mem(

                                const void *buf,

                                size_t buflen,

                                int keyform,

                                uint32_t flags);

EP_STAT                 ep_crypto_key_write_file(

                                EP_CRYPTO_KEY *key,

                                const char *filename,

                                int keyform,

                                int cipher,

                                uint32_t flags);

EP_STAT                 ep_crypto_key_write_fp(

                                EP_CRYPTO_KEY *key,

                                FILE *fp,

                                int keyform,

                                int cipher,

                                uint32_t flags);

EP_STAT                 ep_crypto_key_write_mem(

                                EP_CRYPTO_KEY *key,

                                void *buf,

                                size_t bufsize,

                                int keyform,

                                int cipher,

                                uint32_t flags);</programlisting>

      <para>When finished with a key it must be freed.</para>

      <programlisting>void                    ep_crypto_key_free(

                                EP_CRYPTO_KEY *key);</programlisting>

      <para>There are also some utility routines. A public and a secret key

      can be compared to see if they match each other (same algorithm,

      keysize, etc.) using <function>ep_crypto_key_compat</function>. Various

      conversions are also included:

      <function>ep_crypto_keyform_byname</function> converts a text string

      (e.g., "pem") to an internal code,

      <function>ep_crypto_keytype_fromkey</function> returns the type of a

      key, and <function>ep_crypto_keytype_byname</function> converts a text

      string to a type.</para>

      <programlisting>EP_STAT                 ep_crypto_key_compat(

                                const EP_CRYPTO_KEY *pubkey,

                                const EP_CRYPTO_KEY *seckey);

int                     ep_crypto_keyform_byname(

                                const char *fmt);

int                     ep_crypto_keytype_fromkey(

                                EP_CRYPTO_KEY *key);

int                     ep_crypto_keytype_byname(

                                const char *alg_name);</programlisting>

    </section>

    <section>

      <title>Message Digests (Hashes)</title>

      <para>Several message digest (cryptographic hash) algorithms are

      supported. Text can be converted to one of these values, and the

      algorithm type can be extracted from the internal form.</para>

      <programlisting>// digest algorithms (no more than 4 bits)

# define EP_CRYPTO_MD_NULL      0

# define EP_CRYPTO_MD_SHA1      1

# define EP_CRYPTO_MD_SHA224    2

# define EP_CRYPTO_MD_SHA256    3

# define EP_CRYPTO_MD_SHA384    4

# define EP_CRYPTO_MD_SHA512    5

int                     ep_crypto_md_alg_byname(

                                const char *algname);

int                     ep_crypto_md_type(

                                EP_CRYPTO_MD *md);</programlisting>

      <para>Digests (type <type>EP_CRYPTO_MD</type>) can be created, freed,

      and cloned. Cloning lets an application compute the a fixed part of a

      digest (perhaps an unchanging header) and then produce separate digests

      for individual records.</para>

      <programlisting>EP_CRYPTO_MD            *ep_crypto_md_new(

                                int md_alg_id);

EP_CRYPTO_MD            *ep_crypto_md_clone(

                                EP_CRYPTO_MD *base_md);

void                    ep_crypto_md_free(

                                EP_CRYPTO_MD *md);</programlisting>

      <para>The typical lifetime of a digest is to be created (as above),

      updated with additional data, possibly multiple times, and then

      finalized to give the output hash.</para>

      <programlisting>EP_STAT                 ep_crypto_md_update(

                                EP_CRYPTO_MD *md,

                                void *data,

                                size_t dsize);

EP_STAT                 ep_crypto_md_final(

                                EP_CRYPTO_MD *md,

                                void *dbuf,

                                size_t *dbufsize);</programlisting>

    </section>

    <section>

      <title>Signing and Verification</title>

      <para>Signing and verification are quite similar. A new internal

      structure is created, using the same type as a message digest, data is

      added to the existing hash, possibly multiple times, the signature is

      created or verified, and finally the structure is freed.</para>

      <programlisting># define EP_CRYPTO_MAX_SIG      (1024 * 8)

EP_CRYPTO_MD            *ep_crypto_sign_new(

                                EP_CRYPTO_KEY *skey,

                                int md_alg_id);

void                    ep_crypto_sign_free(

                                EP_CRYPTO_MD *md);

EP_STAT                 ep_crypto_sign_update(

                                EP_CRYPTO_MD *md,

                                void *dbuf,

                                size_t dbufsize);

EP_STAT                 ep_crypto_sign_final(

                                EP_CRYPTO_MD *md,

                                void *sbuf,

                                size_t *sbufsize);

EP_CRYPTO_MD            *ep_crypto_vrfy_new(

                                EP_CRYPTO_KEY *pkey,

                                int md_alg_id);

void                    ep_crypto_vrfy_free(

                                EP_CRYPTO_MD *md);

EP_STAT                 ep_crypto_vrfy_update(

                                EP_CRYPTO_MD *md,

                                void *dbuf,

                                size_t dbufsize);

EP_STAT                 ep_crypto_vrfy_final(

                                EP_CRYPTO_MD *md,

                                void *obuf,

                                size_t obufsize);</programlisting>

    </section>

    <section>

      <title>Encryption and Decryption (Asymmetric)</title>

      <para>To be supplied.</para>

    </section>

    <section>

      <title>Encryption and Decryption (Symmetric Ciphers)</title>

      <para>Symmetric Ciphers are driven by a Chaining Mode (how subsequent

      blocks have the key modified to prevent replay and brute force attacks)

      and the actual cipher itself. The chaining modes are:</para>

      <informaltable border="1">

        <col width="4*"/>

        <col width="6*"/>

        <tr>

          <td width=""><constant>EP_CRYPTO_MODE_CBC</constant></td>

          <td>Cipher Block Chaining</td>

        </tr>

        <tr>

          <td width=""><constant>EP_CRYPTO_MODE_CFB</constant></td>

          <td>Cipher Feedback mode</td>

        </tr>

        <tr>

          <td width=""><constant>EP_CRYPTO_MODE_OFB</constant></td>

          <td>Output Feedback mode</td>

        </tr>

      </informaltable>

      <para>The various cipher algorithms (which is equivalent to the key

      type) are:</para>

      <informaltable border="1">

        <col width="4*"/>

        <col width="6*"/>

        <tr>

          <td><constant>EP_CRYPTO_SYMKEY_NONE</constant></td>

          <td>Error/unencrypted</td>

        </tr>

        <tr>

          <td><constant>EP_CRYPTO_SYMKEY_AES128</constant></td>

          <td>Advanced Encr Std, 128-bit key</td>

        </tr>

        <tr>

          <td><constant>EP_CRYPTO_SYMKEY_AES192</constant></td>

          <td>Advanced Encr Std, 192-bit key</td>

        </tr>

        <tr>

          <td><constant>EP_CRYPTO_SYMKEY_AES256</constant></td>

          <td>Advanced Encr Std, 256-bit key</td>

        </tr>

        <tr>

          <td><constant>EP_CRYPTO_SYMKEY_CAMELLIA128</constant></td>

          <td>Camellia, 128-bit key</td>

        </tr>

        <tr>

          <td><constant>EP_CRYPTO_SYMKEY_CAMELLIA192</constant></td>

          <td>Camellia, 192-bit key</td>

        </tr>

        <tr>

          <td><constant>EP_CRYPTO_SYMKEY_CAMELLIA256</constant></td>

          <td>Camellia, 256-bit key</td>

        </tr>

        <tr>

          <td><constant>EP_CRYPTO_SYMKEY_DES</constant></td>

          <td>Data Encryption Standard, single, 56-bit key</td>

        </tr>

        <tr>

          <td><constant>EP_CRYPTO_SYMKEY_3DES</constant></td>

          <td>Data Encryption Standard, triple, 128-bit key (112-bit

          effective)</td>

        </tr>

        <tr>

          <td><constant>EP_CRYPTO_SYMKEY_IDEA</constant></td>

          <td>International Data Encryption Alg, 128-bit key</td>

        </tr>

      </informaltable>

      <para>One value from each table are "or"ed together to specify a full

      symmetric cipher. The rest of the interface is as follows:</para>

      <programlisting>/*

**      The cipher is set to encrypt or decrypt when the context

**      is created.

**

**      ep_crypto_cipher_crypt is just shorthand for a single

**      call to ep_crypto_cipher_update followed by a single

**      call to ep_crypto_cipher_final.  Final pads out any

**      remaining block and returns that data.

*/

EP_CRYPTO_CIPHER_CTX    *ep_crypto_cipher_new(

                                uint32_t ciphertype,    // mode + keytype & len

                                uint8_t *key,           // the key

                                uint8_t *iv,            // initialization vector

                                bool enc);              // true => encrypt

void                    ep_crypto_cipher_free(

                                EP_CRYPTO_CIPHER_CTX *cipher);

EP_STAT                 ep_crypto_cipher_crypt(

                                EP_CRYPTO_CIPHER_CTX *cipher,

                                void *in,               // input data

                                size_t inlen,           // input length

                                void *out,              // output buffer

                                size_t outlen);         // output buf size

EP_STAT                 ep_crypto_cipher_update(

                                EP_CRYPTO_CIPHER_CTX *cipher,

                                void *in,               // input data

                                size_t inlen,           // input length

                                void *out,              // output buffer

                                size_t outlen);         // output buf size

EP_STAT                 ep_crypto_cipher_final(

                                EP_CRYPTO_CIPHER_CTX *cipher,

                                void *out,              // output buffer

                                size_t outlen);         // output buf size</programlisting>

    </section>

    <section>

      <title>Cryptography-specific Error Codes</title>

      <para>There are several status codes that may be returned from the

      cryptography routines. These are all in module

      EP_STAT_MOD_CRYPTO.</para>

      <informaltable border="1">

        <col width="4*"/>

        <col width="6*"/>

        <tr>

          <td><errorcode>EP_STAT_CRYPTO_DIGEST</errorcode></td>

          <td>Failed to update or finalize a digest (hash)</td>

        </tr>

        <tr>

          <td><errorcode>EP_STAT_CRYPTO_SIGN</errorcode></td>

          <td>Failed to update or finalize a digest for signing</td>

        </tr>

        <tr>

          <td><errorcode>EP_STAT_CRYPTO_VRFY</errorcode></td>

          <td>Failed to update or finalize a digest for verification</td>

        </tr>

        <tr>

          <td><errorcode>EP_STAT_CRYPTO_BADSIG</errorcode></td>

          <td>Signature did not match</td>

        </tr>

        <tr>

          <td><errorcode>EP_STAT_CRYPTO_KEYTYPE</errorcode></td>

          <td>Unknown key type</td>

        </tr>

        <tr>

          <td><errorcode>EP_STAT_CRYPTO_KEYFORM</errorcode></td>

          <td>Unknown key format</td>

        </tr>

        <tr>

          <td><errorcode>EP_STAT_CRYPTO_CONVERT</errorcode></td>

          <td>Couldn't read or write a key</td>

        </tr>

        <tr>

          <td><errorcode>EP_STAT_CRYPTO_KEYCREATE</errorcode></td>

          <td>Couldn't create a new key</td>

        </tr>

        <tr>

          <td><errorcode>EP_STAT_CRYPTO_KEYCOMPAT</errorcode></td>

          <td>Public and secret keys are incompatible</td>

        </tr>

        <tr>

          <td><errorcode>EP_STAT_CRYPTO_CIPHER</errorcode></td>

          <td>Symmetric cipher failure</td>

        </tr>

      </informaltable>

    </section>

  </section>

  <section>

    <title>APPLICATION SUPPORT</title>

    <para>The following routines are intended to provide useful support to

    applications, but are not otherwise fundamental</para>

    <section>

      <title>Printing Flag Words, Etc.</title>

      <programlisting>    #include &lt;ep/ep_prflags.h&gt;

    void

    ep_prflags(