chickadee » protobuf

Outdated egg!

This is an egg for CHICKEN 4, the unsupported old release. You're almost certainly looking for the CHICKEN 5 version of this egg, if it exists.

If it does not exist, there may be equivalent functionality provided by another egg; have a look at the egg index. Otherwise, please consider porting this egg to the current version of CHICKEN.

protobuf

Description

The egg provides facilities to read and write Protocol Buffers in binary format and comes with a plug-in for the protoc compiler that generates CHICKEN code from a .proto definition file.

Programming Interface

Most of the time you will run protoc --chicken_out=... on your definition files and import only the generated modules plus the protobuf module from the protobuf library into your code. The generated modules contain SRFI-99 record types and a convenient constructur for every protocol buffer message type, as well as reflection information used by the (de-)serialization process for every protocol buffer message or enumeration type. The generated modules are written in terms of protobuf-syntax.

If you don't want to bother with application specific protocol buffer definitions, you can also use the protobuf-generic module to serialize almost any value you might encounter in a CHICKEN program.

Additionally the protobuf library contains modules that allow you to hook into the protocol buffer (de-)serialization process or perform low-level manipulations of the wire format manually.

protobuf

This module contains the high-level entry points for protocol buffer (de-)serialization and re-exports message?, message-extensions and message-unknown from the protobuf-reflection module.

deserialize TYPE #!optional PORTprocedure

Deserializes a binary protocol buffer stream from PORT (or (current-input-port)) into a record of the given TYPE, which must be the SRFI-99 record type descriptor of a protocol buffer message type.

serialize MSG #!optional PORTprocedure

Serializes the protocol buffer message MSG to the PORT (or (current-output-port)) as a binary protocol buffer stream. MSG must be a SRFI-99 record instance of a protocol buffer message type.

protobuf-generic

This module contains the high-level entry points for protocol buffer (de-)serialization of CHICKEN values without any application specific schema definition. The module code is optimized for this task and uses only the protobuf-encoding facilities. It emits protocol buffers conforming to the schema defined in extend/protobuf/chicken.proto.

The following values can be serialized:

current-serialization-contextparameter

Holds the current default serialization context, an object that tracks the identity of values that have already been serialized or deserialized.

The parameter value may also be #f, which is its default value.

(make-serialization-context [V ...])procedure

Creates a serialization context and initializes it with a set of well known values. This can be used to replace certain non-serializable values with "external" references upon serialization and to inject those external dependencies upon deserialization. The context always sees itself as an external dependency.

serialization-context? Vprocedure

Checks whether something is a serialization context.

serialization-infotype

A SRFI-99 record type holding a custom reader and writer procedures in its two immutable fields.

prop:serialization-infoconstant

A record property that associates a serialization-info structure with SRFI-99 record types and instances.

(serialize V [PORT] [CONTEXT])procedure

Writes the value V in generic protocol buffer format to the PORT (defaulting to (current-output-port)). The given CONTEXT (defaulting to (current-serialization-context)) is used to track serialized values, allowing the correct external representation of shared structure.

If the context is #f, a new context is allocated and initialized with the current input, output and error ports as external dependencies.

(deserialize [PORT] [CONTEXT])procedure

Reads a value in generic protocol buffer format from the PORT (defaulting to (current-input-port)). The given CONTEXT (defaulting to (current-serialization-context)) is used to track deserialized values, allowing the correct reconstruction of shared structure.

If the context is #f, a new context is allocated and initialized with the current input, output and error ports as external dependencies.

protobuf-reflection

This module contains the data structures used by the (de-)serialization code to guide its operation. It can also be used for purposes of introspection.

type-infotype

A SRFI-99 record type that serves as the basis for primitive, enumeration or message type descriptors. It contains just an immutable name field.

This abstract type has no constructor.

primitive-infotype

The record type of descriptors for primitive protocol buffer wire types. It contains

  • an immutable type field holding the uninterpreted wire type of the primitive,
  • an immutable reader field holding a read-like procedure to deserialize an instance of the primitive and
  • an immutable writer field holding a write-like procedure to serialize an instance of the primitive.
enum-infotype

The record type of descriptors for protocol buffer enumeration types. It contains

  • an immutable integer->enum field holding a procedure that converts the integer representation of an enumeration item into a symbol and
  • an immutable enum->integer field holding a procedure that converts the symbolic representation of an enumeration item into an integer.
message-infotype

The record type of descriptors for protocol buffer message types. It contains

  • an immutable constructor field holding a procedure that creates a new message instance taking only keyword arguments to fill the instance fields,
  • an immutable fields field holding a hash table that maps numeric field tags to field information records and
  • an immutable required field holding a hash table that maps the numeric tags of required fields to their information records.
field-infotype

The record type of message field descriptors. It contains

  • an immutable type field holding a promise,
  • an immutable repeated? field holding a boolean,
  • an immutable packed? field holding a boolean,
  • an immutable accessor field holding a procedure from instances of the containing message type to values of the field, with an optional default argument, and
  • an immutable mutator field holding a procedure that sets the field value in an instance of the containing message type.

The type field refers to a SRFI-99 record type descriptor of another protocol buffer message or to an enumeration or primitive type descriptor record. The field-info-type accessor automatically forces the promise contained in this field.

messagetype

A SRFI-99 record type that serves as the basis for any protocol buffer message type. It contains

  • an immutable extensions field containing a hash table that maps extension tags to their deserialized values and
  • a mutable unknown field containing any serial data that could not be interpreted.

This abstract type has no constructor.

protobuf-syntax

This module contains the syntactic forms used by modules created through protoc-gen-chicken.

int32constant
int64constant
uint32constant
uint64constant
sint32constant
sint64constant
fixed32constant
fixed64constant
sfixed32constant
sfixed64constant
boolconstant
floatconstant
doubleconstant
bytesconstant
stringconstant

Descriptors for the standard primitive protocol buffer types.

uint* MAX-SIZEprocedure
sint* MAX-SIZEprocedure

Constructors for primitive type descriptors compatible with standard unsigned or signed integers, but with arbitrary maximum encoding size. MAX-SIZE can also be #f to allow arbitrary precision integers.

The protocol compiler plugin allows access to these types through a max_size extension to the google.protobuf.FieldOptions message defined in extend/protobuf/bigint.proto.

(define-enum-type NAME (ITEM TAG) ...)syntax

Defines a new enumeration type.

(define-message-type NAME ({required|optional|repeated|packed} TYPE FIELD TAG [DEFAULT]) ...)syntax

Defines a new message type, which is a SRFI-99 record type with a specialized constructor.

The constructors for protocol buffer messages take a keyword argument for every field. Missing fields are initialized to their defaults or a void value.

(define-message-extension NAME ({required|optional|repeated|packed} TYPE FIELD TAG [DEFAULT]) ...)syntax

Extends an existing message type with additional synthetic fields.

protobuf-encoding

This module forms the low-level layer for protocol buffer wire format encoding and decoding.

make-limited-input-port PORT LIMIT CLOSE-ORIG?procedure

Creates an input port reading up to LIMIT bytes from PORT before entering an end-of-file state.

(read-uint* [PORT] [MAX-SIZE])procedure

Reads a variable length base 128 encoded unsigned integer from PORT (defaulting to (current-input-port)). Stops with a syntax error if the encoding is longer than MAX-SIZE bytes (defaulting to 10); pass #f to disable the length check.

(write-uint* N [PORT] [MAX-SIZE])procedure

Writes a variable length base 128 encoded unsigned integer to PORT (defaulting to (current-output-port)). Stops with a syntax error if the encoding is longer than MAX-SIZE bytes (defaulting to 10); pass #f to disable the length check.

(read-sint* [PORT] [MAX-SIZE])procedure

Reads a variable length base 128 zig-zag-encoded signed integer from PORT (defaulting to (current-input-port)). Stops with a syntax error if the encoding is longer than MAX-SIZE bytes (defaulting to 10); pass #f to disable the length check.

(write-sint* I [PORT] [MAX-SIZE])procedure

Writes a variable length base 128 zig-zag-encoded signed integer to PORT (defaulting to (current-output-port)). Stops with a syntax error if the encoding is longer than MAX-SIZE bytes (defaulting to 10); pass #f to disable the length check.

read-int* #!optional PORTprocedure

Reads a variable length base 128 two's complement encoded signed integer from PORT (defaulting to (current-input-port)). Stops with a syntax error if the encoding is longer than 10 bytes.

write-int* I #!optional PORTprocedure

Writes a variable length base 128 two's complement encoded signed integer to PORT (defaulting to (current-output-port)). Stops with a syntax error if the encoding is longer than 10 bytes.

read-bool #!optional PORTprocedure

Reads a boolean from PORT (defaulting to (current-input-port)).

write-bool V #!optional PORTprocedure

Writes a boolean to PORT (defaulting to (current-output-port)).

SIZESIGNED? #!optional PORTprocedure

Reads a fixed length, possibly SIGNED?, integer from PORT (defaulting to (current-input-port)). SIZE must be 4 or 8.

SIZESIGNED? N #!optional PORTprocedure

Writes a fixed length, possibly SIGNED?, integer to PORT (defaulting to (current-output-port)). SIZE must be 4 or 8.

read-fixed32constant

An alias for (read-fixed* 4 #f).

write-fixed32constant

An alias for (write-fixed* 4 #f).

read-fixed64constant

An alias for (read-fixed* 8 #f).

write-fixed64constant

An alias for (write-fixed* 8 #f).

read-sfixed32constant

An alias for (read-fixed* 4 #t).

write-sfixed32constant

An alias for (write-fixed* 4 #t).

read-sfixed64constant

An alias for (read-fixed* 8 #t).

write-sfixed64constant

An alias for (write-fixed* 8 #t).

SIZE #!optional PORTprocedure

Reads a fixed length floating point number from PORT (defaulting to (current-input-port)). SIZE must be 4 or 8.

SIZE X #!optional PORTprocedure

Writes a fixed length floating point number to PORT (defaulting to (current-output-port)). SIZE must be 4 or 8.

read-floatconstant

An alias for (read-float* 4).

write-floatconstant

An alias for (write-float* 4).

read-doubleconstant

An alias for (read-float* 8).

write-doubleconstant

An alias for (write-float* 8).

read-sized-bytes #!optional PORTprocedure

Reads a SRFI-4 u8vector with size information from PORT (defaulting to (current-input-port)).

write-sized-bytes BSTR #!optional PORTprocedure

Writes a SRFI-4 u8vector with size information to PORT (defaulting to (current-output-port)).

read-sized-string #!optional PORTprocedure

Reads a string with size information from PORT (defaulting to (current-input-port)).

write-sized-string STR #!optional PORTprocedure

Writes a string with size information to PORT (defaulting to (current-output-port)).

read-sized READ #!optional PORTprocedure

Reads a block with size information from PORT (defaulting to (current-input-port)). Uses READ to decode the contents of the block.

write-sized WRITE V #!optional PORTprocedure

Uses WRITE to encode a block to PORT (defaulting to (current-output-port)). Automatically adds size information to the output.

read-tag/type #!optional PORTprocedure

Reads the tag and wire type of a protocol buffer message field from PORT (defaulting to (current-input-port)). The type is one of the symbols int*, 32bit, 64bit or sized.

write-tag/type TAG TYPE #!optional PORTprocedure

Writes the tag and wire type of a protocol buffer message field to PORT (defaulting to (current-output-port)). The TYPE must be one of the symbols int*, 32bit, 64bit or sized.

Example

$ cat msg.proto
message Msg {
  optional uint64 ts = 2;
  optional string data = 3; 
}


$ protoc --chicken_out=. msg.proto
$ cat msg.scm
;; Generated by protoc-gen-chicken v1.0.0
(module
  main
  *
  (import (except scheme string) chicken protobuf-syntax)
  (define-message-type msg (optional uint64 ts 2) (optional string data 3)))


$ cat serializer.scm
(use protobuf) (include "msg.scm") (import main)
(serialize (make-msg data: "hello world"))
;; obs: don't concat multiple messages,
;; you need a message boundary.


$ cat deserialize.scm
(use protobuf) (include "msg.scm") (import main)
(pp (msg-data (deserialize msg)))


$ csi -s serializer.scm | csi -s deserialize.scm
"hello world"

Author

Thomas Chust

Requirements

License

 Copyright (C) 2013 Thomas Chust <chust@web.de>.  All rights reserved.
 
 Permission is hereby granted, free of charge, to any person obtaining a
 copy of this software and associated documentation files (the Software),
 to deal in the Software without restriction, including without limitation
 the rights to use, copy, modify, merge, publish, distribute, sublicense,
 and/or sell copies of the Software, and to permit persons to whom the
 Software is furnished to do so, subject to the following conditions:
 
 The above copyright notice and this permission notice shall be included
 in all copies or substantial portions of the Software.
 
 THE SOFTWARE IS PROVIDED ASIS, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
 THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR
 OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
 ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
 OTHER DEALINGS IN THE SOFTWARE.

Contents »