protobufs.pl -- Google's Protocol Buffers ("protobufs")protobufs.pl -- Google's Protocol Buffers ("protobufs")Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data -- think XML, but smaller, faster, and simpler. You define how you want your data to be structured once. This takes the form of a template that describes the data structure. You use this template to encode and decode your data structure into wire-streams that may be sent-to or read-from your peers. The underlying wire stream is platform independent, lossless, and may be used to interwork with a variety of languages and systems regardless of word size or endianness. Techniques exist to safely extend your data structure without breaking deployed programs that are compiled against the "old" format.
The idea behind Google's Protocol Buffers is that you define your structured messages using a domain-specific language and tool set. Further documentation on this is at https://developers.google.com/protocol-buffers.
There are two ways you can use protobufs in Prolog:
.proto file: protobuf_parse_from_codes/3 and
protobuf_serialize_to_codes/3.
The protobuf_parse_from_codes/3 and protobuf_serialize_to_codes/3
interface translates between a "wire stream" and a Prolog term. This
interface takes advantage of SWI-Prolog's
dict.
The protoc plugin (protoc-gen-swipl) generates a
Prolog file of meta-information that captures the .proto file's
definition in the protobufs module, with the following facts:
proto_meta_normalize(Unnormalized, Normalized)proto_meta_package(Package, FileName, Options)proto_meta_message_type(Fqn, Package, Name)proto_meta_message_type_map_entry(Fqn)proto_meta_field_name(Fqn, FieldNumber, FieldName, FqnName)proto_meta_field_json_name(FqnName, JsonName)proto_meta_field_label(FqnName, LabelRepeatOptional) % 'LABEL_OPTIONAL', 'LABEL_REQUIRED', 'LABEL_REPEATED'proto_meta_field_type(FqnName, Type) % 'TYPE_INT32', 'TYPE_MESSAGE', etcproto_meta_field_type_name(FqnName, TypeName)proto_meta_field_default_value(FqnName, DefaultValue)proto_meta_field_option_packed(FqnName)proto_meta_enum_type(FqnName, Fqn, Name)proto_meta_enum_value(FqnName, Name, Number)proto_meta_field_oneof_index(FqnName, Index)proto_meta_oneof(FqnName, Index, Name)
The protobuf_message/2 interface allows you to define your message
template as a list of predefined
Prolog terms that correspond to production rules in the Definite Clause
Grammar (DCG) that realizes the interpreter. Each production rule has an
equivalent rule in the protobuf grammar. The process is not unlike
specifiying the format of a regular expression. To encode a template to
a wire-stream, you pass a grounded template, X, and variable, Y, to
protobuf_message/2. To decode a wire-stream, Y, you pass an ungrounded
template, X, along with a grounded wire-stream, Y, to
protobuf_message/2. The interpreter will unify the unbound variables in
the template with values decoded from the wire-stream.
For an overview and tutorial with examples, see
library(protobufs): Google's Protocol Buffers
Examples of usage may also be found by inspecting
test_protobufs.pl
and the
demo
directory, or by looking at the "addressbook" example that is typically
installed at
/usr/lib/swi-prolog/doc/packages/examples/protobufs/interop/addressbook.pl
protobuf_parse_from_codes(+WireCodes:list(int), +MessageType:atom, -Term) is semidetMessageType), creating a Prolog term.
Protoc must have been run (with the --swipl_out= option and the resulting
top-level _pb.pl file loaded. For more details, see the "protoc" section of the
overview documentation.
Fails if the message can't be parsed or if the appropriate meta-data from protoc
hasn't been loaded.
All fields that are omitted from the WireCodes are set to their
default values (typically the empty string or 0, depending on the
type; or [] for repeated groups). There is no way of testing
whether a value was specified in WireCodes or given its default
value (that is, there is no equivalent of the Python
implementation's =HasField`). Optional embedded messages and groups
do not have any default value -- you must check their existence by
using get_dict/3 or similar. If a field is part of a "oneof" set,
then none of the other fields is set. You can determine which field
had a value by using get_dict/3.
protobuf_serialize_to_codes(+Term:dict, -MessageType:atom, -WireCodes:list(int)) is detMessageType).
Protoc must have been run (with the --swipl_out= option and the resulting
top-level _pb.pl file loaded. For more details, see the "protoc" section of the
overview documentation.
Fails if the term isn't of an appropriate form or if the appropriate
meta-data from protoc hasn't been loaded, or if a field name is incorrect
(and therefore nothing in the meta-data matches it).
protobuf_var_int(?A:int)// is det[private]A is negative.
This is a low-level predicate; normally, you should use
template_message/2 and the appropriate template term.
e.g. phrase(protobuf_var_int(300), S) => S = [172,2]
phrase(protobuf_var_int(A), [172,2]) -> A = 300 protobuf_tag_type(?Tag:int, ?WireType:atom)// is det[private] prolog_type(?Tag:int, ?PrologType:atom)// is semidet[private]Payload, according to PrologType
TODO: payload//2 "mode" is sometimes module-sensitive, sometimes not.
payload(enum, A)// has A as a callable
all other uses of payload//2, the 2nd arg is not callable.
vector_demo.pl, which defines commands/2) single_message(+PrologType:atom, ?Tag, ?Payload)// is det[private]protobuf([...]).
The PrologType, Tag, Payload are from Field =.. [PrologType, Tag, Payload]
in the caller protobuf_message(?Template, ?WireStream) is semidet protobuf_message(?Template, ?WireStream, ?Rest) is nondet protobuf_segment_message(+Segments:list, -WireStream:list(int)) is det[private].proto description, similar to
the processing done by protoc --decode_raw. This means that
field names aren't shown, only field numbers.
For unmarshalling, a simple heuristic is used on length-delimited segments: first interpret it as a message; if that fails, try to interpret as a UTF8 string; otherwise, leave it as a "blob" (if the heuristic was wrong, you can convert to a string or a blob by using protobuf_segment_convert/2). 32-bit and 64-bit numbers are left as codes because they could be either integers or floating point (use int32_codes_when/2, float32_codes_when/2, int64_codes_when/2, uint32_codes_when/2, uint64_codes_when/2, float64_codes_when/2 as appropriate); variable-length numbers ("varint" in the Protocol Buffers encoding documentation), might require "zigzag" conversion, int64_zigzag_when/2.
For marshalling, use the predicates int32_codes_when/2, float32_codes_when/2, int64_codes_when/2, uint32_codes_when/2, uint64_codes_when/2, float64_codes_when/2, int64_zigzag_when/2 to put integer and floating point values into the appropriate form.
detag(+Compound, -Name, -Tag, -Value, List, -CompoundWithList) is semidet[private]Compound or the form Name(Tag,Value) and create a
new CompoundWithList that replaces Value with List. This is
used by packed_list/2 to transform [varint(1,0),varint(1,1)] to
varint(1,[0,1]).
Some of Compound items are impossible for packed with the
current protobuf spec, but they don't do any harm.
protobuf_segment_convert(+Form1, ?Form2) is multi[private]
Form1 is converted back to the original wire stream, then the
predicate non-deterimisticly attempts to convert the wire stream to
a string or length_delimited term (or both: the lattter
always succeeds).
The possible conversions are:
message(Tag,Segments) => string(Tag,String)
message(Tag,Segments) => length_delimited(Tag,Codes)
string(Tag,String) => length_delimited(Tag,Codes)
length_delimited(Tag,Codes) => length_delimited(Tag,Codes)
Note that for fixed32, fixed64, only the signed integer forms are given; if you want the floating point forms, then you need to do use int64_float64_when/2 and int32_float32_when/2.
For example:
?- protobuf_segment_convert(
message(10,[fixed64(13,7309475598860382318)]),
string(10,"inputType")).
?- protobuf_segment_convert(
message(10,[fixed64(13,7309475598860382318)]),
length_delimited(10,[105,110,112,117,116,84,121,112,101])).
?- protobuf_segment_convert(
string(10, "inputType"),
length_delimited(10,[105,110,112,117,116,84,121,112,101])).
?- forall(protobuf_segment_convert(string(1999,"\x1\\x0\\x0\\x0\\x2\\x0\\x0\\x0\"),Z), writeln(Z)).
string(1999, )
packed(1999,fixed64([8589934593]))
packed(1999,fixed32([1,2]))
packed(1999,varint([1,0,0,0,2,0,0,0]))
length_delimited(1999,[1,0,0,0,2,0,0,0])
These come from:
Codes = [82,9,105,110,112,117,116,84,121,112,101], protobuf_message(protobuf([embedded(T1, protobuf([integer64(T2, I)]))]), Codes), protobuf_message(protobuf([string(T,S)]), Codes). T = 10, T1 = 10, T2 = 13, I = 7309475598860382318, S = "inputType".
uint32_codes_when(?Uint32, ?Codes) is det[private]
This predicate delays until either Uint32 or Codes is
sufficiently instantiated.
There is also a non-delayed uint32_codes/2
SWI-Prolog doesn't have a 32-bit integer type, so 32-bit integer is simulated by doing a range check.
int32_codes_when(?Int32, ?Codes) is det[private]
This predicate delays until either Int32 or Codes is
sufficiently instantiated.
There is also a non-delayed int32_codes/2
SWI-Prolog doesn't have a 32-bit integer type, so 32-bit integer is simulated by doing a range check.
float32_codes_when(?Value, ?Codes) is det[private]
This predicate delays until either Value or Codes is
sufficiently instantiated.
There is also a non-delayed float32_codes/2
uint64_codes_when(?Uint64, ?Codes) is det[private]SWI-Prolog allows integer values greater than 64 bits, so a range check is done.
This predicate delays until either Uint64 or Codes is
sufficiently instantiated.
There is also a non-delayed uint64_codes/2
int64_codes_when(?Int64, ?Codes) is det[private]SWI-Prolog allows integer values greater than 64 bits, so a range check is done.
This predicate delays until either Int64 or Codes is
sufficiently instantiated.
There is also a non-delayed int64_codes/2
float64_codes_when(?Value, ?Codes) is det[private]
This predicate delays until either Value or Codes is
sufficiently instantiated.
There is also a non-delayed float64_codes/2
int64_zigzag_when(?Original, ?Encoded) is det[private]sint32 and sint64 types. This is a
low-level predicate; normally, you should use template_message/2 and
the appropriate template term.
SWI-Prolog allows integer values greater than 64 bits, so a range check is done.
This predicate delays until either Original or Encoded is
sufficiently instantiated.
There is also a non-delayed int64_zigzag/2
uint64_int64_when(?Uint64:integer, ?Int64:integer) is det[private]uint64_int64(0xffffffffffffffff,-1).
This predicate delays until either Uint64 or Int64 is
sufficiently instantiated.
There is also a non-delayed uint64_int64/2
uint32_int32_when(?Uint32, ?Int32) is det[private]
This predicate delays until either Uint32 or Int32 is
sufficiently instantiated.
There is also a non-delayed uint32_int32/2
int64_float64_when(?Int64:integer, ?Float64:float) is det[private]int64_float64(3ff0000000000000,1.0).
This predicate delays until either Int64 or Float64 is
sufficiently instantiated.
There is also a non-delayed uint64_int64/2
int32_float32_when(?Int32:integer, ?Float32:float) is det[private]int32_float32(0x3f800000,1.0).
This predicate delays until either Int32 or Float32 is
sufficiently instantiated.
There is also a non-delayed uint32_int32/2
segment_to_term(+ContextType:atom, +Segment, -FieldAndValue) is det[private]TODO: if performance is an issue, this code can be combined with
protobuf_segment_message/2 (and thereby avoid the use of protobuf_segment_convert/2) convert_segment(+Type:atom, +ContextType:atom, Tag:atom, ?Segment, ?Value) is det[private]Value from the combination of descriptor
"type" (in Type) and a Segment.
Reversible on Segment, Values. add_defaulted_fields(+Value0:dict, ContextType:atom, -Value:dict) is det[private] message_field_default(+ContextType:atom, Name:atom, -DefaultValue) is semidet[private] combine_fields(+Fields:list, +MsgDict0, -MsgDict) is det[private] combine_fields_repeat(+Fields:list, Field:atom, -Values:list, RestFields:list) is det[private]Field - the assumption
is that all the items for a field will be together and if they're
not, they would be combined outside this predicate.
field_and_type(+ContextType:atom, +Tag:int, -FieldName:atom, -FqnName:atom, -ContextType2:atom, -RepeatOptional:atom, -Type:atom) is det[private]ContextType and Tag to get the field name, type, etc. fqn_repeat_optional(+FqnName:atom, -RepeatOptional:atom) is det[private]proto_meta_field_label(FqnName, _), proto_meta_field_option_packed(FqnName)
and set RepeatOptional to one of
norepeat, repeat, repeat_packed. fqn_repeat_optional_2(+DescriptorLabelEnum:atom, -RepeatOrEmpty:atom) is det[private]proto_meta_enum_value('.google.protobuf.FieldDescriptorProto.Label', Label, _). field_descriptor_label_repeated(+Label:atom) is semidet[private]proto_meta_enum_value('.google.protobuf.FieldDescriptorProto.Label', 'LABEL_REPEATED', _).
TODO: unused field_descriptor_label_single(+Label:atom) is semidet[private]proto_meta_enum_value('.google.protobuf.FieldDescriptorProto.Label', Label, _). term_to_segments(+Term:dict, +MessageType:atom, Segments) is det[private]Term, generating message segments field_segment_scalar_or_repeated(+Label, +Packed, +FieldType, +Tag, +FieldTypeName, ?Value, Segment) is det[private]FieldType is from the .proto meta information ('TYPE_SINT32', etc.) protobuf_field_is_map(+MessageType, +FieldName) is semidetMessageType's FieldName is defined as a map<...> in
the .proto file. protobuf_map_pairs(+ProtobufTermList:list, ?DictTag:atom, ?Pairs) is detDictTag{key:Key, value:Value} and a key-value list as described
in library(pairs). At least one of ProtobufTermList and Pairs
must be instantiated; DictTag can be uninstantiated. If
ProtobufTermList is from a term created by
protobuf_parse_from_codes/3, the ordering of the items is undefined;
you can order them by using keysort/2 (or by a predicate such as
dict_pairs/3, list_to_assoc/2, or list_to_rbtree/2.The following predicates are exported from this file while their implementation is defined in imported modules or non-module files loaded by this module.
protobuf_message(?Template, ?WireStream) is semidet protobuf_message(?Template, ?WireStream, ?Rest) is nondet