谷歌Protobuf python教程2同时处理多条信息

同时处理多条信息

python实现并没有提供任何内置的方法来同时处理多条信息。然而，Java实现提供了parseDelimitedFrom和writeDelimitedTo等方法。为了避免重新发明轮子，我们把 Java 库的功能翻译成 Python。

ava实现将消息的大小保存在Varint值中。Varints是一种序列化方法，它以一个或多个字节存储整数。

• 序列化

import random

from google.protobuf.internal.encoder import _VarintBytes

import metric_pb2

with open('out.bin', 'wb') as f:
    my_tags = ('my_tag', 'foo:bar')
    for i in range(5):
        my_metric = metric_pb2.Metric()
        my_metric.name = 'sys.cpu'
        my_metric.type = 'gauge'
        my_metric.value = round(random.random(), 2)
        my_metric.tags.extend(my_tags)
        size = my_metric.ByteSize()
        f.write(_VarintBytes(size))
        f.write(my_metric.SerializeToString())
        

• 反序列化

from google.protobuf.internal.decoder import _DecodeVarint32

import metric_pb2

with open('out.bin', 'rb') as f:
    buf = f.read()
    n = 0
    while n < len(buf):
        msg_len, new_pos = _DecodeVarint32(buf, n)
        n = new_pos
        msg_buf = buf[n:n+msg_len]
        n += msg_len
        read_metric = metric_pb2.Metric()
        read_metric.ParseFromString(msg_buf)
        print(read_metric)  

执行结果：

$ python pb_unpacks.py 
name: "sys.cpu"
type: "gauge"
value: 0.4699999988079071
tags: "my_tag"
tags: "foo:bar"

name: "sys.cpu"
type: "gauge"
value: 0.7699999809265137
tags: "my_tag"
tags: "foo:bar"

name: "sys.cpu"
type: "gauge"
value: 0.5400000214576721
tags: "my_tag"
tags: "foo:bar"

name: "sys.cpu"
type: "gauge"
value: 0.44999998807907104
tags: "my_tag"
tags: "foo:bar"

name: "sys.cpu"
type: "gauge"
value: 0.9100000262260437
tags: "my_tag"
tags: "foo:bar"

参考资料

• https://developers.google.com/protocol-buffers/docs/proto
• https://developers.google.com/protocol-buffers/docs/pythontutorial
• https://github.com/simplesteph/protobuf-example-python
• https://www.datadoghq.com/blog/engineering/protobuf-parsing-in-python/
• https://github.com/prometheus/client_python