Streamlining Communication with Protobuf and gRPC

Introduction

In today's rapidly evolving world of software development, efficient and seamless communication between different systems and services is important. Protobuf (Protocol Buffers) and gRPC have emerged as powerful technologies that address the challenges associated with inter-service communication. As a consulting company, it is crucial for us to stay at the forefront of technology and ensure that these technologies are mature enough for production use.

One of the main reasons we are using gRPC is its effectiveness in facilitating inter-service communication. With projects becoming increasingly complex and extending beyond the scope of web service applications, it's becoming crucial to have a reliable means of communication. As you probably know, one of our projects is Spot. Internally, Spot is using Protobuf to communicate with other components. This enables seamless and efficient data exchange within the system.

Another reason why we started working with Protobuf and gRPC is the nature of the projects we are working on. For example, projects that are using computer vision can decrease their latency when they are using gRPC. Projects involving Iot-devices are also great examples of areas where we can decrease latency and reduce packet size.

Without further ado, I will introduce some key aspects of both technologies.

Protobuf

What is Protobuf

Protobuf is a language-agnostic data serialization format developed by Google. It enables efficient serialization, deserialization, and transmission of structured data. With Protobuf, developers define the data structure using a language-agnostic interface definition language (IDL). This definition acts as a contract between different services, ensuring seamless communication. This means that the sender can use a strongly type language like .NET C#. while the receiver can work with JavaScript, without needing knowledge about each other’s programming languages.

Compactness and Efficiency

Protobuf uses a binary format that is highly compact and efficient compared to traditional text-based formats like XML and JSON. The compactness of Protobuf leads to reduced network bandwidth usage and smaller message sizes, resulting in faster transmission and reduced storage.

Versioning and Compatibility

Maintaining compatibility between different versions of software can be a challenging task. However, Protobuf addresses this by allowing developers to evolve their data schema without breaking existing systems. New fields can be added, and old fields can be deprecated without impacting the compatibility of the systems. This flexibility makes Protobuf an ideal choice for projects that require long-term data evolution and seamless backward compatibility.

Language Support

Protobuf offers support for a wide range of programming languages, including but not limited to C++, Java, Python, Go, and C#. This extensive language support ensures that developers can seamlessly integrate Protobuf into their existing technology stack, regardless of the language used for development.

gRPC

What is gRPC

Building upon Protobuf, gRPC is an open-source framework also developed by Google. It leverages Protobuf's efficient data serialization to enable fast and reliable inter-service communication. The inter-service communication can be a server-client architecture or a more intricate setup, such as Spot with its various modules. Let’s consider a specific example with Spot: The process of transmitting camera data from spot. gRPC supports streaming and bi-direction streaming. It's a good choice for all the IoT connected devices.

High-Performance Remote Procedure Calls

gRPC provides an efficient and high-performance RPC (Remote Procedure Call) mechanism by leveraging Protobuf. It offers bi-directional streaming, allowing clients and servers to send and receive multiple messages asynchronously. This capability is particularly beneficial in scenarios where real-time data exchange (Iot object) and continuous communication are required.

To understand bi-directional streaming, let's imagine a conversation between two people, Alice and Bob. They are using walkie-talkies to communicate. Normally, when one person speaks, the other person listens, and vice versa. This is similar to traditional request-response communication, where one side sends a message and waits for a response before sending another message.

However, with bi-directional streaming, Alice and Bob can both speak and listen at the same time. They can have a back-and-forth conversation, exchanging messages without having to wait for each other to finish speaking. This allows them to have real-time communication, like a live chat or a phone call.

gRPC Example

Let’s start by defining a proto file. The .proto extension is used to define the Protobuf language. In the proto file, you define the structure of the messages that will be exchanged between the client and the server. A message represents a unit of data that needs to be serialized (converted to a binary format) when sending it over the “wire” and deserialized (converted back to its original form) when receiving it. Messages can contain fields of various types, such as strings, integers, booleans, or even nested message types.

message CreateTodo {  string task = 4;  optional google.protobuf.Timestamp due = 3;}message Todo {  int32 id = 1;  bool complete = 2;  optional google.protobuf.Timestamp due = 3;  string task = 4;}message TodoId {  int32 id = 1;}

In the example above, I created 3 messages.The first message is to create the Todo task, the next one is the complete object of the todo and the last one is the message to retrieve a task based on an ID. Without a service, we cannot consume these messages. Let’s create some grpc services in the same proto file.

service TodoService {  rpc Create (CreateTodo) returns (Todo) {}  rpc Get(TodoId) returns (Todo) {}}

In the example above, we are defining 2 grpc services. You can imagine rpc service as an endpoint in the REST world. But, RPC and REST are a bit different.

And now, this is where the magic happens. Protobuf and gRPC have binaries that can be used to create code in various languages. For example, I will use Python to generate code for the messages defined above and the gRPC services.

python -m grpc_tools.protoc -Iproto \\  --python_out=api/grpc/ \\  --pyi_out=api/grpc \\  --grpc_python_out=api/grpc \\  proto/todos.proto

• In the command line interface(CLI) above, I use python with the grpc_tools binary to generate the proto message with the --python_out options.
• I also created the python interface to define functions I can override to make an object callable. I used the option --pyi_out to generate this interface file.
• Lastly, I used the grpc_python_out option to have the grpc code to interact with the grpc function. The last argument of the CLI is the path to the proto file.

Conclusion

Protobuf and gRPC have become go-to technologies for modern software development due to their numerous benefits. The compactness and efficiency of Protobuf drastically helps in the optimization of data transmission and storage requirements, all while its versioning and compatibility features ensure long-term viability. Moreover, gRPC builds upon Protobuf to provide a high-performance RPC mechanism, enabling fast and reliable inter-service communication.

The language-independent nature of Protobuf allows developers to work seamlessly across multiple programming languages, making it easier to integrate these technologies into existing software ecosystems. With their combined power, Protobuf and gRPC empower software developers to design scalable and efficient distributed systems.

As the demand for highly performant and interoperable systems continues to grow, Protobuf and gRPC offer a reliable foundation for building communication channels across multiple languages and architecture. By embracing these technologies, developers can unlock the potential for streamlined and efficient data transfer, ultimately leading to robust and scalable applications.

For all the reasons above, Osedea will promote the use of Protobuf and gRPC in future projects.