secture & code

Everything you need to know about gRPC and ProtoBuf

What is gRPC?

gRPC is a high-performance, cross-platform RPC (Remote Procedure Call) communication system developed by Google.

gRPC was born in Google as an evolution of an internal communication system called “Stubby”. Stubby was used for communication between distributed services within Google's infrastructure. However, as Google grew and diversified its services, challenges arose in terms of efficiency and ease of development and maintenance of communication systems.

In this context, Google developed gRPC as a solution to address these challenges. The main objective was to create an RPC communication system (Remote Procedure Call) that was efficient, easy to use and compatible with multiple languages and platforms. gRPC was designed to be a high-performance communication system, using the HTTP/2 protocol to improve data transfer efficiency and offering support for multiple programming languages by automatically generating code from a service definition file. protobuf.

The purpose of gRPC was to provide a standardized and efficient way of communicating between distributed services, which would facilitate the development and integration of systems in complex and scalable environments. In addition, being open source and compatible with multiple languages, gRPC became an attractive option for the software development community outside of Google, which contributed to its rapid adoption and popularity in the industry.

1*4Hdrfc6Kqul7SPMD9OtSPA — *gRPC allows communication between services by means of protobuf messages, even if they are in different languages.*

Currently, gRPC supports the following languages:

What is protobuf?

Protocol Buffers (protobuf) is a data serialization format developed by Google to efficiently structure and serialize data. It is mainly used for communication between distributed systems, as in the case of gRPC, but can also be used to store data or any other task involving the efficient transfer or storage of structured data.

At protobuf, The data are defined using a simple and readable schema language called “Protocol Buffers Language” or simply “proto”. This schema defines the structure of the data to be serialized, including the data types and their names. From this schema, source code is automatically generated in different programming languages to serialize and deserialize the data efficiently.

For example, suppose we want to define a user message with a name and an email address using protobuf.

The schema definition file proto would be something like this:

syntax = "proto3";

message User {
  string name = 1;
  string email = 2;
}

This scheme will be compiled using the compiler of ProtoBuf: protoc. To return a file to use in Python we execute the following:

protoc --python_out=. user.proto

This would generate a file, with a name similar to user_pb2.py (unfortunately we can't choose the name under which it is generated), which we can import and use in our code:

import user_pb2

# Create user
user = user_pb2.User()
user.name = "John Doe"
user.email = "john.doe@example.com"

# Serialize user
serialized_user = user.SerializeToString()

# Deserialize user
deserialized_user = user_pb2.User()
deserialized_user.ParseFromString(serialized_user)

# Print name and email
print("Name:", deserialized_user.name)
print("Email:", deserialized_user.email)

With protobuf we not only define the data models, but we also define the interfaces of the services and the messages that are used to communicate between them, for example, we are going to implement a simple service that greets our users.

To do so, we will define:

A message HelloRequest , which receives as parameter an object of type User
A message HelloResponse
A message User
A service Greeter with an RPC method SayHello, which accepts as input a message of type HelloRequest and returns another of type HelloResponse

service Greeter {
  rpc SayHello (HelloRequest) returns (HelloResponse) {}
}

message HelloRequest {
  User user = 1;
}

message HelloResponse {
  string message = 1;
}

message User {
  string name = 1;
  string email = 2;
}

Implementation of a service

Now that we've seen what gRPC is and what is ProtoBuf and how we can define services and messages, we are going to implement a currency exchange service.

The service will consist of a method Exchange which will accept an object of type Money (composed of an amount and a currency) and a target currency. In turn, it will return a response that will include an object of type Money .

To illustrate the flexibility of communication between languages, we will implement the server in Python, and two clients, one in Python and one in NodeJS.

Definition in ProtoBuf

See the specification in ProtoBuf of this service:

syntax = "proto3";

service ExchangeService {
  rpc Exchange(ExchangeRequest) returns (ExchangeResponse);
}

message ExchangeRequest {
  Money money = 1;
  string to_currency = 2;
}

message ExchangeResponse {
  Money money = 1;
}

message Money {
  double amount = 1;
  string currency = 2;
}

Server implementation in Python

First of all, we will need to have installed the following packages grpcio y grpcio-tools:

python3 -m pip install grpcio grpcio grpcio-tools

Based on the definition in ProtoBuf Before, let's compile the file to generate the code needed to implement the server in Python:

python3 -m grpc_tools.protoc -I. --python_out=. --grpc_python_out=. exchange.proto

With the generated code, we can implement our server as follows:

import grpc
import exchange_pb2
import exchange_pb2_grpc
from concurrent.futures import ThreadPoolExecutor

class ExchangeService(exchange_pb2_grpc.ExchangeServiceServicer):
    def Exchange(self, request, context):
        # We will simply return the same amount by changing the currency for simplicity's sake.
        new_money = exchange_pb2.Money(amount=request.money.amount, currency=request.to_currency)
        return exchange_pb2.ExchangeResponse(
            money=new_money,
        )

def serve():
    server = grpc.server(ThreadPoolExecutor(max_workers=10))
    exchange_pb2_grpc.add_ExchangeServiceServicer_to_server(ExchangeService(), server)
    server.add_insecure_port('[::]:50051')
    server.start()
    print("Server listening on port 50051...")
    server.wait_for_termination()

if __name__ == '__main__':
    serve()

Client implementation in Python

As before, first of all we will need to have installed the packages grpcio y grpcio-tools:

python3 -m pip install grpcio grpcio grpcio-tools

Based on the definition in ProtoBuf Before, let's compile the file to generate the code needed to implement the server in Python:

python3 -m grpc_tools.protoc -I. --python_out=. --grpc_python_out=. exchange.proto

With the generated code, we can implement our client as follows:

import grpc
import exchange_pb2
import exchange_pb2_grpc
from termcolor import colored

def run_client():
    # Connect to gRPC Server
    channel = grpc.insecure_channel('grpc_server_python:50051')

    # Creating a client for the Exchange service
    stub = exchange_pb2_grpc.ExchangeServiceStub(channel)
    amount = float(input(colored('[?] Amount: ', 'blue')))
    currency_from = input(colored('[?] Currency from: ', 'blue'))
    currency_to = input(colored('[?] Currency to: ', 'blue'))

    request = exchange_pb2.ExchangeRequest(
        money=exchange_pb2.Money(amount=amount, currency=currency_from.upper()),
        to_currency=currency_to.upper()
    )

    response = stub.Exchange(request)

    print(colored(f"\n[+] Amount: {response.money.amount} {response.money.currency} {response.money.currency}", 'yellow'))
    print(colored(f"[+] Exchange rate: {response.rate}\n", 'yellow'))

if __name__ == '__main__':
    run_client()

Client implementation in NodeJS

With Node we have the possibility of doing the implementation in two different ways:

With dynamic code generationThe code is generated in runtime.
With static code generationThe code is generated in a previous step.

As we have already seen the example with static code generation, we are going to follow the dynamic approach. I have only seen this approach available in Node, and it has the characteristic that, using it, we can dispense with the objects that represent the messages and use flat NodeJS objects. You can see this as an advantage or a disadvantage, I personally like better to use the defined objects, as it allows us to be more consistent with the rest of the implementations and adds a layer of validation to the messages we send. In fact, if we were to do the implementation with TypeScript we would use the defined messages.

Prior to implementation, we will need to install the following packages @grpc/grpc-js y @grpc/proto-loader

Let's see an example of implementation:

const grpc = require('@grpc/grpc-js');
const protoLoader = require('@grpc/proto-loader');
const input = require('readline-sync');

const packageDefinition = protoLoader.loadSync('protos/exchange.proto', {
    keepCase: true,
    longs: String,
    enums: String,
    defaults: true,
    oneofs: true
});

const exchange_proto = grpc.loadPackageDefinition(packageDefinition);

const client = new exchange_proto.ExchangeService('grpc-server-python:50051', grpc.credentials.createInsecure());

const amountFrom = parseFloat(input.question('[?] Amount: '));
const currencyFrom = input.question('[?] Currency from: ').toUpperCase();
const currencyTo = input.question('[?] Currency to: ').toUpperCase();

const request = {
    money: {
        amount: amountFrom,
        currency: currencyFrom
    },
    to_currency: currencyTo
};

client.Exchange(request, (err, response) => {
    if (err) {
        console.error('Error:', err);
        return;
    }
    console.log('Amount:', response.money.amount.toFixed(2), response.money.currency);
    console.log('Exchange rate:', response.rate.toFixed(2));
});

Functional Demo

You can find a functional demo of everything seen here in the following Github repository.

You will be able to see the implementations and run the server and clients through a simple Makefile and Docker.

Conclusions on gRPC

gRPC seems to be carving out a small niche in the world of microservice communications. I find it a useful and powerful technology, plus the service definitions and messaging seems like a great idea to keep teams aligned and at the same time document our services.

While it is true that I have published only the tests with Python and Node, I have done many others with languages such as PHP or Golang, and I have to say that it has not been easy at all.

Despite having extensive experience in PHP, the installation of the libraries, the compilation of the models... etc, is quite complex, I have encountered dozens of errors that have had me diving in StackOverflow, ChatGPT and Github looking for solutions. Besides that installation of the PHP library is very slow and desperate.

With Golang I have also had problems, but I take my share of the blame, as my experience with Go is quite limited in comparison.