Tutorial

Implementing Remote Procedure Calls With gRPC and Protocol Buffers

Draft updated on Invalid Date
    Default avatar

    By John Kariuki

    Implementing Remote Procedure Calls With gRPC and Protocol Buffers

    This tutorial is out of date and no longer maintained.

    Introduction

    Remote procedure call (RPC) architecture is popular in building scalable distributed client/server model based applications.

    RPC allows a client to a make procedure call (also referred to as subroutine call or function call) to a server on a different address space without understanding the network configuration as if the server was in the same network (making a local procedure call).

    In this tutorial, we will implement an RPC client/server model using Google’s gRPC and Protocol Buffers.

    How RPC Works

    Before we dive into the heart of RPC, let’s take some time to understand how it works.

    1. A client application makes a local procedure call to the client stub containing the parameters to be passed on to the server.
    2. The client stub serializes the parameters through a process called marshalling forwards the request to the local client-time library in the local computer which forwards the request to the server stub.
    3. The server run-time library receives the request and calls the server stub procedure which unmarshalls (unpacks) the passed parameters and calls the actual procedure.
    4. The server stub sends back a response to the client-stub in the same fashion, a point at which the client resumes normal execution.

    How RPC works

    When and Why to use RPC

    While in the process of building out a scalable microservice architecture and Andela, we are implementing REST to expose API endpoints, through and API gateway, to external applications (mostly web apps in AngularJS and ReactJS). Inter service communication is then handled using RPC.

    Remote procedure call’s data serialization into binary format for inter-process (server-to-server) communication makes it ideal for building out scalable microservice architecture by improving performance.

    RPC client and server run-time stubs take care of the network protocol and communication so that you can focus on your application.

    RPC vs REST. Fight!

    While RPC and REST use the request/response HTTP protocol to send and receive data respectively, REST is the most popular and preferred client-server communication approach.

    Before we dive into RPC. Let’s make a comparison of the two.

    • REST as an architectural style implements HTTP protocol between a client and a server through a set of constraints, typically a method (GET) and a resource or endpoint (/users/1).
    • RPC implements client and server stubs that essentially make it possible to make calls to procedures over a network address as if the procedure was local. Simply put, RPC exposes methods (say, getUsers()) in the server to the client.

    The choice of architecture to use entirely remains at the discretion of the development teams and the nature of the system. Some developers have argued against comparing the two, and rightfully so because they can work well together.

    Read more about how the two compare on this article by Joshua Hartman.

    Implementing RPC with gRPC and Protocol Buffers

    Google-managed gRPC is a very popular open source RPC framework with support of languages such as C++, Java, Python, Go, Ruby, Node.js, C#, Objective-C, and PHP.

    gRPC is a modern open source high performance RPC framework that can run in any environment. It can efficiently connect services in and across data centers with pluggable support for load balancing, tracing, health checking and authentication. It is also applicable in last mile of distributed computing to connect devices, mobile applications and browsers to backend services.

    http://www.grpc.io/about/

    As we mentioned, with gRPC, a client application can directly call methods on a server application on a different network as if the method was local. The beauty about RPC is that it is language agnostic. This means you could have a grpc server written in Java handling client calls from Node.js, PHP, and Go.

    how gRPC works

    gRPC allows you to define a service which specifies the methods that you can call remotely, the parameters that can be passed to the procedure call and the return responses. The server then implements this definitions and creates a new grpc server to handle the procedure calls from the client.

    By default, gRPC implements Protocol Buffers to serialize structured data as well as define the parameters and return responses for the callable methods.

    Advantages Of Using The gRPC Framework

    Before we get into your first RPC application with gRPC, it is good to look at why we should go for gRPC as opposed to other RPC frameworks such as Apache’s Thrift.

    • gRPC is built on HTTP/2 under it’s BSD license. What this means is that the procedure calls get the goodness of HTTP/2 to build real time applications taking advantages of features such as bidirectional streaming, flow control, header compression and multiplexing requests.
    • gRPC’s default serialization protocol, Protocol Buffer, also transmits data in binary format which is smaller and faster as compared to good old JSON and XML.
    • Protocol buffer’s latest version proto3 which makes it easy to define sevices and automatically generate client libraries as we will see later.

    Defining Our Sample Application

    We will be building a simple RPC based app that let’s you employees know if they are eligible for work leave or not, if they are, we will proceed to grant them leave days. The rundown of how we will achieve this:

    1. Define our protocol buffer message types and service using the proto3 language guide.
    2. Create a gRPC server (in Node.js) based on the proto file we define and test it out.
    3. Create gRPC clients (in Node.js and Python).
    4. Make RPC calls from the respective clients to the server.

    Our project structure will be laid out as follows.

    ├── package.json
    ├── .gitignore
    ├── .grpclirc
    ├── server/
      ├── index.js
    ├── client/
      ├── node/
      ├── python/
    ├── proto/
      ├── work_leave.proto
    

    Creating Protocol Buffer Message Types And Service

    proto/work_leave.proto

    syntax = "proto3"; //Specify proto3 version.
    
    package work_leave; //Optional: unique package name.
    
    //Service. define the methods that the grpc server can expose to the client.
    service EmployeeLeaveDaysService {
      rpc EligibleForLeave (Employee) returns (LeaveEligibility);
      rpc grantLeave (Employee) returns (LeaveFeedback);
    }
    
    // Message Type fefinition for an Employee.
    message Employee {
      int32 employee_id = 1;
      string name = 2;
      float accrued_leave_days = 3;
      float requested_leave_days = 4;
    }
    
    // Message Type definition for LeaveEligibility response.
    message LeaveEligibility {
      bool eligible = 1;
    }
    
    // Message Type definition for LeaveFeedback response.
    message LeaveFeedback  {
      bool granted = 1;
      float accrued_leave_days = 2;
      float granted_leave_days = 3;
    }
    

    Let’s take a logical walk through of the .proto file.

    • First, we define the protocol buffer language syntax we will be using, in this case, proto3
    • Next we define a unique package name for our file to prevent name clashes between protocol message types. Note that this is optional.
    • Define an RPC service interface that stipulates what parameters each method in the gRPC server takes and returns. In our case, the service name is EmployeeLeaveDaysService
    • Last but not least, we define a Message Type for each data that we will be transmitting between the client and the server. A Message Type typically consists of a name/value pair where you define a data type, the field name and a unique number tag that identifies the field when serialized into binary format.

    Notice how easy it is to interpret the Service method implementations.

    • An RPC call to EligibleForLeave takes an Employee object and returns a LeaveEligibility response.
    • An RPC call to grantLeave takes an Employee object and returns a LeaveFeedback response.

    Creating And Testing The Server

    With our .proto file in hand, let’s go ahead and create our gRPC server. Since our server is written in Node, go ahead and initalize a new node project and install the grpc package.

    1. npm install --save grpc

    Next, create a server directory and add the following code to spin up the server.

    server/index.js

    const grpc = require('grpc');
    
    const proto = grpc.load('proto/work_leave.proto');
    const server = new grpc.Server();
    
    //define the callable methods that correspond to the methods defined in the protofile
    server.addProtoService(proto.work_leave.EmployeeLeaveDaysService.service, {
      /**
      Check if an employee is eligible for leave.
      True If the requested leave days are greater than 0 and within the number
      of accrued days.
      */
      eligibleForLeave(call, callback) {
        if (call.request.requested_leave_days > 0) {
          if (call.request.accrued_leave_days > call.request.requested_leave_days) {
            callback(null, { eligible: true });
          } else {
            callback(null, { eligible: false });
          }-1
        } else {
          callback(new Error('Invalid requested days'));
        }
      },
    
      /**
      Grant an employee leave days
      */
      grantLeave(call, callback) {
        let granted_leave_days = call.request.requested_leave_days;
        let accrued_leave_days = call.request.accrued_leave_days - granted_leave_days;
    
        callback(null, {
          granted: true,
          granted_leave_days,
          accrued_leave_days
        });
      }
    });
    
    //Specify the IP and and port to start the grpc Server, no SSL in test environment
    server.bind('0.0.0.0:50050', grpc.ServerCredentials.createInsecure());
    
    //Start the server
    server.start();
    console.log('grpc server running on port:', '0.0.0.0:50050');
    

    First, we load the grpc package and load the proto file using the exposed load method and then create a new instance of ther gRPC server.

    The server instance gives us access to the following key methods.

    • addProtoService
      • Takes two parameters, a link to the service name and a list of methods defined in the respective service.
      • The service name is referenced as <proto>.<package_name>.<service_name>.service.

    Each of the defined method takes in a call which contains the payload from the client and a callback function that returns an error and response respectively.

    • bind
      • The bind method specifies the IP and port on which to create the server. Since we are on a test environment, we will also create an insecure server(No SSL).
    • start
      • The start method simply starts the server.

    Testing out the gRPC server

    To take our new server for a test drive, let’s globally install a CLI gRPC client before we proceed to create our own clients.

    1. npm install -g grpcli

    Next, let’s start our server.

    1. node server

    To run the client, navigate to the route of your grpc application and run the grpcli command with the path to the protofile, the IP address and port to connect to and a command to start the client as insecured. You can also specify the configurations in a .grpclirc file

    1. grpcli -f proto/work_leave.proto --ip=127.0.0.1 --port=50050 -i

    Once a connection is established, you will be able to list all callable methods with rpc list and call methods with rpc call.

    Test grpc server with grpcli

    Sweet! Now with that working, let’s go ahead and create the grpc clients.

    Creating The Clients

    Like we had mentioned earlier, we will be creating gRPC clients in Node.js and Python just because we can. This will help us appreciate the power of RPC.

    For each of the clients we will basically confirm if an employee is eligible for leave and if they are, we will proceed to grant them the requested leave days.

    Creating a Node Client

    Let’s go ahead and create an index.js file in client/node and proceed to create our first client.

    const grpc = require('grpc');
    
    const protoPath = require('path').join(__dirname, '../..', 'proto');
    const proto = grpc.load({root: protoPath, file: 'work_leave.proto' });
    
    //Create a new client instance that binds to the IP and port of the grpc server.
    const client = new proto.work_leave.EmployeeLeaveDaysService('localhost:50050', grpc.credentials.createInsecure());
    
    const employees = {
      valid: {
        employee_id: 1,
        name: 'John Kariuki',
        accrued_leave_days: 10,
        requested_leave_days: 4
      },
      ineligible: {
        employee_id: 1,
        name: 'John Kariuki',
        accrued_leave_days: 10,
        requested_leave_days: 20
      },
      invalid: {
        employee_id: 1,
        name: 'John Kariuki',
        accrued_leave_days: 10,
        requested_leave_days: -1
      },
      illegal: {
        foo: 'bar'
      }
    }
    
    client.eligibleForLeave(employees.valid, (error, response) => {
      if (!error) {
        if (response.eligible) {
          client.grantLeave(employees.valid, (error, response) => {
            console.log(response);
          })
        } else {
          console.log("You are currently ineligible for leave days");
        }
      } else {
        console.log("Error:", error.message);
      }
    });
    

    Just like we did for the server, we need to load the proto file and the grpc package. We then create a new client instance that binds to our server’s address and port.

    At this point, we can now directly make calls to the methods defined in our server. As we had mentioned, each method take in a payload parameter and a callback function that returns an error and response respectively.

    Let’s take a look at how this looks like on our terminal.

    1. node client/node

    node client reponse

    Simple, but powerful.

    The protocol buffer limits us to how the employee object looks like so an error would be raised if you tried passing the employees.illegal object. Give it a spin for the different employees provided.

    Creating a Python Client

    To build a Python client, here are a few prerequisites.

    Install Python 2.6 or higher

    Install pip version 8 or higher, You can upgrade pip with the following command

    1. python -m pip install --upgrade pip

    Install virtualenvwrapper to setup a virtual environment workspace for your project. Create a new virtualenv and proceed set it as the current environment

    mkvirtualenv grpc_tutorial && workon grpc_tutorial
    

    Last but not least create client/python/client.py where our Python client code will reside.

    Installing gRPC

    In our virtual environment, install the grpc and grpc tools packages

    1. pip install grpcio grpcio-tools

    Compiling the proto file to Python

    The grpcio-tools module ships with protoc, the protocol buffer compiler as well as plugins for generating server and client code from the service definitions in the proto file.

    Create a generate_pb.py file in client/python and add the following code to compile the protocol buffer file to python.

    from grpc.tools import protoc
    
    protoc.main(
        (
          '',
          '--proto_path=../../proto/',
          '--python_out=.',
          '--grpc_python_out=.',
          '../../proto/work_leave.proto'
        )
    )
    

    With that, from the Python client directory, simply run:

    1. python generate_pb.py

    This will generate a client/python/work_leave_pb2.py which we will use to create our Python client.

    Creating the client

    in client/python/client.py, let’s go ahead and load the gRPC module and the Python protobuf. We then create an insecure channel to bind to the server’s address and port and create a stub from the EmployeeLeaveDaysService .

    import grpc
    import work_leave_pb2 as pb
    
    def main():
        """Python Client for Employee leave days"""
    
        # Create channel and stub to server's address and port.
        channel = grpc.insecure_channel('localhost:50050')
        stub = pb.EmployeeLeaveDaysServiceStub(channel)
    
        # Exception handling.
        try:
            # Check if the Employee is eligible or not.
            response = stub.EligibleForLeave(pb.Employee(employee_id=1,
                                                         name='Peter Pan',
                                                         accrued_leave_days=10,
                                                         requested_leave_days=5))
            print(response)
    
            # If the Employee is eligible, grant them leave days.
            if response.eligible:
                leaveRequest = stub.grantLeave(pb.Employee(employee_id=1,
                                                           name='Peter Pan',
                                                           accrued_leave_days=10,
                                                           requested_leave_days=5))
                print(leaveRequest)
        # Catch any raised errors by grpc.
        except grpc.RpcError as e:
            print("Error raised: " + e.details())
    
    if __name__ == '__main__':
        main()
    

    Start the node server and run the python client on a different terminal.

    Python client demo

    Conclusion

    For the PHP developers, gRPC can only support PHP clients currently. You can therefore not create a gRPC server with PHP.

    In this tutorial, we have managed to create a gRPC server in Node.js and made RPC calls from a Node.js and Python client based on a protocol buffer definition.

    For those of you that dare, you may make pull requests to this tutorial’s repository for both gRPC clients and servers in all the other supported languages. Make sure to abide to the directory structure.

    Thanks for learning with the DigitalOcean Community. Check out our offerings for compute, storage, networking, and managed databases.

    Learn more about us


    About the authors
    Default avatar
    John Kariuki

    author

    Still looking for an answer?

    Ask a questionSearch for more help

    Was this helpful?
     
    Leave a comment
    

    This textbox defaults to using Markdown to format your answer.

    You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

    Try DigitalOcean for free

    Click below to sign up and get $200 of credit to try our products over 60 days!

    Sign up

    Join the Tech Talk
    Success! Thank you! Please check your email for further details.

    Please complete your information!

    Get our biweekly newsletter

    Sign up for Infrastructure as a Newsletter.

    Hollie's Hub for Good

    Working on improving health and education, reducing inequality, and spurring economic growth? We'd like to help.

    Become a contributor

    Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.

    Welcome to the developer cloud

    DigitalOcean makes it simple to launch in the cloud and scale up as you grow — whether you're running one virtual machine or ten thousand.

    Learn more
    DigitalOcean Cloud Control Panel