Getting around Protobuf's Python limitations - no option python_package

While working on converting a Thrift codebase to GRPC, I noticed a limitation in GRPC/Protobuf. The Protobuf compiler doesn’t seem to allow me to specify the target Python package of the generated stubs.

I found this odd because it supports directives like this for Java and Golang e.g.:

option go_package = "go.temporal.io/temporal-proto/common";
option java_package = "io.temporal.proto.common";

Without an equivalent option python_package the stubs would be generated in the root source code directory. A few Github issues¹² exists for this but it doesn’t look like it’s a high priority issue for the the Protobuf team.

In my case, I wanted the generated files to be placed in the temporal.proto package. So the first step was to create that equivalent directory structure for the .proto files:

git clone https://github.com/temporalio/temporal-proto.git temporal-proto/temporal/proto
cd temporal-proto

Then I had to prefix any import statements in the .proto files with the directory prefix temporal/proto/ with the exception of any import "goooge/.." statements:

find . -name '*.proto'|xargs -I % sed -i -E '/.*google.*/! s/import "/import "temporal\/proto\//' %

NOTE: This was tested with the GNU version of xargs and sed. If you’re on the Mac, be sure to install these using Homebrew.

Then I ran the protobuf compiler, as usual telling it to generate the Python stub files in the root directory.

python -m grpc_tools.protoc -I .  --python_out=.. --grpc_python_out=.. find . -name '*.proto'

Since the protobuf files were placed in a temporal/proto/ directory, the Protobuf compiler produced an equivalent directory structure in the Python output directory.

I’m then able to use the generated files from Python code by importing the generated modules and packages under the temporal.proto package.

from temporal.proto.workflowservice.service_pb2_grpc import WorkflowServiceStub
channel = grpc.insecure_channel("localhost:7233")
stub = WorkflowServiceStub(channel)
....