As I look to reduce the number of touch points that are required to make a change to my personal system, I’m always thinking about how to reduce complexity and leverage existing tooling. In this way, we can continue to leverage an existing foundation to build out further abstractions. One such lever I have worked on over the last year was using my gRPC protobuf definitions to generate LDAP schema files that could be loaded into the server to match the objects that I’d be working with in Go.
When I initially stated implementing gRPC in my environment, I hadn’t expected to be developing my own code generation methods, even though the gRPC and protobuf definitions support this by default by creating language-native methods for the developer to implement.
To do this, I’m using an interesting project by Moul on github. This project reads the proto file and then injects an object into any Go template that can be used to generate pretty much anything based on the contents of a proto definition, which is super handy.
As I worked on my system, I realized that many of the objects I was trying to work with in Go looked a lot like the objects I wanted to store in my database. In my case, the database I’m using is LDAP, but this method applies more generally than LDAP in particular.
After splitting a monolith proto file into many diferent packages, I then
updated the make
targets to ensure that each proto
file also called the
template plugin to generate the text output I desired.
proto-grpc:
@protoc -I internal/inventory/ -I ./ \
--gotemplate_out=template_dir=internal/inventory/templates,debug=false,single-package-mode=true,all=true:internal/inventory \
internal/inventory/inventory.proto
@protoc -I internal/inventory/ -I ./ \
--gotemplate_out=template_dir=cmd/templates,debug=false,single-package-mode=true,all=true:cmd \
internal/inventory/inventory.proto
The above is a little hard to look at, and took a few minutes to get straight, but now works pretty reliably for my needs.
The next part of the project was to start developing the templates themselves.
I started with the LDAP ldif
file. I’d worked with templates in other
languages and a little Go templates before this, but I still had to work at
getting this right, which took some time. You can see the current template
here and the resulting ldif
file here.
Once the schema is generated, the last piece I’ve not yet automated here is just to modify the database schema itself.
ldapmodify -Z -H ldaps://auth10.znet -D cn=config -W -f internal/inventory/ldap.ldif -vv
Next I wanted some methods that I could use to interact with the database
directly, without needing to implement that client side. This would methods
like Create
, Read
, Update
, List
for the various objects I wanted to
implement. This code ends up being pretty wordy in the template and the
resulting output, so I’ll just link out to it. Here you can
see the template, and the resulting code that was generated.
With the above in place, I’ve been able to scale out the system from just a
couple of objects, to several dozen at this point. I have an easy way to
update the database schema with objects from the protobuf definition, and Go
methods to interact with those database objects. This has come in quite handy,
and even though the templates too a few iterations to get right, I’ve been
happy with the results. Now all it takes to add a new field, or object is
only a couple of lines worth of manual change, and then a make proto
at the
root of the project and everything gets updated. I’m sure there will be
changes to this over time, so the inventory package should
be the place to look for the bigger picture.
I may end up moving this sort of approach to Cuelang in the future
when they support service
definitions from proto
, but for now I expect this
to carry me through the year.