X

Several times, clients have asked me for help increasing developer productivity by creating a micro-service generator. Service generators enable developers to quickly generate functional components provisioned with all the necessary glue and configuration for their target ecosystem. This is a common strategy in midsize to large companies. Standardizing on a service stack, architecture, a configuration and a templatized generator brings a number of advantages that I’ll talk about below..

Let’s call such an automation tool: _X_.

Project Goals

Increase developer productivity

It should be possible to generate an application/service stub and deploy this to production in 5 minutes or less. Developers should need to spend as little time as possible dealing with previously solved problems and configuration details, leaving them to spend the bulk of their time focusing on the business domain.

Increase maintainability

All services should follow a common format. Configuration, logging, exception reporting, open API tracing, secrets, etc. should all have the same look and feel across the ecosystem. Legacy services might not fit the mould, but all the new micro-services should have a similar shape and conform to a common requirements checklist.

Minimize blast radius

Without semantic versioning, dependencies can be tricky to keep up-to-date. Even with proper versioning, large libraries present a correspondingly large surface area for bugs… A defect in a commonly used library can have a blast radius that is company wide. One way to deal with this to keep libraries small and to use semantic versioning.

Minimal scope

Do one thing, and do it well. That is, this micro-service generator does not need to do everything. It just needs to generate new micro-services. Keeping the boundary tight ensures that the end product will be understandable and successful.

Purpose

X was envisioned to ease the burden of on-boarding new developers into the clients ecosystem, provide a standard application structure for new service development, and provide a ‘standards compliant’ implementation, complete with standard components wired in, such as:

• Logging
• Metrics
• Alarms
• Bug reporting
• OpenAPI tracing
• Secrets management

X takes an OpenAPI Specification (also known as a Swagger Spec) and a service name, and outputs a fully functioning application with all these components wired up and ready to use in a production environment.

Architecture

X does two different things:

• Generate custom glue code and configuration

  • This is the metadata that wires an application into the client’s environment
  • This includes configuration artifacts such as secrets, application definition, Docker-files, etc.

• Generate source code in target language

  • Services are structured in a specific way, as specified by the client.
  • Multiple target languages need to be supported

X delegates to an open source library called swagger-codegen to perform all the ‘heavy lifting’ for generating the actual source code. However, because the client had custom requirements for things like file layout, etc., I created custom swagger modules to plug into swagger-codegen. In general, the swagger code-base is large (and a bit messy), so abstracting your specific target language and service implementation into a swagger module can really ease the cognitive load for maintenance.

I won’t go into too much detail, but the overall picture is:

_X_ (go) -> generates Makefile (and other configuration artifacts)
_X_ (go) -> invokes Makefile target -> swagger -> generates code artifacts

Delegating to a Makefile has all sorts of advantages for isolation of concerns. If you are a fan of Makefiles, then you understand.

Implementation

Here are the java classes that swagger applies to the templates to generate go-server artifacts:

• AbstractGoCodegen.java
• GoServerCodegen.java

Here are the corresponding templates:

• Templates

Debugging

Note that it can be challenging to read the mustache templates the first few times through. It would really help if there was an easy way to dump the object model that’s presented to the mustache template when it’s being applied… Not hard to write, actually. Take a look at this Kotlin generator:

• KotlinServerCodegen.java#L207

All you need to do is write a class that implements Mustache.Lambda. Have this class dump the object model. Then wire it into the additional properties here and insert

{% raw %}
{{#lamda.dump}}{{/lambda.dump}}
{% endraw %}

into your template and voila, you are dumping your object model into the generated template.

Key Learnings

• There as a delicate balance between what to expose to developers and what to hide, and this needs to be easy to shift over time
  • If things are too tight, developers will simply not use the tooling
  • If things are too loose, developers lose the benefits of standardization (especially important for maintenance)
  • Developers must be included in the process and their feedback needs to be addressed quickly
  • Treat developers as customers, not as miscreants that do not know how to code
• There will always be multiple languages, so the tooling needs to accommodate that reality from the start
• Use rapid prototyping to develop the POC
• Resist the urge to write everything from scratch. For example, I would have loved to write my own Open API Spec parser… but the specification alone is enormous… so I punted and used a preexisting library, warts and all, rather than invest the time to read, let alone implement, the Open API Spec.

Development Process

The first time I developed X, I created it in stages, using rapid-prototyping for the first 3 stages like so:

• Stage 1 : Rapid Prototype in bash and go-swagger -> generate Go micro-service
• Stage 2 : Rapid prototype in bash and swagger-codegen -> generate Go micro-service
• Stage 3 : Replace bash with Go for front-end, continue delegating to swagger-codegen -> generate Go micro-service
• Stage 4 : Add additional language(s) to the generated outputs -> generate micro-service in multiple target languages
• Stage 5 : Extract micro-service generation from swagger-codegen into swagger modules

Looking at the stages, you can see that I used rapid-prototyping to evaluate tooling choices and to help flesh out the general concepts before committing to more concrete implementations. I also ensured that X was built in a modular fashion so that components could be swapped in or out as necessary. Currently, it is trivial to add a new output language or application format. Some might question why I moved from using go-swagger to swagger-codegen. There were a number of reasons. First, the code that go-swagger generated seem too complex. Second, and more importantly, I did not want to have to develop and support multiple back-end code generators. If swagger could handle my needs, then I just needed to become proficient in one generator.

Scope (Exclusions)

My goal was to create a simple tool that does one thing well: generate micro-services. To that end, I excluded a number of potential features either because they violated that goal or because I could not justify the ROI (even though they would be fun to work on):

• No git/github integration : X does not create your repository for you.
• Minimal Grafana integration : Frankly, it’s easier to manually create a template in Grafana.
• Minimal test generation : X does generate test stubs, but really, this is just there as a reminder to developers to write tests.
• X is a generator, not a code maintenance tool. For example, updating your transitive dependencies is not the purview of X, your various languages and frameworks have existing tools for this sort of thing.

Summary

• The friction for new service development has been reduced to less than a minute
• Because services conform to a similar model and shape
  • Deployment, maintenance, and support burdens are all decreased
  • Cognitive load on developers is decreased

This leads to more time spent on writing business logic components and less time re-writing common boilerplate glue code.