I’m slowly learning Erlang. It’s taken me more than 10 years to get to where I am now…which, frankly, isn’t very far. I thought I’d share some newbie thoughts from a veteran software developer with over 25 years of experience in the industry.

History

From 2009 to 2019…

• read books (Joe, Fred, etc.)
• write little code snippets
• post little how-to-tutorials showing how little I understand about Erlang
• wander away as I lose focus and time
• repeat.

This year, after Joe died, I thought…I’m not getting any younger either. I decided to take the summer off and really study Erlang. So far the process has been:

• read the same books, but only as references
• have a real problem to solve (I’m basing a business on it)
• write little code snippets (escripts) to POC business components
• document what I am learning on my personal website
• design the full scale app
• implement … oops hitting the road blocks…

Road blocks

Erlang installation and tools

I wish I’d known about asdf much earlier. That’s really simplified my dev box configuration. I wish that rebar3 and kerl and other docs just pointed this out. In fact, I wish there was a golden path published that just says, look, here’s the common accepted tool chain you are going to want to use. Stay on this path, and life will be easy. rebar3 is a big part of this story, but not the whole thing.

Migrating from escripts to real apps

$ rebar3 new escript myfooscript

I started with escripts because I didn’t want the hassle of understanding OTP or anything else as I was writing POC snippets to understand the basic Erlang libraries. So I’d write an escript to shell out and curl some resource and gradually build up an application that ingests data and writes into a database, blah blah. But along the way I could learn about xmerl and other useful libraries. Oh, and I could run the app and see STDOUT. Cool! Just like any other language I’m familiar with.

So I roll up my sleeves and decide to create an app…

Naming things is hard

$ rebar3 new release|umbrella myfooapp

Um, why does ‘umbrella’ exist as an option here? Is it the same thing as a ‘release’? Is it necessary? Why is ‘release’ called ‘release’? This is an overloaded term, as there are ‘releases’ which is a thing, and there’s a ‘release’ command, and there’s a ‘release’ target in the rebar3 command. And there’s an ‘umbrella’ in there, too. OK, release == umbrella, so I pick one.

$ rebar3 new release myfooapp

Where is STDOUT?

So, let me say, rebar3 is great. I barely remember what it was like to use sinan or faxien. (And while Loic’s Makefiles look amazing, I’m nowhere near smart enough to use them.) But here’s the thing, I had to watch a number of videos before I found out what the shell command was. This is important to me because the next block I had was… how do I start my application and see the output from it? I want to watch STDOUT, right? I was watching some conference video when someone mentioned that ‘shell’ was the coolest part of rebar3 and I’m like, ‘what is that’? And voila! It’s what I needed.

This should be on the golden path.

$ cd myfooapp
$ rebar3 shell myfooapp

Where is the golden path?

It’s weird that there is no golden path. In this talk, at 14:50, Fred talks about how it’s only far into various Erlang books that authors start talking about running code outside the shell, in contrast to K&R C where it’s discussed on page 6. In fact, the entire (excellent) presentation is on the terrible state of tooling in the Erlang ecosystem.

begin rant

Here’s what’s weird about Erlang : Books are out of date

• The books are already out of date. Even LYSEFGG still talks about systools and reltool even though rebar3 and relx have supplanted them? What’s weird about this is to be reading the rebar3 documentation, then cross-referencing with relx, and reading the various books that describe the old ways of building releases and relups, etc.

Docs are hard to follow

• relx documentation is pretty hard to follow
• Community support seems fragmented
  • Github shows relx has 41 open issues and 6 open pull requests, with 89 contributors
  • Github shows rebar3 has 107 open issues and 11 open pull requests, with 239 contributors

Who’s maintaining these tools?

If I were Tristan or Fred, boy I’d be tired of supporting this stuff.

Where are the videos?

Here’s another thing that’s weird…

I cannot find videos on how to use the Erlang shell. So on top of that weird fact, I’m finding out about erlang by watching Elixir videos. I was watching an Elixir video on ‘wobserver’ and realized, ‘Hey! I bet that’s just a web interface on some Erlang thingy called “observer”’. So then I find out about ‘observer’ and how useful that is. Great! But things like, how to I do basic stuff like navigate from one process to another, what do I do when everything seems hung and broken, what do real live seasoned Erlang devs do when ‘stuff hit’s the fan’? How do they go about examining a running system? The web seems to have nothing on this. But that’s just weird, b/c Erlang has been around for over 25 years, and any other language / framework/ runtime whatever has zillions of youtube videos.

end rant

Culture of Quick Fixes

OK, so enough ranting. Let me get into Fred’s mention of the ‘Culture of Quick Fixes’ (at about 21:00) by introducing my own steaming pile of a quick fix.

So here’s the deal. I’m trying to build a company. I have limited $$$, I can’t screw around learning stuff forever. At some point I have to build a real live app and try to generate some buzz. I’ll roughly describe my first app/service and run through my decision points…

App

The app basically pulls data from publicly available but disparate data sets, ingests them, normalizes them into a common data store, and then surfaces the data store for analysis, reporting, AI/ML, etc, etc. In a nutshell, take data, add magic, produce pretty pictures.

Process

• prototype in bash (or python, or whatever…)

Seriously, with just bash and I can query for my data, extract in, play around with it, etc. If I wanted I could hook it up to cron and probably move on to actually using the data rather than learning Erlang and how to build ‘proper’ infrastructure.

Prototype in escript

Here’s where I take small tasks and learn the Erlang libraries to accomplish what I’m trying to do. It takes about as much time to do this as the previous prototyping step. I find this to be a great way to both decompose a problem and learn a new language.

Start building industrial grade services

So now, rather than focusing on my business, finding customers, analyzing the data I’ve collected, or in any other way providing value that could bring in revenue and allow me to purchase more Erlang books… I’m veering off into the weeds and learning how to:

• build erlang apps for real
• build erlang releases
• wire all this into docker containers
• wire in monitoring and all the ‘industrial grade goodies’

Along the way I tend to post about what I learn here.

• Is any of this worthwhile?
• Am I just engaged in endless pencil sharpening?
• I dunno. I’m too close to the fire.

I’m having fun though. And it’s incredibly useful to my brain in some way I cannot describe.

The Problem

So back to “The Culture of Quick Fixes”. As I’m building my first set of components and learning about rebar.config and config/sys.config and more groovy stuff, I run into the issue of how to deploy Erlang to docker containers. The answer is, run the release with the foreground command. But this wasn’t obvious to me at first and took a lot of digging around the ‘net, posting a question on the rebar3 discussion list, and just general noodling around.

But, if I’m building industrial grade applications, then I need to solve for issues such as:

• How do I emit metrics from my app?
• How do I monitor my metrics (e.g. Prometheus)?
• How do I hook up a visualization tool (e.g. Grafana)? Alarms?
• And how do I wire all this up into a containerized solution so that it’s re-usable and shareable?
• Don’t forget my “awesome” Makefiles! I like to have a common control plane for all my services, and common Makefiles are great for this.

Now, if you watched the end of Fred’s video, he was talking about many of these same things.

And I’ll suggest that what I’ve built is an example of exactly what shouldn’t be done:

• I ignored everything on hex. Why? Um, I don’t really understand hex that well yet, I’d have to go read a bunch of code to understand what folks have done, and I’m still learning Erlang by building things myself. None of these is a good reason, BTW.
• I only published my stuff to github, and not to hex. This may be a good thing as I’d just be duplicating other, probably better, efforts.
• What I built should probably be a rebar3 plugin, and not a Makefile with dependencies on rebar3 etc.

However, I’ll show you what this is because I think it satisfies a need and perhaps we can all come up with a better implementation strategy. Or learn something.

Erlang-Seed

So, here we have ‘yet-another-quick-fix’ called erlang-seed. Let me be honest, this is a ‘quick-fix’ to satisfy my own personal needs…

The Good

What this does is create an erlang application, inject it into a Docker container, and then spin up a local Docker cluster that monitors the Erlang app and surfaces the metrics via Prometheus and Grafana. So with one command I can generate an OTP application, all the infrastructure necessary to monitor it with common tooling, and spin it up and run it on my dev box. I find this to be really useful. YMMV.

The Bad

This is really crufty. It calls into rebar3 to generate the app. It should probably be a rebar3 plugin so that it can better evolve with the tooling. It could then more intelligently inject the necessary dependencies, instrument the code with metrics, etc. Right now it’s gross, as it just overwrites files as necessary using mustache templates.

The Ugly

• Do I have time to maintain this? No.
• Do I have time to make this a rebar3 plugin? Not right now.
• Will this probably break over time due to either rebar3, docker, or some other dependency changing? Certainly, eventually, yes.

Summary

The reason I’m sharing this is to just widen the discussion on what the Erlang education and tooling could look like. I think that generating full stacks that allow developers to focus more on solving business domain issues rather than forcing them to understand all the arcane details of whatever is under the covers is the ‘way to go (TM)’. There’s got to be a way to allow developers to maintain velocity while increasing their depth and scope of knowledge as necessary. I’ve been studying Erlang intensely for at least a month now, read the books countless times, and read the online docs and I still don’t quite understand how this all fits together. I mean I get the big picture. But there’s a lot of details.

Contrast this with Go. I was writing production code in 2 days. The quality was production quality after about a week. Back in the day it took me about a month to become proficient in Java, though it required reading Bloch’s book “Java Concurrency In Practice” to be ‘good’. Python was similar, it took about a week to ramp up, about a month to become comfortable. In all of these languages it was a relatively short and easy ramp up and then smooth sailing for years and years.

With Erlang it’s an incredibly steep ramp that goes on and on. I guess if I remember that I’m learning the equivalent of a language, an orchestration framework (like kubernetes), a database system, etc., then it starts to make sense.

Anyway, for my part, I’ll keep posting things as I figure them out.

-Todd