Jul 1, 2019

Monitoring : Part 3

In the previous post, Monitoring : Part 2 we looked at how Linux exposes host metrics and in particular how the Prometheus agent, called node_exporter, reads the Linux host metrics. In this post we’re going to circle back to the monitor project mentioned in Monitoring : Part 1, and add some basic Prometheus metrics, using Erlang, so that we can explore them in Grafana.

Dependencies

You need to have the following installed to make use of the demo:

git
docker
docker-compose
make

Getting Started

Clone the monitor repo.

mkdir temp
cd temp
git clone https://github.com/toddg/emitter

Start the docker components.

cd emitter
make start

Verify everything started OK.

docker ps

You should see the following components running:

CONTAINER ID        IMAGE                     COMMAND                  CREATED             STATUS              PORTS                    NAMES
02dc8ccb5218        toddg/erlang-wavegen      "/bin/bash -l -c /bu…"   18 hours ago        Up 18 hours         0.0.0.0:4444->4444/tcp   monitor_wavegen_1
d8bb17caa1d0        grafana/grafana:6.2.4     "/run.sh"                18 hours ago        Up 18 hours         0.0.0.0:3000->3000/tcp   monitor_grafana_1
1715763b4326        prom/prometheus:v2.10.0   "/bin/prometheus --c…"   18 hours ago        Up 18 hours         0.0.0.0:9090->9090/tcp   monitor_prometheus_1

Emit Metrics

It’s trivial to write a metrics emitter in Erlang. Here’s why, the receive ... after pattern for responding to messages. The snippet below is a simple timer function that invokes a method F on a data blob D if no message is received after Delay milliseconds:

%%--------------------------------------------------------------------
%% timer calls itself every Delay milliseconds and invokes function F on 
%% Data D
%%--------------------------------------------------------------------
timer(Delay, F, D) ->
    receive cancel  -> 
                void
    after Delay     -> 
              D1 = F(D),
              timer(Delay, F, D1)
    end.

The helper timer function above is invoked by another helper function tick. tick spawns an Erlang process using timer. The timer method in the spawned process waits for an event that never comes, times out, and invokes the function F in theafter stanza. Here’s tick:

%%--------------------------------------------------------------------
%% tick calls itself every N milliseconds and invokes function F on 
%% Data D; returns a cancel function C
%%--------------------------------------------------------------------
tick(N, F, D) ->
    Pid = spawn(fun() -> timer(N, F, D) end),
    fun() -> Pid ! cancel end.

Now that we can invoke methods via a timer, let’s wire in some methods to emit Prometheus metrics. First, we add a helper function to emit data from the various emitter functions. To start with, let’s verify that the timer works, by printing to STDOUT:

%%--------------------------------------------------------------------
%% emit : side-effect for the function F
%%--------------------------------------------------------------------
emit(Name, Value) -> 
    io:format("emit: [~p]~p~n", [Name, Value]).

Let’s wire the emit function into the following functions that actually do something:

flatline
incrementer
flipflop
sinewave

Here’s what these methods look like:

%%--------------------------------------------------------------------
%% flat line : emits the value 1 for a flat line shape
%%--------------------------------------------------------------------
flatline(_) -> emit("flatline", 1), ok.

%%--------------------------------------------------------------------
%% incrementer : always increments value
%%--------------------------------------------------------------------
incrementer(V) -> 
    V1 = V + 1,
    emit("incrementer", V1),
    V1.

%%--------------------------------------------------------------------
%% flip flop : emits the value 1 and then the value 0
%%--------------------------------------------------------------------
flipflop(0) -> emit("flipflop", 1), 1;
flipflop(1) -> emit("flipflop", 0), 0;
flipflop(_) -> emit("flipflop", 0), 0.

%%--------------------------------------------------------------------
%% sine wave
%%--------------------------------------------------------------------
sinewave(V) ->
    sinewave(V, 1).
sinewave(V, Delta) ->
    emit("sinewave", math:sin(V)),
    V1 = V + Delta,
    V1.

The last step is start these emitter methods in main:

%%====================================================================
%% API functions
%%====================================================================

%% escript Entry point
main(Args) ->
    io:format("Args: ~p~n", [Args]),
    CancelFuns = [tick(10, F, 0) || F <- [fun flatline/1, fun flipflop/1, fun sinewave/1, fun incrementer/1]],
    %% this never get's called
    receive
        {ok} -> ok
    end,
    %% cancel all the timers
    [F() || F <- CancelFuns],
    erlang:halt(0).

One of the beauties of Erlang is list comprehensions. Here, we map the tick function across the list of emitter funcitons, thereby spawning an Erlang process for each of them:

    CancelFuns = [tick(10, F, 0) || F <- [fun flatline/1, fun flipflop/1, fun sinewave/1, fun incrementer/1]],

Let’s run this and see what the output looks like:

make start
cd monitor && docker-compose logs -f --tail="all" | grep wavegen

The first line invokes the start task in the Makefile which invokes the docker-compose up. The second line tails the output of the wavegen container:

wavegen_1     | emit: ["sinewave"]-0.7391806966492228
wavegen_1     | emit: ["incrementer"]63
wavegen_1     | emit: ["flatline"]1
wavegen_1     | emit: ["flipflop"]0
wavegen_1     | emit: ["sinewave"]0.16735570030280691
wavegen_1     | emit: ["incrementer"]64
wavegen_1     | emit: ["flatline"]1
wavegen_1     | emit: ["flipflop"]1
wavegen_1     | emit: ["sinewave"]0.9200260381967906
wavegen_1     | emit: ["incrementer"]65
wavegen_1     | emit: ["flatline"]1
wavegen_1     | emit: ["flipflop"]0

The key takeaways are:

flatline always emits : 1
flipflop alternates between 0 and 1
incrementer is monotonically increasing by 1
sinewave is taking the sin() of a number that is increasing by 1 every step

Now that we have verified that he emit function is getting called repeatedly, let’s wire in the prometheus metrics by replacing the print statement with a function to emit metrics:

%%--------------------------------------------------------------------
%% emit : side-effect for the function F
%%--------------------------------------------------------------------
emit(Name, Value) -> 
    multimetric(Name, Value).


%%--------------------------------------------------------------------
%% multimetric : emit all the metrics since we're just trying to
%% sort out how all this works. this way we can see stuff in the
%% grafana dashboard
%%--------------------------------------------------------------------
multimetric(Label, Value) ->
    %% counter for times the method has been invoked
    prometheus_counter:inc(mycounter, [Label], 1),
    % set a gauge to the current value
    prometheus_gauge:set(myguage, [Label], Value),
    % set the summary to the current value
    prometheus_summary:observe(mysummary, [Label], Value),
    % set the histogram to the current value
    prometheus_histogram:observe(myhist, [Label], Value).

Of course the metrics have to be registered, and the proper services started, etc. See the source for further details.

Grafana

Now that we are emitting known metrics, let’s see what they look like in Grafana, coming up in the next post: Monitoring : Part 4.