• https://www.erlang.org/
• https://erlangforums.com/
• https://exercism.org/tracks/erlang
• https://github.com/happi/theBeamBook
• https://learnyousomeerlang.com/content
• https://www.erlang.org/docs

Gleam Resources

• https://gleam.run/
• https://exercism.org/tracks/gleam
• https://tour.gleam.run/everything/
• Getting to know Actors in Gleam - Raúl Chouza | Code BEAM America 2024
• code beam europe 2024
• An Erlanger’s encounter with Gleam | Bartek Gorny

Wisp Resources

• End-to-end types: full-stack Web apps with Gleam by Hayleigh Thompson | Code BEAM Europe 2023
• A Code Centric Journey Into the Gleam Language • Giacomo Cavalieri • GOTO 2024
• Lustre Universal Components: the best of Elm and Phoenix LiveView - H. Thompson | Lambda Days 2024

Misc

• code sync
• code beam

Why

Why are we doing this?

Todd

I want to:
* re-connect with nature
* do this in a way that doesn't leave a huge carbon footprint
* figure out what I want to do with the next chapter in life 
  
Ever since I first smelled the Cascades, I've wanted to spend time there. 
When I was in my teens, I heard about the PCT... I've wanted to hike 
this for most of my life. 

[Kristin] TODO

Details

Contact

We’ll both have our cell phones and will probably check them weekly or so. We’ll also have a Garmin In-Reach Mini so family and close friends can contact us in an emergency.

Plan

Hike from South to North (remember, never trust a southbounder). We’ll start at the US/Mexico border and see what happens.

Photos

We’ll post photos to this google photo album.

Gear

• 
• DIY backpack post

Something that was an easy hack that I’m excited to use:

• 
• DIY USB-C adapter on a sonicare toothbrush 2
• DIY USB-C adapter on a sonicare toothbrush 3

Links

Dehydrated Food

Here are some dehydrated food products that we’ve researched. I think we’ll be be eating a fair bit of dried bean flakes. I’ve tried a bunch of the various things we ordered from 3 of these companies. Overall, this stuff is all OK but not something I would choose over fresh food.

• dehydrated food resources

Protein

Some longtime friends have created these delicious snack sticks. I think these will be a staple on this trip.

• singing pastures

See you later!

History

Prescribed Burn Manager Certification Program was established by House Bill 2733, see also Laws and Regs.

What this law provides:

DNR Program

• https://www.dnr.wa.gov/certifiedburner
• https://www.dnr.wa.gov/prescribedfire
• https://burnportal.dnr.wa.gov/

Pre Class Work

• Videos

Class Work

• TODO: get slide decks from Kyle

Field Work

Process

• write a burn plan
• request a burn permit (with burn plan attached)
• attend weekly smoke meetings
• request a smoke permit
• with burn permit, burn plan, and smoke permit IN HAND, proceed with prescribed burn, but only according to plan

Burn Plan

Also known as a Rx (Prescription)

• objectives
• parameters
• resources
• impacts

Links

• The Big Burn
• Introduction to Prescribed Fire in Southern Ecosystems
• Ignition Techniques
• The Photoload Sampling Technique
• Standard Fire Behavior Fuel Models (Scott/Burgan)
• Prescribed Burning

• South on the Oregon Coast Trail
• North on the Pacific Crest Trail

We plan to spend about a year hiking, using as little carbon as possible. Realistically this means that we plan to either be walking, or traveling on public transportation…bus or train. We will not have a car, and we will not be flying.

As part of this, I designed and sewed my own backpack.

DIY Backpack

Clearly I am standing on the shoulders of giants. For those that want to sew a backpack (or tent), I recommend checking out Ray and Jenny’s site. I have sewed a number of their kits:

• Net Tent
• Quilt

Note:

The umbrella is a key component. First popularized by Peter and Barbara Jenkins. Then, the father of ultralightweight hiking, Ray Jardine, picked it up and talked about it in their book, “Trail Life”, see:

• https://www.rayjardine.com/ray-way/index.php
• https://www.rayjardine.com/ray-way/Books/Trail-Life/index.php
• https://www.rayjardine.com/Papers/Magazine-Articles/index.php?StoryPage=12.95

Kristin purchased (but didn’t end up building):

• Backpack

I have sewed two of the net tents. On the second one, I opted to keep the design simple and ommitted the bat wings. I haven’t used this tent yet, so I’ll report back as to whether that was a good idea or not.

I’ll mention it again, be sure to read Trail Life.

For my backpack, I opted to design mine from scratch. I could (and probably should) have ordered a kit from the Jardine’s… it’s actually cheaper to order their kits than to purchase the fabric directly from my local fabric supplier. However, I just wanted the experience of starting from scratch.

Here’s a photo trail of this journey…

Prototypes

Ultra-Ultralight

I made my first prototype out of 1.9 oz nylon… this is the sort of fabric I make stuff sacks out of. I am certain that a backpack made of this material would be a shredded mess in no time. But when I started out, it was exciting to have a backpack that weighed just a few ounces. I never completed this design with straps as it just seemed silly to continue.

First Full Prototype (with straps)

I used basting stitches to help fasten things as I sorted out where to put the straps and lifeters…

A also used a fair bit of blue painters tape and black electrical tape while sorting out the shoulder straps. If you look closely at the photos, you’ll see sewing with nothing attached… at first this wasn’t a prototype. It was a failure. But then, once I freed my mind and just started hacking it apart and re-sewing it back together, what I wanted it to become started to take shape.

Pattern Making

Once I had a pack that I liked, it was time to spit the pack apart along the side seams.

Then I flipped the pack over so that I could see all the features attached to the outside.

Making the Pack

With the prototype spread out, I laid down my new fabric next to it, marked it, and cut it to length.

• Fabric : 60" 400 x 300 D. Nylon/Polyester Diamond Ripstop @ $13.50/ linear yard

I also cut the bottom reinforcement piece. Note that I have an electric fabric melty thingy in this photo. That tool works magic with meltable fabrics. However, while it’s a must-have tool for cutting sunbrella… for a backpack, the melted ends are a little brittle and might chafe somewhere. So I ended up just using scissors for everything in this build.

• Fabric : 58" 420 D. X 420 D. Large Diamond Ripstop Nylon/Polyester @ $12.95/ linear yard

The bottom is now reinforced.

I’m laying a straight edege across the black fabric over to the blue fabric. The black fabric is the prototype turned into a template. I marked the center location for the bottom, using chalk, on both the template and the new fabric. From this point on I didn’t really measure anything. I just transfered locations from the template to the new fabric, marked with chalk, and started sewing.

I added a black patch for reinforcing the backpack and lifter attachment points.

I added the mesh bag for drying out my socks and underwear. Maybe I’ll also put snacks here.

I love my umbrella. And I want to be able to cary lots of water. So these are the panels for those two features…

Panels sewed in.

I reinforced these loops with some rainbow webbing.

I had intended to run some thin ~1/8" bungee cord on the back of my pack to hang a rain coat or something. These little black tabs worked great on a previous prototype. However, on the finished pack these were a PITA. When the fabric is folded over 2-4 times, this becomes really THICK and hard to sew. And I misjudged how long they need to be and only have of them are available to use in the finished pack. Next time I’ll stagger how these get sewed and pull them back from the edge by 1/2".

Pickup strap and lifters. Again, I sized all of this on the prototype.

The straps are not hard, but there is some complexity in that the front webbing needs to be solidly attached to the strap. Otherwise there’s this piece of webbing digging into you with out a pad between you and the webbing. I took the proto-pieces and turned them into templates.

Attaching straps to the pack.

I hope my stitching holds.

Both straps attached.

With all the features added to the outside of the pack, it’s time to turn it inside out and sew the side seams. For this I switched from V-69 to V-92.

The pack has been sewed inside out, with something like a flat felled seam…resulting in 4-6 layers of fabric… there is no way a normal sewing machine could do this. Make’s me glad I purchased a used Juki walking foot industrial machine for this project.

Right side out

Pack loaded up with umbrella, water bottle, a bear canister and some clothes inside.

Conclusion

Sewing this pack was a blast. It is probably heavier and more expensive than a kit would have been. However, I had fun designing it and testing out prototypes. Actually, I should mention that the design stage was kind of disheartening. I felt like I was wasting material and time and that I’d eventually wind up buying a pack after spending $$$ trying to make one.

I definitely didn’t save any money on this. And I’d do a few things differently. But hey, now I know how to make a backpack and I have a cool holster for my umbrella!

Pack Weights

• My normal pack (Deuters internal frame) : 73oz
• This pack as built : 20oz
• The Jardine’s pack : 13 oz

We hope you are enjoying some of the honey from the Maple Leaf Honey Bee Project! This project is part of our ride on the roller coaster of thoughts and feelings about climate change. On our good days, we believe putting our attention on the climate emergency can generate some small solutions, and more closeness to other people and to nature. Below is a little about our journeys in backyard beekeeping.

We’d love to hear what you’re discovering on your own journey facing climate change. Feel free to email us (bees at zwrob dot com) stories of what is unfolding for you.

Todd…

A few years ago I was wondering, “What could I do, in my backyard, to fight climate change?” Somehow I stumbled upon the idea of raising bees, not for the honey so much as to help pollinators. My thought was that helping pollinators can help our local ecosystem. So I replaced my front yard with a hugelkultur and started raising bees. I involved friends and neighbors by inviting them to host bee hives in their back yards, too. I’m trying to breed a locally adapted variety of bees suited to our wet climate.

Kristin…

When I first met Todd he told me about the food forest he was creating in his yard. Pear and hazelnut trees, blackberry bushes, composting logs (a hugelkultur), and service berry and other native shrubs for the bees and critters to feast. Last Fall I sowed native grass seeds and this Spring we spread wildflower seeds. We’re hoping they’ll grow into a Pacific Northwest prairie that feeds the finches, shelters the mice, and supplies the bees with even more pollen.

Other paths…

The part of the journey we both love most is the connections with people… An old friend that was hosting one of Todd’s hives in his yard has decided to take a bee keeping class and has started keeping bees of his own.

Two young friends of ours have taken a keen interest in bees, and one even drew the picture of a bee on this web page (thanks Mays!)

We’ve had dozens of neighbors stop and ask about the bees (and honey!).

If you are interested in joining us for one of our critter parties, then drop us an email.

–Todd and Kristin

Links

Here are some links to organizations that we enjoy…

• Puget Sound Beekeepers Association
• Seattle Tilth Alliance
• Seattle Tilth + Seattle Public Utilities’ Master Composter/Sustainability Steward program
• Transit Riders Union
• Russ Vernon Jones (uplifting stories)
• Out of the Closet Thrift Store in Capitol Hill takes clothes and household goods donations and supports testing and health care treatment for HIV/AIDS treatment
• Refugee Artisan Initiative, takes textile donations that help women who are refugees create clothes, bags, and more for sale
• Grist, for thoughtful, sometimes hopeful news about the climate
• Seattle waste prevention tips
• King County Conservation District has lots of free videos and workshops about healthy soil and water and wildfire preparation, and they’ll test your soil’s nutrient content for free. P.S. we the public elect their Board of Supervisors!

Related posts

• Eco Restoration 01
• Eco Restoration 02
• Eco Restoration 03

Let’s have a tiny critter house party!

Kristin and I created a bunch of bumble bee and mason bee houses, plus a stack of tiny bird houses. What follows are pics leading up to that, and from the party. At the end are some resource links for things like bees, polinators, and native prarie.

Pics

Fall 2023 - Start a prairie

Earlier this Fall, Kristin started to create a Prairie in the backyard. This involved rototilling the grass that was out there, watering to get weed seeds to germinate, and then covering the backyard in plastic to kill the newly germinated weed seeds. In early October, the plan is to plant native seeds.

This tiny rototiller was not up to the task…

A quick trip to HandyAndy and we are in business.

After the soil has been tilled, we laid plastic.

Fall 2023 - Build Critter Houses

I bought a staple gun and we went to work.

The bumble bee house is the most complicated of the houses. It has a little tube entrance into a ‘mud room’. Then there is the ’nesting room’ with a bunch of bedding material already there. Bumble bees like to nest in abandoned mice nests. Notice the nice bit of bedding…just leaves and grass I picked up off the driveway.

Fall 2023 - Critter House Painting Party

Now that we have built dozens of critter houses, we are ready to have a party and have folks come over and decorate these and then take them home. Here’s the start of the day, before folks show up.

Results from the critter house painting party…

Resources

Bee Resources

• Mason Bee Revolution
• Crown Bees
• Bumble Bee House Plans
• Mason Bee Mud Box

Prarie Resources

Related posts

• Eco Restoration 01
• Eco Restoration 02

"""COP27 must be a reset of the system. A moment to abandon the old,
profiteering, polluting world order, and a time to reimagine and rapidly
implement global collaboration that centers equity, justice, science, and
humanity."""

international

• democracy now
• common dreams
• black agenda report
• russ’s hopeful blog
• the guardian
• npr

environmental

• global justice ecology project
• indigenous environmental network

local (seattle / wa)

• eat the airwaves
• crosscut
• the urbanist
• kexp

tech

• the register
• slashdot

• https://photos.app.goo.gl/djw2KZJwBBiDonTb6

Strawbale links:

• https://strawbale.com/
• https://strawbale.com/straw-bale-fire-resistant-southern-california/
• https://strawbale.com/wildfires-straw-bale-construction/
• https://limes.us/strawbale-walls-survive-california-fires/

We can reverse climate change.

Overview

I started this in Eco Restoration 01 back in 2020. Now it’s 2022, so how are things going, what have I learned, and where are we going with this? 2023 Update …

Pics

Spring 2022

All my bees from last year died. So I ordered more bees, both packages and nucs. I installed bees in my backyard and (key point) got neighbors interested in hosting hives in their back yards. I have bees in 4 yards this year.

I also ordered some trees and shrubs to provide food for critters.

Summer 2022

It’s been a dry hot year. I installed some sprinklers to help the plants.

Learnings

• Honey bees don’t seem to do that well in the PNW winters
• Native bees love the clover I planted on the hugulkultures
• Logs take a long time to decompose
• An old, used, low mileage, low range (~50 miles) Nissan Leaf has been a game changer. I have taken a few longer trips where I needed to fast charge. But mostly, my lifestyle has evolved to just driving in the 5 miles around me…to yoga or to get groceries. Otherwise, I take the bus, lightrail, or train. I still have a gas car, but I almost never use it.
• I am going to increase my focus on planting a variety of flowers and plants for a variety of pollinators

Other thoughts

I sold my sailboat this year. Then my partner and I joined a small sailing club and have been spending our weekends sailing dinghys around the local lake. So now I am spending a lot less money on a sail boat and actually sailing more. It’s pretty fun sailing a dinghy around, and it’s only a 5 minute drive to the beach. I might still go sailing around the world on a full size boat, but it certainly is interesting that I can have tons of fun doing something local, too.

Links

Political

Global Justice Ecology – love this site

• https://globaljusticeecology.org

Maria Thomchick is amazing

• https://kodxseattle.org/eat-the-airwaves/

I’m sure you all know this one…

• https://democracynow.org

Russ sends a hopeful message to us through this site…

• https://www.russvernonjones.org

Building

I’m thinking of ways to make our living spaces more ecological and fire resistant

• https://strawbale.com/
• https://strawbale.com/wp-content/uploads/Fire-ResistiveStrawbaleWalls-CASBA.pdf

Gardening

Check out the tall grass meadow plant list in the project resources section.

• https://www.seattleparksfoundation.org/project/maple-leaf-reservoir-park/

Here’s a good site with local native meadow plants and recommendations:

• https://northwestmeadowscapes.com/

And if you’re gonna do some quantity, these guys are cheaper but have a $500 minimum.

• http://www.heritageseedlings.com/native-seed

I have purchased cover crops from these folks (clover, for example):

• https://www.highmowingseeds.com/

I have purchased several pounds of wildflower mixes here

• https://www.seedsavers.org/bee-feed-mix

And you can get free seeds from psba here:

• https://pugetsoundbees.org/psba-shopping/#!form/Shopping

A few years ago I ordered 100 packets and then walked around my neighborhood and gave them away to my neighbors…one packet at a time. I have no idea how many people planted them, but that sort of outreach can have a huge effect. I’m doing the same thing this year…as soon as they send me the seeds.

I have purchased various trees and shrubs both locally from swansons and also from burnt ridge

• https://www.burntridgenursery.com/

Summary

I’m just playing around with the idea that if I change what’s important to me, then my actions follow suit.And if I start making the planet and our ecosystem my priority, then … who knows what good can come of it?

We can reverse climate change

Overview

To this end, I’ve started several simple projects around my house:

• hugulkultur
• beehives
• plants

My 3 Step Plan

1. Build Soil
2. Encourage Nature
3. Goto 1

Clearly I don’t know what I’m doing, but hey, we all have to start somewhere. In fact, #2 is really part of #1. So this simplifies to:

1. Encourage Nature
2. Goto 1

I guess #1 is:

a. build soil b. encourage bees c. encourage fungi d. … what else?

So, with that nav chart, here we go:

Summer of 2020

Build Soil

A call to an arborist and I have a big pile of logs:

Time with a come-along and my logs are up in my front yard:

While you can try to get wood chips from chipdrop.com, I’ve had better luck just calling arborists directly.

Put chips on logs, plant things, and see what happens:

Ok, hugel is going. I’l check back on this from time 2 time…

Encourage Bees

I’m following the advice of Dr. Leo Sharaskin:

• Sharaskin YouTube Videos
• Swarm Catcher Plans
• Frame Plans
• Beehive Plans
• Book 1
• Book 2

A couple of thoughts…

• I just watched enough of the videos and read enough of the books to know I wanted to do this. Then I just started doing it.
• The swarm catcher and beehives are really well thought out
• I modified the frames to be way way less work to build. I’ll post my modified plans soon, but the key difference is that I just router a 3/4" slot in the underside of the top bar instead of wasting all the material suggested. And instead of slabbing 2"x10" or whatever…I just use 1"x8" and rip 1" strips for both the side and bottom bars. This reduces the work to create the frames by orders of magnitude. TODO: post side-by-side photos.

I started this project by building and deploying 8 swarm catchers:

On to building the full size beehives…

I set primitive stops for all the lumber cutting:

After some time, I have frame parts:

Top bars go in here, and then I run this over a router table with a 3/4" bit:

Hives get heavy:

Cutting stock:

Assembly:

Glueup:

Painting:

Adding Frames:

West side of my house. Dry. Covered in cardboard. I’ll plant clover here in the spring.

Here’s my denuded backyard. Perfect to start a restoration project:

The Summary

Ok, I have bees checking out the lavender in my yard. But so far, no bees have moved into my hives. I’ll add blog posts as things progress.

Links

• https://www.pugetsoundbees.org/
• https://en.wikipedia.org/wiki/H%C3%BCgelkultur
• beehives](http://horizontalhive.com
• https://www.pugetsoundbees.org/wp-content/uploads/2013/06/BeeFriendlyPlantsBrochure-2.1.pdf

Books

• http://horizontalhive.com/keeping-bees-with-a-smile/fedor-lazutin-book.shtml
• http://horizontalhive.com/keeping-bees-with-a-smile/georges-layens-book.shtml
• https://www.powells.com/book/mycelium-running-how-mushrooms-can-help-save-the-world-9781580085793

My informal notes:

Power Committee – Oct 15 2019 8:30am Pacific time – Hoh Room

Update on Progress of GENESYS Redevelopment

NWPCC has developed a finer grained flow modelling system called, GENESYS that aims to either replace or at least complement the venerable HYDSYM?

Context

• GENESYS RFP
• GENESYS
• HSPF
• Wikipedia hydrologic modeling
• proprietary flow modeling software

Note: I cannot find reference to HYDSYM, though I did find reference to HSPF.

Notes

• modeling hourly constraints for thinking about markets
• are currently calibrating the model with BPA and other stakeholders
• aggregate hydro -> plant specific dispatch (shape matters)
• previous model was all in FORTRAN, moving to modernize the model (language?)
• lots of non-power constraints in model, e.g. fish
• vetting with river operators
• model tries to emulate time frames for markets

One key difference that was discussed at length was the difference in the monthly model of HYDSYM vs. the moving to the finer grained weekly model.

• goal: verify that GENESYS results match HYDSYM, which is made a little tricky b/c weekly boundaries do not always fall on monthly boundaries, therefore some interpolation is required in GENESYS

Council member questions

Questions

• are/is GENESYS using the same or similar constraints as BPA?
• will the results be radically different than BPA vs CRSO vs NWPP?

Answers

• No. Results will be roughly the same, modulo minor variance introduced by interpolations.

Buzzwords

• KSFD
• Aurora
• CRSO
• SCCT
• GRAC : generating resources advisory committee
• NWEC : northwest energy coalition

References

The Seventh Power Plan was referenced repeatedly:

• 7th Power Plan

Load Forecast

Load forecast based on:

• population growth and migration
• behind the meter solar
• transport / EV penetration -> results in the load forecast

Data sources

Population

• ACS data for population flows (subset of census)
• Global Insight (apparently this is used all throughout the industry for population modelling)
• IHS-Global Insights : population growth is main driver

Solar

• EIA : energy information administration : annual energy outlook
• SEDS : state energy data systems
• NREL : system advisory model
• Energy2020 : load forecasting tool
• “cogen” : affects energy (aMW) and peak (MW)
• forecast an aggressive build out of solar: ~5 MW additional capacity from 2025 and 2041
• solar will be very useful for peak load reduction in summer

However, a council member questioned this last point, pointing out that the supply and demand are offset by several hours in Idaho and the response was that the model was aggregated at a monthly level. I find this somewhat strange and IMHO this analysis relies on the supply and demand being matched, either naturally or via some sort of load shift (hydro/battery/etc.).

EV Transport

• late afternoon peak as drivers arrive home and charge vehicles
• expect an increased electric load from EVs
• low auto turnover rate will limit velocity of EV penetration
• charging profiles from AVISTA
• drivers : population growth, vehicle retirement, vehicle efficiency
• network transmission costs vs p2p
• look at the interconnection queue

Solar, Battery Storage, and Solar + Battery Storage Reference Plants

This presentation was far too detailed for me to take notes. Hopefully the slide decks are available.

Council Meeting – Oct 15, 2019 1:30pm Pacific time – Nooksack Room

Presentation by the Orca Task Force: Dr. Les Purce and Stephanie Solien, Co-Chairs of the ORCA Task Force.

I did not take notes on this, suffice to say, the situation is looking really really bad for both Salmonids and Orcas.

Briefing on water resources in the Pacific Northwest as Related to Marine Heat Waves, El Nino, and Climate Change: Dr. Nick Bond, Washington State Climatologist.

Summary

• lots of variability, but the trends are
  • warmer weather, less cloud cover? esp summers, leading to increased evapotranspiration, esp in summers
  • shift in precip from snow to rain
  • shift in precip to wetter winters (rain) and drier summers
  • precip changes result in changes to the flow curves of rivers
    • less snow release dominant
    • more rain release dominant
    • release times shifted towards wetter winters
• overall, these systemic changes are going to occur slowly over the next 50 years with lots of variability

My comments: if you look at the cold water refuges plan and the forecast for the change in precip patterns, you should probably be alarmed. Salmonids like cool rivers and tend to die in water that is too warm. We are def looking at lower spring flows which are also warmer. Uh oh.

I chatted with Dr.Bond after his presentation and suggested that one could misinterpret his message as, “we do not have to worry about climate change in the PNW”. He said that is definitely not the message he is intending to deliver. We also chatted about a new term called, ‘climate privilege’, that describes how folks in places like the PNW actually have privilege, much like ‘white privilege’, of not having to react as quickly as people in other locales.

2020 Regional Technical Forum Work Plan – Oct 16 2019 9:00 am Pacific Time

• RTF : regional technical forum
• NG : natural gas analysis
• residential building model
• connect disparate utilities ~ coordinate research
• DR : 3% of budget ~= 50K
  • EE + DR (energy efficiency and demand response)
  • tools need to be improved, historically focused on EE, not DR
• Expand scope of RTF charter to include both EE + DR, and NG efficiency

Update on key Environmental Protection Agency activities in the Columbia River Basin - Oct 16 2019 10 am Pacific Time

• Chris Hladick, Regional Administrator, US EPA Region 10
• State of the River Report
• Action Plan for the Columbia River Basin Restoration Act
  • https://www.congress.gov/bill/114th-congress/house-bill/2469/text
  • https://www.congress.gov/congressional-report/111th-congress/senate-report/358/1?r=5
• 2019 Status Update: Columbia River Basin Toxics Reduction
• $1M Jan 2019 Grant Program
• DOJ appealing water quality models (I don’t understand the issues here)
• Draft Cold Water Refuges Plan
• Identified 23 areas where cool tributaries enter the Columbia, 12 of which are critical b/c they account for ~80%? of cold water in system
• EPA concludes there are enough refuges now but will not be in the future
• Accepting feedback until Dec 3
• River Keeper Agreements
  • permits regulate only some pollutants
• Flex Spill Agreement
  • https://archive.fisheries.noaa.gov/wcr/fish_passage/fcrps_opinion/flexible-spill-operations.html
  • https://www.bpa.gov/news/newsroom/Pages/Flexible-spill-agreement-aims-to-benefit-salmon-and-hydropower.aspx
  • states develop their own guidelines for TDL and then the EPA has to vet/approve them?

Council Question

• main stem TMDL cold water refuge?

Answer:

• TMDL : total maximum daily load : heat contributions
• refuges report -> importance of places for fish during times of warmer temp regimes
  • not embedded in the TMDL…what does this mean?

Council Question:

• What are the contaminants? What are the measurements?
• Do you measure the surface? Where are the points of compliance?

Answer:

• variability in time of year, location on stream system, depth
• mitigation opportunities are cool regions that vary by TOD, TOY, season, etc.
• many different fed/state/tribal entities are involved

Council Question:

• Are requirements different for adult spawning vs juveniles escaping?
• Main stem is a corridor, not a spawning ground
• Year round temp standard unnecessary b/c only important when fish are present, right?

Answer

• Web of water quality standards. Varies by season, and use by fish
• Focus is on TOY when fish are present and temps are up
• Refuges give fish a place to shelter so they can rest before moving on

Council Question:

• Who is going to manage the cold water refuges?

Answer:

• EPA is identifying where they are, how to protect and enhance, but implementation and management is up to the states

Council Question:

• What about upstream pollutants from Canada? Area of the Kootenay…Note: a coal mine and effluent/tailings ponds were mentioned, but the coal company was not identified.

Answer:

• State Department interacts with Canada
• Environment Canada is the Canadian agency

Briefing on regional energy future from an independent power producer perspective: Orijit Ghoshal, Invenergy. - Oct 16 2019 10:45 am Pacific Time

• IPP : Independent Power Producer
  • developer / builder/ operator
  • not affiliated with a utility monopoly
  • no rate payer or guaranteed cash inflow
  • need long term agreements for revenue certainty
  • transmission rights are different (no eminent domain for one thing)
• fuel supply and transmission rights are not firm
• do not require firm supply and transmission
• do not have imminent domain rights
• need voluntary agreements with landowners (currently ~16K locations)
• need close partnerships for those agreements
• Invenergy the largest IPP in North America
  • developing desalination plants
  • 23GW nameplate capacity
  • 33B in capital
  • sell a lot of stuff to Pacificorp
• listed a bunch of wind farms
  • Hood River : 72MW sells directly to LADWP
  • Grand Ridge : batter - PJAM freq regulation market, sub-second response
• 600MW deployed wind generation across 13 facilities
  • used to take 2 years to develop a wind farm, now it’s down to 1.25 years
  • time frame for a battery facility is on order of months (and they can be relocated as necessary)
• BPA owns 75% of PNW transmission
• zero cost dispatch of resources makes economic sense
• energy flows from CA <-> WA are changing
• opportunity for IPP to enter the market
• example scenario
  • low air pressure in winter -> cloud cover, low wind, and cold weather
  • need non solar non wind energy supply
  • (seems like trying to sell their gas gen plant that is in danger of being stranded)
• BPA is not FERC regulated
• all generators are revenue sources for BPA
• resource adequacy requirements for 8GW?
• interconnection challenges
  • siting
  • low hanging fruit (sites that are flat w/ good wind or solar) have been taken
  • constrained by physical characteristics
  • solar cost curve is declining faster than wind
  • federal tax credits for solar expire later than wind
  • (explains the rush on wind farms ATM, gotta build and complete before credits expire)
• still need to address load
• to get financing, need firm transmission rights, but transmission is limited
• need increased transmission capacity in the system
• firm transmission reservations are made 24x7 and typically day ahead…but these reservations are not being used 100%, in fact they are only used a small percentage of the time, so the system is booked but under utilized
• RECS : renewable energy credits - that’s what Facebook is buying with it’s wind farm/solar? project, not the actual electricity, which will be sold on the market
• firm zero carbon energy : code named ‘unicorn’ project : can be done with BPA hydro. but w/o a firm backing source like hydro, this is pretty expensive to do with current batteries, etc.
• transmission congestion
  • need a region wide operator
  • need a single market
  • advocating with american wind energy association and renewable energy northwest, and others
• when to buy / sell an asset?
  • outlook on asset performance
  • transmission injection rights
  • use of money
  • development transfer - build and sell facilities more and more as utilities become more comfortable with RE tech, and the utilities have a preference to own rather than PPA
• given high demand for assets, Invenergy is selling them and using the $$$ to purchase transmission rights

I haven’t made any effort to clean these up other than to find relevant links that I’ll research later.

–Todd

OpenADR

Presenters: Rolf Bienert and Don Duichinos

• Oasis Inter operation created by Oasis Open
• Sends requests not commands, customers can opt in/out
• Service events have start/end/intervals
• Service events can be re-sent with updates (there’s a sequence number)
• Security is PKI based
• Security model vetted by:
  • NIST
  • SGIP #1
  • SGIP #2
  • IEC
  • SCADA

Questions

• How are certs revoked and/or updated (CRL for revocation)
• Can organizations provide their own CA (Yes)
• How are upgrades performed? OTA?
• Are commands idempotent? Is every command ack’d or is this fire-and-forget?

Also discussed

• 2030.5 IEEE 2030.5-2018 - IEEE Standard for Smart Energy Profile Application Protocol
• CTA-2045 Port to dispatch (control) water heaters
• CA Rule 21
• R.A.T.E.S.org – I cannot find this site
• Quality Logic has online learning for OpenADR

DERMS

• What is DERMS? Where are you in that value chain?
• Is it a distribution tool? A transmission tool?
• 3 Forms/lineages
  • VPP Virtual Power Plant
  • ADMSAdvanced Distribution Management System
  • Hybrid
• EMS/GMS ?
• HEM
• Australia soon 40% of their gen will be on the customer side
• Advance forecasting, load flow, load management, asset use
• Devices behind the meter: settlement, delivery, back office management
• “We” have no idea how to change business processes when deploying DRMS:
  • interconnection permits, DR group, ops group
  • utilities actively looking at this
  • have to learn how to manage DERS
• Capacity relief on a substation
  • forecast DERS, monitor, how is it registered?
  • ADMS registration? Need an interconnected update process
  • reactive power? interconnect agreements?
• Operators need response on the system
• Market structures, policies, programs
• How to unify perspectives?

BC Hydro

• Problem With Theft
• Collecting 2M metrics on hourly/5/15minute intervals
• Transformer impedance / voltage profile -> looking for anomalies
• Energy balancing : reconciling billing vs generation, looking for theft or data quality errors
• Drivers: large theft scenario of grow ops
  • secondary issues of safety
  • theft totalled 7% of load, equivalent to ~80K homes
  • leads to premature asset failure (transformer overload)
• AMI - enhanced field investigation team
• Over time, theft teams went from 60% to 5% of investigations resulting in identifying theft
• Result: $732M in savings
• Q: Does the result factor in the theft detection costs (system creation, maintenance, investigation teams, etc.)?
• Used external solution to start, Data Raker, from 2011 to 2014
• Used DR while developing in-house solution
• Deployed DR and in-house solution side-by-side
• A/B systems compared in final analysis
• Key takeaways:
  • IT/OT bridge : the business and technical relationships were key
• Maintenance system is at 78TB currently and grows by 6TB/month
• Stack uses Greenplum and HortonWorks

Oh DER!

• Status Quo is lacking framework for incentivizing storage
• Opportunity to decarbonize / digitize / and democratize
• Create monetization and value
• Pricing and billing engine for TOD pricing took 10 years

Strategy across industry

• Breaking down silos

• What or the goals and objectives?

• Evaluate framework / value streams / DOU / Rate designs

• Need financing mechanisms to bring systems to people that cannot afford 10K for a battery system

• Need confidence in data

What’s needed in the regulatory environment? …crickets.

Pathways to Deep Decarbonization

• Northwest Power and Conservation Council

  • 2021 Power Plan
    • Next Meeting
    • Adequate power system
    • Protect fish and wildlife
    • Do this within the power plan process
      • energy efficiency
      • DR
      • geothermal?
      • transmission
    • 70M buildings using natural gas
    • CTA-2045 Port to dispatch (control) water heaters

• PSE

  • transmission - IRP
  • distribution - planning is somewhat separate
    • generation
    • smart grid & AMI & batteries & self-healing -> increased reliability and flexibility
    • feeder deployments have decreased outages
    • trees lead to outages on the grid
    • w/o digitization decarbonization won’t happen
  • Analytics
    • Get To Zero
    • Multiple types of customers : low-income -> AWS/Nordstroms (data centers)
    • anticipate and proactively solve problems
    • Web/billing/payments/operational
    • AMI - 2023 Landis & Gyr
    • Used to be on their board of directors
    • Something about customer preferences
  • Resiliency/Micro-grid
    • Example 2MW in Glacier Whatcom County
    • 10mi long radial ~ 100" snow/year
    • difficult to restore power
    • battery installed allowing
    • restored power in August
    • large commercial / military customers
  • Pilots & Partnerships Ecosystem
    • Pilots allow controlled ecosystem w/ controlled costs
    • Responsible to customers and the environment
    • Game Changers
      • EV infrastructure
      • WA - 40% of carbon emissions are from transport
      • WA - 50K electric vehicles
      • WA - project 400K - 1M by 2030
      • UpNGo – can test drive EV vehicles? Cannot find link
      • Heavy transport / Maritime
      • Pro LPNG facility in Tacoma
  • Hoped for achievements in 5 years
    • No Cyber Security events
    • Cost effectively meet goals in clean energy
    • D&I - unconscious bias / trust / etc.

Cyber Cyber

What are the top Cyber risks?

Answer

• build relationships before incidents happen
• stuxnet
• halvex
• it’s an evolution
• not going to see the exact same thing, cannot predict threat vectors
• build resiliency models to detect, categorize, react to threats

Who faces Cyber attacks?

Answer

• today, distribution
• targets of opportunity, places w/ most potential for damage
• as distribution becomes hardened, then attacks move
• transmission is expected next

What about vendors? What role do they play?

Answer

• recent attacks via vendor pathways
• old devices w/ old vulnerabilities
• new devices w/ vulnerabilities, un-encrypted firmware is easily examined and rooted
• not signing firmware, even large firmware vendors
• micro-grids with smaller companies in these spaces may not have resources to properly defend against these attacks (however it was also stated that the larger companies are also doing a poor job of defending, so seems a bit FUD).
• STIG Open Source threat modeling software

What role do Regulations Play?

Answer

• DOE
• FERC
• NIST
• NERC
• IC Level Chain
• IC Level Chain
• NERC - focus on legacy environments, on the cusp of engaging with the cloud
• NERC - does not have language? for cloud cyber security
• Threat Vectors
  • Infra
  • Operations
  • Data protection
• Weakest link is humans
• Employees must be aware of attack vectors
• Leaders must start investing in employee awareness
• Awareness training is a constant thing

What happened in March?

Answer

• Fishing attack, stage 1 malware
• Followed by lot’s of, trust us, sorry, we have to keep secrets, blah blah

What happened in the Ukraine?

Answer

• Blackenergy3
• not designed to attack the grid
• stage 1 -> 4
• found value
• attacked SCADA
• deleted stage 1 and started scavenging for VPN connections
• managed routers and switches have VPNs into target environments

How to trust Vendors?

Answer

• Segmented networks
• 2FA
• Scoped roles/access to the secure resources/networks
• Verify vendor compliance
• Do not blindly trust vendors or their compliance
• Pen tasting?
• Validate, check policies, use 3rd parties for risk management
• Need to involve cyber-security from the beginning
• ICS Kill Chain
• Procurement process (RFP) needs to have cyber-security as part of the process

Threat Monitoring

• visibility fabric devices
• anomaly detection devices
• what to do with alerts? need to invest in human capital to deal with alerts and make meaningful decisions
• train from within
• Gridex Government training for incidents
• SANS.org Has paid training for ICS

Other resources

• Defcon
• Blackhat
• ICS Security Conference
• Houston Security Conference
• Follow “Clint Bodungen”
• Follow “Robert Lee”, owner of dragos

Public Power

Flathead Electric

• relying on customer networks and modems was a disaster
• water heater controller has high acquisition cost
• BPA does not supply real time information (costs)
• Degraded the battery installed at a youth home by 20% in 5 years by cycling from 0 -> 100% charge
• TOD rates
• Micro-grid an a 330 mile radial feed

Burbank

• not part of CAISO, part of LADWP
• cannot use market to help w/ intermittent resources
• 100% smart meters
• vertically integrated
• EV charging TOU rates
• fiber over the city
• local gen

Fort Collins

• biz cases
• fund outcomes not departments -> 7 outcome areas
• electric utility is under multiple departments -> outcome is safe community
• 3x bottom line : economic / social / environmental -> changes how you think and do business

Flathead

• need big thinkers, don’t be afraid, take chances

Burbank

• costs now provide benefits later
• hoover dam as an example
• had vision and big thinking
• integrating RE for the environment is new vocabulary
• think in decades

Biggest Forward Thinking Investments?

Burbank

• Inter-mountain Power Project (IPP) - 18MW Coal Plant
• Online since the ’80s
• working to accelerate plant shutdown and replacement with NG plant
• Southern Transmission System
• 50 year commitment to a gas plant
• Renewable hydrogen?
• the best of many imperfect solutions

Flathead

• coop
• know your audience / members
• local gen was asked for
• landfill to gas plant - first in Montana
• timber manufacturer - biomass -> energy
• community solar
• members are willing

Ft Collins

• One of 6 places of invention mentioned in a Smithsonian exhibit
• 500K cost share was from community members
• Collaboration and partnership
• High-tech area
• Climate economy

Key Takeaways?

• No shareholders
• No state regulations
• Local control and accountability
• Not mandated as investor owned utilities are
• Taking shareholders & regulators out of the equation -> yields advantages and more resilient decision making process

GW of DER

…

Regulation and Innovation

“Regulators need the same level of detail as a CFO”

• whole plan
• future sunk costs
• mitigating risks
• societal benefits

Random Bits

• IRP Integrated Resource Plan
• WA State Bill 51162019 Senate Bill 5116: Supporting Washington’s clean energy economy and transitioning to a clean, affordable, and reliable energy future
• WA State Bill 1444Concerning appliance efficiency standards.
• In WA transport is 45% of our GHG emissions
• AMI Smart Meters (bi-directional)

I introduced the Energy Analytics Project in my previous blog post, Energy Analytics Project - Part 1 : BackStory.

This post will dive into how to use this project and hopefully give you some ideas for how to extend and contribute to it.

The following posts will dive into the more technical aspects of how, what, and why as well as highlight some of the false starts, dead-ends, frustrations and successes of trying to create a data-centric application.

How To / Getting Started

I wrote a How To / Getting Started Guide here. Rather than repeat that information, I’ll assume you’ve read that page and want even more of an introduction or ideas for how to use this project.

Beyond Getting Started

OK, so you’ve played around with a data set or two, you’ve read through the CAISO documentation that describes the various data feeds that are available… now what?

Here’s where things take off.

Together, we need to figure out how to answer basic regional questions like:

• What are our real hard vs soft energy needs?
• What are our predicted energy needs over the next 20-50 years?
• How can we most cost-effectively de-carbonize?
• How can we do that in ways that build community, empower citizens, and reverse discriminatory impacts to ‘front line’ communities?

Democratize Data

I propose that freely available data and analysis is critical for our success.

My reason for putting this project together is to serve as a model going forward. CAISO has published enough data that we can reason about that market. I’d like to get similar data sets from Oregon, Washington, Idaho and Montana. If we can get some idea as to the shapes of these different markets and how they can be beneficial to each other, then we can make real progress.

I’ve written some ideas on that here.

One thing that I’m very concerned with is restricted access to data. We need public access to data, public analysis and counter analysis, and informed decision making.

Pitch

So the beyond-getting-started pitch is:

• Help determine which data sets are useful for understanding the California markets
• Help retrieve those data sets from the other states (I can help with this)
• If the data sets are not available, then help cultivate relationships and retrieve those data sets

How can you help?

• Join the mailing list
• Become a contributor – this is Free Software, contributions are welcome
• Become a data curator – help me source new data sets and manage the existing ones
• Become a data scientist – start creating Jupyter Notebooks and sharing your knowledge

And if you are a domain expert in either data science or the energy sector, please jump in and lend your expertise.

In the next post, I’m going to dive in and discuss the platform architecture, the tooling, and how the design trade offs interact with daily use. When posted, it’ll be here: Energy Analytics Project - Part 3 : Architecture.

-Todd

Back Story

I took this summer (2019) off to address the climate crisis. Somehow, there must be a way for software engineers, like myself, to affect climate change… and what better avenue than the energy sector?

For starters, I attended a couple of energy conferences. That’s where I learned about the price of electricity going negative in California. Apparently, solar electricity generation peaks a few hours before consumer demand for cooling peaks. It’s also where I learned about our Pacific Northwest hydro electricity prices going negative during certain flood events. Apparently, during flood events that coincide with salmon runs, the dams run excess water through the turbines to protect the fish. However, because this energy was not asked for and is just dumped onto the grid, this causes the price of electricity to go negative. I also learned that there are significant amounts of wind energy in Idaho and Montana… (Unfortunately, these presentations were anecdotal and I do not have the data to back up any of these assertions. Not quite.)

So the question is, how can we integrate and optimize the energy sector for renewable energy (and most importantly, decarbonizing the energy sector)?

About this time a friend told me about the data available on the CAISO OASIS site, and something clicked. You see, this site exposes an enormous amount of energy related data for the state of California (in fact, CAISO stands for California Independent Service Operator). If someone could consolidate data feeds in the energy sector, across the western region of CA, OR, WA, ID, MT, then perhaps we could analyze that data from the perspective of decarbonization? How can we most effectively use renewable and non-renewable energy sources?

To answer these questions, I created the Energy Analytics Project (EAP) and it is currently pulling down over 90 separate data feeds from CAISO OASIS, totalling about 1.5 million time-series data points.

In this blog post I’m going to introduce this project, describe why I think it’s interesting, and why I think you should join me and help out.

So, read on to find out more about this project…

Primary Objective

Decarbonize the energy sector.

I’ve written a background piece on why I started this project here.

That should give you a general grounding for project intent, namely, let’s get enough data to make informed decisions about our investments in the energy sector.

Goals

Create a community of developers, researchers, data scientists, and policy advocates, each leveraging the skills of the others.

Community

Developers, like me, help by providing the infrastructure and tooling to get data into one place, in an accessible platform. The platform I chose to start with is:

• source code in github
• data in S3 buckets
• databases in Sqlite3
• reports in Jupyter Notebooks
• distributed via github

My hope is that researchers and data scientists will start consuming this data and start generating reports. Then, we need policy advocates to start using these reports to affect the decisions being made at Federal, Regional, State and local levels.

Tooling

Provide adequate tooling for managing all this data. I’ll defer further discussion of the tooling to a later post as that’s the rabbit hole that I am always drawn to.

Research

Data scientists use a common set of tools, so this project should accommodate that. To that end, the tooling is based on Python3, Jupyter Notebooks, and Pandas. However, you can use whatever tooling you want, write your own data pipeline, whatever, it doesn’t matter. One thing that may be unusual about this project, however, is that all the data is surfaced as Sqlite3 databases. The primary artifact of each data feed is a Sqlite3 database.

Why is this interesting?

For me, it’s intrinsically interesting to get such detailed information about the worlds 5th largest economy and how it’s energy sector functions. But what’s more interesting is taking the CAISO model and evangelizing this type of data transparency to other agencies in the West. If we can get some small subset of the OASIS data feeds from entities like Bonneville Power Administration (BPA), and the western states such as Oregon, Washington, Idaho and Montana… then we can start to understand the potential costs and benefits of different types of energy resources, pricing strategies, etc. at a regional scale.

As the need to de-carbonize the energy sector becomes more and more apparent, we will need to make important decisions about where to invest our time and money. Do we build nuclear reactors, and if so why and where? What is the time-cost and ROI for this? Should be invest in solar, wind, or hydro? Are these systems built in a way that is complementary, e.g. are we using our hydro for capacitive storage? What is the cost for each of these solutions? How much of the demand can be shifted by live pricing?

There are a lot of questions to be answered here, and we need this data to be easily and publicly accessible so that we, any and all of us, can get started with trying to make sense out of the situation.

How can you get involved?

Please join the Energy Analytics Project mailing list.

See Energy Analytics Project - Part 2 : How To for instructions on how to get started with this platform.

–Todd

History

From 2009 to 2019…

• read books (Joe, Fred, etc.)
• write little code snippets
• post little how-to-tutorials showing how little I understand about Erlang
• wander away as I lose focus and time
• repeat.

This year, after Joe died, I thought…I’m not getting any younger either. I decided to take the summer off and really study Erlang. So far the process has been:

• read the same books, but only as references
• have a real problem to solve (I’m basing a business on it)
• write little code snippets (escripts) to POC business components
• document what I am learning on my personal website
• design the full scale app
• implement … oops hitting the road blocks…

Road blocks

Erlang installation and tools

I wish I’d known about asdf much earlier. That’s really simplified my dev box configuration. I wish that rebar3 and kerl and other docs just pointed this out. In fact, I wish there was a golden path published that just says, look, here’s the common accepted tool chain you are going to want to use. Stay on this path, and life will be easy. rebar3 is a big part of this story, but not the whole thing.

Migrating from escripts to real apps

$ rebar3 new escript myfooscript

I started with escripts because I didn’t want the hassle of understanding OTP or anything else as I was writing POC snippets to understand the basic Erlang libraries. So I’d write an escript to shell out and curl some resource and gradually build up an application that ingests data and writes into a database, blah blah. But along the way I could learn about xmerl and other useful libraries. Oh, and I could run the app and see STDOUT. Cool! Just like any other language I’m familiar with.

So I roll up my sleeves and decide to create an app…

Naming things is hard

$ rebar3 new release|umbrella myfooapp

Um, why does ‘umbrella’ exist as an option here? Is it the same thing as a ‘release’? Is it necessary? Why is ‘release’ called ‘release’? This is an overloaded term, as there are ‘releases’ which is a thing, and there’s a ‘release’ command, and there’s a ‘release’ target in the rebar3 command. And there’s an ‘umbrella’ in there, too. OK, release == umbrella, so I pick one.

$ rebar3 new release myfooapp

Where is STDOUT?

So, let me say, rebar3 is great. I barely remember what it was like to use sinan or faxien. (And while Loic’s Makefiles look amazing, I’m nowhere near smart enough to use them.) But here’s the thing, I had to watch a number of videos before I found out what the shell command was. This is important to me because the next block I had was… how do I start my application and see the output from it? I want to watch STDOUT, right? I was watching some conference video when someone mentioned that ‘shell’ was the coolest part of rebar3 and I’m like, ‘what is that’? And voila! It’s what I needed.

This should be on the golden path.

$ cd myfooapp
$ rebar3 shell myfooapp

Where is the golden path?

It’s weird that there is no golden path. In this talk, at 14:50, Fred talks about how it’s only far into various Erlang books that authors start talking about running code outside the shell, in contrast to K&R C where it’s discussed on page 6. In fact, the entire (excellent) presentation is on the terrible state of tooling in the Erlang ecosystem.

begin rant

Here’s what’s weird about Erlang : Books are out of date

• The books are already out of date. Even LYSEFGG still talks about systools and reltool even though rebar3 and relx have supplanted them? What’s weird about this is to be reading the rebar3 documentation, then cross-referencing with relx, and reading the various books that describe the old ways of building releases and relups, etc.

Docs are hard to follow

• relx documentation is pretty hard to follow
• Community support seems fragmented
  • Github shows relx has 41 open issues and 6 open pull requests, with 89 contributors
  • Github shows rebar3 has 107 open issues and 11 open pull requests, with 239 contributors

Who’s maintaining these tools?

If I were Tristan or Fred, boy I’d be tired of supporting this stuff.

Where are the videos?

Here’s another thing that’s weird…

I cannot find videos on how to use the Erlang shell. So on top of that weird fact, I’m finding out about erlang by watching Elixir videos. I was watching an Elixir video on ‘wobserver’ and realized, ‘Hey! I bet that’s just a web interface on some Erlang thingy called “observer”’. So then I find out about ‘observer’ and how useful that is. Great! But things like, how to I do basic stuff like navigate from one process to another, what do I do when everything seems hung and broken, what do real live seasoned Erlang devs do when ‘stuff hit’s the fan’? How do they go about examining a running system? The web seems to have nothing on this. But that’s just weird, b/c Erlang has been around for over 25 years, and any other language / framework/ runtime whatever has zillions of youtube videos.

end rant

Culture of Quick Fixes

OK, so enough ranting. Let me get into Fred’s mention of the ‘Culture of Quick Fixes’ (at about 21:00) by introducing my own steaming pile of a quick fix.

So here’s the deal. I’m trying to build a company. I have limited $$$, I can’t screw around learning stuff forever. At some point I have to build a real live app and try to generate some buzz. I’ll roughly describe my first app/service and run through my decision points…

App

The app basically pulls data from publicly available but disparate data sets, ingests them, normalizes them into a common data store, and then surfaces the data store for analysis, reporting, AI/ML, etc, etc. In a nutshell, take data, add magic, produce pretty pictures.

Process

• prototype in bash (or python, or whatever…)

Seriously, with just bash and I can query for my data, extract in, play around with it, etc. If I wanted I could hook it up to cron and probably move on to actually using the data rather than learning Erlang and how to build ‘proper’ infrastructure.

Prototype in escript

Here’s where I take small tasks and learn the Erlang libraries to accomplish what I’m trying to do. It takes about as much time to do this as the previous prototyping step. I find this to be a great way to both decompose a problem and learn a new language.

Start building industrial grade services

So now, rather than focusing on my business, finding customers, analyzing the data I’ve collected, or in any other way providing value that could bring in revenue and allow me to purchase more Erlang books… I’m veering off into the weeds and learning how to:

• build erlang apps for real
• build erlang releases
• wire all this into docker containers
• wire in monitoring and all the ‘industrial grade goodies’

Along the way I tend to post about what I learn here.

• Is any of this worthwhile?
• Am I just engaged in endless pencil sharpening?
• I dunno. I’m too close to the fire.

I’m having fun though. And it’s incredibly useful to my brain in some way I cannot describe.

The Problem

So back to “The Culture of Quick Fixes”. As I’m building my first set of components and learning about rebar.config and config/sys.config and more groovy stuff, I run into the issue of how to deploy Erlang to docker containers. The answer is, run the release with the foreground command. But this wasn’t obvious to me at first and took a lot of digging around the ’net, posting a question on the rebar3 discussion list, and just general noodling around.

But, if I’m building industrial grade applications, then I need to solve for issues such as:

• How do I emit metrics from my app?
• How do I monitor my metrics (e.g. Prometheus)?
• How do I hook up a visualization tool (e.g. Grafana)? Alarms?
• And how do I wire all this up into a containerized solution so that it’s re-usable and shareable?
• Don’t forget my “awesome” Makefiles! I like to have a common control plane for all my services, and common Makefiles are great for this.

Now, if you watched the end of Fred’s video, he was talking about many of these same things.

And I’ll suggest that what I’ve built is an example of exactly what shouldn’t be done:

• I ignored everything on hex. Why? Um, I don’t really understand hex that well yet, I’d have to go read a bunch of code to understand what folks have done, and I’m still learning Erlang by building things myself. None of these is a good reason, BTW.
• I only published my stuff to github, and not to hex. This may be a good thing as I’d just be duplicating other, probably better, efforts.
• What I built should probably be a rebar3 plugin, and not a Makefile with dependencies on rebar3 etc.

However, I’ll show you what this is because I think it satisfies a need and perhaps we can all come up with a better implementation strategy. Or learn something.

Erlang-Seed

So, here we have ‘yet-another-quick-fix’ called erlang-seed. Let me be honest, this is a ‘quick-fix’ to satisfy my own personal needs…

The Good

What this does is create an erlang application, inject it into a Docker container, and then spin up a local Docker cluster that monitors the Erlang app and surfaces the metrics via Prometheus and Grafana. So with one command I can generate an OTP application, all the infrastructure necessary to monitor it with common tooling, and spin it up and run it on my dev box. I find this to be really useful. YMMV.

The Bad

This is really crufty. It calls into rebar3 to generate the app. It should probably be a rebar3 plugin so that it can better evolve with the tooling. It could then more intelligently inject the necessary dependencies, instrument the code with metrics, etc. Right now it’s gross, as it just overwrites files as necessary using mustache templates.

The Ugly

• Do I have time to maintain this? No.
• Do I have time to make this a rebar3 plugin? Not right now.
• Will this probably break over time due to either rebar3, docker, or some other dependency changing? Certainly, eventually, yes.

Summary

The reason I’m sharing this is to just widen the discussion on what the Erlang education and tooling could look like. I think that generating full stacks that allow developers to focus more on solving business domain issues rather than forcing them to understand all the arcane details of whatever is under the covers is the ‘way to go (TM)’. There’s got to be a way to allow developers to maintain velocity while increasing their depth and scope of knowledge as necessary. I’ve been studying Erlang intensely for at least a month now, read the books countless times, and read the online docs and I still don’t quite understand how this all fits together. I mean I get the big picture. But there’s a lot of details.

Contrast this with Go. I was writing production code in 2 days. The quality was production quality after about a week. Back in the day it took me about a month to become proficient in Java, though it required reading Bloch’s book “Java Concurrency In Practice” to be ‘good’. Python was similar, it took about a week to ramp up, about a month to become comfortable. In all of these languages it was a relatively short and easy ramp up and then smooth sailing for years and years.

With Erlang it’s an incredibly steep ramp that goes on and on. I guess if I remember that I’m learning the equivalent of a language, an orchestration framework (like kubernetes), a database system, etc., then it starts to make sense.

Anyway, for my part, I’ll keep posting things as I figure them out.

-Todd

Launch Metrics

mkdir temp
cd temp
git clone https://github.com/toddg/emitter

Start the docker components.

cd emitter
make start

Wait a second and run docker ps to verify the containers started and stayed started:

$ docker ps

CONTAINER ID        IMAGE                     COMMAND                  CREATED             STATUS              PORTS                    NAMES
506443c4d497        toddg/erlang-wavegen      "/bin/bash -l -c /bu…"   12 seconds ago      Up 8 seconds        0.0.0.0:4444->4444/tcp   monitor_wavegen_1
b6b7cc6e0128        grafana/grafana:6.2.4     "/run.sh"                12 seconds ago      Up 7 seconds        0.0.0.0:3000->3000/tcp   monitor_grafana_1
f8cef82f727e        prom/prometheus:v2.10.0   "/bin/prometheus --c…"   12 seconds ago      Up 6 seconds        0.0.0.0:9090->9090/tcp   monitor_prometheus_1

Interesting URLs

wavegen

 curl http://localhost:4444/metrics | grep my | head
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 94392  100 94392    0     0   970k      0 --:--:-- --:--:-- --:--:--  970k
# TYPE mycounter counter
# HELP mycounter count the times each method has been invoked
mycounter{method="flatline"} 11295
mycounter{method="flipflop"} 11295
mycounter{method="incrementer"} 11295
mycounter{method="sinewave"} 11295
# TYPE myguage gauge
# HELP myguage current value of a method
myguage{method="flipflop"} 1
myguage{method="incrementer"} 11295

Prometheus

Cool, we see the same metrics that we were printing out in the previous post. Now let’s verify that the prometheus server can scrape this data from wavegen:

Browse to: http://localhost:9090/targets

You should see both prometheus and wavegen up with a last scrape in the 5-10 second range like here:

Grafana

Browse to: http://localhost:3000/login

• Enter: admin/admin
• Skip changing the password

• From here, select HOME

• Select BEAM

• Select PlayingWithWaveGenMetrics

• We have arrived at the PlayingWithWaveGenMetrics dashboard.

How Prometheus Works

Scrape Interval

The first thing to understand is the scrape_interval. This is the frequency at which Prometheus will query the targets for their metrics. scrape_interval is configured in the prometheus.yml like this:

prometheus.yml

global:
    scrape_interval: XXs     # X seconds

Remember that metrics emitters do not normally include timestamps in their data. So when Prometheus queries the wavegen emitter, this is what that looks like:

mycounter{method="countup"} 25522
mycounter{method="flatline"} 25522
mycounter{method="flipflop"} 25522
mycounter{method="incrementer"} 25522
mycounter{method="sinewave"} 25522
myguage{method="flipflop"} -1
...

Note that there are no timestamps in this data. This means that Prometheus must be timestamping this data at scrape time. Why is this important you ask? Well, because if you are looking at something like changes() below:

https://prometheus.io/docs/prometheus/latest/querying/functions/#changes

changes()

For each input time series, changes(v range-vector) returns the number of times
its value has changed within the provided time range as an instant vector.  

So the question is, does this mean every time a counter is incremented we get some changes() value incremented? No, for several reasons. First, the counters like mycounter above are always increasing. But this counter could increase from say, 1 to 100 in one step, or in 99 steps. We have no way of knowing b/c the emitter is not keeping track of mutations, it’s keeping track of a current value. So when the Prometheus scraper queries wavegen:4444/metrics and grabs the metrics, it’s grabbing a set of current values, timestamping them, and stuffing them into a time series database.

Let’s get back to a real world example. If we have a counter that is monotonically increasing by 1 every second, what will changes() show? It completely depends on the scrape_interval:

We’ll use the following promql:

changes(myguage{method=~"incrementer"}[XXs])

Summary

The point of this entire post

Here we have a graph of the incrementer() function where the scraper_interval was set to 05s:

Here we have a graph of the incrementer() function where the scraper_interval was set to 15s:

                            changes
scrape_interval         10s               20s          30s             60s
--------------------|--------------|-------------|--------------|-------------|
15s                     0               0               1               3
05s                     1               3               5              11

The results show that it doesn’t matter how many times a value changes in between scrapes, what changes() is counting is the number of scrapes that have changed values. This means that if you have 1 scrape and the value changed, then changes() will show 1. If there are 100 scrapes and the value has changed 50 times, then changes() will show 50. This is such a simple concept…but the docs made this clear as mud for me and it took a morning of experimentation to determine how this worked.

The next post discusses Erlang and how to actually wire metrics infrastucture into Erlang applications: Erlang : Where is the Golden Path.

Dependencies

You need to have the following installed to make use of the demo:

• git
• docker
• docker-compose
• make

Getting Started

Clone the monitor repo.

mkdir temp
cd temp
git clone https://github.com/toddg/emitter

Start the docker components.

cd emitter
make start

Verify everything started OK.

docker ps

You should see the following components running:

CONTAINER ID        IMAGE                     COMMAND                  CREATED             STATUS              PORTS                    NAMES
02dc8ccb5218        toddg/erlang-wavegen      "/bin/bash -l -c /bu…"   18 hours ago        Up 18 hours         0.0.0.0:4444->4444/tcp   monitor_wavegen_1
d8bb17caa1d0        grafana/grafana:6.2.4     "/run.sh"                18 hours ago        Up 18 hours         0.0.0.0:3000->3000/tcp   monitor_grafana_1
1715763b4326        prom/prometheus:v2.10.0   "/bin/prometheus --c…"   18 hours ago        Up 18 hours         0.0.0.0:9090->9090/tcp   monitor_prometheus_1

Emit Metrics

It’s trivial to write a metrics emitter in Erlang. Here’s why, the receive ... after pattern for responding to messages. The snippet below is a simple timer function that invokes a method F on a data blob D if no message is received after Delay milliseconds:

%%--------------------------------------------------------------------
%% timer calls itself every Delay milliseconds and invokes function F on 
%% Data D
%%--------------------------------------------------------------------
timer(Delay, F, D) ->
    receive cancel  -> 
                void
    after Delay     -> 
              D1 = F(D),
              timer(Delay, F, D1)
    end.

The helper timer function above is invoked by another helper function tick. tick spawns an Erlang process using timer. The timer method in the spawned process waits for an event that never comes, times out, and invokes the function F in theafter stanza. Here’s tick:

%%--------------------------------------------------------------------
%% tick calls itself every N milliseconds and invokes function F on 
%% Data D; returns a cancel function C
%%--------------------------------------------------------------------
tick(N, F, D) ->
    Pid = spawn(fun() -> timer(N, F, D) end),
    fun() -> Pid ! cancel end.

Now that we can invoke methods via a timer, let’s wire in some methods to emit Prometheus metrics. First, we add a helper function to emit data from the various emitter functions. To start with, let’s verify that the timer works, by printing to STDOUT:

%%--------------------------------------------------------------------
%% emit : side-effect for the function F
%%--------------------------------------------------------------------
emit(Name, Value) -> 
    io:format("emit: [~p]~p~n", [Name, Value]).

Let’s wire the emit function into the following functions that actually do something:

• flatline
• incrementer
• flipflop
• sinewave

Here’s what these methods look like:

%%--------------------------------------------------------------------
%% flat line : emits the value 1 for a flat line shape
%%--------------------------------------------------------------------
flatline(_) -> emit("flatline", 1), ok.

%%--------------------------------------------------------------------
%% incrementer : always increments value
%%--------------------------------------------------------------------
incrementer(V) -> 
    V1 = V + 1,
    emit("incrementer", V1),
    V1.

%%--------------------------------------------------------------------
%% flip flop : emits the value 1 and then the value 0
%%--------------------------------------------------------------------
flipflop(0) -> emit("flipflop", 1), 1;
flipflop(1) -> emit("flipflop", 0), 0;
flipflop(_) -> emit("flipflop", 0), 0.

%%--------------------------------------------------------------------
%% sine wave
%%--------------------------------------------------------------------
sinewave(V) ->
    sinewave(V, 1).
sinewave(V, Delta) ->
    emit("sinewave", math:sin(V)),
    V1 = V + Delta,
    V1.

The last step is start these emitter methods in main:

%%====================================================================
%% API functions
%%====================================================================

%% escript Entry point
main(Args) ->
    io:format("Args: ~p~n", [Args]),
    CancelFuns = [tick(10, F, 0) || F <- [fun flatline/1, fun flipflop/1, fun sinewave/1, fun incrementer/1]],
    %% this never get's called
    receive
        {ok} -> ok
    end,
    %% cancel all the timers
    [F() || F <- CancelFuns],
    erlang:halt(0).

One of the beauties of Erlang is list comprehensions. Here, we map the tick function across the list of emitter funcitons, thereby spawning an Erlang process for each of them:

    CancelFuns = [tick(10, F, 0) || F <- [fun flatline/1, fun flipflop/1, fun sinewave/1, fun incrementer/1]],

Let’s run this and see what the output looks like:

make start
cd monitor && docker-compose logs -f --tail="all" | grep wavegen

The first line invokes the start task in the Makefile which invokes the docker-compose up. The second line tails the output of the wavegen container:

wavegen_1     | emit: ["sinewave"]-0.7391806966492228
wavegen_1     | emit: ["incrementer"]63
wavegen_1     | emit: ["flatline"]1
wavegen_1     | emit: ["flipflop"]0
wavegen_1     | emit: ["sinewave"]0.16735570030280691
wavegen_1     | emit: ["incrementer"]64
wavegen_1     | emit: ["flatline"]1
wavegen_1     | emit: ["flipflop"]1
wavegen_1     | emit: ["sinewave"]0.9200260381967906
wavegen_1     | emit: ["incrementer"]65
wavegen_1     | emit: ["flatline"]1
wavegen_1     | emit: ["flipflop"]0

The key takeaways are:

• flatline always emits : 1
• flipflop alternates between 0 and 1
• incrementer is monotonically increasing by 1
• sinewave is taking the sin() of a number that is increasing by 1 every step

Now that we have verified that he emit function is getting called repeatedly, let’s wire in the prometheus metrics by replacing the print statement with a function to emit metrics:

%%--------------------------------------------------------------------
%% emit : side-effect for the function F
%%--------------------------------------------------------------------
emit(Name, Value) -> 
    multimetric(Name, Value).


%%--------------------------------------------------------------------
%% multimetric : emit all the metrics since we're just trying to
%% sort out how all this works. this way we can see stuff in the
%% grafana dashboard
%%--------------------------------------------------------------------
multimetric(Label, Value) ->
    %% counter for times the method has been invoked
    prometheus_counter:inc(mycounter, [Label], 1),
    % set a gauge to the current value
    prometheus_gauge:set(myguage, [Label], Value),
    % set the summary to the current value
    prometheus_summary:observe(mysummary, [Label], Value),
    % set the histogram to the current value
    prometheus_histogram:observe(myhist, [Label], Value).

Of course the metrics have to be registered, and the proper services started, etc. See the source for further details.

Grafana

Now that we are emitting known metrics, let’s see what they look like in Grafana, coming up in the next post: Monitoring : Part 4.

Overview

There’s a ton of information out there, but here’s what you need to know:

From “Linux Kernel Procfs Guide” by Erik (J.A.K.) Mouw… https://kernelnewbies.org/Documents/Kernel-Docbooks?action=AttachFile&do=get&target=procfs-guide_2.6.29.pdf

The /proc file system (procfs) is a special file system in the linux kernel.
It’s a virtual file system: it is not associated with a block device but exists
only in memory. The files in the procfs are there to allow userland programs
access to certain information from the kernel (like process information in
/proc/[0-9]+/ ), but also for debug purposes (like /proc/ksyms ).  

...  

Instead of reading (or writing) information directly from kernel memory, procfs
works with call back functions for files: functions that are called when a
specific file is being read or written.  

More links:

• Overview
• Deep dive
• Commentary

So basically, procfs and sysfs allow us access to kernel level details via a file-system-like API. Everything is a file.

Documentation

There is this magical kindom called Kernel.org, and in that kingdom, they have documentation…

https://www.kernel.org/doc/Documentation/filesystems/proc.txt

------------------------------------------------------------------------------
                       T H E  /proc   F I L E S Y S T E M
------------------------------------------------------------------------------

Kernel.org is a documentation dream. Section 1.8 deals with what we are interested in, /proc/stat:

  1.8	Miscellaneous kernel statistics in /proc/stat

Paging down to that section, we see the nuts and bolts of how this works:

1.8 Miscellaneous kernel statistics in /proc/stat
-------------------------------------------------

Various pieces   of  information about  kernel activity  are  available in the
/proc/stat file.  All  of  the numbers reported  in  this file are  aggregates
since the system first booted.  For a quick look, simply cat the file:

Ok, let’s try it…

<pre>$ cat /proc/stat
cpu  1274240 19928 407422 72766108 72703 0 12494 0 0 0
cpu0 188950 2084 56891 10272145 10833 0 6916 0 0 0
cpu1 125875 1620 39109 8952369 19718 0 1321 0 0 0
cpu2 193832 2997 58106 8873354 10000 0 569 0 0 0
cpu3 124766 1828 43418 8964723 5675 0 155 0 0 0
cpu4 194548 2834 58202 8873039 9908 0 130 0 0 0
cpu5 126274 1812 44630 8970546 6717 0 333 0 0 0
cpu6 195927 4164 67449 8882986 5034 0 392 0 0 0
cpu7 124064 2586 39614 8976943 4814 0 2676 0 0 0
intr 77943879 6459077 6315 0 0 0 0 0 0 1 25484 0 0 829151 0 0 0 747 0 16 430 0 0 0 43772 6105365 2416471 6126426 3089783 6233849 833826 0 0 0 0 0 0 0 19 1623130 640778 174 385550 841313 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
ctxt 220842322
btime 1561498948
processes 58050
procs_running 2
procs_blocked 0
softirq 28105901 2528 9356000 121075 1863721 835876 56 249179 9047058 0 6630408

So it says I have 8 processors on my current laptop, and I do. Check.

Summary

I need to spend more time here.

• Prometheus is implemented in Go, so the code is easy to follow
• The source is commented, not well, but enough
• Prometheus includes both a host agent and a collector/aggregator

Let’s take a look at the Prometheus agent.

Agent

Prometheus has a metrics collection agent, written in ‘Go’, called node_exporter.

The code in node_exporter can be thought of as broken into 3 separate chunks: startup, metrics collection, and metrics serving.

(The code listings below have all but the nuts and bolts elided. Look at the original source via the provided links for full context.)

Startup

We enter into all Go code via main:

https://github.com/prometheus/node_exporter/blob/e972e38b423632a4f45326bd33eb127a4935f1f6/node_exporter.go#L131-L131

func main() {
	http.Handle(*metricsPath, newHandler(!*disableExporterMetrics, *maxRequests))
	if err := http.ListenAndServe(*listenAddress, nil); err != nil { log.Fatal(err) }

So main is constructing an httpHandler via newHandler and then handing that off to the http server. Does this mean that this collector is only collecting metrics upon receiving a web request to serve up those metrics? Hmmmm….

newHandler is essentially creating an innerHandler. Let’s ignore the code path where we pass in a custom set of collection filters in the query string to the server and stick with the default behaviour. In this case, newHandler creates an innerHandler here:

https://github.com/prometheus/node_exporter/blob/e972e38b423632a4f45326bd33eb127a4935f1f6/node_exporter.go#L42

func newHandler(includeExporterMetrics bool, maxRequests int) *handler {
	h := &handler{... }
	if innerHandler, err := h.innerHandler(); err != nil {...}
	return h
}

innerHandler is where all the meat is. This is the function that actually set’s up the metrics collectors:

https://github.com/prometheus/node_exporter/blob/e972e38b423632a4f45326bd33eb127a4935f1f6/node_exporter.go#L83-L83

// innerHandler is used to create buth the one unfiltered http.Handler to be
// wrapped by the outer handler and also the filtered handlers created on the
// fly. The former is accomplished by calling innerHandler without any arguments
// (in which case it will log all the collectors enabled via command-line
// flags).
func (h *handler) innerHandler(filters ...string) (http.Handler, error) {
	nc := collector.NewNodeCollector(filters...)
	r := prometheus.NewRegistry()
	r.MustRegister(version.NewCollector("node_exporter"))
	r.Register(nc)
	handler := promhttp.HandlerFor(
		prometheus.Gatherers{h.exporterMetricsRegistry, r},
		promhttp.HandlerOpts{
			ErrorLog:            log.NewErrorLogger(),
			ErrorHandling:       promhttp.ContinueOnError,
			MaxRequestsInFlight: h.maxRequests,
		},
	)
	return handler, nil
}

The key takeaway above is that the innerHandler() method is constructing a list of collectors, registering the collectors into a new registry, and then passing the registry into a constructor for prometheus.Gatherers, and returning a handler that wraps all of this goodness. Now we know how the application is started and how the handler is constructed. The next question is what happens when the handler is invoked? So on to the next section…

Collection

What is the prometheus.Gatherers all about and what is inside the handler returned by promhttp.HondlerFor()?

https://github.com/prometheus/node_exporter/blob/e972e38b423632a4f45326bd33eb127a4935f1f6/vendor/github.com/prometheus/client_golang/prometheus/promhttp/http.go#L80-L80

// HandlerFor returns an uninstrumented http.Handler for the provided Gatherer. 
func HandlerFor(reg prometheus.Gatherer, opts HandlerOpts) http.Handler {
	h := http.HandlerFunc(func(rsp http.ResponseWriter, req *http.Request) {
		mfs := reg.Gather()
		w := io.Writer(rsp)
		enc := expfmt.NewEncoder(w, contentType)
		for _, mf := range mfs {
			if err := enc.Encode(mf); err != nil {
}

So HandlerFor returns a handlerFunc that invokes Gather() and then writes the encoded data onto the response. Pretty straight forward.

What does Gather() do? Well the Gatherer interface implements the Gather() method. And this is ultimately the method that get’s invoked by the registry here:

https://github.com/prometheus/node_exporter/blob/e972e38b423632a4f45326bd33eb127a4935f1f6/vendor/github.com/prometheus/client_golang/prometheus/registry.go#L405-L405

// Gather implements Gatherer.
func (r *Registry) Gather() ([]*dto.MetricFamily, error) {
	r.mtx.RLock()
	checkedCollectors := make(chan Collector, len(r.collectorsByID))
	for _, collector := range r.collectorsByID {
		checkedCollectors <- collector
	}
	r.mtx.RUnlock()

	collectWorker := func() {
            collector := <-checkedCollectors:
            collector.Collect(checkedMetricChan)
	}

	// Start the first worker now to make sure at least one is running.
	go collectWorker()
	return internal.NormalizeMetricFamilies(metricFamiliesByName), errs.MaybeUnwrap()
}

The Gatherer.Gather() function iterates over all the available collectors and pushes them onto a channel. Then it fires up go routine(s) to invoke the Collector.Collect() method on these collectors. When complete, the function returns a bunch of metrics data.

What is the Collector interface all about you ask? Here it is:

https://github.com/prometheus/node_exporter/blob/e972e38b423632a4f45326bd33eb127a4935f1f6/vendor/github.com/prometheus/client_golang/prometheus/collector.go#L16-L16

// Collector is the interface implemented by anything that can be used by
// Prometheus to collect metrics. 
type Collector interface {
	// Collect is called by the Prometheus registry when collecting
	// metrics. The implementation sends each collected metric via the
	// provided channel and returns once the last metric has been sent. 
	Collect(chan<- Metric)
}

So that’s the code at a high level. Now let’s deep dive into one particular collector, the CPU collector, to see how these metrics are being pulled from the operating system.

CPU Metrics

The source code related to CPU metrics is organized under the collector directory, with files named as {METRIC_NAME}_{OPERATING_SYSTEM}.go.

Here’s a listing of the cpu metrics collector go source files:

~/repos/external/prom $ tree node_exporter/collector/ -L 1 | grep cpu_
├── cpu_common.go
├── cpu_darwin.go
├── cpu_dragonfly.go
├── cpu_dragonfly_test.go
├── cpu_freebsd.go
├── cpu_linux.go
├── cpu_openbsd.go
├── cpu_solaris.go

As you can see, there is a different source code file for each OS. Since I’m testing on Linux platforms, let’s look at cpu_linux.go.

https://github.com/prometheus/node_exporter/blob/7e684f16aecf8121e41b39fa45a3ab969e9b1cd1/collector/cpu_linux.go#L156

// updateStat reads /proc/stat through procfs and exports cpu related metrics.
func (c *cpuCollector) updateStat(ch chan<- prometheus.Metric) error {
	stats, err := c.fs.Stat()
	if err != nil {
		return err
}

• Q: What’s this c.fs object?

https://github.com/prometheus/node_exporter/blob/e972e38b423632a4f45326bd33eb127a4935f1f6/vendor/github.com/prometheus/procfs/fs.go#L20

// FS represents the pseudo-filesystem sys, which provides an interface to
// kernel data structures.
type FS struct {
	proc fs.FS
}

• Q: I’ve heard of sys and procfs… what are these again?

I have a whole writeup on Linux’s Procfs.

Prometheus’ Procfs Library

• Q: So what’s c.fs.Stat() doing?
• A: It’s calling into a separate prometheus library called procfs.

Here’s what the prometheus procfs library is all about:

https://github.com/prometheus/procfs#usage

The procfs library is organized by packages based on whether the gathered data
is coming from /proc, /sys, or both. Each package contains an FS type which
represents the path to either /proc, /sys, or both. For example, current cpu
statistics are gathered from /proc/stat and are available via the root procfs
package.

Back to the call to stats, err := c.fs.Stat():

https://github.com/prometheus/procfs/blob/master/stat.go#L164

// Stat returns information about current cpu/process statistics.
// See https://www.kernel.org/doc/Documentation/filesystems/proc.txt
func (fs FS) Stat() (Stat, error) {

	f, err := os.Open(fs.proc.Path("stat"))
	if err != nil {
		return Stat{}, err
	}
	defer f.Close()

stat := Stat{}

Cool! Not only can we see that it’s opening a standard file to read from, it’s also well documented code! Let’s take a look at the referenced docs at kernel.org.

Procfs Library Usage

The rest of the code at procfs/stat.go is basically reading the from the file, parsing the data into lines, then splitting the lines by whitespace, and returning a struct with UINTs parsed from the strings. It’s not that interesting… frankly the only thing that is interesting about it is how straight forward it is:

https://github.com/prometheus/procfs/blob/39e1aff1547e9a628f5714cc5f73d058ba44258e/stat.go#L176

        // ----------------------------------------------
        // [ME] read from file
        // ----------------------------------------------
	scanner := bufio.NewScanner(f)
	for scanner.Scan() {
                // ----------------------------------------------
                // [ME] break file into lines
                // ----------------------------------------------
		line := scanner.Text()
                // ----------------------------------------------
                // [ME] break lines into parts based on white space separation
                // ----------------------------------------------
		parts := strings.Fields(scanner.Text())
		// require at least <key> <value>
		if len(parts) < 2 {
			continue
		}
		switch {
		case parts[0] == "btime":
                        // ----------------------------------------------
                        // [ME] parse string into UINT
                        // ----------------------------------------------
			if stat.BootTime, err = strconv.ParseUint(parts[1], 10, 64); err != nil {
				return Stat{}, fmt.Errorf("couldn't parse %s (btime): %s", parts[1], err)

                ...

This stat object is then returned to the caller and the results written to a go channel here:

https://github.com/prometheus/node_exporter/blob/7e684f16aecf8121e41b39fa45a3ab969e9b1cd1/collector/cpu_linux.go#L163

	for cpuID, cpuStat := range stats.CPU {
		cpuNum := fmt.Sprintf("%d", cpuID)
                ch <- prometheus.MustNewConstMetric(c.cpu, prometheus.CounterValue, cpuStat.User, cpuNum, "user")
                ...

This bit is a little confusing. Why is this code calling prometheus.MustNewConstMetric() and not one of the standard emitters such as a Counter or Guage? We know from the kernel.org docs that the CPU values are cumulative totals since the machine was last booted. We also know that the values never decrease. Perhaps it’s because these values are pass-through and don’t quite follow the semantics of a Counter or Guage? Frankly, I’m not sure why these are defined as MustNewConstMetric() and not NewConstMetric() or NewGauge…

Perhaps the documentation can help:

https://godoc.org/github.com/prometheus/client_golang/prometheus#hdr-Custom_Collectors_and_constant_Metrics

If you already have metrics available, created outside of the Prometheus
context, you don't need the interface of the various Metric types. You
essentially want to mirror the existing numbers into Prometheus Metrics during
collection. An own implementation of the Collector interface is perfect for
that.

Since cpu_linux is simply mirroring (pass-through, whatever) the metrics that the OS has already collected, then we can use these purpose-built metrics for them. Perhaps they are faster or more memory efficient?

https://github.com/prometheus/node_exporter/blob/e972e38b423632a4f45326bd33eb127a4935f1f6/vendor/github.com/prometheus/client_golang/prometheus/value.go#L79-L79

// NewConstMetric returns an error if the length of
// labelValues is not consistent with the variable labels in Desc or if Desc is
// invalid.

I suspect the reason to use MustNewConstMetric is a simpler code path while still checking for naming consistency of the emitted metrics.

Summary

Prometheus agent reading CPU stats, on Linux, via Procfs. The call chain goes something like this:

node_exporter 
    -> http.Handler 
        -> onRequest 
            -> Gather 
                -> Collect 
                    -> CPU_Collector 
                        -> Procfs 
                            -> read from the linux procfs pseudo file system
                                -> prometheus.MustNewConstMetric(...)

Now that we know the pattern, it’s fairly easy to deconstruct the rest of the collectors, as needed.

Overview

I’ve broken this post out into subsections to make this more digestable:

• Monitoring : Part 1
• Linux Metrics : procfs
• Prometheus Client : golang
• Prometheus Agent : node_exporter

Once you have digested the contents of those two posts, the rest of this will be pretty straight forward.

1. Linux servers up kernel level information such as host stats via sys and procfs
2. The Prometheus agent slurps up those metrics and parses them by parsing the text in the pseudo file /proc/stats
3. The Prometheus agent then serves those metrics via a url as : http://localhost:[PORT]/metrics
4. The Prometheus collector/aggregator makes an http GET request to that url to retrieve the metrics

Example

The serving part was essentially covered in 1 and 2 where we showed the metrics getting written to the response. In order to see this in action, fire up the node_exporter and curl the listed url to get a metrics report.

First, build the node_exporter…

$ cd node_exporter
$ make
...

Now, start the node_exporter…

 $ ./node_exporter
INFO[0000] Starting node_exporter (version=0.18.1, branch=master, revision=e972e38b423632a4f45326bd33eb127a4935f1f6)  source="node_exporter.go:156"
INFO[0000] Build context (go=go1.12.6, user=toddg@dino, date=20190628-15:55:33)  source="node_exporter.go:157"
INFO[0000] Enabled collectors:                           source="node_exporter.go:97"
INFO[0000]  - arp                                        source="node_exporter.go:104"
INFO[0000]  - bcache                                     source="node_exporter.go:104"
INFO[0000]  - bonding                                    source="node_exporter.go:104"
INFO[0000]  - conntrack                                  source="node_exporter.go:104"
INFO[0000]  - cpu                                        source="node_exporter.go:104"
INFO[0000]  - cpufreq                                    source="node_exporter.go:104"
INFO[0000]  - diskstats                                  source="node_exporter.go:104"
INFO[0000]  - edac                                       source="node_exporter.go:104"
INFO[0000]  - entropy                                    source="node_exporter.go:104"
INFO[0000]  - filefd                                     source="node_exporter.go:104"
INFO[0000]  - filesystem                                 source="node_exporter.go:104"
INFO[0000]  - hwmon                                      source="node_exporter.go:104"
INFO[0000]  - infiniband                                 source="node_exporter.go:104"
INFO[0000]  - ipvs                                       source="node_exporter.go:104"
INFO[0000]  - loadavg                                    source="node_exporter.go:104"
INFO[0000]  - mdadm                                      source="node_exporter.go:104"
INFO[0000]  - meminfo                                    source="node_exporter.go:104"
INFO[0000]  - netclass                                   source="node_exporter.go:104"
INFO[0000]  - netdev                                     source="node_exporter.go:104"
INFO[0000]  - netstat                                    source="node_exporter.go:104"
INFO[0000]  - nfs                                        source="node_exporter.go:104"
INFO[0000]  - nfsd                                       source="node_exporter.go:104"
INFO[0000]  - pressure                                   source="node_exporter.go:104"
INFO[0000]  - sockstat                                   source="node_exporter.go:104"
INFO[0000]  - stat                                       source="node_exporter.go:104"
INFO[0000]  - textfile                                   source="node_exporter.go:104"
INFO[0000]  - time                                       source="node_exporter.go:104"
INFO[0000]  - timex                                      source="node_exporter.go:104"
INFO[0000]  - uname                                      source="node_exporter.go:104"
INFO[0000]  - vmstat                                     source="node_exporter.go:104"
INFO[0000]  - xfs                                        source="node_exporter.go:104"
INFO[0000]  - zfs                                        source="node_exporter.go:104"
INFO[0000] Listening on :9100                            source="node_exporter.go:170"

Try curling localhost:9100…

$ curl localhost:9100
<html>
			<head><title>Node Exporter</title></head>
			<body>
			<h1>Node Exporter</h1>
			<p><a href="/metrics">Metrics</a></p>
			</body>
			</html>~/repos/external/prom/node_exporter

That’s exactly what we saw in the code. Perfect. Append metrics to the url and curl again…

$ curl localhost:9100/metrics
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
go_gc_duration_seconds{quantile="0.5"} 0
go_gc_duration_seconds{quantile="0.75"} 0
go_gc_duration_seconds{quantile="1"} 0
go_gc_duration_seconds_sum 0
go_gc_duration_seconds_count 0
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 8
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.12.6"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 906112
...

There’s a lot of output generated here, so see this gist for a full sample output.

Verification

Does all this work as expected? Let’s take a top level view, knowing what we know now:

$ curl localhost:9100/metrics | grep cpu | grep idle; cat /proc/stat

node_cpu_seconds_total{cpu="0",mode="idle"} 173053.7
node_cpu_seconds_total{cpu="1",mode="idle"} 60628.78
node_cpu_seconds_total{cpu="2",mode="idle"} 60319.59
node_cpu_seconds_total{cpu="3",mode="idle"} 60668.45
node_cpu_seconds_total{cpu="4",mode="idle"} 60367.33
node_cpu_seconds_total{cpu="5",mode="idle"} 60750.17
node_cpu_seconds_total{cpu="6",mode="idle"} 60363.55
node_cpu_seconds_total{cpu="7",mode="idle"} 60724.35

// from the docs, 'idle' is the 4th column reported by /proc/stat

cpu  1769022 21772 573346 59687683 52777 0 16659 0 0 0
cpu0 261911 2143 80436 17305381 16340 0 9367 0 0 0
cpu1 175662 1669 54010 6062890 9968 0 1779 0 0 0
cpu2 269207 3138 82611 6031970 7467 0 778 0 0 0
cpu3 172678 2143 60796 6066852 4373 0 208 0 0 0
cpu4 269848 2975 82810 6036745 3139 0 181 0 0 0
cpu5 175312 2104 60435 6075029 2152 0 386 0 0 0
cpu6 270884 4214 96732 6036367 5641 0 672 0 0 0
cpu7 173518 3383 55513 6072446 3693 0 3286 0 0 0

Let’s compare the results… The docs also say that “““Time units are in USER_HZ (typically hundredths of a second)””” so normalizing by X/100 gives us:

idle            idle          idle 
localhost:      cat           cat
/metrics        /proc/stat    /proc/stat
(seconds)       (USER_HZ)     (normalized to seconds) X/100
-----------------------------------------------
173053.7      17305381        173053.8
 60628.78      6062890         60628.90
 60319.59      6031970         60319.70
 60668.45      6066852         60668.52
 60367.33      6036745         60367.45
 60750.17      6075029         60750.29
 60363.55      6036367         60363.67
 60724.35      6072446         60724.46

So things look good down to the second. There seems to be a difference in rounding at the subsecond level.

Summary

The Prometheus node_exporter is doing the exact same thing I was doing by hand… cat’ing out a pseduo file and scraping the results. Only node_exporter also exposes the metrics via a URL.

Background

• Intro to monitoring
• System Comparison

Monitor

So, the first thing I tackled is creating a docker-compose based monitoring and reporting stack:

monitor

‘Monitor’ can be plugged into any build system (Makefile, etc.) and turns on monitoring on my dev box as part of a test run. It collects metrics from both collectd and from prometheus’ node-exporter. ‘Monitor’ includes some basic grafana dashboards so that all you have to do is fire up a browser and start looking at the collected metrics in a standard and repeatable way. However, here be (tiny) dragons.

Questions

Plugging in a monitoring stack immediately raised a bunch of questions:

• Why are there so many different collectors and stores out there? How are they different?
• What are some basic dashboards that would be useful to analyze applications written in Java, Python, Go, and Erlang?
• How can I make application monitoring just a standard part of my workflow, with a low friction much like running unit tests?

As I started using grafana and looking at the metrics being surfaced via collectd and prometheus collectors, I couldn’t help but notice that the cpu and memory graphs looked different, and the units didn’t make sense, either. Basically, what I was running into was my full lack of understanding of how metrics were being collected, what they mean, and how to surface them in a meaningful way.

It’s time to deep dive into some code and docs and try to sort this out.

All that and more coming up next in Monitoring : Part 2.

rpc:pmap()

Questions

• Does Erlang follow Amdahl’s Law?
• What are the constant time factors for Erlang?
• How easy is it to use?

Analysis

Jupyter Notebook

Summary

Parallelization of an embarrassingly parallel problem is certainly easy to do in Erlang, though the benefits of spreading a calculation across processors diminishes after the 2nd or 3rd processor.

• Does Erlang follow Amdahl’s Law? (Yes)
• What are the constant time factors for Erlang? (I didn’t measure them but they are high)
• How easy is it to use? (Yes, very easy to use)

Links

• mailing list : skel
• skel
• rpc docs
• Amdahl’s Law

What sort would you use if you required tight 
max time bounds and wanted highly regular 
performance?

Discussion

There were some answers but nothing beats implementing and measuring for yourself.

Inputs

• Constant value : [1,1,1,…]
• Sorted : [1,2,3,…]
• Reverse sorted: [N, N-1, N-2, …]
• Random: […]
• Interleaved : [Constant, Sorted, Reverse-sorted, Random, …]

Algorithms

• Insertion Sort
• Quicksort
• Heap Sort
• Bucket Sort

Input Data

Data

Results

• ‘heapsort’ had the most consistent performance
• ‘quicksort’ had the fastest performance
• ‘quicksort’ had the worst performance

Determining which sorting algorithm is ‘best’ for a given use case actually depends on your requirements. If you need the fastest sorter possible, then that’s probably quicksort. But if you need good overall performance for a wide variety of input types, you’re probably going to want heapsort. The full analysis is captured here as a Jupyter Notebook (with lots of pretty graphs):

Notebook