Posts

This post talks about Caddy, a HTTP server that’s easy to get up and running, lightweight and has a module for exposing metrics in the native Prometheus format, so we we like it a lot. In this case we’re using Caddy to host a small static gallery, generated from the image post-processing suite Lightroom. Caddy runs in a FreeBSD jail (OS-level virtualisation), which is hosted on a fairly powerful physical machine.

For one reason or another, Scandinavian countries always seemed like excellent places to visit and live in. It was many years before I actually visited any of them, but they are renowned for quality of life & quality of output, professionally and artistically. So naturally I was thoroughly excited when we received an invitation to a wedding held in Stockholm. Don’t get me wrong, two of my good friends getting married was most excellent news, but, as is common these days, we knew about their plans well in advance, so there was no surprise when the official invitation finally reached us in Singapore.

Somewhat unsurprisingly, I’m arriving late, but thankfully still find some breakfast being served, nice :) I have just have enough time before the next talk starts to chat with a person from Zendesk (they’re one of the sponsors and actually paid for all the coffee, thanks!). Zendesk seem to be all over the radar recently - hosting a “wine & dine” aka “we’re hiring” event a few weeks ago and coming up high in a recent review of best places to work at; good for them.

Problem statement Recently I had to send a sizeable amount of logs into our log pipeline, a whole 23781261 log lines to be exact. Our log pipeline is the standard ELK stack, plus Filebeat, a lightweight log shipper from Elastic that forwards logs from the central log server into Logstash. Here’s a diagram to illustrate the entire flow of logs in our system: Nodes -> Syslog-NG -> Central logserver Filebeat -> Logstash -> Elasticsearch Pushing this number of logs with pretty much standard configurations of Filebeat and ELK completely saturated disk IO on the server running Elasticsearch (this node is using spinning disks so not that difficult of a feat to achieve).

This is my first attendance to SREcon (or any other USENIX conference for that matter), so I decide to arrive early to mingle over breakfast. Somewhat surprisingly, there are only a dozen or so people here (the rest are arriving late I subsequently find). Breakfast is pretty good and even the pre-brewed coffee is drinkable, so that’s a good start. I notice that Facebook have a desk with people present already, maybe they’re still jetlagged, most of them are from USA or European offices.