Data retention with the Serverless Framework, DynamoDB, and Go

At Honeybadger we have standard retention periods for data from which our customers can choose. Based on which subscription plan they choose, we’ll store their error data up to 180 days. Some customers, though, need to have a custom retention period. Due to compliance or other reasons, they may want to have enforce a data retention period of 30 days even though they subscribe to a plan that offers a longer retention period. We allow our customers to configure this custom retention period on a per-project basis, and we then delete each error notification based on the schedule that they have set. Since we store customer error data on S3, we need to keep track of every S3 object we create and when it should be expired so that we can delete it at the right time. This blog post describes how we use S3, DynamoDB, Lambda, and the Serverless Framework to accomplish this task.

More …

Writing Again

A while back I changed the stack I used for publishing this blog with the hope that I would write more because it would be easier to publish. Looking back at how little I’ve written since I’ve made that change, I can see that that didn’t work out so well. :) I’m not going to change my stack again (just yet), but I am going to try and write a bit more. Hopefully it won’t be another year before my next post.

More …

Solr Recovery

At Honeybadger this morning we had a failure of our SolrCloud cluster (of three servers). Each of the three servers has a replica of the eight shards of our notices collection. Theoretically this means that two of the three servers can go away and we can still serve search traffic. Sadly, reality doesn’t always match the theory.

More …