Post on Dan Kleiman

Roll Your Own Database: Part 1

Sat, 16 Feb 2019 11:31:03 -0500

Warning: This post is NSFW. In this series, we are going to build a really, really simple database management system that you should by no means use in a production work environment. Here’s the experiment: Start with a naive implementation of a database – read and write from a local csv file Reach for all the normal features we use every day: basic CRUD, aggregate queries, complex joins Realize that our basic implementation falls short Look at how these problems are solved in modern systems Think of the whole series as one giant experiment in Cunningham’s Law: “the best way to get the right answer on the internet is not to ask a question; it’s to post the wrong answer.

Seed Data for Your WIP SQL Queries

Tue, 05 Feb 2019 20:41:59 -0500

In my last post, I wrote about steps you can take to make writing complicated queries more manageable. One aspect that I didn’t cover in that post is how to set sample data to work with during the writing process. Assuming you’re not working directly in your production database as you test out new queries (right? right?? right???), you need some way to work on your new ideas. In this post, I want to share some tips and tricks for creating reliable, reproduceable test data to help you develop new ideas in SQL.

Stop Writing SQL Backwards

Wed, 02 Jan 2019 09:21:26 -0500

How many times have you started off building a complicated analytical SQL query like this? SELECT . . . . uh??? . . SELECT * FROM . . . SELECT . . . And you get stuck trying to figure out exactly what you want to select. You’re thinking about averages, group by’s, the order of your results or some change you want to see over time and the query editor is just sitting there, taunting you, because in SQL, you have to know up front what you want to select into your final results.

3 Ways to Level Up Your SQL as a Software Engineer

Tue, 06 Feb 2018 12:31:44 -0500

If you are a software engineer and you have just enough SQL to write queries that count, sum, average join and maybe sub-select, then I’m writing this post for you. If, when you need more complicated analysis or computation, you pull your query results into excel or your favorite scripting language to do more processing, then I have some good news.

There’s a whole lot more you can do right in SQL and it’s not too bad to learn how to do it.

In this post, I’m going to cover a few concepts that have recently helped me do more computation, better analysis, and it turns out, more efficient querying…and I hope they help you too.

Keeping an Engineering Notebook

Sun, 28 Jan 2018 22:52:26 -0500

The best upgrade I’ve made to my workflow in the past year was to start keeping an engineering notebook.

Whenever I start a new project, the first thing I do is create a new section in my engineering notebook. It’s really simple. With a tiny script, I generate a dedicated folder for the new project, plus three sub-folders and a README:

    some_new_work
        |__ notes/
        |__ data/
        |__ scripts/
        |__ README.md

As simple as this structure seems, it has had a tremendous impact on my work. In this post, I want to try to unpack why and how I think that’s working.

Top N Per Group in BigQuery

Mon, 30 Oct 2017 19:29:32 -0400

EDIT: After I posted this initially, I got some great feedback, so I wrote a follow-up post here.

In this post, we are going to explore a strategy for collecting the Top N results per Group over a mixed dataset, all in a single query.

I stumbled onto this solution the other day, mostly driven by the fear that I was re-scanning my BigQuery data too often. At the time, the only way I knew how to look at a Top 10 list of a subset of the data was to add a WHERE clause limiting the whole data set to a single group and combine with ORDER BY and LIMIT clauses.

For each group, I would just modify the WHERE clause, rescan all the data, and get new results. I thought there had to be an easier way to get the same ordered subset for any particular group in the data, all at once.

It turns out, there is a much more efficient way to solve this problem.

Don't Blow Your BigQuery Budget on Unknown Data!

Fri, 06 Oct 2017 14:55:37 -0400

It’s easy to blow your BigQuery budget when you are exploring a new data set. Because you’re billed for the amount of data scanned, not the ultimate result set, when you don’t know what you’re looking for, you can end up with wasteful queries.

In this post, I’m going to share some tips for more efficiently scanning data in BigQuery when you don’t quite know what you need.

GoBridge with Bill Kennedy

Sun, 12 Feb 2017 15:18:04 -0500

Last weekend, I had the chance to volunteer at a GoBridge event taught by Bill Kennedy of Ardan Labs. I’m trying to make 2017 my year of learning Go, so helping out at the event felt like a natural extension and a great way to connect with more people in the Go community.

Going in with Ruby as my first language, I braced myself for static typing and wanted concurrent programming to bend my brain, but that’s not really what happened at all.

My 5 Strategies for Learning Go in 2017

Thu, 29 Dec 2016 09:29:22 -0500

Over the past couple of years, one thing I’ve become more and more aware of is the unease and uncertainty of diving into a new project. Not matter what the new X is, I find I always go through the same set of uncomfortable feelings on my initial approach.

Now, though, I’m starting to become familiar enough with this process that – even though the discomfort doesn’t go away in the initial learning stages – I can embrace it, coexist with it, and forge ahead in learning, because of the strategies I’m going to lay out here.

Rails Security Exercises from Bearclaw

Sat, 24 Sep 2016 06:05:49 -0400

In this post, I’m going to tell you what I learned doing a series of Rails security exercises developed by Bearclaw, a Rails security consultancy. Before I go into the exercises, though, I want to send a huge thank you to Ali Najaf, founder of Bearclaw. What I’ve learned here is due to the thoughtfulness of the exercises he’s put together and his willingness to try something new by sharing them with me.

Nice Try, NilClass

Sun, 10 Apr 2016 07:13:27 -0400

I love that feeling when a new concept starts to come together in your mind and you can point to all the converging sources of insight. Right now, I can’t tell if I’m fooling myself, hiding some logic, or making my code more readable with this particular concept, but when I put together these three pieces of information, I think I start to see something emerge. I’ve been inspired to do some more digging into these kinds of questions lately thanks to the awesome new Ruby Book Club Podcast.

Migrating Posts and Pages from Wordpress to Jekyll

Fri, 11 Mar 2016 13:55:15 -0500

This is Part 1 in a series on Migrating from Wordpress to Jekyll.

The documentation for getting started with Jekyll is great. I’m not going to rehash everything that’s covered there.

Instead, this post and the others in the series will be more like, “here’s the order I wish I had done things in” or “here’s everything I ended up needing to pull together to get stuff working”. I hope it helps you and saves you time if you ever decide to do a similar migration from a self-hosted Wordpress install to Jekyll.

So here we go….

Migrating from Wordpress to Jekyll

Wed, 09 Mar 2016 13:09:18 -0500

So, I’ve decided to migrate my Tai Chi site at dankleiman.com from Wordpress to a new static site using Jekyll.

Since I haven’t posted there in almost two years, but I get a steady stream of new subscribers who want to learn about Tai Chi, qigong, and meditation, I thought it would be good to give the 300+ pages and posts a more evergreen feel.

I Hate Voicemail: Straight to SMS with Twilio

Tue, 15 Sep 2015 08:02:19 -0400

I hate checking my voicemail. I hate the nagging voicemail icon on my phone that won’t be dismissed unless I actually check my voicemail. I hate how listening to a voicemail, capturing relevant information from it, and calling someone back breaks up the normal flow of how I communicate with people all day long: asynchronously, via written communication like email or text.

These days, when I want to talk to someone, we plan a time to talk and that’s fine. But some people still call and leave me a black-box-of-a-voicemail that I have to wade into.

I needed some way to cut voicemail out of my life…so I hooked up a Twilio app to handle it for me.