Damn! It Worked Last Week!
It takes ages for the newsfeed to load. Scrolling on the tablet is slower than usual. Applications aren’t starting as quickly as they were a week ago.
You’re getting frustrated. You recall the last change you pushed to production. No, it couldn’t be that simple change. Are you sure?
We are ready to see the reality of software development. Enter the world of benchmarking and code archaeology.
What is benchmarking and why you should care
What is benchmarking? A quick search unveils a few definitions. They focus on business performance, product sales, corporate growth, and universities’ and students’ learning goals. Hmm, there is nothing about programming, hacking, or testing. Not even about technology.
The next intuitive place to search is Wikipedia. What do we get? A business definition again.
Benchmarking is the practice of comparing business processes and performance metrics to industry bests and best practices from other companies. Dimensions typically measured are quality, time and cost.
But we get more ideas when reading the last sentence in the definition.
Dimensions typically measured are quality, time and cost.
Aha! The quality, time and cost. We know already how to verify the quality of functions. Don’t we? We trust our unit tests, developed over months. They cover both positive and negative scenarios. We know the application works as it should. We’re confident we have got a rock-solid piece of code.
What about time and cost? The two words help us define benchmarking in software development and quality context.
The time measures how long a function takes to compute and return results. We can agree it’s intuitive. What about the cost? Let’s agree that the function’s cost is the memory it needs to run and return results.
Now the picture has become clearer.
Two most important questions to ask
We write and run benchmarks to learn:
- How fast does the function compute and return results?
- How much memory does the function need to compute and return results successfully?
Ok, but why do we need this information at all? Why bother? Is it not enought that the function pass unit tests?
How does your untested code impact customers
Let’s change our hats for a moment and wear the end user’s, a customer’s, hat.
It’s time to forget that we know how things work under the hood. In the end, users don’t realise how much duct tape we use to keep the application in a working state.
You’re waking up, going to the kitchen and taking a sip of your favourite beverage. At the same time, you scroll on your mobile, trying to check the news and emails.
It takes ages for the newsfeed to load. Scrolling on the tablet is slower than usual. Applications aren’t starting as fast as they were a week or a month ago.
You go for a short walk to the park nearby and start enjoying the morning sunshine. Initial frustration disappears.
You put the tech hat on again and start analysing the entire situation.
- How come the app started complaining about low memory?
- Was this app even tested? If yes, under what circumstances, in what environments?
- Why did everything work last week?
The investigation journey begins
You’re coming home, opening your laptop, and checking recent updates in the Git repo.
The “Aha!” moment arrives! It turns out that the only change in the app was yours. But the tests were green. You hit ENTER and, after a few moments, the buggy app was in the app store. Ready to give a headache to thousands of users.
You realise that current tests are not enough. You need to find the bottlenecks that slow down the app.
You need to dive deeper into the internals to find and fix performance problems. You know that you need to start writing benchmarks.
Take the red pill and enter the world of Sherlock Holmes and Dr Watson. The world of code investigators and archaeologists!
Soon, we’ll start writing benchmarks and making sense of the reported numbers.
Till next time!
Jakub
To join the benchmarking journey, subscribe to my mailing list for updates on my upcoming book.