Damn! It worked last week!

It takes ages for the newsfeed to load. Scrolling on the tablet is slower than normal. Applications are not starting as smoothly as they did a week ago.

You are getting frustrated. You recall the last change you pushed to production. No, it couldn’t be that simple change! Are you sure?

We are ready to experience the reality. Enter the world of benchmarking and code archaeology!

What is benchmarking and why you should care

What is benchmarking? Your quick search online brings us a list of definitions. They all are about: business performance, product sales, corporate growth, and universities’ and students’ learning goals. Hmm, there is nothing about programming, hacking, or testing. Not even about technology!

The next intuitive place to search for the benchmarking definition is, of course, Wikipedia. What do we get? A business definition again.

Benchmarking is the practice of comparing business processes and performance metrics to industry bests and best practices from other companies. Dimensions typically measured are quality, time and cost.

But there is some clue in the last sentence!

Dimensions typically measured are quality, time and cost.

Aha! The quality, time and cost. We know already how to verify the quality of functions we write. Don’t we? Our extensive set of unit tests gives us confidence. Our tests cover both positive and negative scenarios. We know the functions work as intended. We are confident we have a rock-solid piece of code. Don’t we?

What about time and cost? The two words help us define benchmarking in software development and quality context.

The time measures how long a function takes to compute and return results. We can agree it’s intuitive. What about the cost? Let’s agree that the function’s cost is the memory it needs to run and return results.

Now the picture has become clearer.

Two most important questions to ask

We write and run benchmark tests to learn:

• How fast does the function compute and return results?
• How much memory does the function need to compute and return results successfully?

Ok, but why do we need this information at all? Why bother?

How does your untested code impact customers

Let’s change our hats for a moment and wear the end user’s, a customer’s, hat.

It’s time to forget that we know how things work under the hood. More frightening, we must forget why the “things” work. They work because we assembled them using a “duct tape” solution.

You are waking up, going to the kitchen and taking the first sip of your favourite morning beverage. At the same time, you are holding a smartphone or a tablet and trying to check the news or emails.

It takes ages for the newsfeed to load. Scrolling on the tablet is slower than normal. Applications are not starting as smoothly as they did a week or a month ago.

You go for a short walk to the park nearby. You start enjoying the morning sunshine. Initial frustration disappears.

You put the tech hat on again and start analysing the entire situation. You start asking questions:

• How come the app started complaining about low memory?
• Was this app even tested? If yes, under what circumstances, in what environments?
• Why did everything work last week?

The investigation journey begins

You are coming home. You open your laptop and scroll through your recent changes in the Git repository.

The “Aha!” moment arrives! It turns out that the only change in the app you complained about in the early morning was yours. All tests were green. You pressed Enter, and millions of users were ready to download the app.

You realise that current tests are not enough. You need to find “the issue”.

You know you need to start benchmarking your application. You need to dive deeper into the internals to find and fix real problems that have surfaced!

Take the “red pill” and enter the world of Sherlock Holmes and Dr Watson. The world of code investigators and archaeologists!

In the next part, we will start writing benchmarks and making sense of the reported numbers.

Till next time!

Jakub

To join the benchmarking journey, subscribe to my mailing list for updates on my upcoming book.