“Flash, Flash, hundred-yard dash!” — do you remember this very funny and u-n-h-u-r-r-i-e-d sloth named Blitz from Zootopia? It pops up in the memory whenever we face someone’s slow performance or reaction. In fact, Blitz required only 10 seconds to reply. 10 seconds is not so much. In real life, we often have to wait much longer and waiting does not look so funny at all. Waiting is unpleasant and very expensive.
In this series of articles, I would like to talk about why we have to wait, how to stop waiting and what an interesting world opens up if we live at the same speed as reality. More precisely, let’s talk about the evolution of technological processes underlying the speed of data processing and decision making.
What did Tyler Akidau mean?
In 2015, Tyler Akidau, a senior software engineer at Google, published the manifesto “Streaming 101: The world beyond batch”. In 2018, this publication formed the fundamental pillar of a book on the evolution of data processing systems “Streaming Systems: The What, Where, When, and How of Large-Scale Data Processing”.
It seemed from the title, that in 2015 the world came close to replacing batch processing with the streaming approach. The following definition was given: stream processing is a kind of data processing designed with infinite data in mind.
The data streams need to be processed continuously, even if they are highly intensive. Otherwise, the data will begin accumulating and at some point, will reach a critical mass, at which point the processing ceases to keep up with the arrival of newer and newer data. The processing must be carried out at the speed of data arrival and immediately.
There is no such a problem with batch processing, since the data is always finite. Even if there is a lot of data, processing can be performed for an arbitrary time, can be stopped, started over again, repeated, split into smaller data sets.
The publication focused on the technical aspects of batch vs. streaming, but also provided business prerequisites for moving to streaming:
- Business needs more and more up-to-date data and moving to streaming is a good solution to close the gap.
- The continuous data flows become increasingly important for business. The systems that are specifically designed for infinite streams handle flows better.
- Processing data as it comes allows better load balancing and more predictable resource consumption.
The second and the third arguments are important but more technical. The first argument is really significant from a business perspective. Let’s talk about it in detail.
In the 6 years since the publication of the book, we can say that batch processing has not disappeared anywhere and the world has not been divided into two epochs: before and after. Maybe Tyler Akidau meant something else when stating: the world beyond batch?
Closing the gap
Let’s repeat once again the thesis “Business needs more and more up-to-date data and moving to streaming is a good solution to close the gap.”
Why does business need more and more “up-to-date data”? What is “up-to-date data”? What kind of a “gap” are we looking to “close”?
“Up-to-date data” can mean different things. Events could have happened a second, a minute, an hour, a day, a week, a month or a year ago.
Events that have just happened within the last milliseconds, seconds or minutes are usually referred to as real-time or near real-time. Events that occurred an hour, a day or more ago are referred to as historical data.
When Tyler Akidau mentions “closing the gap”, he assumes moving to millisecond and second time lags from the precise moment described as “right now”.
Perhaps, Tyler Akidau had a different meaning in mind with the title “Streaming: The world beyond batch”. Does business need to switch to a new technological process that is beyond the capabilities of the batch? Does it mean new opportunities are going to open up?
Actually, batch, as a technological process, is used where it is permissible to lag behind reality by an hour or more. It is well suited for working with historical data.
Streaming, as a technological process, is able to lag behind reality of milliseconds and seconds; in other words, almost matching the speed of events.
What closing the gap brings?
Why does a business may have a “need” to close the gap from hours and days to milliseconds and seconds?
Because we live in the real-time world, events and changes take place in the here and now. We do not live in a world that existed an hour, a day, a week, a month, or a year ago.
There are many tasks where a lag of an hour, a day, or a week is acceptable. For example, if a retailer or a bank is planning to open a new store or a branch, then a day’s lag seems reasonable. Of course, this only applies if tomorrow a new lockdown does not happen, or the street does not undergo redevelopment, or something else occurs.
At the same time, there are a growing number of tasks when a lag of an hour is unacceptable. A customer is phoning a call-center — predicting the reason for the call enables the staff to serve the customer faster. A product is leaving an assembly line — predicting quality helps to detect defects as quickly as possible. A courier is carrying a parcel — predicting a precise delivery time is desirable.
The “need” arises exactly at the moment when business realizes that lagging behind reality by an hour is too long and expensive — it is necessary to make predictions as events occur.
Let’s remind ourselves of the results of continuous improvements in the technological processes in electronics from 10 microns in 1971 to the expected 2 nm in 2023. Each step in the improvement of technology has led to an increase in productivity, reduction in power consumption, higher operating frequencies, lower costs, diminishing chip sizes, and market changes.
Likewise, “closing the gap” and moving to streaming, as a new technological process, leads to improved customer service, reduced service times and costs, improved product quality, prevention of design failures and reduced downtime.
What this technological transformation means for each of us
Every 5 minutes of waiting for each person in a country with a population of 10 million people means a national loss of 4 million hours every day. It doesn’t matter where we waste these 5 minutes — in the checkout queue, waiting for a connection with an operator or for our data to be processed.
Closing the gap and making decisions at the same speed as events occur mean we can wait less, work less, spend more time with friends and family, and burn fewer resources.