Making of design pattern
Design patterns are very abstract. Difficult to see relevance in day-to-day life of a developer. As a developer myself when I started, I felt that those were difficult to see their purpose. But with time and many projects I see all coming together. Background processing is one such design pattern. If you have come across only the Gang of Four as the set of design patterns, please expand your horizons. Patterns are classified as design and architectural I guess design pattern is oversold and architectural are talked about in niche circles only. But as developers we need to be pragmatic and see them as repeated problems that we solve. If we walk with this definition and try to look at the problem which background processing solves it essentially boils down to –
Some tasks require to be performed on each user request that does not require feedback to user immediately.
In .NET framework one would have something like this for background processing let us say decades back would be
|
Thread backgroundWork = new Thread(backgroundProcessing); … private void backgroundProcessing(..) { … }
|
This style of processing background task was rooted on the section of problem statement that read “does not require feedback to user immediately”. True to the ask a new thread spins off and performs in a parallel timeline. The request might have very well finished by the time this thread finishes the background processing.
There were challenges with the way problem was attempted to be solved. It was a thread and thus required the host process to be around for it to finish its work and post the result of its processing. Otherwise, the result of processing was lost in ether. The other challenge was managing the thread as it spun off. By managing we mean matching the request to a particular background process, retrying (if possible) in case of failure, maintaining order if applicable in processing between two threads, and many such real-life side kick challenges that come along the main challenge.
In case you wonder what could really be different, than the sample code that you would have seen across many articles. Let us break the myth, these articles are to narrate the nature of the problem not to expose the problem in entirety to you. Not just in this article but elsewhere and everywhere that is the nature of tech articles.
In real life one of the prominent examples taken for background processing is
Creating a thumbnail of larger images that get uploaded. One caneasily this happening in e-commerce websites.
Another close cousin of this kind of problem is creating profile pictures of individuals from a larger picture. This dominantly involves cropping and sizing both.
Then there are data dependent challenges that needs to be solved in background particularly in applications that have amazing visualizations for data like charts and graphs. These require some summarization that is to be performed on the fly.
There are many more but these are few top ones that remind us of background processing as of now. Let us resume our focus on .NET code, so there was this thread-based processing whichserved the purpose to some extent but carried challenges of its own.
Then there was another approach where event-based model was used delegate the background processing which did lot of thread managed by itself and this one focused on not requiring a feedback in UI immediately.
|
BackgroundWorker worker = new System.ComponentModel.BackgroundWorker(); worker.DoWork += new DoWorkEventHandler(…); worker.RunWorkCompleted += new RunWorkCompletedEventHandler(..); worker.ProgressChanged += new ProgressChangedEventHandler(…);
|
The challenge was not solved yet because both solve some specific portions of the problem statement but when bought into the world of ASP.NET i.e., web-based applications, they both had severe shortcomings. To start with we know the overhead of using threads for background processing. BackgroundWorkerposes similar challenges as of thread but requires additional effort to make it all work on HTTP request and response.
Now, in the modern business of software development many are web applications. Thus, the shortcomings of above will have to be addressed to be actually practiced as solution for the problem statement.
Enter the world of IPC – Inter Process Communication. This has been known for long time right from the UNIX times. To some extent IPC has been attempted as a solution to address background processing problem in entirety. E.g., bespoke solution which uses a Console application and works with persistent storage. However, in the past challenge was the ability to run multiple processes in an environment. It was resource consuming to bootstrap a process for a specific task and put it to rest once completed with repeating this cycle time and again.
The challenge thus was –
Bootstrapping time for the process
Communication between processes as well that was not tolerant.
The eventual solution which adorns a very popular name is now looked upon as a cookie cutter solution which removes the limitations of IPC – Microservices. The only shortcoming of this is more oriented to the complexity and design time required to complete a solution using microservices. Thus, many projects did not exercise microservice or as a solution of background tasks until the task itself was so complex that it required microservicesexplicitly. One of the popular approaches though might not qualify as based on IPC but definitely as out-of-process processing (daemon processing) is a solution like Hangfire. Hangfire is a framework that you could add reference to your project and spin off a background processing task like this –
|
BackgroundJobClient worker = new BackgroundJobClient(); worker.Enqueue(backgroundProcessing);
|
This is similar to the thread-based approach with difference that now one has to take care of infrastructure required for theframework to work. More than framework one might say this is a 3rd party library/tool that performs background processing. Since, it requires infrastructure and has support for few in the open-sourced version. Popular ones like Redis cache, SQL database, Azure SQL are supported but exotic ones like MySQL, Oracle or any other like that might need careful selection effort. One of the best benefits of having out-of-process based solution for background processing is any fault with the background task processing will not pull down the front-end to which users are interacting.
As a developer depending on your where you are at this point on your career graph you might find various design pattern that are helpful not just the Gang of Four design patterns. You can build your own catalogue of problem statements that you have encountered many times at your work and in your pet projects. You will notice as you evolve as developer how these design pattern catalogue of yours evolve and probably converge with GoF or diverge to something more interesting than GoF. Happy programming and discovering!
