For the most part of software development, I’ve been using scheduler quite often, For example, when there is a need to expire something, depends on how quickly data needed to be expired–the scheduler wake up every 1 minute to look for what it has to do.

This is not efficient, why?

Because even the scheduler has nothing to do (no task) it still has to wake up every minute to fetch database and figure out that there is nothing to do!

So, what are the solution ?

The best way is to wake up the system only when it needed to. In this case a task scheduler with ability to set “when-to-execute” would do the job.

Aren’t there existing solutions ?

There is! it is called Google Cloud Tasks ! But I’m using AWS, sad

so, after some researches I see that people are having the same problem, they want to do something like Google Cloud Tasks, basically all people want is something like “hey scheduler, run this thing at this time please”

Let’s do it

Redis has a data structure called “SortedSet”, when we added somethings into “SortedSet” Redis will sort them by “Score”, and that’s it!

Will put “when to run a tak” as a score, then Redis will take care of sorting what tasks need to run first–by just quering the “SortedSet” by score

Data structure for Redis' Sorted Set is consisting of “Score” and “Member”, which “Score” is int64 and “Member” is string

We can put Task’s detail into “Member” field

Here is the overview diagram

The only job for Task Processor is to dispatch Http Request to somewhere, instead of having workers doing some specific jobs here, delegate jobs to resource owner would be a good idea.

and the payload to the Api looks like this

{
    "when": "2022-06-05T03:45:00+07:00",
    "http": {
        "method": "GET",
        "url": "http://localhost:8443/purchase/rn=123456789",
        "headers": [{"key":"Content-Type", "value":"application/json"}],
        "payload": "{\"name\":\"Katie\",\"personInfo\":{\"age\": 21}}"
    }
}

How about it, the idea is pretty simple right?

• Leverage Redis SortedSet to be a Sorted Tasks
• Delegate job that needs to be done, to resource owner do it by themself

Here is the repo: https://github.com/tjindapitak/sched

Cheers guys!

Improvements

• There can be race condition when adding a task to Redis and reading a task to Redis, to solve is using Redis Lock
• Support rich http request spec
• Authentication
• Log
• Retry mechanism (base on Job type and Error type)
• Test