Dispatch and run distributed work asynchronously, brokered by Redis
Source Code: https://github.com/rafalstapinski/redispatcher
What is redispatcher
redispatcher allows you to dispatch work that needs to be done in a process separate from your main program loop. This is useful in cases when you need to process some long running work that doesn't necessarily need to be done synchronously within your code. A classic example of this is sending a welcome email to a user as they sign up for your service. It's not necessary to wait for the results of sending an email, and it may take a few seconds to do so. redispatcher lets you fire-and-forget this work (as a message put into a Redis server) and have it be executed in the background, asynchronously, in a separate process (or machine) entirely.
redispatcher comes in two parts: 1. A library that lets you define workers, define strongly typed messages sent to workers, and provides helper functions to facilitate dispatching that work 2. A daemon that runs your workers in a pool, consistently listening for any incoming messages to be processed
Why use it
There are certainly other solutions for orchestrating distributed workers. redispatcher aims to be super lightweight, very fast and simple to set up (there are many free cloud-hosted Redis solutions available), and has robust type validation and intellisense support.
- Full intellisense support across your code, despite a distributed workload
- Strongly typed message contract between your publishing code and consumer
- Minimal boilerplate required to setup and start publishing compared than most alternatives
- Minimal performance overhead and completely non-blocking, thanks to
asyncio(and works with alternatives like
aioredisis used under the hood to publish message to and read messages from Redis
pydanticis used to to validate messages conform to your strongly typed contracts
Defining your worker
redispatcher is already used in production, but is still in its infancy.
If you find a bug, open an issue with a detailed description and steps to reproduce.