Internal design

The focus of this server was to handle a common way for multiple languages/services to deal with background jobs in a common way, especially for dealing with jobs that may take hours/days to run.

Designing for the very high throughput low-latency use case (e.g. tens of thousands of workers, millions of jobs per day) was not a focus for this project, there are plenty of good systems out there which deal with that very well.

Ocypod makes heavy use of Redis transactions, in order to allow clients/workers to modify jobs as they are in progress, while ensuring the integrity of the system.

The ocypod-server itself is stateless, so there should be no issues with running multiple instances of the server with the same Redis instance as a backend.

Internally, several Redis data structures are used:

The ocypod-server runs three background tasks which monitor different queues and modify job state as necessary: