Welcome to redis-tasks’ documentation!

redis tasks is a framework for reliable background tasks execution, specifically for environments where workers are expected to restart frequently, like Heroku. The task queueing is handled by redis.

Disclaimer

redis-tasks does not store any kind of task results. If you need to store task results, you have to use a persistence layer of your choice in your task code.

Installation

Your can install redis-tasks using pip like this:

$ pip install redis-tasks

Note

redis-tasks only runs on python versions 3.7 and above.

You have to set the RT_SETTINGS_MODULE environent variable before you can use redis-tasks in your code.

Alternativaly you can call redis_tasks.conf.settings.configure() passing in a python settings module or call redis_tasks.conf.settings.configure_from_dict() passing in a dict of settings.

See redis_tasks.conf.Settings for more details.

Quickstart Guide

Prerequisite: You have successfully installed redis-tasks and have a running redis instance. In this Quickstart Guide we’ll use "redis://localhost:6379" as our REDIS_URL

1. Create a tasks.py module and write a function that you want to execute in a worker using redis-tasks

import urllib.request
from redis_tasks import redis_task

@redis_task()
def print_len_of_url_content(url):
    with urllib.request.urlopen(url) as f:
        print(len(f.read()))

2. Create a settings.py module to supply your custom config to redis-tasks In this example we just pass the REDIS_URL from the enviroment

import os
REDIS_URL = os.environ['REDIS_URL']

3. In a separate terminal/shell start a worker:

   $ REDIS_URL=redis://127.0.0.1:6379 RT_SETTINGS_MODULE=settings redis_tasks worker
   

You should see a message that the worker started successfully:

16:15:39 INFO    redis_tasks.worker: Worker iw-T460p.14152 [0e015d7a-79c7-42be-8b02-add7161c5951] started

4. In a separate terminal/shell you can now enqueue this tasks to be executed by the worker at any time like this:

Note

Make sure that the python process is also started with the same ENV VARS as the worker process, see export calls in step 3.

from redis_tasks import Queue
from tasks import print_len_of_url_content
Queue().enqueue_call(
    print_len_of_url_content,
    kwargs={'url': 'https://www.readthedocs.org/'})

In the worker logs you should see the task being started and finished:

16:15:53 INFO      redis_tasks.task: Task tasks.print_len_of_url_content(url='https://www.readthedocs.org/') [095c3eb9-7073-4ad6-a7cc-16885ee962a4] started
12862
16:15:53 INFO      redis_tasks.task: Task tasks.print_len_of_url_content(url='https://www.readthedocs.org/') [095c3eb9-7073-4ad6-a7cc-16885ee962a4] finished

Adding Scheduled Tasks

So far the configuration allowed you to run the tasks almost immediately. What if you want to specify the exact time?

5. To add regularly executed tasks we extend the settings.py module to this:

import os
from redis_tasks import run_every

REDIS_URL = os.environ['REDIS_URL']

TIMEZONE = "UTC"  # timezone to be used by scheduler
SCHEDULE = {
    'print_len_of_readthedocs_org': {
        'task': 'tasks.print_len_of_url_content',
        'schedule': run_every(minutes=20),
        'args': [],
        'kwargs': {'url': 'https://readthedocs.org/'},
    },
}

For more details on SCHEDULE configuration see Scheduler

6. In a separate shell/terminal start a scheduler process:

   $ export REDIS_URL='redis://127.0.0.1:6379'
   $ export RT_SETTINGS_MODULE=rt_settings
   $ redis_tasks scheduler
   

Verify that you see a successful startup message like:

16:50:33 INFO  redis_tasks.scheduler: redis_tasks scheduler started

Every time the scheduler enqueues something, you should see a message like this in it’s logs:

16:53:00 INFO      redis_tasks.task: Task tasks.print_len_of_url_content(url='https://readthedocs.org/') [f7d4f223-929c-4eb6-b3a7-66f61dd027b7] enqueued

And the worker will log the following messages when it starts/finishes the task processing:

16:55:53 INFO      redis_tasks.task: Task tasks.print_len_of_url_content(url='https://readthedocs.org/') [095c3eb9-7073-4ad6-a7cc-16885ee962a4] started
12862
16:55:53 INFO      redis_tasks.task: Task tasks.print_len_of_url_content(url='https://readthedocs.org/') [095c3eb9-7073-4ad6-a7cc-16885ee962a4] finished

Usage:

• Command Line Interface
  • empty
  • info
  • scheduler
  • worker
• Available Settings
  • REDIS_URL
  • REDIS_PREFIX
  • TIMEZONE
  • DEFAULT_TASK_TIMEOUT
  • EXPIRING_REGISTRIES_TTL
  • DEAD_WORKER_TTL
  • WORKER_HEARTBEAT_FREQ
  • WORKER_HEARTBEAT_TIMEOUT
  • WORKER_PRELOAD_FUNCTION
  • WORKER_DESCRIPTION_FUNCTION
  • MIDDLEWARE
  • SCHEDULE
  • SCHEDULER_QUEUE
  • SCHEDULER_MAX_CATCHUP
• Scheduler
• The Middleware
• Ordering the tasks

API Docs:

• redis_tasks package
  • Submodules
  • redis_tasks.cli module
  • redis_tasks.conf module
  • redis_tasks.defaults module
  • redis_tasks.exceptions module
  • redis_tasks.queue module
  • redis_tasks.registries module
  • redis_tasks.scheduler module
  • redis_tasks.smear_dst module
  • redis_tasks.task module
  • redis_tasks.utils module
  • redis_tasks.worker module
  • redis_tasks.worker_process module
  • Module contents

Indices and tables

• Index

• Module Index

• Search Page

How it works

Signal handling

In order to assure reliable task processing, redis-tasks ships with the SIGTERM and SIGINT signals handling, making it compatible with platforms like Heroku. When one of the signals is sent, the WorkerProcess requests shutdown from the WorkHorse (a process taking care of a task). The WorkHorse ensures that the task was handled properly and interrupts the task processing only during the actual task run. The outcome of the task execution is determined based on whether the task is reentrant or not. Reentrant tasks are re-enqueued to be the first to run after the worker restart, while not reentrant tasks are considered failed. Only after that the WorkerProcess is shut down.