Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pull from queues matching a pattern #1095

Open
glacials opened this issue Jul 1, 2019 · 0 comments
Open

Pull from queues matching a pattern #1095

glacials opened this issue Jul 1, 2019 · 0 comments

Comments

@glacials
Copy link

glacials commented Jul 1, 2019

We're running delayed_job on a beta environment and a production environment simultaneously. Both access the same database.

When we push a change to beta that e.g. deletes a job type, beta will continue to pull those job types from the queues (put there by production) and try to process them, failing every time, until we promote beta to production. Same in reverse if we add a job type on beta, production starts failing to process jobs.

The natural solution seems to be

# ensure the environment doesn't pull jobs it doesn't know about
export QUEUES=job1,job2,job3

However, manually keeping QUEUES in sync with the various jobs we have seems subject to forgetfulness and poor knowledge transfer when jobs are added or deleted.

Additionally as the number of jobs increases, the environment variable gets more and more unwieldy to manage.

I would love a DRY way to achieve this same result, like matching against the existing queue prefix:

module MyApp
  class Application < Rails::Application
    config.active_job.queue_name_prefix = ENV['QUEUE_PREFIX'] # beta or production
  end
end

such that any job that starts with the prefix is processed, and others are not. This would enforce that an environment can only process jobs it knows how to without needing to keep QUEUES in sync manually.

glacials added a commit to glacials/splits-io that referenced this issue Jul 1, 2019
Previous commit didn't work because delayed_job takes from all queues
indiscriminately unless you manually specify the QUEUES env var. There
is no way to do this based on prefix, but I've opened
collectiveidea/delayed_job#1095.

For now we'll have to keep QUEUES in sync with the jobs we have.
glacials added a commit to glacials/splits-io that referenced this issue Jul 1, 2019
* Fix poisoned queues

Fixes beta and production trying to process jobs enqueued by the other,
which can make a box error indefinitely as it tries to run a job it
doesn't know how to run.

Along with this I'll set QUEUE_PREFIX to `production` and `beta` on the
respective boxes.

* Prevent delayed_job from taking from every queue

Previous commit didn't work because delayed_job takes from all queues
indiscriminately unless you manually specify the QUEUES env var. There
is no way to do this based on prefix, but I've opened
collectiveidea/delayed_job#1095.

For now we'll have to keep QUEUES in sync with the jobs we have.

* Fix a comment and remove an unused config option
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant