Minion-6.02/000755 000765 000024 00000000000 13032222265 012627 5ustar00sristaff000000 000000 Minion-6.02/Changes000644 000765 000024 00000031430 13032221720 014116 0ustar00sristaff000000 000000 6.02 2017-01-02 - Updated Mojo::Pg requirement to 2.18. 6.01 2017-01-01 - Updated Mojo::Pg requirement to 2.33. - Improved performance of Minion::Backend::Pg with a new index. 6.0 2016-09-17 - Removed TTIN, TTOU and USR1 signals from worker command. - Changed return value of start method in Minion::Job. - Added support for worker remote control commands. - Added commands attribute to Minion::Worker. - Added add_command and process_commands methods to Minion::Worker. - Added pid and stop methods to Minion::Job. - Added broadcast and receive methods to Minion::Backend::Pg. - Added -b option to job command. - Improved worker command with support for jobs and stop remote control commands. 5.09 2016-08-31 - Added EXPERIMENTAL enqueued_jobs field to stats methods in Minion and Minion::Backend::Pg. - Improved Minion::Backend::Pg performance slightly with a new index. 5.08 2016-05-20 - Improved repair methods not to remove finished jobs with unresolved dependencies. 5.07 2016-05-17 - Added support for job dependencies. (jberger, sri) - Added parents option to enqueue methods in Minion and Minion::Backend::Pg. (jberger, sri) - Added children and parents fields to info method in Minion::Job and job_info method in Minion::Backend::Pg. - Added -P option to job command. - Improved stats methods to include jobs with unresolved dependencies in delayed_jobs count. 5.06 2016-05-05 - Improved worker command to support the TTIN, TTOU and USR1 signals. - Improved Minion::Backend::Pg to handle delayed and retried jobs more efficiently. 5.05 2016-04-20 - Added queue option to list_jobs method in Minion::Backend::Pg. - Improved performance of stats method in Minion::Backend::Pg slightly. 5.04 2016-04-19 - Added EXPERIMENTAL delayed_jobs field to stats methods in Minion and Minion::Backend::Pg. - Updated Mojo::Pg requirement to 2.18. - Improved job command to show more detailed information for jobs and workers. 5.03 2016-04-10 - Added enqueue event to Minion. (jberger) 5.02 2016-03-23 - Fixed copyright notice. 5.01 2016-02-24 - Fixed worker command to repair in regular intervals. 5.0 2016-02-17 - Minion::Backend::Pg now requires PostgreSQL 9.5. - Added start event to Minion::Job. - Added -R option to worker command. - Reduced default missing_after value to 30 minutes. - Reduced default remove_after value to 2 days. - Improved Minion::Backend::Pg performance significantly with a new index and PostgreSQL 9.5 features. - Improved Minion::Job to capture more exceptions. - Improved worker command to support the QUIT signal. - Improved worker command to repair in less regular intervals. 4.06 2016-02-06 - Improved performance of Minion::Backend::Pg slightly. 4.05 2016-02-05 - Improved Minion::Backend::Pg to check the PostgreSQL version. 4.04 2016-01-23 - Updated Minion::Backend::Pg to use new Mojo::Pg features. 4.03 2016-01-17 - Removed an unused index from Minion::Backend::Pg. - Fixed a bug where the worker command would always watch the default queue. (avkhozov) 4.02 2016-01-03 - Updated links to Mojolicious website. 4.01 2015-11-12 - Improved retry methods to allow options to be changed for already inactive jobs. 4.0 2015-11-09 - Removed attempts attribute from Minion::Job. - Improved Minion::Backend::Pg to preserve more information across retries for debugging. - Fixed bug where jobs could not be retried automatically if a worker went away. 3.03 2015-11-08 - Added queues option to perform_jobs method in Minion. 3.02 2015-10-31 - Fixed portability issues in tests. 3.01 2015-10-30 - Added support for retrying failed jobs automatically. - Added backoff attribute to Minion. - Added attempts attribute to Minion::Job. - Added attempts option to enqueue methods in Minion and Minion::Backend::Pg. - Added -A option to job command. 3.0 2015-10-30 - Removed Minion::Backend::File, because DBM::Deep quickly becomes unusably slow, you can use the CPAN module Minion::Backend::SQLite instead. 2.05 2015-10-15 - Fixed bug where jobs could sometimes not be finished correctly by the worker command. 2.04 2015-10-14 - Fixed portability issue in worker command. 2.03 2015-10-09 - Improved commands to show all options that can affect their behavior. 2.02 2015-10-08 - Improved job command to show the queue in job lists. 2.01 2015-10-02 - Fixed Windows portability issues in tests. 2.0 2015-10-01 - Removed -t option from worker command. - Added support for multiple named queues. - Added retries attribute to Minion::Job. - Added retries argument to fail_job, finish_job and retry_job methods in Minion::Backend::File and Minion::Backend::Pg. - Added queue option to enqueue method in Minion. - Added queue option to enqueue and retry_job methods in Minion::Backend::File and Minion::Backend::Pg. - Added queues option to dequeue methods in Minion::Worker, Minion::Backend::File and Minion::Backend::Pg. - Added -q option to job and worker commands. - Improved worker command to be more resilient to time jumps. - Fixed a race condition in Minion::Backend::File and Minion::Backend::Pg where a retried job did not have to be dequeued again before it could be finished. 1.19 2015-09-28 - Added support for retrying jobs with a new priority. - Added priority option to retry method in Minion::Job. - Added priority option to retry_job methods in Minion::Backend::File and Minion::Backend::Pg. 1.18 2015-08-30 - Fixed Makefile.PL to be compliant with version 2 of the CPAN distribution metadata specification. 1.17 2015-08-29 - Fixed bug in worker command where new jobs would still be dequeued after receiving an INT/TERM signal. 1.16 2015-08-28 - Improved worker command to detect workers without heartbeat a little faster. 1.15 2015-05-15 - Added support for retrying jobs with a delay. (kwa) - Added delay option to retry method in Minion::Job. (kwa) - Added delay option to retry_job methods in Minion::Backend::File and Minion::Backend::Pg. (kwa) 1.14 2015-04-21 - Improved performance of Minion::Backend::Pg with a new index. (avkhozov) 1.13 2015-03-25 - Improved Minion::Backend::Pg to reset the job queue a little faster. 1.12 2015-03-17 - Improved portability of some tests. 1.11 2015-03-10 - Fixed tests to work without Mojo::Pg. 1.10 2015-03-09 - Added support for performing multiple jobs concurrently with a single worker. (bpmedley, sri) - Added is_finished and start methods to Minion::Job. (bpmedley, sri) - Added -j option to worker command. (bpmedley, sri) - Fixed concurrency bugs in Minion::Backend::File. - Fixed bug in job command where timing information was not displayed correctly. 1.09 2015-03-02 - Added support for monitoring workers with heartbeats instead of signals. - Added missing_after attribute to Minion. - Added -I option to worker command. - Fixed bug where workers were considered active even if they had no active jobs assigned. 1.08 2015-02-20 - Updated for Mojolicious 5.81. 1.07 2015-02-12 - Updated Minion::Backend::Pg for Mojo::Pg 1.08. 1.06 2015-01-26 - Improved commands to use less punctuation. 1.05 2015-01-05 - Improved repair methods in Minion::Backend::File and Minion::Backend::Pg to mention the current process in results of failed jobs. 1.04 2015-01-03 - Improved Minion::Backend::Pg to use new JSON features of Mojo::Pg. 1.03 2014-12-19 - Added -t option to worker command. 1.02 2014-11-22 - Renamed -L and -T options to -l and -S. - Improved job command table formatting. 1.01 2014-11-20 - Improved documentation. 1.0 2014-11-19 - Removed experimental status from distribution. 0.45 2014-11-18 - Improved dequeue performance in Minion::Backend::Pg significantly. (bpmedley) 0.44 2014-11-17 - Fixed bug where jobs could not be dequeued inside a running event loop. 0.43 2014-11-17 - Fixed bug where advisory locks would run out of shared memory. 0.42 2014-11-16 - Improved Minion::Backend::Pg performance with advisory locks and notifications. (bpmedley, sri) 0.41 2014-11-15 - Improved Minion::Backend::Pg performance. 0.40 2014-11-11 - Added PostgreSQL support with Mojo::Pg. (bpmedley, sri) - Added support for job results. - Added Minion::Backend::Pg. (bpmedley, sri) 0.39 2014-10-05 - Added DBM::Deep support to Minion::Backend::File. - Renamed -S option to -o. 0.38 2014-10-04 - Removed support for non-blocking enqueue. - Removed Minion::Backend::Mango. 0.37 2014-09-22 - Fixed packaging bug. 0.36 2014-09-21 - Updated Makefile.PL for version 2 of the CPAN distribution metadata specification. 0.35 2014-09-06 - Improved Minion::Backend::File to write less often. 0.34 2014-08-26 - Improved job command to show higher precision times. 0.33 2014-07-10 - Replaced state argument of list_jobs methods in Minion::Backend::File and Minion::Backend::Mango with more versatile options argument. - Added -t option to job command. 0.32 2014-07-10 - Added state argument to list_jobs methods in Minion::Backend::File and Minion::Backend::Mango. - Added -T option to job command. 0.31 2014-07-09 - Reduced CPU usage of Minion::Backend::Mango when waiting for new jobs. 0.30 2014-07-08 - Added timeout argument to dequeue methods in Minion::Backend::File, Minion::Backend::Mango and Minion::Worker. 0.29 2014-07-07 - Renamed restart_job method to retry_job in Minion::Backend, Minion::Backend::File and Minion::Backend::Mango. - Renamed restart method to retry in Minion::Job. - Improved worker command to repair in regular intervals. 0.28 2014-06-28 - Added spawn event to Minion::Job. 0.27 2014-06-21 - Replaced delayed option with delay. 0.26 2014-06-18 - Renamed clean_up_after attribute in Minion to remove_after. 0.25 2014-06-17 - Removed auto_perform attribute from Minion. - Added perform_jobs method to Minion. - Fixed multiple Windows bugs. 0.24 2014-06-16 - Improved Minion::Job to reset Mojo::IOLoop. - Fixed Windows bugs in tests. 0.23 2014-06-15 - Fixed Minion::Backend::File Windows support. 0.22 2014-06-15 - Reduced default clean_up_after value to 10 days. 0.21 2014-06-14 - Added clean_up_after attribute to Minion. - Improved performance of repair methods. 0.20 2014-06-13 - Added module Minion::Backend::File. - Improved Minion::Backend to provide a generic repair method. 0.15 2014-06-04 - Fixed a few small bugs in Minion::Backend::Mango. 0.14 2014-06-04 - Fixed Minion::Backend::Mango to work with strings in addition to object ids. 0.13 2014-06-03 - Added list_workers methods to Minion::Backend and Minion::Backend::Mango. 0.12 2014-06-03 - Fixed enqueue to use the correct time format. 0.11 2014-06-03 - Changed a few return values in Minion::Backend::Mango. 0.10 2014-06-02 - Removed created, delayed, error, finished, priority, restarted, restarts, started and state methods from Minion::Job. - Removed started method from Minion::Worker. - Added support for pluggable backends. - Added modules Minion::Backend and Minion::Backend::Mango. - Added backend attribute to Minion. - Added reset method to Minion. - Added info methods to Minion::Job and Minion::Worker. - Added -L and -S options to job command. 0.09 2014-04-05 - Added worker event to Minion. - Added dequeue event to Minion::Worker. - Added failed and finished events to Minion::Job. - Added restarted method to Minion::Job. - Changed return values of fail, finish and restart methods in Minion::Job. 0.08 2014-04-04 - Added support for non-blocking enqueue. 0.07 2014-04-03 - Improved performance by reusing Mango connections. 0.06 2014-04-03 - Added delayed and priority methods to Minion::Job. - Renamed after option to delayed. - Improved job command to use a human readable time format and allow new jobs to be enqueued. 0.05 2014-04-03 - Improved job command to stream job lists and show more information. 0.04 2014-04-02 - Removed all_jobs and one_job methods from Minion::Job. - Removed repair method from Minion::Worker; - Added module Minion::Command::minion::job. - Added auto_perform attribute to Minion. - Added repair method to Minion. - Added created, error, finished, remove, restart, restarts and started methods to Minion::Job. - Added started method to Minion::Worker. 0.03 2014-03-30 - Removed doc and worker attributes from Minion::Job. - Added args, id and minion attributes to Minion::Job. - Added id attribute to Minion::Worker. - Added job method to Minion. - Added state method to Minion::Job. 0.02 2014-03-28 - Added support for delayed jobs. - Added stats method to Minion. - Added app method to Minion::Job. - Reduced Perl requirement to 5.10.1. 0.01 2014-03-27 - First release. Minion-6.02/examples/000755 000765 000024 00000000000 13032222265 014445 5ustar00sristaff000000 000000 Minion-6.02/lib/000755 000765 000024 00000000000 13032222265 013375 5ustar00sristaff000000 000000 Minion-6.02/LICENSE000644 000765 000024 00000021413 12733154016 013642 0ustar00sristaff000000 000000 The Artistic License 2.0 Copyright (c) 2000-2006, The Perl Foundation. Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Preamble This license establishes the terms under which a given free software Package may be copied, modified, distributed, and/or redistributed. The intent is that the Copyright Holder maintains some artistic control over the development of that Package while still keeping the Package available as open source and free software. You are always permitted to make arrangements wholly outside of this license directly with the Copyright Holder of a given Package. If the terms of this license do not permit the full use that you propose to make of the Package, you should contact the Copyright Holder and seek a different licensing arrangement. Definitions "Copyright Holder" means the individual(s) or organization(s) named in the copyright notice for the entire Package. "Contributor" means any party that has contributed code or other material to the Package, in accordance with the Copyright Holder's procedures. "You" and "your" means any person who would like to copy, distribute, or modify the Package. "Package" means the collection of files distributed by the Copyright Holder, and derivatives of that collection and/or of those files. A given Package may consist of either the Standard Version, or a Modified Version. "Distribute" means providing a copy of the Package or making it accessible to anyone else, or in the case of a company or organization, to others outside of your company or organization. "Distributor Fee" means any fee that you charge for Distributing this Package or providing support for this Package to another party. It does not mean licensing fees. "Standard Version" refers to the Package if it has not been modified, or has been modified only in ways explicitly requested by the Copyright Holder. "Modified Version" means the Package, if it has been changed, and such changes were not explicitly requested by the Copyright Holder. "Original License" means this Artistic License as Distributed with the Standard Version of the Package, in its current version or as it may be modified by The Perl Foundation in the future. "Source" form means the source code, documentation source, and configuration files for the Package. "Compiled" form means the compiled bytecode, object code, binary, or any other form resulting from mechanical transformation or translation of the Source form. Permission for Use and Modification Without Distribution (1) You are permitted to use the Standard Version and create and use Modified Versions for any purpose without restriction, provided that you do not Distribute the Modified Version. Permissions for Redistribution of the Standard Version (2) You may Distribute verbatim copies of the Source form of the Standard Version of this Package in any medium without restriction, either gratis or for a Distributor Fee, provided that you duplicate all of the original copyright notices and associated disclaimers. At your discretion, such verbatim copies may or may not include a Compiled form of the Package. (3) You may apply any bug fixes, portability changes, and other modifications made available from the Copyright Holder. The resulting Package will still be considered the Standard Version, and as such will be subject to the Original License. Distribution of Modified Versions of the Package as Source (4) You may Distribute your Modified Version as Source (either gratis or for a Distributor Fee, and with or without a Compiled form of the Modified Version) provided that you clearly document how it differs from the Standard Version, including, but not limited to, documenting any non-standard features, executables, or modules, and provided that you do at least ONE of the following: (a) make the Modified Version available to the Copyright Holder of the Standard Version, under the Original License, so that the Copyright Holder may include your modifications in the Standard Version. (b) ensure that installation of your Modified Version does not prevent the user installing or running the Standard Version. In addition, the Modified Version must bear a name that is different from the name of the Standard Version. (c) allow anyone who receives a copy of the Modified Version to make the Source form of the Modified Version available to others under (i) the Original License or (ii) a license that permits the licensee to freely copy, modify and redistribute the Modified Version using the same licensing terms that apply to the copy that the licensee received, and requires that the Source form of the Modified Version, and of any works derived from it, be made freely available in that license fees are prohibited but Distributor Fees are allowed. Distribution of Compiled Forms of the Standard Version or Modified Versions without the Source (5) You may Distribute Compiled forms of the Standard Version without the Source, provided that you include complete instructions on how to get the Source of the Standard Version. Such instructions must be valid at the time of your distribution. If these instructions, at any time while you are carrying out such distribution, become invalid, you must provide new instructions on demand or cease further distribution. If you provide valid instructions or cease distribution within thirty days after you become aware that the instructions are invalid, then you do not forfeit any of your rights under this license. (6) You may Distribute a Modified Version in Compiled form without the Source, provided that you comply with Section 4 with respect to the Source of the Modified Version. Aggregating or Linking the Package (7) You may aggregate the Package (either the Standard Version or Modified Version) with other packages and Distribute the resulting aggregation provided that you do not charge a licensing fee for the Package. Distributor Fees are permitted, and licensing fees for other components in the aggregation are permitted. The terms of this license apply to the use and Distribution of the Standard or Modified Versions as included in the aggregation. (8) You are permitted to link Modified and Standard Versions with other works, to embed the Package in a larger work of your own, or to build stand-alone binary or bytecode versions of applications that include the Package, and Distribute the result without restriction, provided the result does not expose a direct interface to the Package. Items That are Not Considered Part of a Modified Version (9) Works (including, but not limited to, modules and scripts) that merely extend or make use of the Package, do not, by themselves, cause the Package to be a Modified Version. In addition, such works are not considered parts of the Package itself, and are not subject to the terms of this license. General Provisions (10) Any use, modification, and distribution of the Standard or Modified Versions is governed by this Artistic License. By using, modifying or distributing the Package, you accept this license. Do not use, modify, or distribute the Package, if you do not accept this license. (11) If your Modified Version has been derived from a Modified Version made by someone other than you, you are nevertheless required to ensure that your Modified Version complies with the requirements of this license. (12) This license does not grant you the right to use any trademark, service mark, tradename, or logo of the Copyright Holder. (13) This license includes the non-exclusive, worldwide, free-of-charge patent license to make, have made, use, offer to sell, sell, import and otherwise transfer the Package with respect to any patent claims licensable by the Copyright Holder that are necessarily infringed by the Package. If you institute patent litigation (including a cross-claim or counterclaim) against any party alleging that the Package constitutes direct or contributory patent infringement, then this Artistic License to you shall terminate on the date that such litigation is filed. (14) Disclaimer of Warranty: THE PACKAGE IS PROVIDED BY THE COPYRIGHT HOLDER AND CONTRIBUTORS "AS IS" AND WITHOUT ANY EXPRESS OR IMPLIED WARRANTIES. THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT ARE DISCLAIMED TO THE EXTENT PERMITTED BY YOUR LOCAL LAW. UNLESS REQUIRED BY LAW, NO COPYRIGHT HOLDER OR CONTRIBUTOR WILL BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING IN ANY WAY OUT OF THE USE OF THE PACKAGE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. Minion-6.02/Makefile.PL000644 000765 000024 00000001732 12733154016 014611 0ustar00sristaff000000 000000 use 5.010001; use strict; use warnings; use ExtUtils::MakeMaker; WriteMakefile( NAME => 'Minion', VERSION_FROM => 'lib/Minion.pm', ABSTRACT => 'Job queue', AUTHOR => 'Sebastian Riedel ', LICENSE => 'artistic_2', META_MERGE => { dynamic_config => 0, 'meta-spec' => {version => 2}, no_index => {directory => ['t']}, prereqs => {runtime => {requires => {perl => '5.010001'}}}, resources => { bugtracker => {web => 'https://github.com/kraih/minion/issues'}, homepage => 'http://mojolicious.org', license => ['http://www.opensource.org/licenses/artistic-license-2.0'], repository => { type => 'git', url => 'https://github.com/kraih/minion.git', web => 'https://github.com/kraih/minion', }, x_IRC => 'irc://irc.perl.org/#mojo' }, }, PREREQ_PM => {Mojolicious => '6.0'}, test => {TESTS => 't/*.t t/*/*.t'} ); Minion-6.02/MANIFEST000644 000765 000024 00000001104 13032222266 013755 0ustar00sristaff000000 000000 Changes examples/minion_bench.pl lib/Minion.pm lib/Minion/Backend.pm lib/Minion/Backend/Pg.pm lib/Minion/Command/minion.pm lib/Minion/Command/minion/job.pm lib/Minion/Command/minion/worker.pm lib/Minion/Job.pm lib/Minion/Worker.pm lib/Mojolicious/Plugin/Minion.pm LICENSE Makefile.PL MANIFEST This list of files MANIFEST.bak MANIFEST.SKIP README.md t/commands.t t/pg.t t/pg_lite_app.t t/pod.t t/pod_coverage.t META.yml Module YAML meta-data (added by MakeMaker) META.json Module JSON meta-data (added by MakeMaker) Minion-6.02/MANIFEST.bak000644 000765 000024 00000000654 12733154016 014526 0ustar00sristaff000000 000000 Changes CONTRIBUTING.md lib/Minion.pm lib/Minion/Backend.pm lib/Minion/Backend/File.pm lib/Minion/Backend/Pg.pm lib/Minion/Command/minion.pm lib/Minion/Command/minion/job.pm lib/Minion/Command/minion/worker.pm lib/Minion/Job.pm lib/Minion/Worker.pm lib/Mojolicious/Plugin/Minion.pm LICENSE Makefile.PL MANIFEST This list of files MANIFEST.SKIP README.md t/commands.t t/file.t t/file_lite_app.t t/pg.t t/pod.t t/pod_coverage.t Minion-6.02/MANIFEST.SKIP000644 000765 000024 00000000115 12733154016 014527 0ustar00sristaff000000 000000 ^\.(?!perltidyrc) .*\.old$ \.tar\.gz$ ^Makefile$ ^MYMETA\. ^blib ^pm_to_blib Minion-6.02/META.json000644 000765 000024 00000002622 13032222265 014252 0ustar00sristaff000000 000000 { "abstract" : "Job queue", "author" : [ "Sebastian Riedel " ], "dynamic_config" : 0, "generated_by" : "ExtUtils::MakeMaker version 7.24, CPAN::Meta::Converter version 2.150010", "license" : [ "artistic_2" ], "meta-spec" : { "url" : "http://search.cpan.org/perldoc?CPAN::Meta::Spec", "version" : "2" }, "name" : "Minion", "no_index" : { "directory" : [ "t", "inc", "t" ] }, "prereqs" : { "build" : { "requires" : { "ExtUtils::MakeMaker" : "0" } }, "configure" : { "requires" : { "ExtUtils::MakeMaker" : "0" } }, "runtime" : { "requires" : { "Mojolicious" : "6.0", "perl" : "5.010001" } } }, "release_status" : "stable", "resources" : { "bugtracker" : { "web" : "https://github.com/kraih/minion/issues" }, "homepage" : "http://mojolicious.org", "license" : [ "http://www.opensource.org/licenses/artistic-license-2.0" ], "repository" : { "type" : "git", "url" : "https://github.com/kraih/minion.git", "web" : "https://github.com/kraih/minion" }, "x_IRC" : "irc://irc.perl.org/#mojo" }, "version" : "6.02", "x_serialization_backend" : "JSON::PP version 2.27400" } Minion-6.02/META.yml000644 000765 000024 00000001456 13032222265 014106 0ustar00sristaff000000 000000 --- abstract: 'Job queue' author: - 'Sebastian Riedel ' build_requires: ExtUtils::MakeMaker: '0' configure_requires: ExtUtils::MakeMaker: '0' dynamic_config: 0 generated_by: 'ExtUtils::MakeMaker version 7.24, CPAN::Meta::Converter version 2.150010' license: artistic_2 meta-spec: url: http://module-build.sourceforge.net/META-spec-v1.4.html version: '1.4' name: Minion no_index: directory: - t - inc - t requires: Mojolicious: '6.0' perl: '5.010001' resources: IRC: irc://irc.perl.org/#mojo bugtracker: https://github.com/kraih/minion/issues homepage: http://mojolicious.org license: http://www.opensource.org/licenses/artistic-license-2.0 repository: https://github.com/kraih/minion.git version: '6.02' x_serialization_backend: 'CPAN::Meta::YAML version 0.018' Minion-6.02/README.md000644 000765 000024 00000003351 12733154016 014115 0ustar00sristaff000000 000000 # Minion [![Build Status](https://travis-ci.org/kraih/minion.svg?branch=master)](https://travis-ci.org/kraih/minion) A job queue for the [Mojolicious](http://mojolicious.org) real-time web framework, with support for multiple named queues, priorities, delayed jobs, job results, retries with backoff, statistics, distributed workers, parallel processing, autoscaling, resource leak protection and multiple backends (such as [PostgreSQL](http://www.postgresql.org)). Job queues allow you to process time and/or computationally intensive tasks in background processes, outside of the request/response lifecycle. Among those tasks you'll commonly find image resizing, spam filtering, HTTP downloads, building tarballs, warming caches and basically everything else you can imagine that's not super fast. ```perl use Mojolicious::Lite; use 5.20.0; use experimental 'signatures'; plugin Minion => {Pg => 'postgresql://postgres@/test'}; # Slow task app->minion->add_task(slow_log => sub ($job, $msg) { sleep 5; $job->app->log->debug(qq{Received message "$msg"}); }); # Perform job in a background worker process get '/log' => sub ($c) { $c->minion->enqueue(slow_log => [$c->param('msg') // 'no message']); $c->render(text => 'Your message will be logged soon.'); }; app->start; ``` Just start one or more background worker processes in addition to your web server. $ ./myapp.pl minion worker ## Installation All you need is a one-liner, it takes less than a minute. $ curl -L https://cpanmin.us | perl - -M https://cpan.metacpan.org -n Minion We recommend the use of a [Perlbrew](http://perlbrew.pl) environment. ## Want to know more? Take a look at our excellent [documentation](http://mojolicious.org/perldoc>)! Minion-6.02/t/000755 000765 000024 00000000000 13032222265 013072 5ustar00sristaff000000 000000 Minion-6.02/t/commands.t000644 000765 000024 00000001254 12733154016 015067 0ustar00sristaff000000 000000 use Mojo::Base -strict; use Test::More; # minion require Minion::Command::minion; my $minion = Minion::Command::minion->new; ok $minion->description, 'has a description'; like $minion->message, qr/minion/, 'has a message'; like $minion->hint, qr/help/, 'has a hint'; # job require Minion::Command::minion::job; my $job = Minion::Command::minion::job->new; ok $job->description, 'has a description'; like $job->usage, qr/job/, 'has usage information'; # worker require Minion::Command::minion::worker; my $worker = Minion::Command::minion::worker->new; ok $worker->description, 'has a description'; like $worker->usage, qr/worker/, 'has usage information'; done_testing(); Minion-6.02/t/pg.t000644 000765 000024 00000065532 13032222055 013675 0ustar00sristaff000000 000000 use Mojo::Base -strict; BEGIN { $ENV{MOJO_REACTOR} = 'Mojo::Reactor::Poll' } use Test::More; plan skip_all => 'set TEST_ONLINE to enable this test' unless $ENV{TEST_ONLINE}; use Minion; use Mojo::IOLoop; use Sys::Hostname 'hostname'; use Time::HiRes qw(time usleep); # Isolate tests require Mojo::Pg; my $pg = Mojo::Pg->new($ENV{TEST_ONLINE}); $pg->db->query('drop schema if exists minion_test cascade'); $pg->db->query('create schema minion_test'); my $minion = Minion->new(Pg => $ENV{TEST_ONLINE}); $minion->backend->pg->search_path(['minion_test']); # Nothing to repair my $worker = $minion->repair->worker; isa_ok $worker->minion->app, 'Mojolicious', 'has default application'; # Migrate up and down is $minion->backend->pg->migrations->active, 13, 'active version is 13'; is $minion->backend->pg->migrations->migrate(0)->active, 0, 'active version is 0'; is $minion->backend->pg->migrations->migrate->active, 13, 'active version is 13'; # Register and unregister $worker->register; like $worker->info->{started}, qr/^[\d.]+$/, 'has timestamp'; my $notified = $worker->info->{notified}; like $notified, qr/^[\d.]+$/, 'has timestamp'; my $id = $worker->id; is $worker->register->id, $id, 'same id'; usleep 50000; ok $worker->register->info->{notified} > $notified, 'new timestamp'; is $worker->unregister->info, undef, 'no information'; my $host = hostname; is $worker->register->info->{host}, $host, 'right host'; is $worker->info->{pid}, $$, 'right pid'; is $worker->unregister->info, undef, 'no information'; # Repair missing worker $minion->add_task(test => sub { }); my $worker2 = $minion->worker->register; isnt $worker2->id, $worker->id, 'new id'; $id = $minion->enqueue('test'); my $job = $worker2->dequeue(0); is $job->id, $id, 'right id'; is $worker2->info->{jobs}[0], $job->id, 'right id'; $id = $worker2->id; undef $worker2; is $job->info->{state}, 'active', 'job is still active'; ok !!$minion->backend->worker_info($id), 'is registered'; $minion->backend->pg->db->query( "update minion_workers set notified = now() - interval '1 second' * ? where id = ?", $minion->missing_after + 1, $id ); $minion->repair; ok !$minion->backend->worker_info($id), 'not registered'; like $job->info->{finished}, qr/^[\d.]+$/, 'has finished timestamp'; is $job->info->{state}, 'failed', 'job is no longer active'; is $job->info->{result}, 'Worker went away', 'right result'; # Repair abandoned job $worker->register; $id = $minion->enqueue('test'); $job = $worker->dequeue(0); is $job->id, $id, 'right id'; $worker->unregister; $minion->repair; is $job->info->{state}, 'failed', 'job is no longer active'; is $job->info->{result}, 'Worker went away', 'right result'; # Repair old jobs $worker->register; $id = $minion->enqueue('test'); my $id2 = $minion->enqueue('test'); my $id3 = $minion->enqueue('test'); $worker->dequeue(0)->perform for 1 .. 3; my $finished = $minion->backend->pg->db->query( 'select extract(epoch from finished) as finished from minion_jobs where id = ?', $id2 )->hash->{finished}; $minion->backend->pg->db->query( 'update minion_jobs set finished = to_timestamp(?) where id = ?', $finished - ($minion->remove_after + 1), $id2); $finished = $minion->backend->pg->db->query( 'select extract(epoch from finished) as finished from minion_jobs where id = ?', $id3 )->hash->{finished}; $minion->backend->pg->db->query( 'update minion_jobs set finished = to_timestamp(?) where id = ?', $finished - ($minion->remove_after + 1), $id3); $worker->unregister; $minion->repair; ok $minion->job($id), 'job has not been cleaned up'; ok !$minion->job($id2), 'job has been cleaned up'; ok !$minion->job($id3), 'job has been cleaned up'; # List workers $worker = $minion->worker->register; $worker2 = $minion->worker->register; my $batch = $minion->backend->list_workers(0, 10); ok $batch->[0]{id}, 'has id'; is $batch->[0]{host}, $host, 'right host'; is $batch->[0]{pid}, $$, 'right pid'; like $batch->[0]{started}, qr/^[\d.]+$/, 'has timestamp'; is $batch->[1]{host}, $host, 'right host'; is $batch->[1]{pid}, $$, 'right pid'; ok !$batch->[2], 'no more results'; $batch = $minion->backend->list_workers(0, 1); is $batch->[0]{id}, $worker2->id, 'right id'; ok !$batch->[1], 'no more results'; $batch = $minion->backend->list_workers(1, 1); is $batch->[0]{id}, $worker->id, 'right id'; ok !$batch->[1], 'no more results'; $worker->unregister; $worker2->unregister; # Reset $minion->reset->repair; ok !$minion->backend->pg->db->query( 'select count(id) as count from minion_jobs')->hash->{count}, 'no jobs'; ok !$minion->backend->pg->db->query( 'select count(id) as count from minion_workers')->hash->{count}, 'no workers'; # Stats $minion->add_task( add => sub { my ($job, $first, $second) = @_; $job->finish({added => $first + $second}); } ); $minion->add_task(fail => sub { die "Intentional failure!\n" }); my $stats = $minion->stats; is $stats->{active_workers}, 0, 'no active workers'; is $stats->{inactive_workers}, 0, 'no inactive workers'; is $stats->{enqueued_jobs}, 0, 'no enqueued jobs'; is $stats->{active_jobs}, 0, 'no active jobs'; is $stats->{failed_jobs}, 0, 'no failed jobs'; is $stats->{finished_jobs}, 0, 'no finished jobs'; is $stats->{inactive_jobs}, 0, 'no inactive jobs'; is $stats->{delayed_jobs}, 0, 'no delayed jobs'; $worker = $minion->worker->register; is $minion->stats->{inactive_workers}, 1, 'one inactive worker'; $minion->enqueue('fail'); is $minion->stats->{enqueued_jobs}, 1, 'one enqueued job'; $minion->enqueue('fail'); is $minion->stats->{enqueued_jobs}, 2, 'two enqueued jobs'; is $minion->stats->{inactive_jobs}, 2, 'two inactive jobs'; $job = $worker->dequeue(0); $stats = $minion->stats; is $stats->{active_workers}, 1, 'one active worker'; is $stats->{active_jobs}, 1, 'one active job'; is $stats->{inactive_jobs}, 1, 'one inactive job'; $minion->enqueue('fail'); my $job2 = $worker->dequeue(0); $stats = $minion->stats; is $stats->{active_workers}, 1, 'one active worker'; is $stats->{active_jobs}, 2, 'two active jobs'; is $stats->{inactive_jobs}, 1, 'one inactive job'; ok $job2->finish, 'job finished'; ok $job->finish, 'job finished'; is $minion->stats->{finished_jobs}, 2, 'two finished jobs'; $job = $worker->dequeue(0); ok $job->fail, 'job failed'; is $minion->stats->{failed_jobs}, 1, 'one failed job'; ok $job->retry, 'job retried'; is $minion->stats->{failed_jobs}, 0, 'no failed jobs'; ok $worker->dequeue(0)->finish(['works']), 'job finished'; $worker->unregister; $stats = $minion->stats; is $stats->{active_workers}, 0, 'no active workers'; is $stats->{inactive_workers}, 0, 'no inactive workers'; is $stats->{active_jobs}, 0, 'no active jobs'; is $stats->{failed_jobs}, 0, 'no failed jobs'; is $stats->{finished_jobs}, 3, 'three finished jobs'; is $stats->{inactive_jobs}, 0, 'no inactive jobs'; is $stats->{delayed_jobs}, 0, 'no delayed jobs'; # List jobs $id = $minion->enqueue('add'); $batch = $minion->backend->list_jobs(0, 10); ok $batch->[0]{id}, 'has id'; is $batch->[0]{task}, 'add', 'right task'; is $batch->[0]{state}, 'inactive', 'right state'; is $batch->[0]{retries}, 0, 'job has not been retried'; like $batch->[0]{created}, qr/^[\d.]+$/, 'has created timestamp'; is $batch->[1]{task}, 'fail', 'right task'; is_deeply $batch->[1]{args}, [], 'right arguments'; is_deeply $batch->[1]{result}, ['works'], 'right result'; is $batch->[1]{state}, 'finished', 'right state'; is $batch->[1]{priority}, 0, 'right priority'; is_deeply $batch->[1]{parents}, [], 'right parents'; is_deeply $batch->[1]{children}, [], 'right children'; is $batch->[1]{retries}, 1, 'job has been retried'; like $batch->[1]{created}, qr/^[\d.]+$/, 'has created timestamp'; like $batch->[1]{delayed}, qr/^[\d.]+$/, 'has delayed timestamp'; like $batch->[1]{finished}, qr/^[\d.]+$/, 'has finished timestamp'; like $batch->[1]{retried}, qr/^[\d.]+$/, 'has retried timestamp'; like $batch->[1]{started}, qr/^[\d.]+$/, 'has started timestamp'; is $batch->[2]{task}, 'fail', 'right task'; is $batch->[2]{state}, 'finished', 'right state'; is $batch->[2]{retries}, 0, 'job has not been retried'; is $batch->[3]{task}, 'fail', 'right task'; is $batch->[3]{state}, 'finished', 'right state'; is $batch->[3]{retries}, 0, 'job has not been retried'; ok !$batch->[4], 'no more results'; $batch = $minion->backend->list_jobs(0, 10, {state => 'inactive'}); is $batch->[0]{state}, 'inactive', 'right state'; is $batch->[0]{retries}, 0, 'job has not been retried'; ok !$batch->[1], 'no more results'; $batch = $minion->backend->list_jobs(0, 10, {task => 'add'}); is $batch->[0]{task}, 'add', 'right task'; is $batch->[0]{retries}, 0, 'job has not been retried'; ok !$batch->[1], 'no more results'; $batch = $minion->backend->list_jobs(0, 10, {queue => 'default'}); is $batch->[0]{queue}, 'default', 'right queue'; is $batch->[1]{queue}, 'default', 'right queue'; is $batch->[2]{queue}, 'default', 'right queue'; is $batch->[3]{queue}, 'default', 'right queue'; ok !$batch->[4], 'no more results'; $batch = $minion->backend->list_jobs(0, 10, {queue => 'does_not_exist'}); is_deeply $batch, [], 'no results'; $batch = $minion->backend->list_jobs(0, 1); is $batch->[0]{state}, 'inactive', 'right state'; is $batch->[0]{retries}, 0, 'job has not been retried'; ok !$batch->[1], 'no more results'; $batch = $minion->backend->list_jobs(1, 1); is $batch->[0]{state}, 'finished', 'right state'; is $batch->[0]{retries}, 1, 'job has been retried'; ok !$batch->[1], 'no more results'; ok $minion->job($id)->remove, 'job removed'; # Enqueue, dequeue and perform is $minion->job(12345), undef, 'job does not exist'; $id = $minion->enqueue(add => [2, 2]); ok $minion->job($id), 'job does exist'; my $info = $minion->job($id)->info; is_deeply $info->{args}, [2, 2], 'right arguments'; is $info->{priority}, 0, 'right priority'; is $info->{state}, 'inactive', 'right state'; $worker = $minion->worker; is $worker->dequeue(0), undef, 'not registered'; ok !$minion->job($id)->info->{started}, 'no started timestamp'; $job = $worker->register->dequeue(0); is $worker->info->{jobs}[0], $job->id, 'right job'; like $job->info->{created}, qr/^[\d.]+$/, 'has created timestamp'; like $job->info->{started}, qr/^[\d.]+$/, 'has started timestamp'; is_deeply $job->args, [2, 2], 'right arguments'; is $job->info->{state}, 'active', 'right state'; is $job->task, 'add', 'right task'; is $job->retries, 0, 'job has not been retried'; $id = $job->info->{worker}; is $minion->backend->worker_info($id)->{pid}, $$, 'right worker'; ok !$job->info->{finished}, 'no finished timestamp'; $job->perform; is $worker->info->{jobs}[0], undef, 'no jobs'; like $job->info->{finished}, qr/^[\d.]+$/, 'has finished timestamp'; is_deeply $job->info->{result}, {added => 4}, 'right result'; is $job->info->{state}, 'finished', 'right state'; $worker->unregister; $job = $minion->job($job->id); is_deeply $job->args, [2, 2], 'right arguments'; is $job->retries, 0, 'job has not been retried'; is $job->info->{state}, 'finished', 'right state'; is $job->task, 'add', 'right task'; # Retry and remove $id = $minion->enqueue(add => [5, 6]); $job = $worker->register->dequeue(0); is $job->info->{attempts}, 1, 'job will be attempted once'; is $job->info->{retries}, 0, 'job has not been retried'; is $job->id, $id, 'right id'; ok $job->finish, 'job finished'; ok !$worker->dequeue(0), 'no more jobs'; $job = $minion->job($id); ok !$job->info->{retried}, 'no retried timestamp'; ok $job->retry, 'job retried'; like $job->info->{retried}, qr/^[\d.]+$/, 'has retried timestamp'; is $job->info->{state}, 'inactive', 'right state'; is $job->info->{retries}, 1, 'job has been retried once'; $job = $worker->dequeue(0); is $job->retries, 1, 'job has been retried once'; ok !$job->retry, 'job not retried'; is $job->id, $id, 'right id'; ok !$job->remove, 'job has not been removed'; ok $job->fail, 'job failed'; ok $job->retry, 'job retried'; is $job->info->{retries}, 2, 'job has been retried twice'; $job = $worker->dequeue(0); is $job->info->{state}, 'active', 'right state'; ok $job->finish, 'job finished'; ok $job->remove, 'job has been removed'; is $job->info, undef, 'no information'; $id = $minion->enqueue(add => [6, 5]); $job = $minion->job($id); is $job->info->{state}, 'inactive', 'right state'; is $job->info->{retries}, 0, 'job has not been retried'; ok $job->retry, 'job retried'; is $job->info->{state}, 'inactive', 'right state'; is $job->info->{retries}, 1, 'job has been retried once'; $job = $worker->dequeue(0); is $job->id, $id, 'right id'; ok $job->fail, 'job failed'; ok $job->remove, 'job has been removed'; is $job->info, undef, 'no information'; $id = $minion->enqueue(add => [5, 5]); $job = $minion->job("$id"); ok $job->remove, 'job has been removed'; $worker->unregister; # Jobs with priority $minion->enqueue(add => [1, 2]); $id = $minion->enqueue(add => [2, 4], {priority => 1}); $job = $worker->register->dequeue(0); is $job->id, $id, 'right id'; is $job->info->{priority}, 1, 'right priority'; ok $job->finish, 'job finished'; isnt $worker->dequeue(0)->id, $id, 'different id'; $id = $minion->enqueue(add => [2, 5]); $job = $worker->register->dequeue(0); is $job->id, $id, 'right id'; is $job->info->{priority}, 0, 'right priority'; ok $job->finish, 'job finished'; ok $job->retry({priority => 100}), 'job retried with higher priority'; $job = $worker->dequeue(0); is $job->id, $id, 'right id'; is $job->info->{retries}, 1, 'job has been retried once'; is $job->info->{priority}, 100, 'high priority'; ok $job->finish, 'job finished'; ok $job->retry({priority => 0}), 'job retried with lower priority'; $job = $worker->dequeue(0); is $job->id, $id, 'right id'; is $job->info->{retries}, 2, 'job has been retried twice'; is $job->info->{priority}, 0, 'low priority'; ok $job->finish, 'job finished'; $worker->unregister; # Delayed jobs $id = $minion->enqueue(add => [2, 1] => {delay => 100}); is $minion->stats->{delayed_jobs}, 1, 'one delayed job'; is $worker->register->dequeue(0), undef, 'too early for job'; ok $minion->job($id)->info->{delayed} > time, 'delayed timestamp'; $minion->backend->pg->db->query( "update minion_jobs set delayed = now() - interval '1 day' where id = ?", $id); $job = $worker->dequeue(0); is $job->id, $id, 'right id'; like $job->info->{delayed}, qr/^[\d.]+$/, 'has delayed timestamp'; ok $job->finish, 'job finished'; ok $job->retry, 'job retried'; ok $minion->job($id)->info->{delayed} < time, 'no delayed timestamp'; ok $job->remove, 'job removed'; ok !$job->retry, 'job not retried'; $id = $minion->enqueue(add => [6, 9]); $job = $worker->dequeue(0); ok $job->info->{delayed} < time, 'no delayed timestamp'; ok $job->fail, 'job failed'; ok $job->retry({delay => 100}), 'job retried with delay'; is $job->info->{retries}, 1, 'job has been retried once'; ok $job->info->{delayed} > time, 'delayed timestamp'; ok $minion->job($id)->remove, 'job has been removed'; $worker->unregister; # Events my ($enqueue, $pid); my $failed = $finished = 0; $minion->once(enqueue => sub { $enqueue = pop }); $minion->once( worker => sub { my ($minion, $worker) = @_; $worker->on( dequeue => sub { my ($worker, $job) = @_; $job->on(failed => sub { $failed++ }); $job->on(finished => sub { $finished++ }); $job->on(spawn => sub { $pid = pop }); $job->on( start => sub { my $job = shift; return unless $job->task eq 'switcheroo'; $job->task('add')->args->[-1] += 1; } ); } ); } ); $worker = $minion->worker->register; $id = $minion->enqueue(add => [3, 3]); is $enqueue, $id, 'enqueue event has been emitted'; $minion->enqueue(add => [4, 3]); $job = $worker->dequeue(0); is $failed, 0, 'failed event has not been emitted'; is $finished, 0, 'finished event has not been emitted'; my $result; $job->on(finished => sub { $result = pop }); ok $job->finish('Everything is fine!'), 'job finished'; $job->perform; is $result, 'Everything is fine!', 'right result'; is $failed, 0, 'failed event has not been emitted'; is $finished, 1, 'finished event has been emitted once'; isnt $pid, $$, 'new process id'; $job = $worker->dequeue(0); my $err; $job->on(failed => sub { $err = pop }); $job->fail("test\n"); $job->fail; is $err, "test\n", 'right error'; is $failed, 1, 'failed event has been emitted once'; is $finished, 1, 'finished event has been emitted once'; $minion->add_task(switcheroo => sub { }); $minion->enqueue(switcheroo => [5, 3]); $job = $worker->dequeue(0); $job->perform; is_deeply $job->info->{result}, {added => 9}, 'right result'; $worker->unregister; # Queues $id = $minion->enqueue(add => [100, 1]); is $worker->register->dequeue(0 => {queues => ['test1']}), undef, 'wrong queue'; $job = $worker->dequeue(0); is $job->id, $id, 'right id'; is $job->info->{queue}, 'default', 'right queue'; ok $job->finish, 'job finished'; $id = $minion->enqueue(add => [100, 3] => {queue => 'test1'}); is $worker->dequeue(0), undef, 'wrong queue'; $job = $worker->dequeue(0 => {queues => ['test1']}); is $job->id, $id, 'right id'; is $job->info->{queue}, 'test1', 'right queue'; ok $job->finish, 'job finished'; ok $job->retry({queue => 'test2'}), 'job retried'; $job = $worker->dequeue(0 => {queues => ['default', 'test2']}); is $job->id, $id, 'right id'; is $job->info->{queue}, 'test2', 'right queue'; ok $job->finish, 'job finished'; $worker->unregister; # Failed jobs $id = $minion->enqueue(add => [5, 6]); $job = $worker->register->dequeue(0); is $job->id, $id, 'right id'; is $job->info->{result}, undef, 'no result'; ok $job->fail, 'job failed'; ok !$job->finish, 'job not finished'; is $job->info->{state}, 'failed', 'right state'; is $job->info->{result}, 'Unknown error', 'right result'; $id = $minion->enqueue(add => [6, 7]); $job = $worker->dequeue(0); is $job->id, $id, 'right id'; ok $job->fail('Something bad happened!'), 'job failed'; is $job->info->{state}, 'failed', 'right state'; is $job->info->{result}, 'Something bad happened!', 'right result'; $id = $minion->enqueue('fail'); $job = $worker->dequeue(0); is $job->id, $id, 'right id'; $job->perform; is $job->info->{state}, 'failed', 'right state'; is $job->info->{result}, "Intentional failure!\n", 'right result'; $worker->unregister; # Nested data structures $minion->add_task( nested => sub { my ($job, $hash, $array) = @_; $job->finish([{23 => $hash->{first}[0]{second} x $array->[0][0]}]); } ); $minion->enqueue(nested => [{first => [{second => 'test'}]}, [[3]]]); $job = $worker->register->dequeue(0); $job->perform; is $job->info->{state}, 'finished', 'right state'; is_deeply $job->info->{result}, [{23 => 'testtesttest'}], 'right structure'; $worker->unregister; # Perform job in a running event loop $id = $minion->enqueue(add => [8, 9]); Mojo::IOLoop->delay(sub { $minion->perform_jobs })->wait; is $minion->job($id)->info->{state}, 'finished', 'right state'; is_deeply $minion->job($id)->info->{result}, {added => 17}, 'right result'; # Non-zero exit status $minion->add_task(exit => sub { exit 1 }); $id = $minion->enqueue('exit'); $job = $worker->register->dequeue(0); is $job->id, $id, 'right id'; $job->perform; is $job->info->{state}, 'failed', 'right state'; is $job->info->{result}, 'Non-zero exit status (1)', 'right result'; $worker->unregister; # Multiple attempts while processing is $minion->backoff->(0), 15, 'right result'; is $minion->backoff->(1), 16, 'right result'; is $minion->backoff->(2), 31, 'right result'; is $minion->backoff->(3), 96, 'right result'; is $minion->backoff->(4), 271, 'right result'; is $minion->backoff->(5), 640, 'right result'; is $minion->backoff->(25), 390640, 'right result'; $id = $minion->enqueue(exit => [] => {attempts => 2}); $job = $worker->register->dequeue(0); is $job->id, $id, 'right id'; is $job->retries, 0, 'job has not been retried'; $job->perform; is $job->info->{attempts}, 2, 'job will be attempted twice'; is $job->info->{state}, 'inactive', 'right state'; is $job->info->{result}, 'Non-zero exit status (1)', 'right result'; ok $job->info->{retried} < $job->info->{delayed}, 'delayed timestamp'; $minion->backend->pg->db->query( 'update minion_jobs set delayed = now() where id = ?', $id); $job = $worker->register->dequeue(0); is $job->id, $id, 'right id'; is $job->retries, 1, 'job has been retried once'; $job->perform; is $job->info->{attempts}, 2, 'job will be attempted twice'; is $job->info->{state}, 'failed', 'right state'; is $job->info->{result}, 'Non-zero exit status (1)', 'right result'; $worker->unregister; # Multiple attempts during maintenance $id = $minion->enqueue(exit => [] => {attempts => 2}); $job = $worker->register->dequeue(0); is $job->id, $id, 'right id'; is $job->retries, 0, 'job has not been retried'; is $job->info->{attempts}, 2, 'job will be attempted twice'; is $job->info->{state}, 'active', 'right state'; $worker->unregister; $minion->repair; is $job->info->{state}, 'inactive', 'right state'; is $job->info->{result}, 'Worker went away', 'right result'; ok $job->info->{retried} < $job->info->{delayed}, 'delayed timestamp'; $minion->backend->pg->db->query( 'update minion_jobs set delayed = now() where id = ?', $id); $job = $worker->register->dequeue(0); is $job->id, $id, 'right id'; is $job->retries, 1, 'job has been retried once'; $worker->unregister; $minion->repair; is $job->info->{state}, 'failed', 'right state'; is $job->info->{result}, 'Worker went away', 'right result'; # A job needs to be dequeued again after a retry $minion->add_task(restart => sub { }); $id = $minion->enqueue('restart'); $job = $worker->register->dequeue(0); is $job->id, $id, 'right id'; ok $job->finish, 'job finished'; is $job->info->{state}, 'finished', 'right state'; ok $job->retry, 'job retried'; is $job->info->{state}, 'inactive', 'right state'; $job2 = $worker->dequeue(0); is $job->info->{state}, 'active', 'right state'; ok !$job->finish, 'job not finished'; is $job->info->{state}, 'active', 'right state'; is $job2->id, $id, 'right id'; ok $job2->finish, 'job finished'; ok !$job->retry, 'job not retried'; is $job->info->{state}, 'finished', 'right state'; $worker->unregister; # Perform jobs concurrently $id = $minion->enqueue(add => [10, 11]); $id2 = $minion->enqueue(add => [12, 13]); $id3 = $minion->enqueue('test'); my $id4 = $minion->enqueue('exit'); $worker = $minion->worker->register; $job = $worker->dequeue(0); $job2 = $worker->dequeue(0); my $job3 = $worker->dequeue(0); my $job4 = $worker->dequeue(0); $job->start; $job2->start; $job3->start; $job4->start; my ($first, $second, $third, $fourth); usleep 50000 until $first ||= $job->is_finished and $second ||= $job2->is_finished and $third ||= $job3->is_finished and $fourth ||= $job4->is_finished; is $minion->job($id)->info->{state}, 'finished', 'right state'; is_deeply $minion->job($id)->info->{result}, {added => 21}, 'right result'; is $minion->job($id2)->info->{state}, 'finished', 'right state'; is_deeply $minion->job($id2)->info->{result}, {added => 25}, 'right result'; is $minion->job($id3)->info->{state}, 'finished', 'right state'; is $minion->job($id3)->info->{result}, undef, 'no result'; is $minion->job($id4)->info->{state}, 'failed', 'right state'; is $minion->job($id4)->info->{result}, 'Non-zero exit status (1)', 'right result'; $worker->unregister; # Stopping jobs $minion->add_task(long_running => sub { sleep 1000 }); $worker = $minion->worker->register; $minion->enqueue('long_running'); $job = $worker->dequeue(0); ok $job->start->pid, 'has a process id'; ok !$job->is_finished, 'job is not finished'; $job->stop; usleep 5000 until $job->is_finished; is $job->info->{state}, 'failed', 'right state'; like $job->info->{result}, qr/Non-zero exit status/, 'right result'; $worker->unregister; # Job dependencies $worker = $minion->remove_after(0)->worker->register; is $minion->repair->stats->{finished_jobs}, 0, 'no finished jobs'; $id = $minion->enqueue('test'); $id2 = $minion->enqueue('test'); $id3 = $minion->enqueue(test => [] => {parents => [$id, $id2]}); is $minion->stats->{delayed_jobs}, 1, 'one delayed job'; $job = $worker->dequeue(0); is $job->id, $id, 'right id'; is_deeply $job->info->{children}, [$id3], 'right children'; is_deeply $job->info->{parents}, [], 'right parents'; $job2 = $worker->dequeue(0); is $job2->id, $id2, 'right id'; is_deeply $job2->info->{children}, [$id3], 'right children'; is_deeply $job2->info->{parents}, [], 'right parents'; ok !$worker->dequeue(0), 'parents are not ready yet'; ok $job->finish, 'job finished'; ok !$worker->dequeue(0), 'parents are not ready yet'; ok $job2->fail, 'job failed'; ok !$worker->dequeue(0), 'parents are not ready yet'; ok $job2->retry, 'job retried'; $job2 = $worker->dequeue(0); is $job2->id, $id2, 'right id'; ok $job2->finish, 'job finished'; $job = $worker->dequeue(0); is $job->id, $id3, 'right id'; is_deeply $job->info->{children}, [], 'right children'; is_deeply $job->info->{parents}, [$id, $id2], 'right parents'; is $minion->stats->{finished_jobs}, 2, 'two finished jobs'; is $minion->repair->stats->{finished_jobs}, 2, 'two finished jobs'; ok $job->finish, 'job finished'; is $minion->stats->{finished_jobs}, 3, 'three finished jobs'; is $minion->repair->stats->{finished_jobs}, 0, 'no finished jobs'; $id = $minion->enqueue(test => [] => {parents => [-1]}); ok !$worker->dequeue(0), 'job with missing parent will never be ready'; $minion->repair; like $minion->job($id)->info->{finished}, qr/^[\d.]+$/, 'has finished timestamp'; is $minion->job($id)->info->{state}, 'failed', 'right state'; is $minion->job($id)->info->{result}, 'Parent went away', 'right result'; $worker->unregister; # Worker remote control commands $worker = $minion->worker->register->process_commands; $worker2 = $minion->worker->register; my @commands; $_->add_command(test_id => sub { push @commands, shift->id }) for $worker, $worker2; $worker->add_command(test_args => sub { shift and push @commands, [@_] }) ->register; ok $minion->backend->broadcast('test_id', [], [$worker->id]), 'sent command'; ok $minion->backend->broadcast('test_id', [], [$worker->id, $worker2->id]), 'sent command'; $worker->process_commands->register; $worker2->process_commands; is_deeply \@commands, [$worker->id, $worker->id, $worker2->id], 'right structure'; @commands = (); ok $minion->backend->broadcast('test_id'), 'sent command'; ok $minion->backend->broadcast('test_whatever'), 'sent command'; ok $minion->backend->broadcast('test_args', [23], []), 'sent command'; ok $minion->backend->broadcast('test_args', [1, [2], {3 => 'three'}], [$worker->id]), 'sent command'; $_->process_commands for $worker, $worker2; is_deeply \@commands, [$worker->id, [23], [1, [2], {3 => 'three'}], $worker2->id], 'right structure'; $_->unregister for $worker, $worker2; ok !$minion->backend->broadcast('test_id', []), 'command not sent'; # Clean up once we are done $pg->db->query('drop schema minion_test cascade'); done_testing(); Minion-6.02/t/pg_lite_app.t000644 000765 000024 00000004000 13032222141 015525 0ustar00sristaff000000 000000 use Mojo::Base -strict; BEGIN { $ENV{MOJO_REACTOR} = 'Mojo::Reactor::Poll' } use Test::More; plan skip_all => 'set TEST_ONLINE to enable this test' unless $ENV{TEST_ONLINE}; use Mojo::IOLoop; use Mojolicious::Lite; use Test::Mojo; # Missing backend eval { plugin Minion => {Something => 'fun'} }; like $@, qr/^Backend "Minion::Backend::Something" missing/, 'right error'; # Isolate tests require Mojo::Pg; my $pg = Mojo::Pg->new($ENV{TEST_ONLINE}); $pg->db->query('drop schema if exists minion_app_test cascade'); $pg->db->query('create schema minion_app_test'); plugin Minion => {Pg => $ENV{TEST_ONLINE}}; app->minion->backend->pg->search_path(['minion_app_test']); app->minion->add_task( add => sub { my ($job, $first, $second) = @_; Mojo::IOLoop->next_tick( sub { $job->finish($first + $second); Mojo::IOLoop->stop; } ); Mojo::IOLoop->start; } ); get '/add' => sub { my $c = shift; my $id = $c->minion->enqueue( add => [$c->param('first'), $c->param('second')] => {queue => 'test'}); $c->render(text => $id); }; get '/result' => sub { my $c = shift; $c->render(text => $c->minion->job($c->param('id'))->info->{result}); }; my $t = Test::Mojo->new; # Perform jobs automatically $t->get_ok('/add' => form => {first => 1, second => 2})->status_is(200); $t->app->minion->perform_jobs({queues => ['test']}); $t->get_ok('/result' => form => {id => $t->tx->res->text})->status_is(200) ->content_is('3'); $t->get_ok('/add' => form => {first => 2, second => 3})->status_is(200); my $first = $t->tx->res->text; $t->get_ok('/add' => form => {first => 4, second => 5})->status_is(200); my $second = $t->tx->res->text; Mojo::IOLoop->delay(sub { $t->app->minion->perform_jobs({queues => ['test']}) }) ->wait; $t->get_ok('/result' => form => {id => $first})->status_is(200) ->content_is('5'); $t->get_ok('/result' => form => {id => $second})->status_is(200) ->content_is('9'); # Clean up once we are done $pg->db->query('drop schema minion_app_test cascade'); done_testing(); Minion-6.02/t/pod.t000644 000765 000024 00000000400 12733154016 014040 0ustar00sristaff000000 000000 use Mojo::Base -strict; use Test::More; plan skip_all => 'set TEST_POD to enable this test (developer only!)' unless $ENV{TEST_POD}; plan skip_all => 'Test::Pod 1.14+ required for this test!' unless eval 'use Test::Pod 1.14; 1'; all_pod_files_ok(); Minion-6.02/t/pod_coverage.t000644 000765 000024 00000000427 12733154016 015724 0ustar00sristaff000000 000000 use Mojo::Base -strict; use Test::More; plan skip_all => 'set TEST_POD to enable this test (developer only!)' unless $ENV{TEST_POD}; plan skip_all => 'Test::Pod::Coverage 1.04+ required for this test!' unless eval 'use Test::Pod::Coverage 1.04; 1'; all_pod_coverage_ok(); Minion-6.02/lib/Minion/000755 000765 000024 00000000000 13032222265 014626 5ustar00sristaff000000 000000 Minion-6.02/lib/Minion.pm000644 000765 000024 00000030124 13032217270 015164 0ustar00sristaff000000 000000 package Minion; use Mojo::Base 'Mojo::EventEmitter'; use Carp 'croak'; use Config; use Minion::Job; use Minion::Worker; use Mojo::Loader 'load_class'; use Mojo::Server; use Scalar::Util 'weaken'; has app => sub { Mojo::Server->new->build_app('Mojo::HelloWorld') }; has 'backend'; has backoff => sub { \&_backoff }; has missing_after => 1800; has remove_after => 172800; has tasks => sub { {} }; our $VERSION = '6.02'; sub add_task { ($_[0]->tasks->{$_[1]} = $_[2]) and return $_[0] } sub enqueue { my $self = shift; my $id = $self->backend->enqueue(@_); $self->emit(enqueue => $id); return $id; } sub job { my ($self, $id) = @_; return undef unless my $job = $self->backend->job_info($id); return Minion::Job->new( args => $job->{args}, id => $job->{id}, minion => $self, retries => $job->{retries}, task => $job->{task} ); } sub new { my $self = shift->SUPER::new; my $class = 'Minion::Backend::' . shift; my $e = load_class $class; croak ref $e ? $e : qq{Backend "$class" missing} if $e; $self->backend($class->new(@_)); weaken $self->backend->minion($self)->{minion}; return $self; } sub perform_jobs { my ($self, $options) = @_; my $worker = $self->worker; while (my $job = $worker->register->dequeue(0, $options)) { $job->perform } $worker->unregister; } sub repair { shift->_delegate('repair') } sub reset { shift->_delegate('reset') } sub stats { shift->backend->stats } sub worker { my $self = shift; # No fork emulation support croak 'Minion workers do not support fork emulation' if $Config{d_pseudofork}; my $worker = Minion::Worker->new(minion => $self); $self->emit(worker => $worker); return $worker; } sub _backoff { (shift()**4) + 15 } sub _delegate { my ($self, $method) = @_; $self->backend->$method; return $self; } 1; =encoding utf8 =head1 NAME Minion - Job queue =head1 SYNOPSIS use Minion; # Connect to backend my $minion = Minion->new(Pg => 'postgresql://postgres@/test'); # Add tasks $minion->add_task(something_slow => sub { my ($job, @args) = @_; sleep 5; say 'This is a background worker process.'; }); # Enqueue jobs $minion->enqueue(something_slow => ['foo', 'bar']); $minion->enqueue(something_slow => [1, 2, 3] => {priority => 5}); # Perform jobs for testing $minion->enqueue(something_slow => ['foo', 'bar']); $minion->perform_jobs; # Build more sophisticated workers my $worker = $minion->repair->worker; while (int rand 2) { if (my $job = $worker->register->dequeue(5)) { $job->perform } } $worker->unregister; =head1 DESCRIPTION L is a job queue for the L real-time web framework, with support for multiple named queues, priorities, delayed jobs, job dependencies, job results, retries with backoff, statistics, distributed workers, parallel processing, autoscaling, remote control, resource leak protection and multiple backends (such as L). Job queues allow you to process time and/or computationally intensive tasks in background processes, outside of the request/response lifecycle. Among those tasks you'll commonly find image resizing, spam filtering, HTTP downloads, building tarballs, warming caches and basically everything else you can imagine that's not super fast. use Mojolicious::Lite; plugin Minion => {Pg => 'postgresql://sri:s3cret@localhost/test'}; # Slow task app->minion->add_task(poke_mojo => sub { my $job = shift; $job->app->ua->get('mojolicious.org'); $job->app->log->debug('We have poked mojolicious.org for a visitor'); }); # Perform job in a background worker process get '/' => sub { my $c = shift; $c->minion->enqueue('poke_mojo'); $c->render(text => 'We will poke mojolicious.org for you soon.'); }; app->start; Background worker processes are usually started with the command L, which becomes automatically available when an application loads the plugin L. $ ./myapp.pl minion worker Jobs can be managed right from the command line with L. $ ./myapp.pl minion job To manage background worker processes with systemd, you can use a unit configuration file like this. [Unit] Description=My Mojolicious application workers After=postgresql.service [Service] Type=simple ExecStart=/home/sri/myapp/myapp.pl minion worker -m production KillMode=process [Install] WantedBy=multi-user.target Every job can fail or succeed, but not get lost, the system is eventually consistent and will preserve job results for as long as you like, depending on L. While individual workers can fail in the middle of processing a job, the system will detect this and ensure that no job is left in an uncertain state, depending on L. =head1 GROWING And as your application grows, you can move tasks into application specific plugins. package MyApp::Task::PokeMojo; use Mojo::Base 'Mojolicious::Plugin'; sub register { my ($self, $app) = @_; $app->minion->add_task(poke_mojo => sub { my $job = shift; $job->app->ua->get('mojolicious.org'); $job->app->log->debug('We have poked mojolicious.org for a visitor'); }); } 1; Which are loaded like any other plugin from your application. # Mojolicious $app->plugin('MyApp::Task::PokeMojo'); # Mojolicious::Lite plugin 'MyApp::Task::PokeMojo'; =head1 EVENTS L inherits all events from L and can emit the following new ones. =head2 enqueue $minion->on(enqueue => sub { my ($minion, $id) = @_; ... }); Emitted after a job has been enqueued, in the process that enqueued it. $minion->on(enqueue => sub { my ($minion, $id) = @_; say "Job $id has been enqueued."; }); =head2 worker $minion->on(worker => sub { my ($minion, $worker) = @_; ... }); Emitted in the worker process after it has been created. $minion->on(worker => sub { my ($minion, $worker) = @_; my $id = $worker->id; say "Worker $$:$id started."; }); =head1 ATTRIBUTES L implements the following attributes. =head2 app my $app = $minion->app; $minion = $minion->app(MyApp->new); Application for job queue, defaults to a L object. =head2 backend my $backend = $minion->backend; $minion = $minion->backend(Minion::Backend::Pg->new); Backend, usually a L object. =head2 backoff my $cb = $minion->backoff; $minion = $minion->backoff(sub {...}); A callback used to calculate the delay for automatically retried jobs, defaults to C<(retries ** 4) + 15> (15, 16, 31, 96, 271, 640...), which means that roughly C<25> attempts can be made in C<21> days. $minion->backoff(sub { my $retries = shift; return ($retries ** 4) + 15 + int(rand 30); }); =head2 missing_after my $after = $minion->missing_after; $minion = $minion->missing_after(172800); Amount of time in seconds after which workers without a heartbeat will be considered missing and removed from the registry by L, defaults to C<1800> (30 minutes). =head2 remove_after my $after = $minion->remove_after; $minion = $minion->remove_after(86400); Amount of time in seconds after which jobs that have reached the state C and have no unresolved dependencies will be removed automatically by L, defaults to C<172800> (2 days). =head2 tasks my $tasks = $minion->tasks; $minion = $minion->tasks({foo => sub {...}}); Registered tasks. =head1 METHODS L inherits all methods from L and implements the following new ones. =head2 add_task $minion = $minion->add_task(foo => sub {...}); Register a task. # Job with result $minion->add_task(add => sub { my ($job, $first, $second) = @_; $job->finish($first + $second); }); my $id = $minion->enqueue(add => [1, 1]); my $result = $minion->job($id)->info->{result}; =head2 enqueue my $id = $minion->enqueue('foo'); my $id = $minion->enqueue(foo => [@args]); my $id = $minion->enqueue(foo => [@args] => {priority => 1}); Enqueue a new job with C state. Arguments get serialized by the L (often with L), so you shouldn't send objects and be careful with binary data, nested data structures with hash and array references are fine though. These options are currently available: =over 2 =item attempts attempts => 25 Number of times performing this job will be attempted, with a delay based on L after the first attempt, defaults to C<1>. =item delay delay => 10 Delay job for this many seconds (from now). =item parents parents => [$id1, $id2, $id3] One or more existing jobs this job depends on, and that need to have transitioned to the state C before it can be processed. =item priority priority => 5 Job priority, defaults to C<0>. Jobs with a higher priority get performed first. =item queue queue => 'important' Queue to put job in, defaults to C. =back =head2 job my $job = $minion->job($id); Get L object without making any changes to the actual job or return C if job does not exist. # Check job state my $state = $minion->job($id)->info->{state}; # Get job result my $result = $minion->job($id)->info->{result}; =head2 new my $minion = Minion->new(Pg => 'postgresql://postgres@/test'); Construct a new L object. =head2 perform_jobs $minion->perform_jobs; $minion->perform_jobs({queues => ['important']}); Perform all jobs with a temporary worker, very useful for testing. # Longer version my $worker = $minion->worker; while (my $job = $worker->register->dequeue(0)) { $job->perform } $worker->unregister; These options are currently available: =over 2 =item queues queues => ['important'] One or more queues to dequeue jobs from, defaults to C. =back =head2 repair $minion = $minion->repair; Repair worker registry and job queue if necessary. =head2 reset $minion = $minion->reset; Reset job queue. =head2 stats my $stats = $minion->stats; Get statistics for jobs and workers. # Check idle workers my $idle = $minion->stats->{inactive_workers}; These fields are currently available: =over 2 =item active_jobs active_jobs => 100 Number of jobs in C state. =item active_workers active_workers => 100 Number of workers that are currently processing a job. =item delayed_jobs delayed_jobs => 100 Number of jobs in C state that are scheduled to run at specific time in the future or have unresolved dependencies. Note that this field is EXPERIMENTAL and might change without warning! =item enqueued_jobs enqueued_jobs => 100000 Rough estimate of how many jobs have ever been enqueued. Note that this field is EXPERIMENTAL and might change without warning! =item failed_jobs failed_jobs => 100 Number of jobs in C state. =item finished_jobs finished_jobs => 100 Number of jobs in C state. =item inactive_jobs inactive_jobs => 100 Number of jobs in C state. =item inactive_workers inactive_workers => 100 Number of workers that are currently not processing a job. =back =head2 worker my $worker = $minion->worker; Build L object. =head1 REFERENCE This is the class hierarchy of the L distribution. =over 2 =item * L =item * L =over 2 =item * L =back =item * L =item * L =item * L =item * L =item * L =item * L =back =head1 AUTHOR Sebastian Riedel, C. =head1 CREDITS In alphabetical order: =over 2 Andrey Khozov Brian Medley Joel Berger Paul Williams =back =head1 COPYRIGHT AND LICENSE Copyright (C) 2014-2017, Sebastian Riedel and others. This program is free software, you can redistribute it and/or modify it under the terms of the Artistic License version 2.0. =head1 SEE ALSO L, L, L. =cut Minion-6.02/lib/Mojolicious/000755 000765 000024 00000000000 13032222265 015671 5ustar00sristaff000000 000000 Minion-6.02/lib/Mojolicious/Plugin/000755 000765 000024 00000000000 13032222265 017127 5ustar00sristaff000000 000000 Minion-6.02/lib/Mojolicious/Plugin/Minion.pm000644 000765 000024 00000003547 12733154016 020734 0ustar00sristaff000000 000000 package Mojolicious::Plugin::Minion; use Mojo::Base 'Mojolicious::Plugin'; use Minion; use Scalar::Util 'weaken'; sub register { my ($self, $app, $conf) = @_; push @{$app->commands->namespaces}, 'Minion::Command'; my $minion = Minion->new(each %$conf); weaken $minion->app($app)->{app}; $app->helper(minion => sub {$minion}); } 1; =encoding utf8 =head1 NAME Mojolicious::Plugin::Minion - Minion job queue plugin =head1 SYNOPSIS # Mojolicious (choose a backend) $self->plugin(Minion => {Pg => 'postgresql://postgres@/test'}); # Mojolicious::Lite (choose a backend) plugin Minion => {Pg => 'postgresql://postgres@/test'}; # Add tasks to your application app->minion->add_task(slow_log => sub { my ($job, $msg) = @_; sleep 5; $job->app->log->debug(qq{Received message "$msg"}); }); # Start jobs from anywhere in your application $c->minion->enqueue(slow_log => ['test 123']); # Perform jobs in your tests $t->get_ok('/start_slow_log_job')->status_is(200); $t->get_ok('/start_another_job')->status_is(200); $t->app->minion->perform_jobs; =head1 DESCRIPTION L is a L plugin for the L job queue. =head1 HELPERS L implements the following helpers. =head2 minion my $minion = $app->minion; my $minion = $c->minion; Get L object for application. # Add job to the queue $c->minion->enqueue(foo => ['bar', 'baz']); # Perform jobs for testing $app->minion->perform_jobs; =head1 METHODS L inherits all methods from L and implements the following new ones. =head2 register $plugin->register(Mojolicious->new, {Pg => 'postgresql://postgres@/test'}); Register plugin in L application. =head1 SEE ALSO L, L, L. =cut Minion-6.02/lib/Minion/Backend/000755 000765 000024 00000000000 13032222265 016155 5ustar00sristaff000000 000000 Minion-6.02/lib/Minion/Backend.pm000644 000765 000024 00000024471 12772223573 016540 0ustar00sristaff000000 000000 package Minion::Backend; use Mojo::Base -base; use Carp 'croak'; has 'minion'; sub broadcast { croak 'Method "broadcast" not implemented by subclass' } sub dequeue { croak 'Method "dequeue" not implemented by subclass' } sub enqueue { croak 'Method "enqueue" not implemented by subclass' } sub fail_job { croak 'Method "fail_job" not implemented by subclass' } sub finish_job { croak 'Method "finish_job" not implemented by subclass' } sub job_info { croak 'Method "job_info" not implemented by subclass' } sub list_jobs { croak 'Method "list_jobs" not implemented by subclass' } sub list_workers { croak 'Method "list_workers" not implemented by subclass' } sub receive { croak 'Method "receive" not implemented by subclass' } sub register_worker { croak 'Method "register_worker" not implemented by subclass'; } sub remove_job { croak 'Method "remove_job" not implemented by subclass' } sub repair { croak 'Method "repair" not implemented by subclass' } sub reset { croak 'Method "reset" not implemented by subclass' } sub retry_job { croak 'Method "retry_job" not implemented by subclass' } sub stats { croak 'Method "stats" not implemented by subclass' } sub unregister_worker { croak 'Method "unregister_worker" not implemented by subclass'; } sub worker_info { croak 'Method "worker_info" not implemented by subclass' } 1; =encoding utf8 =head1 NAME Minion::Backend - Backend base class =head1 SYNOPSIS package Minion::Backend::MyBackend; use Mojo::Base 'Minion::Backend'; sub broadcast {...} sub dequeue {...} sub enqueue {...} sub fail_job {...} sub finish_job {...} sub job_info {...} sub list_jobs {...} sub list_workers {...} sub receive {...} sub register_worker {...} sub remove_job {...} sub repair {...} sub reset {...} sub retry_job {...} sub stats {...} sub unregister_worker {...} sub worker_info {...} =head1 DESCRIPTION L is an abstract base class for L backends, like L. =head1 ATTRIBUTES L implements the following attributes. =head2 minion my $minion = $backend->minion; $backend = $backend->minion(Minion->new); L object this backend belongs to. =head1 METHODS L inherits all methods from L and implements the following new ones. =head2 broadcast my $bool = $backend->broadcast('some_command'); my $bool = $backend->broadcast('some_command', [@args]); my $bool = $backend->broadcast('some_command', [@args], [$id1, $id2, $id3]); Broadcast remote control command to one or more workers. =head2 dequeue my $job_info = $backend->dequeue($worker_id, 0.5); my $job_info = $backend->dequeue($worker_id, 0.5, {queues => ['important']}); Wait a given amount of time in seconds for a job, dequeue it and transition from C to C state, or return C if queues were empty. Meant to be overloaded in a subclass. These options are currently available: =over 2 =item queues queues => ['important'] One or more queues to dequeue jobs from, defaults to C. =back These fields are currently available: =over 2 =item args args => ['foo', 'bar'] Job arguments. =item id id => '10023' Job ID. =item retries retries => 3 Number of times job has been retried. =item task task => 'foo' Task name. =back =head2 enqueue my $job_id = $backend->enqueue('foo'); my $job_id = $backend->enqueue(foo => [@args]); my $job_id = $backend->enqueue(foo => [@args] => {priority => 1}); Enqueue a new job with C state. Meant to be overloaded in a subclass. These options are currently available: =over 2 =item attempts attempts => 25 Number of times performing this job will be attempted, with a delay based on L after the first attempt, defaults to C<1>. =item delay delay => 10 Delay job for this many seconds (from now). =item parents parents => [$id1, $id2, $id3] One or more existing jobs this job depends on, and that need to have transitioned to the state C before it can be processed. =item priority priority => 5 Job priority, defaults to C<0>. Jobs with a higher priority get performed first. =item queue queue => 'important' Queue to put job in, defaults to C. =back =head2 fail_job my $bool = $backend->fail_job($job_id, $retries); my $bool = $backend->fail_job($job_id, $retries, 'Something went wrong!'); my $bool = $backend->fail_job( $job_id, $retries, {whatever => 'Something went wrong!'}); Transition from C to C state, and if there are attempts remaining, transition back to C with a delay based on L. Meant to be overloaded in a subclass. =head2 finish_job my $bool = $backend->finish_job($job_id, $retries); my $bool = $backend->finish_job($job_id, $retries, 'All went well!'); my $bool = $backend->finish_job( $job_id, $retries, {whatever => 'All went well!'}); Transition from C to C state. Meant to be overloaded in a subclass. =head2 job_info my $job_info = $backend->job_info($job_id); Get information about a job, or return C if job does not exist. Meant to be overloaded in a subclass. # Check job state my $state = $backend->job_info($job_id)->{state}; # Get job result my $result = $backend->job_info($job_id)->{result}; These fields are currently available: =over 2 =item args args => ['foo', 'bar'] Job arguments. =item attempts attempts => 25 Number of times performing this job will be attempted. =item children children => ['10026', '10027', '10028'] Jobs depending on this job. =item created created => 784111777 Epoch time job was created. =item delayed delayed => 784111777 Epoch time job was delayed to. =item finished finished => 784111777 Epoch time job was finished. =item parents parents => ['10023', '10024', '10025'] Jobs this job depends on. =item priority priority => 3 Job priority. =item queue queue => 'important' Queue name. =item result result => 'All went well!' Job result. =item retried retried => 784111777 Epoch time job has been retried. =item retries retries => 3 Number of times job has been retried. =item started started => 784111777 Epoch time job was started. =item state state => 'inactive' Current job state, usually C, C, C or C. =item task task => 'foo' Task name. =item worker worker => '154' Id of worker that is processing the job. =back =head2 list_jobs my $batch = $backend->list_jobs($offset, $limit); my $batch = $backend->list_jobs($offset, $limit, {state => 'inactive'}); Returns the same information as L but in batches. Meant to be overloaded in a subclass. These options are currently available: =over 2 =item queue queue => 'important' List only jobs in this queue. =item state state => 'inactive' List only jobs in this state. =item task task => 'test' List only jobs for this task. =back =head2 list_workers my $batch = $backend->list_workers($offset, $limit); Returns the same information as L but in batches. Meant to be overloaded in a subclass. =head2 receive my $commands = $backend->receive($worker_id); Receive remote control commands for worker. =head2 register_worker my $worker_id = $backend->register_worker; my $worker_id = $backend->register_worker($worker_id); Register worker or send heartbeat to show that this worker is still alive. Meant to be overloaded in a subclass. =head2 remove_job my $bool = $backend->remove_job($job_id); Remove C, C or C job from queue. Meant to be overloaded in a subclass. =head2 repair $backend->repair; Repair worker registry and job queue if necessary. Meant to be overloaded in a subclass. =head2 reset $backend->reset; Reset job queue. Meant to be overloaded in a subclass. =head2 retry_job my $bool = $backend->retry_job($job_id, $retries); my $bool = $backend->retry_job($job_id, $retries, {delay => 10}); Transition from C or C state back to C, already C jobs may also be retried to change options. Meant to be overloaded in a subclass. These options are currently available: =over 2 =item delay delay => 10 Delay job for this many seconds (from now). =item priority priority => 5 Job priority. =item queue queue => 'important' Queue to put job in. =back =head2 stats my $stats = $backend->stats; Get statistics for jobs and workers. Meant to be overloaded in a subclass. These fields are currently available: =over 2 =item active_jobs active_jobs => 100 Number of jobs in C state. =item active_workers active_workers => 100 Number of workers that are currently processing a job. =item delayed_jobs delayed_jobs => 100 Number of jobs in C state that are scheduled to run at specific time in the future or have unresolved dependencies. Note that this field is EXPERIMENTAL and might change without warning! =item failed_jobs failed_jobs => 100 Number of jobs in C state. =item finished_jobs finished_jobs => 100 Number of jobs in C state. =item inactive_jobs inactive_jobs => 100 Number of jobs in C state. =item inactive_workers inactive_workers => 100 Number of workers that are currently not processing a job. =back =head2 unregister_worker $backend->unregister_worker($worker_id); Unregister worker. Meant to be overloaded in a subclass. =head2 worker_info my $worker_info = $backend->worker_info($worker_id); Get information about a worker, or return C if worker does not exist. Meant to be overloaded in a subclass. # Check worker host my $host = $backend->worker_info($worker_id)->{host}; These fields are currently available: =over 2 =item host host => 'localhost' Worker host. =item jobs jobs => ['10023', '10024', '10025', '10029'] Ids of jobs the worker is currently processing. =item notified notified => 784111777 Epoch time worker sent the last heartbeat. =item pid pid => 12345 Process id of worker. =item started started => 784111777 Epoch time worker was started. =back =head1 SEE ALSO L, L, L. =cut Minion-6.02/lib/Minion/Command/000755 000765 000024 00000000000 13032222265 016204 5ustar00sristaff000000 000000 Minion-6.02/lib/Minion/Job.pm000644 000765 000024 00000015627 12767205330 015721 0ustar00sristaff000000 000000 package Minion::Job; use Mojo::Base 'Mojo::EventEmitter'; use Mojo::IOLoop; use POSIX 'WNOHANG'; has [qw(args id minion retries task)]; sub app { shift->minion->app } sub fail { my ($self, $err) = (shift, shift // 'Unknown error'); my $ok = $self->minion->backend->fail_job($self->id, $self->retries, $err); return $ok ? !!$self->emit(failed => $err) : undef; } sub finish { my ($self, $result) = @_; my $ok = $self->minion->backend->finish_job($self->id, $self->retries, $result); return $ok ? !!$self->emit(finished => $result) : undef; } sub info { $_[0]->minion->backend->job_info($_[0]->id) } sub is_finished { my $self = shift; return undef unless waitpid($self->{pid}, WNOHANG) == $self->{pid}; $? ? $self->fail("Non-zero exit status (@{[$? >> 8]})") : $self->finish; return 1; } sub stop { kill 'KILL', shift->{pid} } sub perform { my $self = shift; waitpid $self->start->pid, 0; $? ? $self->fail("Non-zero exit status (@{[$? >> 8]})") : $self->finish; } sub pid { shift->{pid} } sub remove { $_[0]->minion->backend->remove_job($_[0]->id) } sub retry { my $self = shift; return $self->minion->backend->retry_job($self->id, $self->retries, @_); } sub start { my $self = shift; # Parent die "Can't fork: $!" unless defined(my $pid = fork); return $self->emit(spawn => $pid) if $self->{pid} = $pid; # Child eval { # Reset event loop Mojo::IOLoop->reset; $self->minion->tasks->{$self->emit('start')->task}->($self, @{$self->args}); 1; } or $self->fail($@); POSIX::_exit(0); } 1; =encoding utf8 =head1 NAME Minion::Job - Minion job =head1 SYNOPSIS use Minion::Job; my $job = Minion::Job->new(id => $id, minion => $minion, task => 'foo'); =head1 DESCRIPTION L is a container for L jobs. =head1 EVENTS L inherits all events from L and can emit the following new ones. =head2 failed $job->on(failed => sub { my ($job, $err) = @_; ... }); Emitted in the worker process managing this job or the process performing it, after it has transitioned to the C state. $job->on(failed => sub { my ($job, $err) = @_; say "Something went wrong: $err"; }); =head2 finished $job->on(finished => sub { my ($job, $result) = @_; ... }); Emitted in the worker process managing this job or the process performing it, after it has transitioned to the C state. $job->on(finished => sub { my ($job, $result) = @_; my $id = $job->id; say "Job $id is finished."; }); =head2 spawn $job->on(spawn => sub { my ($job, $pid) = @_; ... }); Emitted in the worker process managing this job, after a new process has been spawned for processing. $job->on(spawn => sub { my ($job, $pid) = @_; my $id = $job->id; say "Job $id running in process $pid"; }); =head2 start $job->on(start => sub { my $job = shift; ... }); Emitted in the process performing this job, after it has been spawned. $job->on(start => sub { my $job = shift; $0 = $job->id; }); =head1 ATTRIBUTES L implements the following attributes. =head2 args my $args = $job->args; $job = $job->args([]); Arguments passed to task. =head2 id my $id = $job->id; $job = $job->id($id); Job id. =head2 minion my $minion = $job->minion; $job = $job->minion(Minion->new); L object this job belongs to. =head2 retries my $retries = $job->retries; $job = $job->retries(5); Number of times job has been retried. =head2 task my $task = $job->task; $job = $job->task('foo'); Task name. =head1 METHODS L inherits all methods from L and implements the following new ones. =head2 app my $app = $job->app; Get application from L. # Longer version my $app = $job->minion->app; =head2 fail my $bool = $job->fail; my $bool = $job->fail('Something went wrong!'); my $bool = $job->fail({whatever => 'Something went wrong!'}); Transition from C to C state, and if there are attempts remaining, transition back to C with a delay based on L. =head2 finish my $bool = $job->finish; my $bool = $job->finish('All went well!'); my $bool = $job->finish({whatever => 'All went well!'}); Transition from C to C state. =head2 info my $info = $job->info; Get job information. # Check job state my $state = $job->info->{state}; # Get job result my $result = $job->info->{result}; These fields are currently available: =over 2 =item args args => ['foo', 'bar'] Job arguments. =item attempts attempts => 25 Number of times performing this job will be attempted. =item children children => ['10026', '10027', '10028'] Jobs depending on this job. =item created created => 784111777 Epoch time job was created. =item delayed delayed => 784111777 Epoch time job was delayed to. =item finished finished => 784111777 Epoch time job was finished. =item parents parents => ['10023', '10024', '10025'] Jobs this job depends on. =item priority priority => 3 Job priority. =item queue queue => 'important' Queue name. =item result result => 'All went well!' Job result. =item retried retried => 784111777 Epoch time job has been retried. =item retries retries => 3 Number of times job has been retried. =item started started => 784111777 Epoch time job was started. =item state state => 'inactive' Current job state, usually C, C, C or C. =item task task => 'foo' Task name. =item worker worker => '154' Id of worker that is processing the job. =back =head2 is_finished my $bool = $job->is_finished; Check if job performed with L is finished. =head2 perform $job->perform; Perform job in new process and wait for it to finish. =head2 pid my $pid = $job->pid; Process id of the process spawned by L if available. =head2 remove my $bool = $job->remove; Remove C, C or C job from queue. =head2 retry my $bool = $job->retry; my $bool = $job->retry({delay => 10}); Transition from C or C state back to C, already C jobs may also be retried to change options. These options are currently available: =over 2 =item delay delay => 10 Delay job for this many seconds (from now). =item priority priority => 5 Job priority. =item queue queue => 'important' Queue to put job in. =back =head2 start $job = $job->start; Perform job in new process, but do not wait for it to finish. # Perform two jobs concurrently $job1->start; $job2->start; my ($first, $second); sleep 1 until $first ||= $job1->is_finished and $second ||= $job2->is_finished; =head2 stop $job->stop; Stop job performed with L immediately. =head1 SEE ALSO L, L, L. =cut Minion-6.02/lib/Minion/Worker.pm000644 000765 000024 00000007475 12767202352 016463 0ustar00sristaff000000 000000 package Minion::Worker; use Mojo::Base 'Mojo::EventEmitter'; has commands => sub { {} }; has [qw(id minion)]; sub add_command { $_[0]->commands->{$_[1]} = $_[2] and return $_[0] } sub dequeue { my ($self, $wait, $options) = @_; # Worker not registered return undef unless my $id = $self->id; my $minion = $self->minion; return undef unless my $job = $minion->backend->dequeue($id, $wait, $options); $job = Minion::Job->new( args => $job->{args}, id => $job->{id}, minion => $minion, retries => $job->{retries}, task => $job->{task} ); $self->emit(dequeue => $job); return $job; } sub info { $_[0]->minion->backend->worker_info($_[0]->id) } sub process_commands { my $self = shift; for my $command (@{$self->minion->backend->receive($self->id)}) { next unless my $cb = $self->commands->{shift @$command}; $self->$cb(@$command); } return $self; } sub register { $_[0]->id($_[0]->minion->backend->register_worker($_[0]->id)) } sub unregister { my $self = shift; $self->minion->backend->unregister_worker(delete $self->{id}); return $self; } 1; =encoding utf8 =head1 NAME Minion::Worker - Minion worker =head1 SYNOPSIS use Minion::Worker; my $worker = Minion::Worker->new(minion => $minion); =head1 DESCRIPTION L performs jobs for L. =head1 EVENTS L inherits all events from L and can emit the following new ones. =head2 dequeue $worker->on(dequeue => sub { my ($worker, $job) = @_; ... }); Emitted in the worker process after a job has been dequeued. $worker->on(dequeue => sub { my ($worker, $job) = @_; my $id = $job->id; say "Job $id has been dequeued."; }); =head1 ATTRIBUTES L implements the following attributes. =head2 commands my $commands = $worker->commands; $worker = $worker->commands({jobs => sub {...}}); Registered worker remote control commands. =head2 id my $id = $worker->id; $worker = $worker->id($id); Worker id. =head2 minion my $minion = $worker->minion; $worker = $worker->minion(Minion->new); L object this worker belongs to. =head1 METHODS L inherits all methods from L and implements the following new ones. =head2 add_command $worker = $worker->add_command(jobs => sub {...}); Register a worker remote control command. $worker->add_command(foo => sub { my ($worker, @args) = @_; ... }); =head2 dequeue my $job = $worker->dequeue(0.5); my $job = $worker->dequeue(0.5 => {queues => ['important']}); Wait a given amount of time in seconds for a job, dequeue L object and transition from C to C state, or return C if queues were empty. These options are currently available: =over 2 =item queues queues => ['important'] One or more queues to dequeue jobs from, defaults to C. =back =head2 info my $info = $worker->info; Get worker information. # Check worker host my $host = $worker->info->{host}; These fields are currently available: =over 2 =item host host => 'localhost' Worker host. =item jobs jobs => ['10023', '10024', '10025', '10029'] Ids of jobs the worker is currently processing. =item notified notified => 784111777 Epoch time worker sent the last heartbeat. =item pid pid => 12345 Process id of worker. =item started started => 784111777 Epoch time worker was started. =back =head2 process_commands $worker = $worker->process_commands; Process worker remote control commands. =head2 register $worker = $worker->register; Register worker or send heartbeat to show that this worker is still alive. =head2 unregister $worker = $worker->unregister; Unregister worker. =head1 SEE ALSO L, L, L. =cut Minion-6.02/lib/Minion/Command/minion/000755 000765 000024 00000000000 13032222265 017475 5ustar00sristaff000000 000000 Minion-6.02/lib/Minion/Command/minion.pm000644 000765 000024 00000003313 12733154016 020040 0ustar00sristaff000000 000000 package Minion::Command::minion; use Mojo::Base 'Mojolicious::Commands'; has description => 'Minion job queue'; has hint => < sub { shift->extract_usage . "\nCommands:\n" }; has namespaces => sub { ['Minion::Command::minion'] }; sub help { shift->run(@_) } 1; =encoding utf8 =head1 NAME Minion::Command::minion - Minion command =head1 SYNOPSIS Usage: APPLICATION minion COMMAND [OPTIONS] =head1 DESCRIPTION L lists available L commands. =head1 ATTRIBUTES L inherits all attributes from L and implements the following new ones. =head2 description my $description = $minion->description; $minion = $minion->description('Foo'); Short description of this command, used for the command list. =head2 hint my $hint = $minion->hint; $minion = $minion->hint('Foo'); Short hint shown after listing available L commands. =head2 message my $msg = $minion->message; $minion = $minion->message('Bar'); Short usage message shown before listing available L commands. =head2 namespaces my $namespaces = $minion->namespaces; $minion = $minion->namespaces(['MyApp::Command::minion']); Namespaces to search for available L commands, defaults to L. =head1 METHODS L inherits all methods from L and implements the following new ones. =head2 help $minion->help('app'); Print usage information for L command. =head1 SEE ALSO L, L, L. =cut Minion-6.02/lib/Minion/Command/minion/job.pm000644 000765 000024 00000013225 12767774333 020636 0ustar00sristaff000000 000000 package Minion::Command::minion::job; use Mojo::Base 'Mojolicious::Command'; use Getopt::Long qw(GetOptionsFromArray :config no_auto_abbrev no_ignore_case); use Mojo::JSON 'decode_json'; use Mojo::Util qw(dumper tablify); has description => 'Manage Minion jobs'; has usage => sub { shift->extract_usage }; sub run { my ($self, @args) = @_; my ($args, $options) = ([], {}); GetOptionsFromArray \@args, 'A|attempts=i' => \$options->{attempts}, 'a|args=s' => sub { $args = decode_json($_[1]) }, 'b|broadcast=s' => (\my $command), 'd|delay=i' => \$options->{delay}, 'e|enqueue=s' => \my $enqueue, 'l|limit=i' => \(my $limit = 100), 'o|offset=i' => \(my $offset = 0), 'P|parent=s' => ($options->{parents} = []), 'p|priority=i' => \$options->{priority}, 'q|queue=s' => \$options->{queue}, 'R|retry' => \my $retry, 'r|remove' => \my $remove, 'S|state=s' => \$options->{state}, 's|stats' => \my $stats, 't|task=s' => \$options->{task}, 'w|workers' => \my $workers; # Worker remote control command return $self->app->minion->backend->broadcast($command, $args, \@args) if $command; # Enqueue return say $self->app->minion->enqueue($enqueue, $args, $options) if $enqueue; # Show stats return $self->_stats if $stats; # List jobs/workers my $id = @args ? shift @args : undef; return $id ? $self->_worker($id) : $self->_list_workers($offset, $limit) if $workers; return $self->_list_jobs($offset, $limit, $options) unless defined $id; die "Job does not exist.\n" unless my $job = $self->app->minion->job($id); # Remove job return $job->remove || die "Job is active.\n" if $remove; # Retry job return $job->retry($options) || die "Job is active.\n" if $retry; # Job info print dumper $job->info; } sub _list_jobs { my $jobs = shift->app->minion->backend->list_jobs(@_); print tablify [map { [@$_{qw(id state queue task)}] } @$jobs]; } sub _list_workers { my $workers = shift->app->minion->backend->list_workers(@_); my @workers = map { [$_->{id}, $_->{host} . ':' . $_->{pid}] } @$workers; print tablify \@workers; } sub _stats { print dumper shift->app->minion->stats } sub _worker { die "Worker does not exist.\n" unless my $worker = shift->app->minion->backend->worker_info(@_); print dumper $worker; } 1; =encoding utf8 =head1 NAME Minion::Command::minion::job - Minion job command =head1 SYNOPSIS Usage: APPLICATION minion job [OPTIONS] [IDS] ./myapp.pl minion job ./myapp.pl minion job 10023 ./myapp.pl minion job -w ./myapp.pl minion job -w 23 ./myapp.pl minion job -s ./myapp.pl minion job -q important -t foo -S inactive ./myapp.pl minion job -e foo -a '[23, "bar"]' ./myapp.pl minion job -e foo -P 10023 -P 10024 -p 5 -q important ./myapp.pl minion job -R -d 10 10023 ./myapp.pl minion job -r 10023 ./myapp.pl minion job -b jobs -a '[12]' ./myapp.pl minion job -b jobs -a '[12]' 23 24 25 Options: -A, --attempts Number of times performing this new job will be attempted, defaults to 1 -a, --args Arguments for new job or worker remote control command in JSON format -b, --broadcast Broadcast remote control command to one or more workers -d, --delay Delay new job for this many seconds -e, --enqueue New job to be enqueued -h, --help Show this summary of available options --home Path to home directory of your application, defaults to the value of MOJO_HOME or auto-detection -l, --limit Number of jobs/workers to show when listing them, defaults to 100 -m, --mode Operating mode for your application, defaults to the value of MOJO_MODE/PLACK_ENV or "development" -o, --offset Number of jobs/workers to skip when listing them, defaults to 0 -P, --parent One or more jobs the new job depends on -p, --priority Priority of new job, defaults to 0 -q, --queue Queue to put new job in, defaults to "default", or list only jobs in this queue -R, --retry Retry job -r, --remove Remove job -S, --state List only jobs in this state -s, --stats Show queue statistics -t, --task List only jobs for this task -w, --workers List workers instead of jobs, or show information for a specific worker =head1 DESCRIPTION L manages the L job queue. =head1 ATTRIBUTES L inherits all attributes from L and implements the following new ones. =head2 description my $description = $job->description; $job = $job->description('Foo'); Short description of this command, used for the command list. =head2 usage my $usage = $job->usage; $job = $job->usage('Foo'); Usage information for this command, used for the help screen. =head1 METHODS L inherits all methods from L and implements the following new ones. =head2 run $job->run(@ARGV); Run this command. =head1 SEE ALSO L, L, L. =cut Minion-6.02/lib/Minion/Command/minion/worker.pm000644 000765 000024 00000014144 12767776500 021374 0ustar00sristaff000000 000000 package Minion::Command::minion::worker; use Mojo::Base 'Mojolicious::Command'; use Getopt::Long qw(GetOptionsFromArray :config no_auto_abbrev no_ignore_case); use Mojo::Util 'steady_time'; has description => 'Start Minion worker'; has usage => sub { shift->extract_usage }; sub run { my ($self, @args) = @_; GetOptionsFromArray \@args, 'C|command-interval=i' => \($self->{commands} = 10), 'I|heartbeat-interval=i' => \($self->{heartbeat} = 60), 'j|jobs=i' => \($self->{max} = 4), 'q|queue=s' => \my @queues, 'R|repair-interval=i' => \($self->{repair} = 21600); $self->{queues} = @queues ? \@queues : ['default']; local $SIG{CHLD} = 'DEFAULT'; local $SIG{INT} = local $SIG{TERM} = sub { $self->{finished}++ }; local $SIG{QUIT} = sub { ++$self->{finished} and kill 'KILL', keys %{$self->{jobs}} }; # Remote control commands need to validate arguments carefully my $app = $self->app; my $worker = $self->{worker} = $app->minion->worker; $worker->add_command( jobs => sub { $self->{max} = $_[1] if ($_[1] // '') =~ /^\d+$/ }); $worker->add_command( stop => sub { $self->{jobs}{$_[1]}->stop if $self->{jobs}{$_[1] // ''} }); # Log fatal errors $app->log->debug("Worker $$ started"); eval { $self->_work until $self->{finished} && !keys %{$self->{jobs}}; 1 } or $app->log->fatal("Worker error: $@"); $worker->unregister; } sub _work { my $self = shift; # Send heartbeats in regular intervals my $worker = $self->{worker}; $worker->register and $self->{lr} = steady_time + $self->{heartbeat} if ($self->{lr} || 0) < steady_time; # Process worker remote control commands in regular intervals $worker->process_commands and $self->{lp} = steady_time + $self->{commands} if ($self->{lp} || 0) < steady_time; # Repair in regular intervals (randomize to avoid congestion) if (($self->{check} || 0) < steady_time) { my $app = $self->app; $app->log->debug('Checking worker registry and job queue'); $app->minion->repair; $self->{check} = steady_time + ($self->{repair} - int rand $self->{repair} / 2); } # Check if jobs are finished my $jobs = $self->{jobs} ||= {}; $jobs->{$_}->is_finished and delete $jobs->{$_} for keys %$jobs; # Wait if job limit has been reached or worker is stopping if (($self->{max} <= keys %$jobs) || $self->{finished}) { sleep 1 } # Try to get more jobs elsif (my $job = $worker->dequeue(5 => {queues => $self->{queues}})) { $jobs->{my $id = $job->id} = $job->start; my ($pid, $task) = ($job->pid, $job->task); $self->app->log->debug( qq{Performing job "$id" with task "$task" in process $pid}); } } 1; =encoding utf8 =head1 NAME Minion::Command::minion::worker - Minion worker command =head1 SYNOPSIS Usage: APPLICATION minion worker [OPTIONS] ./myapp.pl minion worker ./myapp.pl minion worker -m production -I 15 -C 5 -R 3600 -j 10 ./myapp.pl minion worker -q important -q default Options: -C, --command-interval Worker remote control command interval, defaults to 10 -h, --help Show this summary of available options --home Path to home directory of your application, defaults to the value of MOJO_HOME or auto-detection -I, --heartbeat-interval Heartbeat interval, defaults to 60 -j, --jobs Number of jobs to perform concurrently, defaults to 4 -m, --mode Operating mode for your application, defaults to the value of MOJO_MODE/PLACK_ENV or "development" -q, --queue One or more queues to get jobs from, defaults to "default" -R, --repair-interval Repair interval, defaults to 21600 (6 hours) =head1 DESCRIPTION L starts a L worker. You can have as many workers as you like. =head1 SIGNALS The L process can be controlled at runtime with the following signals. =head2 INT, TERM Stop gracefully after finishing the current jobs. =head2 QUIT Stop immediately without finishing the current jobs. =head1 REMOTE CONTROL COMMANDS The L process can be controlled at runtime through L, from anywhere in the network, by broadcasting the following remote control commands. =head2 jobs $ ./myapp.pl minion job -b jobs -a '[10]' $ ./myapp.pl minion job -b jobs -a '[10]' 23 Instruct one or more workers to change the number of jobs to perform concurrently. Setting this value to C<0> will effectively pause the worker. That means all current jobs will be finished, but no new ones accepted, until the number is increased again. =head2 stop $ ./myapp.pl minion job -b stop -a '[10025]' $ ./myapp.pl minion job -b stop -a '[10025]' 23 Instruct one or more workers to stop a job that is currently being performed immediately. This command will be ignored by workers that do not have a job matching the id. That means it is safe to broadcast this command to all workers. =head1 ATTRIBUTES L inherits all attributes from L and implements the following new ones. =head2 description my $description = $worker->description; $worker = $worker->description('Foo'); Short description of this command, used for the command list. =head2 usage my $usage = $worker->usage; $worker = $worker->usage('Foo'); Usage information for this command, used for the help screen. =head1 METHODS L inherits all methods from L and implements the following new ones. =head2 run $worker->run(@ARGV); Run this command. =head1 SEE ALSO L, L, L. =cut Minion-6.02/lib/Minion/Backend/Pg.pm000644 000765 000024 00000046602 13032221713 017066 0ustar00sristaff000000 000000 package Minion::Backend::Pg; use Mojo::Base 'Minion::Backend'; use Carp 'croak'; use Mojo::IOLoop; use Mojo::Pg 2.18; use Sys::Hostname 'hostname'; has 'pg'; sub broadcast { my ($self, $command, $args, $ids) = (shift, shift, shift || [], shift || []); return !!$self->pg->db->query( q{update minion_workers set inbox = inbox || $1::jsonb where (id = any($2) or $2 = '{}')}, {json => [[$command, @$args]]}, $ids )->rows; } sub dequeue { my ($self, $id, $wait, $options) = @_; if ((my $job = $self->_try($id, $options))) { return $job } return undef if Mojo::IOLoop->is_running; my $db = $self->pg->db; $db->listen('minion.job')->on(notification => sub { Mojo::IOLoop->stop }); my $timer = Mojo::IOLoop->timer($wait => sub { Mojo::IOLoop->stop }); Mojo::IOLoop->start; $db->unlisten('*') and Mojo::IOLoop->remove($timer); undef $db; return $self->_try($id, $options); } sub enqueue { my ($self, $task, $args, $options) = (shift, shift, shift || [], shift || {}); my $db = $self->pg->db; return $db->query( "insert into minion_jobs (args, attempts, delayed, parents, priority, queue, task) values (?, ?, (now() + (interval '1 second' * ?)), ?, ?, ?, ?) returning id", {json => $args}, $options->{attempts} // 1, $options->{delay} // 0, $options->{parents} || [], $options->{priority} // 0, $options->{queue} // 'default', $task )->hash->{id}; } sub fail_job { shift->_update(1, @_) } sub finish_job { shift->_update(0, @_) } sub job_info { shift->pg->db->query( 'select id, args, attempts, array(select id from minion_jobs where parents @> ARRAY[j.id]) as children, extract(epoch from created) as created, extract(epoch from delayed) as delayed, extract(epoch from finished) as finished, parents, priority, queue, result, extract(epoch from retried) as retried, retries, extract(epoch from started) as started, state, task, worker from minion_jobs as j where id = ?', shift )->expand->hash; } sub list_jobs { my ($self, $offset, $limit, $options) = @_; return $self->pg->db->query( 'select id from minion_jobs where (queue = $1 or $1 is null) and (state = $2 or $2 is null) and (task = $3 or $3 is null) order by id desc limit $4 offset $5', @$options{qw(queue state task)}, $limit, $offset )->arrays->map(sub { $self->job_info($_->[0]) })->to_array; } sub list_workers { my ($self, $offset, $limit) = @_; my $sql = 'select id from minion_workers order by id desc limit ? offset ?'; return $self->pg->db->query($sql, $limit, $offset) ->arrays->map(sub { $self->worker_info($_->[0]) })->to_array; } sub new { my $self = shift->SUPER::new(pg => Mojo::Pg->new(@_)); croak 'PostgreSQL 9.5 or later is required' if Mojo::Pg->new(@_)->db->dbh->{pg_server_version} < 90500; my $pg = $self->pg->auto_migrate(1)->max_connections(1); $pg->migrations->name('minion')->from_data; return $self; } sub receive { my $array = shift->pg->db->query( "update minion_workers as new set inbox = '[]' from (select id, inbox from minion_workers where id = ? for update) as old where new.id = old.id and old.inbox != '[]' returning old.inbox", shift )->expand->array; return $array ? $array->[0] : []; } sub register_worker { my ($self, $id) = @_; return shift->pg->db->query( "insert into minion_workers (id, host, pid) values (coalesce(?, nextval('minion_workers_id_seq')), ?, ?) on conflict(id) do update set notified = now() returning id", $id, $self->{host} //= hostname, $$ )->hash->{id}; } sub remove_job { !!shift->pg->db->query( "delete from minion_jobs where id = ? and state in ('inactive', 'failed', 'finished') returning 1", shift )->rows; } sub repair { my $self = shift; # Workers without heartbeat my $db = $self->pg->db; my $minion = $self->minion; $db->query( "delete from minion_workers where notified < now() - interval '1 second' * ?", $minion->missing_after ); # Jobs with missing worker (can be retried) my $fail = $db->query( "select id, retries from minion_jobs as j where state = 'active' and not exists (select 1 from minion_workers where id = j.worker)" )->hashes; $fail->each(sub { $self->fail_job(@$_{qw(id retries)}, 'Worker went away') }); # Jobs with missing parents (can't be retried) $db->query( "update minion_jobs as j set finished = now(), result = to_json('Parent went away'::text), state = 'failed' where parents <> '{}' and cardinality(parents) <> ( select count(*) from minion_jobs where j.parents @> ARRAY[id] ) and state = 'inactive'" ); # Old jobs with no unresolved dependencies $db->query( "delete from minion_jobs as j where finished <= now() - interval '1 second' * ? and not exists ( select 1 from minion_jobs where parents @> ARRAY[j.id] and state <> 'finished' ) and state = 'finished'", $minion->remove_after ); } sub reset { shift->pg->db->query('truncate minion_jobs, minion_workers restart identity'); } sub retry_job { my ($self, $id, $retries, $options) = (shift, shift, shift, shift || {}); return !!$self->pg->db->query( "update minion_jobs set delayed = (now() + (interval '1 second' * ?)), priority = coalesce(?, priority), queue = coalesce(?, queue), retried = now(), retries = retries + 1, state = 'inactive' where id = ? and retries = ? and state in ('inactive', 'failed', 'finished') returning 1", $options->{delay} // 0, @$options{qw(priority queue)}, $id, $retries )->rows; } sub stats { my $self = shift; my $stats = $self->pg->db->query( "select 'enqueued_jobs', case when is_called then last_value else 0 end from minion_jobs_id_seq union all select state::text || '_jobs', count(*) from minion_jobs group by state union all select 'delayed_jobs', count(*) from minion_jobs where (delayed > now() or parents <> '{}') and state = 'inactive' union all select 'inactive_workers', count(*) from minion_workers union all select 'active_workers', count(distinct worker) from minion_jobs where state = 'active'" )->arrays->reduce(sub { $a->{$b->[0]} = $b->[1]; $a }, {}); $stats->{inactive_workers} -= $stats->{active_workers}; $stats->{"${_}_jobs"} ||= 0 for qw(inactive active failed finished); return $stats; } sub unregister_worker { shift->pg->db->query('delete from minion_workers where id = ?', shift); } sub worker_info { shift->pg->db->query( "select id, extract(epoch from notified) as notified, array( select id from minion_jobs where state = 'active' and worker = minion_workers.id ) as jobs, host, pid, extract(epoch from started) as started from minion_workers where id = ?", shift )->hash; } sub _try { my ($self, $id, $options) = @_; return $self->pg->db->query( "update minion_jobs set started = now(), state = 'active', worker = ? where id = ( select id from minion_jobs as j where delayed <= now() and (parents = '{}' or cardinality(parents) = ( select count(*) from minion_jobs where j.parents @> ARRAY[id] and state = 'finished' )) and queue = any(?) and state = 'inactive' and task = any(?) order by priority desc, id limit 1 for update skip locked ) returning id, args, retries, task", $id, $options->{queues} || ['default'], [keys %{$self->minion->tasks}] )->expand->hash; } sub _update { my ($self, $fail, $id, $retries, $result) = @_; return undef unless my $row = $self->pg->db->query( "update minion_jobs set finished = now(), result = ?, state = ? where id = ? and retries = ? and state = 'active' returning attempts", {json => $result}, $fail ? 'failed' : 'finished', $id, $retries )->array; return 1 if !$fail || (my $attempts = $row->[0]) == 1; return 1 if $retries >= ($attempts - 1); my $delay = $self->minion->backoff->($retries); return $self->retry_job($id, $retries, {delay => $delay}); } 1; =encoding utf8 =head1 NAME Minion::Backend::Pg - PostgreSQL backend =head1 SYNOPSIS use Minion::Backend::Pg; my $backend = Minion::Backend::Pg->new('postgresql://postgres@/test'); =head1 DESCRIPTION L is a backend for L based on L. All necessary tables will be created automatically with a set of migrations named C. Note that this backend uses many bleeding edge features, so only the latest, stable version of PostgreSQL is fully supported. =head1 ATTRIBUTES L inherits all attributes from L and implements the following new ones. =head2 pg my $pg = $backend->pg; $backend = $backend->pg(Mojo::Pg->new); L object used to store all data. =head1 METHODS L inherits all methods from L and implements the following new ones. =head2 broadcast my $bool = $backend->broadcast('some_command'); my $bool = $backend->broadcast('some_command', [@args]); my $bool = $backend->broadcast('some_command', [@args], [$id1, $id2, $id3]); Broadcast remote control command to one or more workers. =head2 dequeue my $job_info = $backend->dequeue($worker_id, 0.5); my $job_info = $backend->dequeue($worker_id, 0.5, {queues => ['important']}); Wait a given amount of time in seconds for a job, dequeue it and transition from C to C state, or return C if queues were empty. These options are currently available: =over 2 =item queues queues => ['important'] One or more queues to dequeue jobs from, defaults to C. =back These fields are currently available: =over 2 =item args args => ['foo', 'bar'] Job arguments. =item id id => '10023' Job ID. =item retries retries => 3 Number of times job has been retried. =item task task => 'foo' Task name. =back =head2 enqueue my $job_id = $backend->enqueue('foo'); my $job_id = $backend->enqueue(foo => [@args]); my $job_id = $backend->enqueue(foo => [@args] => {priority => 1}); Enqueue a new job with C state. These options are currently available: =over 2 =item attempts attempts => 25 Number of times performing this job will be attempted, with a delay based on L after the first attempt, defaults to C<1>. =item delay delay => 10 Delay job for this many seconds (from now). =item parents parents => [$id1, $id2, $id3] One or more existing jobs this job depends on, and that need to have transitioned to the state C before it can be processed. =item priority priority => 5 Job priority, defaults to C<0>. Jobs with a higher priority get performed first. =item queue queue => 'important' Queue to put job in, defaults to C. =back =head2 fail_job my $bool = $backend->fail_job($job_id, $retries); my $bool = $backend->fail_job($job_id, $retries, 'Something went wrong!'); my $bool = $backend->fail_job( $job_id, $retries, {whatever => 'Something went wrong!'}); Transition from C to C state, and if there are attempts remaining, transition back to C with a delay based on L. =head2 finish_job my $bool = $backend->finish_job($job_id, $retries); my $bool = $backend->finish_job($job_id, $retries, 'All went well!'); my $bool = $backend->finish_job( $job_id, $retries, {whatever => 'All went well!'}); Transition from C to C state. =head2 job_info my $job_info = $backend->job_info($job_id); Get information about a job, or return C if job does not exist. # Check job state my $state = $backend->job_info($job_id)->{state}; # Get job result my $result = $backend->job_info($job_id)->{result}; These fields are currently available: =over 2 =item args args => ['foo', 'bar'] Job arguments. =item attempts attempts => 25 Number of times performing this job will be attempted. =item children children => ['10026', '10027', '10028'] Jobs depending on this job. =item created created => 784111777 Epoch time job was created. =item delayed delayed => 784111777 Epoch time job was delayed to. =item finished finished => 784111777 Epoch time job was finished. =item parents parents => ['10023', '10024', '10025'] Jobs this job depends on. =item priority priority => 3 Job priority. =item queue queue => 'important' Queue name. =item result result => 'All went well!' Job result. =item retried retried => 784111777 Epoch time job has been retried. =item retries retries => 3 Number of times job has been retried. =item started started => 784111777 Epoch time job was started. =item state state => 'inactive' Current job state, usually C, C, C or C. =item task task => 'foo' Task name. =item worker worker => '154' Id of worker that is processing the job. =back =head2 list_jobs my $batch = $backend->list_jobs($offset, $limit); my $batch = $backend->list_jobs($offset, $limit, {state => 'inactive'}); Returns the same information as L but in batches. These options are currently available: =over 2 =item queue queue => 'important' List only jobs in this queue. =item state state => 'inactive' List only jobs in this state. =item task task => 'test' List only jobs for this task. =back =head2 list_workers my $batch = $backend->list_workers($offset, $limit); Returns the same information as L but in batches. =head2 new my $backend = Minion::Backend::Pg->new('postgresql://postgres@/test'); Construct a new L object. =head2 receive my $commands = $backend->receive($worker_id); Receive remote control commands for worker. =head2 register_worker my $worker_id = $backend->register_worker; my $worker_id = $backend->register_worker($worker_id); Register worker or send heartbeat to show that this worker is still alive. =head2 remove_job my $bool = $backend->remove_job($job_id); Remove C, C or C job from queue. =head2 repair $backend->repair; Repair worker registry and job queue if necessary. =head2 reset $backend->reset; Reset job queue. =head2 retry_job my $bool = $backend->retry_job($job_id, $retries); my $bool = $backend->retry_job($job_id, $retries, {delay => 10}); Transition from C or C state back to C, already C jobs may also be retried to change options. These options are currently available: =over 2 =item delay delay => 10 Delay job for this many seconds (from now). =item priority priority => 5 Job priority. =item queue queue => 'important' Queue to put job in. =back =head2 stats my $stats = $backend->stats; Get statistics for jobs and workers. These fields are currently available: =over 2 =item active_jobs active_jobs => 100 Number of jobs in C state. =item active_workers active_workers => 100 Number of workers that are currently processing a job. =item delayed_jobs delayed_jobs => 100 Number of jobs in C state that are scheduled to run at specific time in the future or have unresolved dependencies. Note that this field is EXPERIMENTAL and might change without warning! =item enqueued_jobs enqueued_jobs => 100000 Rough estimate of how many jobs have ever been enqueued. Note that this field is EXPERIMENTAL and might change without warning! =item failed_jobs failed_jobs => 100 Number of jobs in C state. =item finished_jobs finished_jobs => 100 Number of jobs in C state. =item inactive_jobs inactive_jobs => 100 Number of jobs in C state. =item inactive_workers inactive_workers => 100 Number of workers that are currently not processing a job. =back =head2 unregister_worker $backend->unregister_worker($worker_id); Unregister worker. =head2 worker_info my $worker_info = $backend->worker_info($worker_id); Get information about a worker, or return C if worker does not exist. # Check worker host my $host = $backend->worker_info($worker_id)->{host}; These fields are currently available: =over 2 =item host host => 'localhost' Worker host. =item jobs jobs => ['10023', '10024', '10025', '10029'] Ids of jobs the worker is currently processing. =item notified notified => 784111777 Epoch time worker sent the last heartbeat. =item pid pid => 12345 Process id of worker. =item started started => 784111777 Epoch time worker was started. =back =head1 SEE ALSO L, L, L. =cut __DATA__ @@ minion -- 1 up create table if not exists minion_jobs ( id bigserial not null primary key, args json not null, created timestamp with time zone not null, delayed timestamp with time zone not null, finished timestamp with time zone, priority int not null, result json, retried timestamp with time zone, retries int not null, started timestamp with time zone, state text not null, task text not null, worker bigint ); create table if not exists minion_workers ( id bigserial not null primary key, host text not null, pid int not null, started timestamp with time zone not null ); -- 1 down drop table if exists minion_jobs; drop table if exists minion_workers; -- 2 up alter table minion_jobs alter column created set default now(); alter table minion_jobs alter column state set default 'inactive'; alter table minion_jobs alter column retries set default 0; alter table minion_workers add column notified timestamp with time zone not null default now(); alter table minion_workers alter column started set default now(); -- 4 up alter table minion_jobs add column queue text not null default 'default'; -- 5 up alter table minion_jobs add column attempts int not null default 1; -- 7 up create type minion_state as enum ('inactive', 'active', 'failed', 'finished'); alter table minion_jobs alter column state set default 'inactive'::minion_state; alter table minion_jobs alter column state type minion_state using state::minion_state; alter table minion_jobs alter column args type jsonb using args::jsonb; alter table minion_jobs alter column result type jsonb using result::jsonb; -- 7 down alter table minion_jobs alter column state type text using state; alter table minion_jobs alter column state set default 'inactive'; drop type if exists minion_state; -- 8 up alter table minion_jobs add constraint args check(jsonb_typeof(args) = 'array'); -- 9 up create or replace function minion_jobs_notify_workers() returns trigger as $$ begin if new.delayed <= now() then notify "minion.job"; end if; return null; end; $$ language plpgsql; set client_min_messages to warning; drop trigger if exists minion_jobs_insert_trigger on minion_jobs; drop trigger if exists minion_jobs_notify_workers_trigger on minion_jobs; set client_min_messages to notice; create trigger minion_jobs_notify_workers_trigger after insert or update of retries on minion_jobs for each row execute procedure minion_jobs_notify_workers(); -- 9 down drop trigger if exists minion_jobs_notify_workers_trigger on minion_jobs; drop function if exists minion_jobs_notify_workers(); -- 10 up alter table minion_jobs add column parents bigint[] default '{}'; -- 11 up create index on minion_jobs (state, priority desc, id); -- 12 up alter table minion_workers add column inbox jsonb check(jsonb_typeof(inbox) = 'array') default '[]'; -- 13 up create index on minion_jobs using gin (parents); Minion-6.02/examples/minion_bench.pl000644 000765 000024 00000004412 13003212143 017423 0ustar00sristaff000000 000000 use Mojo::Base -strict; use Minion; use Time::HiRes 'time'; my $ENQUEUE = 10000; my $DEQUEUE = 1000; my $REPETITIONS = 2; my $WORKERS = 4; my $STATS = 100; my $REPAIR = 100; my $INFO = 100; # A benchmark script for comparing backends and evaluating the performance # impact of proposed changes my $minion = Minion->new(Pg => 'postgresql://tester:testing@/test'); $minion->add_task(foo => sub { }); $minion->add_task(bar => sub { }); $minion->reset; # Enqueue say "Clean start with $ENQUEUE jobs"; my $before = time; $minion->enqueue($_ % 2 ? 'foo' : 'bar') for 1 .. $ENQUEUE; my $elapsed = time - $before; my $avg = sprintf '%.3f', $ENQUEUE / $elapsed; say "Enqueued $ENQUEUE jobs in $elapsed seconds ($avg/s)"; # Dequeue sub dequeue { my @pids; for (1 .. $WORKERS) { die "Couldn't fork: $!" unless defined(my $pid = fork); unless ($pid) { my $worker = $minion->worker->register; say "$$ will finish $DEQUEUE jobs"; my $before = time; $worker->dequeue(0.5)->finish for 1 .. $DEQUEUE; my $elapsed = time - $before; my $avg = sprintf '%.3f', $DEQUEUE / $elapsed; say "$$ finished $DEQUEUE jobs in $elapsed seconds ($avg/s)"; $worker->unregister; exit; } push @pids, $pid; } say "$$ has started $WORKERS workers"; my $before = time; waitpid $_, 0 for @pids; my $elapsed = time - $before; my $avg = sprintf '%.3f', ($DEQUEUE * $WORKERS) / $elapsed; say "$WORKERS workers finished $DEQUEUE jobs each in $elapsed seconds ($avg/s)"; } dequeue() for 1 .. $REPETITIONS; # Stats say "Requesting stats $STATS times"; $before = time; $minion->stats for 1 .. $STATS; $elapsed = time - $before; $avg = sprintf '%.3f', $STATS / $elapsed; say "Received stats $STATS times in $elapsed seconds ($avg/s)"; # Repair say "Repairing $REPAIR times"; $before = time; $minion->repair for 1 .. $REPAIR; $elapsed = time - $before; $avg = sprintf '%.3f', $REPAIR / $elapsed; say "Repaired $REPAIR times in $elapsed seconds ($avg/s)"; # Job info say "Requesting job info $INFO times"; $before = time; my $backend = $minion->backend; $backend->job_info($ENQUEUE) for 1 .. $INFO; $elapsed = time - $before; $avg = sprintf '%.3f', $INFO / $elapsed; say "Received job info $INFO times in $elapsed seconds ($avg/s)";