pax_global_header00006660000000000000000000000064130270015400014502gustar00rootroot0000000000000052 comment=57cb667cf975bececbe093e636cdee1aa37352ab curator-4.2.5/000077500000000000000000000000001302700154000131715ustar00rootroot00000000000000curator-4.2.5/.gitignore000066400000000000000000000004671302700154000151700ustar00rootroot00000000000000.*.swp *~ *.py[co] build dist scratch docs/_build *.egg .eggs elasticsearch_curator.egg-info elasticsearch_curator_dev.egg-info .coverage .idea coverage.xml nosetests.xml index.html docs/asciidoc/html_docs wheelhouse Elastic.ico Vagrant/centos/6/.vagrant Vagrant/centos/7/.vagrant Vagrant/ubuntu/14.04/.vagrant curator-4.2.5/.travis.yml000066400000000000000000000010351302700154000153010ustar00rootroot00000000000000language: python python: - "2.7" - "3.4" - "3.5" env: - ES_VERSION=2.0.0 - ES_VERSION=2.1.1 - ES_VERSION=2.2.2 - ES_VERSION=2.3.5 - ES_VERSION=2.4.2 - ES_VERSION=5.0.2 - ES_VERSION=5.1.1 os: linux cache: pip: true jdk: - oraclejdk8 install: - pip install -r requirements.txt - pip install . script: - sudo apt-get update && sudo apt-get install oracle-java8-installer - java -version - sudo update-alternatives --set java /usr/lib/jvm/java-8-oracle/jre/bin/java - java -version - ./travis-run.sh curator-4.2.5/CONTRIBUTING.md000066400000000000000000000073161302700154000154310ustar00rootroot00000000000000# Contributing to Curator All contributions are welcome: ideas, patches, documentation, bug reports, complaints, etc! Programming is not a required skill, and there are many ways to help out! It is more important to us that you are able to contribute. That said, some basic guidelines, which you are free to ignore :) ## Want to learn? Want to write your own code to do something Curator doesn't do out of the box? * [Curator API Documentation](http://curator.readthedocs.io/) Since version 2.0, Curator ships with both an API and wrapper scripts (which are actually defined as entry points). This allows you to write your own scripts to accomplish similar goals, or even new and different things with the [Curator API](http://curator.readthedocs.io/), and the [Elasticsearch Python API](http://elasticsearch-py.readthedocs.io/). Want to know how to use the command-line interface (CLI)? * [Curator CLI Documentation](http://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html) The Curator CLI Documentation is now a part of the document repository at http://elastic.co/guide at http://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html Want to lurk about and see what others are doing with Curator? * The irc channels (#logstash and #elasticsearch on irc.freenode.org) are good places for this ## Got Questions? Have a problem you want Curator to solve for you? * You are welcome to join the IRC channel #logstash (or #elasticsearch) on irc.freenode.org and ask for help there! ## Have an Idea or Feature Request? * File a ticket on [github](https://github.com/elastic/curator/issues) ## Something Not Working? Found a Bug? If you think you found a bug, it probably is a bug. * File it on [github](https://github.com/elastic/curator/issues) # Contributing Documentation and Code Changes If you have a bugfix or new feature that you would like to contribute to Curator, and you think it will take more than a few minutes to produce the fix (ie; write code), it is worth discussing the change with the Curator users and developers first! You can reach us via [github](https://github.com/elastic/curator/issues), or via IRC (#logstash or #elasticsearch on freenode irc) Documentation is in two parts: API and CLI documentation. API documentation is generated from comments inside the classes and methods within the code. This documentation is rendered and hosted at http://curator.readthedocs.io CLI documentation is in Asciidoc format in the GitHub repository at https://github.com/elastic/curator/tree/master/docs/asciidoc. This documentation can be changed via a pull request as with any other code change. ## Contribution Steps 1. Test your changes! Run the test suite ('python setup.py test'). Please note that this requires an Elasticsearch instance. The tests will try to connect to your local elasticsearch instance and run integration tests against it. **This will delete all the data stored there!** You can use the env variable `TEST_ES_SERVER` to point to a different instance (for example 'otherhost:9203'). 2. Please make sure you have signed our [Contributor License Agreement](http://www.elastic.co/contributor-agreement/). We are not asking you to assign copyright to us, but to give us the right to distribute your code without restriction. We ask this of all contributors in order to assure our users of the origin and continuing existence of the code. You only need to sign the CLA once. 3. Send a pull request! Push your changes to your fork of the repository and [submit a pull request](https://help.github.com/articles/using-pull-requests). In the pull request, describe what your changes do and mention any bugs/issues related to the pull request. curator-4.2.5/CONTRIBUTORS000066400000000000000000000041131302700154000150500ustar00rootroot00000000000000The following is a list of people who have contributed ideas, code, bug reports, or in general have helped curator along its way. Contributors: * Jordan Sissel (jordansissel) (For Logstash, first and foremost) * Shay Banon (kimchy) (For Elasticsearch, of course!) * Aaron Mildenstein (untergeek) * Njal Karevoll * François Deppierraz * Honza Kral (HonzaKral) * Benjamin Smith (benjaminws) * Colin Moller (LeftyBC) * Elliot (edgeofnite) * Ram Viswanadha (ramv) * Chris Meisinger (cmeisinger) * Stuart Warren (stuart-warren) * (gitshaw) * (sfritz) * (sjoelsam) * Jose Diaz-Gonzalez (josegonzalez) * Arie Bro (arieb) * David Harrigan (dharrigan) * Mathieu Geli (gelim) * Nick Ethier (nickethier) * Mohab Usama (mohabusama) * (gitshaw) * Stuart Warren (stuart-warren) * Xavier Calland (xavier-calland) * Chad Schellenger (cschellenger) * Kamil Essekkat (ekamil) * (gbutt) * Ben Buchacher (bbuchacher) * Ehtesh Choudhury (shurane) * Markus Fischer (mfn) * Fabien Wernli (faxm0dem) * Michael Weiser (michaelweiser) * (digital-wonderland) * cassiano (cassianoleal) * Matt Dainty (bodgit) * Alex Philipp (alex-sf) * (krzaczek) * Justin Lintz (jlintz) * Jeremy Falling (jjfalling) * Ian Babrou (bobrik) * Ferenc Erki (ferki) * George Heppner (gheppner) * Matt Hughes (matthughes) * Brian Lalor (blalor) * Paweł Krzaczkowski (krzaczek) * Ben Tse (bt5e) * Tom Hendrikx (whyscream) * Christian Vozar (christianvozar) * Magnus Baeck (magnusbaeck) * Robin Kearney (rk295) * (cfeio) * (malagoli) * Dan Sheridan (djs52) * Michael-Keith Bernard (SegFaultAX) * Simon Lundström (simmel) * (pkr1234) * Mark Feltner (feltnerm) * William Jimenez (wjimenez5271) * Jeremy Canady (jrmycanady) * Steven Ottenhoff (steffo) * Ole Rößner (Basster) * Jack (univerio) * Tomáš Mózes (hydrapolic) * Gary Gao (garyelephant) * Panagiotis Moustafellos (pmoust) * (pbamba) * Pavel Strashkin (xaka) * Wadim Kruse (wkruse) * Richard Megginson (richm) * Thibaut Ackermann (thib-ack) * (zzugg) * Julien Mancuso (petitout) * Spencer Herzberg (sherzberg) * Luke Waite (lukewaite) * (dtrv) * Christopher "Chief" Najewicz (chiefy) * Filipe Gonçalves (basex) curator-4.2.5/Changelog.rst000077700000000000000000000000001302700154000211562docs/Changelog.rstustar00rootroot00000000000000curator-4.2.5/Dockerfile000066400000000000000000000003271302700154000151650ustar00rootroot00000000000000# Docker Definition for ElasticSearch Curator FROM python:2.7.8-slim MAINTAINER Christian R. Vozar RUN pip install --quiet elasticsearch-curator ENTRYPOINT [ "/usr/local/bin/curator" ] curator-4.2.5/LICENSE.txt000066400000000000000000000011261302700154000150140ustar00rootroot00000000000000Copyright 2011–2016 Elasticsearch and contributors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. curator-4.2.5/MANIFEST.in000066400000000000000000000007441302700154000147340ustar00rootroot00000000000000include Changelog.rst include CONTRIBUTORS CONTRIBUTING.md include LICENSE.txt include README.rst include Dockerfile recursive-exclude * __pycache__ recursive-exclude * *.py[co] recursive-include curator * recursive-include test * recursive-include docs * recursive-exclude curator *.pyc recursive-exclude curator *.pyo recursive-exclude docs *.pyc recursive-exclude docs *.pyo recursive-exclude test *.pyc recursive-exclude test *.pyo prune docs/_build prune docs/asciidoc/html_docs curator-4.2.5/NOTICE000066400000000000000000000103641302700154000141010ustar00rootroot00000000000000In accordance with section 4d of the Apache 2.0 license (http://www.apache.org/licenses/LICENSE-2.0), this NOTICE file is included. All users mentioned in the CONTRIBUTORS file at https://github.com/elastic/curator/blob/master/CONTRIBUTORS must be included in any derivative work. All conditions of section 4 of the Apache 2.0 license will be enforced: 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: a. You must give any other recipients of the Work or Derivative Works a copy of this License; and b. You must cause any modified files to carry prominent notices stating that You changed the files; and c. You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and d. If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. Contributors: * Jordan Sissel (jordansissel) (For Logstash, first and foremost) * Shay Banon (kimchy) (For Elasticsearch, of course!) * Aaron Mildenstein (untergeek) * Njal Karevoll * François Deppierraz * Honza Kral (HonzaKral) * Benjamin Smith (benjaminws) * Colin Moller (LeftyBC) * Elliot (edgeofnite) * Ram Viswanadha (ramv) * Chris Meisinger (cmeisinger) * Stuart Warren (stuart-warren) * (gitshaw) * (sfritz) * (sjoelsam) * Jose Diaz-Gonzalez (josegonzalez) * Arie Bro (arieb) * David Harrigan (dharrigan) * Mathieu Geli (gelim) * Nick Ethier (nickethier) * Mohab Usama (mohabusama) * (gitshaw) * Stuart Warren (stuart-warren) * Xavier Calland (xavier-calland) * Chad Schellenger (cschellenger) * Kamil Essekkat (ekamil) * (gbutt) * Ben Buchacher (bbuchacher) * Ehtesh Choudhury (shurane) * Markus Fischer (mfn) * Fabien Wernli (faxm0dem) * Michael Weiser (michaelweiser) * (digital-wonderland) * cassiano (cassianoleal) * Matt Dainty (bodgit) * Alex Philipp (alex-sf) * (krzaczek) * Justin Lintz (jlintz) * Jeremy Falling (jjfalling) * Ian Babrou (bobrik) * Ferenc Erki (ferki) * George Heppner (gheppner) * Matt Hughes (matthughes) * Brian Lalor (blalor) * Paweł Krzaczkowski (krzaczek) * Ben Tse (bt5e) * Tom Hendrikx (whyscream) * Christian Vozar (christianvozar) * Magnus Baeck (magnusbaeck) * Robin Kearney (rk295) * (cfeio) * (malagoli) * Dan Sheridan (djs52) * Michael-Keith Bernard (SegFaultAX) * Simon Lundström (simmel) * (pkr1234) * Mark Feltner (feltnerm) * William Jimenez (wjimenez5271) * Jeremy Canady (jrmycanady) * Steven Ottenhoff (steffo) * Ole Rößner (Basster) * Jack (univerio) * Tomáš Mózes (hydrapolic) * Gary Gao (garyelephant) * Panagiotis Moustafellos (pmoust) * (pbamba) * Pavel Strashkin (xaka) * Wadim Kruse (wkruse) * Richard Megginson (richm) * Thibaut Ackermann (thib-ack) * (zzugg) * Julien Mancuso (petitout) curator-4.2.5/README.rst000066400000000000000000000232641302700154000146670ustar00rootroot00000000000000.. _readme: Curator ======= Have indices in Elasticsearch? This is the tool for you! Like a museum curator manages the exhibits and collections on display, Elasticsearch Curator helps you curate, or manage your indices. Compatibility Matrix ==================== +--------+----------+------------+----------+------------+----------+ |Version | ES 1.x | AWS ES 1.x | ES 2.x | AWS ES 2.x | ES 5.x | +========+==========+============+==========+============+==========+ | 3 | yes | yes* | yes | yes* | no | +--------+----------+------------+----------+------------+----------+ | 4 | no | no | yes | no | yes | +--------+----------+------------+----------+------------+----------+ It is important to note that Curator 4 will not work with indices created in versions of Elasticsearch older than 1.4 (if they have been subsequently re-indexed, they will work). This is because those older indices lack index metadata that Curator 4 requires. Curator 4 will simply exclude any such indices from being acted on, and you will get a warning message like the following: :: 2016-07-31 10:36:17,423 WARNING Index: YOUR_INDEX_NAME has no "creation_date"! This implies that the index predates Elasticsearch v1.4. For safety, this index will be removed from the actionable list. It is also important to note that Curator 4 requires access to the ``/_cluster/state/metadata`` endpoint. Forks of Elasticsearch which do not support this endpoint (such as AWS ES, see #717) *will not* be able to use Curator version 4. \* It appears that AWS ES `does not allow access to the snapshot status endpoint`_ for either 1.x or 2.x versions. This prevents Curator 3 from being used to make snapshots. .. _does not allow access to the snapshot status endpoint: https://github.com/elastic/curator/issues/796 Build Status ------------ +--------+----------+ | Branch | Status | +========+==========+ | Master | |master| | +--------+----------+ | 4.x | |4_x| | +--------+----------+ | 4.0 | |4_0| | +--------+----------+ | 4.1 | |4_1| | +--------+----------+ | 4.2 | |4_2| | +--------+----------+ PyPI: |pypi_pkg| .. |master| image:: https://travis-ci.org/elastic/curator.svg?branch=master :target: https://travis-ci.org/elastic/curator .. |4_x| image:: https://travis-ci.org/elastic/curator.svg?branch=4.x :target: https://travis-ci.org/elastic/curator .. |4_0| image:: https://travis-ci.org/elastic/curator.svg?branch=4.0 :target: https://travis-ci.org/elastic/curator .. |4_1| image:: https://travis-ci.org/elastic/curator.svg?branch=4.1 :target: https://travis-ci.org/elastic/curator .. |4_2| image:: https://travis-ci.org/elastic/curator.svg?branch=4.2 :target: https://travis-ci.org/elastic/curator .. |pypi_pkg| image:: https://badge.fury.io/py/elasticsearch-curator.svg :target: https://badge.fury.io/py/elasticsearch-curator `Curator API Documentation`_ ---------------------------- Version 4 of Curator ships with both an API and a wrapper script (which is actually defined as an entry point). The API allows you to write your own scripts to accomplish similar goals, or even new and different things with the `Curator API`_, and the `Elasticsearch Python API`_. .. _Curator API: http://curator.readthedocs.io/ .. _Curator API Documentation: `Curator API`_ .. _Elasticsearch Python API: http://elasticsearch-py.readthedocs.io/ `Curator CLI Documentation`_ ---------------------------- The `Curator CLI Documentation`_ is now a part of the document repository at http://elastic.co/guide at http://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html .. _Curator CLI Documentation: http://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html `Getting Started`_ ------------------ .. _Getting Started: https://www.elastic.co/guide/en/elasticsearch/client/curator/current/getting-started.html See the `Installation guide `_ and the `command-line usage guide `_ Running ``curator --help`` will also show usage information. `Frequently Asked Questions`_ ----------------------------- .. _Frequently Asked Questions: http://www.elastic.co/guide/en/elasticsearch/client/curator/current/faq.html Encountering issues like ``DistributionNotFound``? See the FAQ_ for that issue, and more. .. _FAQ: http://www.elastic.co/guide/en/elasticsearch/client/curator/current/entrypoint-fix.html `Documentation & Examples`_ --------------------------- .. _Documentation & Examples: http://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html The documentation for the CLI is now part of the document repository at http://elastic.co/guide at http://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html The `Curator Wiki `_ on Github is now a place to add your own examples and ideas. Contributing ------------ * fork the repo * make changes in your fork * add tests to cover your changes (if necessary) * run tests * sign the `CLA `_ * send a pull request! To run from source, use the ``run_curator.py`` script in the root directory of the project. Running Tests ------------- To run the test suite just run ``python setup.py test`` When changing code, contributing new code or fixing a bug please make sure you include tests in your PR (or mark it as without tests so that someone else can pick it up to add the tests). When fixing a bug please make sure the test actually tests the bug - it should fail without the code changes and pass after they're applied (it can still be one commit of course). The tests will try to connect to your local elasticsearch instance and run integration tests against it. This will delete all the data stored there! You can use the env variable ``TEST_ES_SERVER`` to point to a different instance (for example, 'otherhost:9203'). Binary Executables ------------------ The combination of `setuptools `_ and `cx_Freeze `_ allows for Curator to be compiled into binary packages. These consist of a binary file placed in a directory which contains all the libraries required to run it. In order to make a binary package you must manually install the ``cx_freeze`` python module. You can do this via ``pip``, or ``python setup.py install``, or by package, if such exists for your platform. In order to make it compile on recent Debian/Ubuntu platforms, a patch had to be applied to the ``setup.py`` file in the extracted folder. This patch file is in the ``unix_packages`` directory in this repository. With ``cx_freeze`` installed, building a binary package is as simple as running ``python setup.py build_exe``. In Linux distributions, the results will be in the ``build`` directory, in a subdirectory labelled ``exe.linux-x86_64-${PYVER}``, where `${PYVER}` is the current major/minor version of Python, e.g. ``2.7``. This directory can be renamed as desired. Other entry-points that are defined in the ``setup.py`` file, such as ``es_repo_mgr``, will also appear in this directory. The process is identical for building the binary package for Windows. It must be run from a Windows machine with all dependencies installed. Executables in Windows will have the ``.exe`` suffix attached. The directory in ``build`` will be named ``exe.win-amd64-${PYVER}``, where `${PYVER}` is the current major/minor version of Python, e.g. ``2.7``. This directory can be renamed as desired. In Windows, cx_Freeze also allows for building rudimentary MSI installers. This can be done by invoking ``python setup.py bdist_msi``. The MSI fill will be in the ``dist`` directory, and will be named ``elasticsearch-curator-#.#.#-amd64.msi``, where the major, minor, and patch version numbers are substituted accordingly. One drawback to this rudimentary MSI is that it does not allow updates to be installed on top of the existing installation. You must uninstall the old version before installing the newer one. The ``unix_packages`` directory contains the ``build_packages.sh`` script used to generate the packages for the Curator YUM and APT repositories. The ``Vagrant`` directory has the Vagrantfiles used in conjunction with the ``build_packages.sh`` script. If you wish to use this method on your own, you must ensure that the shared folders exist. ``/curator_packages`` is where the packages will be placed after building. ``/curator_source`` is the path to the Curator source code, so that the ``build_packages.sh`` script can be called from there. The ``build_packages.sh`` script does `not` use the local source code, but rather pulls the version specified as an argument directly from GitHub. Versioning ---------- Version 4 of Curator is the current ``master`` branch. It supports Elasticsearch versions 2.0 through 5.0. This is the first release of Curator that is not fully reverse compatible. The ``3.x`` branch will continue to be available to support earlier versions of Elasticsearch. No new development is being done with the ``3.x`` branch, but bug fixes may be merged as necessary. Origins ------- Curator was first called ``clearESindices.py`` [1] and was almost immediately renamed to ``logstash_index_cleaner.py`` [1]. After a time it was migrated under the [logstash](https://github.com/elastic/logstash) repository as ``expire_logs``. Soon thereafter, Jordan Sissel was hired by Elasticsearch, as was the original author of this tool. It became Elasticsearch Curator after that and is now hosted at [1] curator-4.2.5/Vagrant/000077500000000000000000000000001302700154000145735ustar00rootroot00000000000000curator-4.2.5/Vagrant/centos/000077500000000000000000000000001302700154000160665ustar00rootroot00000000000000curator-4.2.5/Vagrant/centos/6/000077500000000000000000000000001302700154000162335ustar00rootroot00000000000000curator-4.2.5/Vagrant/centos/6/Vagrantfile000066400000000000000000000012241302700154000204170ustar00rootroot00000000000000# -*- mode: ruby -*- # vi: set ft=ruby : Vagrant.configure(2) do |config| config.vm.box = "elastic/centos-6-x86_64" config.vm.provision "shell", inline: <<-SHELL sudo yum -y groupinstall "Development Tools" sudo yum -y install python-devel zlib-devel bzip2-devel sqlite sqlite-devel openssl-devel SHELL config.vm.synced_folder "/curator_packages", "/curator_packages", create: true, owner: "vagrant", group: "vagrant" config.vm.synced_folder "/curator_source", "/curator_source", create: true, owner: "vagrant", group: "vagrant" config.vm.provider "virtualbox" do |v| v.customize ["modifyvm", :id, "--nictype1", "virtio"] end end curator-4.2.5/Vagrant/centos/7/000077500000000000000000000000001302700154000162345ustar00rootroot00000000000000curator-4.2.5/Vagrant/centos/7/Vagrantfile000066400000000000000000000012241302700154000204200ustar00rootroot00000000000000# -*- mode: ruby -*- # vi: set ft=ruby : Vagrant.configure(2) do |config| config.vm.box = "elastic/centos-7-x86_64" config.vm.provision "shell", inline: <<-SHELL sudo yum -y groupinstall "Development Tools" sudo yum -y install python-devel zlib-devel bzip2-devel sqlite sqlite-devel openssl-devel SHELL config.vm.synced_folder "/curator_packages", "/curator_packages", create: true, owner: "vagrant", group: "vagrant" config.vm.synced_folder "/curator_source", "/curator_source", create: true, owner: "vagrant", group: "vagrant" config.vm.provider "virtualbox" do |v| v.customize ["modifyvm", :id, "--nictype1", "virtio"] end end curator-4.2.5/Vagrant/ubuntu/000077500000000000000000000000001302700154000161155ustar00rootroot00000000000000curator-4.2.5/Vagrant/ubuntu/14.04/000077500000000000000000000000001302700154000165635ustar00rootroot00000000000000curator-4.2.5/Vagrant/ubuntu/14.04/Vagrantfile000066400000000000000000000012671302700154000207560ustar00rootroot00000000000000# -*- mode: ruby -*- # vi: set ft=ruby : Vagrant.configure(2) do |config| config.vm.box = "ubuntu/trusty64" config.vm.provision "shell", inline: <<-SHELL sudo apt-get -y autoremove sudo apt-get update sudo apt-get install -y libxml2-dev zlib1g-dev pkg-config python-dev make build-essential libssl-dev libbz2-dev libsqlite3-dev SHELL config.vm.synced_folder "/curator_packages", "/curator_packages", create: true, owner: "vagrant", group: "vagrant" config.vm.synced_folder "/curator_source", "/curator_source", create: true, owner: "vagrant", group: "vagrant" config.vm.provider "virtualbox" do |v| v.customize ["modifyvm", :id, "--nictype1", "virtio"] end end curator-4.2.5/binary_release.py000066400000000000000000000073511302700154000165350ustar00rootroot00000000000000import os import re import sys import shutil import hashlib # This script simply takes the output of `python setup.py build_exe` and makes # a compressed archive (zip for windows, tar.gz for Linux) for distribution. # Utility function to read from file. def fread(fname): return open(os.path.join(os.path.dirname(__file__), fname)).read() def get_version(): VERSIONFILE="curator/_version.py" verstrline = fread(VERSIONFILE).strip() vsre = r"^__version__ = ['\"]([^'\"]*)['\"]" mo = re.search(vsre, verstrline, re.M) if mo: VERSION = mo.group(1) else: raise RuntimeError("Unable to find version string in %s." % (VERSIONFILE,)) build_number = os.environ.get('CURATOR_BUILD_NUMBER', None) if build_number: return VERSION + "b{}".format(build_number) return VERSION archive_format = 'gztar' enviro = dict(os.environ) platform = sys.platform pyver = str(sys.version_info[0]) + '.' + str(sys.version_info[1]) if platform == 'win32': # Win32 stuff archive_format = 'zip' build_name = 'exe.win-' + enviro['PROCESSOR_ARCHITECTURE'].lower() + '-' + pyver target_name = "curator-" + str(get_version()) + "-win32" elif platform == 'linux' or platform == 'linux2': sys_string = enviro['_system_type'].lower() + '-' + enviro['_system_arch'].lower() build_name = 'exe.' + sys_string + '-' + pyver target_name = "curator-" + str(get_version()) + "-" + sys_string else: # Unsupported platform? print('Your platform ({0}) is not yet supported for binary build/distribution.'.format(platform)) sys.exit(1) #sys_string = sys_type + '-' + sys_arch #build_name = 'exe.' + sys_string + '-' + pyver #print('Expected build directory: {0}'.format(build_name)) build_path = os.path.join('build', build_name) if os.path.exists(build_path): #print("I found the path: {0}".format(build_path)) target_path = os.path.join('.', target_name) # Check to see if an older directory exists... if os.path.exists(target_path): print('An older build exists at {0}. Please delete this before continuing.'.format(target_path)) sys.exit(1) else: os.rename(build_path, target_path) # Ensure the rename went smoothly, then continue if os.path.exists(target_path): #print("Build successfully renamed") if float(pyver) >= 2.7: shutil.make_archive(target_name, archive_format, '.', target_path) if platform == 'win32': fname = target_name + '.zip' else: fname = target_name + '.tar.gz' # Clean up directory if we made a viable archive. if os.path.exists(fname): shutil.rmtree(target_path) else: print('Something went wrong creating the archive {0}'.format(fname)) sys.exit(1) md5sum = hashlib.md5(open(fname, 'rb').read()).hexdigest() sha1sum = hashlib.sha1(open(fname, 'rb').read()).hexdigest() with open(fname + ".md5.txt", "w") as md5_file: md5_file.write("{0}".format(md5sum)) with open(fname + ".sha1.txt", "w") as sha1_file: sha1_file.write("{0}".format(sha1sum)) print('Archive: {0}'.format(fname)) print('{0} = {1}'.format(fname + ".md5.txt", md5sum)) print('{0} = {1}'.format(fname + ".sha1.txt", sha1sum)) else: print('Your python version ({0}) is too old to use with shutil.make_archive.'.format(pyver)) print('You can manually compress the {0} directory to achieve the same result.'.format(target_name)) else: # We couldn't find a build_path print("Build not found. Please run 'python setup.py build_exe' to create the build directory.") sys.exit(1) curator-4.2.5/curator/000077500000000000000000000000001302700154000146505ustar00rootroot00000000000000curator-4.2.5/curator/__init__.py000066400000000000000000000004051302700154000167600ustar00rootroot00000000000000from .exceptions import * from .defaults import * from .validators import * from .logtools import * from .utils import * from .indexlist import IndexList from .snapshotlist import SnapshotList from .actions import * from .cli import * from .repomgrcli import * curator-4.2.5/curator/__main__.py000066400000000000000000000001371302700154000167430ustar00rootroot00000000000000 """Executed when package directory is called as a script""" from .curator import main main() curator-4.2.5/curator/_version.py000066400000000000000000000000261302700154000170440ustar00rootroot00000000000000__version__ = '4.2.5' curator-4.2.5/curator/actions.py000066400000000000000000001240421302700154000166650ustar00rootroot00000000000000from .exceptions import * from .utils import * import logging import time from datetime import datetime class Alias(object): def __init__(self, name=None, extra_settings={}): """ Define the Alias object. :arg name: The alias name :arg extra_settings: Extra settings, including filters and routing. For more information see https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-aliases.html :type extra_settings: dict, representing the settings. """ if not name: raise MissingArgument('No value for "name" provided.') #: Instance variable #: The strftime parsed version of `name`. self.name = parse_date_pattern(name) #: The list of actions to perform. Populated by #: :mod:`curator.actions.Alias.add` and #: :mod:`curator.actions.Alias.remove` self.actions = [] #: Instance variable. #: The Elasticsearch Client object derived from `ilo` self.client = None #: Instance variable. #: Any extra things to add to the alias, like filters, or routing. self.extra_settings = extra_settings self.loggit = logging.getLogger('curator.actions.alias') def add(self, ilo): """ Create `add` statements for each index in `ilo` for `alias`, then append them to `actions`. Add any `extras` that may be there. :arg ilo: A :class:`curator.indexlist.IndexList` object """ verify_index_list(ilo) if not self.client: self.client = ilo.client ilo.empty_list_check() for index in ilo.working_list(): self.loggit.debug( 'Adding index {0} to alias {1} with extra settings ' '{2}'.format(index, self.name, self.extra_settings) ) add_dict = { 'add' : { 'index' : index, 'alias': self.name } } add_dict['add'].update(self.extra_settings) self.actions.append(add_dict) def remove(self, ilo): """ Create `remove` statements for each index in `ilo` for `alias`, then append them to `actions`. :arg ilo: A :class:`curator.indexlist.IndexList` object """ verify_index_list(ilo) if not self.client: self.client = ilo.client ilo.empty_list_check() for index in ilo.working_list(): self.loggit.debug( 'Removing index {0} from alias {1}'.format(index, self.name)) self.actions.append( { 'remove' : { 'index' : index, 'alias': self.name } }) def body(self): """ Return a `body` string suitable for use with the `update_aliases` API call. """ if not self.actions: raise ActionError('No "add" or "remove" operations') self.loggit.debug('Alias actions: {0}'.format(self.actions)) return { 'actions' : self.actions } def do_dry_run(self): """ Log what the output would be, but take no action. """ self.loggit.info('DRY-RUN MODE. No changes will be made.') for item in self.body()['actions']: job = list(item.keys())[0] index = item[job]['index'] alias = item[job]['alias'] # We want our log to look clever, so if job is "remove", strip the # 'e' so "remove" can become "removing". "adding" works already. self.loggit.info( 'DRY-RUN: alias: {0}ing index "{1}" {2} alias ' '"{3}"'.format( job.rstrip('e'), index, 'to' if job is 'add' else 'from', alias ) ) def do_action(self): """ Run the API call `update_aliases` with the results of `body()` """ self.loggit.info('Updating aliases...') self.loggit.info('Alias actions: {0}'.format(self.body())) try: self.client.indices.update_aliases(body=self.body()) except Exception as e: report_failure(e) class Allocation(object): def __init__(self, ilo, key=None, value=None, allocation_type='require', wait_for_completion=False, timeout=30, ): """ :arg ilo: A :class:`curator.indexlist.IndexList` object :arg key: An arbitrary metadata attribute key. Must match the key assigned to at least some of your nodes to have any effect. :arg value: An arbitrary metadata attribute value. Must correspond to values associated with `key` assigned to at least some of your nodes to have any effect. :arg allocation_type: Type of allocation to apply. Default is `require` :arg wait_for_completion: Wait (or not) for the operation to complete before returning. (default: `False`) :type wait_for_completion: bool :arg timeout: Number of seconds to `wait_for_completion` .. note:: See: https://www.elastic.co/guide/en/elasticsearch/reference/current/shard-allocation-filtering.html """ verify_index_list(ilo) if not key: raise MissingArgument('No value for "key" provided') if not value: raise MissingArgument('No value for "value" provided') if allocation_type not in ['require', 'include', 'exclude']: raise ValueError( '{0} is an invalid allocation_type. Must be one of "require", ' '"include", "exclude".'.format(allocation_type) ) #: Instance variable. #: Internal reference to `ilo` self.index_list = ilo #: Instance variable. #: The Elasticsearch Client object derived from `ilo` self.client = ilo.client self.loggit = logging.getLogger('curator.actions.allocation') #: Instance variable. #: Populated at instance creation time. Value is #: ``index.routing.allocation.`` `allocation_type` ``.`` `key` ``.`` `value` bkey = 'index.routing.allocation.{0}.{1}'.format(allocation_type, key) self.body = { bkey : value } #: Instance variable. #: Internal reference to `wait_for_completion` self.wfc = wait_for_completion #: Instance variable. #: How long in seconds to `wait_for_completion` before returning with an #: exception self.timeout = '{0}s'.format(timeout) def do_dry_run(self): """ Log what the output would be, but take no action. """ show_dry_run(self.index_list, 'allocation', body=self.body) def do_action(self): """ Change allocation settings for indices in `index_list.indices` with the settings in `body`. """ self.loggit.debug( 'Cannot get change shard routing allocation of closed indices. ' 'Omitting any closed indices.' ) self.index_list.filter_closed() self.index_list.empty_list_check() self.loggit.info('Updating index setting {0}'.format(self.body)) try: index_lists = chunk_index_list(self.index_list.indices) for l in index_lists: self.client.indices.put_settings( index=to_csv(l), body=self.body ) if self.wfc: logger.debug( 'Waiting for shards to complete relocation for indices:' ' {0}'.format(to_csv(l)) ) self.client.cluster.health(index=to_csv(l), level='indices', wait_for_relocating_shards=0, timeout=self.timeout, ) except Exception as e: report_failure(e) class Close(object): def __init__(self, ilo, delete_aliases=False): """ :arg ilo: A :class:`curator.indexlist.IndexList` object :arg delete_aliases: If `True`, will delete any associated aliases before closing indices. :type delete_aliases: bool """ verify_index_list(ilo) #: Instance variable. #: Internal reference to `ilo` self.index_list = ilo #: Instance variable. #: Internal reference to `delete_aliases` self.delete_aliases = delete_aliases #: Instance variable. #: The Elasticsearch Client object derived from `ilo` self.client = ilo.client self.loggit = logging.getLogger('curator.actions.close') def do_dry_run(self): """ Log what the output would be, but take no action. """ show_dry_run( self.index_list, 'close', **{'delete_aliases':self.delete_aliases}) def do_action(self): """ Close open indices in `index_list.indices` """ self.index_list.filter_closed() self.index_list.empty_list_check() self.loggit.info( 'Closing selected indices: {0}'.format(self.index_list.indices)) try: index_lists = chunk_index_list(self.index_list.indices) for l in index_lists: if self.delete_aliases: self.loggit.info( 'Deleting aliases from indices before closing.') self.loggit.debug('Deleting aliases from: {0}'.format(l)) try: self.client.indices.delete_alias( index=to_csv(l), name='_all') except Exception as e: self.loggit.warn( 'Some indices may not have had aliases. Exception:' ' {0}'.format(e) ) self.client.indices.flush( index=to_csv(l), ignore_unavailable=True) self.client.indices.close( index=to_csv(l), ignore_unavailable=True) except Exception as e: report_failure(e) class ClusterRouting(object): def __init__( self, client, routing_type=None, setting=None, value=None, wait_for_completion=False, timeout=30, ): """ For now, the cluster routing settings are hardcoded to be ``transient`` :arg client: An :class:`elasticsearch.Elasticsearch` client object :arg routing_type: Type of routing to apply. Either `allocation` or `rebalance` :arg setting: Currently, the only acceptable value for `setting` is ``enable``. This is here in case that changes. :arg value: Used only if `setting` is `enable`. Semi-dependent on `routing_type`. Acceptable values for `allocation` and `rebalance` are ``all``, ``primaries``, and ``none`` (string, not `NoneType`). If `routing_type` is `allocation`, this can also be ``new_primaries``, and if `rebalance`, it can be ``replicas``. :arg wait_for_completion: Wait (or not) for the operation to complete before returning. (default: `False`) :type wait_for_completion: bool :arg timeout: Number of seconds to `wait_for_completion` """ verify_client_object(client) #: Instance variable. #: An :class:`elasticsearch.Elasticsearch` client object self.client = client self.loggit = logging.getLogger('curator.actions.cluster_routing') #: Instance variable. #: Internal reference to `wait_for_completion` self.wfc = wait_for_completion #: Instance variable. #: How long in seconds to `wait_for_completion` before returning with an #: exception self.timeout = '{0}s'.format(timeout) if setting != 'enable': raise ValueError( 'Invalid value for "setting": {0}.'.format(setting) ) if routing_type == 'allocation': if value not in ['all', 'primaries', 'new_primaries', 'none']: raise ValueError( 'Invalid "value": {0} with "routing_type":' '{1}.'.format(value, routing_type) ) elif routing_type == 'rebalance': if value not in ['all', 'primaries', 'replicas', 'none']: raise ValueError( 'Invalid "value": {0} with "routing_type":' '{1}.'.format(value, routing_type) ) else: raise ValueError( 'Invalid value for "routing_type": {0}.'.format(routing_type) ) bkey = 'cluster.routing.{0}.{1}'.format(routing_type,setting) self.body = { 'transient' : { bkey : value } } def do_dry_run(self): """ Log what the output would be, but take no action. """ logger.info('DRY-RUN MODE. No changes will be made.') self.loggit.info( 'DRY-RUN: Update cluster routing settings with arguments: ' '{0}'.format(self.body) ) def do_action(self): """ Change cluster routing settings with the settings in `body`. """ self.loggit.info('Updating cluster settings: {0}'.format(self.body)) try: self.client.cluster.put_settings(body=self.body) if self.wfc: logger.debug( 'Waiting for shards to complete routing and/or rebalancing' ) self.client.cluster.health( level='indices', wait_for_relocating_shards=0, timeout=self.timeout, ) except Exception as e: report_failure(e) class CreateIndex(object): def __init__(self, client, name, extra_settings={}): """ :arg client: An :class:`elasticsearch.Elasticsearch` client object :arg name: A name, which can contain :py:func:`time.strftime` strings :arg extra_settings: The `settings` and `mappings` for the index. For more information see https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html :type extra_settings: dict, representing the settings and mappings. """ verify_client_object(client) if not name: raise ConfigurationError('Value for "name" not provided.') #: Instance variable. #: The parsed version of `name` self.name = parse_date_pattern(name) #: Instance variable. #: Extracted from the config yaml, it should be a dictionary of #: mappings and settings suitable for index creation. self.body = extra_settings #: Instance variable. #: An :class:`elasticsearch.Elasticsearch` client object self.client = client self.loggit = logging.getLogger('curator.actions.create_index') def do_dry_run(self): """ Log what the output would be, but take no action. """ logger.info('DRY-RUN MODE. No changes will be made.') self.loggit.info( 'DRY-RUN: create_index "{0}" with arguments: ' '{1}'.format(self.name, self.body) ) def do_action(self): """ Create index identified by `name` with settings in `body` """ self.loggit.info( 'Creating index "{0}" with settings: ' '{1}'.format(self.name, self.body) ) try: self.client.indices.create(index=self.name, body=self.body) except Exception as e: report_failure(e) class DeleteIndices(object): def __init__(self, ilo, master_timeout=30): """ :arg ilo: A :class:`curator.indexlist.IndexList` object :arg master_timeout: Number of seconds to wait for master node response """ verify_index_list(ilo) if not type(master_timeout) == type(int()): raise TypeError( 'Incorrect type for "master_timeout": {0}. ' 'Should be integer value.'.format(type(master_timeout)) ) #: Instance variable. #: Internal reference to `ilo` self.index_list = ilo #: Instance variable. #: The Elasticsearch Client object derived from `ilo` self.client = ilo.client #: Instance variable. #: String value of `master_timeout` + 's', for seconds. self.master_timeout = str(master_timeout) + 's' self.loggit = logging.getLogger('curator.actions.delete_indices') self.loggit.debug('master_timeout value: {0}'.format( self.master_timeout)) def _verify_result(self, result, count): """ Breakout method to aid readability :arg result: A list of indices from `_get_result_list` :arg count: The number of tries that have occurred :rtype: bool """ if len(result) > 0: self.loggit.error( 'The following indices failed to delete on try ' '#{0}:'.format(count) ) for idx in result: self.loggit.error("---{0}".format(idx)) return False else: self.loggit.debug( 'Successfully deleted all indices on try #{0}'.format(count) ) return True def __chunk_loop(self, chunk_list): """ Loop through deletes 3 times to ensure they complete :arg chunk_list: A list of indices pre-chunked so it won't overload the URL size limit. """ working_list = chunk_list for count in range(1, 4): # Try 3 times for i in working_list: self.loggit.info("---deleting index {0}".format(i)) self.client.indices.delete( index=to_csv(working_list), master_timeout=self.master_timeout) result = [ i for i in working_list if i in get_indices(self.client)] if self._verify_result(result, count): return else: working_list = result self.loggit.error( 'Unable to delete the following indices after 3 attempts: ' '{0}'.format(result) ) def do_dry_run(self): """ Log what the output would be, but take no action. """ show_dry_run(self.index_list, 'delete_indices') def do_action(self): """ Delete indices in `index_list.indices` """ self.index_list.empty_list_check() self.loggit.info( 'Deleting selected indices: {0}'.format(self.index_list.indices)) try: index_lists = chunk_index_list(self.index_list.indices) for l in index_lists: self.__chunk_loop(l) except Exception as e: report_failure(e) class ForceMerge(object): def __init__(self, ilo, max_num_segments=None, delay=0): """ :arg ilo: A :class:`curator.indexlist.IndexList` object :arg max_num_segments: Number of segments per shard to forceMerge :arg delay: Number of seconds to delay between forceMerge operations """ verify_index_list(ilo) if not max_num_segments: raise MissingArgument('Missing value for "max_num_segments"') #: Instance variable. #: The Elasticsearch Client object derived from `ilo` self.client = ilo.client #: Instance variable. #: Internal reference to `ilo` self.index_list = ilo #: Instance variable. #: Internally accessible copy of `max_num_segments` self.max_num_segments = max_num_segments #: Instance variable. #: Internally accessible copy of `delay` self.delay = delay self.loggit = logging.getLogger('curator.actions.forcemerge') def do_dry_run(self): """ Log what the output would be, but take no action. """ show_dry_run( self.index_list, 'forcemerge', max_num_segments=self.max_num_segments, delay=self.delay, ) def do_action(self): """ forcemerge indices in `index_list.indices` """ self.index_list.empty_list_check() self.index_list.filter_forceMerged( max_num_segments=self.max_num_segments) self.loggit.info('forceMerging selected indices') try: for index_name in self.index_list.indices: self.loggit.info( 'forceMerging index {0} to {1} segments per shard. ' 'Please wait...'.format(index_name, self.max_num_segments) ) if get_version(self.client) < (2, 1, 0): self.client.indices.optimize(index=index_name, max_num_segments=self.max_num_segments) else: self.client.indices.forcemerge(index=index_name, max_num_segments=self.max_num_segments) if self.delay > 0: self.loggit.info( 'Pausing for {0} seconds before continuing...'.format( self.delay) ) time.sleep(self.delay) except Exception as e: report_failure(e) class Open(object): def __init__(self, ilo): """ :arg ilo: A :class:`curator.indexlist.IndexList` object """ verify_index_list(ilo) #: Instance variable. #: The Elasticsearch Client object derived from `ilo` self.client = ilo.client #: Instance variable. #: Internal reference to `ilo` self.index_list = ilo self.loggit = logging.getLogger('curator.actions.open') def do_dry_run(self): """ Log what the output would be, but take no action. """ show_dry_run(self.index_list, 'open') def do_action(self): """ Open closed indices in `index_list.indices` """ self.index_list.empty_list_check() self.loggit.info( 'Opening selected indices: {0}'.format(self.index_list.indices)) try: index_lists = chunk_index_list(self.index_list.indices) for l in index_lists: self.client.indices.open(index=to_csv(l)) except Exception as e: report_failure(e) class Replicas(object): def __init__(self, ilo, count=None, wait_for_completion=False, timeout=30): """ :arg ilo: A :class:`curator.indexlist.IndexList` object :arg count: The count of replicas per shard :arg wait_for_completion: Wait (or not) for the operation to complete before returning. (default: `False`) :type wait_for_completion: bool """ verify_index_list(ilo) # It's okay for count to be zero if count == 0: pass elif not count: raise MissingArgument('Missing value for "count"') #: Instance variable. #: The Elasticsearch Client object derived from `ilo` self.client = ilo.client #: Instance variable. #: Internal reference to `ilo` self.index_list = ilo #: Instance variable. #: Internally accessible copy of `count` self.count = count #: Instance variable. #: Internal reference to `wait_for_completion` self.wfc = wait_for_completion #: Instance variable. #: How long in seconds to `wait_for_completion` before returning with an #: exception self.timeout = '{0}s'.format(timeout) self.loggit = logging.getLogger('curator.actions.replicas') def do_dry_run(self): """ Log what the output would be, but take no action. """ show_dry_run(self.index_list, 'replicas', count=self.count) def do_action(self): """ Update the replica count of indices in `index_list.indices` """ self.index_list.empty_list_check() self.loggit.debug( 'Cannot get update replica count of closed indices. ' 'Omitting any closed indices.' ) self.index_list.filter_closed() self.loggit.info( 'Setting the replica count to {0} for indices: ' '{1}'.format(self.count, self.index_list.indices) ) try: index_lists = chunk_index_list(self.index_list.indices) for l in index_lists: self.client.indices.put_settings(index=to_csv(l), body={'number_of_replicas' : self.count}) if self.wfc and self.count > 0: logger.debug( 'Waiting for shards to complete replication for ' 'indices: {0}'.format(to_csv(l)) ) self.client.cluster.health( index=to_csv(l), wait_for_status='green', timeout=self.timeout, ) except Exception as e: report_failure(e) class DeleteSnapshots(object): def __init__(self, slo, retry_interval=120, retry_count=3): """ :arg slo: A :class:`curator.snapshotlist.SnapshotList` object :arg retry_interval: Number of seconds to delay betwen retries. Default: 120 (seconds) :arg retry_count: Number of attempts to make. Default: 3 """ verify_snapshot_list(slo) #: Instance variable. #: The Elasticsearch Client object derived from `slo` self.client = slo.client #: Instance variable. #: Internally accessible copy of `retry_interval` self.retry_interval = retry_interval #: Instance variable. #: Internally accessible copy of `retry_count` self.retry_count = retry_count #: Instance variable. #: Internal reference to `slo` self.snapshot_list = slo #: Instance variable. #: The repository name derived from `slo` self.repository = slo.repository self.loggit = logging.getLogger('curator.actions.delete_snapshots') def do_dry_run(self): """ Log what the output would be, but take no action. """ logger.info('DRY-RUN MODE. No changes will be made.') mykwargs = { 'repository' : self.repository, 'retry_interval' : self.retry_interval, 'retry_count' : self.retry_count, } for snap in self.snapshot_list.snapshots: logger.info('DRY-RUN: delete_snapshot: {0} with arguments: ' '{1}'.format(snap, mykwargs)) def do_action(self): """ Delete snapshots in `slo` Retry up to `retry_count` times, pausing `retry_interval` seconds between retries. """ self.snapshot_list.empty_list_check() self.loggit.info('Deleting selected snapshots') if not safe_to_snap( self.client, repository=self.repository, retry_interval=self.retry_interval, retry_count=self.retry_count): raise FailedExecution( 'Unable to delete snapshot(s) because a snapshot is in ' 'state "IN_PROGRESS"') try: for s in self.snapshot_list.snapshots: self.loggit.info('Deleting snapshot {0}...'.format(s)) self.client.snapshot.delete( repository=self.repository, snapshot=s) except Exception as e: report_failure(e) class Snapshot(object): def __init__(self, ilo, repository=None, name=None, ignore_unavailable=False, include_global_state=True, partial=False, wait_for_completion=True, skip_repo_fs_check=False): """ :arg ilo: A :class:`curator.indexlist.IndexList` object :arg repository: The Elasticsearch snapshot repository to use :arg name: What to name the snapshot. :arg wait_for_completion: Wait (or not) for the operation to complete before returning. (default: `True`) :type wait_for_completion: bool :arg ignore_unavailable: Ignore unavailable shards/indices. (default: `False`) :type ignore_unavailable: bool :arg include_global_state: Store cluster global state with snapshot. (default: `True`) :type include_global_state: bool :arg partial: Do not fail if primary shard is unavailable. (default: `False`) :type partial: bool :arg skip_repo_fs_check: Do not validate write access to repository on all cluster nodes before proceeding. (default: `False`). Useful for shared filesystems where intermittent timeouts can affect validation, but won't likely affect snapshot success. :type skip_repo_fs_check: bool """ verify_index_list(ilo) # Check here and don't bother with the rest of this if there are no # indices in the index list. ilo.empty_list_check() if not repository_exists(ilo.client, repository=repository): raise ActionError( 'Cannot snapshot indices to missing repository: ' '{0}'.format(repository) ) if not name: raise MissingArgument('No value for "name" provided.') #: Instance variable. #: The parsed version of `name` self.name = parse_date_pattern(name) #: Instance variable. #: The Elasticsearch Client object derived from `ilo` self.client = ilo.client #: Instance variable. #: Internal reference to `ilo` self.index_list = ilo #: Instance variable. #: Internally accessible copy of `repository` self.repository = repository #: Instance variable. #: Internally accessible copy of `wait_for_completion` self.wait_for_completion = wait_for_completion #: Instance variable. #: Internally accessible copy of `skip_repo_fs_check` self.skip_repo_fs_check = skip_repo_fs_check self.state = None #: Instance variable. #: Populated at instance creation time by calling #: :mod:`curator.utils.create_snapshot_body` with `ilo.indices` and the #: provided arguments: `ignore_unavailable`, `include_global_state`, #: `partial` self.body = create_snapshot_body( ilo.indices, ignore_unavailable=ignore_unavailable, include_global_state=include_global_state, partial=partial ) self.loggit = logging.getLogger('curator.actions.snapshot') def get_state(self): """ Get the state of the snapshot """ try: self.state = self.client.snapshot.get( repository=self.repository, snapshot=self.name)['snapshots'][0]['state'] return self.state except IndexError: raise CuratorException( 'Snapshot "{0}" not found in repository ' '"{1}"'.format(self.name, self.repository) ) def report_state(self): """ Log the state of the snapshot """ self.get_state() if self.state == 'SUCCESS': self.loggit.info( 'Snapshot {0} successfully completed.'.format(self.name)) else: self.loggit.warn( 'Snapshot {0} completed with state: {0}'.format(self.state)) def do_dry_run(self): """ Log what the output would be, but take no action. """ self.loggit.info('DRY-RUN MODE. No changes will be made.') self.loggit.info( 'DRY-RUN: snapshot: {0} in repository {1} with arguments: ' '{2}'.format(self.name, self.repository, self.body) ) def do_action(self): """ Snapshot indices in `index_list.indices`, with options passed. """ if not self.skip_repo_fs_check: test_repo_fs(self.client, self.repository) if snapshot_running(self.client): raise SnapshotInProgress('Snapshot already in progress.') try: self.loggit.info('Creating snapshot "{0}" from indices: ' '{1}'.format(self.name, self.index_list.indices) ) self.client.snapshot.create( repository=self.repository, snapshot=self.name, body=self.body, wait_for_completion=self.wait_for_completion ) if self.wait_for_completion: self.report_state() else: self.loggit.warn( '"wait_for_completion" set to {0}. ' 'Remember to check for successful completion ' 'manually.'.format(self.wait_for_completion) ) except Exception as e: report_failure(e) class Restore(object): def __init__(self, slo, name=None, indices=None, include_aliases=False, ignore_unavailable=False, include_global_state=True, partial=False, rename_pattern=None, rename_replacement=None, extra_settings={}, wait_for_completion=True, skip_repo_fs_check=False): """ :arg slo: A :class:`curator.snapshotlist.SnapshotList` object :arg name: Name of the snapshot to restore. If no name is provided, it will restore the most recent snapshot by age. :type name: str :arg indices: A list of indices to restore. If no indices are provided, it will restore all indices in the snapshot. :type indices: list :arg include_aliases: If set to `True`, restore aliases with the indices. (default: `False`) :type include_aliases: bool :arg ignore_unavailable: Ignore unavailable shards/indices. (default: `False`) :type ignore_unavailable: bool :arg include_global_state: Store cluster global state with snapshot. (default: `True`) :type include_global_state: bool :arg partial: Do not fail if primary shard is unavailable. (default: `False`) :type partial: bool :arg rename_pattern: A regular expression pattern with one or more captures, e.g. ``index_(.+)`` :type rename_pattern: str :arg rename_replacement: A target index name pattern with `$#` numbered references to the captures in ``rename_pattern``, e.g. ``restored_index_$1`` :type rename_replacement: str :arg extra_settings: Extra settings, including shard count and settings to omit. For more information see https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html#_changing_index_settings_during_restore :type extra_settings: dict, representing the settings. :arg wait_for_completion: Wait (or not) for the operation to complete before returning. (default: `True`) :type wait_for_completion: bool :arg skip_repo_fs_check: Do not validate write access to repository on all cluster nodes before proceeding. (default: `False`). Useful for shared filesystems where intermittent timeouts can affect validation, but won't likely affect snapshot success. :type skip_repo_fs_check: bool """ self.loggit = logging.getLogger('curator.actions.snapshot') verify_snapshot_list(slo) # Get the most recent snapshot. most_recent = slo.most_recent() self.loggit.debug('"most_recent" snapshot: {0}'.format(most_recent)) #: Instance variable. #: Will use a provided snapshot name, or the most recent snapshot in slo self.name = name if name else most_recent # Stop here now, if it's not a successful snapshot. if slo.snapshot_info[self.name]['state'] == 'PARTIAL' \ and partial == True: self.loggit.warn( 'Performing restore of snapshot in state PARTIAL.') elif slo.snapshot_info[self.name]['state'] != 'SUCCESS': raise CuratorException( 'Restore operation can only be performed on snapshots with ' 'state "SUCCESS", or "PARTIAL" if partial=True.' ) #: Instance variable. #: The Elasticsearch Client object derived from `slo` self.client = slo.client #: Instance variable. #: Internal reference to `slo` self.snapshot_list = slo #: Instance variable. #: `repository` derived from `slo` self.repository = slo.repository if indices: self.indices = ensure_list(indices) else: self.indices = slo.snapshot_info[self.name]['indices'] self.wfc = wait_for_completion #: Instance variable version of ``rename_pattern`` self.rename_pattern = rename_pattern if rename_replacement is not None \ else '' #: Instance variable version of ``rename_replacement`` self.rename_replacement = rename_replacement if rename_replacement \ is not None else '' #: Also an instance variable version of ``rename_replacement`` #: but with Java regex group designations of ``$#`` #: converted to Python's ``\\#`` style. self.py_rename_replacement = self.rename_replacement.replace('$', '\\') #: Instance variable. #: Internally accessible copy of `skip_repo_fs_check` self.skip_repo_fs_check = skip_repo_fs_check #: Instance variable. #: Populated at instance creation time from the other options self.body = { 'indices' : self.indices, 'include_aliases' : include_aliases, 'ignore_unavailable' : ignore_unavailable, 'include_global_state' : include_global_state, 'partial' : partial, 'rename_pattern' : self.rename_pattern, 'rename_replacement' : self.rename_replacement, } if extra_settings: self.loggit.debug( 'Adding extra_settings to restore body: ' '{0}'.format(extra_settings) ) try: self.body.update(extra_settings) except: self.loggit.error( 'Unable to apply extra settings to restore body') self.loggit.debug('REPOSITORY: {0}'.format(self.repository)) self.loggit.debug('WAIT_FOR_COMPLETION: {0}'.format(self.wfc)) self.loggit.debug( 'SKIP_REPO_FS_CHECK: {0}'.format(self.skip_repo_fs_check)) self.loggit.debug('BODY: {0}'.format(self.body)) # Populate the expected output index list. self._get_expected_output() def _get_expected_output(self): if not self.rename_pattern and not self.rename_replacement: self.expected_output = self.indices return # Don't stick around if we're not replacing anything self.expected_output = [] for index in self.indices: self.expected_output.append( re.sub( self.rename_pattern, self.py_rename_replacement, index ) ) self.loggit.debug('index: {0} replacement: ' '{1}'.format(index, self.expected_output[-1]) ) def report_state(self): """ Log the state of the restore This should only be done if ``wait_for_completion`` is `True`, and only after completing the restore. """ all_indices = get_indices(self.client) found_count = 0 missing = [] for index in self.expected_output: if index in all_indices: found_count += 1 self.loggit.info('Found restored index {0}'.format(index)) else: missing.append(index) if found_count == len(self.expected_output): self.loggit.info('All indices appear to have been restored.') else: self.loggit.error( 'Some of the indices do not appear to have been restored. ' 'Missing: {0}'.format(missing) ) def do_dry_run(self): """ Log what the output would be, but take no action. """ logger.info('DRY-RUN MODE. No changes will be made.') logger.info( 'DRY-RUN: restore: Repository: {0} Snapshot name: {1} Arguments: ' '{2}'.format( self.name, self.repository, { 'wait_for_completion' : self.wfc, 'body' : self.body } ) ) for index in self.indices: if self.rename_pattern and self.rename_replacement: replacement_msg = 'as {0}'.format( re.sub( self.rename_pattern, self.py_rename_replacement, index ) ) else: replacement_msg = '' logger.info( 'DRY-RUN: restore: Index {0} {1}'.format(index, replacement_msg) ) def do_action(self): """ Restore indices with options passed. """ if not self.skip_repo_fs_check: test_repo_fs(self.client, self.repository) if snapshot_running(self.client): raise SnapshotInProgress( 'Cannot restore while a snapshot is in progress.') try: self.loggit.info('Restoring indices "{0}" from snapshot: ' '{1}'.format(self.indices, self.name) ) self.client.snapshot.restore( repository=self.repository, snapshot=self.name, body=self.body, wait_for_completion=self.wfc ) if self.wfc: self.report_state() else: self.loggit.warn( '"wait_for_completion" set to {0}. ' 'Remember to check for successful completion ' 'manually.'.format(self.wfc) ) except Exception as e: report_failure(e) curator-4.2.5/curator/cli.py000066400000000000000000000200621302700154000157710ustar00rootroot00000000000000import os, sys import yaml import logging import click from voluptuous import Schema from .defaults import settings from .validators import SchemaCheck from .config_utils import process_config from .exceptions import * from .utils import * from .indexlist import IndexList from .snapshotlist import SnapshotList from .actions import * from ._version import __version__ CLASS_MAP = { 'alias' : Alias, 'allocation' : Allocation, 'close' : Close, 'cluster_routing' : ClusterRouting, 'create_index' : CreateIndex, 'delete_indices' : DeleteIndices, 'delete_snapshots' : DeleteSnapshots, 'forcemerge' : ForceMerge, 'open' : Open, 'replicas' : Replicas, 'restore' : Restore, 'snapshot' : Snapshot, } def process_action(client, config, **kwargs): """ Do the `action` in the configuration dictionary, using the associated args. Other necessary args may be passed as keyword arguments :arg config: An `action` dictionary. """ logger = logging.getLogger(__name__) # Make some placeholder variables here for readability logger.debug('Configuration dictionary: {0}'.format(config)) logger.debug('kwargs: {0}'.format(kwargs)) action = config['action'] # This will always have some defaults now, so no need to do the if... # # OLD WAY: opts = config['options'] if 'options' in config else {} opts = config['options'] logger.debug('opts: {0}'.format(opts)) mykwargs = {} action_class = CLASS_MAP[action] # Add some settings to mykwargs... if action == 'delete_indices': mykwargs['master_timeout'] = ( kwargs['master_timeout'] if 'master_timeout' in kwargs else 30) if action == 'allocation' or action == 'replicas': # Setting the operation timeout to the client timeout mykwargs['timeout'] = ( kwargs['timeout'] if 'timeout' in kwargs else 30) ### Update the defaults with whatever came with opts, minus any Nones mykwargs.update(prune_nones(opts)) logger.debug('Action kwargs: {0}'.format(mykwargs)) ### Set up the action ### if action == 'alias': # Special behavior for this action, as it has 2 index lists logger.debug('Running "{0}" action'.format(action.upper())) action_obj = action_class(**mykwargs) if 'add' in config: logger.debug('Adding indices to alias "{0}"'.format(opts['name'])) adds = IndexList(client) adds.iterate_filters(config['add']) action_obj.add(adds) if 'remove' in config: logger.debug( 'Removing indices from alias "{0}"'.format(opts['name'])) removes = IndexList(client) removes.iterate_filters(config['remove']) action_obj.remove(removes) elif action in [ 'cluster_routing', 'create_index' ]: action_obj = action_class(client, **mykwargs) elif action == 'delete_snapshots' or action == 'restore': logger.debug('Running "{0}"'.format(action)) slo = SnapshotList(client, repository=opts['repository']) slo.iterate_filters(config) # We don't need to send this value to the action mykwargs.pop('repository') action_obj = action_class(slo, **mykwargs) else: logger.debug('Running "{0}"'.format(action.upper())) ilo = IndexList(client) ilo.iterate_filters(config) action_obj = action_class(ilo, **mykwargs) ### Do the action if 'dry_run' in kwargs and kwargs['dry_run'] == True: action_obj.do_dry_run() else: logger.debug('Doing the action here.') action_obj.do_action() @click.command() @click.option('--config', help="Path to configuration file. Default: ~/.curator/curator.yml", type=click.Path(exists=True), default=settings.config_file() ) @click.option('--dry-run', is_flag=True, help='Do not perform any changes.') @click.argument('action_file', type=click.Path(exists=True), nargs=1) @click.version_option(version=__version__) def cli(config, dry_run, action_file): """ Curator for Elasticsearch indices. See http://elastic.co/guide/en/elasticsearch/client/curator/current """ client_args = process_config(config) logger = logging.getLogger(__name__) logger.debug('Client and logging options validated.') # Extract this and save it for later, in case there's no timeout_override. default_timeout = client_args.pop('timeout') logger.debug('default_timeout = {0}'.format(default_timeout)) ######################################### ### Start working on the actions here ### ######################################### action_config = get_yaml(action_file) action_dict = validate_actions(action_config) actions = action_dict['actions'] logger.debug('Full list of actions: {0}'.format(actions)) action_keys = sorted(list(actions.keys())) for idx in action_keys: action = actions[idx]['action'] action_disabled = actions[idx]['options'].pop('disable_action') logger.debug('action_disabled = {0}'.format(action_disabled)) continue_if_exception = ( actions[idx]['options'].pop('continue_if_exception')) logger.debug( 'continue_if_exception = {0}'.format(continue_if_exception)) timeout_override = actions[idx]['options'].pop('timeout_override') logger.debug('timeout_override = {0}'.format(timeout_override)) ignore_empty_list = actions[idx]['options'].pop('ignore_empty_list') logger.debug('ignore_empty_list = {0}'.format(ignore_empty_list)) ### Skip to next action if 'disabled' if action_disabled: logger.info( 'Action ID: {0}: "{1}" not performed because "disable_action" ' 'is set to True'.format(idx, action) ) continue else: logger.info('Preparing Action ID: {0}, "{1}"'.format(idx, action)) # Override the timeout, if specified, otherwise use the default. if type(timeout_override) == type(int()): client_args['timeout'] = timeout_override else: client_args['timeout'] = default_timeout # Set up action kwargs kwargs = {} kwargs['master_timeout'] = ( client_args['timeout'] if client_args['timeout'] <= 300 else 300) kwargs['dry_run'] = dry_run kwargs['timeout'] = client_args['timeout'] # Create a client object for each action... client = get_client(**client_args) logger.debug('client is {0}'.format(type(client))) ########################## ### Process the action ### ########################## try: logger.info('Trying Action ID: {0}, "{1}": ' '{2}'.format(idx, action, actions[idx]['description']) ) process_action(client, actions[idx], **kwargs) except Exception as e: if str(type(e)) == "" or \ str(type(e)) == "": if ignore_empty_list: logger.info( 'Skipping action "{0}" due to empty list: ' '{1}'.format(action, type(e)) ) else: logger.error( 'Unable to complete action "{0}". No actionable items ' 'in list: {1}'.format(action, type(e)) ) sys.exit(1) else: logger.error( 'Failed to complete action: {0}. {1}: ' '{2}'.format(action, type(e), e) ) if continue_if_exception: logger.info( 'Continuing execution with next action because ' '"continue_if_exception" is set to True for action ' '{0}'.format(action) ) else: sys.exit(1) logger.info('Action ID: {0}, "{1}" completed.'.format(idx, action)) logger.info('Job completed.') curator-4.2.5/curator/config_utils.py000066400000000000000000000034741302700154000177170ustar00rootroot00000000000000from voluptuous import Schema # from .defaults import settings from .validators import SchemaCheck, config_file from .utils import * from .logtools import LogInfo, Whitelist, Blacklist def test_config(config): # Get config from yaml file yaml_config = get_yaml(config) # if the file is empty, which is still valid yaml, set as an empty dict yaml_config = {} if not yaml_config else prune_nones(yaml_config) # Voluptuous can't verify the schema of a dict if it doesn't have keys, # so make sure the keys are at least there and are dict() for k in ['client', 'logging']: if k not in yaml_config: yaml_config[k] = {} else: yaml_config[k] = prune_nones(yaml_config[k]) return SchemaCheck(yaml_config, config_file.client(), 'Client Configuration', 'full configuration dictionary').result() def set_logging(log_opts): try: from logging import NullHandler except ImportError: from logging import Handler class NullHandler(Handler): def emit(self, record): pass # Set up logging loginfo = LogInfo(log_opts) logging.root.addHandler(loginfo.handler) logging.root.setLevel(loginfo.numeric_log_level) logger = logging.getLogger('curator.cli') # Set up NullHandler() to handle nested elasticsearch.trace Logger # instance in elasticsearch python client logging.getLogger('elasticsearch.trace').addHandler(NullHandler()) if log_opts['blacklist']: for bl_entry in ensure_list(log_opts['blacklist']): for handler in logging.root.handlers: handler.addFilter(Blacklist(bl_entry)) def process_config(yaml_file): config = test_config(yaml_file) set_logging(config['logging']) test_client_options(config['client']) return config['client'] curator-4.2.5/curator/curator_cli.py000077500000000000000000000001061302700154000175300ustar00rootroot00000000000000import click from .singletons import cli def main(): cli(obj={}) curator-4.2.5/curator/defaults/000077500000000000000000000000001302700154000164575ustar00rootroot00000000000000curator-4.2.5/curator/defaults/__init__.py000066400000000000000000000000001302700154000205560ustar00rootroot00000000000000curator-4.2.5/curator/defaults/settings.py000066400000000000000000000035111302700154000206710ustar00rootroot00000000000000import os from voluptuous import * # Elasticsearch versions supported def version_max(): return (5, 1, 99) def version_min(): return (2, 0, 0) # Default Config file location def config_file(): return os.path.join(os.path.expanduser('~'), '.curator', 'curator.yml') # Default filter patterns (regular expressions) def regex_map(): return { 'timestring': r'^.*{0}.*$', 'regex': r'{0}', 'prefix': r'^{0}.*$', 'suffix': r'^.*{0}$', } def date_regex(): return { 'Y' : '4', 'y' : '2', 'm' : '2', 'W' : '2', 'U' : '2', 'd' : '2', 'H' : '2', 'M' : '2', 'S' : '2', 'j' : '3', } # Actions def cluster_actions(): return [ 'cluster_routing' ] def index_actions(): return [ 'alias', 'allocation', 'close', 'create_index', 'delete_indices', 'forcemerge', 'open', 'replicas', 'snapshot', ] def snapshot_actions(): return [ 'delete_snapshots', 'restore' ] def all_actions(): return sorted(cluster_actions() + index_actions() + snapshot_actions()) def index_filtertypes(): return [ 'alias', 'allocated', 'age', 'closed', 'count', 'forcemerged', 'kibana', 'none', 'opened', 'pattern', 'space', ] def snapshot_filtertypes(): return ['age', 'count', 'none', 'pattern', 'state'] def all_filtertypes(): return sorted(list(set(index_filtertypes() + snapshot_filtertypes()))) def default_options(): return { 'continue_if_exception': False, 'disable_action': False, 'ignore_empty_list': False, 'timeout_override': None, } def default_filters(): return { 'filters' : [{ 'filtertype' : 'none' }] } curator-4.2.5/curator/exceptions.py000066400000000000000000000020231302700154000174000ustar00rootroot00000000000000class CuratorException(Exception): """ Base class for all exceptions raised by Curator which are not Elasticsearch exceptions. """ class ConfigurationError(CuratorException): """ Exception raised when a misconfiguration is detected """ class MissingArgument(CuratorException): """ Exception raised when a needed argument is not passed. """ class NoIndices(CuratorException): """ Exception raised when an operation is attempted against an empty index_list """ class NoSnapshots(CuratorException): """ Exception raised when an operation is attempted against an empty snapshot_list """ class ActionError(CuratorException): """ Exception raised when an action (against an index_list or snapshot_list) cannot be taken. """ class FailedExecution(CuratorException): """ Exception raised when an action fails to execute for some reason. """ class SnapshotInProgress(ActionError): """ Exception raised when a snapshot is already in progress """ curator-4.2.5/curator/indexlist.py000066400000000000000000001064231302700154000172330ustar00rootroot00000000000000from datetime import timedelta, datetime, date import time import re import logging import elasticsearch from .defaults import settings from .validators import SchemaCheck, filters from .exceptions import * from .utils import * class IndexList(object): def __init__(self, client): verify_client_object(client) self.loggit = logging.getLogger('curator.indexlist') #: An Elasticsearch Client object #: Also accessible as an instance variable. self.client = client #: Instance variable. #: Information extracted from indices, such as segment count, age, etc. #: Populated at instance creation time, and by other private helper #: methods, as needed. **Type:** ``dict()`` self.index_info = {} #: Instance variable. #: The running list of indices which will be used by an Action class. #: Populated at instance creation time. **Type:** ``list()`` self.indices = [] #: Instance variable. #: All indices in the cluster at instance creation time. #: **Type:** ``list()`` self.all_indices = [] self.__get_indices() def __actionable(self, idx): self.loggit.debug( 'Index {0} is actionable and remains in the list.'.format(idx)) def __not_actionable(self, idx): self.loggit.debug( 'Index {0} is not actionable, removing from list.'.format(idx)) self.indices.remove(idx) def __excludify(self, condition, exclude, index, msg=None): if condition == True: if exclude: text = "Removed from actionable list" self.__not_actionable(index) else: text = "Remains in actionable list" self.__actionable(index) else: if exclude: text = "Remains in actionable list" self.__actionable(index) else: text = "Removed from actionable list" self.__not_actionable(index) if msg: self.loggit.debug('{0}: {1}'.format(text, msg)) def __get_indices(self): """ Pull all indices into `all_indices`, then populate `indices` and `index_info` """ self.loggit.debug('Getting all indices') self.all_indices = get_indices(self.client) self.indices = self.all_indices[:] self.empty_list_check() for index in self.indices: self.__build_index_info(index) self._get_metadata() self._get_index_stats() def __build_index_info(self, index): """ Ensure that `index` is a key in `index_info`. If not, create a sub-dictionary structure under that key. """ self.loggit.debug( 'Building preliminary index metadata for {0}'.format(index)) if not index in self.index_info: self.index_info[index] = { "age" : {}, "number_of_replicas" : 0, "number_of_shards" : 0, "segments" : 0, "size_in_bytes" : 0, "docs" : 0, "state" : "", } def __map_method(self, ft): methods = { 'alias': self.filter_by_alias, 'age': self.filter_by_age, 'allocated': self.filter_allocated, 'closed': self.filter_closed, 'count': self.filter_by_count, 'forcemerged': self.filter_forceMerged, 'kibana': self.filter_kibana, 'none': self.filter_none, 'opened': self.filter_opened, 'pattern': self.filter_by_regex, 'space': self.filter_by_space, } return methods[ft] def _get_index_stats(self): """ Populate `index_info` with index `size_in_bytes` and doc count information for each index. """ self.loggit.debug('Getting index stats') self.empty_list_check() # Subroutine to do the dirty work def iterate_over_stats(stats): for index in stats['indices']: size = stats['indices'][index]['total']['store']['size_in_bytes'] docs = stats['indices'][index]['total']['docs']['count'] self.loggit.debug( 'Index: {0} Size: {1} Docs: {2}'.format( index, byte_size(size), docs ) ) self.index_info[index]['size_in_bytes'] = size self.index_info[index]['docs'] = docs working_list = self.working_list() for index in self.working_list(): if self.index_info[index]['state'] == 'close': working_list.remove(index) if working_list: index_lists = chunk_index_list(working_list) for l in index_lists: iterate_over_stats( self.client.indices.stats(index=to_csv(l), metric='store,docs') ) def _get_metadata(self): """ Populate `index_info` with index `size_in_bytes` and doc count information for each index. """ self.loggit.debug('Getting index metadata') self.empty_list_check() index_lists = chunk_index_list(self.indices) for l in index_lists: working_list = ( self.client.cluster.state( index=to_csv(l),metric='metadata' )['metadata']['indices'] ) if working_list: for index in list(working_list.keys()): s = self.index_info[index] wl = working_list[index] if not 'creation_date' in wl['settings']['index']: self.loggit.warn( 'Index: {0} has no "creation_date"! This implies ' 'that the index predates Elasticsearch v1.4. For ' 'safety, this index will be removed from the ' 'actionable list.'.format(index) ) self.__not_actionable(index) else: s['age']['creation_date'] = ( fix_epoch(wl['settings']['index']['creation_date']) ) s['number_of_replicas'] = ( wl['settings']['index']['number_of_replicas'] ) s['number_of_shards'] = ( wl['settings']['index']['number_of_shards'] ) s['state'] = wl['state'] if 'routing' in wl['settings']['index']: s['routing'] = wl['settings']['index']['routing'] def empty_list_check(self): """Raise exception if `indices` is empty""" self.loggit.debug('Checking for empty list') if not self.indices: raise NoIndices('index_list object is empty.') def working_list(self): """ Return the current value of `indices` as copy-by-value to prevent list stomping during iterations """ # Copy by value, rather than reference to prevent list stomping during # iterations self.loggit.debug('Generating working list of indices') return self.indices[:] def _get_segmentcounts(self): """ Populate `index_info` with segment information for each index. """ self.loggit.debug('Getting index segment counts') self.empty_list_check() index_lists = chunk_index_list(self.indices) for l in index_lists: working_list = ( self.client.indices.segments(index=to_csv(l))['indices'] ) if working_list: for index in list(working_list.keys()): shards = working_list[index]['shards'] segmentcount = 0 for shardnum in shards: for shard in range(0,len(shards[shardnum])): segmentcount += ( shards[shardnum][shard]['num_search_segments'] ) self.index_info[index]['segments'] = segmentcount def _get_name_based_ages(self, timestring): """ Add indices to `index_info` based on the age as indicated by the index name pattern, if it matches `timestring` :arg timestring: An strftime pattern """ # Check for empty list before proceeding here to prevent non-iterable # condition self.loggit.debug('Getting ages of indices by "name"') self.empty_list_check() ts = TimestringSearch(timestring) for index in self.working_list(): epoch = ts.get_epoch(index) if epoch: self.index_info[index]['age']['name'] = epoch def _get_field_stats_dates(self, field='@timestamp'): """ Add indices to `index_info` based on the value the stats api returns, as determined by `field` :arg field: The field with the date value. The field must be mapped in elasticsearch as a date datatype. Default: ``@timestamp`` """ self.loggit.debug('Getting index date from field_stats API') self.loggit.debug( 'Cannot use field_stats on closed indices. ' 'Omitting any closed indices.' ) self.filter_closed() index_lists = chunk_index_list(self.indices) for l in index_lists: working_list = self.client.field_stats( index=to_csv(l), fields=field, level='indices' )['indices'] if working_list: for index in list(working_list.keys()): try: s = self.index_info[index]['age'] wl = working_list[index]['fields'][field] # Use these new references to keep these lines more # readable s['min_value'] = fix_epoch(wl['min_value']) s['max_value'] = fix_epoch(wl['max_value']) except KeyError as e: raise ActionError( 'Field "{0}" not found in index ' '"{1}"'.format(field, index) ) def _calculate_ages(self, source=None, timestring=None, field=None, stats_result=None ): """ This method initiates index age calculation based on the given parameters. Exceptions are raised when they are improperly configured. Set instance variable `age_keyfield` for use later, if needed. :arg source: Source of index age. Can be one of 'name', 'creation_date', or 'field_stats' :arg timestring: An strftime string to match the datestamp in an index name. Only used for index filtering by ``name``. :arg field: A timestamp field name. Only used for ``field_stats`` based calculations. :arg stats_result: Either `min_value` or `max_value`. Only used in conjunction with `source`=``field_stats`` to choose whether to reference the minimum or maximum result value. """ self.age_keyfield = source if source == 'name': if not timestring: raise MissingArgument( 'source "name" requires the "timestring" keyword argument' ) self._get_name_based_ages(timestring) elif source == 'creation_date': # Nothing to do here as this comes from `get_metadata` in __init__ pass elif source == 'field_stats': if not field: raise MissingArgument( 'source "field_stats" requires the "field" keyword argument' ) if stats_result not in ['min_value', 'max_value']: raise ValueError( 'Invalid value for "stats_result": {0}'.format(stats_result) ) self.age_keyfield = stats_result self._get_field_stats_dates(field=field) else: raise ValueError( 'Invalid source: {0}. ' 'Must be one of "name", ' '"creation_date", "field_stats".'.format(source) ) def _sort_by_age(self, index_list, reverse=True): """ Take a list of indices and sort them by date. By default, the youngest are first with `reverse=True`, but the oldest can be first by setting `reverse=False` """ # Do the age-based sorting here. # First, build an temporary dictionary with just index and age # as the key and value, respectively temp = {} for index in index_list: if self.age_keyfield in self.index_info[index]['age']: temp[index] = self.index_info[index]['age'][self.age_keyfield] else: msg = ( '{0} does not have age key "{1}" in IndexList ' ' metadata'.format(index, self.age_keyfield) ) self.__excludify(True, True, index, msg) # If reverse is True, this will sort so the youngest indices are first. # However, if you want oldest first, set reverse to False. # Effectively, this should set us up to act on everything older than # meets the other set criteria. # It starts as a tuple, but then becomes a list. sorted_tuple = ( sorted(temp.items(), key=lambda k: k[1], reverse=reverse) ) return [x[0] for x in sorted_tuple] def filter_by_regex(self, kind=None, value=None, exclude=False): """ Match indices by regular expression (pattern). :arg kind: Can be one of: ``suffix``, ``prefix``, ``regex``, or ``timestring``. This option defines what kind of filter you will be building. :arg value: Depends on `kind`. It is the strftime string if `kind` is ``timestring``. It's used to build the regular expression for other kinds. :arg exclude: If `exclude` is `True`, this filter will remove matching indices from `indices`. If `exclude` is `False`, then only matching indices will be kept in `indices`. Default is `False` """ self.loggit.debug('Filtering indices by regex') if kind not in [ 'regex', 'prefix', 'suffix', 'timestring' ]: raise ValueError('{0}: Invalid value for kind'.format(kind)) # Stop here if None or empty value, but zero is okay if value == 0: pass elif not value: raise ValueError( '{0}: Invalid value for "value". ' 'Cannot be "None" type, empty, or False' ) if kind == 'timestring': regex = settings.regex_map()[kind].format(get_date_regex(value)) else: regex = settings.regex_map()[kind].format(value) self.empty_list_check() pattern = re.compile(regex) for index in self.working_list(): self.loggit.debug('Filter by regex: Index: {0}'.format(index)) match = pattern.match(index) if match: self.__excludify(True, exclude, index) else: self.__excludify(False, exclude, index) def filter_by_age(self, source='name', direction=None, timestring=None, unit=None, unit_count=None, field=None, stats_result='min_value', epoch=None, exclude=False, ): """ Match `indices` by relative age calculations. :arg source: Source of index age. Can be one of 'name', 'creation_date', or 'field_stats' :arg direction: Time to filter, either ``older`` or ``younger`` :arg timestring: An strftime string to match the datestamp in an index name. Only used for index filtering by ``name``. :arg unit: One of ``seconds``, ``minutes``, ``hours``, ``days``, ``weeks``, ``months``, or ``years``. :arg unit_count: The number of ``unit``s. ``unit_count`` * ``unit`` will be calculated out to the relative number of seconds. :arg field: A timestamp field name. Only used for ``field_stats`` based calculations. :arg stats_result: Either `min_value` or `max_value`. Only used in conjunction with `source`=``field_stats`` to choose whether to reference the minimum or maximum result value. :arg epoch: An epoch timestamp used in conjunction with ``unit`` and ``unit_count`` to establish a point of reference for calculations. If not provided, the current time will be used. :arg exclude: If `exclude` is `True`, this filter will remove matching indices from `indices`. If `exclude` is `False`, then only matching indices will be kept in `indices`. Default is `False` """ self.loggit.debug('Filtering indices by age') # Get timestamp point of reference, PoR PoR = get_point_of_reference(unit, unit_count, epoch) if not direction: raise MissingArgument('Must provide a value for "direction"') if direction not in ['older', 'younger']: raise ValueError( 'Invalid value for "direction": {0}'.format(direction) ) self._calculate_ages( source=source, timestring=timestring, field=field, stats_result=stats_result ) for index in self.working_list(): try: msg = ( 'Index "{0}" age ({1}), direction: "{2}", point of ' 'reference, ({3})'.format( index, int(self.index_info[index]['age'][self.age_keyfield]), direction, PoR ) ) # Because time adds to epoch, smaller numbers are actually older # timestamps. if direction == 'older': agetest = self.index_info[index]['age'][self.age_keyfield] < PoR else: agetest = self.index_info[index]['age'][self.age_keyfield] > PoR self.__excludify(agetest, exclude, index, msg) except KeyError: self.loggit.debug( 'Index "{0}" does not meet provided criteria. ' 'Removing from list.'.format(index, source)) self.indices.remove(index) def filter_by_space( self, disk_space=None, reverse=True, use_age=False, source='creation_date', timestring=None, field=None, stats_result='min_value', exclude=False): """ Remove indices from the actionable list based on space consumed, sorted reverse-alphabetically by default. If you set `reverse` to `False`, it will be sorted alphabetically. The default is usually what you will want. If only one kind of index is provided--for example, indices matching ``logstash-%Y.%m.%d``--then reverse alphabetical sorting will mean the oldest will remain in the list, because lower numbers in the dates mean older indices. By setting `reverse` to `False`, then ``index3`` will be deleted before ``index2``, which will be deleted before ``index1`` `use_age` allows ordering indices by age. Age is determined by the index creation date by default, but you can specify an `source` of ``name``, ``max_value``, or ``min_value``. The ``name`` `source` requires the timestring argument. :arg disk_space: Filter indices over *n* gigabytes :arg reverse: The filtering direction. (default: `True`). Ignored if `use_age` is `True` :arg use_age: Sort indices by age. ``source`` is required in this case. :arg source: Source of index age. Can be one of ``name``, ``creation_date``, or ``field_stats``. Default: ``creation_date`` :arg timestring: An strftime string to match the datestamp in an index name. Only used if `source` ``name`` is selected. :arg field: A timestamp field name. Only used if `source` ``field_stats`` is selected. :arg stats_result: Either `min_value` or `max_value`. Only used if `source` ``field_stats`` is selected. It determines whether to reference the minimum or maximum value of `field` in each index. :arg exclude: If `exclude` is `True`, this filter will remove matching indices from `indices`. If `exclude` is `False`, then only matching indices will be kept in `indices`. Default is `False` """ self.loggit.debug('Filtering indices by disk space') # Ensure that disk_space is a float if not disk_space: raise MissingArgument('No value for "disk_space" provided') disk_space = float(disk_space) disk_usage = 0.0 disk_limit = disk_space * 2**30 self.loggit.debug( 'Cannot get disk usage info from closed indices. ' 'Omitting any closed indices.' ) self.filter_closed() # Create a copy-by-value working list working_list = self.working_list() if use_age: self._calculate_ages( source=source, timestring=timestring, field=field, stats_result=stats_result ) # Using default value of reverse=True in self._sort_by_age() sorted_indices = self._sort_by_age(working_list) else: # Default to sorting by index name sorted_indices = sorted(working_list, reverse=reverse) for index in sorted_indices: disk_usage += self.index_info[index]['size_in_bytes'] msg = ( '{0}, summed disk usage is {1} and disk limit is {2}.'.format( index, byte_size(disk_usage), byte_size(disk_limit) ) ) self.__excludify((disk_usage > disk_limit), exclude, index, msg) def filter_kibana(self, exclude=True): """ Match any index named ``.kibana``, ``kibana-int``, ``.marvel-kibana``, or ``.marvel-es-data`` in `indices`. :arg exclude: If `exclude` is `True`, this filter will remove matching indices from `indices`. If `exclude` is `False`, then only matching indices will be kept in `indices`. Default is `True` """ self.loggit.debug('Filtering kibana indices') self.empty_list_check() for index in self.working_list(): if index in [ '.kibana', '.marvel-kibana', 'kibana-int', '.marvel-es-data' ]: self.__excludify(True, exclude, index) def filter_forceMerged(self, max_num_segments=None, exclude=True): """ Match any index which has `max_num_segments` per shard or fewer in the actionable list. :arg max_num_segments: Cutoff number of segments per shard. :arg exclude: If `exclude` is `True`, this filter will remove matching indices from `indices`. If `exclude` is `False`, then only matching indices will be kept in `indices`. Default is `True` """ self.loggit.debug('Filtering forceMerged indices') if not max_num_segments: raise MissingArgument('Missing value for "max_num_segments"') self.loggit.debug( 'Cannot get segment count of closed indices. ' 'Omitting any closed indices.' ) self.filter_closed() self._get_segmentcounts() for index in self.working_list(): # Do this to reduce long lines and make it more readable... shards = int(self.index_info[index]['number_of_shards']) replicas = int(self.index_info[index]['number_of_replicas']) segments = int(self.index_info[index]['segments']) msg = ( '{0} has {1} shard(s) + {2} replica(s) ' 'with a sum total of {3} segments.'.format( index, shards, replicas, segments ) ) expected_count = ((shards + (shards * replicas)) * max_num_segments) self.__excludify((segments <= expected_count), exclude, index, msg) def filter_closed(self, exclude=True): """ Filter out closed indices from `indices` :arg exclude: If `exclude` is `True`, this filter will remove matching indices from `indices`. If `exclude` is `False`, then only matching indices will be kept in `indices`. Default is `True` """ self.loggit.debug('Filtering closed indices') self.empty_list_check() for index in self.working_list(): condition = self.index_info[index]['state'] == 'close' self.loggit.debug('Index {0} state: {1}'.format( index, self.index_info[index]['state'] ) ) self.__excludify(condition, exclude, index) def filter_opened(self, exclude=True): """ Filter out opened indices from `indices` :arg exclude: If `exclude` is `True`, this filter will remove matching indices from `indices`. If `exclude` is `False`, then only matching indices will be kept in `indices`. Default is `True` """ self.loggit.debug('Filtering open indices') self.empty_list_check() for index in self.working_list(): condition = self.index_info[index]['state'] == 'open' self.loggit.debug('Index {0} state: {1}'.format( index, self.index_info[index]['state'] ) ) self.__excludify(condition, exclude, index) def filter_allocated(self, key=None, value=None, allocation_type='require', exclude=True, ): """ Match indices that have the routing allocation rule of `key=value` from `indices` :arg key: The allocation attribute to check for :arg value: The value to check for :arg allocation_type: Type of allocation to apply :arg exclude: If `exclude` is `True`, this filter will remove matching indices from `indices`. If `exclude` is `False`, then only matching indices will be kept in `indices`. Default is `True` """ self.loggit.debug( 'Filtering indices with shard routing allocation rules') if not key: raise MissingArgument('No value for "key" provided') if not value: raise MissingArgument('No value for "value" provided') if not allocation_type in ['include', 'exclude', 'require']: raise ValueError( 'Invalid "allocation_type": {0}'.format(allocation_type) ) self.empty_list_check() index_lists = chunk_index_list(self.indices) for l in index_lists: working_list = self.client.indices.get_settings(index=to_csv(l)) if working_list: for index in list(working_list.keys()): try: has_routing = ( working_list[index]['settings']['index']['routing']['allocation'][allocation_type][key] == value ) except KeyError: has_routing = False # if has_routing: msg = ( '{0}: Routing (mis)match: ' 'index.routing.allocation.{1}.{2}={3}.'.format( index, allocation_type, key, value ) ) # self.indices.remove(index) self.__excludify(has_routing, exclude, index, msg) def filter_none(self): self.loggit.debug('"None" filter selected. No filtering will be done.') def filter_by_alias(self, aliases=None, exclude=False): """ Match indices which are associated with the alias identified by `name` :arg aliases: A list of alias names. :type aliases: list :arg exclude: If `exclude` is `True`, this filter will remove matching indices from `indices`. If `exclude` is `False`, then only matching indices will be kept in `indices`. Default is `False` """ self.loggit.debug( 'Filtering indices matching aliases: "{0}"'.format(aliases)) if not aliases: raise MissingArgument('No value for "aliases" provided') aliases = ensure_list(aliases) self.empty_list_check() index_lists = chunk_index_list(self.indices) for l in index_lists: try: # get_alias will either return {} or a NotFoundError. has_alias = list(self.client.indices.get_alias( index=to_csv(l), name=to_csv(aliases) ).keys()) self.loggit.debug('has_alias: {0}'.format(has_alias)) except elasticsearch.exceptions.NotFoundError: # if we see the NotFoundError, we need to set working_list to {} has_alias = [] for index in l: if index in has_alias: isOrNot = 'is' condition = True else: isOrNot = 'is not' condition = False msg = ( '{0} {1} associated with aliases: {2}'.format( index, isOrNot, aliases ) ) self.__excludify(condition, exclude, index, msg) def filter_by_count( self, count=None, reverse=True, use_age=False, source='creation_date', timestring=None, field=None, stats_result='min_value', exclude=True): """ Remove indices from the actionable list beyond the number `count`, sorted reverse-alphabetically by default. If you set `reverse` to `False`, it will be sorted alphabetically. The default is usually what you will want. If only one kind of index is provided--for example, indices matching ``logstash-%Y.%m.%d``--then reverse alphabetical sorting will mean the oldest will remain in the list, because lower numbers in the dates mean older indices. By setting `reverse` to `False`, then ``index3`` will be deleted before ``index2``, which will be deleted before ``index1`` `use_age` allows ordering indices by age. Age is determined by the index creation date by default, but you can specify an `source` of ``name``, ``max_value``, or ``min_value``. The ``name`` `source` requires the timestring argument. :arg count: Filter indices beyond `count`. :arg reverse: The filtering direction. (default: `True`). :arg use_age: Sort indices by age. ``source`` is required in this case. :arg source: Source of index age. Can be one of ``name``, ``creation_date``, or ``field_stats``. Default: ``creation_date`` :arg timestring: An strftime string to match the datestamp in an index name. Only used if `source` ``name`` is selected. :arg field: A timestamp field name. Only used if `source` ``field_stats`` is selected. :arg stats_result: Either `min_value` or `max_value`. Only used if `source` ``field_stats`` is selected. It determines whether to reference the minimum or maximum value of `field` in each index. :arg exclude: If `exclude` is `True`, this filter will remove matching indices from `indices`. If `exclude` is `False`, then only matching indices will be kept in `indices`. Default is `True` """ self.loggit.debug('Filtering indices by count') if not count: raise MissingArgument('No value for "count" provided') # Create a copy-by-value working list working_list = self.working_list() if use_age: if source is not 'name': self.loggit.warn( 'Cannot get age information from closed indices unless ' 'source="name". Omitting any closed indices.' ) self.filter_closed() self._calculate_ages( source=source, timestring=timestring, field=field, stats_result=stats_result ) # Using default value of reverse=True in self._sort_by_age() sorted_indices = self._sort_by_age(working_list, reverse=reverse) else: # Default to sorting by index name sorted_indices = sorted(working_list, reverse=reverse) idx = 1 for index in sorted_indices: msg = ( '{0} is {1} of specified count of {2}.'.format( index, idx, count ) ) condition = True if idx <= count else False self.__excludify(condition, exclude, index, msg) idx += 1 def iterate_filters(self, filter_dict): """ Iterate over the filters defined in `config` and execute them. :arg filter_dict: The configuration dictionary .. note:: `filter_dict` should be a dictionary with the following form: .. code-block:: python { 'filters' : [ { 'filtertype': 'the_filter_type', 'key1' : 'value1', ... 'keyN' : 'valueN' } ] } """ self.loggit.debug('Iterating over a list of filters') # Make sure we actually _have_ filters to act on if not 'filters' in filter_dict or len(filter_dict['filters']) < 1: logger.info('No filters in config. Returning unaltered object.') return self.loggit.debug('All filters: {0}'.format(filter_dict['filters'])) for f in filter_dict['filters']: self.loggit.debug('Top of the loop: {0}'.format(self.indices)) self.loggit.debug('Un-parsed filter args: {0}'.format(f)) # Make sure we got at least this much in the configuration self.loggit.debug('Parsed filter args: {0}'.format( SchemaCheck( f, filters.structure(), 'filter', 'IndexList.iterate_filters' ).result() ) ) method = self.__map_method(f['filtertype']) del f['filtertype'] # If it's a filtertype with arguments, update the defaults with the # provided settings. if f: logger.debug('Filter args: {0}'.format(f)) logger.debug('Pre-instance: {0}'.format(self.indices)) method(**f) logger.debug('Post-instance: {0}'.format(self.indices)) else: # Otherwise, it's a settingless filter. method() curator-4.2.5/curator/logtools.py000066400000000000000000000046541302700154000170750ustar00rootroot00000000000000import sys import json import logging import time class LogstashFormatter(logging.Formatter): # The LogRecord attributes we want to carry over to the Logstash message, # mapped to the corresponding output key. WANTED_ATTRS = {'levelname': 'loglevel', 'funcName': 'function', 'lineno': 'linenum', 'message': 'message', 'name': 'name'} # def converter(self, timevalue): # return time.gmtime(timevalue) def format(self, record): self.converter = time.gmtime timestamp = '%s.%03dZ' % ( self.formatTime(record, datefmt='%Y-%m-%dT%H:%M:%S'), record.msecs) result = {'message': record.getMessage(), '@timestamp': timestamp} for attribute in set(self.WANTED_ATTRS).intersection(record.__dict__): result[self.WANTED_ATTRS[attribute]] = getattr(record, attribute) return json.dumps(result, sort_keys=True) class Whitelist(logging.Filter): def __init__(self, *whitelist): self.whitelist = [logging.Filter(name) for name in whitelist] def filter(self, record): return any(f.filter(record) for f in self.whitelist) class Blacklist(Whitelist): def filter(self, record): return not Whitelist.filter(self, record) class LogInfo(object): def __init__(self, cfg): cfg['loglevel'] = 'INFO' if not 'loglevel' in cfg else cfg['loglevel'] cfg['logfile'] = None if not 'logfile' in cfg else cfg['logfile'] cfg['logformat'] = 'default' if not 'logformat' in cfg else cfg['logformat'] self.numeric_log_level = getattr(logging, cfg['loglevel'].upper(), None) self.format_string = '%(asctime)s %(levelname)-9s %(message)s' if not isinstance(self.numeric_log_level, int): raise ValueError('Invalid log level: {0}'.format(cfg['loglevel'])) self.handler = logging.StreamHandler( open(cfg['logfile'], 'a') if cfg['logfile'] else sys.stdout ) if self.numeric_log_level == 10: # DEBUG self.format_string = ( '%(asctime)s %(levelname)-9s %(name)22s ' '%(funcName)22s:%(lineno)-4d %(message)s' ) if cfg['logformat'] == 'json' or cfg['logformat'] == 'logstash': self.handler.setFormatter(LogstashFormatter()) else: self.handler.setFormatter(logging.Formatter(self.format_string)) curator-4.2.5/curator/repomgrcli.py000066400000000000000000000116411302700154000173700ustar00rootroot00000000000000import elasticsearch import click import re import sys import logging from .defaults import settings from .exceptions import * from .config_utils import process_config from .utils import * from ._version import __version__ logger = logging.getLogger('curator.repomgrcli') def delete_callback(ctx, param, value): if not value: ctx.abort() def show_repos(client): for repository in sorted(get_repository(client, '_all').keys()): print('{0}'.format(repository)) sys.exit(0) @click.command(short_help='Filesystem Repository') @click.option('--repository', required=True, type=str, help='Repository name') @click.option( '--location', required=True, type=str, help=( 'Shared file-system location. ' 'Must match remote path, & be accessible to all master & data nodes' ) ) @click.option('--compression', type=bool, default=True, show_default=True, help='Enable/Disable metadata compression.') @click.option('--chunk_size', type=str, help='Chunk size, e.g. 1g, 10m, 5k. [unbounded]') @click.option('--max_restore_bytes_per_sec', type=str, default='20mb', show_default=True, help='Throttles per node restore rate (per second).') @click.option('--max_snapshot_bytes_per_sec', type=str, default='20mb', show_default=True, help='Throttles per node snapshot rate (per second).') @click.pass_context def fs( ctx, repository, location, compression, chunk_size, max_restore_bytes_per_sec, max_snapshot_bytes_per_sec): """ Create a filesystem repository. """ logger = logging.getLogger('curator.repomgrcli.fs') client = get_client(**ctx.obj['client_args']) try: create_repository(client, repo_type='fs', **ctx.params) except FailedExecution as e: logger.critical(e) sys.exit(1) @click.command(short_help='S3 Repository') @click.option('--repository', required=True, type=str, help='Repository name') @click.option('--bucket', required=True, type=str, help='S3 bucket name') @click.option('--region', type=str, help='S3 region. [US Standard]') @click.option('--base_path', type=str, help='S3 base path. [root]') @click.option('--access_key', type=str, help='S3 access key. [value of cloud.aws.access_key]') @click.option('--secret_key', type=str, help='S3 secret key. [value of cloud.aws.secret_key]') @click.option('--compression', type=bool, default=True, show_default=True, help='Enable/Disable metadata compression.') @click.option('--chunk_size', type=str, help='Chunk size, e.g. 1g, 10m, 5k. [unbounded]') @click.option('--max_restore_bytes_per_sec', type=str, default='20mb', show_default=True, help='Throttles per node restore rate (per second).') @click.option('--max_snapshot_bytes_per_sec', type=str, default='20mb', show_default=True, help='Throttles per node snapshot rate (per second).') @click.pass_context def s3( ctx, repository, bucket, region, base_path, access_key, secret_key, compression, chunk_size, max_restore_bytes_per_sec, max_snapshot_bytes_per_sec): """ Create an S3 repository. """ logger = logging.getLogger('curator.repomgrcli.s3') client = get_client(**ctx.obj['client_args']) try: create_repository(client, repo_type='s3', **ctx.params) except FailedExecution as e: logger.critical(e) sys.exit(1) @click.group() @click.option( '--config', help="Path to configuration file. Default: ~/.curator/curator.yml", type=click.Path(exists=True), default=settings.config_file() ) @click.pass_context def repo_mgr_cli(ctx, config): """ Repository manager for Elasticsearch Curator. """ ctx.obj = {} ctx.obj['client_args'] = process_config(config) logger = logging.getLogger(__name__) logger.debug('Client and logging options validated.') @repo_mgr_cli.group('create') @click.pass_context def _create(ctx): """Create an Elasticsearch repository""" _create.add_command(fs) _create.add_command(s3) @repo_mgr_cli.command('show') @click.pass_context def show(ctx): """ Show all repositories """ client = get_client(**ctx.obj['client_args']) show_repos(client) @repo_mgr_cli.command('delete') @click.option('--repository', required=True, help='Repository name', type=str) @click.option('--yes', is_flag=True, callback=delete_callback, expose_value=False, prompt='Are you sure you want to delete the repository?') @click.pass_context def _delete(ctx, repository): """Delete an Elasticsearch repository""" client = get_client(**ctx.obj['client_args']) try: logger.info('Deleting repository {0}...'.format(repository)) client.snapshot.delete_repository(repository=repository) except elasticsearch.NotFoundError: logger.error( 'Unable to delete repository: {0} Not Found.'.format(repository)) sys.exit(1) curator-4.2.5/curator/singletons.py000066400000000000000000000557221302700154000174220ustar00rootroot00000000000000import os, sys import yaml, json import logging import click from voluptuous import Schema from .defaults import settings from .validators import SchemaCheck, config_file, options from .config_utils import test_config, set_logging from .exceptions import * from .utils import * from .indexlist import IndexList from .snapshotlist import SnapshotList from .actions import * from ._version import __version__ CLASS_MAP = { 'alias' : Alias, 'allocation' : Allocation, 'close' : Close, 'cluster_routing' : ClusterRouting, 'create_index' : CreateIndex, 'delete_indices' : DeleteIndices, 'delete_snapshots' : DeleteSnapshots, 'forcemerge' : ForceMerge, 'open' : Open, 'replicas' : Replicas, 'restore' : Restore, 'snapshot' : Snapshot, } EXCLUDED_OPTIONS = [ 'ignore_empty_list', 'timeout_override', 'continue_if_exception', 'disable_action' ] def validate_filter_json(ctx, param, value): try: filter_list = ensure_list(json.loads(value)) return filter_list except ValueError: raise click.BadParameter('Invalid JSON: {0}'.format(value)) def false_to_none(ctx, param, value): try: if value: return True else: return None except ValueError: raise click.BadParameter('Invalid value: {0}'.format(value)) def filter_schema_check(action, filter_dict): valid_filters = SchemaCheck( filter_dict, Schema(filters.Filters(action, location='singleton')), 'filters', '{0} singleton action "filters"'.format(action) ).result() return validate_filters(action, valid_filters) def _actionator(action, action_obj, dry_run=True): logger = logging.getLogger(__name__) logger.debug('Doing the singleton "{0}" action here.'.format(action)) try: if dry_run: action_obj.do_dry_run() else: action_obj.do_action() except Exception as e: if str(type(e)) == "" or \ str(type(e)) == "": logger.error( 'Unable to complete action "{0}". No actionable items ' 'in list: {1}'.format(action, type(e)) ) else: logger.error( 'Failed to complete action: {0}. {1}: ' '{2}'.format(action, type(e), e) ) sys.exit(1) logger.info('Singleton "{0}" action completed.'.format(action)) def _do_filters(list_object, filters, ignore=False): logger = logging.getLogger(__name__) logger.debug('Running filters and testing for empty list object') try: list_object.iterate_filters(filters) list_object.empty_list_check() except (NoIndices, NoSnapshots) as e: if str(type(e)) == "": otype = 'index' else: otype = 'snapshot' if ignore: logger.info( 'Singleton action not performed: empty {0} list'.format(otype) ) sys.exit(0) else: logger.error( 'Singleton action failed due to empty {0} list'.format(otype) ) sys.exit(1) def _prune_excluded(option_dict): for k in list(option_dict.keys()): if k in EXCLUDED_OPTIONS: del option_dict[k] return option_dict def option_schema_check(action, option_dict): clean_options = SchemaCheck( prune_nones(option_dict), options.get_schema(action), 'options', '{0} singleton action "options"'.format(action) ).result() return _prune_excluded(clean_options) def config_override(ctx, config_dict): if config_dict == None: config_dict = {} for k in ['client', 'logging']: if not k in config_dict: config_dict[k] = {} for k in list(ctx.params.keys()): if k in ['dry_run', 'config']: pass elif k == 'host': if 'host' in ctx.params and ctx.params['host'] is not None: config_dict['client']['hosts'] = ctx.params[k] elif k in ['loglevel', 'logfile', 'logformat']: if k in ctx.params and ctx.params[k] is not None: config_dict['logging'][k] = ctx.params[k] else: if k in ctx.params and ctx.params[k] is not None: config_dict['client'][k] = ctx.params[k] # After override, prune the nones for k in ['client', 'logging']: config_dict[k] = prune_nones(config_dict[k]) return SchemaCheck(config_dict, config_file.client(), 'Client Configuration', 'full configuration dictionary').result() @click.command(name='allocation') @click.option( '--key', type=str, required=True, help='Node identification tag' ) @click.option( '--value', type=str, required=True, help='Value associated with --key' ) @click.option( '--allocation_type', type=str, help='Must be one of: require, include, or exclude' ) @click.option( '--wait_for_completion', is_flag=True, help='Wait for operation to complete' ) @click.option( '--ignore_empty_list', is_flag=True, help='Do not raise exception if there are no actionable indices' ) @click.option( '--filter_list', callback=validate_filter_json, help='JSON string representing an array of filters.', required=True ) @click.pass_context def allocation_singleton( ctx, key, value, allocation_type, wait_for_completion, ignore_empty_list, filter_list): """ Shard Routing Allocation """ action = 'allocation' action_class = CLASS_MAP[action] c_args = ctx.obj['config']['client'] client = get_client(**c_args) logger = logging.getLogger(__name__) raw_options = { 'key': key, 'value': value, 'allocation_type': allocation_type, 'wait_for_completion': wait_for_completion, } logger.debug('Validating provided options: {0}'.format(raw_options)) mykwargs = option_schema_check(action, raw_options) mykwargs.update( { 'timeout': c_args['timeout'] if c_args['timeout'] else 30 } ) logger.debug('Validating provided filters: {0}'.format(filter_list)) clean_filters = { 'filters': filter_schema_check(action, filter_list) } ilo = IndexList(client) _do_filters(ilo, clean_filters, ignore_empty_list) action_obj = action_class(ilo, **mykwargs) ### Do the action _actionator(action, action_obj, dry_run=ctx.parent.params['dry_run']) @click.command(name='close') @click.option( '--delete_aliases', is_flag=True, help='Delete all aliases from indices to be closed' ) @click.option( '--ignore_empty_list', is_flag=True, help='Do not raise exception if there are no actionable indices' ) @click.option( '--filter_list', callback=validate_filter_json, help='JSON string representing an array of filters.', required=True ) @click.pass_context def close_singleton( ctx, delete_aliases, ignore_empty_list, filter_list): """ Close indices """ action = 'close' action_class = CLASS_MAP[action] c_args = ctx.obj['config']['client'] client = get_client(**c_args) logger = logging.getLogger(__name__) raw_options = { 'delete_aliases': delete_aliases } logger.debug('Validating provided options: {0}'.format(raw_options)) mykwargs = option_schema_check(action, raw_options) logger.debug('Validating provided filters: {0}'.format(filter_list)) clean_filters = { 'filters': filter_schema_check(action, filter_list) } ilo = IndexList(client) _do_filters(ilo, clean_filters, ignore_empty_list) action_obj = action_class(ilo, **mykwargs) ### Do the action _actionator(action, action_obj, dry_run=ctx.parent.params['dry_run']) @click.command(name='delete_indices') @click.option( '--ignore_empty_list', is_flag=True, help='Do not raise exception if there are no actionable indices' ) @click.option( '--filter_list', callback=validate_filter_json, help='JSON string representing an array of filters.', required=True ) @click.pass_context def delete_indices_singleton(ctx, ignore_empty_list, filter_list): """ Delete indices """ action = 'delete_indices' action_class = CLASS_MAP[action] c_args = ctx.obj['config']['client'] client = get_client(**c_args) logger = logging.getLogger(__name__) mykwargs = { 'master_timeout': c_args['timeout'] if c_args['timeout'] <= 300 else 300 } logger.debug('Validating provided filters: {0}'.format(filter_list)) clean_filters = { 'filters': filter_schema_check(action, filter_list) } ilo = IndexList(client) _do_filters(ilo, clean_filters, ignore_empty_list) action_obj = action_class(ilo, **mykwargs) ### Do the action _actionator(action, action_obj, dry_run=ctx.parent.params['dry_run']) @click.command(name='delete_snapshots') @click.option( '--repository', type=str, required=True, help='Snapshot repository name' ) @click.option( '--retry_count', type=int, help='Number of times to retry (max 3)' ) @click.option( '--retry_interval', type=int, help='Time in seconds between retries' ) @click.option( '--ignore_empty_list', is_flag=True, help='Do not raise exception if there are no actionable snapshots' ) @click.option( '--filter_list', callback=validate_filter_json, help='JSON string representing an array of filters.', required=True ) @click.pass_context def delete_snapshots_singleton( ctx, repository, retry_count, retry_interval, ignore_empty_list, filter_list): """ Delete snapshots """ action = 'delete_snapshots' action_class = CLASS_MAP[action] c_args = ctx.obj['config']['client'] client = get_client(**c_args) logger = logging.getLogger(__name__) raw_options = { 'repository': repository, 'retry_count': retry_count, 'retry_interval': retry_interval } logger.debug('Validating provided options: {0}'.format(raw_options)) mykwargs = option_schema_check(action, raw_options) # Repo arg Not necessary after schema check. It's only for the slo object del mykwargs['repository'] logger.debug('Validating provided filters: {0}'.format(filter_list)) clean_filters = { 'filters': filter_schema_check(action, filter_list) } slo = SnapshotList(client, repository=repository) _do_filters(slo, clean_filters, ignore_empty_list) action_obj = action_class(slo, **mykwargs) ### Do the action _actionator(action, action_obj, dry_run=ctx.parent.params['dry_run']) @click.command(name='open') @click.option( '--ignore_empty_list', is_flag=True, help='Do not raise exception if there are no actionable indices' ) @click.option( '--filter_list', callback=validate_filter_json, help='JSON string representing an array of filters.', required=True ) @click.pass_context def open_singleton( ctx, ignore_empty_list, filter_list): """ Open indices """ action = 'open' action_class = CLASS_MAP[action] c_args = ctx.obj['config']['client'] client = get_client(**c_args) logger = logging.getLogger(__name__) logger.debug('Validating provided filters: {0}'.format(filter_list)) clean_filters = { 'filters': filter_schema_check(action, filter_list) } ilo = IndexList(client) _do_filters(ilo, clean_filters, ignore_empty_list) action_obj = action_class(ilo) ### Do the action _actionator(action, action_obj, dry_run=ctx.parent.params['dry_run']) @click.command(name='forcemerge') @click.option( '--max_num_segments', type=int, required=True, help='Maximum number of segments per shard (minimum of 1)' ) @click.option( '--delay', type=float, help='Time in seconds to delay between operations. Default 0, maximum 3600' ) @click.option( '--ignore_empty_list', is_flag=True, help='Do not raise exception if there are no actionable indices' ) @click.option( '--filter_list', callback=validate_filter_json, help='JSON string representing an array of filters.', required=True ) @click.pass_context def forcemerge_singleton( ctx, max_num_segments, delay, ignore_empty_list, filter_list): """ forceMerge index/shard segments """ action = 'forcemerge' action_class = CLASS_MAP[action] c_args = ctx.obj['config']['client'] client = get_client(**c_args) logger = logging.getLogger(__name__) raw_options = { 'max_num_segments': max_num_segments, 'delay': delay, } logger.debug('Validating provided options: {0}'.format(raw_options)) mykwargs = option_schema_check(action, raw_options) logger.debug('Validating provided filters: {0}'.format(filter_list)) clean_filters = { 'filters': filter_schema_check(action, filter_list) } ilo = IndexList(client) _do_filters(ilo, clean_filters, ignore_empty_list) action_obj = action_class(ilo, **mykwargs) ### Do the action _actionator(action, action_obj, dry_run=ctx.parent.params['dry_run']) @click.command(name='replicas') @click.option( '--count', type=int, required=True, help='Number of replicas (max 10)' ) @click.option( '--wait_for_completion', is_flag=True, help='Wait for operation to complete' ) @click.option( '--ignore_empty_list', is_flag=True, help='Do not raise exception if there are no actionable indices' ) @click.option( '--filter_list', callback=validate_filter_json, help='JSON string representing an array of filters.', required=True ) @click.pass_context def replicas_singleton( ctx, count, wait_for_completion, ignore_empty_list, filter_list): """ Change replica count """ action = 'replicas' action_class = CLASS_MAP[action] c_args = ctx.obj['config']['client'] client = get_client(**c_args) logger = logging.getLogger(__name__) raw_options = { 'count': count, 'wait_for_completion': wait_for_completion, } logger.debug('Validating provided options: {0}'.format(raw_options)) mykwargs = option_schema_check(action, raw_options) logger.debug('Validating provided filters: {0}'.format(filter_list)) clean_filters = { 'filters': filter_schema_check(action, filter_list) } ilo = IndexList(client) _do_filters(ilo, clean_filters, ignore_empty_list) action_obj = action_class(ilo, **mykwargs) ### Do the action _actionator(action, action_obj, dry_run=ctx.parent.params['dry_run']) @click.command(name='snapshot') @click.option( '--repository', type=str, required=True, help='Snapshot repository') @click.option( '--name', type=str, help='Snapshot name', show_default=True, default='curator-%Y%m%d%H%M%S' ) @click.option( '--ignore_unavailable', is_flag=True, show_default=True, help='Ignore unavailable shards/indices.' ) @click.option( '--include_global_state', type=bool, show_default=True, default=True, expose_value=True, help='Store cluster global state with snapshot.' ) @click.option( '--partial', is_flag=True, show_default=True, help='Do not fail if primary shard is unavailable.' ) @click.option( '--wait_for_completion', type=bool, show_default=True, default=True, help='Wait for operation to complete' ) @click.option( '--skip_repo_fs_check', is_flag=True, expose_value=True, help='Skip repository filesystem access validation.' ) @click.option( '--ignore_empty_list', is_flag=True, help='Do not raise exception if there are no actionable indices' ) @click.option( '--filter_list', callback=validate_filter_json, default='{"filtertype":"none"}', help='JSON string representing an array of filters.' ) @click.pass_context def snapshot_singleton( ctx, repository, name, ignore_unavailable, include_global_state, partial, skip_repo_fs_check, wait_for_completion, ignore_empty_list, filter_list): """ Snapshot indices """ action = 'snapshot' action_class = CLASS_MAP[action] c_args = ctx.obj['config']['client'] client = get_client(**c_args) logger = logging.getLogger(__name__) raw_options = { 'repository': repository, 'name': name, 'ignore_unavailable': ignore_unavailable, 'include_global_state': include_global_state, 'partial': partial, 'skip_repo_fs_check': skip_repo_fs_check, 'wait_for_completion': wait_for_completion, } logger.debug('Validating provided options: {0}'.format(raw_options)) mykwargs = option_schema_check(action, raw_options) logger.debug('Validating provided filters: {0}'.format(filter_list)) clean_filters = { 'filters': filter_schema_check(action, filter_list) } ilo = IndexList(client) _do_filters(ilo, clean_filters, ignore_empty_list) action_obj = action_class(ilo, **mykwargs) ### Do the action _actionator(action, action_obj, dry_run=ctx.parent.params['dry_run']) @click.command(name='show_indices') @click.option('--verbose', help='Show verbose output.', is_flag=True) @click.option('--header', help='Print header if --verbose', is_flag=True) @click.option('--epoch', help='Print time as epoch if --verbose', is_flag=True) @click.option( '--ignore_empty_list', is_flag=True, help='Do not raise exception if there are no actionable indices' ) @click.option( '--filter_list', callback=validate_filter_json, default='{"filtertype":"none"}', help='JSON string representing an array of filters.' ) @click.pass_context def show_indices_singleton( ctx, epoch, header, verbose, ignore_empty_list, filter_list): """ Show indices """ action = "open" c_args = ctx.obj['config']['client'] client = get_client(**c_args) logger = logging.getLogger(__name__) logger.debug( 'Using dummy "open" action for show_indices singleton. ' 'No action will be taken.' ) logger.debug('Validating provided filters: {0}'.format(filter_list)) clean_filters = { 'filters': filter_schema_check(action, filter_list) } ilo = IndexList(client) _do_filters(ilo, clean_filters, ignore_empty_list) indices = sorted(ilo.indices) # Do some calculations to figure out the proper column sizes allbytes = [] alldocs = [] for idx in indices: allbytes.append(byte_size(ilo.index_info[idx]['size_in_bytes'])) alldocs.append(str(ilo.index_info[idx]['docs'])) if epoch: timeformat = '{6:>13}' column = 'creation_date' else: timeformat = '{6:>20}' column = 'Creation Timestamp' formatting = ( '{0:' + str(len(max(indices, key=len))) + '} ' '{1:>5} ' '{2:>' + str(len(max(allbytes, key=len)) + 1) + '} ' '{3:>' + str(len(max(alldocs, key=len)) + 1) + '} ' '{4:>3} {5:>3} ' + timeformat ) # Print the header, if both verbose and header are enabled if header and verbose: click.secho( formatting.format( 'Index', 'State', 'Size', 'Docs', 'Pri', 'Rep', column ), bold=True, underline=True ) # Loop through indices and print info, if verbose for idx in indices: p = ilo.index_info[idx] if verbose: if epoch: datefield = p['age']['creation_date'] if 'creation_date' in p['age'] else 0 else: datefield = '{0}Z'.format( datetime.utcfromtimestamp(p['age']['creation_date'] ).isoformat()) if 'creation_date' in p['age'] else 'unknown/closed' click.echo( formatting.format( idx, p['state'], byte_size(p['size_in_bytes']), p['docs'], p['number_of_shards'], p['number_of_replicas'], datefield ) ) else: click.echo('{0}'.format(idx)) @click.command(name='show_snapshots') @click.option( '--repository', type=str, required=True, help='Snapshot repository name' ) @click.option( '--ignore_empty_list', is_flag=True, help='Do not raise exception if there are no actionable snapshots' ) @click.option( '--filter_list', callback=validate_filter_json, default='{"filtertype":"none"}', help='JSON string representing an array of filters.' ) @click.pass_context def show_snapshots_singleton( ctx, repository, ignore_empty_list, filter_list): """ Show snapshots """ action = 'delete_snapshots' c_args = ctx.obj['config']['client'] client = get_client(**c_args) logger = logging.getLogger(__name__) logger.debug('Validating provided filters: {0}'.format(filter_list)) clean_filters = { 'filters': filter_schema_check(action, filter_list) } slo = SnapshotList(client, repository=repository) _do_filters(slo, clean_filters, ignore_empty_list) snapshots = sorted(slo.snapshots) for idx in snapshots: click.secho('{0}'.format(idx)) @click.group() @click.option( '--config', help='Path to configuration file. Default: ~/.curator/curator.yml', type=click.Path(), default=settings.config_file() ) @click.option('--host', help='Elasticsearch host.') @click.option('--url_prefix', help='Elasticsearch http url prefix.') @click.option('--port', help='Elasticsearch port.') @click.option( '--use_ssl', is_flag=True, callback=false_to_none, help='Connect to Elasticsearch through SSL.' ) @click.option( '--certificate', help='Path to certificate to use for SSL validation.') @click.option( '--client-cert', help='Path to file containing SSL certificate for client auth.', type=str ) @click.option( '--client-key', help='Path to file containing SSL key for client auth.', type=str ) @click.option( '--ssl-no-validate', is_flag=True, callback=false_to_none, help='Do not validate SSL certificate' ) @click.option('--http_auth', help='Use Basic Authentication ex: user:pass') @click.option('--timeout', help='Connection timeout in seconds.', type=int) @click.option( '--master-only', is_flag=True, callback=false_to_none, help='Only operate on elected master node.' ) @click.option('--dry-run', is_flag=True, help='Do not perform any changes.') @click.option('--loglevel', help='Log level') @click.option('--logfile', help='log file') @click.option('--logformat', help='Log output format [default|logstash|json].') @click.version_option(version=__version__) @click.pass_context def cli( ctx, config, host, url_prefix, port, use_ssl, certificate, client_cert, client_key, ssl_no_validate, http_auth, timeout, master_only, dry_run, loglevel, logfile, logformat): if os.path.isfile(config): initial_config = test_config(config) else: initial_config = None configuration = config_override(ctx, initial_config) set_logging(configuration['logging']) test_client_options(configuration['client']) logger = logging.getLogger(__name__) ctx.obj['config'] = configuration cli.add_command(allocation_singleton) cli.add_command(close_singleton) cli.add_command(delete_indices_singleton) cli.add_command(delete_snapshots_singleton) cli.add_command(forcemerge_singleton) cli.add_command(open_singleton) cli.add_command(replicas_singleton) cli.add_command(snapshot_singleton) cli.add_command(show_indices_singleton) cli.add_command(show_snapshots_singleton) curator-4.2.5/curator/snapshotlist.py000066400000000000000000000435451302700154000177700ustar00rootroot00000000000000from datetime import timedelta, datetime, date import time import re import logging from .defaults import settings from .validators import SchemaCheck, filters from .exceptions import * from .utils import * class SnapshotList(object): def __init__(self, client, repository=None): verify_client_object(client) if not repository: raise MissingArgument('No value for "repository" provided') if not repository_exists(client, repository): raise FailedExecution( 'Unable to verify existence of repository ' '{0}'.format(repository) ) self.loggit = logging.getLogger('curator.snapshotlist') #: An Elasticsearch Client object. #: Also accessible as an instance variable. self.client = client #: An Elasticsearch repository. #: Also accessible as an instance variable. self.repository = repository #: Instance variable. #: Information extracted from snapshots, such as age, etc. #: Populated by internal method `__get_snapshots` at instance creation #: time. **Type:** ``dict()`` self.snapshot_info = {} #: Instance variable. #: The running list of snapshots which will be used by an Action class. #: Populated by internal methods `__get_snapshots` at instance creation #: time. **Type:** ``list()`` self.snapshots = [] #: Instance variable. #: Raw data dump of all snapshots in the repository at instance creation #: time. **Type:** ``list()`` of ``dict()`` data. self.__get_snapshots() def __actionable(self, snap): self.loggit.debug( 'Snapshot {0} is actionable and remains in the list.'.format(snap)) def __not_actionable(self, snap): self.loggit.debug( 'Snapshot {0} is not actionable, removing from ' 'list.'.format(snap) ) self.snapshots.remove(snap) def __excludify(self, condition, exclude, snap, msg=None): if condition == True: if exclude: text = "Removed from actionable list" self.__not_actionable(snap) else: text = "Remains in actionable list" self.__actionable(snap) else: if exclude: text = "Remains in actionable list" self.__actionable(snap) else: text = "Removed from actionable list" self.__not_actionable(snap) if msg: self.loggit.debug('{0}: {1}'.format(text, msg)) def __get_snapshots(self): """ Pull all snapshots into `snapshots` and populate `snapshot_info` """ self.all_snapshots = get_snapshot_data(self.client, self.repository) for list_item in self.all_snapshots: if 'snapshot' in list_item.keys(): self.snapshots.append(list_item['snapshot']) self.snapshot_info[list_item['snapshot']] = list_item self.empty_list_check() def __map_method(self, ft): methods = { 'age': self.filter_by_age, 'count': self.filter_by_count, 'none': self.filter_none, 'pattern': self.filter_by_regex, 'state': self.filter_by_state, } return methods[ft] def empty_list_check(self): """Raise exception if `snapshots` is empty""" if not self.snapshots: raise NoSnapshots('snapshot_list object is empty.') def working_list(self): """ Return the current value of `snapshots` as copy-by-value to prevent list stomping during iterations """ # Copy by value, rather than reference to prevent list stomping during # iterations return self.snapshots[:] def _get_name_based_ages(self, timestring): """ Add a snapshot age to `snapshot_info` based on the age as indicated by the snapshot name pattern, if it matches `timestring`. This is stored at key ``age_by_name``. :arg timestring: An strftime pattern """ # Check for empty list before proceeding here to prevent non-iterable # condition self.empty_list_check() ts = TimestringSearch(timestring) for snapshot in self.working_list(): epoch = ts.get_epoch(snapshot) if epoch: self.snapshot_info[snapshot]['age_by_name'] = epoch else: self.snapshot_info[snapshot]['age_by_name'] = None def _calculate_ages(self, source='creation_date', timestring=None): """ This method initiates snapshot age calculation based on the given parameters. Exceptions are raised when they are improperly configured. Set instance variable `age_keyfield` for use later, if needed. :arg source: Source of snapshot age. Can be 'name' or 'creation_date'. :arg timestring: An strftime string to match the datestamp in an snapshot name. Only used if ``source`` is ``name``. """ if source == 'name': self.age_keyfield = 'age_by_name' if not timestring: raise MissingArgument( 'source "name" requires the "timestring" keyword argument' ) self._get_name_based_ages(timestring) elif source == 'creation_date': self.age_keyfield = 'start_time_in_millis' else: raise ValueError( 'Invalid source: {0}. ' 'Must be "name", or "creation_date".'.format(source) ) def _sort_by_age(self, snapshot_list, reverse=True): """ Take a list of snapshots and sort them by date. By default, the youngest are first with `reverse=True`, but the oldest can be first by setting `reverse=False` """ # Do the age-based sorting here. # First, build an temporary dictionary with just snapshot and age # as the key and value, respectively temp = {} for snap in snapshot_list: if self.age_keyfield in self.snapshot_info[snap]: temp[snap] = self.snapshot_info[snap][self.age_keyfield] else: msg = ( '{0} does not have age key "{1}" in SnapshotList ' ' metadata'.format(snap, self.age_keyfield) ) self.__excludify(True, True, snap, msg) # If reverse is True, this will sort so the youngest snapshots are # first. However, if you want oldest first, set reverse to False. # Effectively, this should set us up to act on everything older than # meets the other set criteria. # It starts as a tuple, but then becomes a list. sorted_tuple = ( sorted(temp.items(), key=lambda k: k[1], reverse=reverse) ) return [x[0] for x in sorted_tuple] def most_recent(self): """ Return the most recent snapshot based on `start_time_in_millis`. """ self.empty_list_check() most_recent_time = 0 most_recent_snap = '' for snapshot in self.snapshots: snaptime = fix_epoch( self.snapshot_info[snapshot]['start_time_in_millis']) if snaptime > most_recent_time: most_recent_snap = snapshot most_recent_time = snaptime return most_recent_snap def filter_by_regex(self, kind=None, value=None, exclude=False): """ Filter out snapshots not matching the pattern, or in the case of exclude, filter those matching the pattern. :arg kind: Can be one of: ``suffix``, ``prefix``, ``regex``, or ``timestring``. This option defines what kind of filter you will be building. :arg value: Depends on `kind`. It is the strftime string if `kind` is `timestring`. It's used to build the regular expression for other kinds. :arg exclude: If `exclude` is `True`, this filter will remove matching snapshots from `snapshots`. If `exclude` is `False`, then only matching snapshots will be kept in `snapshots`. Default is `False` """ if kind not in [ 'regex', 'prefix', 'suffix', 'timestring' ]: raise ValueError('{0}: Invalid value for kind'.format(kind)) # Stop here if None or empty value, but zero is okay if value == 0: pass elif not value: raise ValueError( '{0}: Invalid value for "value". ' 'Cannot be "None" type, empty, or False' ) if kind == 'timestring': regex = settings.regex_map()[kind].format(get_date_regex(value)) else: regex = settings.regex_map()[kind].format(value) self.empty_list_check() pattern = re.compile(regex) for snapshot in self.working_list(): match = pattern.match(snapshot) self.loggit.debug('Filter by regex: Snapshot: {0}'.format(snapshot)) if match: self.__excludify(True, exclude, snapshot) else: self.__excludify(False, exclude, snapshot) def filter_by_age(self, source='creation_date', direction=None, timestring=None, unit=None, unit_count=None, epoch=None, exclude=False ): """ Remove snapshots from `snapshots` by relative age calculations. :arg source: Source of snapshot age. Can be 'name', or 'creation_date'. :arg direction: Time to filter, either ``older`` or ``younger`` :arg timestring: An strftime string to match the datestamp in an snapshot name. Only used for snapshot filtering by ``name``. :arg unit: One of ``seconds``, ``minutes``, ``hours``, ``days``, ``weeks``, ``months``, or ``years``. :arg unit_count: The number of ``unit``s. ``unit_count`` * ``unit`` will be calculated out to the relative number of seconds. :arg epoch: An epoch timestamp used in conjunction with ``unit`` and ``unit_count`` to establish a point of reference for calculations. If not provided, the current time will be used. :arg exclude: If `exclude` is `True`, this filter will remove matching snapshots from `snapshots`. If `exclude` is `False`, then only matching snapshots will be kept in `snapshots`. Default is `False` """ self.loggit.debug('Starting filter_by_age') # Get timestamp point of reference, PoR PoR = get_point_of_reference(unit, unit_count, epoch) self.loggit.debug('Point of Reference: {0}'.format(PoR)) if not direction: raise MissingArgument('Must provide a value for "direction"') if direction not in ['older', 'younger']: raise ValueError( 'Invalid value for "direction": {0}'.format(direction) ) self._calculate_ages(source=source, timestring=timestring) for snapshot in self.working_list(): if not self.snapshot_info[snapshot][self.age_keyfield]: self.loggit.debug('Removing snapshot {0} for having no age') self.snapshots.remove(snapshot) continue msg = ( 'Snapshot "{0}" age ({1}), direction: "{2}", point of ' 'reference, ({3})'.format( snapshot, fix_epoch(self.snapshot_info[snapshot][self.age_keyfield]), direction, PoR ) ) # Because time adds to epoch, smaller numbers are actually older # timestamps. snapshot_age = fix_epoch( self.snapshot_info[snapshot][self.age_keyfield]) if direction == 'older': agetest = snapshot_age < PoR else: # 'younger' agetest = snapshot_age > PoR self.__excludify(agetest, exclude, snapshot, msg) def filter_by_state(self, state=None, exclude=False): """ Filter out snapshots not matching ``state``, or in the case of exclude, filter those matching ``state``. :arg state: The snapshot state to filter for. Must be one of ``SUCCESS``, ``PARTIAL``, ``FAILED``, or ``IN_PROGRESS``. :arg exclude: If `exclude` is `True`, this filter will remove matching snapshots from `snapshots`. If `exclude` is `False`, then only matching snapshots will be kept in `snapshots`. Default is `False` """ if state.upper() not in ['SUCCESS', 'PARTIAL', 'FAILED', 'IN_PROGRESS']: raise ValueError('{0}: Invalid value for state'.format(state)) self.empty_list_check() for snapshot in self.working_list(): self.loggit.debug('Filter by state: Snapshot: {0}'.format(snapshot)) if self.snapshot_info[snapshot]['state'] == state: self.__excludify(True, exclude, snapshot) else: self.__excludify(False, exclude, snapshot) def filter_none(self): self.loggit.debug('"None" filter selected. No filtering will be done.') def filter_by_count( self, count=None, reverse=True, use_age=False, source='creation_date', timestring=None, exclude=True ): """ Remove snapshots from the actionable list beyond the number `count`, sorted reverse-alphabetically by default. If you set `reverse` to `False`, it will be sorted alphabetically. The default is usually what you will want. If only one kind of snapshot is provided--for example, snapshots matching ``curator-%Y%m%d%H%M%S``-- then reverse alphabetical sorting will mean the oldest will remain in the list, because lower numbers in the dates mean older snapshots. By setting `reverse` to `False`, then ``snapshot3`` will be acted on before ``snapshot2``, which will be acted on before ``snapshot1`` `use_age` allows ordering snapshots by age. Age is determined by the snapshot creation date (as identified by ``start_time_in_millis``) by default, but you can also specify a `source` of ``name``. The ``name`` `source` requires the timestring argument. :arg count: Filter snapshots beyond `count`. :arg reverse: The filtering direction. (default: `True`). :arg use_age: Sort snapshots by age. ``source`` is required in this case. :arg source: Source of snapshot age. Can be one of ``name``, or ``creation_date``. Default: ``creation_date`` :arg timestring: An strftime string to match the datestamp in a snapshot name. Only used if `source` ``name`` is selected. :arg exclude: If `exclude` is `True`, this filter will remove matching snapshots from `snapshots`. If `exclude` is `False`, then only matching snapshots will be kept in `snapshots`. Default is `True` """ self.loggit.debug('Filtering snapshots by count') if not count: raise MissingArgument('No value for "count" provided') # Create a copy-by-value working list working_list = self.working_list() if use_age: self._calculate_ages(source=source, timestring=timestring) # Using default value of reverse=True in self._sort_by_age() sorted_snapshots = self._sort_by_age(working_list, reverse=reverse) else: # Default to sorting by snapshot name sorted_snapshots = sorted(working_list, reverse=reverse) idx = 1 for snap in sorted_snapshots: msg = ( '{0} is {1} of specified count of {2}.'.format( snap, idx, count ) ) condition = True if idx <= count else False self.__excludify(condition, exclude, snap, msg) idx += 1 def iterate_filters(self, config): """ Iterate over the filters defined in `config` and execute them. :arg config: A dictionary of filters, as extracted from the YAML configuration file. .. note:: `config` should be a dictionary with the following form: .. code-block:: python { 'filters' : [ { 'filtertype': 'the_filter_type', 'key1' : 'value1', ... 'keyN' : 'valueN' } ] } """ # Make sure we actually _have_ filters to act on if not 'filters' in config or len(config['filters']) < 1: logger.info('No filters in config. Returning unaltered object.') return self.loggit.debug('All filters: {0}'.format(config['filters'])) for f in config['filters']: self.loggit.debug('Top of the loop: {0}'.format(self.snapshots)) self.loggit.debug('Un-parsed filter args: {0}'.format(f)) self.loggit.debug('Parsed filter args: {0}'.format( SchemaCheck( f, filters.structure(), 'filter', 'SnapshotList.iterate_filters' ).result() ) ) method = self.__map_method(f['filtertype']) # Remove key 'filtertype' from dictionary 'f' del f['filtertype'] # If it's a filtertype with arguments, update the defaults with the # provided settings. logger.debug('Filter args: {0}'.format(f)) logger.debug('Pre-instance: {0}'.format(self.snapshots)) method(**f) logger.debug('Post-instance: {0}'.format(self.snapshots)) curator-4.2.5/curator/utils.py000066400000000000000000001250761302700154000163750ustar00rootroot00000000000000from datetime import timedelta, datetime, date import elasticsearch import time import logging import yaml, os, re, sys from voluptuous import Schema from .exceptions import * from .defaults import settings from .validators import SchemaCheck, actions, filters, options from ._version import __version__ logger = logging.getLogger(__name__) def read_file(myfile): """ Read a file and return the resulting data. :arg myfile: A file to read. :rtype: str """ try: with open(myfile, 'r') as f: data = f.read() return data except IOError as e: raise FailedExecution( 'Unable to read file {0}'.format(myfile) ) def get_yaml(path): """ Read the file identified by `path` and import its YAML contents. :arg path: The path to a YAML configuration file. :rtype: dict """ # Set the stage here to parse single scalar value environment vars from # the YAML file being read single = re.compile( r'^\$\{(.*)\}$' ) yaml.add_implicit_resolver ( "!single", single ) def single_constructor(loader,node): value = loader.construct_scalar(node) proto = single.match(value).group(1) default = None if len(proto.split(':')) > 1: envvar, default = proto.split(':') else: envvar = proto return os.environ[envvar] if envvar in os.environ else default yaml.add_constructor('!single', single_constructor) raw = read_file(path) try: cfg = yaml.load(raw) except yaml.scanner.ScannerError as e: raise ConfigurationError( 'Unable to parse YAML file. Error: {0}'.format(e)) return cfg def test_client_options(config): """ Test whether a SSL/TLS files exist. Will raise an exception if the files cannot be read. :arg config: A client configuration file data dictionary :rtype: None """ if config['use_ssl']: # Test whether certificate is a valid file path if 'certificate' in config and config['certificate']: read_file(config['certificate']) # Test whether client_cert is a valid file path if 'client_cert' in config and config['client_cert']: read_file(config['client_cert']) # Test whether client_key is a valid file path if 'client_key' in config and config['client_key']: read_file(config['client_key']) def verify_client_object(test): """ Test if `test` is a proper :class:`elasticsearch.Elasticsearch` client object and raise an exception if it is not. :arg test: The variable or object to test :rtype: None """ # Ignore mock type for testing if str(type(test)) == "" or \ str(type(test)) == "": pass elif not type(test) == type(elasticsearch.Elasticsearch()): raise TypeError( 'Not a client object. Type: {0}'.format(type(test)) ) def verify_index_list(test): """ Test if `test` is a proper :class:`curator.indexlist.IndexList` object and raise an exception if it is not. :arg test: The variable or object to test :rtype: None """ if not str(type(test)) == "": raise TypeError( 'Not an IndexList object. Type: {0}.'.format(type(test)) ) def verify_snapshot_list(test): """ Test if `test` is a proper :class:`curator.snapshotlist.SnapshotList` object and raise an exception if it is not. :arg test: The variable or object to test :rtype: None """ if not str(type(test)) == "": raise TypeError( 'Not an SnapshotList object. Type: {0}.'.format(type(test)) ) def report_failure(exception): """ Raise a `FailedExecution` exception and include the original error message. :arg exception: The upstream exception. :rtype: None """ raise FailedExecution( 'Exception encountered. Rerun with loglevel DEBUG and/or check ' 'Elasticsearch logs for more information. ' 'Exception: {0}'.format(exception) ) def get_date_regex(timestring): """ Return a regex string based on a provided strftime timestring. :arg timestring: An strftime pattern :rtype: str """ prev = ''; curr = ''; regex = '' for s in range(0, len(timestring)): curr = timestring[s] if curr == '%': pass elif curr in settings.date_regex() and prev == '%': regex += '\d{' + settings.date_regex()[curr] + '}' elif curr in ['.', '-']: regex += "\\" + curr else: regex += curr prev = curr logger.debug("regex = {0}".format(regex)) return regex def get_datetime(index_timestamp, timestring): """ Return the datetime extracted from the index name, which is the index creation time. :arg index_timestamp: The timestamp extracted from an index name :arg timestring: An strftime pattern :rtype: :py:class:`datetime.datetime` """ # Compensate for week of year by appending '%w' to the timestring # and '1' (Monday) to index_timestamp if '%W' in timestring: timestring += '%w' index_timestamp += '1' elif '%U' in timestring: timestring += '%w' index_timestamp += '1' elif '%m' in timestring: if not '%d' in timestring: timestring += '%d' index_timestamp += '1' return datetime.strptime(index_timestamp, timestring) def fix_epoch(epoch): """ Fix value of `epoch` to be epoch, which should be 10 or fewer digits long. :arg epoch: An epoch timestamp, in epoch + milliseconds, or microsecond, or even nanoseconds. :rtype: int """ # No decimals allowed epoch = int(epoch) # If we're still using this script past January, 2038, we have bigger # problems than my hacky math here... if len(str(epoch)) <= 10: return epoch elif len(str(epoch)) == 13: return int(epoch/1000) elif len(str(epoch)) > 10 and len(str(epoch)) < 13: raise ValueError( 'Unusually formatted epoch timestamp. ' 'Should be 10, 13, or more digits' ) else: orders_of_magnitude = len(str(epoch)) - 10 powers_of_ten = 10**orders_of_magnitude epoch = int(epoch/powers_of_ten) return epoch class TimestringSearch(object): """ An object to allow repetitive search against a string, `searchme`, without having to repeatedly recreate the regex. :arg timestring: An strftime pattern """ def __init__(self, timestring): regex = r'(?P{0})'.format(get_date_regex(timestring)) self.pattern = re.compile(regex) self.timestring = timestring def get_epoch(self, searchme): """ Return the epoch timestamp extracted from the `timestring` appearing in `searchme`. :arg searchme: A string to be searched for a date pattern that matches `timestring` :rtype: int """ match = self.pattern.search(searchme) if match: if match.group("date"): timestamp = match.group("date") # I would have used `total_seconds`, but apparently that's new # to Python 2.7+, and due to so many people still using # RHEL/CentOS 6, I need this to support Python 2.6. tdelta = ( get_datetime(timestamp, self.timestring) - datetime(1970,1,1) ) return tdelta.seconds + tdelta.days * 24 * 3600 def get_point_of_reference(unit, count, epoch=None): """ Get a point-of-reference timestamp in epoch + milliseconds by deriving from a `unit` and a `count`, and an optional reference timestamp, `epoch` :arg unit: One of ``seconds``, ``minutes``, ``hours``, ``days``, ``weeks``, ``months``, or ``years``. :arg unit_count: The number of ``units``. ``unit_count`` * ``unit`` will be calculated out to the relative number of seconds. :arg epoch: An epoch timestamp used in conjunction with ``unit`` and ``unit_count`` to establish a point of reference for calculations. :rtype: int """ if unit == 'seconds': multiplier = 1 elif unit == 'minutes': multiplier = 60 elif unit == 'hours': multiplier = 3600 elif unit == 'days': multiplier = 3600*24 elif unit == 'weeks': multiplier = 3600*24*7 elif unit == 'months': multiplier = 3600*24*30 elif unit == 'years': multiplier = 3600*24*365 else: raise ValueError('Invalid unit: {0}.'.format(unit)) # Use this moment as a reference point, if one is not provided. if not epoch: epoch = time.time() epoch = fix_epoch(epoch) return epoch - multiplier * count def byte_size(num, suffix='B'): """ Return a formatted string indicating the size in bytes, with the proper unit, e.g. KB, MB, GB, TB, etc. :arg num: The number of byte :arg suffix: An arbitrary suffix, like `Bytes` :rtype: float """ for unit in ['','K','M','G','T','P','E','Z']: if abs(num) < 1024.0: return "%3.1f%s%s" % (num, unit, suffix) num /= 1024.0 return "%.1f%s%s" % (num, 'Y', suffix) def ensure_list(indices): """ Return a list, even if indices is a single value :arg indices: A list of indices to act upon :rtype: list """ if type(indices) is not type(list()): # in case of a single value passed indices = [indices] return indices def to_csv(indices): """ Return a csv string from a list of indices, or a single value if only one value is present :arg indices: A list of indices to act on, or a single value, which could be in the format of a csv string already. :rtype: str """ indices = ensure_list(indices) # in case of a single value passed if indices: return ','.join(sorted(indices)) else: return None def check_csv(value): """ Some of the curator methods should not operate against multiple indices at once. This method can be used to check if a list or csv has been sent. :arg value: The value to test, if list or csv string :rtype: bool """ if type(value) is type(list()): return True string = False # Python3 hack because it doesn't recognize unicode as a type anymore if sys.version_info < (3, 0): if type(value) is type(unicode()): value = str(value) if type(value) is type(str()): if len(value.split(',')) > 1: # It's a csv string. return True else: # There's only one value here, so it's not a csv string return False else: raise TypeError( 'Passed value: {0} is not a list or a string ' 'but is of type {1}'.format(value, type(value)) ) def chunk_index_list(indices): """ This utility chunks very large index lists into 3KB chunks It measures the size as a csv string, then converts back into a list for the return value. :arg indices: A list of indices to act on. :rtype: list """ chunks = [] chunk = "" for index in indices: if len(chunk) < 3072: if not chunk: chunk = index else: chunk += "," + index else: chunks.append(chunk.split(',')) chunk = index chunks.append(chunk.split(',')) return chunks def get_indices(client): """ Get the current list of indices from the cluster. :arg client: An :class:`elasticsearch.Elasticsearch` client object :rtype: list """ try: indices = list( client.indices.get_settings( index='_all', params={'expand_wildcards': 'open,closed'}) ) version_number = get_version(client) logger.debug( 'Detected Elasticsearch version ' '{0}'.format(".".join(map(str,version_number))) ) # This hack ONLY works if you're using 2.4.2 or higher, but is unneeded # if you are using 5.0 or higher. See issue #826 if version_number >= (2, 4, 2) \ and version_number < (5, 0, 0): logger.debug('Using Elasticsearch >= 2.4.2 < 5.0.0') if client.indices.exists(index='.security'): logger.debug( 'Found the ".security" index. ' 'Adding to list of all indices' ) # Double check to see if it's there before appending if not '.security' in indices: indices.append('.security') logger.debug("All indices: {0}".format(indices)) return indices except Exception as e: raise FailedExecution('Failed to get indices. Error: {0}'.format(e)) def get_version(client): """ Return the ES version number as a tuple. Omits trailing tags like -dev, or Beta :arg client: An :class:`elasticsearch.Elasticsearch` client object :rtype: tuple """ version = client.info()['version']['number'] version = version.split('-')[0] if len(version.split('.')) > 3: version = version.split('.')[:-1] else: version = version.split('.') return tuple(map(int, version)) def is_master_node(client): """ Return `True` if the connected client node is the elected master node in the Elasticsearch cluster, otherwise return `False`. :arg client: An :class:`elasticsearch.Elasticsearch` client object :rtype: bool """ my_node_id = list(client.nodes.info('_local')['nodes'])[0] master_node_id = client.cluster.state(metric='master_node')['master_node'] return my_node_id == master_node_id def check_version(client): """ Verify version is within acceptable range. Raise an exception if it is not. :arg client: An :class:`elasticsearch.Elasticsearch` client object :rtype: None """ version_number = get_version(client) logger.debug( 'Detected Elasticsearch version ' '{0}'.format(".".join(map(str,version_number))) ) if version_number >= settings.version_max() \ or version_number < settings.version_min(): raise CuratorException( 'Elasticsearch version {0} incompatible ' 'with this version of Curator ' '({0})'.format(".".join(map(str,version_number)), __version__) ) def check_master(client, master_only=False): """ Check if connected client is the elected master node of the cluster. If not, cleanly exit with a log message. :arg client: An :class:`elasticsearch.Elasticsearch` client object :rtype: None """ if master_only and not is_master_node(client): logger.info( 'Master-only flag detected. ' 'Connected to non-master node. Aborting.' ) sys.exit(0) def get_client(**kwargs): """ NOTE: AWS IAM parameters `aws_key`, `aws_secret_key`, and `aws_region` are provided for future compatibility, should AWS ES support the ``/_cluster/state/metadata`` endpoint. So long as this endpoint does not function in AWS ES, the client will not be able to use :class:`curator.indexlist.IndexList`, which is the backbone of Curator 4 Return an :class:`elasticsearch.Elasticsearch` client object using the provided parameters. Any of the keyword arguments the :class:`elasticsearch.Elasticsearch` client object can receive are valid, such as: :arg hosts: A list of one or more Elasticsearch client hostnames or IP addresses to connect to. Can send a single host. :type hosts: list :arg port: The Elasticsearch client port to connect to. :type port: int :arg url_prefix: `Optional` url prefix, if needed to reach the Elasticsearch API (i.e., it's not at the root level) :type url_prefix: str :arg use_ssl: Whether to connect to the client via SSL/TLS :type use_ssl: bool :arg certificate: Path to SSL/TLS certificate :arg client_cert: Path to SSL/TLS client certificate (public key) :arg client_key: Path to SSL/TLS private key :arg aws_key: AWS IAM Access Key (Only used if the :mod:`requests-aws4auth` python module is installed) :arg aws_secret_key: AWS IAM Secret Access Key (Only used if the :mod:`requests-aws4auth` python module is installed) :arg aws_region: AWS Region (Only used if the :mod:`requests-aws4auth` python module is installed) :arg ssl_no_validate: If `True`, do not validate the certificate chain. This is an insecure option and you will see warnings in the log output. :type ssl_no_validate: bool :arg http_auth: Authentication credentials in `user:pass` format. :type http_auth: str :arg timeout: Number of seconds before the client will timeout. :type timeout: int :arg master_only: If `True`, the client will `only` connect if the endpoint is the elected master node of the cluster. **This option does not work if `hosts` has more than one value.** It will raise an Exception in that case. :type master_only: bool :rtype: :class:`elasticsearch.Elasticsearch` """ if 'url_prefix' in kwargs: if ( type(kwargs['url_prefix']) == type(None) or kwargs['url_prefix'] == "None" ): kwargs['url_prefix'] = '' if 'host' in kwargs and 'hosts' in kwargs: raise ConfigurationError( 'Both "host" and "hosts" are defined. Pick only one.') elif 'host' in kwargs and not 'hosts' in kwargs: kwargs['hosts'] = kwargs['host'] del kwargs['host'] kwargs['hosts'] = '127.0.0.1' if not 'hosts' in kwargs else kwargs['hosts'] kwargs['master_only'] = False if not 'master_only' in kwargs \ else kwargs['master_only'] kwargs['use_ssl'] = False if not 'use_ssl' in kwargs else kwargs['use_ssl'] kwargs['ssl_no_validate'] = False if not 'ssl_no_validate' in kwargs \ else kwargs['ssl_no_validate'] kwargs['certificate'] = False if not 'certificate' in kwargs \ else kwargs['certificate'] kwargs['client_cert'] = False if not 'client_cert' in kwargs \ else kwargs['client_cert'] kwargs['client_key'] = False if not 'client_key' in kwargs \ else kwargs['client_key'] kwargs['hosts'] = ensure_list(kwargs['hosts']) logger.debug("kwargs = {0}".format(kwargs)) master_only = kwargs.pop('master_only') if kwargs['use_ssl']: if kwargs['ssl_no_validate']: kwargs['verify_certs'] = False # Not needed, but explicitly defined else: logger.debug('Attempting to verify SSL certificate.') # If user provides a certificate: if kwargs['certificate']: kwargs['verify_certs'] = True kwargs['ca_certs'] = kwargs['certificate'] else: # Try to use bundled certifi certificates if getattr(sys, 'frozen', False): # The application is frozen (compiled) datadir = os.path.dirname(sys.executable) kwargs['verify_certs'] = True kwargs['ca_certs'] = os.path.join(datadir, 'cacert.pem') else: # Use certifi certificates via certifi.where(): import certifi kwargs['verify_certs'] = True kwargs['ca_certs'] = certifi.where() try: from requests_aws4auth import AWS4Auth kwargs['aws_key'] = False if not 'aws_key' in kwargs \ else kwargs['aws_key'] kwargs['aws_secret_key'] = False if not 'aws_secret_key' in kwargs \ else kwargs['aws_secret_key'] kwargs['aws_region'] = False if not 'aws_region' in kwargs \ else kwargs['aws_region'] if kwargs['aws_key'] or kwargs['aws_secret_key'] or kwargs['aws_region']: if not kwargs['aws_key'] and kwargs['aws_secret_key'] \ and kwargs['aws_region']: raise MissingArgument( 'Missing one or more of "aws_key", "aws_secret_key", ' 'or "aws_region".' ) # Override these kwargs kwargs['use_ssl'] = True kwargs['verify_certs'] = True kwargs['connection_class'] = elasticsearch.RequestsHttpConnection kwargs['http_auth'] = ( AWS4Auth( kwargs['aws_key'], kwargs['aws_secret_key'], kwargs['aws_region'], 'es') ) else: logger.debug('"requests_aws4auth" module present, but not used.') except ImportError: logger.debug('Not using "requests_aws4auth" python module to connect.') if master_only: if len(kwargs['hosts']) > 1: raise ConfigurationError( '"master_only" cannot be True if more than one host is ' 'specified. Hosts = {0}'.format(kwargs['hosts']) ) try: client = elasticsearch.Elasticsearch(**kwargs) # Verify the version is acceptable. check_version(client) # Verify "master_only" status, if applicable check_master(client, master_only=master_only) return client except Exception as e: raise elasticsearch.ElasticsearchException( 'Unable to create client connection to Elasticsearch. ' 'Error: {0}'.format(e) ) def override_timeout(timeout, action): """ Override the default timeout for `forcemerge`, `snapshot`, and `sync_flush` operations if the default value of ``30`` is provided. :arg timeout: Number of seconds before the client will timeout. :arg action: The `action` to be performed. """ retval = timeout if action in ['forcemerge', 'snapshot', 'sync_flush']: # Check for default timeout of 30s if timeout == 30: if action in ['forcemerge', 'snapshot']: retval = 21600 elif action == 'sync_flush': retval = 180 logger.debug( 'Overriding default connection timeout for {0} action. ' 'New timeout: {1}'.format(action.upper(),timeout) ) return retval def show_dry_run(ilo, action, **kwargs): """ Log dry run output with the action which would have been executed. :arg ilo: A :class:`curator.indexlist.IndexList` :arg action: The `action` to be performed. :arg kwargs: Any other args to show in the log output """ logger.info('DRY-RUN MODE. No changes will be made.') logger.info( '(CLOSED) indices may be shown that may not be acted on by ' 'action "{0}".'.format(action) ) indices = sorted(ilo.indices) for idx in indices: index_closed = ilo.index_info[idx]['state'] == 'close' logger.info( 'DRY-RUN: {0}: {1}{2} with arguments: {3}'.format( action, idx, ' (CLOSED)' if index_closed else '', kwargs ) ) ### SNAPSHOT STUFF ### def get_repository(client, repository=''): """ Return configuration information for the indicated repository. :arg client: An :class:`elasticsearch.Elasticsearch` client object :arg repository: The Elasticsearch snapshot repository to use :rtype: dict """ try: return client.snapshot.get_repository(repository=repository) except (elasticsearch.TransportError, elasticsearch.NotFoundError): logger.error("Repository {0} not found.".format(repository)) return False def get_snapshot(client, repository=None, snapshot=''): """ Return information about a snapshot (or a comma-separated list of snapshots) If no snapshot specified, it will return all snapshots. If none exist, an empty dictionary will be returned. :arg client: An :class:`elasticsearch.Elasticsearch` client object :arg repository: The Elasticsearch snapshot repository to use :arg snapshot: The snapshot name, or a comma-separated list of snapshots :rtype: dict """ if not repository: raise MissingArgument('No value for "repository" provided') snapname = '_all' if snapshot == '' else snapshot try: return client.snapshot.get(repository=repository, snapshot=snapshot) except (elasticsearch.TransportError, elasticsearch.NotFoundError) as e: raise FailedExecution( 'Unable to get information about snapshot {0} from repository: ' '{1}. Error: {2}'.format(snapname, repository, e) ) def get_snapshot_data(client, repository=None): """ Get ``_all`` snapshots from repository and return a list. :arg client: An :class:`elasticsearch.Elasticsearch` client object :arg repository: The Elasticsearch snapshot repository to use :rtype: list """ if not repository: raise MissingArgument('No value for "repository" provided') try: return client.snapshot.get( repository=repository, snapshot="_all")['snapshots'] except (elasticsearch.TransportError, elasticsearch.NotFoundError) as e: raise FailedExecution( 'Unable to get snapshot information from repository: {0}. ' 'Error: {1}'.format(repository, e) ) def snapshot_in_progress(client, repository=None, snapshot=None): """ Determine whether the provided snapshot in `repository` is ``IN_PROGRESS``. If no value is provided for `snapshot`, then check all of them. Return `snapshot` if it is found to be in progress, or `False` :arg client: An :class:`elasticsearch.Elasticsearch` client object :arg repository: The Elasticsearch snapshot repository to use :arg snapshot: The snapshot name """ allsnaps = get_snapshot_data(client, repository=repository) inprogress = ( [snap['snapshot'] for snap in allsnaps if 'state' in snap.keys() \ and snap['state'] == 'IN_PROGRESS'] ) if snapshot: return snapshot if snapshot in inprogress else False else: if len(inprogress) == 0: return False elif len(inprogress) == 1: return inprogress[0] else: # This should not be possible raise CuratorException( 'More than 1 snapshot in progress: {0}'.format(inprogress) ) def safe_to_snap(client, repository=None, retry_interval=120, retry_count=3): """ Ensure there are no snapshots in progress. Pause and retry accordingly :arg client: An :class:`elasticsearch.Elasticsearch` client object :arg repository: The Elasticsearch snapshot repository to use :arg retry_interval: Number of seconds to delay betwen retries. Default: 120 (seconds) :arg retry_count: Number of attempts to make. Default: 3 :rtype: bool """ if not repository: raise MissingArgument('No value for "repository" provided') for count in range(1, retry_count+1): in_progress = snapshot_in_progress( client, repository=repository ) if in_progress: logger.info( 'Snapshot already in progress: {0}'.format(in_progress)) logger.info( 'Pausing {0} seconds before retrying...'.format( retry_interval) ) time.sleep(retry_interval) logger.info('Retry {0} of {1}'.format(count, retry_count)) else: return True return False def create_snapshot_body(indices, ignore_unavailable=False, include_global_state=True, partial=False): """ Create the request body for creating a snapshot from the provided arguments. :arg indices: A single index, or list of indices to snapshot. :arg ignore_unavailable: Ignore unavailable shards/indices. (default: `False`) :type ignore_unavailable: bool :arg include_global_state: Store cluster global state with snapshot. (default: `True`) :type include_global_state: bool :arg partial: Do not fail if primary shard is unavailable. (default: `False`) :type partial: bool :rtype: dict """ if not indices: logger.error('No indices provided.') return False body = { "ignore_unavailable": ignore_unavailable, "include_global_state": include_global_state, "partial": partial, } if indices == '_all': body["indices"] = indices else: body["indices"] = to_csv(indices) return body def create_repo_body(repo_type=None, compress=True, chunk_size=None, max_restore_bytes_per_sec=None, max_snapshot_bytes_per_sec=None, location=None, bucket=None, region=None, base_path=None, access_key=None, secret_key=None, **kwargs): """ Build the 'body' portion for use in creating a repository. :arg repo_type: The type of repository (presently only `fs` and `s3`) :arg compress: Turn on compression of the snapshot files. Compression is applied only to metadata files (index mapping and settings). Data files are not compressed. (Default: `True`) :arg chunk_size: The chunk size can be specified in bytes or by using size value notation, i.e. 1g, 10m, 5k. Defaults to `null` (unlimited chunk size). :arg max_restore_bytes_per_sec: Throttles per node restore rate. Defaults to ``20mb`` per second. :arg max_snapshot_bytes_per_sec: Throttles per node snapshot rate. Defaults to ``20mb`` per second. :arg location: Location of the snapshots. Required. :arg bucket: `S3 only.` The name of the bucket to be used for snapshots. Required. :arg region: `S3 only.` The region where bucket is located. Defaults to `US Standard` :arg base_path: `S3 only.` Specifies the path within bucket to repository data. Defaults to value of ``repositories.s3.base_path`` or to root directory if not set. :arg access_key: `S3 only.` The access key to use for authentication. Defaults to value of ``cloud.aws.access_key``. :arg secret_key: `S3 only.` The secret key to use for authentication. Defaults to value of ``cloud.aws.secret_key``. :returns: A dictionary suitable for creating a repository from the provided arguments. :rtype: dict """ # This shouldn't happen, but just in case... if not repo_type: raise MissingArgument('Missing required parameter --repo_type') argdict = locals() body = {} body['type'] = argdict['repo_type'] body['settings'] = {} settingz = [] # Differentiate from module settings maybes = [ 'compress', 'chunk_size', 'max_restore_bytes_per_sec', 'max_snapshot_bytes_per_sec' ] s3 = ['bucket', 'region', 'base_path', 'access_key', 'secret_key'] settingz += [i for i in maybes if argdict[i]] # Type 'fs' if argdict['repo_type'] == 'fs': settingz.append('location') # Type 's3' if argdict['repo_type'] == 's3': settingz += [i for i in s3 if argdict[i]] for k in settingz: body['settings'][k] = argdict[k] return body def create_repository(client, **kwargs): """ Create repository with repository and body settings :arg client: An :class:`elasticsearch.Elasticsearch` client object :arg repo_type: The type of repository (presently only `fs` and `s3`) :arg compress: Turn on compression of the snapshot files. Compression is applied only to metadata files (index mapping and settings). Data files are not compressed. (Default: `True`) :arg chunk_size: The chunk size can be specified in bytes or by using size value notation, i.e. 1g, 10m, 5k. Defaults to `null` (unlimited chunk size). :arg max_restore_bytes_per_sec: Throttles per node restore rate. Defaults to ``20mb`` per second. :arg max_snapshot_bytes_per_sec: Throttles per node snapshot rate. Defaults to ``20mb`` per second. :arg location: Location of the snapshots. Required. :arg bucket: `S3 only.` The name of the bucket to be used for snapshots. Required. :arg region: `S3 only.` The region where bucket is located. Defaults to `US Standard` :arg base_path: `S3 only.` Specifies the path within bucket to repository data. Defaults to value of ``repositories.s3.base_path`` or to root directory if not set. :arg access_key: `S3 only.` The access key to use for authentication. Defaults to value of ``cloud.aws.access_key``. :arg secret_key: `S3 only.` The secret key to use for authentication. Defaults to value of ``cloud.aws.secret_key``. :returns: A boolean value indicating success or failure. :rtype: bool """ if not 'repository' in kwargs: raise MissingArgument('Missing required parameter --repository') else: repository = kwargs['repository'] try: body = create_repo_body(**kwargs) logger.debug( 'Checking if repository {0} already exists...'.format(repository) ) result = get_repository(client, repository=repository) logger.debug("Result = {0}".format(result)) if not result: logger.debug( 'Repository {0} not in Elasticsearch. Continuing...'.format( repository ) ) client.snapshot.create_repository(repository=repository, body=body) elif result is not None and repository not in result: logger.debug( 'Repository {0} not in Elasticsearch. Continuing...'.format( repository ) ) client.snapshot.create_repository(repository=repository, body=body) else: raise FailedExecution( 'Unable to create repository {0}. A repository with that name ' 'already exists.'.format(repository) ) except elasticsearch.TransportError as e: raise FailedExecution( """ Unable to create repository {0}. Response Code: {1}. Error: {2}. Check curator and elasticsearch logs for more information. """.format( repository, e.status_code, e.error ) ) logger.debug("Repository {0} creation initiated...".format(repository)) return True def repository_exists(client, repository=None): """ Verify the existence of a repository :arg client: An :class:`elasticsearch.Elasticsearch` client object :arg repository: The Elasticsearch snapshot repository to use :rtype: bool """ if not repository: raise MissingArgument('No value for "repository" provided') test_result = get_repository(client, repository) if repository in test_result: logger.debug("Repository {0} exists.".format(repository)) return True else: logger.debug("Repository {0} not found...".format(repository)) return False def test_repo_fs(client, repository=None): """ Test whether all nodes have write access to the repository :arg client: An :class:`elasticsearch.Elasticsearch` client object :arg repository: The Elasticsearch snapshot repository to use """ try: nodes = client.snapshot.verify_repository( repository=repository)['nodes'] logger.debug('All nodes can write to the repository') logger.debug( 'Nodes with verified repository access: {0}'.format(nodes)) except Exception as e: try: if e.status_code == 404: msg = ( '--- Repository "{0}" not found. Error: ' '{1}, {2}'.format(repository, e.status_code, e.error) ) else: msg = ( '--- Got a {0} response from Elasticsearch. ' 'Error message: {1}'.format(e.status_code, e.error) ) except AttributeError: msg = ('--- Error message: {0}'.format(e)) raise ActionError( 'Failed to verify all nodes have repository access: ' '{0}'.format(msg) ) def snapshot_running(client): """ Return `True` if a snapshot is in progress, and `False` if not :arg client: An :class:`elasticsearch.Elasticsearch` client object :rtype: bool """ try: status = client.snapshot.status()['snapshots'] except Exception as e: report_failure(e) # We will only accept a positively identified False. Anything else is # suspect. return False if status == [] else True def parse_date_pattern(name): """ Scan and parse `name` for :py:func:`time.strftime` strings, replacing them with the associated value when found, but otherwise returning lowercase values, as uppercase snapshot names are not allowed. The :py:func:`time.strftime` identifiers that Curator currently recognizes as acceptable include: * ``Y``: A 4 digit year * ``y``: A 2 digit year * ``m``: The 2 digit month * ``W``: The 2 digit week of the year * ``d``: The 2 digit day of the month * ``H``: The 2 digit hour of the day, in 24 hour notation * ``M``: The 2 digit minute of the hour * ``S``: The 2 digit number of second of the minute * ``j``: The 3 digit day of the year :arg name: A name, which can contain :py:func:`time.strftime` strings """ prev = ''; curr = ''; rendered = '' for s in range(0, len(name)): curr = name[s] if curr == '%': pass elif curr in settings.date_regex() and prev == '%': rendered += str(datetime.utcnow().strftime('%{0}'.format(curr))) else: rendered += curr logger.debug('Partially rendered name: {0}'.format(rendered)) prev = curr logger.debug('Fully rendered name: {0}'.format(rendered)) return rendered def prune_nones(mydict): """ Remove keys from `mydict` whose values are `None` :arg mydict: The dictionary to act on :rtype: dict """ # Test for `None` instead of existence or zero values will be caught return dict([(k,v) for k, v in mydict.items() if v != None and v != 'None']) def validate_filters(action, filters): """ Validate that the filters are appropriate for the action type, e.g. no index filters applied to a snapshot list. :arg action: An action name :arg filters: A list of filters to test. """ # Define which set of filtertypes to use for testing if action in settings.snapshot_actions(): filtertypes = settings.snapshot_filtertypes() else: filtertypes = settings.index_filtertypes() for f in filters: if f['filtertype'] not in filtertypes: raise ConfigurationError( '"{0}" filtertype is not compatible with action "{1}"'.format( f['filtertype'], action ) ) # If we get to this point, we're still valid. Return the original list return filters def validate_actions(data): """ Validate an Action configuration dictionary, as imported from actions.yml, for example. The method returns a validated and sanitized configuration dictionary. :arg data: The configuration dictionary :rtype: dict """ # data is the ENTIRE schema... clean_config = { } # Let's break it down into smaller chunks... # First, let's make sure it has "actions" as a key, with a subdictionary root = SchemaCheck(data, actions.root(), 'Actions File', 'root').result() # We've passed the first step. Now let's iterate over the actions... for action_id in root['actions']: # Now, let's ensure that the basic action structure is correct, with # the proper possibilities for 'action' action_dict = root['actions'][action_id] loc = 'Action ID "{0}"'.format(action_id) valid_structure = SchemaCheck( action_dict, actions.structure(action_dict, loc), 'structure', loc ).result() # With the basic structure validated, now we extract the action name current_action = valid_structure['action'] # And let's update the location with the action. loc = 'Action ID "{0}", action "{1}"'.format( action_id, current_action) clean_options = SchemaCheck( prune_nones(valid_structure['options']), options.get_schema(current_action), 'options', loc ).result() clean_config[action_id] = { 'action' : current_action, 'description' : valid_structure['description'], 'options' : clean_options, } if current_action == 'alias': add_remove = {} for k in ['add', 'remove']: if k in valid_structure: current_filters = SchemaCheck( valid_structure[k]['filters'], Schema(filters.Filters(current_action, location=loc)), '"{0}" filters', '{1}, "filters"'.format(k, loc) ).result() add_remove.update( { k: { 'filters' : SchemaCheck( current_filters, Schema( filters.Filters( current_action, location=loc ) ), 'filters', '{0}, "{1}", "filters"'.format(loc, k) ).result() } } ) # Add/Remove here clean_config[action_id].update(add_remove) elif current_action in [ 'cluster_routing', 'create_index' ]: # neither cluster_routing nor create_index should have filters pass else: # Filters key only appears in non-alias actions valid_filters = SchemaCheck( valid_structure['filters'], Schema(filters.Filters(current_action, location=loc)), 'filters', '{0}, "filters"'.format(loc) ).result() clean_filters = validate_filters(current_action, valid_filters) clean_config[action_id].update({'filters' : clean_filters}) # if we've gotten this far without any Exceptions raised, it's valid! return { 'actions' : clean_config } curator-4.2.5/curator/validators/000077500000000000000000000000001302700154000170205ustar00rootroot00000000000000curator-4.2.5/curator/validators/__init__.py000066400000000000000000000000451302700154000211300ustar00rootroot00000000000000from .schemacheck import SchemaCheck curator-4.2.5/curator/validators/actions.py000066400000000000000000000032471302700154000210400ustar00rootroot00000000000000from voluptuous import * from ..defaults import settings from . import SchemaCheck ### Schema information ### # Actions: root level def root(): return Schema({ Required('actions'): dict }) def valid_action(): return { Required('action'): Any( In(settings.all_actions()), msg='action must be one of {0}'.format( settings.all_actions() ) ) } # Basic action structure def structure(data, location): # Validate the action type first, so we can use it for other tests valid_action_type = SchemaCheck( data, Schema(valid_action(), extra=True), 'action type', location, ).result() # Build a valid schema knowing that the action has already been validated retval = valid_action() retval.update( { Optional('description', default='No description given'): Any( str, unicode ) } ) retval.update( { Optional('options', default=settings.default_options()): dict } ) action = data['action'] if action in [ 'cluster_routing', 'create_index' ]: # Neither the cluster_routing nor create_index actions should have a # 'filters' block pass elif action == 'alias': # The alias action should not have a filters block, but should have # an add and/or remove block. retval.update( { Optional('add'): dict, Optional('remove'): dict, } ) else: retval.update( { Optional('filters', default=settings.default_filters()): list } ) return Schema(retval) curator-4.2.5/curator/validators/config_file.py000066400000000000000000000036101302700154000216360ustar00rootroot00000000000000from voluptuous import * # Configuration file: client def config_client(): return { Optional('hosts', default='127.0.0.1'): Any(None, str, unicode, list), Optional('port', default=9200): Any( None, All(Coerce(int), Range(min=1, max=65535)) ), Optional('url_prefix', default=''): Any(None, str, unicode), Optional('use_ssl', default=False): Boolean(), Optional('certificate', default=None): Any(None, str, unicode), Optional('client_cert', default=None): Any(None, str, unicode), Optional('client_key', default=None): Any(None, str, unicode), Optional('aws_key', default=None): Any(None, str, unicode), Optional('aws_secret_key', default=None): Any(None, str, unicode), Optional('aws_region', default=None): Any(None, str, unicode), Optional('ssl_no_validate', default=False): Boolean(), Optional('http_auth', default=None): Any(None, str, unicode), Optional('timeout', default=30): All( Coerce(int), Range(min=1, max=86400)), Optional('master_only', default=False): Boolean(), } # Configuration file: logging def config_logging(): return { Optional( 'loglevel', default='INFO'): Any(None, 'NOTSET', 'DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL', All(Coerce(int), Any(0, 10, 20, 30, 40, 50)) ), Optional('logfile', default=None): Any(None, str, unicode), Optional( 'logformat', default='default'): Any(None, All( Any(str, unicode), Any('default', 'json', 'logstash') ) ), Optional( 'blacklist', default=['elasticsearch', 'urllib3']): Any(None, list), } def client(): return Schema( { Optional('client'): config_client(), Optional('logging'): config_logging(), } ) curator-4.2.5/curator/validators/filter_elements.py000066400000000000000000000103151302700154000225530ustar00rootroot00000000000000from voluptuous import * from ..defaults import settings ### Schema information ### def aliases(**kwargs): # This setting is used by the alias filtertype and is required return { Required('aliases'): Any(str, [str], unicode, [unicode]) } def allocation_type(**kwargs): return { Optional('allocation_type', default='require'): All( Any(str, unicode), Any('require', 'include', 'exclude')) } def count(**kwargs): # This setting is only used with the count filtertype and is required return { Required('count'): All(Coerce(int), Range(min=1)) } def direction(**kwargs): # This setting is only used with the age filtertype. return { Required('direction'): Any('older', 'younger') } def disk_space(**kwargs): # This setting is only used with the space filtertype and is required return { Required('disk_space'): Any(Coerce(float)) } def epoch(**kwargs): # This setting is only used with the age filtertype. return { Optional('epoch', default=None): Any(Coerce(int), None) } def exclude(**kwargs): # This setting is available in all filter types. if 'exclude' in kwargs and kwargs['exclude']: val = True else: # False by default val = False return { Optional('exclude', default=val): Boolean() } def field(**kwargs): # This setting is only used with the age filtertype. if 'required' in kwargs and kwargs['required']: return { Required('field'): Any(str, unicode) } else: return { Optional('field'): Any(str, unicode) } def key(**kwargs): # This setting is only used with the allocated filtertype. return { Required('key'): Any(str, unicode) } def kind(**kwargs): # This setting is only used with the pattern filtertype and is required return { Required('kind'): Any('prefix', 'suffix', 'timestring', 'regex') } def max_num_segments(**kwargs): return { Required('max_num_segments'): All(Coerce(int), Range(min=1)) } def reverse(**kwargs): # Only used with space filtertype # Should be ignored if `use_age` is True return { Optional('reverse', default=True): Boolean() } def source(**kwargs): # This setting is only used with the age filtertype, or with the space # filtertype when use_age is set to True. if 'action' in kwargs and kwargs['action'] in settings.snapshot_actions(): valuelist = Any('name', 'creation_date') else: valuelist = Any('name', 'creation_date', 'field_stats') if 'required' in kwargs and kwargs['required']: return { Required('source'): valuelist } else: return { Optional('source'): valuelist } def state(**kwargs): # This setting is only used with the state filtertype. return { Optional('state', default='SUCCESS'): Any( 'SUCCESS', 'PARTIAL', 'FAILED', 'IN_PROGRESS') } def stats_result(**kwargs): # This setting is only used with the age filtertype. return { Optional('stats_result', default='min_value'): Any( 'min_value', 'max_value') } def timestring(**kwargs): # This setting is only used with the age filtertype, or with the space # filtertype if use_age is set to True. if 'required' in kwargs and kwargs['required']: return { Required('timestring'): Any(str, unicode) } else: return { Optional('timestring', default=None): Any(str, unicode, None) } def unit(**kwargs): # This setting is only used with the age filtertype, or with the space # filtertype if use_age is set to True. return { Required('unit'): Any( 'seconds', 'minutes', 'hours', 'days', 'weeks', 'months', 'years' ) } def unit_count(**kwargs): # This setting is only used with the age filtertype, or with the space # filtertype if use_age is set to True. return { Required('unit_count'): Coerce(int) } def use_age(**kwargs): # Use of this setting requires the additional setting, source. return { Optional('use_age', default=False): Boolean() } def value(**kwargs): # This setting is only used with the pattern filtertype and is a required # setting. There is a separate value option associated with the allocation # action, and the allocated filtertype. return { Required('value'): Any(str, unicode) } curator-4.2.5/curator/validators/filters.py000066400000000000000000000052311302700154000210430ustar00rootroot00000000000000from voluptuous import * from ..defaults import settings from ..exceptions import ConfigurationError from . import SchemaCheck, filtertypes import logging logger = logging.getLogger(__name__) def filtertype(): return { Required('filtertype'): Any( In(settings.all_filtertypes()), msg='filtertype must be one of {0}'.format( settings.all_filtertypes() ) ) } def structure(): # This is to first ensure that only the possible keys/filter elements are # there, and get a dictionary back to work with. retval = { Optional('aliases'): Any(str, [str], unicode, [unicode]), Optional('allocation_type'): Any(str, unicode), Optional('count'): Coerce(int), Optional('direction'): Any(str, unicode), Optional('disk_space'): float, Optional('epoch'): Any(Coerce(int), None), Optional('exclude'): Any(int, str, unicode, bool, None), Optional('field'): Any(str, unicode, None), Optional('key'): Any(str, unicode), Optional('kind'): Any(str, unicode), Optional('max_num_segments'): Coerce(int), Optional('reverse'): Any(int, str, unicode, bool, None), Optional('source'): Any(str, unicode), Optional('state'): Any(str, unicode), Optional('stats_result'): Any(str, unicode, None), Optional('timestring'): Any(str, unicode, None), Optional('unit'): Any(str, unicode), Optional('unit_count'): Coerce(int), Optional('use_age'): Boolean(), Optional('value'): Any(int, float, str, unicode, bool), } retval.update(filtertype()) return Schema(retval) def single(action, data): try: ft = data['filtertype'] except KeyError: raise ConfigurationError('Missing key "filtertype"') f = filtertype() for each in getattr(filtertypes, ft)(action, data): f.update(each) return Schema(f) def Filters(action, location=None): def f(v): def prune_nones(mydict): return dict([(k,v) for k, v in mydict.items() if v != None and v != 'None']) # This validator method simply validates all filters in the list. for idx in range(0, len(v)): pruned = prune_nones(v[idx]) filter_dict = SchemaCheck( pruned, single(action, pruned), 'filter', '{0}, filter #{1}: {2}'.format(location, idx, pruned) ).result() logger.debug('Filter #{0}: {1}'.format(idx, filter_dict)) v[idx] = filter_dict # If we've made it here without raising an Exception, it's valid return v return f curator-4.2.5/curator/validators/filtertypes.py000066400000000000000000000064311302700154000217500ustar00rootroot00000000000000from voluptuous import * from ..defaults import settings from . import filter_elements import logging logger = logging.getLogger(__name__) ## Helpers ## def _age_elements(action, config): retval = [] is_req = True if config['filtertype'] in ['count', 'space']: is_req = True if 'use_age' in config and config['use_age'] else False retval.append(filter_elements.source(action=action, required=is_req)) if action in settings.index_actions(): retval.append(filter_elements.stats_result()) # This is a silly thing here, because the absence of 'source' will # show up in the actual schema check, but it keeps code from breaking here ts_req = False if 'source' in config: if config['source'] == 'name': ts_req = True elif action in settings.index_actions(): # field_stats must _only_ exist for Index actions (not Snapshot) if config['source'] == 'field_stats': retval.append(filter_elements.field(required=True)) else: retval.append(filter_elements.field(required=False)) retval.append(filter_elements.timestring(required=ts_req)) else: # If source isn't in the config, then the other elements are not # required, but should be Optional to prevent false positives retval.append(filter_elements.field(required=False)) retval.append(filter_elements.timestring(required=ts_req)) return retval ### Schema information ### def alias(action, config): return [ filter_elements.aliases(), filter_elements.exclude(), ] def age(action, config): # Required & Optional retval = [ filter_elements.direction(), filter_elements.unit(), filter_elements.unit_count(), filter_elements.epoch(), filter_elements.exclude(), ] retval += _age_elements(action, config) return retval def allocated(action, config): return [ filter_elements.key(), filter_elements.value(), filter_elements.allocation_type(), filter_elements.exclude(exclude=True), ] def closed(action, config): return [ filter_elements.exclude(exclude=True) ] def count(action, config): retval = [ filter_elements.count(), filter_elements.use_age(), filter_elements.reverse(), filter_elements.exclude(exclude=True), ] retval += _age_elements(action, config) return retval def forcemerged(action, config): return [ filter_elements.max_num_segments(), filter_elements.exclude(exclude=True), ] def kibana(action, config): return [ filter_elements.exclude(exclude=True) ] def none(action, config): return [ ] def opened(action, config): return [ filter_elements.exclude(exclude=True) ] def pattern(action, config): return [ filter_elements.kind(), filter_elements.value(), filter_elements.exclude(), ] def space(action, config): retval = [ filter_elements.disk_space(), filter_elements.reverse(), filter_elements.use_age(), filter_elements.exclude(), ] retval += _age_elements(action, config) return retval def state(action, config): return [ filter_elements.state(), filter_elements.exclude(), ] curator-4.2.5/curator/validators/options.py000066400000000000000000000135211302700154000210670ustar00rootroot00000000000000from voluptuous import * from ..defaults import settings ### Schema information ### def allocation_type(): return { Optional('allocation_type', default='require'): All( Any(str, unicode), Any('require', 'include', 'exclude')) } def continue_if_exception(): return { Optional('continue_if_exception', default=False): Boolean() } def count(): return { Required('count'): All(Coerce(int), Range(min=0, max=10)) } def delay(): return { Optional('delay', default=0): All( Coerce(float), Range(min=0.0, max=3600.0) ) } def delete_aliases(): return { Optional('delete_aliases', default=False): Boolean() } def disable_action(): return { Optional('disable_action', default=False): Boolean() } def extra_settings(): return { Optional('extra_settings', default={}): dict } def ignore_empty_list(): return { Optional('ignore_empty_list', default=False): Boolean() } def ignore_unavailable(): return { Optional('ignore_unavailable', default=False): Boolean() } def include_aliases(): return { Optional('include_aliases', default=False): Boolean() } def include_global_state(): return { Optional('include_global_state', default=True): Boolean() } def indices(): return { Optional('indices', default=None): Any(None, list) } def key(): return { Required('key'): Any(str, unicode) } def max_num_segments(): return { Required('max_num_segments'): All(Coerce(int), Range(min=1, max=32768)) } def name(action): if action in ['alias', 'create_index']: return { Required('name'): Any(str, unicode) } elif action == 'snapshot': return { Optional('name', default='curator-%Y%m%d%H%M%S'): Any(str, unicode) } elif action == 'restore': return { Optional('name'): Any(str, unicode) } def partial(): return { Optional('partial', default=False): Boolean() } def rename_pattern(): return { Optional('rename_pattern'): Any(str, unicode) } def rename_replacement(): return { Optional('rename_replacement'): Any(str, unicode) } def repository(): return { Required('repository'): Any(str, unicode) } def retry_count(): return { Optional('retry_count', default=3): All( Coerce(int), Range(min=0, max=100) ) } def retry_interval(): return { Optional('retry_interval', default=120): All( Coerce(int), Range(min=1, max=600) ) } def routing_type(): return { Required('routing_type'): Any('allocation', 'rebalance') } def cluster_routing_setting(): return { Required('setting'): Any('enable') } def cluster_routing_value(): return { Required('value'): Any( 'all', 'primaries', 'none', 'new_primaries', 'replicas' ) } def skip_repo_fs_check(): return { Optional('skip_repo_fs_check', default=False): Boolean() } def timeout_override(action): if action in ['forcemerge', 'restore', 'snapshot']: return { Optional('timeout_override', default=21600): Any(Coerce(int), None) } if action in ['close']: # This is due to the synced flush operation before closing. return { Optional('timeout_override', default=180): Any(Coerce(int), None) } else: return { Optional('timeout_override', default=None): Any(Coerce(int), None) } def value(): return { Required('value'): Any(str, unicode) } def wait_for_completion(action): if action in ['allocation', 'cluster_routing', 'replicas']: return { Optional('wait_for_completion', default=False): Boolean() } elif action in ['restore', 'snapshot']: return { Optional('wait_for_completion', default=True): Boolean() } ## Methods for building the schema def action_specific(action): options = { 'alias' : [ name(action), extra_settings(), ], 'allocation' : [ key(), value(), allocation_type(), wait_for_completion(action), ], 'close' : [ delete_aliases() ], 'cluster_routing' : [ routing_type(), cluster_routing_setting(), cluster_routing_value(), wait_for_completion(action), ], 'create_index' : [ name(action), extra_settings(), ], 'delete_indices' : [], 'delete_snapshots' : [ repository(), retry_interval(), retry_count(), ], 'forcemerge' : [ delay(), max_num_segments(), ], 'open' : [], 'replicas' : [ count(), wait_for_completion(action), ], 'restore' : [ repository(), name(action), indices(), ignore_unavailable(), include_aliases(), include_global_state(), partial(), rename_pattern(), rename_replacement(), extra_settings(), wait_for_completion(action), skip_repo_fs_check(), ], 'snapshot' : [ repository(), name(action), ignore_unavailable(), include_global_state(), partial(), wait_for_completion(action), skip_repo_fs_check(), ], } return options[action] def get_schema(action): # Appending the options dictionary seems to be the best way, since the # "Required" and "Optional" elements are hashes themselves. options = {} defaults = [ continue_if_exception(), disable_action(), ignore_empty_list(), timeout_override(action), ] for each in defaults: options.update(each) for each in action_specific(action): options.update(each) return Schema(options) curator-4.2.5/curator/validators/schemacheck.py000066400000000000000000000050771302700154000216410ustar00rootroot00000000000000from voluptuous import * from ..exceptions import ConfigurationError import re import logging class SchemaCheck(object): def __init__(self, config, schema, test_what, location): """ Validate ``config`` with the provided voluptuous ``schema``. ``test_what`` and ``location`` are for reporting the results, in case of failure. If validation is successful, the method returns ``config`` as valid. :arg config: A configuration dictionary. :type config: dict :arg schema: A voluptuous schema definition :type schema: :class:`voluptuous.Schema` :arg test_what: which configuration block is being validated :type test_what: str :arg location: An string to report which configuration sub-block is being tested. :type location: str """ self.loggit = logging.getLogger('curator.validators.SchemaCheck') # Set the Schema for validation... self.loggit.debug('Schema: {0}'.format(schema)) self.loggit.debug('"{0}" config: {1}'.format(test_what, config)) self.config = config self.schema = schema self.test_what = test_what self.location = location def __parse_error(self): """ Report the error, and try to report the bad key or value as well. """ def get_badvalue(data_string, data): elements = re.sub('[\'\]]', '', data_string).split('[') elements.pop(0) # Get rid of data as the first element value = None for k in elements: try: key = int(k) except ValueError: key = k if value == None: value = data[key] # if this fails, it's caught below return value try: self.badvalue = get_badvalue(str(self.error).split()[-1], self.config) except: self.badvalue = '(could not determine)' def result(self): try: return self.schema(self.config) except Exception as e: try: self.error = e.errors[0] except: self.error = '{0}'.format(e) self.__parse_error() self.loggit.error('Schema error: {0}'.format(self.error)) raise ConfigurationError( 'Configuration: {0}: Location: {1}: Bad Value: "{2}", {3}. ' 'Check configuration file.'.format( self.test_what, self.location, self.badvalue, self.error) ) curator-4.2.5/docs/000077500000000000000000000000001302700154000141215ustar00rootroot00000000000000curator-4.2.5/docs/Changelog.rst000066400000000000000000001454231302700154000165530ustar00rootroot00000000000000.. _changelog: Changelog ========= 4.2.5 (22 December 2016) ------------------------ **General** * Add and increment test versions for Travis CI. #839 (untergeek) * Make `filter_list` optional in snapshot, show_snapshot and show_indices singleton actions. #853 (alexef) **Bug Fixes** * Fix cli integration test when different host/port are specified. Reported in #843 (untergeek) * Catch empty list condition during filter iteration in singleton actions. Reported in #848 (untergeek) **Documentation** * Add docs regarding how filters are ANDed together, and how to do an OR with the regex pattern filter type. Requested in #842 (untergeek) * Fix typo in Click version in docs. #850 (breml) * Where applicable, replace `[source,text]` with `[source,yaml]` for better formatting in the resulting docs. 4.2.4 (7 December 2016) ----------------------- **Bug Fixes** * ``--wait_for_completion`` should be `True` by default for Snapshot singleton action. Reported in #829 (untergeek) * Increase `version_max` to 5.1.99. Prematurely reported in #832 (untergeek) * Make the '.security' index visible for snapshots so long as proper credentials are used. Reported in #826 (untergeek) 4.2.3.post1 (22 November 2016) ------------------------------ This fix is `only` going in for ``pip``-based installs. There are no other code changes. **Bug Fixes** * Fixed incorrect assumption of PyPI picking up dependency for certifi. It is still a dependency, but should not affect ``pip`` installs with an error any more. Reported in #821 (untergeek) 4.2.3 (21 November 2016) ------------------------ 4.2.2 was pulled immediately after release after it was discovered that the Windows binary distributions were still not including the certifi-provided certificates. This has now been remedied. **General** * ``certifi`` is now officially a requirement. * ``setup.py`` now forcibly includes the ``certifi`` certificate PEM file in the "frozen" distributions (i.e., the compiled versions). The ``get_client`` method was updated to reflect this and catch it for both the Linux and Windows binary distributions. This should `finally` put to rest #810 4.2.2 (21 November 2016) ------------------------ **Bug Fixes** * The certifi-provided certificates were not propagating to the compiled RPM/DEB packages. This has been corrected. Reported in #810 (untergeek) **General** * Added missing ``--ignore_empty_list`` option to singleton actions. Requested in #812 (untergeek) **Documentation** * Add a FAQ entry regarding the click module's need for Unicode when using Python 3. Kind of a bug fix too, as the entry_points were altered to catch this omission and report a potential solution on the command-line. Reported in #814 (untergeek) * Change the "Command-Line" documentation header to be "Running Curator" 4.2.1 (8 November 2016) ----------------------- **Bug Fixes** * In the course of package release testing, an undesirable scenario was caught where boolean flags default values for ``curator_cli`` were improperly overriding values from a yaml config file. **General** * Adding in direct download URLs for the RPM, DEB, tarball and zip packages. 4.2.0 (4 November 2016) ----------------------- **New Features** * Shard routing allocation enable/disable. This will allow you to disable shard allocation routing before performing one or more actions, and then re-enable after it is complete. Requested in #446 (untergeek) * Curator 3.x-style command-line. This is now ``curator_cli``, to differentiate between the current binary. Not all actions are available, but the most commonly used ones are. With the addition in 4.1.0 of schema and configuration validation, there's even a way to still do filter chaining on the command-line! Requested in #767, and by many other users (untergeek) **General** * Update testing to the most recent versions. * Lock elasticsearch-py module version at >= 2.4.0 and <= 3.0.0. There are API changes in the 5.0 release that cause tests to fail. **Bug Fixes** * Guarantee that binary packages are built from the latest Python + libraries. This ensures that SSL/TLS will work without warning messages about insecure connections, unless they actually are insecure. Reported in #780, though the reported problem isn't what was fixed. The fix is needed based on what was discovered while troubleshooting the problem. (untergeek) 4.1.2 (6 October 2016) ---------------------- This release does not actually add any new code to Curator, but instead improves documentation and includes new linux binary packages. **General** * New Curator binary packages for common Linux systems! These will be found in the same repositories that the python-based packages are in, but have no dependencies. All necessary libraries/modules are bundled with the binary, so everything should work out of the box. This feature doesn't change any other behavior, so it's not a major release. These binaries have been tested in: * CentOS 6 & 7 * Ubuntu 12.04, 14.04, 16.04 * Debian 8 They do not work in Debian 7 (library mismatch). They may work in other systems, but that is untested. The script used is in the unix_packages directory. The Vagrantfiles for the various build systems are in the Vagrant directory. **Bug Fixes** * The only bug that can be called a bug is actually a stray ``.exe`` suffix in the binary package creation section (cx_freeze) of ``setup.py``. The Windows binaries should have ``.exe`` extensions, but not unix variants. * Elasticsearch 5.0.0-beta1 testing revealed that a document ID is required during document creation in tests. This has been fixed, and a redundant bit of code in the forcemerge integration test was removed. **Documentation** * The documentation has been updated and improved. Examples and installation are now top-level events, with the sub-sections each having their own link. They also now show how to install and use the binary packages, and the section on installation from source has been improved. The missing section on installing the voluptuous schema verification module has been written and included. #776 (untergeek) 4.1.1 (27 September 2016) ------------------------- **Bug Fixes** * String-based booleans are now properly coerced. This fixes an issue where `True`/`False` were used in environment variables, but not recognized. #765 (untergeek) * Fix missing `count` method in ``__map_method`` in SnapshotList. Reported in #766 (untergeek) **General** * Update es_repo_mgr to use the same client/logging YAML config file. Requested in #752 (untergeek) **Schema Validation** * Cases where ``source`` was not defined in a filter (but should have been) were informing users that a `timestring` field was there that shouldn't have been. This edge case has been corrected. **Documentation** * Added notifications and FAQ entry to explain that AWS ES is not supported. 4.1.0 (6 September 2016) ------------------------ **New Features** * Configuration and Action file schema validation. Requested in #674 (untergeek) * Alias filtertype! With this filter, you can select indices based on whether they are part of an alias. Merged in #748 (untergeek) * Count filtertype! With this filter, you can now configure Curator to only keep the most recent _n_ indices (or snapshots!). Merged in #749 (untergeek) * Experimental! Use environment variables in your YAML configuration files. This was a popular request, #697. (untergeek) **General** * New requirement! ``voluptuous`` Python schema validation module * Requirement version bump: Now requires ``elasticsearch-py`` 2.4.0 **Bug Fixes** * ``delete_aliases`` option in ``close`` action no longer results in an error if not all selected indices have an alias. Add test to confirm expected behavior. Reported in #736 (untergeek) **Documentation** * Add information to FAQ regarding indices created before Elasticsearch 1.4. Merged in #747 4.0.6 (15 August 2016) ---------------------- **Bug Fixes** * Update old calls used with ES 1.x to reflect changes in 2.x+. This was necessary to work with Elasticsearch 5.0.0-alpha5. Fixed in #728 (untergeek) **Doc Fixes** * Add section detailing that the value of a ``value`` filter element should be encapsulated in single quotes. Reported in #726. (untergeek) 4.0.5 (3 August 2016) --------------------- **Bug Fixes** * Fix incorrect variable name for AWS Region reported in #679 (basex) * Fix ``filter_by_space()`` to not fail when index age metadata is not present. Indices without the appropriate age metadata will instead be excluded, with a debug-level message. Reported in #724 (untergeek) **Doc Fixes** * Fix documentation for the space filter and the source filter element. 4.0.4 (1 August 2016) --------------------- **Bug Fixes** * Fix incorrect variable name in Allocation action. #706 (lukewaite) * Incorrect error message in ``create_snapshot_body`` reported in #711 (untergeek) * Test for empty index list object should happen in action initialization for snapshot action. Discovered in #711. (untergeek) **Doc Fixes** * Add menus to asciidoc chapters #704 (untergeek) * Add pyyaml dependency #710 (dtrv) 4.0.3 (22 July 2016) -------------------- **General** * 4.0.2 didn't work for ``pip`` installs due to an omission in the MANIFEST.in file. This came up during release testing, but before the release was fully published. As the release was never fully published, this should not have actually affected anyone. **Bug Fixes** * These are the same as 4.0.2, but it was never fully released. * All default settings are now values returned from functions instead of constants. This was resulting in settings getting stomped on. New test addresses the original complaint. This removes the need for ``deepcopy``. See issue #687 (untergeek) * Fix ``host`` vs. ``hosts`` issue in ``get_client()`` rather than the non-functional function in ``repomgrcli.py``. * Update versions being tested. * Community contributed doc fixes. * Reduced logging verbosity by making most messages debug level. #684 (untergeek) * Fixed log whitelist behavior (and switched to blacklisting instead). Default behavior will now filter traffic from the ``elasticsearch`` and ``urllib3`` modules. * Fix Travis CI testing to accept some skipped tests, as needed. #695 (untergeek) * Fix missing empty index test in snapshot action. #682 (sherzberg) 4.0.2 (22 July 2016) -------------------- **Bug Fixes** * All default settings are now values returned from functions instead of constants. This was resulting in settings getting stomped on. New test addresses the original complaint. This removes the need for ``deepcopy``. See issue #687 (untergeek) * Fix ``host`` vs. ``hosts`` issue in ``get_client()`` rather than the non-functional function in ``repomgrcli.py``. * Update versions being tested. * Community contributed doc fixes. * Reduced logging verbosity by making most messages debug level. #684 (untergeek) * Fixed log whitelist behavior (and switched to blacklisting instead). Default behavior will now filter traffic from the ``elasticsearch`` and ``urllib3`` modules. * Fix Travis CI testing to accept some skipped tests, as needed. #695 (untergeek) * Fix missing empty index test in snapshot action. #682 (sherzberg) 4.0.1 (1 July 2016) ------------------- **Bug Fixes** * Coerce Logstash/JSON logformat type timestamp value to always use UTC. #661 (untergeek) * Catch and remove indices from the actionable list if they do not have a `creation_date` field in settings. This field was introduced in ES v1.4, so that indicates a rather old index. #663 (untergeek) * Replace missing ``state`` filter for ``snapshotlist``. #665 (untergeek) * Restore ``es_repo_mgr`` as a stopgap until other CLI scripts are added. It will remain undocumented for now, as I am debating whether to make repository creation its own action in the API. #668 (untergeek) * Fix dry run results for snapshot action. #673 (untergeek) 4.0.0 (24 June 2016) -------------------- It's official! Curator 4.0.0 is released! **Breaking Changes** * New and improved API! * Command-line changes. No more command-line args, except for ``--config``, ``--actions``, and ``--dry-run``: - ``--config`` points to a YAML client and logging configuration file. The default location is ``~/.curator/curator.yml`` - ``--actions`` arg points to a YAML action configuration file - ``--dry-run`` will simulate the action(s) which would have taken place, but not actually make any changes to the cluster or its indices. **New Features** * Snapshot restore is here! * YAML configuration files. Now a single file can define an entire batch of commands, each with their own filters, to be performed in sequence. * Sort by index age not only by index name (as with previous versions of Curator), but also by index `creation_date`, or by calculations from the Field Stats API on a timestamp field. * Atomically add/remove indices from aliases! This is possible by way of the new `IndexList` class and YAML configuration files. * State of indices pulled and stored in `IndexList` instance. Fewer API calls required to serially test for open/close, `size_in_bytes`, etc. * Filter by space now allows sorting by age! * Experimental! Use AWS IAM credentials to sign requests to Elasticsearch. This requires the end user to *manually* install the `requests_aws4auth` python module. * Optionally delete aliases from indices before closing. * An empty index or snapshot list no longer results in an error if you set ``ignore_empty_list`` to `True`. If `True` it will still log that the action was not performed, but will continue to the next action. If 'False' it will log an ERROR and exit with code 1. **API** * Updated API documentation * Class: `IndexList`. This pulls all indices at instantiation, and you apply filters, which are class methods. You can iterate over as many filters as you like, in fact, due to the YAML config file. * Class: `SnapshotList`. This pulls all snapshots from the given repository at instantiation, and you apply filters, which are class methods. You can iterate over as many filters as you like, in fact, due to the YAML config file. * Add `wait_for_completion` to Allocation and Replicas actions. These will use the client timeout, as set by default or `timeout_override`, to determine how long to wait for timeout. These are handled in batches of indices for now. * Allow `timeout_override` option for all actions. This allows for different timeout values per action. * Improve API by giving each action its own `do_dry_run()` method. **General** * Updated use documentation for Elastic main site. * Include example files for ``--config`` and ``--actions``. 4.0.0b2 (16 June 2016) ---------------------- **Second beta release of the 4.0 branch** **New Feature** * An empty index or snapshot list no longer results in an error if you set ``ignore_empty_list`` to `True`. If `True` it will still log that the action was not performed, but will continue to the next action. If 'False' it will log an ERROR and exit with code 1. (untergeek) 4.0.0b1 (13 June 2016) ---------------------- **First beta release of the 4.0 branch!** The release notes will be rehashing the new features in 4.0, rather than the bug fixes done during the alphas. **Breaking Changes** * New and improved API! * Command-line changes. No more command-line args, except for ``--config``, ``--actions``, and ``--dry-run``: - ``--config`` points to a YAML client and logging configuration file. The default location is ``~/.curator/curator.yml`` - ``--actions`` arg points to a YAML action configuration file - ``--dry-run`` will simulate the action(s) which would have taken place, but not actually make any changes to the cluster or its indices. **New Features** * Snapshot restore is here! * YAML configuration files. Now a single file can define an entire batch of commands, each with their own filters, to be performed in sequence. * Sort by index age not only by index name (as with previous versions of Curator), but also by index `creation_date`, or by calculations from the Field Stats API on a timestamp field. * Atomically add/remove indices from aliases! This is possible by way of the new `IndexList` class and YAML configuration files. * State of indices pulled and stored in `IndexList` instance. Fewer API calls required to serially test for open/close, `size_in_bytes`, etc. * Filter by space now allows sorting by age! * Experimental! Use AWS IAM credentials to sign requests to Elasticsearch. This requires the end user to *manually* install the `requests_aws4auth` python module. * Optionally delete aliases from indices before closing. **API** * Updated API documentation * Class: `IndexList`. This pulls all indices at instantiation, and you apply filters, which are class methods. You can iterate over as many filters as you like, in fact, due to the YAML config file. * Class: `SnapshotList`. This pulls all snapshots from the given repository at instantiation, and you apply filters, which are class methods. You can iterate over as many filters as you like, in fact, due to the YAML config file. * Add `wait_for_completion` to Allocation and Replicas actions. These will use the client timeout, as set by default or `timeout_override`, to determine how long to wait for timeout. These are handled in batches of indices for now. * Allow `timeout_override` option for all actions. This allows for different timeout values per action. * Improve API by giving each action its own `do_dry_run()` method. **General** * Updated use documentation for Elastic main site. * Include example files for ``--config`` and ``--actions``. 4.0.0a10 (10 June 2016) ----------------------- **New Features** * Snapshot restore is here! * Optionally delete aliases from indices before closing. Fixes #644 (untergeek) **General** * Add `wait_for_completion` to Allocation and Replicas actions. These will use the client timeout, as set by default or `timeout_override`, to determine how long to wait for timeout. These are handled in batches of indices for now. * Allow `timeout_override` option for all actions. This allows for different timeout values per action. **Bug Fixes** * Disallow use of `master_only` if multiple hosts are used. Fixes #615 (untergeek) * Fix an issue where arguments weren't being properly passed and populated. * ForceMerge replaced Optimize in ES 2.1.0. * Fix prune_nones to work with Python 2.6. Fixes #619 (untergeek) * Fix TimestringSearch to work with Python 2.6. Fixes #622 (untergeek) * Add language classifiers to ``setup.py``. Fixes #640 (untergeek) * Changed references to readthedocs.org to be readthedocs.io. 4.0.0a9 (27 Apr 2016) --------------------- **General** * Changed `create_index` API to use kwarg `extra_settings` instead of `body` * Normalized Alias action to use `name` instead of `alias`. This simplifies documentation by reducing the number of option elements. * Streamlined some code * Made `exclude` a filter element setting for all filters. Updated all examples to show this. * Improved documentation **New Features** * Alias action can now accept `extra_settings` to allow adding filters, and/or routing. 4.0.0a8 (26 Apr 2016) --------------------- **Bug Fixes** * Fix to use `optimize` with versions of Elasticsearch < 5.0 * Fix missing setting in testvars 4.0.0a7 (25 Apr 2016) --------------------- **Bug Fixes** * Fix AWS4Auth error. 4.0.0a6 (25 Apr 2016) --------------------- **General** * Documentation updates. * Improve API by giving each action its own `do_dry_run()` method. **Bug Fixes** * Do not escape characters other than ``.`` and ``-`` in timestrings. Fixes #602 (untergeek) ** New Features** * Added `CreateIndex` action. 4.0.0a4 (21 Apr 2016) --------------------- **Bug Fixes** * Require `pyyaml` 3.10 or better. * In the case that no `options` are in an action, apply the defaults. 4.0.0a3 (21 Apr 2016) --------------------- It's time for Curator 4.0 alpha! **Breaking Changes** * New API! (again?!) * Command-line changes. No more command-line args, except for ``--config``, ``--actions``, and ``--dry-run``: - ``--config`` points to a YAML client and logging configuration file. The default location is ``~/.curator/curator.yml`` - ``--actions`` arg points to a YAML action configuration file - ``--dry-run`` will simulate the action(s) which would have taken place, but not actually make any changes to the cluster or its indices. **General** * Updated API documentation * Updated use documentation for Elastic main site. * Include example files for ``--config`` and ``--actions``. **New Features** * Sort by index age not only by index name (as with previous versions of Curator), but also by index `creation_date`, or by calculations from the Field Stats API on a timestamp field. * Class: `IndexList`. This pulls all indices at instantiation, and you apply filters, which are class methods. You can iterate over as many filters as you like, in fact, due to the YAML config file. * Class: `SnapshotList`. This pulls all snapshots from the given repository at instantiation, and you apply filters, which are class methods. You can iterate over as many filters as you like, in fact, due to the YAML config file. * YAML configuration files. Now a single file can define an entire batch of commands, each with their own filters, to be performed in sequence. * Atomically add/remove indices from aliases! This is possible by way of the new `IndexList` class and YAML configuration files. * State of indices pulled and stored in `IndexList` instance. Fewer API calls required to serially test for open/close, `size_in_bytes`, etc. * Filter by space now allows sorting by age! * Experimental! Use AWS IAM credentials to sign requests to Elasticsearch. This requires the end user to *manually* install the `requests_aws4auth` python module. 3.5.1 (21 March 2016) --------------------- **Bug fixes** * Add more logging information to snapshot delete method #582 (untergeek) * Improve default timeout, logging, and exception handling for `seal` command #583 (untergeek) * Fix use of default snapshot name. #584 (untergeek) 3.5.0 (16 March 2016) --------------------- **General** * Add support for the `--client-cert` and `--client-key` command line parameters and client_cert and client_key parameters to the get_client() call. #520 (richm) **Bug fixes** * Disallow users from creating snapshots with upper-case letters, which is not permitted by Elasticsearch. #562 (untergeek) * Remove `print()` command from ``setup.py`` as it causes issues with command- line retrieval of ``--url``, etc. #568 (thib-ack) * Remove unnecessary argument from `build_filter()` #530 (zzugg) * Allow day of year filter to be made up with 1, 2 or 3 digits #578 (petitout) 3.4.1 (10 February 2016) ------------------------ **General** * Update license copyright to 2016 * Use slim python version with Docker #527 (xaka) * Changed ``--master-only`` exit code to 0 when connected to non-master node #540 (wkruse) * Add ``cx_Freeze`` capability to ``setup.py``, plus a ``binary_release.py`` script to simplify binary package creation. #554 (untergeek) * Set Elastic as author. #555 (untergeek) * Put repository creation methods into API and document them. Requested in #550 (untergeek) **Bug fixes** * Fix sphinx documentation build error #506 (hydrapolic) * Ensure snapshots are found before iterating #507 (garyelephant) * Fix a doc inconsistency #509 (pmoust) * Fix a typo in `show` documentation #513 (pbamba) * Default to trying the cluster state for checking whether indices are closed, and then fall back to using the _cat API (for Amazon ES instances). #519 (untergeek) * Improve logging to show time delay between optimize runs, if selected. #525 (untergeek) * Allow elasticsearch-py module versions through 2.3.0 (a presumption at this point) #524 (untergeek) * Improve logging in snapshot api method to reveal when a repository appears to be missing. Reported in #551 (untergeek) * Test that ``--timestring`` has the correct variable for ``--time-unit``. Reported in #544 (untergeek) * Allocation will exit with exit_code 0 now when there are no indices to work on. Reported in #531 (untergeek) 3.4.0 (28 October 2015) ----------------------- **General** * API change in elasticsearch-py 1.7.0 prevented alias operations. Fixed in #486 (HonzaKral) * During index selection you can now select only closed indices with ``--closed-only``. Does not impact ``--all-indices`` Reported in #476. Fixed in #487 (Basster) * API Changes in Elasticsearch 2.0.0 required some refactoring. All tests pass for ES versions 1.0.3 through 2.0.0-rc1. Fixed in #488 (untergeek) * es_repo_mgr now has access to the same SSL options from #462. #489 (untergeek) * Logging improvements requested in #475. (untergeek) * Added ``--quiet`` flag. #494 (untergeek) * Fixed ``index_closed`` to work with AWS Elasticsearch. #499 (univerio) * Acceptable versions of Elasticsearch-py module are 1.8.0 up to 2.1.0 (untergeek) 3.3.0 (31 August 2015) ---------------------- **Announcement** * Curator is tested in Jenkins. Each commit to the master branch is tested with both Python versions 2.7.6 and 3.4.0 against each of the following Elasticsearch versions: * 1.7_nightly * 1.6_nightly * 1.7.0 * 1.6.1 * 1.5.1 * 1.4.4 * 1.3.9 * 1.2.4 * 1.1.2 * 1.0.3 * If you are using a version different from this, your results may vary. **General** * Allocation type can now also be ``include`` or ``exclude``, in addition to the the existing default ``require`` type. Add ``--type`` to the allocation command to specify the type. #443 (steffo) * Bump elasticsearch python module dependency to 1.6.0+ to enable synced_flush API call. Reported in #447 (untergeek) * Add SSL features, ``--ssl-no-validate`` and ``certificate`` to provide other ways to validate SSL connections to Elasticsearch. #436 (untergeek) **Bug fixes** * Delete by space was only reporting space used by primary shards. Fixed to show all space consumed. Reported in #455 (untergeek) * Update exit codes and messages for snapshot selection. Reported in #452 (untergeek) * Fix potential int/float casting issues. Reported in #465 (untergeek) 3.2.3 (16 July 2015) -------------------- **Bug fix** * In order to address customer and community issues with bulk deletes, the ``master_timeout`` is now invoked for delete operations. This should address 503s with 30s timeouts in the debug log, even when ``--timeout`` is set to a much higher value. The ``master_timeout`` is tied to the ``--timeout`` flag value, but will not exceed 300 seconds. #420 (untergeek) **General** * Mixing it up a bit here by putting `General` second! The only other changes are that logging has been improved for deletes so you won't need to have the ``--debug`` flag to see if you have error codes >= 400, and some code documentation improvements. 3.2.2 (13 July 2015) -------------------- **General** * This is a very minor change. The ``mock`` library recently removed support for Python 2.6. As many Curator users are using RHEL/CentOS 6, which is pinned to Python 2.6, this requires the mock version referenced by Curator to also be pinned to a supported version (``mock==1.0.1``). 3.2.1 (10 July 2015) -------------------- **General** * Added delete verification & retry (fixed at 3x) to potentially cover an edge case in #420 (untergeek) * Since GitHub allows rST (reStructuredText) README documents, and that's what PyPI wants also, the README has been rebuilt in rST. (untergeek) **Bug fixes** * If closing indices with ES 1.6+, and all indices are closed, ensure that the seal command does not try to seal all indices. Reported in #426 (untergeek) * Capture AttributeError when sealing indices if a non-TransportError occurs. Reported in #429 (untergeek) 3.2.0 (25 June 2015) -------------------- **New!** * Added support to manually seal, or perform a [synced flush](http://www.elastic.co/guide/en/elasticsearch/reference/current/indices-synced-flush.html) on indices with the ``seal`` command. #394 (untergeek) * Added *experimental* support for SSL certificate validation. In order for this to work, you must install the ``certifi`` python module: ``pip install certifi`` This feature *should* automatically work if the ``certifi`` module is installed. Please report any issues. **General** * Changed logging to go to stdout rather than stderr. Reopened #121 and figured they were right. This is better. (untergeek) * Exit code 99 was unpopular. It has been removed. Reported in #371 and #391 (untergeek) * Add ``--skip-repo-validation`` flag for snapshots. Do not validate write access to repository on all cluster nodes before proceeding. Useful for shared filesystems where intermittent timeouts can affect validation, but won't likely affect snapshot success. Requested in #396 (untergeek) * An alias no longer needs to be pre-existent in order to use the alias command. #317 (untergeek) * es_repo_mgr now passes through upstream errors in the event a repository fails to be created. Requested in #405 (untergeek) **Bug fixes** * In rare cases, ``*`` wildcard would not expand. Replaced with _all. Reported in #399 (untergeek) * Beginning with Elasticsearch 1.6, closed indices cannot have their replica count altered. Attempting to do so results in this error: ``org.elasticsearch.ElasticsearchIllegalArgumentException: Can't update [index.number_of_replicas] on closed indices [[test_index]] - can leave index in an unopenable state`` As a result, the ``change_replicas`` method has been updated to prune closed indices. This change will apply to all versions of Elasticsearch. Reported in #400 (untergeek) * Fixed es_repo_mgr repository creation verification error. Reported in #389 (untergeek) 3.1.0 (21 May 2015) ------------------- **General** * If ``wait_for_completion`` is true, snapshot success is now tested and logged. Reported in #253 (untergeek) * Log & return false if a snapshot is already in progress (untergeek) * Logs individual deletes per index, even though they happen in batch mode. Also log individual snapshot deletions. Reported in #372 (untergeek) * Moved ``chunk_index_list`` from cli to api utils as it's now also used by ``filter.py`` * Added a warning and 10 second timer countdown if you use ``--timestring`` to filter indices, but do not use ``--older-than`` or ``--newer-than`` in conjunction with it. This is to address #348, which behavior isn't a bug, but prevents accidental action against all of your time-series indices. The warning and timer are not displayed for ``show`` and ``--dry-run`` operations. * Added tests for ``es_repo_mgr`` in #350 * Doc fixes **Bug fixes** * delete-by-space needed the same fix used for #245. Fixed in #353 (untergeek) * Increase default client timeout for ``es_repo_mgr`` as node discovery and availability checks for S3 repositories can take a bit. Fixed in #352 (untergeek) * If an index is closed, indicate in ``show`` and ``--dry-run`` output. Reported in #327. (untergeek) * Fix issue where CLI parameters were not being passed to the ``es_repo_mgr`` create sub-command. Reported in #337. (feltnerm) 3.0.3 (27 Mar 2015) ------------------- **Announcement** This is a bug fix release. #319 and #320 are affecting a few users, so this release is being expedited. Test count: 228 Code coverage: 99% **General** * Documentation for the CLI converted to Asciidoc and moved to http://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html * Improved logging, and refactored a few methods to help with this. * Dry-run output is now more like v2, with the index or snapshot in the log line, along with the command. Several tests needed refactoring with this change, along with a bit of documentation. **Bug fixes** * Fix links to repository in setup.py. Reported in #318 (untergeek) * No more ``--delay`` with optimized indices. Reported in #319 (untergeek) * ``--request_timeout`` not working as expected. Reinstate the version 2 timeout override feature to prevent default timeouts for ``optimize`` and ``snapshot`` operations. Reported in #320 (untergeek) * Reduce index count to 200 for test.integration.test_cli_commands.TestCLISnapshot.test_cli_snapshot_huge_list in order to reduce or eliminate Jenkins CI test timeouts. Reported in #324 (untergeek) * ``--dry-run`` no longer calls ``show``, but will show output in the log, as in v2. This was a recurring complaint. See #328 (untergeek) 3.0.2 (23 Mar 2015) ------------------- **Announcement** This is a bug fix release. #307 and #309 were big enough to warrant an expedited release. **Bug fixes** * Purge unneeded constants, and clean up config options for snapshot. Reported in #303 (untergeek) * Don't split large index list if performing snapshots. Reported in #307 (untergeek) * Act correctly if a zero value for `--older-than` or `--newer-than` is provided. #309 (untergeek) 3.0.1 (16 Mar 2015) ------------------- **Announcement** The ``regex_iterate`` method was horribly named. It has been renamed to ``apply_filter``. Methods have been added to allow API users to build a filtered list of indices similarly to how the CLI does. This was an oversight. Props to @SegFaultAX for pointing this out. **General** * In conjunction with the rebrand to Elastic, URLs and documentation were updated. * Renamed horribly named `regex_iterate` method to `apply_filter` #298 (untergeek) * Added `build_filter` method to mimic CLI calls. #298 (untergeek) * Added Examples page in the API documentation. #298 (untergeek) **Bug fixes** * Refactored to show `--dry-run` info for `--disk-space` calls. Reported in #290 (untergeek) * Added list chunking so acting on huge lists of indices won't result in a URL bigger than 4096 bytes (Elasticsearch's default limit.) Reported in https://github.com/elastic/curator/issues/245#issuecomment-77916081 * Refactored `to_csv()` method to be simpler. * Added and removed tests according to changes. Code coverage still at 99% 3.0.0 (9 March 2015) -------------------- **Release Notes** The full release of Curator 3.0 is out! Check out all of the changes here! *Note:* This release is _not_ reverse compatible with any previous version. Because 3.0 is a major point release, there have been some major changes to both the API as well as the CLI arguments and structure. Be sure to read the updated command-line specific docs in the [wiki](https://github.com/elasticsearch/curator/wiki) and change your command-line arguments accordingly. The API docs are still at http://curator.readthedocs.io. Be sure to read the latest docs, or select the docs for 3.0.0. **General** * **Breaking changes to the API.** Because this is a major point revision, changes to the API have been made which are non-reverse compatible. Before upgrading, be sure to update your scripts and test them thoroughly. * **Python 3 support** Somewhere along the line, Curator would no longer work with curator. All tests now pass for both Python2 and Python3, with 99% code coverage in both environments. * **New CLI library.** Using Click now. http://click.pocoo.org/3/ This change is especially important as it allows very easy CLI integration testing. * **Pipelined filtering!** You can now use ``--older-than`` & ``--newer-than`` in the same command! You can also provide your own regex via the ``--regex`` parameter. You can use multiple instances of the ``--exclude`` flag. * **Manually include indices!** With the ``--index`` paramter, you can add an index to the working list. You can provide multiple instances of the ``--index`` parameter as well! * **Tests!** So many tests now. Test coverage of the API methods is at 100% now, and at 99% for the CLI methods. This doesn't mean that all of the tests are perfect, or that I haven't missed some scenarios. It does mean, however, that it will be much easier to write tests if something turns up missed. It also means that any new functionality will now need to have tests. * **Iteration changes** Methods now only iterate through each index when appropriate! In fact, the only commands that iterate are `alias` and `optimize`. The `bloom` command will iterate, but only if you have added the `--delay` flag with a value greater than zero. * **Improved packaging!** Methods have been moved into categories of ``api`` and ``cli``, and further broken out into individual modules to help them be easier to find and read. * Check for allocation before potentially re-applying an allocation rule. #273 (ferki) * Assigning replica count and routing allocation rules _can_ be done to closed indices. #283 (ferki) **Bug fixes** * Don't accidentally delete ``.kibana`` index. #261 (malagoli) * Fix segment count for empty indices. #265 (untergeek) * Change bloom filter cutoff Elasticsearch version to 1.4. Reported in #267 (untergeek) 3.0.0rc1 (5 March 2015) ----------------------- **Release Notes** RC1 is here! I'm re-releasing the Changes from all betas here, minus the intra-beta code fixes. Barring any show stoppers, the official release will be soon. **General** * **Breaking changes to the API.** Because this is a major point revision, changes to the API have been made which are non-reverse compatible. Before upgrading, be sure to update your scripts and test them thoroughly. * **Python 3 support** Somewhere along the line, Curator would no longer work with curator. All tests now pass for both Python2 and Python3, with 99% code coverage in both environments. * **New CLI library.** Using Click now. http://click.pocoo.org/3/ This change is especially important as it allows very easy CLI integration testing. * **Pipelined filtering!** You can now use ``--older-than`` & ``--newer-than`` in the same command! You can also provide your own regex via the ``--regex`` parameter. You can use multiple instances of the ``--exclude`` flag. * **Manually include indices!** With the ``--index`` paramter, you can add an index to the working list. You can provide multiple instances of the ``--index`` parameter as well! * **Tests!** So many tests now. Test coverage of the API methods is at 100% now, and at 99% for the CLI methods. This doesn't mean that all of the tests are perfect, or that I haven't missed some scenarios. It does mean, however, that it will be much easier to write tests if something turns up missed. It also means that any new functionality will now need to have tests. * Methods now only iterate through each index when appropriate! * Improved packaging! Hopefully the ``entry_point`` issues some users have had will be addressed by this. Methods have been moved into categories of ``api`` and ``cli``, and further broken out into individual modules to help them be easier to find and read. * Check for allocation before potentially re-applying an allocation rule. #273 (ferki) * Assigning replica count and routing allocation rules _can_ be done to closed indices. #283 (ferki) **Bug fixes** * Don't accidentally delete ``.kibana`` index. #261 (malagoli) * Fix segment count for empty indices. #265 (untergeek) * Change bloom filter cutoff Elasticsearch version to 1.4. Reported in #267 (untergeek) 3.0.0b4 (5 March 2015) ---------------------- **Notes** Integration testing! Because I finally figured out how to use the Click Testing API, I now have a good collection of command-line simulations, complete with a real back-end. This testing found a few bugs (this is why testing exists, right?), and fixed a few of them. **Bug fixes** * HUGE! `curator show snapshots` would _delete_ snapshots. This is fixed. * Return values are now being sent from the commands. * `scripttest` is no longer necessary (click.Test works!) * Calling `get_snapshot` without a snapshot name returns all snapshots 3.0.0b3 (4 March 2015) ---------------------- **Bug fixes** * setup.py was lacking the new packages "curator.api" and "curator.cli" The package works now. * Python3 suggested I had to normalize the beta tag to just b3, so that's also changed. * Cleaned out superfluous imports and logger references from the __init__.py files. 3.0.0-beta2 (3 March 2015) -------------------------- **Bug fixes** * Python3 issues resolved. Tests now pass on both Python2 and Python3 3.0.0-beta1 (3 March 2015) -------------------------- **General** * **Breaking changes to the API.** Because this is a major point revision, changes to the API have been made which are non-reverse compatible. Before upgrading, be sure to update your scripts and test them thoroughly. * **New CLI library.** Using Click now. http://click.pocoo.org/3/ * **Pipelined filtering!** You can now use ``--older-than`` & ``--newer-than`` in the same command! You can also provide your own regex via the ``--regex`` parameter. You can use multiple instances of the ``--exclude`` flag. * **Manually include indices!** With the ``--index`` paramter, you can add an index to the working list. You can provide multiple instances of the ``--index`` parameter as well! * **Tests!** So many tests now. Unit test coverage of the API methods is at 100% now. This doesn't mean that all of the tests are perfect, or that I haven't missed some scenarios. It does mean that any new functionality will need to also have tests, now. * Methods now only iterate through each index when appropriate! * Improved packaging! Hopefully the ``entry_point`` issues some users have had will be addressed by this. Methods have been moved into categories of ``api`` and ``cli``, and further broken out into individual modules to help them be easier to find and read. * Check for allocation before potentially re-applying an allocation rule. #273 (ferki) **Bug fixes** * Don't accidentally delete ``.kibana`` index. #261 (malagoli) * Fix segment count for empty indices. #265 (untergeek) * Change bloom filter cutoff Elasticsearch version to 1.4. Reported in #267 (untergeek) 2.1.2 (22 January 2015) ----------------------- **Bug fixes** * Do not try to set replica count if count matches provided argument. #247 (bobrik) * Fix JSON logging (Logstash format). #250 (magnusbaeck) * Fix bug in `filter_by_space()` which would match all indices if the provided patterns found no matches. Reported in #254 (untergeek) 2.1.1 (30 December 2014) ------------------------ **Bug fixes** * Renamed unnecessarily redundant ``--replicas`` to ``--count`` in args for ``curator_script.py`` 2.1.0 (30 December 2014) ------------------------ **General** * Snapshot name now appears in log output or STDOUT. #178 (untergeek) * Replicas! You can now change the replica count of indices. Requested in #175 (untergeek) * Delay option added to Bloom Filter functionality. #206 (untergeek) * Add 2-digit years as acceptable pattern (y vs. Y). Reported in #209 (untergeek) * Add Docker container definition #226 (christianvozar) * Allow the use of 0 with --older-than, --most-recent and --delete-older-than. See #208. #211 (bobrik) **Bug fixes** * Edge case where 1.4.0.Beta1-SNAPSHOT would break version check. Reported in #183 (untergeek) * Typo fixed. #193 (ferki) * Type fixed. #204 (gheppner) * Shows proper error in the event of concurrent snapshots. #177 (untergeek) * Fixes erroneous index display of ``_, a, l, l`` when --all-indices selected. Reported in #222 (untergeek) * Use json.dumps() to escape exceptions. Reported in #210 (untergeek) * Check if index is closed before adding to alias. Reported in #214 (bt5e) * No longer force-install argparse if pre-installed #216 (whyscream) * Bloom filters have been removed from Elasticsearch 1.5.0. Update methods and tests to act accordingly. #233 (untergeek) 2.0.2 (8 October 2014) ---------------------- **Bug fixes** * Snapshot name not displayed in log or STDOUT #185 (untergeek) * Variable name collision in delete_snapshot() #186 (untergeek) 2.0.1 (1 October 2014) ---------------------- **Bug fix** * Override default timeout when snapshotting --all-indices #179 (untergeek) 2.0.0 (25 September 2014) ------------------------- **General** * New! Separation of Elasticsearch Curator Python API and curator_script.py (untergeek) * New! ``--delay`` after optimize to allow cluster to quiesce #131 (untergeek) * New! ``--suffix`` option in addition to ``--prefix`` #136 (untergeek) * New! Support for wildcards in prefix & suffix #136 (untergeek) * Complete refactor of snapshots. Now supporting incrementals! (untergeek) **Bug fix** * Incorrect error msg if no indices sent to create_snapshot (untergeek) * Correct for API change coming in ES 1.4 #168 (untergeek) * Missing ``"`` in Logstash log format #143 (cassianoleal) * Change non-master node test to exit code 0, log as ``INFO``. #145 (untergeek) * `months` option missing from validate_timestring() (untergeek) 1.2.2 (29 July 2014) -------------------- **Bug fix** * Updated ``README.md`` to briefly explain what curator does #117 (untergeek) * Fixed ``es_repo_mgr`` logging whitelist #119 (untergeek) * Fixed absent ``months`` time-unit #120 (untergeek) * Filter out ``.marvel-kibana`` when prefix is ``.marvel-`` #120 (untergeek) * Clean up arg parsing code where redundancy exists #123 (untergeek) * Properly divide debug from non-debug logging #125 (untergeek) * Fixed ``show`` command bug caused by changes to command structure #126 (michaelweiser) 1.2.1 (24 July 2014) -------------------- **Bug fix** * Fixed the new logging when called by ``curator`` entrypoint. 1.2.0 (24 July 2014) -------------------- **General** * New! Allow user-specified date patterns: ``--timestring`` #111 (untergeek) * New! Curate weekly indices (must use week of year) #111 (untergeek) * New! Log output in logstash format ``--logformat logstash`` #111 (untergeek) * Updated! Cleaner default logs (debug still shows everything) (untergeek) * Improved! Dry runs are more visible in log output (untergeek) Errata * The ``--separator`` option was removed in lieu of user-specified date patterns. * Default ``--timestring`` for days: ``%Y.%m.%d`` (Same as before) * Default ``--timestring`` for hours: ``%Y.%m.%d.%H`` (Same as before) * Default ``--timestring`` for weeks: ``%Y.%W`` 1.1.3 (18 July 2014) -------------------- **Bug fix** * Prefix not passed in ``get_object_list()`` #106 (untergeek) * Use ``os.devnull`` instead of ``/dev/null`` for Windows #102 (untergeek) * The http auth feature was erroneously omitted #100 (bbuchacher) 1.1.2 (13 June 2014) -------------------- **Bug fix** * This was a showstopper bug for anyone using RHEL/CentOS with a Python 2.6 dependency for yum * Python 2.6 does not like format calls without an index. #96 via #95 (untergeek) * We won't talk about what happened to 1.1.1. No really. I hate git today :( 1.1.0 (12 June 2014) -------------------- **General** * Updated! New command structure * New! Snapshot to fs or s3 #82 (untergeek) * New! Add/Remove indices to alias #82 via #86 (cschellenger) * New! ``--exclude-pattern`` #80 (ekamil) * New! (sort of) Restored ``--log-level`` support #73 (xavier-calland) * New! show command-line options #82 via #68 (untergeek) * New! Shard Allocation Routing #82 via #62 (nickethier) **Bug fix** * Fix ``--max_num_segments`` not being passed correctly #74 (untergeek) * Change ``BUILD_NUMBER`` to ``CURATOR_BUILD_NUMBER`` in ``setup.py`` #60 (mohabusama) * Fix off-by-one error in time calculations #66 (untergeek) * Fix testing with python3 #92 (untergeek) Errata * Removed ``optparse`` compatibility. Now requires ``argparse``. 1.0.0 (25 Mar 2014) ------------------- **General** * compatible with ``elasticsearch-py`` 1.0 and Elasticsearch 1.0 (honzakral) * Lots of tests! (honzakral) * Streamline code for 1.0 ES versions (honzakral) **Bug fix** * Fix ``find_expired_indices()`` to not skip closed indices (honzakral) 0.6.2 (18 Feb 2014) ------------------- **General** * Documentation fixes #38 (dharrigan) * Add support for HTTPS URI scheme and ``optparse`` compatibility for Python 2.6 (gelim) * Add elasticsearch module version checking for future compatibility checks (untergeek) 0.6.1 (08 Feb 2014) ------------------- **General** * Added tarball versioning to ``setup.py`` (untergeek) **Bug fix** * Fix ``long_description`` by including ``README.md`` in ``MANIFEST.in`` (untergeek) * Incorrect version number in ``curator.py`` (untergeek) 0.6.0 (08 Feb 2014) ------------------- **General** * Restructured repository to a be a proper python package. (arieb) * Added ``setup.py`` file. (arieb) * Removed the deprecated file ``logstash_index_cleaner.py`` (arieb) * Updated ``README.md`` to fit the new package, most importantly the usage and installation. (arieb) * Fixes and package push to PyPI (untergeek) 0.5.2 (26 Jan 2014) ------------------- **General** * Fix boolean logic determining hours or days for time selection (untergeek) 0.5.1 (20 Jan 2014) ------------------- **General** * Fix ``can_bloom`` to compare numbers (HonzaKral) * Switched ``find_expired_indices()`` to use ``datetime`` and ``timedelta`` * Do not try and catch unrecoverable exceptions. (HonzaKral) * Future proofing the use of the elasticsearch client (i.e. work with version 1.0+ of Elasticsearch) (HonzaKral) Needs more testing, but should work. * Add tests for these scenarios (HonzaKral) 0.5.0 (17 Jan 2014) ------------------- **General** * Deprecated ``logstash_index_cleaner.py`` Use new ``curator.py`` instead (untergeek) * new script change: ``curator.py`` (untergeek) * new add index optimization (Lucene forceMerge) to reduce segments and therefore memory usage. (untergeek) * update refactor of args and several functions to streamline operation and make it more readable (untergeek) * update refactor further to clean up and allow immediate (and future) portability (HonzaKral) 0.4.0 ----- **General** * First version logged in ``CHANGELOG`` * new ``--disable-bloom-days`` feature requires 0.90.9+ http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-codec.html#bloom-postings This can save a lot of heap space on cold indexes (i.e. not actively indexing documents) curator-4.2.5/docs/Makefile000066400000000000000000000152061302700154000155650ustar00rootroot00000000000000# Makefile for Sphinx documentation # # You can set these variables from the command line. SPHINXOPTS = SPHINXBUILD = sphinx-build PAPER = BUILDDIR = _build # User-friendly check for sphinx-build ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1) $(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/) endif # Internal variables. PAPEROPT_a4 = -D latex_paper_size=a4 PAPEROPT_letter = -D latex_paper_size=letter ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . # the i18n builder cannot share the environment and doctrees with the others I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . .PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest gettext help: @echo "Please use \`make ' where is one of" @echo " html to make standalone HTML files" @echo " dirhtml to make HTML files named index.html in directories" @echo " singlehtml to make a single large HTML file" @echo " pickle to make pickle files" @echo " json to make JSON files" @echo " htmlhelp to make HTML files and a HTML help project" @echo " qthelp to make HTML files and a qthelp project" @echo " devhelp to make HTML files and a Devhelp project" @echo " epub to make an epub" @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" @echo " latexpdf to make LaTeX files and run them through pdflatex" @echo " latexpdfja to make LaTeX files and run them through platex/dvipdfmx" @echo " text to make text files" @echo " man to make manual pages" @echo " texinfo to make Texinfo files" @echo " info to make Texinfo files and run them through makeinfo" @echo " gettext to make PO message catalogs" @echo " changes to make an overview of all changed/added/deprecated items" @echo " xml to make Docutils-native XML files" @echo " pseudoxml to make pseudoxml-XML files for display purposes" @echo " linkcheck to check all external links for integrity" @echo " doctest to run all doctests embedded in the documentation (if enabled)" clean: rm -rf $(BUILDDIR)/* html: $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html @echo @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." dirhtml: $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml @echo @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." singlehtml: $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml @echo @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml." pickle: $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle @echo @echo "Build finished; now you can process the pickle files." json: $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json @echo @echo "Build finished; now you can process the JSON files." htmlhelp: $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp @echo @echo "Build finished; now you can run HTML Help Workshop with the" \ ".hhp project file in $(BUILDDIR)/htmlhelp." qthelp: $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp @echo @echo "Build finished; now you can run "qcollectiongenerator" with the" \ ".qhcp project file in $(BUILDDIR)/qthelp, like this:" @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/Elasticsearch.qhcp" @echo "To view the help file:" @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/Elasticsearch.qhc" devhelp: $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp @echo @echo "Build finished." @echo "To view the help file:" @echo "# mkdir -p $$HOME/.local/share/devhelp/Elasticsearch" @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/Elasticsearch" @echo "# devhelp" epub: $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub @echo @echo "Build finished. The epub file is in $(BUILDDIR)/epub." latex: $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex @echo @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." @echo "Run \`make' in that directory to run these through (pdf)latex" \ "(use \`make latexpdf' here to do that automatically)." latexpdf: $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex @echo "Running LaTeX files through pdflatex..." $(MAKE) -C $(BUILDDIR)/latex all-pdf @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." latexpdfja: $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex @echo "Running LaTeX files through platex and dvipdfmx..." $(MAKE) -C $(BUILDDIR)/latex all-pdf-ja @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." text: $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text @echo @echo "Build finished. The text files are in $(BUILDDIR)/text." man: $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man @echo @echo "Build finished. The manual pages are in $(BUILDDIR)/man." texinfo: $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo @echo @echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo." @echo "Run \`make' in that directory to run these through makeinfo" \ "(use \`make info' here to do that automatically)." info: $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo @echo "Running Texinfo files through makeinfo..." make -C $(BUILDDIR)/texinfo info @echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo." gettext: $(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale @echo @echo "Build finished. The message catalogs are in $(BUILDDIR)/locale." changes: $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes @echo @echo "The overview file is in $(BUILDDIR)/changes." linkcheck: $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck @echo @echo "Link check complete; look for any errors in the above output " \ "or in $(BUILDDIR)/linkcheck/output.txt." doctest: $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest @echo "Testing of doctests in the sources finished, look at the " \ "results in $(BUILDDIR)/doctest/output.txt." xml: $(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml @echo @echo "Build finished. The XML files are in $(BUILDDIR)/xml." pseudoxml: $(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml @echo @echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml." curator-4.2.5/docs/actionclasses.rst000066400000000000000000000022661302700154000175140ustar00rootroot00000000000000.. _actionclasses: Action Classes ============== .. seealso:: It is important to note that each action has a `do_action()` method, which accepts no arguments. This is the means by which all actions are executed. * `Alias`_ * `Allocation`_ * `Close`_ * `ClusterRouting`_ * `DeleteIndices`_ * `DeleteSnapshots`_ * `ForceMerge`_ * `Open`_ * `Replicas`_ * `Snapshot`_ Alias ----- .. autoclass:: curator.actions.Alias :members: Allocation ---------- .. autoclass:: curator.actions.Allocation :members: Close ----- .. autoclass:: curator.actions.Close :members: ClusterRouting -------------- .. autoclass:: curator.actions.ClusterRouting :members: DeleteIndices ------------- .. autoclass:: curator.actions.DeleteIndices :members: DeleteSnapshots --------------- .. autoclass:: curator.actions.DeleteSnapshots :members: ForceMerge ---------- .. autoclass:: curator.actions.ForceMerge :members: Open ---- .. autoclass:: curator.actions.Open :members: Replicas -------- .. autoclass:: curator.actions.Replicas :members: Snapshot -------- .. autoclass:: curator.actions.Snapshot :members: Restore ------- .. autoclass:: curator.actions.Restore :members: curator-4.2.5/docs/asciidoc/000077500000000000000000000000001302700154000156775ustar00rootroot00000000000000curator-4.2.5/docs/asciidoc/about.asciidoc000066400000000000000000000125201302700154000205110ustar00rootroot00000000000000[[about]] = About [partintro] -- Elasticsearch Curator helps you curate, or manage, your Elasticsearch indices and snapshots by: 1. Obtaining the full list of indices (or snapshots) from the cluster, as the _actionable list_ 2. Iterate through a list of user-defined <> to progressively remove indices (or snapshots) from this _actionable list_ as needed. 3. Perform various <> on the items which remain in the _actionable list._ Learn More: * <> * <> * <> * <> * <> * <> * <> -- [[about-origin]] == Origin Curator was first called https://logstash.jira.com/browse/LOGSTASH-211[`clearESindices.py`]. Its sole function was to delete indices. It was almost immediately renamed to https://logstash.jira.com/browse/LOGSTASH-211[`logstash_index_cleaner.py`]. After a time it was briefly relocated under the https://github.com/elastic/logstash[logstash] repository as `expire_logs`, at which point it began to gain new functionality. Soon thereafter, Jordan Sissel was hired by Elastic (then still Elasticsearch), as was the original author of Curator. Not long after that it became Elasticsearch Curator and is now hosted at https://github.com/elastic/curator Curator now performs many operations on your Elasticsearch indices, from delete to snapshot to shard allocation routing. [[about-features]] == Features Curator allows for many different operations to be performed to both indices and snapshots, including: * Add or remove indices (or both!) from an <> * Change shard routing <> * <> indices * <> * <> * <> * <> closed indices * <> indices * Change the number of <> per shard for indices * Take a <> (backup) of indices * <> snapshots [[about-cli]] == Command-Line Interface (CLI) Curator has always been a command-line tool. When Curator 2.0 was released, it was the first attempt to separate the <> from the CLI. This site provides the documentation for how to use Curator on the command-line. TIP: Learn more about <>. [[about-api]] == Application Program Interface (API) Since version 2.0, Curator ships with both an API and wrapper scripts (which are actually defined as entry points). The API, or Application Program Interface, allows you to write your own scripts to accomplish similar goals--or even new and different things--with the same code that Curator uses. The API documentation is not on this site, but is available at http://curator.readthedocs.io/. The Curator API is built using the http://www.elastic.co/guide/en/elasticsearch/client/python-api/current/index.html[Elasticsearch Python API]. [[license]] == License Copyright (c) 2011–2016 Elasticsearch Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. [[site-corrections]] == Site Corrections All documentation on this site allows for quick correction submission. To do this, use the "Edit" link to the right on any page. * You will need to log in with your GitHub account. * Make your corrections or additions to the documentation. * Please use the option to "Create a new branch for this commit and start a pull request." * Please make sure you have signed our http://www.elastic.co/contributor-agreement/[Contributor License Agreement]. We are not asking you to assign copyright to us, but to give us the right to distribute your code (even documentation corrections) without restriction. We ask this of all contributors in order to assure our users of the origin and continuing existence of the code. You only need to sign the CLA once. If you are uncomfortable with this, feel free to submit a https://github.com/elastic/curator/issues[GitHub Issue] with your suggested correction instead. * Changes will be reviewed and merged, if acceptable. [[about-contributing]] == Contributing We welcome contributions and bug fixes to Curator's API and CLI. We are grateful for the many https://github.com/elastic/curator/blob/master/CONTRIBUTORS[contributors] who have helped Curator become what it is today. Please read through our https://github.com/elastic/curator/blob/master/CONTRIBUTING.md[contribution] guide, and the Curator https://github.com/elastic/curator/blob/master/README.md[readme] document. A brief overview of the steps * fork the repo * make changes in your fork * add tests to cover your changes (if necessary) * run tests * sign the http://elastic.co/contributor-agreement/[CLA] * send a pull request! TIP: To submit documentation fixes for this site, see <> curator-4.2.5/docs/asciidoc/actions.asciidoc000066400000000000000000000511061302700154000210420ustar00rootroot00000000000000[[actions]] = Actions [partintro] -- Actions are the tasks which Curator can perform on your indices. Snapshots, once created, can only be deleted. * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> -- [[alias]] == Alias [source,yaml] ------------- action: alias description: "Add/Remove selected indices to or from the specified alias" options: name: alias_name extra_settings: timeout_override: continue_if_exception: False disable_action: False add: filters: - filtertype: ... remove: filters: - filtertype: ... ------------- NOTE: Empty values and commented lines will result in the default value, if any, being selected. If a setting is set, but not used by a given action, it will be ignored. This action adds and/or removes indices from the alias identified by <> The <> under the `add` and `remove` directives define which indices will be added and/or removed. This is an atomic action, so adds and removes happen instantaneously. The <> option allows the addition of extra settings with the `add` directive. These settings are ignored for `remove`. An example of how these settings can be used to create a filtered alias might be: [source,yaml] ------------- extra_settings: filter: term: user: kimchy ------------- WARNING: Before creating a filtered alias, first ensure that the fields already exist in the mapping. Learn more about adding filtering and routing to aliases in the {ref}/indices-aliases.html[Elasticsearch Alias API documentation]. [float] Required settings ~~~~~~~~~~~~~~~~~ * <> (required) [float] Optional settings ~~~~~~~~~~~~~~~~~ * <> (No default value.) * <> (can override the default) * <> (can override the default <>) * <> (has a default value which can optionally be changed) * <> (has a default value which can optionally be changed) TIP: See an example of this action in an <> <>. [[allocation]] == Allocation [source,yaml] ------------- action: allocation description: "Apply shard allocation filtering rules to the specified indices" options: key: value: allocation_type: wait_for_completion: False timeout_override: continue_if_exception: False disable_action: False filters: - filtertype: ... ------------- NOTE: Empty values and commented lines will result in the default value, if any, being selected. If a setting is set, but not used by a given action, it will be ignored. This action changes the shard routing allocation for the selected indices. See https://www.elastic.co/guide/en/elasticsearch/reference/current/shard-allocation-filtering.html for more information. You can optionally set `wait_for_completion` to `True` to have Curator wait for the shard routing to complete before continuing. If this option is chosen, it is advisable to use the <> option in order to avoid client timeouts. [float] Required settings ~~~~~~~~~~~~~~~~~ * <> (required) * <> (required) [float] Optional settings ~~~~~~~~~~~~~~~~~ * <> (has a default value which can optionally be changed) * <> (has a default value which can optionally be changed) * <> (can override the default) * <> (can override the default <>) * <> (has a default value which can optionally be changed) * <> (has a default value which can optionally be changed) TIP: See an example of this action in an <> <>. [[close]] == Close [source,yaml] ------------- action: close description: "Close selected indices" options: delete_aliases: False timeout_override: continue_if_exception: False disable_action: False filters: - filtertype: ... ------------- NOTE: Empty values and commented lines will result in the default value, if any, being selected. If a setting is set, but not used by a given action, it will be ignored. This action closes the selected indices, and optionally deletes associated aliases beforehand. [float] Optional settings ~~~~~~~~~~~~~~~~~ * <> (has a default value which can optionally be changed) * <> (can override the default) * <> (can override the default <>) * <> (has a default value which can optionally be changed) * <> (has a default value which can optionally be changed) TIP: See an example of this action in an <> <>. [[cluster_routing]] == Cluster Routing [source,yaml] ------------- action: cluster_routing description: "Apply routing rules to the entire cluster" options: routing_type: value: setting: enable wait_for_completion: False timeout_override: continue_if_exception: False disable_action: False ------------- NOTE: Empty values and commented lines will result in the default value, if any, being selected. If a setting is set, but not used by a given action, it will be ignored. This action changes the shard routing allocation for the selected indices. See https://www.elastic.co/guide/en/elasticsearch/reference/current/shards-allocation.html for more information. You can optionally set `wait_for_completion` to `True` to have Curator wait for the shard routing to complete before continuing. If this option is chosen, it is advisable to use the <> option in order to avoid client timeouts. [float] Required settings ~~~~~~~~~~~~~~~~~ * <> (required) * <> (required) * <> Currently must be set to `enable`. This setting is a placeholder for potential future expansion. [float] Optional settings ~~~~~~~~~~~~~~~~~ * <> (has a default value which can optionally be changed) * <> (can override the default <>) * <> (has a default value which can optionally be changed) * <> (has a default value which can optionally be changed) TIP: See an example of this action in an <> <>. [[create_index]] == Create Index [source,yaml] ------------- action: create_index description: "Create index as named" options: name: extra_settings: timeout_override: continue_if_exception: False disable_action: False ------------- NOTE: Empty values and commented lines will result in the default value, if any, being selected. If a setting is set, but not used by a given action, it will be ignored. This action creates the named index. The <> option allows the addition of extra settings, such as index settings and mappings. An example of how these settings can be used to create an index might be: [source,yaml] ------------- extra_settings: settings: number_of_shards: 1 number_of_replicas: 0 mappings: type1 : properties: field1: type: string index: not_analyzed ------------- [float] Required settings ~~~~~~~~~~~~~~~~~ * <> [float] Optional settings ~~~~~~~~~~~~~~~~~ * <> No default value. You can add any acceptable index settings and mappings as nested YAML. See the {ref}/indices-create-index.html[Elasticsearch Create Index API documentation] for more information. * <> (can override the default <>) * <> (has a default value which can optionally be changed) * <> (has a default value which can optionally be changed) TIP: See an example of this action in an <> <>. [[delete_indices]] == Delete Indices [source,yaml] ------------- action: delete_indices description: "Delete selected indices" options: timeout_override: continue_if_exception: False disable_action: False filters: - filtertype: ... ------------- NOTE: Empty values and commented lines will result in the default value, if any, being selected. If a setting is set, but not used by a given action, it will be ignored. This action deletes the selected indices. [float] Optional settings ~~~~~~~~~~~~~~~~~ * <> (can override the default) * <> (can override the default <>) * <> (has a default value which can optionally be changed) * <> (has a default value which can optionally be changed) TIP: See an example of this action in an <> <>. [[delete_snapshots]] == Delete Snapshots [source,yaml] ------------- action: delete_snapshots description: "Delete selected snapshots from 'repository'" options: repository: retry_interval: retry_count: timeout_override: continue_if_exception: False disable_action: False filters: - filtertype: ... ------------- NOTE: Empty values and commented lines will result in the default value, if any, being selected. If a setting is set, but not used by a given action, it will be ignored. This action deletes the selected snapshots from the selected <>. If issues are encountered, it will retry up to <> times, with a delay of <> seconds between retries. [float] Required settings ~~~~~~~~~~~~~~~~~ * <> (required) [float] Optional settings ~~~~~~~~~~~~~~~~~ * <> (has a default value which can optionally be changed) * <> (has a default value which can optionally be changed) * <> (can override the default) * <> (can override the default <>) * <> (has a default value which can optionally be changed) * <> (has a default value which can optionally be changed) TIP: See an example of this action in an <> <>. [[open]] == Open [source,yaml] ------------- action: open description: "open selected indices" options: timeout_override: continue_if_exception: False disable_action: False filters: - filtertype: ... ------------- NOTE: Empty values and commented lines will result in the default value, if any, being selected. If a setting is set, but not used by a given action, it will be ignored. This action opens the selected indices. [float] Optional settings ~~~~~~~~~~~~~~~~~ * <> (can override the default) * <> (can override the default <>) * <> (has a default value which can optionally be changed) * <> (has a default value which can optionally be changed) TIP: See an example of this action in an <> <>. [[forcemerge]] == Forcemerge [source,yaml] ------------- action: forcemerge description: "Perform a forceMerge on selected indices to 'max_num_segments' per shard" options: max_num_segments: delay: timeout_override: continue_if_exception: False disable_action: False filters: - filtertype: ... ------------- NOTE: Empty values and commented lines will result in the default value, if any, being selected. If a setting is set, but not used by a given action, it will be ignored. This action performs a forceMerge on the selected indices, merging them to <> per shard, with an optional pause between each merge of <> seconds to allow the cluster to quiesce. [float] Required settings ~~~~~~~~~~~~~~~~~ * <> (required) [float] Optional settings ~~~~~~~~~~~~~~~~~ * <> (has a default value which can optionally be changed) * <> (can override the default) * <> (can override the default <>) * <> (has a default value which can optionally be changed) * <> (has a default value which can optionally be changed) TIP: See an example of this action in an <> <>. [[replicas]] == Replicas [source,yaml] ------------- action: replicas description: >- Set the number of replicas per shard for selected indices to 'count' options: count: wait_for_completion: False timeout_override: continue_if_exception: False disable_action: False filters: - filtertype: ... ------------- NOTE: Empty values and commented lines will result in the default value, if any, being selected. If a setting is set, but not used by a given action, it will be ignored. This action will set the number of replicas per shard to the value of <>. You can optionally set `wait_for_completion` to `True` to have Curator wait for the replication operation to complete before continuing. If this option is chosen, it is advisable to use the <> option in order to avoid client timeouts. [float] Required settings ~~~~~~~~~~~~~~~~~ * <> (required) [float] Optional settings ~~~~~~~~~~~~~~~~~ * <> (has a default value which can optionally be changed) * <> (can override the default) * <> (can override the default <>) * <> (has a default value which can optionally be changed) * <> (has a default value which can optionally be changed) TIP: See an example of this action in an <> <>. [[restore]] == Restore [source,yaml] ------------- actions: 1: action: restore description: >- Restore all indices in the most recent snapshot with state SUCCESS. Wait for the restore to complete before continuing. Do not skip the repository filesystem access check. Use the other options to define the index/shard settings for the restore. options: repository: # Leaving name blank will result in restoring the most recent snapshot by age name: # Leaving indices blank will result in restoring all indices in the snapshot indices: include_aliases: False ignore_unavailable: False include_global_state: True partial: False rename_pattern: rename_replacement: extra_settings: wait_for_completion: True skip_repo_fs_check: False timeout_override: continue_if_exception: False disable_action: False filters: - filtertype: state state: SUCCESS exclude: - filtertype: ... ------------- NOTE: Empty values and commented lines will result in the default value, if any, being selected. If a setting is set, but not used by a given action, it will be ignored. This action will restore indices from the indicated <>, from the most recent snapshot identified by the applied filters, or the snapshot identified by <>. The other options are usually okay to leave at the defaults, but feel free to read about them and change them accordingly. The <> option allows the addition of extra settings, such as index settings and mappings. An example of how these settings can be used to change settings for an index being restored might be: [source,yaml] ------------- extra_settings: settings: number_of_shards: 1 number_of_replicas: 0 mappings: type1 : properties: field1: type: string index: not_analyzed ------------- [float] Required settings ~~~~~~~~~~~~~~~~~ * <> (required) [float] Optional settings ~~~~~~~~~~~~~~~~~ * <> (has no default value) * <> (has a default value which can optionally be changed) * <> (has a default value which can optionally be changed) * <> (has a default value which can optionally be changed) * <> (has a default value which can optionally be changed) * <> (has a default value which can optionally be changed) * <> (has no default value) * <> (has no default value) * <> (has no default value.) * <> (has a default value which can optionally be changed) * <> (has a default value which can optionally be changed) * <> (can override the default) * <> (can override the default <>) * <> (has a default value which can optionally be changed) * <> (has a default value which can optionally be changed) TIP: See an example of this action in an <> <>. [[snapshot]] == Snapshot [source,yaml] ------------- action: snapshot description: >- Snapshot selected indices to 'repository' with the snapshot name or name pattern in 'name'. Use all other options as assigned options: repository: # Leaving name blank will result in the default 'curator-%Y%m%d%H%M%S' name: ignore_unavailable: False include_global_state: True partial: False wait_for_completion: True skip_repo_fs_check: False timeout_override: continue_if_exception: False disable_action: False filters: - filtertype: ... ------------- NOTE: Empty values and commented lines will result in the default value, if any, being selected. If a setting is set, but not used by a given action, it will be ignored. This action will snapshot indices to the indicated <>, with a name, or name pattern, as identified by <>. The other options are usually okay to leave at the defaults, but feel free to read about them and change them accordingly. [float] Required settings ~~~~~~~~~~~~~~~~~ * <> (required) [float] Optional settings ~~~~~~~~~~~~~~~~~ * <> (has a default value which can optionally be changed) * <> (has a default value which can optionally be changed) * <> (has a default value which can optionally be changed) * <> (has a default value which can optionally be changed) * <> (has a default value which can optionally be changed) * <> (has a default value which can optionally be changed) * <> (can override the default) * <> (can override the default <>) * <> (has a default value which can optionally be changed) * <> (has a default value which can optionally be changed) TIP: See an example of this action in an <> <>. curator-4.2.5/docs/asciidoc/command-line.asciidoc000066400000000000000000000205101302700154000217400ustar00rootroot00000000000000[[cli]] = Running Curator [partintro] -- * <> * <> * <> -- [[command-line]] == Command Line Interface The command-line arguments are as follows: [source,sh] ------- curator [--config CONFIG.YML] [--dry-run] ACTION_FILE.YML ------- The square braces indicate optional elements. If `--config` and `CONFIG.YML` are not provided, Curator will look in `~/.curator/curator.yml` for the configuration file. `~` is the home directory of the user executing Curator. In a Unix system, this might be `/home/username/.curator/curator.yml`, while on a Windows system, it might be `C:\Users\username\.curator\curator.yml` If `--dry-run` is included, Curator will simulate the action(s) in ACTION_FILE.YML as closely as possible without actually making any changes. The results will be in the logfile, or STDOUT/command-line if no logfile is specified. `ACTION_FILE.YML` is a YAML <>. Command-line help is never far away: [source,sh] ------- curator --help ------- The help output looks like this: [source,sh] ------- $ curator --help Usage: curator [OPTIONS] ACTION_FILE Curator for Elasticsearch indices. See http://elastic.co/guide/en/elasticsearch/client/curator/current Options: --config PATH Path to configuration file. Default: ~/.curator/curator.yml --dry-run Do not perform any changes. --version Show the version and exit. --help Show this message and exit. ------- Starting in Curator 4.1, you can use <> in your configuration files. TIP: See <>   [[singleton-cli]] == Singleton Command Line Interface Starting in Curator 4.2, a partial restoration of the command-line from the 3.x version is back as the `curator_cli` command. This allows users to run a single, supported action from the command-line, without needing either the client or action YAML configuration file, though it does support using the client configuration file if you want. As an important bonus, the command-line options allow you to override the settings in the `curator.yml` file! [source,sh] --------- $ curator_cli --help Usage: curator_cli [OPTIONS] COMMAND [ARGS]... Options: --config PATH Path to configuration file. Default: ~/.curator/curator.yml --host TEXT Elasticsearch host. --url_prefix TEXT Elasticsearch http url prefix. --port TEXT Elasticsearch port. --use_ssl Connect to Elasticsearch through SSL. --certificate TEXT Path to certificate to use for SSL validation. --client-cert TEXT Path to file containing SSL certificate for client auth. --client-key TEXT Path to file containing SSL key for client auth. --ssl-no-validate Do not validate SSL certificate --http_auth TEXT Use Basic Authentication ex: user:pass --timeout INTEGER Connection timeout in seconds. --master-only Only operate on elected master node. --dry-run Do not perform any changes. --loglevel TEXT Log level --logfile TEXT log file --logformat TEXT Log output format [default|logstash|json]. --version Show the version and exit. --help Show this message and exit. Commands: allocation Shard Routing Allocation close Close indices delete_indices Delete indices delete_snapshots Delete snapshots forcemerge forceMerge index/shard segments open Open indices replicas Change replica count show_indices Show indices show_snapshots Show snapshots snapshot Snapshot indices --------- The option flags for the given commands match those used for the same <>. The only difference is how filtering is handled. TIP: See <> === Command-line filtering Recent improvements in Curator include schema and setting validation. With these improvements, it is possible to validate filters and their many permutations if passed in a way that Curator can easily digest. [source,sh] ----------- --filter_list TEXT JSON string representing an array of filters. ----------- This means that filters need to be passed as a single object, or an array of objects in JSON format. Single: [source,sh] ----------- --filter_list '{"filtertype":"none"}' ----------- Multiple: [source,sh] ----------- --filter_list '[{"filtertype":"age","source":"creation_date","direction":"older","unit":"days","unit_count":13},{"filtertype":"pattern","kind":"prefix","value":"logstash"}]' ----------- This preserves the power of chained filters, making them available on the command line. NOTE: You may need to escape all of the double quotes on some platforms, or shells like PowerShell, for instance. Caveats to this approach: 1. Only one action can be taken at a time. 2. You must still specify a <> filter if you want to act + on all indices or snapshots. 3. Not all actions have singleton analogs. <> and + <> do not have singleton actions. === Show Indices/Snapshots One feature that the singleton command offers that the other cannot is to show which indices and snapshots are in the system. It's a great way to visually test your filters without causing any harm to the system. [source,sh] ----------- $ curator_cli show_indices --help Usage: curator_cli show_indices [OPTIONS] Show indices Options: --verbose Show verbose output. --header Print header if --verbose --epoch Print time as epoch if --verbose --filter_list TEXT JSON string representing an array of filters. [required] --help Show this message and exit. ----------- [source,sh] ----------- $ curator_cli show_snapshots --help Usage: curator_cli show_snapshots [OPTIONS] Show snapshots Options: --repository TEXT Snapshot repository name [required] --filter_list TEXT JSON string representing an array of filters. [required] --help Show this message and exit. ----------- The `show_snapshots` command will only show snapshots matching the provided filters. The `show_indices` command will also do this, but also offers a few extra features. * `--verbose` adds state, total size of primary and all replicas, the document count, the number of primary and replica shards, and the creation date in ISO8601 format. * `--header` adds a header that shows the column names. This only occurs if `--verbose` is also selected. * `--epoch` changes the date format from ISO8601 to epoch time. If `--header` is also selected, the column header title will change to `creation_date` There are no extra columns or `--verbose` output for the `show_snapshots` command. Without `--epoch` [source,sh] ----------- Index State Size Docs Pri Rep Creation Timestamp logstash-2016.10.20 close 0.0B 0 5 1 2016-10-20T00:00:03Z logstash-2016.10.21 open 763.3MB 5860016 5 1 2016-10-21T00:00:03Z logstash-2016.10.22 open 759.1MB 5858450 5 1 2016-10-22T00:00:04Z logstash-2016.10.23 open 757.8MB 5857456 5 1 2016-10-23T00:00:04Z logstash-2016.10.24 open 771.5MB 5859720 5 1 2016-10-24T00:00:00Z logstash-2016.10.25 open 771.0MB 5860112 5 1 2016-10-25T00:00:01Z logstash-2016.10.27 open 658.3MB 4872830 5 1 2016-10-27T00:00:03Z logstash-2016.10.28 open 655.1MB 5237250 5 1 2016-10-28T00:00:00Z ----------- With `--epoch` [source,sh] ----------- Index State Size Docs Pri Rep creation_date logstash-2016.10.20 close 0.0B 0 5 1 1476921603 logstash-2016.10.21 open 763.3MB 5860016 5 1 1477008003 logstash-2016.10.22 open 759.1MB 5858450 5 1 1477094404 logstash-2016.10.23 open 757.8MB 5857456 5 1 1477180804 logstash-2016.10.24 open 771.5MB 5859720 5 1 1477267200 logstash-2016.10.25 open 771.0MB 5860112 5 1 1477353601 logstash-2016.10.27 open 658.3MB 4872830 5 1 1477526403 logstash-2016.10.28 open 655.1MB 5237250 5 1 1477612800 -----------   [[exit-codes]] == Exit Codes Exit codes will indicate success or failure. * `0` — Success * `1` — Failure * `-1` - Exception raised that does not result in a `1` exit code.   curator-4.2.5/docs/asciidoc/configuration.asciidoc000066400000000000000000000345751302700154000222640ustar00rootroot00000000000000[[configuration]] = Configuration [partintro] -- These are the higher-level configuration settings used by the configuration files. <> and <> are documented separately. * <> * <> * <> -- [[envvars]] == Environment Variables WARNING: This functionality is experimental and may be changed or removed + completely in a future release. You can use environment variable references in the both the <> and the <> to set values that need to be configurable at runtime. To do this, use: [source,sh] ------- ${VAR} ------- Where `VAR` is the name of the environment variable. Each variable reference is replaced at startup by the value of the environment variable. The replacement is case-sensitive and occurs while the YAML file is parsed, but before configuration schema validation. References to undefined variables are replaced by `None` unless you specify a default value. To specify a default value, use: [source,sh] ------- ${VAR:default_value} ------- Where `default_value` is the value to use if the environment variable is undefined. [IMPORTANT] .Unsupported use cases ================================================================= When using environment variables, the value must _only_ be the environment variable. Using extra text, such as: [source,sh] ------- logfile: ${LOGPATH}/extra/path/information/file.log ------- is not supported at this time. ================================================================= === Examples Here are some examples of configurations that use environment variables and what each configuration looks like after replacement: [options="header"] |================================== |Config source |Environment setting |Config after replacement |`unit: ${UNIT}` |`export UNIT=days` |`unit: days` |`unit: ${UNIT}` |no setting |`unit:` |`unit: ${UNIT:days}` |no setting |`unit: days` |`unit: ${UNIT:days}` |`export UNIT=hours` |`unit: hours` |================================== [[actionfile]] == Action File NOTE: Starting in Curator 4.1, you can use <> in your configuration files. An action file has the following structure: [source,sh] ----------- --- # Remember, leave a key empty if there is no value. None will be a string, # not a Python "NoneType" # # Also remember that all examples have 'disable_action' set to True. If you # want to use this action as a template, be sure to set this to False after # copying it. actions: 1: action: ACTION1 description: OPTIONAL DESCRIPTION options: option1: value1 ... optionN: valueN continue_if_exception: False disable_action: True filters: - filtertype: *first* filter_element1: value1 ... filter_elementN: valueN - filtertype: *second* filter_element1: value1 ... filter_elementN: valueN 2: action: ACTION2 description: OPTIONAL DESCRIPTION options: option1: value1 ... optionN: valueN continue_if_exception: False disable_action: True filters: - filtertype: *first* filter_element1: value1 ... filter_elementN: valueN - filtertype: *second* filter_element1: value1 ... filter_elementN: valueN 3: action: ACTION3 ... 4: action: ACTION4 ... ----------- It is a YAML configuration file. The root key must be `actions`, after which there can be any number of actions, nested underneath numbers. Actions will be taken in the order they are completed. The high-level elements of each numbered action are: * <> * <> * <> * <> In the case of the <>, there are two additional high-level elements: `add` and `remove`, which are described in the <> documentation. [[description]] === description This is an optional description which can help describe what the action and its filters are supposed to do. [source,yaml] ------------- description: >- I can make the description span multiple lines by putting ">-" at the beginning of the line, as seen above. Subsequent lines must also be indented. options: option1: ... ------------- [[configfile]] == Configuration File NOTE: The default location of the configuration file is `~/.curator/curator.yml`, but another location can be specified using the `--config` flag on the <>. NOTE: Starting in Curator 4.1, you can use <> in your configuration files. The configuration file contains client connection and settings for logging. It looks like this: [source,sh] ----------- --- # Remember, leave a key empty if there is no value. None will be a string, # not a Python "NoneType" client: hosts: - 127.0.0.1 port: 9200 url_prefix: use_ssl: False certificate: client_cert: client_key: ssl_no_validate: False http_auth: timeout: 30 master_only: False logging: loglevel: INFO logfile: logformat: default blacklist: ['elasticsearch', 'urllib3'] ----------- It is a YAML configuration file. The two root keys must be `client` and `logging`. The subkeys of each of these will be described here. [[hosts]] === hosts This can be a single value: [source,sh] ----------- hosts: 127.0.0.1 ----------- Or multiple values in the 3 acceptable YAML ways to render sequences, or arrays: Flow: [source,sh] ----------- hosts: [ "10.0.0.1", "10.0.0.2" ] ----------- Spanning: [source,sh] ----------- hosts: [ "10.0.0.1", "10.0.0.2" ] ----------- Block: [source,sh] ----------- hosts: - 10.0.0.1 - 10.0.0.2 ----------- You can also provide these hosts with optional ports, and bypass the port option: [source,sh] ----------- hosts: - 10.0.0.1:9200 - 10.0.0.2:9201 ----------- WARNING: When adding a port to the end of a host or IP, the YAML Flow and Spanning styles require `host:port` to be single `'` or double `"` quote encapsulated or you will receive an error. The Block style does not have this limitation. [[port]] === port This should be a single value: [source,sh] ----------- port: 9200 ----------- The default is `9200`. This value will only be applied to <> without a port affixed, e.g. `localhost:9202`. [[url_prefix]] === url_prefix This should be a single value or left empty. [source,sh] ----------- url_prefix: ----------- In some cases you may be obliged to connect to your Elasticsearch cluster through a proxy of some kind. There may be a URL prefix before the API URI items, e.g. http://example.com/elasticsearch/ as opposed to http://localhost:9200. In such a case, the set the `url_prefix` to the appropriate value, 'elasticsearch' in this example. The default is an empty string. [[use_ssl]] === use_ssl This should be `True`, `False` or left empty. [source,sh] ----------- use_ssl: ----------- If access to your Elasticsearch instance is protected by SSL encryption, you must use set `use_ssl` to `True`. The default is `False` [[certificate]] === certificate This should be a file path to your CA certificate, or left empty. [source,sh] ----------- certificate: ----------- This setting allows the use of a specified CA certificate file to validate the SSL certificate used by Elasticsearch. There is no default. [[client_cert]] === client_cert This should be a file path to a client certificate (public key), or left empty. [source,sh] ----------- client_cert: ----------- Allows the use of a specified SSL client cert file to authenticate to Elasticsearch. The file may contain both an SSL client certificate and an SSL key, in which case <> is not used. If specifying `client_cert`, and the file specified does not also contain the key, use <> to specify the file containing the SSL key. The file must be in PEM format, and the key part, if used, must be an unencrypted key in PEM format as well. [[client_key]] === client_key This should be a file path to a client key (private key), or left empty. [source,sh] ----------- client_key: ----------- Allows the use of a specified SSL client key file to authenticate to Elasticsearch. If using <> and the file specified does not also contain the key, use `client_key` to specify the file containing the SSL key. The key file must be an unencrypted key in PEM format. [[aws_key]] === aws_key WARNING: This feature allows connection to AWS using IAM credentials, but <>. WARNING: This setting will not work unless the `requests-aws4auth` Python module has been manually installed first. This should be an AWS IAM access key, or left empty. [source,sh] ----------- aws_key: ----------- IMPORTANT: You must set your <> to the proper hostname _with_ port. It may not work setting <> and <> to only a host name due to the different connection module used. [[aws_secret_key]] === aws_secret_key WARNING: This feature allows connection to AWS using IAM credentials, but <>. WARNING: This setting will not work unless the `requests-aws4auth` Python module has been manually installed first. This should be an AWS IAM secret access key, or left empty. [source,sh] ----------- aws_secret_key: ----------- IMPORTANT: You must set your <> to the proper hostname _with_ port. It may not work setting <> and <> to only a host name due to the different connection module used. [[aws_region]] === aws_region WARNING: This feature allows connection to AWS using IAM credentials, but <>. WARNING: This setting will not work unless the `requests-aws4auth` Python module has been manually installed first. This should be an AWS region, or left empty. [source,sh] ----------- aws_region: ----------- IMPORTANT: You must set your <> to the proper hostname _with_ port. It may not work setting <> and <> to only a host name due to the different connection module used. [[ssl_no_validate]] === ssl_no_validate This should be `True`, `False` or left empty. [source,sh] ----------- ssl_no_validate: ----------- If access to your Elasticsearch instance is protected by SSL encryption, you may set `ssl_no_validate` to `True` to disable SSL certificate verification. Valid use cases for doing so include the use of self-signed certificates that cannot be otherwise verified and would generate error messages. WARNING: Setting `ssl_no_validate` to `True` will likely result in a warning message that your SSL certificates are not trusted. This is expected behavior. The default value is `False`. [[http_auth]] === http_auth This should be a authentication credentials (e.g. `user:pass`), or left empty. [source,sh] ----------- http_auth: ----------- This setting allows basic HTTP authentication to an Elasticsearch instance. The default is empty. [[timeout]] === timeout This should be an integer number of seconds, or left empty. [source,sh] ----------- timeout: ----------- You can change the default client connection timeout value with this setting. The default value is `30` (seconds) should typically not be changed to be very large. If a longer timeout is necessary for a given action, such as <>, <>, or <>, the client timeout can be overridden on per action basis by setting <> in the action <>. There are default override values for some of those longer running actions. [[master_only]] === master_only This should be `True`, `False` or left empty. [source,sh] ----------- master_only: ----------- In some situations, primarily with automated deployments, it makes sense to install Curator on every node. But you wouldn’t want it to run on each node. By setting `master_only` to `True`, this is possible. It tests for, and will only continue running on the node that is the elected master. WARNING: If `master_only` is `True`, and <> has more than one value, Curator will raise an Exception. This setting should _only_ be used with a single host in <>, as its utility centers around deploying to all nodes in the cluster. The default value is `False`. [[loglevel]] === loglevel This should be `CRITICAL`, `ERROR`, `WARNING`, `INFO`, `DEBUG`, or left empty. [source,sh] ----------- loglevel: ----------- Set the minimum acceptable log severity to display. * `CRITICAL` will only display critical messages. * `ERROR` will only display error and critical messages. * `WARN` will display error, warning, and critical messages. * `INFO` will display informational, error, warning, and critical messages. * `DEBUG` will display debug messages, in addition to all of the above. The default value is `INFO`. [[logfile]] === logfile This should be a path to a log file, or left empty. [source,sh] ----------- logfile: ----------- The default value is empty, which will result in logging to `STDOUT`, or the console. [[logformat]] === logformat This should `default`, `json`, `logstash`, or left empty. [source,sh] ----------- logformat: ----------- The `default` format looks like: [source,sh] ----------- 2016-04-22 11:53:09,972 INFO Action #1: ACTIONNAME ----------- The `json` or `logstash` formats look like: [source,sh] ----------- {"@timestamp": "2016-04-22T11:54:29.033Z", "function": "cli", "linenum": 178, "loglevel": "INFO", "message": "Action #1: ACTIONNAME", "name": "curator.cli"} ----------- The default value is `default`. [[blacklist]] === blacklist This should be an empty array `[]`, an array of log handler strings, or left empty. [source,sh] ----------- blacklist: ['elasticsearch', 'urllib3'] ----------- The default value is `['elasticsearch', 'urllib3']`, which will result in logs for the `elasticsearch` and `urllib3` Python modules _not_ being output. These can be quite verbose, so unless you need them to debug an issue, you should accept the default value. TIP: If you do need to troubleshoot an issue, set `blacklist` to `[]`, which is an empty array. Leaving it unset will result in the default behavior, which is to filter out `elasticsearch` and `urllib3` log traffic. curator-4.2.5/docs/asciidoc/examples.asciidoc000066400000000000000000000321161302700154000212200ustar00rootroot00000000000000[[examples]] = Examples [partintro] -- These examples should help illustrate how to build your own <>. Starting in Curator 4.1, you can use <> in your configuration files. * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> -- [[ex_alias]] == alias [source,yaml] ------------- --- # Remember, leave a key empty if there is no value. None will be a string, # not a Python "NoneType" # # Also remember that all examples have 'disable_action' set to True. If you # want to use this action as a template, be sure to set this to False after # copying it. actions: 1: action: alias description: >- Alias indices older than 7 days but newer than 14 days, with a prefix of logstash- to 'last_week', remove indices older than 14 days. options: name: last_week extra_settings: timeout_override: continue_if_exception: False disable_action: True add: filters: - filtertype: pattern kind: prefix value: logstash- exclude: - filtertype: age source: name direction: older timestring: '%Y.%m.%d' unit: days unit_count: 7 exclude: - filtertype: age source: name direction: younger timestring: '%Y.%m.%d' unit: days unit_count: 14 exclude: remove: filters: - filtertype: pattern kind: prefix value: logstash- - filtertype: age source: name direction: older timestring: '%Y.%m.%d' unit: days unit_count: 14 exclude: ------------- [[ex_allocation]] == allocation [source,yaml] ------------- --- # Remember, leave a key empty if there is no value. None will be a string, # not a Python "NoneType" # # Also remember that all examples have 'disable_action' set to True. If you # want to use this action as a template, be sure to set this to False after # copying it. actions: 1: action: allocation description: >- Apply shard allocation routing to 'require' 'tag=cold' for hot/cold node setup for logstash- indices older than 3 days, based on index_creation date options: key: tag value: cold allocation_type: require wait_for_completion: False timeout_override: continue_if_exception: False disable_action: True filters: - filtertype: pattern kind: prefix value: logstash- exclude: - filtertype: age source: creation_date direction: older unit: days unit_count: 3 exclude: ------------- [[ex_close]] == close [source,yaml] ------------- --- # Remember, leave a key empty if there is no value. None will be a string, # not a Python "NoneType" # # Also remember that all examples have 'disable_action' set to True. If you # want to use this action as a template, be sure to set this to False after # copying it. actions: 1: action: close description: >- Close indices older than 30 days (based on index name), for logstash- prefixed indices. options: delete_aliases: False timeout_override: continue_if_exception: False disable_action: True filters: - filtertype: pattern kind: prefix value: logstash- exclude: - filtertype: age source: name direction: older timestring: '%Y.%m.%d' unit: days unit_count: 30 exclude: ------------- [[ex_cluster_routing]] == cluster_routing [source,yaml] ------------- --- # Remember, leave a key empty if there is no value. None will be a string, # not a Python "NoneType" # # Also remember that all examples have 'disable_action' set to True. If you # want to use this action as a template, be sure to set this to False after # copying it. # # This action example has a blank spot at action ID 2. This is to show that # Curator can disable allocation before one or more actions, and then re-enable # it afterward. actions: 1: action: cluster_routing description: >- Disable shard routing for the entire cluster. options: routing_type: allocation value: none setting: enable wait_for_completion: False timeout_override: continue_if_exception: False disable_action: True 2: action: (any other action details go here) ... 3: action: cluster_routing description: >- Re-enable shard routing for the entire cluster. options: routing_type: allocation value: all setting: enable wait_for_completion: False timeout_override: continue_if_exception: False disable_action: True ------------- [[ex_create_index]] == create_index [source,yaml] ------------- --- # Remember, leave a key empty if there is no value. None will be a string, # not a Python "NoneType" # # Also remember that all examples have 'disable_action' set to True. If you # want to use this action as a template, be sure to set this to False after # copying it. actions: 1: action: create_index description: Create the index as named, with the specified extra settings. options: name: myindex extra_settings: settings: number_of_shards: 2 number_of_replicas: 1 timeout_override: continue_if_exception: False disable_action: True ------------- [[ex_delete_indices]] == delete_indices [source,yaml] ------------- --- # Remember, leave a key empty if there is no value. None will be a string, # not a Python "NoneType" # # Also remember that all examples have 'disable_action' set to True. If you # want to use this action as a template, be sure to set this to False after # copying it. actions: 1: action: delete_indices description: >- Delete indices older than 45 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly. options: ignore_empty_list: True timeout_override: continue_if_exception: False disable_action: True filters: - filtertype: pattern kind: prefix value: logstash- exclude: - filtertype: age source: name direction: older timestring: '%Y.%m.%d' unit: days unit_count: 45 exclude: ------------- [[ex_delete_snapshots]] == delete_snapshots [source,yaml] ------------- --- # Remember, leave a key empty if there is no value. None will be a string, # not a Python "NoneType" # # Also remember that all examples have 'disable_action' set to True. If you # want to use this action as a template, be sure to set this to False after # copying it. actions: 1: action: delete_snapshots description: >- Delete snapshots from the selected repository older than 45 days (based on creation_date), for 'curator-' prefixed snapshots. options: repository: timeout_override: continue_if_exception: False disable_action: True filters: - filtertype: pattern kind: prefix value: curator- exclude: - filtertype: age source: creation_date direction: older unit: days unit_count: 45 exclude: ------------- [[ex_forcemerge]] == forcemerge [source,yaml] ------------- --- # Remember, leave a key empty if there is no value. None will be a string, # not a Python "NoneType" # # Also remember that all examples have 'disable_action' set to True. If you # want to use this action as a template, be sure to set this to False after # copying it. actions: 1: action: forcemerge description: >- forceMerge logstash- prefixed indices older than 2 days (based on index creation_date) to 2 segments per shard. Delay 120 seconds between each forceMerge operation to allow the cluster to quiesce. This action will ignore indices already forceMerged to the same or fewer number of segments per shard, so the 'forcemerged' filter is unneeded. options: max_num_segments: 2 delay: 120 timeout_override: continue_if_exception: False disable_action: True filters: - filtertype: pattern kind: prefix value: logstash- exclude: - filtertype: age source: creation_date direction: older unit: days unit_count: 2 exclude: ------------- [[ex_open]] == open [source,yaml] ------------- --- # Remember, leave a key empty if there is no value. None will be a string, # not a Python "NoneType" # # Also remember that all examples have 'disable_action' set to True. If you # want to use this action as a template, be sure to set this to False after # copying it. actions: 1: action: open description: >- Open indices older than 30 days but younger than 60 days (based on index name), for logstash- prefixed indices. options: timeout_override: continue_if_exception: False disable_action: True filters: - filtertype: pattern kind: prefix value: logstash- exclude: - filtertype: age source: name direction: older timestring: '%Y.%m.%d' unit: days unit_count: 30 exclude: - filtertype: age source: name direction: younger timestring: '%Y.%m.%d' unit: days unit_count: 60 exclude: ------------- [[ex_replicas]] == replicas [source,yaml] ------------- --- # Remember, leave a key empty if there is no value. None will be a string, # not a Python "NoneType" # # Also remember that all examples have 'disable_action' set to True. If you # want to use this action as a template, be sure to set this to False after # copying it. actions: 1: action: replicas description: >- Reduce the replica count to 0 for logstash- prefixed indices older than 10 days (based on index creation_date) options: count: 0 wait_for_completion: False timeout_override: continue_if_exception: False disable_action: True filters: - filtertype: pattern kind: prefix value: logstash- exclude: - filtertype: age source: creation_date direction: older unit: days unit_count: 10 exclude: ------------- [[ex_restore]] == restore [source,yaml] ------------- --- # Remember, leave a key empty if there is no value. None will be a string, # not a Python "NoneType" # # Also remember that all examples have 'disable_action' set to True. If you # want to use this action as a template, be sure to set this to False after # copying it. actions: 1: action: restore description: >- Restore all indices in the most recent curator-* snapshot with state SUCCESS. Wait for the restore to complete before continuing. Do not skip the repository filesystem access check. Use the other options to define the index/shard settings for the restore. options: repository: # Leaving name blank will result in restoring the most recent snapshot by age name: # Leaving indices blank will result in restoring all indices in the snapshot indices: include_aliases: False ignore_unavailable: False include_global_state: True partial: False rename_pattern: rename_replacement: extra_settings: wait_for_completion: True skip_repo_fs_check: False timeout_override: continue_if_exception: False disable_action: True filters: - filtertype: pattern kind: prefix value: curator- exclude: - filtertype: state state: SUCCESS exclude: ------------- [[ex_snapshot]] == snapshot [source,yaml] ------------- --- # Remember, leave a key empty if there is no value. None will be a string, # not a Python "NoneType" # # Also remember that all examples have 'disable_action' set to True. If you # want to use this action as a template, be sure to set this to False after # copying it. actions: 1: action: snapshot description: >- Snapshot logstash- prefixed indices older than 1 day (based on index creation_date) with the default snapshot name pattern of 'curator-%Y%m%d%H%M%S'. Wait for the snapshot to complete. Do not skip the repository filesystem access check. Use the other options to create the snapshot. options: repository: # Leaving name blank will result in the default 'curator-%Y%m%d%H%M%S' name: ignore_unavailable: False include_global_state: True partial: False wait_for_completion: True skip_repo_fs_check: False timeout_override: continue_if_exception: False disable_action: True filters: - filtertype: pattern kind: prefix value: logstash- exclude: - filtertype: age source: creation_date direction: older unit: days unit_count: 1 exclude: ------------- curator-4.2.5/docs/asciidoc/faq.asciidoc000066400000000000000000000320531302700154000201510ustar00rootroot00000000000000[[faq]] = Frequently Asked Questions [partintro] -- This section will be updated as more frequently asked questions arise * <> * <> * <> * <> * <> * <> * <> * <> -- [[faq_doc_error]] == Q: How can I report an error in the documentation? [float] A: Use the "Edit" link on any page ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ See <>. [[faq_partial_delete]] == Q: Can I delete only certain data from within indices? [float] A: It's complicated ~~~~~~~~~~~~~~~~~~~ [float] TL;DR: No. Curator can only delete entire indices. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [float] Full answer: ^^^^^^^^^^^^ As a thought exercise, think of Elasticsearch indices as being like databases, or tablespaces within a database. If you had hundreds of millions of rows to delete from your database, would you run a separate `DELETE from TABLE where date> with <> set to `regex`, and <> set to the needed regular expression. [float] The Problem: ^^^^^^^^^^^^ Illegal characters make it hard to delete indices. ------------------ % curl logs.example.com:9200/_cat/indices red }?ebc-2015.04.08.03 sip-request{ 5 1 0 0 632b 316b red }?ebc-2015.04.08.03 sip-response 5 1 0 0 474b 237b red ?ebc-2015.04.08.02 sip-request{ 5 1 0 0 474b 316b red eb 5 1 0 0 632b 316b red ?e 5 1 0 0 632b 316b ------------------   You can see it looks like there are some tab characters and maybe newline characters. This makes it hard to use the HTTP API to delete the indices. Dumping all the index settings out: [source,sh] ------- curl -XGET localhost:9200/*/_settings?pretty -------   ...reveals the index names as the first key in the resulting JSON. In this case, the names were very atypical: ------- }\b?\u0011ebc-2015.04.08.02\u000Bsip-request{ }\u0006?\u0011ebc-2015.04.08.03\u000Bsip-request{ }\u0003?\u0011ebc-2015.04.08.03\fsip-response ... -------   Curator lets you use regular expressions to select indices to perform actions on. WARNING: Before attempting an action, see what will be affected by using the `--dry-run` flag first. To delete the first three from the above example, use `'.*sip.*'` as your regular expression. NOTE: In an <>, regular expressions and strftime date strings _must_ be encapsulated in single-quotes. The next one is trickier. The real name of the index was `\n\u0011eb`. The regular expression `.*b$` did not work, but `'\n.*'` did. The last index can be deleted with a regular expression of `'.*e$'`. The resulting <> might look like this: [source,yaml] -------- actions: 1: description: Delete indices with strange characters that match regex '.*sip.*' action: delete_indices options: continue_if_exception: False disable_action: False filters: - filtertype: pattern kind: regex value: '.*sip.*' 2: description: Delete indices with strange characters that match regex '\n.*' action: delete_indices options: continue_if_exception: False disable_action: False filters: - filtertype: pattern kind: regex value: '\n.*' 3: description: Delete indices with strange characters that match regex '.*e$' action: delete_indices options: continue_if_exception: False disable_action: False filters: - filtertype: pattern kind: regex value: '.*e$' --------   ''''' [[entrypoint-fix]] == Q: I'm getting `DistributionNotFound` and `entry_point` errors when I try to run Curator. What am I doing wrong? [float] A: You likely need to upgrade `setuptools` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If you are still unable to install, or get strange errors about dependencies you know you've installed, or messages mentioning `entry_point`, you may need to upgrade the `setuptools` package. This is especially common with RHEL and CentOS installs, and their variants, as they depend on Python 2.6. If you can run `pip install -U setuptools`, it should correct the problem. You may also be able to download and install manually: . `wget https://pypi.python.org/packages/source/s/setuptools/setuptools-15.1.tar.gz -O setuptools-15.1.tar.gz` . `pip install setuptools-15.1.tar.gz` Any dependencies this version of setuptools may require will have to be manually acquired and installed for your platform. For more information about setuptools, see https://pypi.python.org/pypi/setuptools This fix originally appeared https://github.com/elastic/curator/issues/56#issuecomment-77843587[here]. ''''' [[faq_lb_timeout]] == Q: Snapshots seem to be working. Why am I getting `SnapshotMissingException` messages? [float] A: A load balancer, or other network fixture could be timing out your connection. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ A gateway, proxy, or load balancer timeout will usually result in http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html[504 HTTP responses]: 504 Gateway Timeout:: The server, while acting as a gateway or proxy, did not receive a timely response from the upstream server specified by the URI (e.g. HTTP, FTP, LDAP) or some other auxiliary server (e.g. DNS) it needed to access in attempting to complete the request. An easy way to check is to change the <> to DEBUG and look for 504 errors and long requests, like these: [source,txt] ------- 2015-04-02 13:18:37,702 DEBUG urllib3.connectionpool _make_request:368 "PUT /_snapshot/my_backup_s3_repository/curator-20150402121735?wait_for_completion=true HTTP/1.1" 504 0 2015-04-02 13:18:37,703 WARNING elasticsearch log_request_fail:81 PUT /_snapshot/my_backup_s3_repository/curator-20150402121735?wait_for_completion=true [status:504 request:59.777s] ------- Note the `status:504` and the `request:59.777s` at the right end of the line. In the case of these error messages, it turned out to be an AWS load balancer that had a 60 second timeout for connections. The fix was to increase the timeout. Another potential resolution would be to have Curator connect to a client directly, rather than through a load balancer or proxy. IMPORTANT: You will need to set <> in your <> to catch these 504 errors, as they are filtered by default. ''''' [[faq_old_indices]] == Q: Why doesn't Curator 4 work with older indices? [float] A: Because Curator 4 requires that all indices have a `creation_date`. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The actual error message might look something like this: [source,txt] ------- 2016-08-30 13:18:37,702 WARNING Index: indexname has no "creation_date"! This implies that the index predates Elasticsearch v1.4. For safety, this index will be removed from the actionable list. ------- Older indices (made before Elasticsearch 1.4) do not have this metadata. Trying to support this in Curator 4 would result in having to catch edge cases for some indices, but not for others. In order to keep the code clean, and with fewer exceptions, a design decision was made to not work with indices without a `creation_date`. Curator will not change them in any way, ever. They are automatically removed from the list of actionable indices. You can use the Sense plugin for Kibana to act on these indices manually, or your preference of API method. ''''' [[faq_aws_iam]] == Q: Why doesn't Curator 4 work with AWS Elasticsearch? [float] A: Because Curator 4 requires access to the `/_cluster/state/metadata` endpoint. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ There is some confusion because Curator 3 supported AWS ES, but Curator 4 does not. There are even some IAM credentials listed as options for client connections. These are currently available, but not able to be used. This may change at some point, so they remain at the ready until then. Curator 4 requires access to the `/_cluster/state/metadata` endpoint in order to pull metadata at IndexList initialization time for _all_ indices. This metadata is used to determine index routing information, index sizing, index state (either `open` or `close`), aliases, and more. Curator 4 switched to doing this in order to reduce the number of repetitive client calls that were made in the previous versions. After Curator 4 was released, AWS announced it's 2.3 version of Elasticsearch. Unfortunately, this version did not support the `/_cluster/state/metadata` endpoint, which means that Curator cannot be used to manage indices in AWS. https://forums.aws.amazon.com/thread.jspa?threadID=236643&tstart=0[This thread] (requires AWS login credentials) tracks the issue on the Amazon side. ''''' [[faq_unicode]] == Q: Why am I getting an error message about ASCII encoding? If you see messages like this: [source,sh] ----------- Click will abort further execution because Python 3 was configured to use ASCII as encoding for the environment. Either run this under Python 2 or consult http://click.pocoo.org/python3/ for mitigation steps. This system lists a couple of UTF-8 supporting locales that you can pick from. The following suitable locales where discovered: aa_DJ.utf8, aa_ER.utf8, aa_ET.utf8, ... ----------- You are likely running Curator with Python 3, or the RPM/DEB package, which was compiled with Python 3. Using the command-line library http://click.pocoo.org[click] with Python 3 requires your locale to be Unicode. You can set this up by exporting the `LC_ALL` environment variable like this: [source,sh] ----------- $ export LC_ALL=mylocale.utf8 ----------- Where `mylocale.utf8` is one of the listed "suitable locales." You can also set the locale on the command-line before the Curator command: [source,sh] ----------- $ LC_ALL=mylocale.utf8 curator [ARGS] ... ----------- IMPORTANT: If you use `export`, be sure to choose the correct locale as it will be set for the duration of your terminal session. ''''' curator-4.2.5/docs/asciidoc/filter_elements.asciidoc000066400000000000000000000345351302700154000225720ustar00rootroot00000000000000 [[filter_elements]] = Filter Elements [partintro] -- * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> Starting in Curator 4.1, you can use <> in your configuration files. -- [[fe_aliases]] == aliases NOTE: This setting is used only when using the <> filter. The value of this setting must be a single alias name, or a list of alias names. This can be done in any of the ways YAML allows for lists or arrays. Here are a few examples. **Single** [source,txt] ------- filters: - filtertype: alias aliases: my_alias exclude: False ------- **List** - Flow style: + [source,txt] ------- filters: - filtertype: alias aliases: [ my_alias, another_alias ] exclude: False ------- - Block style: + [source,txt] ------- filters: - filtertype: alias aliases: - my_alias - another_alias exclude: False ------- There is no default value. This setting must be set by the user or an exception will be raised, and execution will halt. [[fe_allocation_type]] == allocation_type NOTE: This setting is used only when using the <> filter. The value of this setting must be one of `require`, `include`, or `exclude`. Read more about these settings at {ref}/shard-allocation-filtering.html The default value for this setting is `require`. [[fe_count]] == count NOTE: This setting is only used with the <> filtertype + and is a required setting. The value for this setting is a number of indices or snapshots to match. Items will remain in the actionable list depending on the value of <>, and <>. There is no default value. This setting must be set by the user or an exception will be raised, and execution will halt. [[fe_direction]] == direction NOTE: This setting is only used with the <> filtertype. This setting must be either `older` or `younger`. This setting is used to determine whether indices or snapshots are `older` or `younger` than the reference point in time determined by <>, <>, and optionally, <>. If `direction` is `older`, then indices (or snapshots) which are _older_ than the reference point in time will be matched. Likewise, if `direction` is `younger`, then indices (or snapshots) which are _younger_ than the reference point in time will be matched. There is no default value. This setting must be set by the user or an exception will be raised, and execution will halt. [[fe_disk_space]] == disk_space NOTE: This setting is only used with the <> filtertype + and is a required setting. The value for this setting is a number of gigabytes. Indices in excess of this number of gigabytes will be matched. There is no default value. This setting must be set by the user or an exception will be raised, and execution will halt. [[fe_epoch]] == epoch NOTE: This setting is only used with the <> filtertype. NOTE: This setting is optional. The value for this setting must be an epoch timestamp. <>, <>, and optionally, <>, are used by Curator to establish the moment in time point of reference with this formula: [source,sh] ----------- point_of_reference = epoch - ((number of seconds in unit) * unit_count) ----------- If <> is unset, the current time is used. It is possible to set a point of reference in the future by using a negative value for <>. [[fe_exclude]] == exclude NOTE: This setting is available in _all_ filter types. If `exclude` is `True`, the filter will remove matches from the actionable list. If `exclude` is `False`, then only matches will be kept in the actionable list. The default value for this setting is different for each filter type. [[fe_field]] == field NOTE: This setting is only used with the <> filtertype. The value of this setting must be a timestamp field name. This field must be present in the indices being filtered or an exception will be raised, and execution will halt. This setting is only used when <> is `field_stats`. The default value for this setting is `@timestamp`. [[fe_key]] == key NOTE: This setting is required when using the <>. The value of this setting should correspond to a node setting on one or more nodes in your cluster. For example, you might have set [source,sh] ----------- node.tag: myvalue ----------- in your `elasticsearch.yml` file for one or more of your nodes. To match allocation in this case, set key to `tag`. These special attributes are also supported: [cols="2*", options="header"] |=== |attribute |description |`_name` |Match nodes by node name |`_host_ip` |Match nodes by host IP address (IP associated with hostname) |`_publish_ip` |Match nodes by publish IP address |`_ip` |Match either `_host_ip` or `_publish_ip` |`_host` |Match nodes by hostname |=== There is no default value. This setting must be set by the user or an exception will be raised, and execution will halt. [[fe_kind]] == kind NOTE: This setting is only used with the <> + filtertype and is a required setting. This setting tells the <> what pattern type to match. Acceptable values for this setting are `prefix`, `suffix`, `timestring`, and `regex`. include::inc_filter_chaining.asciidoc[] There is no default value. This setting must be set by the user or an exception will be raised, and execution will halt. [[fe_max_num_segments]] == max_num_segments NOTE: This setting is only used with the <> filtertype. The value for this setting is the cutoff number of segments per shard. Indices which have this number of segments per shard, or fewer, will be actionable depending on the value of <>, which is `True` by default for the <> filter type. There is no default value. This setting must be set by the user or an exception will be raised, and execution will halt. [[fe_reverse]] == reverse NOTE: This setting is only used with the <> filtertype This setting affects the sort order of the indices. `True` means reverse-alphabetical. This means that if all index names share the same pattern with a date--e.g. index-2016.03.01--older indices will be selected first. The default value of this setting is `True`. This setting is ignored if <> is `True`. [[fe_source]] == source The _source_ from which to derive the index or snapshot age. Can be one of `name`, `creation_date`, or `field_stats`. NOTE: This setting is only used with the <> filtertype, or + with the <> filtertype when <> is set to `True`. NOTE: When using the <> filtertype, source requires + <>, <>, <>, + and additionally, the optional setting, <>. [float] name ~~~~ Setting source to `name` tells Curator to derive an epoch timestamp from an index or snapshot name based on a <>. This was the only available behavior in previous versions of Curator. This source requires <> to be set, in addition to <>, <>, and <>. [float] creation_date ~~~~~~~~~~~~~ Setting source to `creation_date` tells Curator to reference the index or snapshot creation time, as stored in the cluster metadata. This source requires <>, <>, and <>. [float] field_stats ~~~~~~~~~~~ Setting source to `field_stats` tells Curator to use a newer Elasticsearch API call which returns the minimum and maximum value of a field in an index. Curator will only use timestamp fields, which should be mapped in Elasticsearch as type `date`. The <> and <> settings are required to use the `field_stats` source. NOTE: Source `field_stats` only works with indices. [[fe_state]] == state NOTE: This setting is only used with the <> filtertype. The value for this setting must be one of `SUCCESS`, `PARTIAL`, `FAILED`, or `IN_PROGRESS`. This setting determines what kind of snapshots will be passed. The default value for this setting is `SUCCESS`. [[fe_stats_result]] == stats_result NOTE: This setting is only used with the <> filtertype. The value for this setting can be either `min_value` or `max_value`. This setting is only used when <> is `field_stats`, and determines whether Curator will use the minimum or maximum value of <> for time calculations. The default value for this setting is `min_value`. [[fe_timestring]] == timestring NOTE: This setting is only used with the <> filtertype, or + with the <> filtertype if <> is set to `True`. This setting must be a valid Python strftime string. It is used to match and extract the timestamp in an index or snapshot name. The identifiers that Curator currently recognizes include: * `Y`: A 4 digit year * `y`: A 2 digit year * `m`: The 2 digit month * `W`: The 2 digit week of the year * `d`: The 2 digit day of the month * `H`: The 2 digit hour of the day, in 24 hour notation * `M`: The 2 digit minute of the hour * `S`: The 2 digit number of second of the minute * `j`: The 3 digit day of the year These identifiers may be combined with each other, and/or separated from each other with hyphens `-`, periods `.`, underscores `_`, or other characters valid in an index name. Each identifier must be preceded by a `%` character in the timestring. For example, an index like `index-2016.04.01` would use a timestring of `'%Y.%m.%d'`. When <> is `name`, this setting must be set by the user or an exception will be raised, and execution will halt. There is no default value. [[fe_unit]] == unit NOTE: This setting is only used with the <> filtertype, or + with the <> filtertype if <> is set to `True`. This setting must be one of `seconds`, `minutes`, `hours`, `days`, `weeks`, `months`, or `years`. <>, <>, and optionally, <>, are used by Curator to establish the moment in time point of reference with this formula: [source,sh] ----------- point_of_reference = epoch - ((number of seconds in unit) * unit_count) ----------- The following table helps to illustrate this: .Time unit reference [width="50%", cols="> is unset, the current time is used. It is possible to set a point of reference in the future by using a negative value for <>. This setting must be set by the user or an exception will be raised, and execution will halt. [[fe_unit_count]] == unit_count NOTE: This setting is only used with the <> filtertype, or + with the <> filtertype if <> is set to `True`. The value of this setting will be used as a multiplier for <>. <>, <>, and optionally, <>, are used by Curator to establish the moment in time point of reference with this formula: [source,sh] ----------- point_of_reference = epoch - ((number of seconds in unit) * unit_count) ----------- If <> is unset, the current time is used. It is possible to set a point of reference in the future by using a negative value for <>. This setting must be set by the user or an exception will be raised, and execution will halt. [[fe_use_age]] == use_age This setting allows matching of indices by their age _and_ the space they consume. In other words, it sorts all indices in the list by age, then starts adding the space consumed by each index, beginning from the youngest. Once the value of <> is reached, all remaining indices, which are the oldest, will remain in the actionable list, omitting all of the younger indices. The default value of this setting is `False` NOTE: Use of this setting requires the additional setting, <>. [[fe_value]] == value NOTE: This setting is only used with the <> + filtertype and is a required setting. There is a separate <> associated with the <>, and the <>. The value of this setting is used by <> as follows: * `prefix`: Search the first part of an index name for the provided value * `suffix`: Search the last part of an index name for the provided value * `regex`: Provide your own regular expression, and Curator will find the matches. * `timestring`: An strftime string to extrapolate and find indices that match. For example, given a `timestring` of `'%Y.%m.%d'`, matching indices would include `logstash-2016.04.01` and `.marvel-2016.04.01`, but not `myindex-2016-04-01`, as the pattern is different. IMPORTANT: Whatever you provide for `value` is always going to be a part of a + regular expression. The safest practice is to always encapsulate within single quotes. For example: `value: '-suffix'`, or `value: 'prefix-'` There is no default value. This setting must be set by the user or an exception will be raised, and execution will halt. curator-4.2.5/docs/asciidoc/filters.asciidoc000066400000000000000000000331671302700154000210610ustar00rootroot00000000000000[[filters]] = Filters [partintro] -- Filters are the way to select only the indices (or snapshots) you want. include::inc_filter_chaining.asciidoc[] The index filtertypes are: * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> The snapshot filtertypes are: * <> * <> * <> * <> * <> Starting in Curator 4.1, you can use <> in your configuration files. -- [[filtertype]] == filtertype Each filter is defined first by a `filtertype`. Each filtertype has its own settings, or no settings at all. In a configuration file, filters are defined as follows: [source,yaml] ------------- - filtertype: *first* setting1: ... ... settingN: ... - filtertype: *second* setting1: ... ... settingN: ... - filtertype: *third* ------------- The `-` indicates in the YAML that this is an array element. Each filtertype declaration must be preceded by a `-` for the filters to be read properly. This is how Curator can chain filters together. Anywhere filters can be used, multiple can be chained together in this manner. The index filtertypes are: * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> The snapshot filtertypes are: * <> * <> * <> * <> * <> [[filtertype_age]] == age [source,yaml] ------------- - filtertype: age source: name direction: older timestring: '%Y.%m.%d' unit: days unit_count: 3 field: stats_result: epoch: exclude: False ------------- NOTE: Empty values and commented lines will result in the default value, if any, being selected. If a setting is set, but not used by a given <>, it may generate an error. This <> will iterate over the actionable list and match indices based on their age. They will remain in, or be removed from the actionable list based on the value of <>. Read more about the different settings for this <>: [float] Required settings ~~~~~~~~~~~~~~~~~ * <> (required) * <> (required) * <> (required) * <> (required) [float] Dependent settings ~~~~~~~~~~~~~~~~~~ * <> (required if `source` is `name`) * <> (required if `source` is `field_stats`) [Indices only] * <> (only used if `source` is `field_stats`) [Indices only] [float] Optional settings ~~~~~~~~~~~~~~~~~ * <> (optional) * <> (default is `False`) [[filtertype_alias]] == alias [source,yaml] ------------- - filtertype: alias aliases: exclude: True ------------- NOTE: Empty values and commented lines will result in the default value, if any, being selected. If a setting is set, but not used by a given <>, it may generate an error. This <> will iterate over the actionable list and match indices based on whether they are associated with the given <>. They will remain in, or be removed from the actionable list based on the value of <>. [float] Required settings ~~~~~~~~~~~~~~~~~ * <> (required) [float] Optional settings ~~~~~~~~~~~~~~~~~ * <> (default is `False`) [[filtertype_allocated]] == allocated [source,yaml] ------------- - filtertype: allocated key: value: allocation_type: exclude: True ------------- NOTE: Empty values and commented lines will result in the default value, if any, being selected. If a setting is set, but not used by a given <>, it may generate an error. This <> will iterate over the actionable list and match indices based on their shard routing allocation settings. They will remain in, or be removed from the actionable list based on the value of <>. [float] Required settings ~~~~~~~~~~~~~~~~~ * <> (required) * <> (required) [float] Optional settings ~~~~~~~~~~~~~~~~~ * <> (has a default value which can optionally be changed) * <> (default is `True`) [[filtertype_closed]] == closed [source,yaml] ------------- - filtertype: closed exclude: True ------------- This <> will iterate over the actionable list and match indices which are closed. They will remain in, or be removed from the actionable list based on the value of <>. [float] Optional settings ~~~~~~~~~~~~~~~~~ * <> (default is `True`) [[filtertype_count]] == count [source,yaml] ------------- - filtertype: count count: 10 reverse: True use_age: False source: creation_date timestring: field: stats_result: exclude: False ------------- NOTE: Empty values and commented lines will result in the default value, if any, being selected. If a setting is set, but not used by a given <>, it may generate an error. This <> will iterate over the actionable list of indices _or_ snapshots. They are ordered by age, or by alphabet, so as to guarantee that the correct items will remain in, or be removed from the actionable list based on the values of <>, <>, and <>. For use cases, where "like" items are being counted, and their name pattern guarantees date sorting is equal to alphabetical sorting, it is unnecessary to set <> to `True`, as item names will be sorted in <> order by default. This means that the item count will start beginning with the _newest_ indices or snapshots, and proceed through to the oldest. Read more about the different settings for this <>: [float] Required settings ~~~~~~~~~~~~~~~~~ * <> (required) [float] Optional settings ~~~~~~~~~~~~~~~~~ * <> (optional) * <> (optional) * <> (required if `use_age` is `True`) * <> (required if `source` is `name`) * <> (default is `False`) [float] Index-only settings ~~~~~~~~~~~~~~~~~~~ * <> (required if `source` is `field_stats`) * <> (only used if `source` is `field_stats`) [[filtertype_forcemerged]] == forcemerged [source,yaml] ------------- - filtertype: forcemerged max_num_segments: 2 exclude: True ------------- NOTE: Empty values and commented lines will result in the default value, if any, being selected. If a setting is set, but not used by a given <>, it may generate an error. This <> will iterate over the actionable list and match indices which have `max_num_segments` segments per shard, or fewer. They will remain in, or be removed from the actionable list based on the value of <>. [float] Required settings ~~~~~~~~~~~~~~~~~ * <> (required) [float] Optional settings ~~~~~~~~~~~~~~~~~ * <> (default is `True`) [[filtertype_kibana]] == kibana [source,yaml] ------------- - filtertype: kibana exclude: True ------------- This <> will remove indices `.kibana`, `.marvel-kibana`, `kibana-int`, and `.marvel-es-data` from the list of indices, if present. This <> will iterate over the actionable list and match indices `.kibana`, `.marvel-kibana`, `kibana-int`, or `.marvel-es-data`. They will remain in, or be removed from the actionable list based on the value of <>. [float] Optional settings ~~~~~~~~~~~~~~~~~ * <> (default is `True`) [[filtertype_none]] == none [source,yaml] ------------- - filtertype: none ------------- This <> will not filter anything, returning the full list of indices or snapshots. There are no settings for this <>. [[filtertype_opened]] == opened [source,yaml] ------------- - filtertype: opened exclude: True ------------- This <> will iterate over the actionable list and match indices which are opened. They will remain in, or be removed from the actionable list based on the value of <>. [float] Optional settings ~~~~~~~~~~~~~~~~~ * <> (default is `True`) [[filtertype_pattern]] == pattern [source,yaml] ------------- - filtertype: pattern kind: prefix value: logstash- exclude: False ------------- NOTE: Empty values and commented lines will result in the default value, if any, being selected. If a setting is set, but not used by a given <>, it may generate an error. This <> will iterate over the actionable list and match indices matching a given pattern. They will remain in, or be removed from the actionable list based on the value of <>. include::inc_filter_chaining.asciidoc[] Read more about the different settings for this <>: [float] Required settings ~~~~~~~~~~~~~~~~~ * <> (required) * <> (required) [float] Optional settings ~~~~~~~~~~~~~~~~~ * <> (default is `False`) [[filtertype_space]] == space [source,yaml] ------------- - filtertype: space disk_space: 100 reverse: True use_age: False source: creation_date timestring: field: stats_result: exclude: False ------------- NOTE: Empty values and commented lines will result in the default value, if any, being selected. If a setting is set, but not used by a given <>, it may generate an error. This <> will iterate over the actionable list and match indices when their cumulative disk consumption exceeds <> gigabytes. They are first ordered by age, or by alphabet, so as to guarantee the oldest indices are deleted first. They will remain in, or be removed from the actionable list based on the value of <>. [float] Deleting Indices By Space ~~~~~~~~~~~~~~~~~~~~~~~~~ This <> is for those who want to retain indices based on disk consumption, rather than by a set number of days. There are some important caveats regarding this choice: * Elasticsearch cannot calculate the size of closed indices. Elasticsearch does not keep tabs on how much disk-space closed indices consume. If you close indices, your space calculations will be inaccurate. * Indices consume resources just by existing. You could run into performance and/or operational snags in Elasticsearch as the count of indices climbs. * You need to manually calculate how much space across all nodes. The total you give will be the sum of all space consumed across all nodes in your cluster. If you use shard allocation to put more shards or indices on a single node, it will not affect the total space reported by the cluster, but you may still run out of space on that node. These are only a few of the caveats. This is still a valid use-case, especially for those running a single-node test box. NOTE: The <> option will be ignored if <> is `True` For use cases, where "like" indices are being counted, and their name pattern guarantees date sorting is equal to alphabetical sorting, it is unnecessary to set <> to `True`, as index names will be sorted in <> order by default. For this case, this means that disk space calculations will start beginning with the _newest_ indices, and proceeding through to the oldest. Read more about the different settings for this <>: [float] Required settings ~~~~~~~~~~~~~~~~~ * <> (required) [float] Optional settings ~~~~~~~~~~~~~~~~~ * <> (optional) * <> (optional) * <> (required if `use_age` is `True`) * <> (required if `source` is `name`) * <> (required if `source` is `field_stats`) * <> (only used if `source` is `field_stats`) * <> (default is `False`) [[filtertype_state]] == state [source,yaml] ------------- - filtertype: state state: SUCCESS exclude: False ------------- NOTE: Empty values and commented lines will result in the default value, if any, being selected. If a setting is set, but not used by a given <>, it may generate an error. This <> will iterate over the actionable list and match snapshots based on the value of <>. They will remain in, or be removed from the actionable list based on the value of <>. [float] Required settings ~~~~~~~~~~~~~~~~~ * <> (required) [float] Optional settings ~~~~~~~~~~~~~~~~~ * <> (default is `False`) curator-4.2.5/docs/asciidoc/inc_filter_chaining.asciidoc000066400000000000000000000017141302700154000233600ustar00rootroot00000000000000[NOTE] .Filter chaining ===================================================================== It is important to note that while filters can be chained, each is linked by an implied logical *AND* operation. If you want to match from one of several different patterns, as with a logical *OR* operation, you can do so with the <> filtertype using _regex_ as the <>. This example shows how to select multiple indices based on them beginning with either `alpha-`, `bravo-`, or `charlie-`: [source,yaml] ------------- filters: - filtertype: pattern kind: regex value: '^(alpha-|bravo-|charlie-).*$' ------------- Explaining all of the different ways in which regular expressions can be used is outside the scope of this document, but hopefully this gives you some idea of how a regular expression pattern can be used when a logical *OR* is desired. ===================================================================== curator-4.2.5/docs/asciidoc/index.asciidoc000066400000000000000000000007721302700154000205140ustar00rootroot00000000000000:curator_version: 4.2.5 :curator_major: 4 :es_py_version: 2.4.0 :ref: http://www.elastic.co/guide/en/elasticsearch/reference/current [[curator-reference]] = Curator Reference include::about.asciidoc[] include::installation.asciidoc[] include::command-line.asciidoc[] include::configuration.asciidoc[] include::actions.asciidoc[] include::options.asciidoc[] include::filters.asciidoc[] include::filter_elements.asciidoc[] include::examples.asciidoc[] include::security.asciidoc[] include::faq.asciidoc[] curator-4.2.5/docs/asciidoc/installation.asciidoc000066400000000000000000000347001302700154000221040ustar00rootroot00000000000000[[installation]] = Installation [partintro] -- Curator can be installed in a variety of ways, depending on what meets your needs. It is important to note that Curator only requires access to a client node in the Elasticsearch cluster to work. It need not be installed on one of the nodes in the cluster if you do not want it that way. IMPORTANT: If you plan on using SSL/TLS to connect to Elasticsearch, you may encounter issues with older versions of Python and outdated libraries. The binary packages have been compiled with up-to-date versions for your convenience. See the <> page for more information. * <>, the easiest way to use and upgrade. * <>, installs a single, binary package! * <>, installs a single, binary package! * <> * <> * <> -- [[pip]] == pip This installation procedure utilizes https://pip.pypa.io/en/latest/installing.html[python pip], and requires that the target machine has internet connectivity. --------------------------------- pip install elasticsearch-curator --------------------------------- === Upgrading with pip If you already have Elasticsearch Curator installed, and want to upgrade to the latest version, use the `-U` flag: ------------------------------------ pip install -U elasticsearch-curator ------------------------------------ === Installing a specific version with pip The `-U` flag uninstalls the current version (if any), then installs the latest version, or a specified one. Specify a specific version by adding `==` followed by the version you'd like to install, like this: ------------------------------------------- pip install -U elasticsearch-curator==3.5.1 ------------------------------------------- === System-wide vs. User-only installation The above commands each imply a system-wide installation. This usually requires super-user access, or the `sudo` command. There is a way to install Curator into a path for just the current user, using the `--user` flag. ---------------------------------------- pip install --user elasticsearch-curator ---------------------------------------- This will result in the `curator` end-point being installed in the current user's home directory, in the `.local` directory, in the `bin` subdirectory. The full path might look something like this: ----------------------------- /home/user/.local/bin/curator ----------------------------- You can make an alias or a symlink to this so you can call it more easily. The `--user` flag can also be used in conjunction with the `-U` flag: ---------------------------------------- pip install -U --user elasticsearch-curator==3.5.1 ----------------------------------------   [[apt-repository]] == APT repository We use the PGP key http://pgp.mit.edu/pks/lookup?op=vindex&search=0xD27D666CD88E42B4[D88E42B4], Elastic's Signing Key, with fingerprint 4609 5ACC 8548 582C 1A26 99A9 D27D 666C D88E 42B4 to sign all our packages. PGP is available from http://pgp.mit.edu. === Signing Key Download and install the Public Signing Key: [source,sh] -------------------------------------------------- wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - -------------------------------------------------- === Direct Package Download Link * https://packages.elastic.co/curator/{curator_major}/debian/pool/main/p/python/elasticsearch-curator_{curator_version}_amd64.deb[Elasticsearch Curator {curator_version} Binary Package (DEB)] === Repository Configuration Add the following in your `/etc/apt/sources.list.d/` directory in a file with a `.list` suffix, for example `curator.list` ["source","sh",subs="attributes,callouts"] -------------------------------------------------- deb http://packages.elastic.co/curator/{curator_major}/debian stable main -------------------------------------------------- After running `sudo apt-get update`, the repository is ready for use. [WARNING] ================================================== Use the file edit method described above to add the Curator repository. Do not use `add-apt-repository` as it will add a `deb-src` entry as well, but we do not provide a source package. If you have added the `deb-src` entry, you will see an error like the following: Unable to find expected entry 'main/source/Sources' in Release file (Wrong sources.list entry or malformed file) Just delete the `deb-src` entry from the `/etc/apt/sources.list.d/curator.list` file and the installation should work as expected. ================================================== [[apt-binary]] === Binary Package Installation Starting with version 4.1.2, Curator comes with a binary version. What this really means is that the source is compiled, and all required libraries are bundled with the `curator` binary, so there are no conflicts. The binary packages resulting from the creation process have been tested on Debian 8, and Ubuntu 12.04, 14.04, & 16.04. It was also tested on Debian 7, but that failed due to a libc library mismatch. This binary package may work on other similar or derivative variants, but they have not been tested. [source,sh] -------------------------------------------------- sudo apt-get update && sudo apt-get install elasticsearch-curator -------------------------------------------------- This will install the necessary files into `/opt/elasticsearch-curator` and make a symlink at `/usr/bin/curator` that points to the `curator` binary in the aforementioned directory. [WARNING] ==================================================== The `elasticsearch-curator` binary package cannot coexist with the older `python-elasticsearch-curator` packages. This collision is explicitly defined in packages since 4.1.2, but not in previous packages. If it is installed, you _must_ uninstall the `python-elasticsearch-curator` package before installing the `elasticsearch-curator` binary package. ==================================================== [[yum-repository]] == YUM repository We use the PGP key http://pgp.mit.edu/pks/lookup?op=vindex&search=0xD27D666CD88E42B4[D88E42B4], Elastic's Signing Key, with fingerprint 4609 5ACC 8548 582C 1A26 99A9 D27D 666C D88E 42B4 to sign all our packages. It is available from http://pgp.mit.edu. === Signing Key Download and install the public signing key: [source,sh] -------------------------------------------------- rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch -------------------------------------------------- === Direct Package Download Link * https://packages.elastic.co/curator/{curator_major}/centos/6/Packages/elasticsearch-curator-{curator_version}-1.x86_64.rpm[Elasticsearch Curator {curator_version} RHEL/CentOS 6 Binary Package (RPM)] * https://packages.elastic.co/curator/{curator_major}/centos/7/Packages/elasticsearch-curator-{curator_version}-1.x86_64.rpm[Elasticsearch Curator {curator_version} RHEL/CentOS 7 Binary Package (RPM)] === Repository Configuration Add the following in your `/etc/yum.repos.d/` directory in a file with a `.repo` suffix, for example `curator.repo` [WARNING] ======================================== The repositories are different for CentOS/RHEL 6 and 7 due to library and path differences. Be sure to use the correct version for your system! RHEL/CentOS 6: ["source","sh",subs="attributes,callouts"] -------------------------------------------------- [curator-{curator_major}] name=CentOS/RHEL 6 repository for Elasticsearch Curator {curator_major}.x packages baseurl=http://packages.elastic.co/curator/{curator_major}/centos/6 gpgcheck=1 gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch enabled=1 -------------------------------------------------- RHEL/CentOS 7: ["source","sh",subs="attributes,callouts"] -------------------------------------------------- [curator-{curator_major}] name=CentOS/RHEL 7 repository for Elasticsearch Curator {curator_major}.x packages baseurl=http://packages.elastic.co/curator/{curator_major}/centos/7 gpgcheck=1 gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch enabled=1 -------------------------------------------------- ========================================= [[yum-binary]] === Binary Package Installation Starting with version 4.1.2, Curator comes with a binary version. What this really means is that the source is compiled, and all required libraries are bundled with the `curator` binary, so there are no conflicts. There are separate binary packages for RedHat variants. The binary packages resulting from the creation process have been tested on CentOS 6 & 7, with a different binary for each. They may work on similar variants and/or derivatives, but they have not been tested. [source,sh] ---------------------------------------- yum install elasticsearch-curator ---------------------------------------- This will install the necessary files into `/opt/elasticsearch-curator` and make a symlink at `/usr/bin/curator` that points to the `curator` binary in the aforementioned directory. [WARNING] ==================================================== The `elasticsearch-curator` binary package cannot coexist with the older `python-elasticsearch-curator` packages. This collision is explicitly defined in packages since 4.1.2, but not in previous packages. If it is installed, you _must_ uninstall the `python-elasticsearch-curator` package before installing the `elasticsearch-curator` binary package. ==================================================== [[windows-zip]] == Windows Binary Zip Package If you do not wish to install and maintain Python on Windows, there is a compiled binary version available. It is in a directory with EXE files and all necessary libraries that Python requires. You can navigate to the directory and run the `curator` command just as you otherwise would. WARNING: If you do have Python installed, do not uncompress the zip file into your Python directory. It can cause library path collisions which will prevent Curator from properly functioning. * https://packages.elastic.co/curator/{curator_major}/windows/curator-{curator_version}-win32.zip[Download Curator] ** https://packages.elastic.co/curator/{curator_major}/windows/curator-{curator_version}-win32.zip.md5.txt[MD5] ** https://packages.elastic.co/curator/{curator_major}/windows/curator-{curator_version}-win32.zip.sha1.txt[SHA1] [[windows-msi]] == Windows MSI Installer There is now a rudimentary MSI installer available for you to try. One known issue is that in-place upgrades are not possible. Subsequent installs will be side-by-side. The recommended course of action is to uninstall the old version, then install the new one. The installation will default to `"C:\Program Files (x86)\elasticsearch-curator"`. The same binaries and libraries found in the Windows Binary Package will be installed into this directory. * https://packages.elastic.co/curator/{curator_major}/windows/elasticsearch-curator-{curator_version}-win32.msi[Download Curator Installer] ** https://packages.elastic.co/curator/{curator_major}/windows/elasticsearch-curator-{curator_version}-win32.msi.md5.txt[MD5] ** https://packages.elastic.co/curator/{curator_major}/windows/elasticsearch-curator-{curator_version}-win32.msi.sha1.txt[SHA1] [[python-source]] == Installation from source Installing or Curator from source is also possible. In order to do so requires that all dependent libraries are installed first. If you have `pip` installed, then you can install from a gzipped file. If not, you have to uncompress the gzipped file and run `python setup.py install`. That might look like this: [source,sh] -------------------------------------- wget https://pypi.python.org/packages/source/p/package/package-#.#.#.tar.gz tar zxf package-#.#.#.tar.gz cd package-#.#.# python setup.py install -------------------------------------- The dependencies are as follows === setuptools Download https://bootstrap.pypa.io/ez_setup.py[ez_setup.py] and run it using the target Python version. The script will download the appropriate version and install it for you: [source,sh] ----------- wget https://bootstrap.pypa.io/ez_setup.py -O - | python ----------- Note that you will need to invoke the command with superuser privileges to install to the system Python: [source,sh] ----------- wget https://bootstrap.pypa.io/ez_setup.py -O - | sudo python ----------- Alternatively, setuptools may be installed to a user-local path: [source,sh] ----------- wget https://bootstrap.pypa.io/ez_setup.py -O - | python - --user -----------   === Urllib3 Download and install the https://github.com/shazow/urllib3[urllib3] dependency (1.8.3 or greater): . `wget https://pypi.python.org/packages/source/u/urllib3/urllib3-1.12.tar.gz` . `pip install urllib3-1.12.tar.gz` or uncompress and run `python setup.py install`   === click Download and install the http://click.pocoo.org/[click] dependency (6.0 or greater): . `wget https://pypi.python.org/packages/source/c/click/click-6.3.tar.gz -O click-6.3.tar.gz` . `pip install click-6.3.tar.gz` or uncompress and run `python setup.py install`   === certifi Download and install the `certifi` dependency. Always use the most recent version: . `wget https://github.com/certifi/python-certifi/archive/2016.09.26.tar.gz -O certifi.tar.gz` . `pip install certifi.tar.gz`   === PyYAML Download and install the http://pyyaml.org/wiki/PyYAML/[PyYAML] dependency (3.10 or greater): . `wget http://pyyaml.org/download/pyyaml/PyYAML-3.11.tar.gz -O PyYAML-3.11.tar.gz` . `pip install PyYAML-3.11.tar.gz` or uncompress and run `python setup.py install`   === voluptuous Download and install the https://github.com/alecthomas/voluptuous[voluptuous] dependency (0.9.3 or greater): . `wget https://github.com/alecthomas/voluptuous/archive/0.9.3.tar.gz` . `pip install 0.9.3.tar.gz` or uncompress and run `python setup.py install`   === elasticsearch (python module) Download and install the https://github.com/elastic/elasticsearch-py[elasticsearch-py] dependency: . `wget https://github.com/elastic/elasticsearch-py/archive/`+pass:attributes[{es_py_version}].tar.gz -O elasticsearch-py.tar.gz+ . `pip install elasticsearch-py.tar.gz` or uncompress and run `python setup.py install`   === elasticsearch-curator (python module) Download and install Curator: . `wget https://github.com/elastic/curator/archive/v`+pass:attributes[{curator_version}].tar.gz -O elasticsearch-curator.tar.gz+ . `pip install elasticsearch-curator.tar.gz` or uncompress and run `python setup.py install`. At this point you could also run `run_curator.py` from the source directory as well. curator-4.2.5/docs/asciidoc/options.asciidoc000066400000000000000000000342201302700154000210730ustar00rootroot00000000000000[[options]] = Options [partintro] -- Options are settings used by <>. * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> * <> Starting in Curator 4.1, you can use <> in your configuration files. -- [[option_allocation_type]] == allocation_type NOTE: This setting is used only when using the <> The value of this setting must be one of `require`, `include`, or `exclude`. Read more about these settings at {ref}/shard-allocation-filtering.html The default value for this setting is `require`. [[option_continue]] == continue_if_exception NOTE: This setting is available in all actions. If `continue_if_exception` is set to `True`, Curator will attempt to continue on to the next action, if any, even if an exception is encountered. Curator will log but ignore the exception that was raised. The default value for this setting is `False` [[option_count]] == count NOTE: This setting is required when using the <>. The value for this setting is the number of replicas to assign to matching indices. There is no default value. This setting must be set by the user or an exception will be raised, and execution will halt. [[option_delay]] == delay NOTE: This setting is only used by the <>, and is optional. The value for this setting is the number of seconds to delay between forceMerging indices, to allow the cluster to quiesce. There is no default value. [[option_delete_aliases]] == delete_aliases NOTE: This setting is only used by the <>, and is optional. The value for this setting determines whether aliases will be deleted from indices before closing. The default value is `False`. [[option_disable]] == disable_action NOTE: This setting is available in all actions. If `disable_action` is set to `True`, Curator will ignore the current action. This may be useful for temporarily disabling actions in a large configuration file. The default value for this setting is `False` [[option_extra_settings]] == extra_settings This setting should be nested YAML. The values beneath `extra_settings` will be used by whichever action uses the option. There is no default value. [[option_ignore_empty]] == ignore_empty_list This setting must be either `True` or `False`. Depending on your indices, and how you've filtered them, an empty list may be presented to the action. This results in an error condition. When the ignore_empty_list option is set to `True`, the action will exit with an INFO level log message indicating such. If ignore_empty_list is set to `False`, an ERROR level message will be logged, and Curator will exit with code 1. The default value of this setting is `False` [[option_ignore]] == ignore_unavailable NOTE: This setting is only used by the <> action. This setting must be either `True` or `False`. When the ignore_unavailable option is `False` and an index is missing the snapshot request will fail. The default value of this setting is `False` [[option_include_aliases]] == include_aliases NOTE: This setting is only used by the <> action. This setting must be either `True` or `False`. The value of this setting determines whether Elasticsearch should include index aliases when restoring the snapshot. The default value of this setting is `False` [[option_include_gs]] == include_global_state NOTE: This setting is only used by the <> action. This setting must be either `True` or `False`. The value of this setting determines whether Elasticsearch should include the global cluster state with the snapshot. The default value of this setting is `True` [[option_indices]] == indices NOTE: This setting is only used by the <> action. This setting must be a list of indices to restore. Any valid YAML format for lists are acceptable here. If `indices` is left empty, or unset, all indices in the snapshot will be restored. The default value of this setting is `None` [[option_key]] == key NOTE: This setting is required when using the <>. The value of this setting should correspond to a node setting on one or more nodes in your cluster. For example, you might have set [source,sh] ----------- node.tag: myvalue ----------- in your `elasticsearch.yml` file for one or more of your nodes. To match allocation in this case, set key to `tag`. These special attributes are also supported: [cols="2*", options="header"] |=== |attribute |description |`_name` |Match nodes by node name |`_host_ip` |Match nodes by host IP address (IP associated with hostname) |`_publish_ip` |Match nodes by publish IP address |`_ip` |Match either `_host_ip` or `_publish_ip` |`_host` |Match nodes by hostname |=== There is no default value. This setting must be set by the user or an exception will be raised, and execution will halt. [[option_mns]] == max_num_segments NOTE: This setting is required when using the <> or the <>. The value for this setting is the cutoff number of segments per shard. Indices which have more than this number of segments per shard will remain in the index list. There is no default value. This setting must be set by the user or an exception will be raised, and execution will halt. [[option_name]] == name NOTE: This setting is used by the <>, <>, and <> actions. The value of this setting is the name of the alias, snapshot, or index, depending on which action makes use of `name`. This setting may contain a valid Python strftime string. Curator will extract the strftime identifiers and replace them with the corresponding values. The Python strftime identifiers that Curator currently recognizes include: * `Y`: A 4 digit year * `y`: A 2 digit year * `m`: The 2 digit month * `W`: The 2 digit week of the year * `d`: The 2 digit day of the month * `H`: The 2 digit hour of the day, in 24 hour notation * `M`: The 2 digit minute of the hour * `S`: The 2 digit number of second of the minute * `j`: The 3 digit day of the year This option is required by the <> action, and has no default setting in that context. For the <> action, there is no default setting. For the <> action, the default value of this setting is `curator-%Y%m%d%H%M%S` [[option_partial]] == partial NOTE: This setting is only used by the <> action. This setting must be either `True` or `False`. The entire snapshot will fail if one or more indices being added to the snapshot don’t have all primary shards available. This behavior can be changed by setting partial to `True`. The default value of this setting is `False` [[option_rename_pattern]] == rename_pattern NOTE: This setting is only used by the <> action. [TIP] .from the Elasticsearch documentation ====================================== The <> and <> options can be also used to rename indices on restore using regular expression that supports referencing the original text as explained http://docs.oracle.com/javase/6/docs/api/java/util/regex/Matcher.html#appendReplacement(java.lang.StringBuffer,%20java.lang.String)[here]. ====================================== Read more about this setting at {ref}/modules-snapshots.html#_restore There is no default value. [[option_rename_replacement]] == rename_replacement NOTE: This setting is only used by the <> action. [TIP] .From the Elasticsearch documentation ====================================== The <> and <> options can be also used to rename indices on restore using regular expression that supports referencing the original text as explained http://docs.oracle.com/javase/6/docs/api/java/util/regex/Matcher.html#appendReplacement(java.lang.StringBuffer,%20java.lang.String)[here]. ====================================== Read more about this setting at {ref}/modules-snapshots.html#_restore There is no default value. [[option_repository]] == repository NOTE: This setting is only used by the <>, and <> actions. There is no default value. This setting must be set by the user or an exception will be raised, and execution will halt. [[option_retry_count]] == retry_count NOTE: This setting is only used by the <>. The value of this setting is the number of times to retry deleting a snapshot. The default for this setting is `3`. [[option_retry_interval]] == retry_interval NOTE: This setting is only used by the <>. The value of this setting is the number of seconds to delay between retries. The default for this setting is `120`. [[option_routing_type]] == routing_type NOTE: This setting is only used by the <>. The value of this setting must be either `allocation` or `rebalance` There is no default value. This setting must be set by the user or an exception will be raised, and execution will halt. [[option_setting]] == setting NOTE: This setting is only used by the <>. The value of this must be `enable` at present. It is a placeholder for future expansion. There is no default value. This setting must be set by the user or an exception will be raised, and execution will halt. [[option_skip_fsck]] == skip_repo_fs_check NOTE: This setting is only used by the <> action. This setting must be either `True` or `False`. Each master and data node in the cluster _must_ have write access to the shared filesystem used by the repository. Some filesystems may take longer to respond to a check, which results in a false positive for the filesystem access check. For these cases, it is desirable to bypass this verification step, by setting this to `True.` The default value of this setting is `False` [[option_timeout_override]] == timeout_override NOTE: This setting is available in all actions. Actions <>, <>, and <> will override this value to `21600` if `timeout_override` is unset. The <> action will override the value to 180 if unset. Some actions have a default value for `timeout_override`. The following table shows these default values: [cols="m,", options="header"] |=== |Action Name |Default `timeout_override` Value |close |180 |forcemerge |21600 |restore |21600 |snapshot |21600 |=== All other actions have no default value for `timeout_override`. This setting must be an integer number of seconds, or an error will result. This setting is particularly useful for the <> action, the <> action, and any other action where a longer timeout from the client-defined default would be desirable. [[option_value]] == value NOTE: This setting is required when using the <> or the <>. === <> value For the <>, the value of this setting should correspond to a node setting on one or more nodes in your cluster. For example, you might have set [source,sh] ----------- node.tag: myvalue ----------- in your `elasticsearch.yml` file for one or more of your nodes. To match allocation in this case, set value to `myvalue`. Additonally, if you used one of the special attribute names `_ip`, `_name`, `_id`, or `_host` for <>, value can match the one of the node ip addresses, names, ids, or host names, respectively. === <> value For the <>, the acceptable values for this setting depend on the value of <>. Acceptable values when <> is either `allocation` or `rebalance` are `all`, `primaries`, and `none` (string, not `NoneType`). If `routing_type` is `allocation`, this can also be `new_primaries`. If `routing_type` is `rebalance`, then the value can also be `replicas`. There is no default value. This setting must be set by the user or an exception will be raised, and execution will halt. [[option_wfc]] == wait_for_completion NOTE: This setting is used by the <>, <>, <>, and <> actions. This setting must be either `True` or `False`. This setting specifies whether or not the request should return immediately or wait for the operation to complete before returning. TIP: During snapshot initialization, information about all previous snapshots is loaded into the memory, which means that in large repositories it may take several seconds (or even minutes) for this command to return even if the `wait_for_completion` setting is set to `False`. The default value of this setting differs for each action. The default for the <> and <> actions is `False`. The default for the <> and <> actions is `True`. curator-4.2.5/docs/asciidoc/security.asciidoc000066400000000000000000000112701302700154000212470ustar00rootroot00000000000000[[security]] = Security [partintro] -- Please read the following sections for help with securing the connection between Curator and Elasticsearch. * <> * <> -- [[python-security]] == Python and Secure Connectivity Curator was written in Python, which allows it to be distributed as code which can run across a wide variety of systems, including Linux, Windows, Mac OS, and any other system or architecture for which a Python interpreter has been written. Curator was also written to be usable by the 4 most recent major release branches of Python: 2.6, 2.7, 3.4, and 3.5. It may even run on other versions, but those versions are not tested. Unfortunately, this broad support comes at a cost. While Curator happily runs on Python version 2.6, this version had its last update more than 3 years ago. There have been many improvements to security, SSL/TLS and the libraries that support them since then. Not all of these have been back-ported, which results in Curator not being able to communicate securely via SSL/TLS, or in some cases even connect securely. Because it is impossible to know if a given system has the correct Python version, leave alone the most recent libraries and modules, it becomes nearly impossible to guarantee that Curator will be able to make a secure and error-free connection to a secured Elasticsearch instance for any `pip` or RPM/DEB installed modules. This has lead to an increased amount of troubleshooting and support work for Curator. The precompiled binary packages were created to address this. The precompiled binary packages (APT/YUM, Windows) have been compiled with Python 3.5.2, which has all of the up-to-date libraries needed for secure transactions. These packages have been tested connecting to Shield (2.x) and Security (5.x X-Pack) with self-signed PKI certificates. Connectivity via SSL or TLS to other open-source plugins may work, but is not guaranteed. If you are encountering SSL/TLS errors in Curator, please see the list of <>. [[security-errors]] == Common Security Error Messages === Elasticsearch ConnectionError [source,sh] ----------- Unable to create client connection to Elasticsearch. Error:ConnectionError(error return without exception set) caused by: SystemError(error return without exception set) ----------- This error can happen on non-secured connections as well. If it happens with a secured instance, it will usually be accompanied by one or more of the following messages === SNIMissingWarning [source,sh] ----------- SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings ----------- This happens on Python 2 versions older than 2.7.9. These older versions lack https://en.wikipedia.org/wiki/Server_Name_Indication[SNI] support. This can cause servers to present a certificate that the client thinks is invalid. Follow the https://urllib3.readthedocs.io/en/latest/user-guide.html#ssl-py2[pyOpenSSL] guide to resolve this warning. === InsecurePlatformWarning [source,sh] ----------- InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings ----------- This happens on Python 2 platforms that have an outdated **ssl** module. These older **ssl** modules can cause some insecure requests to succeed where they should fail and secure requests to fail where they should succeed. Follow the https://urllib3.readthedocs.io/en/latest/user-guide.html#ssl-py2[pyOpenSSL] guide to resolve this warning. === InsecureRequestWarning [source,sh] ----------- InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html ----------- This happens when an request is made to an HTTPS URL without certificate verification enabled. Follow the https://urllib3.readthedocs.io/en/latest/user-guide.html#ssl[certificate verification] guide to resolve this warning. Related: [source,sh] ----------- SSLError: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed ----------- curator-4.2.5/docs/conf.py000066400000000000000000000213771302700154000154320ustar00rootroot00000000000000# -*- coding: utf-8 -*- # # Elasticsearch documentation build configuration file, created by # sphinx-quickstart on Mon May 6 15:38:41 2013. # # This file is execfile()d with the current directory set to its containing dir. # # Note that not all possible configuration values are present in this # autogenerated file. # # All configuration values have a default; values that are commented out # serve to show the default. import sys, os, re # Utility function to read from file. def fread(fname): return open(os.path.join(os.path.dirname(__file__), fname)).read() def get_version(): VERSIONFILE="../curator/_version.py" verstrline = fread(VERSIONFILE).strip() vsre = r"^__version__ = ['\"]([^'\"]*)['\"]" mo = re.search(vsre, verstrline, re.M) if mo: VERSION = mo.group(1) else: raise RuntimeError("Unable to find version string in %s." % (VERSIONFILE,)) build_number = os.environ.get('CURATOR_BUILD_NUMBER', None) if build_number: return VERSION + "b{}".format(build_number) return VERSION # If extensions (or modules to document with autodoc) are in another directory, # add these directories to sys.path here. If the directory is relative to the # documentation root, use os.path.abspath to make it absolute, like shown here. #sys.path.insert(0, os.path.abspath('.')) sys.path.insert(0, os.path.abspath('../')) # -- General configuration ----------------------------------------------------- # If your documentation needs a minimal Sphinx version, state it here. #needs_sphinx = '1.0' # Add any Sphinx extension module names here, as strings. They can be extensions # coming with Sphinx (named 'sphinx.ext.*') or your custom ones. extensions = ['sphinx.ext.autodoc', 'sphinx.ext.doctest', 'sphinx.ext.intersphinx'] intersphinx_mapping = { 'python': ('https://docs.python.org/3.5', None), 'elasticsearch': ('http://elasticsearch-py.readthedocs.io/en/2.3.0', None), } autoclass_content = "both" # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] # The suffix of source filenames. source_suffix = '.rst' # The encoding of source files. #source_encoding = 'utf-8-sig' # The master toctree document. master_doc = 'index' # General information about the project. project = u'Elasticsearch Curator' copyright = u'2011-2016, Elasticsearch' # The version info for the project you're documenting, acts as replacement for # |version| and |release|, also used in various other places throughout the # built documents. # # The full version, including alpha/beta/rc tags. release = get_version() # The short X.Y version. version = '.'.join(release.split('.')[:2]) # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. #language = None # There are two options for replacing |today|: either, you set today to some # non-false value, then it is used: #today = '' # Else, today_fmt is used as the format for a strftime call. #today_fmt = '%B %d, %Y' # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. exclude_patterns = ['_build'] # The reST default role (used for this markup: `text`) to use for all documents. #default_role = None # If true, '()' will be appended to :func: etc. cross-reference text. #add_function_parentheses = True # If true, the current module name will be prepended to all description # unit titles (such as .. function::). #add_module_names = True # If true, sectionauthor and moduleauthor directives will be shown in the # output. They are ignored by default. #show_authors = False # The name of the Pygments (syntax highlighting) style to use. pygments_style = 'sphinx' # A list of ignored prefixes for module index sorting. #modindex_common_prefix = [] # If true, keep warnings as "system message" paragraphs in the built documents. #keep_warnings = False # -- Options for HTML output --------------------------------------------------- # The theme to use for HTML and HTML Help pages. See the documentation for # a list of builtin themes. html_theme = 'default' # Theme options are theme-specific and customize the look and feel of a theme # further. For a list of options available for each theme, see the # documentation. #html_theme_options = {} # Add any paths that contain custom themes here, relative to this directory. #html_theme_path = [] # The name for this set of Sphinx documents. If None, it defaults to # " v documentation". #html_title = None # A shorter title for the navigation bar. Default is the same as html_title. #html_short_title = None # The name of an image file (relative to this directory) to place at the top # of the sidebar. #html_logo = None # The name of an image file (within the static path) to use as favicon of the # docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 # pixels large. #html_favicon = None # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". html_static_path = ['_static'] # If not '', a 'Last updated on:' timestamp is inserted at every page bottom, # using the given strftime format. #html_last_updated_fmt = '%b %d, %Y' # If true, SmartyPants will be used to convert quotes and dashes to # typographically correct entities. #html_use_smartypants = True # Custom sidebar templates, maps document names to template names. #html_sidebars = {} # Additional templates that should be rendered to pages, maps page names to # template names. #html_additional_pages = {} # If false, no module index is generated. #html_domain_indices = True # If false, no index is generated. #html_use_index = True # If true, the index is split into individual pages for each letter. #html_split_index = False # If true, links to the reST sources are added to the pages. #html_show_sourcelink = True # If true, "Created using Sphinx" is shown in the HTML footer. Default is True. #html_show_sphinx = True # If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. #html_show_copyright = True # If true, an OpenSearch description file will be output, and all pages will # contain a tag referring to it. The value of this option must be the # base URL from which the finished HTML is served. #html_use_opensearch = '' # This is the file name suffix for HTML files (e.g. ".xhtml"). #html_file_suffix = None # Output file base name for HTML help builder. htmlhelp_basename = 'Curatordoc' # -- Options for LaTeX output -------------------------------------------------- latex_elements = { # The paper size ('letterpaper' or 'a4paper'). #'papersize': 'letterpaper', # The font size ('10pt', '11pt' or '12pt'). #'pointsize': '10pt', # Additional stuff for the LaTeX preamble. #'preamble': '', } # Grouping the document tree into LaTeX files. List of tuples # (source start file, target name, title, author, documentclass [howto/manual]). latex_documents = [ ('index', 'ES_Curator.tex', u'Elasticsearch Curator Documentation', u'Aaron Mildenstein', 'manual'), ] # The name of an image file (relative to this directory) to place at the top of # the title page. #latex_logo = None # For "manual" documents, if this is true, then toplevel headings are parts, # not chapters. #latex_use_parts = False # If true, show page references after internal links. #latex_show_pagerefs = False # If true, show URL addresses after external links. #latex_show_urls = False # Documents to append as an appendix to all manuals. #latex_appendices = [] # If false, no module index is generated. #latex_domain_indices = True # -- Options for manual page output -------------------------------------------- # One entry per manual page. List of tuples # (source start file, name, description, authors, manual section). man_pages = [ ('index', 'curator', u'Elasticsearch Curator Documentation', [u'Aaron Mildenstein'], 1) ] # If true, show URL addresses after external links. #man_show_urls = False # -- Options for Texinfo output ------------------------------------------------ # Grouping the document tree into Texinfo files. List of tuples # (source start file, target name, title, author, # dir menu entry, description, category) texinfo_documents = [ ('index', 'Curator', u'Elasticsearch Curator Documentation', u'Aaron Mildenstein', 'Curator', 'One line description of project.', 'Miscellaneous'), ] # Documents to append as an appendix to all manuals. #texinfo_appendices = [] # If false, no module index is generated. #texinfo_domain_indices = True # How to display URL addresses: 'footnote', 'no', or 'inline'. #texinfo_show_urls = 'footnote' # If true, do not generate a @detailmenu in the "Top" node's menu. #texinfo_no_detailmenu = False curator-4.2.5/docs/examples.rst000066400000000000000000000043771302700154000165040ustar00rootroot00000000000000.. _examples: Examples ======== Each of these examples presupposes that the requisite modules have been imported and an instance of the Elasticsearch client object has been created: :: import elasticsearch import curator client = elasticsearch.Elasticsearch() Filter indices by prefix ++++++++++++++++++++++++ :: ilo = curator.IndexList(client) ilo.filter_by_regex(kind='prefix', value='logstash-') The contents of `ilo.indices` would then only be indices matching the `prefix`. Filter indices by suffix ++++++++++++++++++++++++ :: ilo = curator.IndexList(client) ilo.filter_by_regex(kind='suffix', value='-prod') The contents of `ilo.indices` would then only be indices matching the `suffix`. Filter indices by age (name) ++++++++++++++++++++++++++++ This example will match indices with the following criteria: * Have a date string of ``%Y.%m.%d`` * Use `days` as the unit of time measurement * Filter indices `older` than 5 `days` :: ilo = curator.IndexList(client) ilo.filter_by_age(source='name', direction='older', timestring='%Y.%m.%d', unit='days', unit_count=5 ) The contents of `ilo.indices` would then only be indices matching these criteria. Filter indices by age (creation_date) +++++++++++++++++++++++++++++++++++++ This example will match indices with the following criteria: * Use `months` as the unit of time measurement * Filter indices where the index creation date is `older` than 2 `months` from this moment. :: ilo = curator.IndexList(client) ilo.filter_by_age(source='creation_date', direction='older', unit='months', unit_count=2 ) The contents of `ilo.indices` would then only be indices matching these criteria. Filter indices by age (field_stats) +++++++++++++++++++++++++++++++++++ This example will match indices with the following criteria: * Use `days` as the unit of time measurement * Filter indices where the `timestamp` field's `min_value` is a date `older` than 3 `weeks` from this moment. :: ilo = curator.IndexList(client) ilo.filter_by_age(source='field_stats', direction='older', unit='weeks', unit_count=3, field='timestamp', stats_result='min_value' ) The contents of `ilo.indices` would then only be indices matching these criteria. curator-4.2.5/docs/filters.rst000066400000000000000000000024511302700154000163250ustar00rootroot00000000000000.. _filters: Filter Methods ============== * `IndexList`_ * `SnapshotList`_ IndexList --------- .. automethod:: curator.indexlist.IndexList.filter_allocated :noindex: .. automethod:: curator.indexlist.IndexList.filter_by_age :noindex: .. automethod:: curator.indexlist.IndexList.filter_by_regex :noindex: .. automethod:: curator.indexlist.IndexList.filter_by_space :noindex: .. automethod:: curator.indexlist.IndexList.filter_closed :noindex: .. automethod:: curator.indexlist.IndexList.filter_forceMerged :noindex: .. automethod:: curator.indexlist.IndexList.filter_kibana :noindex: .. automethod:: curator.indexlist.IndexList.filter_opened :noindex: .. automethod:: curator.indexlist.IndexList.filter_none :noindex: .. automethod:: curator.indexlist.IndexList.filter_by_alias :noindex: .. automethod:: curator.indexlist.IndexList.filter_by_count :noindex: SnapshotList ------------ .. automethod:: curator.snapshotlist.SnapshotList.filter_by_age :noindex: .. automethod:: curator.snapshotlist.SnapshotList.filter_by_regex :noindex: .. automethod:: curator.snapshotlist.SnapshotList.filter_by_state :noindex: .. automethod:: curator.snapshotlist.SnapshotList.filter_none :noindex: .. automethod:: curator.snapshotlist.SnapshotList.filter_by_count :noindex: curator-4.2.5/docs/index.rst000066400000000000000000000055071302700154000157710ustar00rootroot00000000000000Elasticsearch Curator Python API ================================ The Elasticsearch Curator Python API helps you manage your indices and snapshots. .. note:: This documentation is for the Elasticsearch Curator Python API. Documentation for the Elasticsearch Curator *CLI* -- which uses this API and is installed as an entry_point as part of the package -- is available in the `Elastic guide`_. .. _Elastic guide: http://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html Compatibility ------------- The Elasticsearch Curator Python API is compatible with Elasticsearch versions 2.x through 5.0, and supports Python versions 2.6 and later. Example Usage ------------- :: import elasticsearch import curator client = elasticsearch.Elasticsearch() ilo = curator.IndexList(client) ilo.filter_by_regex(kind='prefix', value='logstash-') ilo.filter_by_age(source='name', direction='older', timestring='%Y.%m.%d', unit='days', unit_count=30) delete_indices = curator.DeleteIndices(ilo) delete_indices.do_action() .. TIP:: See more examples in the :doc:`Examples ` page. Features -------- The API methods fall into the following categories: * :doc:`Object Classes ` build and filter index list or snapshot list objects. * :doc:`Action Classes ` act on object classes. * :doc:`Utilities ` are helper methods. Logging ~~~~~~~ The Elasticsearch Curator Python API uses the standard `logging library`_ from Python. It inherits two loggers from ``elasticsearch-py``: ``elasticsearch`` and ``elasticsearch.trace``. Clients use the ``elasticsearch`` logger to log standard activity, depending on the log level. The ``elasticsearch.trace`` logger logs requests to the server in JSON format as pretty-printed ``curl`` commands that you can execute from the command line. The ``elasticsearch.trace`` logger is not inherited from the base logger and must be activated separately. .. _logging library: http://docs.python.org/3.5/library/logging.html Contents -------- .. toctree:: :maxdepth: 2 objectclasses actionclasses filters utilities examples Changelog License ------- Copyright (c) 2012–2016 Elasticsearch Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Indices and tables ------------------ * :ref:`genindex` * :ref:`search` curator-4.2.5/docs/objectclasses.rst000066400000000000000000000003751302700154000175040ustar00rootroot00000000000000.. _objectclasses: Object Classes ============== * `IndexList`_ * `SnapshotList`_ IndexList --------- .. autoclass:: curator.indexlist.IndexList :members: SnapshotList ------------ .. autoclass:: curator.snapshotlist.SnapshotList :members: curator-4.2.5/docs/utilities.rst000066400000000000000000000002371302700154000166700ustar00rootroot00000000000000.. _utilities: Utility & Helper Methods ======================== .. automodule:: curator.utils :members: .. autoclass:: curator.SchemaCheck :members: curator-4.2.5/examples/000077500000000000000000000000001302700154000150075ustar00rootroot00000000000000curator-4.2.5/examples/actions/000077500000000000000000000000001302700154000164475ustar00rootroot00000000000000curator-4.2.5/examples/actions/alias.yml000066400000000000000000000024351302700154000202670ustar00rootroot00000000000000--- # Remember, leave a key empty if there is no value. None will be a string, # not a Python "NoneType" # # Also remember that all examples have 'disable_action' set to True. If you # want to use this action as a template, be sure to set this to False after # copying it. actions: 1: action: alias description: >- Alias indices older than 7 days but newer than 14 days, with a prefix of logstash- to 'last_week', remove indices older than 14 days. options: name: last_week extra_settings: timeout_override: continue_if_exception: False disable_action: True add: filters: - filtertype: pattern kind: prefix value: logstash- exclude: - filtertype: age source: name direction: older timestring: '%Y.%m.%d' unit: days unit_count: 7 exclude: - filtertype: age direction: younger timestring: '%Y.%m.%d' unit: days unit_count: 14 exclude: remove: filters: - filtertype: pattern kind: prefix value: logstash- exclude: - filtertype: age source: name direction: older timestring: '%Y.%m.%d' unit: days unit_count: 14 exclude: curator-4.2.5/examples/actions/allocation.yml000066400000000000000000000015741302700154000213260ustar00rootroot00000000000000--- # Remember, leave a key empty if there is no value. None will be a string, # not a Python "NoneType" # # Also remember that all examples have 'disable_action' set to True. If you # want to use this action as a template, be sure to set this to False after # copying it. actions: 1: action: allocation description: >- Apply shard allocation routing to 'require' 'tag=cold' for hot/cold node setup for logstash- indices older than 3 days, based on index_creation date options: key: tag value: cold allocation_type: require wait_for_completion: False continue_if_exception: False disable_action: True filters: - filtertype: pattern kind: prefix value: logstash- exclude: - filtertype: age source: creation_date direction: older unit: days unit_count: 3 exclude: curator-4.2.5/examples/actions/close.yml000066400000000000000000000014341302700154000203010ustar00rootroot00000000000000--- # Remember, leave a key empty if there is no value. None will be a string, # not a Python "NoneType" # # Also remember that all examples have 'disable_action' set to True. If you # want to use this action as a template, be sure to set this to False after # copying it. actions: 1: action: close description: >- Close indices older than 30 days (based on index name), for logstash- prefixed indices. options: delete_aliases: False timeout_override: continue_if_exception: False disable_action: True filters: - filtertype: pattern kind: prefix value: logstash- exclude: - filtertype: age source: name direction: older timestring: '%Y.%m.%d' unit: days unit_count: 30 exclude: curator-4.2.5/examples/actions/create_index.yml000066400000000000000000000011461302700154000216260ustar00rootroot00000000000000--- # Remember, leave a key empty if there is no value. None will be a string, # not a Python "NoneType" # # Also remember that all examples have 'disable_action' set to True. If you # want to use this action as a template, be sure to set this to False after # copying it. actions: 1: action: create_index description: Create the index as named, with the specified extra settings. options: name: myindex extra_settings: settings: number_of_shards: 2 number_of_replicas: 1 timeout_override: continue_if_exception: False disable_action: True curator-4.2.5/examples/actions/delete_indices.yml000066400000000000000000000016441302700154000221370ustar00rootroot00000000000000--- # Remember, leave a key empty if there is no value. None will be a string, # not a Python "NoneType" # # Also remember that all examples have 'disable_action' set to True. If you # want to use this action as a template, be sure to set this to False after # copying it. actions: 1: action: delete_indices description: >- Delete indices older than 45 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly. options: ignore_empty_list: True timeout_override: continue_if_exception: False disable_action: True filters: - filtertype: pattern kind: prefix value: logstash- exclude: - filtertype: age source: name direction: older timestring: '%Y.%m.%d' unit: days unit_count: 45 exclude: curator-4.2.5/examples/actions/delete_snapshots.yml000066400000000000000000000014561302700154000225440ustar00rootroot00000000000000--- # Remember, leave a key empty if there is no value. None will be a string, # not a Python "NoneType" # # Also remember that all examples have 'disable_action' set to True. If you # want to use this action as a template, be sure to set this to False after # copying it. actions: 1: action: delete_snapshots description: >- Delete snapshots from the selected repository older than 45 days (based on creation_date), for 'curator-' prefixed snapshots. options: repository: timeout_override: continue_if_exception: False disable_action: True filters: - filtertype: pattern kind: prefix value: curator- exclude: - filtertype: age source: creation_date direction: older unit: days unit_count: 45 exclude: curator-4.2.5/examples/actions/forcemerge.yml000066400000000000000000000020531302700154000213100ustar00rootroot00000000000000--- # Remember, leave a key empty if there is no value. None will be a string, # not a Python "NoneType" # # Also remember that all examples have 'disable_action' set to True. If you # want to use this action as a template, be sure to set this to False after # copying it. actions: 1: action: forcemerge description: >- forceMerge logstash- prefixed indices older than 2 days (based on index creation_date) to 2 segments per shard. Delay 120 seconds between each forceMerge operation to allow the cluster to quiesce. This action will ignore indices already forceMerged to the same or fewer number of segments per shard, so the 'forcemerged' filter is unneeded. options: max_num_segments: 2 delay: 120 timeout_override: continue_if_exception: False disable_action: True filters: - filtertype: pattern kind: prefix value: logstash- exclude: - filtertype: age source: creation_date direction: older unit: days unit_count: 2 exclude: curator-4.2.5/examples/actions/open.yml000066400000000000000000000016531302700154000201400ustar00rootroot00000000000000--- # Remember, leave a key empty if there is no value. None will be a string, # not a Python "NoneType" # # Also remember that all examples have 'disable_action' set to True. If you # want to use this action as a template, be sure to set this to False after # copying it. actions: 1: action: open description: >- Open indices older than 30 days but younger than 60 days (based on index name), for logstash- prefixed indices. options: timeout_override: continue_if_exception: False disable_action: True filters: - filtertype: pattern kind: prefix value: logstash- exclude: - filtertype: age source: name direction: older timestring: '%Y.%m.%d' unit: days unit_count: 30 exclude: - filtertype: age source: name direction: younger timestring: '%Y.%m.%d' unit: days unit_count: 60 exclude: curator-4.2.5/examples/actions/replicas.yml000066400000000000000000000014661302700154000210030ustar00rootroot00000000000000--- # Remember, leave a key empty if there is no value. None will be a string, # not a Python "NoneType" # # Also remember that all examples have 'disable_action' set to True. If you # want to use this action as a template, be sure to set this to False after # copying it. actions: 1: action: replicas description: >- Reduce the replica count to 0 for logstash- prefixed indices older than 10 days (based on index creation_date) options: count: 0 wait_for_completion: False timeout_override: continue_if_exception: False disable_action: True filters: - filtertype: pattern kind: prefix value: logstash- exclude: - filtertype: age source: creation_date direction: older unit: days unit_count: 10 exclude: curator-4.2.5/examples/actions/restore.yml000066400000000000000000000024601302700154000206570ustar00rootroot00000000000000--- # Remember, leave a key empty if there is no value. None will be a string, # not a Python "NoneType" # # Also remember that all examples have 'disable_action' set to True. If you # want to use this action as a template, be sure to set this to False after # copying it. actions: 1: action: restore description: >- Restore all indices in the most recent curator-* snapshot with state SUCCESS. Wait for the restore to complete before continuing. Do not skip the repository filesystem access check. Use the other options to define the index/shard settings for the restore. options: repository: # Leaving name blank will result in restoring the most recent snapshot by age name: # Leaving indices blank will result in restoring all indices in the snapshot indices: include_aliases: False ignore_unavailable: False include_global_state: True partial: False rename_pattern: rename_replacement: extra_settings: wait_for_completion: True skip_repo_fs_check: False timeout_override: continue_if_exception: False disable_action: False filters: - filtertype: pattern kind: prefix value: curator- exclude: - filtertype: state state: SUCCESS exclude: curator-4.2.5/examples/actions/snapshot.yml000066400000000000000000000023101302700154000210250ustar00rootroot00000000000000--- # Remember, leave a key empty if there is no value. None will be a string, # not a Python "NoneType" # # Also remember that all examples have 'disable_action' set to True. If you # want to use this action as a template, be sure to set this to False after # copying it. actions: 1: action: snapshot description: >- Snapshot logstash- prefixed indices older than 1 day (based on index creation_date) with the default snapshot name pattern of 'curator-%Y%m%d%H%M%S'. Wait for the snapshot to complete. Do not skip the repository filesystem access check. Use the other options to create the snapshot. options: repository: # Leaving name blank will result in the default 'curator-%Y%m%d%H%M%S' name: ignore_unavailable: False include_global_state: True partial: False wait_for_completion: True skip_repo_fs_check: False timeout_override: continue_if_exception: False disable_action: False filters: - filtertype: pattern kind: prefix value: logstash- exclude: - filtertype: age source: creation_date direction: older unit: days unit_count: 1 exclude: curator-4.2.5/examples/curator.yml000066400000000000000000000006741302700154000172200ustar00rootroot00000000000000--- # Remember, leave a key empty if there is no value. None will be a string, # not a Python "NoneType" client: hosts: - 127.0.0.1 port: 9200 url_prefix: use_ssl: False certificate: client_cert: client_key: aws_key: aws_secret_key: aws_region: ssl_no_validate: False http_auth: timeout: 30 master_only: False logging: loglevel: INFO logfile: logformat: default blacklist: ['elasticsearch', 'urllib3'] curator-4.2.5/requirements.txt000066400000000000000000000001511302700154000164520ustar00rootroot00000000000000urllib3>=1.8.3 elasticsearch>=2.4.0,<3.0.0 click>=6.0 pyyaml>=3.10 voluptuous>=0.9.3 certifi>=2016.09.26 curator-4.2.5/run_curator.py000077500000000000000000000015761302700154000161220ustar00rootroot00000000000000#!/usr/bin/env python """Wrapper for running curator from source.""" from curator.cli import cli if __name__ == '__main__': try: cli() except Exception as e: if type(e) == type(RuntimeError()): if 'ASCII' in str(e): print('{0}'.format(e)) print( ''' When used with Python 3 (and the DEB and RPM packages of Curator are compiled and bundled with Python 3), Curator requires the locale to be unicode. Any of the above unicode definitions are acceptable. To set the locale to be unicode, try: $ export LC_ALL=en_US.utf8 $ curator [ARGS] Alternately, you should be able to specify the locale on the command-line: $ LC_ALL=en_US.utf8 curator [ARGS] Be sure to substitute your unicode variant for en_US.utf8 ''' ) else: import sys print('{0}'.format(e)) sys.exit(1) curator-4.2.5/run_es_repo_mgr.py000077500000000000000000000016431302700154000167370ustar00rootroot00000000000000#!/usr/bin/env python """Wrapper for running es_repo_mgr from source.""" from curator.repomgrcli import repo_mgr_cli if __name__ == '__main__': try: repo_mgr_cli() except Exception as e: if type(e) == type(RuntimeError()): if 'ASCII' in str(e): print('{0}'.format(e)) print( ''' When used with Python 3 (and the DEB and RPM packages of Curator are compiled and bundled with Python 3), Curator requires the locale to be unicode. Any of the above unicode definitions are acceptable. To set the locale to be unicode, try: $ export LC_ALL=en_US.utf8 $ es_repo_mgr [ARGS] Alternately, you should be able to specify the locale on the command-line: $ LC_ALL=en_US.utf8 es_repo_mgr [ARGS] Be sure to substitute your unicode variant for en_US.utf8 ''' ) else: import sys print('{0}'.format(e)) sys.exit(1) curator-4.2.5/run_singleton.py000077500000000000000000000016421302700154000164370ustar00rootroot00000000000000#!/usr/bin/env python """Wrapper for running singletons from source.""" import click from curator.singletons import cli if __name__ == '__main__': try: cli(obj={}) except Exception as e: if type(e) == type(RuntimeError()): if 'ASCII' in str(e): print('{0}'.format(e)) print( ''' When used with Python 3 (and the DEB and RPM packages of Curator are compiled and bundled with Python 3), Curator requires the locale to be unicode. Any of the above unicode definitions are acceptable. To set the locale to be unicode, try: $ export LC_ALL=en_US.utf8 $ curator_cli [ARGS] Alternately, you should be able to specify the locale on the command-line: $ LC_ALL=en_US.utf8 curator_cli [ARGS] Be sure to substitute your unicode variant for en_US.utf8 ''' ) else: import sys print('{0}'.format(e)) sys.exit(1) curator-4.2.5/setup.cfg000066400000000000000000000001031302700154000150040ustar00rootroot00000000000000[metadata] description-file = README.md [bdist_wheel] universal=1 curator-4.2.5/setup.py000066400000000000000000000125401302700154000147050ustar00rootroot00000000000000import os import re import sys from setuptools import setup # Utility function to read from file. def fread(fname): return open(os.path.join(os.path.dirname(__file__), fname)).read() def get_version(): VERSIONFILE="curator/_version.py" verstrline = fread(VERSIONFILE).strip() vsre = r"^__version__ = ['\"]([^'\"]*)['\"]" mo = re.search(vsre, verstrline, re.M) if mo: VERSION = mo.group(1) else: raise RuntimeError("Unable to find version string in %s." % (VERSIONFILE,)) build_number = os.environ.get('CURATOR_BUILD_NUMBER', None) if build_number: return VERSION + "b{}".format(build_number) return VERSION def get_install_requires(): res = ['elasticsearch>=2.4.0,<3.0.0' ] res.append('click>=6.0') res.append('pyyaml>=3.10') res.append('voluptuous>=0.9.3') res.append('certifi>=2016.09.26') return res try: ### cx_Freeze ### from cx_Freeze import setup, Executable try: import certifi cert_file = certifi.where() except ImportError: cert_file = '' # Dependencies are automatically detected, but it might need # fine tuning. buildOptions = dict( packages = [], excludes = [], include_files = [cert_file], ) base = 'Console' icon = None if os.path.exists('Elastic.ico'): icon = 'Elastic.ico' curator_exe = Executable( "run_curator.py", base=base, targetName = "curator", ) curator_cli_exe = Executable( "run_singleton.py", base=base, targetName = "curator_cli", ) repomgr_exe = Executable( "run_es_repo_mgr.py", base=base, targetName = "es_repo_mgr", ) if sys.platform == "win32": curator_exe = Executable( "run_curator.py", base=base, targetName = "curator.exe", icon = icon ) curator_cli_exe = Executable( "run_singleton.py", base=base, targetName = "curator_cli.exe", icon = icon ) repomgr_exe = Executable( "run_es_repo_mgr.py", base=base, targetName = "es_repo_mgr.exe", icon = icon ) setup( name = "elasticsearch-curator", version = get_version(), author = "Elastic", author_email = "info@elastic.co", description = "Tending your Elasticsearch indices", long_description=fread('README.rst'), url = "http://github.com/elastic/curator", download_url = "https://github.com/elastic/curator/tarball/v" + get_version(), license = "Apache License, Version 2.0", install_requires = get_install_requires(), keywords = "elasticsearch time-series indexed index-expiry", packages = ["curator"], include_package_data=True, entry_points = { "console_scripts" : [ "curator = curator.cli:cli", "curator_cli = curator.curator_cli:main", "es_repo_mgr = curator.repomgrcli:repo_mgr_cli", ] }, classifiers=[ "Intended Audience :: Developers", "Intended Audience :: System Administrators", "License :: OSI Approved :: Apache Software License", "Operating System :: OS Independent", "Programming Language :: Python", "Programming Language :: Python :: 2.6", "Programming Language :: Python :: 2.7", "Programming Language :: Python :: 3.4", "Programming Language :: Python :: 3.5", ], test_suite = "test.run_tests.run_all", tests_require = ["mock", "nose", "coverage", "nosexcover"], options = {"build_exe" : buildOptions}, executables = [curator_exe,curator_cli_exe,repomgr_exe] ) ### end cx_Freeze ### except ImportError: setup( name = "elasticsearch-curator", version = get_version(), author = "Elastic", author_email = "info@elastic.co", description = "Tending your Elasticsearch indices", long_description=fread('README.rst'), url = "http://github.com/elastic/curator", download_url = "https://github.com/elastic/curator/tarball/v" + get_version(), license = "Apache License, Version 2.0", install_requires = get_install_requires(), keywords = "elasticsearch time-series indexed index-expiry", packages = ["curator"], include_package_data=True, entry_points = { "console_scripts" : [ "curator = curator.cli:cli", "curator_cli = curator.curator_cli:main", "es_repo_mgr = curator.repomgrcli:repo_mgr_cli", ] }, classifiers=[ "Intended Audience :: Developers", "Intended Audience :: System Administrators", "License :: OSI Approved :: Apache Software License", "Operating System :: OS Independent", "Programming Language :: Python", "Programming Language :: Python :: 2.6", "Programming Language :: Python :: 2.7", "Programming Language :: Python :: 3.4", "Programming Language :: Python :: 3.5", ], test_suite = "test.run_tests.run_all", tests_require = ["mock", "nose", "coverage", "nosexcover"] ) curator-4.2.5/test/000077500000000000000000000000001302700154000141505ustar00rootroot00000000000000curator-4.2.5/test/__init__.py000066400000000000000000000000001302700154000162470ustar00rootroot00000000000000curator-4.2.5/test/integration/000077500000000000000000000000001302700154000164735ustar00rootroot00000000000000curator-4.2.5/test/integration/__init__.py000066400000000000000000000127371302700154000206160ustar00rootroot00000000000000import time import os import shutil import tempfile import random import string from datetime import timedelta, datetime, date from elasticsearch import Elasticsearch from elasticsearch.exceptions import ConnectionError from unittest import SkipTest, TestCase from mock import Mock client = None DATEMAP = { 'months': '%Y.%m', 'weeks': '%Y.%W', 'days': '%Y.%m.%d', 'hours': '%Y.%m.%d.%H', } host, port = os.environ.get('TEST_ES_SERVER', 'localhost:9200').split(':') port = int(port) if port else 9200 def random_directory(): dirname = ''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(8)) directory = tempfile.mkdtemp(suffix=dirname) if not os.path.exists(directory): os.makedirs(directory) return directory def get_client(): global client if client is not None: return client client = Elasticsearch([os.environ.get('TEST_ES_SERVER', {})], timeout=300) # wait for yellow status for _ in range(100): time.sleep(.1) try: client.cluster.health(wait_for_status='yellow') return client except ConnectionError: continue else: # timeout raise SkipTest("Elasticsearch failed to start.") def setup(): get_client() class Args(dict): def __getattr__(self, att_name): return self.get(att_name, None) class CuratorTestCase(TestCase): def setUp(self): super(CuratorTestCase, self).setUp() self.client = get_client() args = {} args['host'], args['port'] = host, port args['time_unit'] = 'days' args['prefix'] = 'logstash-' self.args = args # dirname = ''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(8)) # This will create a psuedo-random temporary directory on the machine # which runs the unit tests, but NOT on the machine where elasticsearch # is running. This means tests may fail if run against remote instances # unless you explicitly set `self.args['location']` to a proper spot # on the target machine. self.args['location'] = random_directory() self.args['configdir'] = random_directory() self.args['configfile'] = os.path.join(self.args['configdir'], 'curator.yml') self.args['actionfile'] = os.path.join(self.args['configdir'], 'actions.yml') self.args['repository'] = 'TEST_REPOSITORY' # if not os.path.exists(self.args['location']): # os.makedirs(self.args['location']) def tearDown(self): self.delete_repositories() self.client.indices.delete(index='*') self.client.indices.delete_template(name='*', ignore=404) for path_arg in ['location', 'configdir']: if os.path.exists(self.args[path_arg]): shutil.rmtree(self.args[path_arg]) def parse_args(self): return Args(self.args) def create_indices(self, count, unit=None): now = datetime.utcnow() unit = unit if unit else self.args['time_unit'] format = DATEMAP[unit] if not unit == 'months': step = timedelta(**{unit: 1}) for x in range(count): self.create_index(self.args['prefix'] + now.strftime(format), wait_for_yellow=False) now -= step else: # months now = date.today() d = date(now.year, now.month, 1) self.create_index(self.args['prefix'] + now.strftime(format), wait_for_yellow=False) for i in range(1, count): if d.month == 1: d = date(d.year-1, 12, 1) else: d = date(d.year, d.month-1, 1) self.create_index(self.args['prefix'] + datetime(d.year, d.month, 1).strftime(format), wait_for_yellow=False) self.client.cluster.health(wait_for_status='yellow') def create_index(self, name, shards=1, wait_for_yellow=True): self.client.indices.create( index=name, body={'settings': {'number_of_shards': shards, 'number_of_replicas': 0}} ) if wait_for_yellow: self.client.cluster.health(wait_for_status='yellow') def add_docs(self, idx): for i in ["1", "2", "3"]: self.client.create( index=idx, doc_type='log', id=i, body={"doc" + i :'TEST DOCUMENT'}, ) # This should force each doc to be in its own segment. self.client.indices.flush(index=idx, force=True) def create_snapshot(self, name, csv_indices): body = { "indices": csv_indices, "ignore_unavailable": False, "include_global_state": True, "partial": False, } self.create_repository() self.client.snapshot.create( repository=self.args['repository'], snapshot=name, body=body, wait_for_completion=True ) def create_repository(self): body = {'type':'fs', 'settings':{'location':self.args['location']}} self.client.snapshot.create_repository(repository=self.args['repository'], body=body) def delete_repositories(self): result = self.client.snapshot.get_repository(repository='_all') for repo in result: self.client.snapshot.delete_repository(repository=repo) def close_index(self, name): self.client.indices.close(index=name) def write_config(self, fname, data): with open(fname, 'w') as f: f.write(data) curator-4.2.5/test/integration/test_alias.py000066400000000000000000000167011302700154000212020ustar00rootroot00000000000000import elasticsearch import curator import os import json import string, random, tempfile import click from click import testing as clicktest from mock import patch, Mock from . import CuratorTestCase from . import testvars as testvars import logging logger = logging.getLogger(__name__) host, port = os.environ.get('TEST_ES_SERVER', 'localhost:9200').split(':') port = int(port) if port else 9200 class TestCLIAlias(CuratorTestCase): def test_add_only(self): alias = 'testalias' self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.alias_add_only.format(alias)) self.create_index('my_index') self.create_index('dummy') self.client.indices.put_alias(index='dummy', name=alias) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertEquals(2, len(self.client.indices.get_alias(name=alias))) def test_add_only_with_extra_settings(self): alias = 'testalias' self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.alias_add_only_with_extra_settings.format(alias)) self.create_index('my_index') test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertEquals( { 'my_index': { 'aliases': { 'testalias': { 'filter': { 'term': { 'user': 'kimchy' } } } } } }, self.client.indices.get_alias(name=alias) ) def test_alias_remove_only(self): alias = 'testalias' self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.alias_remove_only.format(alias)) self.create_index('my_index') self.create_index('dummy') self.client.indices.put_alias(index='dummy', name=alias) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertEqual( {'dummy': {'aliases': {}}, 'my_index': {'aliases': {}}}, self.client.indices.get_aliases() ) def test_add_only_skip_closed(self): alias = 'testalias' self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.alias_add_only.format(alias)) self.create_index('my_index') self.client.indices.close(index='my_index') self.create_index('dummy') self.client.indices.put_alias(index='dummy', name=alias) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) version = curator.get_version(self.client) if version > (3,0,0): self.assertEquals(2, len(self.client.indices.get_alias(name=alias))) else: self.assertEquals(1, len(self.client.indices.get_alias(name=alias))) def test_add_and_remove(self): alias = 'testalias' self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.alias_add_remove.format(alias)) self.create_index('my_index') self.create_index('dummy') self.client.indices.put_alias(index='dummy', name=alias) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertEqual( {'dummy': {'aliases': {}}, 'my_index': {'aliases': {alias: {}}}}, self.client.indices.get_aliases() ) # def test_add_only_huge_list(self): # alias = 'testalias' # self.write_config( # self.args['configfile'], testvars.client_config.format(host, port)) # self.write_config(self.args['actionfile'], # testvars.alias_add_only.format(alias)) # self.create_indices(365) # test = clicktest.CliRunner() # result = test.invoke( # curator.cli, # [ # '--config', self.args['configfile'], # self.args['actionfile'] # ], # ) # self.assertEquals(365, len(self.client.indices.get_alias(name=alias))) def test_no_add_remove(self): alias = 'testalias' self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.alias_no_add_remove.format(alias)) self.create_index('my_index') self.create_index('dummy') test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertEqual(1, result.exit_code) def test_no_alias(self): somevar = 'testalias' self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.alias_no_alias) self.create_index('my_index') self.create_index('dummy') test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertEqual(-1, result.exit_code) def test_extra_options(self): somevar = 'testalias' self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.bad_option_proto_test.format('alias')) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertEqual(-1, result.exit_code) curator-4.2.5/test/integration/test_allocation.py000066400000000000000000000133051302700154000222330ustar00rootroot00000000000000import elasticsearch import curator import os import json import string, random, tempfile import click from click import testing as clicktest from mock import patch, Mock from . import CuratorTestCase from . import testvars as testvars import logging logger = logging.getLogger(__name__) host, port = os.environ.get('TEST_ES_SERVER', 'localhost:9200').split(':') port = int(port) if port else 9200 class TestCLIAllocation(CuratorTestCase): def test_include(self): key = 'tag' value = 'value' at = 'include' self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.allocation_test.format(key, value, at)) self.create_index('my_index') self.create_index('not_my_index') test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertEquals(value, self.client.indices.get_settings(index='my_index')['my_index']['settings']['index']['routing']['allocation'][at][key]) self.assertNotIn('routing', self.client.indices.get_settings(index='not_my_index')['not_my_index']['settings']['index']) def test_require(self): key = 'tag' value = 'value' at = 'require' self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.allocation_test.format(key, value, at)) self.create_index('my_index') self.create_index('not_my_index') test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertEquals(value, self.client.indices.get_settings(index='my_index')['my_index']['settings']['index']['routing']['allocation'][at][key]) self.assertNotIn('routing', self.client.indices.get_settings(index='not_my_index')['not_my_index']['settings']['index']) def test_exclude(self): key = 'tag' value = 'value' at = 'exclude' self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.allocation_test.format(key, value, at)) self.create_index('my_index') self.create_index('not_my_index') test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertEquals(value, self.client.indices.get_settings(index='my_index')['my_index']['settings']['index']['routing']['allocation'][at][key]) self.assertNotIn('routing', self.client.indices.get_settings(index='not_my_index')['not_my_index']['settings']['index']) def test_invalid_allocation_type(self): key = 'tag' value = 'value' at = 'invalid' self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.allocation_test.format(key, value, at)) self.create_index('my_index') self.create_index('not_my_index') test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertEqual(-1, result.exit_code) def test_extra_option(self): key = 'tag' value = 'value' at = 'invalid' self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.bad_option_proto_test.format('allocation')) self.create_index('my_index') self.create_index('not_my_index') test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertEqual(-1, result.exit_code) def test_skip_closed(self): key = 'tag' value = 'value' at = 'include' self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.allocation_test.format(key, value, at)) self.create_index('my_index') self.client.indices.close(index='my_index') self.create_index('not_my_index') test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertNotIn('routing', self.client.indices.get_settings(index='my_index')['my_index']['settings']['index']) self.assertNotIn('routing', self.client.indices.get_settings(index='not_my_index')['not_my_index']['settings']['index']) curator-4.2.5/test/integration/test_cli.py000066400000000000000000000223251302700154000206570ustar00rootroot00000000000000import elasticsearch import curator import os import json import string, random, tempfile import click from click import testing as clicktest from mock import patch, Mock from . import CuratorTestCase from . import testvars as testvars import logging logger = logging.getLogger(__name__) host, port = os.environ.get('TEST_ES_SERVER', 'localhost:9200').split(':') port = int(port) if port else 9200 class TestCLIMethods(CuratorTestCase): def test_bad_client_config(self): self.create_indices(10) self.write_config( self.args['configfile'], testvars.bad_client_config.format(host, port) ) self.write_config(self.args['actionfile'], testvars.disabled_proto.format('close', 'delete_indices')) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], '--dry-run', self.args['actionfile'] ], ) self.assertEqual(-1, result.exit_code) def test_no_config(self): # This test checks whether localhost:9200 is provided if no hosts or # port are in the configuration. But in testing, sometimes # TEST_ES_SERVER is set to something other than localhost:9200. In this # case, the test here would fail. The if statement at the end now # compensates. See https://github.com/elastic/curator/issues/843 localtest = False if (host == 'localhost' or host == '127.0.0.1') and \ port == 9200: localtest = True self.create_indices(10) self.write_config( self.args['configfile'], ' \n' ) self.write_config(self.args['actionfile'], testvars.disabled_proto.format('close', 'delete_indices')) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], '--dry-run', self.args['actionfile'] ], ) if localtest: self.assertEqual(0, result.exit_code) else: self.assertEqual(-1, result.exit_code) def test_no_logging_config(self): self.create_indices(10) self.write_config( self.args['configfile'], testvars.no_logging_config.format(host, port) ) self.write_config(self.args['actionfile'], testvars.disabled_proto.format('close', 'delete_indices')) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], '--dry-run', self.args['actionfile'] ], ) self.assertEqual(0, result.exit_code) def test_logging_none(self): self.create_indices(10) self.write_config( self.args['configfile'], testvars.none_logging_config.format(host, port) ) self.write_config(self.args['actionfile'], testvars.disabled_proto.format('close', 'delete_indices')) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], '--dry-run', self.args['actionfile'] ], ) self.assertEqual(0, result.exit_code) def test_invalid_action(self): self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.optionless_proto.format('invalid_action')) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertEqual(-1, result.exit_code) def test_action_is_None(self): self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.optionless_proto.format(' ')) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertEqual( type(curator.ConfigurationError()), type(result.exception)) def test_no_action(self): self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.actionless_proto) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertEqual( type(curator.ConfigurationError()), type(result.exception)) def test_dry_run(self): self.create_indices(10) self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.delete_proto.format( 'age', 'name', 'older', '\'%Y.%m.%d\'', 'days', 5, ' ', ' ', ' ' ) ) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], '--dry-run', self.args['actionfile'] ], ) self.assertEquals(10, len(curator.get_indices(self.client))) def test_action_disabled(self): self.create_indices(10) self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.disabled_proto.format('close', 'delete_indices')) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertEquals(0, len(curator.get_indices(self.client))) self.assertEqual(0, result.exit_code) # I'll have to think up another way to create an exception. # The exception that using "alias" created, a missing argument, # is caught too early for this to actually run the test now :/ # def test_continue_if_exception(self): name = 'log1' self.create_index(name) self.create_index('log2') self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.continue_proto.format( name, True, 'delete_indices', False ) ) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertEquals(0, len(curator.get_indices(self.client))) self.assertEqual(0, result.exit_code) def test_continue_if_exception_False(self): name = 'log1' self.create_index(name) self.create_index('log2') self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.continue_proto.format( name, False, 'delete_indices', False ) ) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertEquals(2, len(curator.get_indices(self.client))) self.assertEqual(1, result.exit_code) def test_no_options_in_action(self): self.create_indices(10) self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.no_options_proto.format('delete_indices')) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], '--dry-run', self.args['actionfile'] ], ) self.assertEqual(0, result.exit_code) curator-4.2.5/test/integration/test_close.py000066400000000000000000000125551302700154000212210ustar00rootroot00000000000000import elasticsearch import curator import os import json import string, random, tempfile import click from click import testing as clicktest from mock import patch, Mock from . import CuratorTestCase from . import testvars as testvars import logging logger = logging.getLogger(__name__) host, port = os.environ.get('TEST_ES_SERVER', 'localhost:9200').split(':') port = int(port) if port else 9200 class TestCLIClose(CuratorTestCase): def test_close_opened(self): self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.optionless_proto.format('close')) self.create_index('my_index') self.create_index('dummy') test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertEquals( 'close', self.client.cluster.state( index='my_index', metric='metadata', )['metadata']['indices']['my_index']['state'] ) self.assertNotEqual( 'close', self.client.cluster.state( index='dummy', metric='metadata', )['metadata']['indices']['dummy']['state'] ) def test_close_closed(self): self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.optionless_proto.format('close')) self.create_index('my_index') self.client.indices.close( index='my_index', ignore_unavailable=True) self.create_index('dummy') test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertEquals( 'close', self.client.cluster.state( index='my_index', metric='metadata', )['metadata']['indices']['my_index']['state'] ) self.assertNotEqual( 'close', self.client.cluster.state( index='dummy', metric='metadata', )['metadata']['indices']['dummy']['state'] ) def test_close_delete_aliases(self): # Create aliases first alias = 'testalias' index = 'my_index' self.create_index(index) self.create_index('dummy') self.create_index('my_other') self.client.indices.put_alias(index='my_index,dummy', name=alias) self.assertEquals( { "dummy":{"aliases":{"testalias":{}}}, "my_index":{"aliases":{"testalias":{}}} }, self.client.indices.get_alias(name=alias) ) # Now close `index` with delete_aliases=True (dummy stays open) self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.close_delete_aliases) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertEquals( 'close', self.client.cluster.state( index=index, metric='metadata', )['metadata']['indices'][index]['state'] ) self.assertEquals( 'close', self.client.cluster.state( index='my_other', metric='metadata', )['metadata']['indices']['my_other']['state'] ) # Now open the indices and verify that the alias is still gone. self.client.indices.open(index=index) self.assertEquals( {"dummy":{"aliases":{"testalias":{}}}}, self.client.indices.get_alias(name=alias) ) def test_extra_option(self): self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.bad_option_proto_test.format('close')) self.create_index('my_index') self.create_index('dummy') test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertNotEqual( 'close', self.client.cluster.state( index='my_index', metric='metadata', )['metadata']['indices']['my_index']['state'] ) self.assertNotEqual( 'close', self.client.cluster.state( index='dummy', metric='metadata', )['metadata']['indices']['dummy']['state'] ) self.assertEqual(-1, result.exit_code) curator-4.2.5/test/integration/test_clusterrouting.py000066400000000000000000000035341302700154000232020ustar00rootroot00000000000000import elasticsearch import curator import os import json import string, random, tempfile import click from click import testing as clicktest from mock import patch, Mock from . import CuratorTestCase from . import testvars as testvars import logging logger = logging.getLogger(__name__) host, port = os.environ.get('TEST_ES_SERVER', 'localhost:9200').split(':') port = int(port) if port else 9200 class TestCLIClusterRouting(CuratorTestCase): def test_allocation_all(self): routing_type = 'allocation' value = 'all' self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.cluster_routing_test.format(routing_type, value)) self.create_index('my_index') self.create_index('not_my_index') test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertEquals(testvars.CRA_all, self.client.cluster.get_settings()) def test_extra_option(self): self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.bad_option_proto_test.format('cluster_routing')) self.create_index('my_index') self.create_index('not_my_index') test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertEqual(-1, result.exit_code) curator-4.2.5/test/integration/test_create_index.py000066400000000000000000000063651302700154000225500ustar00rootroot00000000000000import elasticsearch import curator import os import json import string, random, tempfile import click from click import testing as clicktest from mock import patch, Mock from . import CuratorTestCase from . import testvars as testvars import logging logger = logging.getLogger(__name__) host, port = os.environ.get('TEST_ES_SERVER', 'localhost:9200').split(':') port = int(port) if port else 9200 class TestCLICreateIndex(CuratorTestCase): def test_plain(self): self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.create_index.format('testing')) self.assertEqual([], curator.get_indices(self.client)) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertEqual(['testing'], curator.get_indices(self.client)) def test_with_extra_settings(self): self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.create_index_with_extra_settings.format('testing')) self.assertEqual([], curator.get_indices(self.client)) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) ilo = curator.IndexList(self.client) self.assertEqual(['testing'], ilo.indices) self.assertEqual(ilo.index_info['testing']['number_of_shards'], '1') self.assertEqual(ilo.index_info['testing']['number_of_replicas'], '0') def test_with_date(self): self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.create_index.format('testing-%Y.%m.%d')) self.assertEqual([], curator.get_indices(self.client)) name = curator.parse_date_pattern('testing-%Y.%m.%d') test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertEqual([name], curator.get_indices(self.client)) def test_extra_option(self): self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.bad_option_proto_test.format('create_index')) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertEqual([], curator.get_indices(self.client)) self.assertEqual(-1, result.exit_code) curator-4.2.5/test/integration/test_delete_indices.py000066400000000000000000000162601302700154000230510ustar00rootroot00000000000000import elasticsearch import curator import os import json import string, random, tempfile import time from click import testing as clicktest from mock import patch, Mock import unittest from . import CuratorTestCase from . import testvars as testvars import logging logger = logging.getLogger(__name__) host, port = os.environ.get('TEST_ES_SERVER', 'localhost:9200').split(':') port = int(port) if port else 9200 # ' - filtertype: {0}\n' # ' source: {1}\n' # ' direction: {2}\n' # ' timestring: {3}\n' # ' unit: {4}\n' # ' unit_count: {5}\n' # ' field: {6}\n' # ' stats_result: {7}\n' # ' epoch: {8}\n') global_client = elasticsearch.Elasticsearch(host=host, port=port) class TestCLIDeleteIndices(CuratorTestCase): def test_name_older_than_now(self): self.create_indices(10) self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.delete_proto.format( 'age', 'name', 'older', '\'%Y.%m.%d\'', 'days', 5, ' ', ' ', ' ' ) ) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertEquals(5, len(curator.get_indices(self.client))) def test_creation_date_newer_than_epoch(self): self.create_indices(10) self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.delete_proto.format( 'age', 'creation_date', 'younger', ' ', 'seconds', 0, ' ', ' ', int(time.time()) - 60 ) ) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertEquals(0, len(curator.get_indices(self.client))) def test_empty_list(self): self.create_indices(10) self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.delete_proto.format( 'age', 'creation_date', 'older', ' ', 'days', 90, ' ', ' ', int(time.time()) ) ) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertEquals(10, len(curator.get_indices(self.client))) self.assertEqual(1, result.exit_code) def test_ignore_empty_list(self): self.create_indices(10) self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.delete_ignore_proto.format( 'age', 'creation_date', 'older', ' ', 'days', 90, ' ', ' ', int(time.time()) ) ) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertEquals(10, len(curator.get_indices(self.client))) self.assertEqual(0, result.exit_code) def test_extra_options(self): self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.bad_option_proto_test.format('delete_indices')) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertEqual(-1, result.exit_code) class TestCLIFixFor687(CuratorTestCase): @unittest.skipIf(curator.get_version(global_client) >= (3, 0, 0), 'not supported for this version of ES') def test_fix_for_687(self): self.create_repository() snap_name = 'test687' # 7 june (week 23) self.client.indices.create( index='logstash-2016.23', body={ 'settings': { 'creation_date': 1465293737000, 'number_of_shards': 1, 'number_of_replicas': 0 } } ) # 14 june (week 24) self.client.indices.create( index='logstash-2016.24', body={ 'settings': { 'creation_date': 1465898537000, 'number_of_shards': 1, 'number_of_replicas': 0 } } ) # 21 june (week 25) self.client.indices.create( index='logstash-2016.25', body={ 'settings': { 'creation_date': 1466503337000, 'number_of_shards': 1, 'number_of_replicas': 0 } } ) # 28 july (week 26) self.client.indices.create( index='logstash-2016.26', body={ 'settings': { 'creation_date': 1467108137000, 'number_of_shards': 1, 'number_of_replicas': 0 } } ) # 5 july (week 27) self.client.indices.create( index='logstash-2016.27', body={ 'settings': { 'creation_date': 1467712937000, 'number_of_shards': 1, 'number_of_replicas': 0 } } ) # 12 july (week 28) self.client.indices.create( index='logstash-2016.28', body={ 'settings': { 'creation_date': 1468317737000, 'number_of_shards': 1, 'number_of_replicas': 0 } } ) self.client.cluster.health(wait_for_status='yellow') self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.test_687.format(self.args['repository'], snap_name)) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) snapshot = curator.get_snapshot( self.client, self.args['repository'], '_all' ) self.assertEquals(6, len(curator.get_indices(self.client))) curator-4.2.5/test/integration/test_delete_snapshots.py000066400000000000000000000067121302700154000234560ustar00rootroot00000000000000import elasticsearch import curator import os import json import string, random, tempfile import time from click import testing as clicktest from mock import patch, Mock from . import CuratorTestCase from . import testvars as testvars import logging logger = logging.getLogger(__name__) host, port = os.environ.get('TEST_ES_SERVER', 'localhost:9200').split(':') port = int(port) if port else 9200 # ' repository: {0}\n' # ' - filtertype: {1}\n' # ' source: {2}\n' # ' direction: {3}\n' # ' timestring: {4}\n' # ' unit: {5}\n' # ' unit_count: {6}\n' # ' epoch: {7}\n') class TestCLIDeleteSnapshots(CuratorTestCase): def test_deletesnapshot(self): ### Create snapshots to delete and verify them self.create_repository() timestamps = [] for i in range(1,4): self.add_docs('my_index{0}'.format(i)) ilo = curator.IndexList(self.client) snap = curator.Snapshot(ilo, repository=self.args['repository'], name='curator-%Y%m%d%H%M%S' ) snap.do_action() snapshot = curator.get_snapshot( self.client, self.args['repository'], '_all' ) self.assertEqual(i, len(snapshot['snapshots'])) time.sleep(1.0) timestamps.append(int(time.time())) time.sleep(1.0) ### Setup the actual delete self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.delete_snap_proto.format( self.args['repository'], 'age', 'creation_date', 'older', ' ', 'seconds', '0', timestamps[0] ) ) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) snapshot = curator.get_snapshot( self.client, self.args['repository'], '_all' ) self.assertEqual(2, len(snapshot['snapshots'])) def test_no_repository(self): self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.delete_snap_proto.format( ' ', 'age', 'creation_date', 'older', ' ', 'seconds', '0', ' ' ) ) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertEqual(-1, result.exit_code) def test_extra_options(self): self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.bad_option_proto_test.format('delete_snapshots')) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertEqual(-1, result.exit_code) curator-4.2.5/test/integration/test_envvars.py000066400000000000000000000055621302700154000216000ustar00rootroot00000000000000import elasticsearch import curator import os import json import string, random, tempfile import click from click import testing as clicktest from mock import patch, Mock from . import CuratorTestCase from . import testvars as testvars import logging logger = logging.getLogger(__name__) host, port = os.environ.get('TEST_ES_SERVER', 'localhost:9200').split(':') port = int(port) if port else 9200 def random_envvar(size): return ''.join( random.SystemRandom().choice( string.ascii_uppercase + string.digits ) for _ in range(size) ) class TestEnvVars(CuratorTestCase): def test_present(self): evar = random_envvar(8) os.environ[evar] = "1234" dollar = '${' + evar + '}' self.write_config( self.args['configfile'], testvars.client_config_envvars.format(dollar, port, 30) ) cfg = curator.get_yaml(self.args['configfile']) self.assertEqual( cfg['client']['hosts'], os.environ.get(evar) ) del os.environ[evar] def test_not_present(self): evar = random_envvar(8) dollar = '${' + evar + '}' self.write_config( self.args['configfile'], testvars.client_config_envvars.format(dollar, port, 30) ) cfg = curator.get_yaml(self.args['configfile']) self.assertIsNone(cfg['client']['hosts']) def test_not_present_with_default(self): evar = random_envvar(8) default = random_envvar(8) dollar = '${' + evar + ':' + default + '}' self.write_config( self.args['configfile'], testvars.client_config_envvars.format(dollar, port, 30) ) cfg = curator.get_yaml(self.args['configfile']) self.assertEqual( cfg['client']['hosts'], default ) def test_do_something_with_int_value(self): self.create_indices(10) evar = random_envvar(8) os.environ[evar] = "1234" dollar = '${' + evar + '}' self.write_config( self.args['configfile'], testvars.client_config_envvars.format(host, port, dollar) ) cfg = curator.get_yaml(self.args['configfile']) self.assertEqual( cfg['client']['timeout'], os.environ.get(evar) ) self.write_config(self.args['actionfile'], testvars.delete_proto.format( 'age', 'name', 'older', '\'%Y.%m.%d\'', 'days', 5, ' ', ' ', ' ' ) ) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertEquals(5, len(curator.get_indices(self.client))) del os.environ[evar] curator-4.2.5/test/integration/test_es_repo_mgr.py000066400000000000000000000122451302700154000224110ustar00rootroot00000000000000import elasticsearch import curator import os import json import click import string, random, tempfile from click import testing as clicktest from mock import patch, Mock, MagicMock from . import CuratorTestCase from . import testvars as testvars import logging logger = logging.getLogger(__name__) host, port = os.environ.get('TEST_ES_SERVER', 'localhost:9200').split(':') port = int(port) if port else 9200 class TestLoggingModules(CuratorTestCase): def test_logger_without_null_handler(self): mock = Mock() modules = {'logger': mock, 'logger.NullHandler': mock.module} self.write_config( self.args['configfile'], testvars.client_conf_logfile.format(host, port, os.devnull) ) with patch.dict('sys.modules', modules): self.create_repository() test = clicktest.CliRunner() result = test.invoke( curator.repo_mgr_cli, [ '--config', self.args['configfile'], 'show' ] ) self.assertEqual(self.args['repository'], result.output.rstrip()) class TestCLIRepositoryCreate(CuratorTestCase): def test_create_fs_repository_success(self): self.write_config( self.args['configfile'], testvars.client_conf_logfile.format(host, port, os.devnull) ) test = clicktest.CliRunner() result = test.invoke( curator.repo_mgr_cli, [ '--config', self.args['configfile'], 'create', 'fs', '--repository', self.args['repository'], '--location', self.args['location'] ] ) self.assertTrue(1, len(self.client.snapshot.get_repository(repository=self.args['repository']))) self.assertEqual(0, result.exit_code) def test_create_fs_repository_fail(self): self.write_config( self.args['configfile'], testvars.client_conf_logfile.format(host, port, os.devnull) ) test = clicktest.CliRunner() result = test.invoke( curator.repo_mgr_cli, [ '--config', self.args['configfile'], 'create', 'fs', '--repository', self.args['repository'], '--location', os.devnull ] ) self.assertEqual(1, result.exit_code) def test_create_s3_repository_fail(self): self.write_config( self.args['configfile'], testvars.client_conf_logfile.format(host, port, os.devnull) ) test = clicktest.CliRunner() result = test.invoke( curator.repo_mgr_cli, [ '--config', self.args['configfile'], 'create', 's3', '--bucket', 'mybucket', '--repository', self.args['repository'], ] ) self.assertEqual(1, result.exit_code) class TestCLIDeleteRepository(CuratorTestCase): def test_delete_repository_success(self): self.create_repository() self.write_config( self.args['configfile'], testvars.client_conf_logfile.format(host, port, os.devnull) ) test = clicktest.CliRunner() result = test.invoke( curator.repo_mgr_cli, [ '--config', self.args['configfile'], 'delete', '--yes', # This ensures no prompting will happen '--repository', self.args['repository'] ] ) self.assertFalse( curator.get_repository(self.client, self.args['repository']) ) def test_delete_repository_notfound(self): self.write_config( self.args['configfile'], testvars.client_conf_logfile.format(host, port, os.devnull) ) test = clicktest.CliRunner() result = test.invoke( curator.repo_mgr_cli, [ '--config', self.args['configfile'], 'delete', '--yes', # This ensures no prompting will happen '--repository', self.args['repository'] ] ) self.assertEqual(1, result.exit_code) class TestCLIShowRepositories(CuratorTestCase): def test_show_repository(self): self.create_repository() self.write_config( self.args['configfile'], testvars.client_conf_logfile.format(host, port, os.devnull) ) test = clicktest.CliRunner() result = test.invoke( curator.repo_mgr_cli, [ '--config', self.args['configfile'], 'show' ] ) self.assertEqual(self.args['repository'], result.output.rstrip()) curator-4.2.5/test/integration/test_forcemerge.py000066400000000000000000000036121302700154000222240ustar00rootroot00000000000000import elasticsearch import curator import os import json import string, random, tempfile import click from click import testing as clicktest from mock import patch, Mock from . import CuratorTestCase from . import testvars as testvars import logging logger = logging.getLogger(__name__) host, port = os.environ.get('TEST_ES_SERVER', 'localhost:9200').split(':') port = int(port) if port else 9200 class TestCLIforceMerge(CuratorTestCase): def test_merge(self): count = 1 idx = 'my_index' self.create_index(idx) self.add_docs(idx) ilo1 = curator.IndexList(self.client) ilo1._get_segmentcounts() self.assertEqual(3, ilo1.index_info[idx]['segments']) self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.forcemerge_test.format(count, 0.20)) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) ilo2 = curator.IndexList(self.client) ilo2._get_segmentcounts() self.assertEqual(count, ilo2.index_info[idx]['segments']) def test_extra_option(self): self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.bad_option_proto_test.format('forcemerge')) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertEqual(-1, result.exit_code) curator-4.2.5/test/integration/test_integrations.py000066400000000000000000000055061302700154000226200ustar00rootroot00000000000000import elasticsearch import curator import os import json import string, random, tempfile import click from click import testing as clicktest from mock import patch, Mock from . import CuratorTestCase from . import testvars as testvars import logging logger = logging.getLogger(__name__) host, port = os.environ.get('TEST_ES_SERVER', 'localhost:9200').split(':') port = int(port) if port else 9200 class TestFilters(CuratorTestCase): def test_filter_by_alias(self): alias = 'testalias' self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.filter_by_alias.format('testalias', False)) self.create_index('my_index') self.create_index('dummy') self.client.indices.put_alias(index='dummy', name=alias) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertEquals(1, len(curator.get_indices(self.client))) def test_filter_by_array_of_aliases(self): alias = 'testalias' self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.filter_by_alias.format(' [ testalias, foo ]', False)) self.create_index('my_index') self.create_index('dummy') self.client.indices.put_alias(index='dummy', name=alias) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertEquals(1, len(curator.get_indices(self.client))) def test_filter_by_alias_bad_aliases(self): alias = 'testalias' self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.filter_by_alias.format('{"this":"isadict"}', False)) self.create_index('my_index') self.create_index('dummy') self.client.indices.put_alias(index='dummy', name=alias) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertEquals( type(curator.ConfigurationError()), type(result.exception)) self.assertEquals(2, len(curator.get_indices(self.client))) curator-4.2.5/test/integration/test_open.py000066400000000000000000000073201302700154000210470ustar00rootroot00000000000000import elasticsearch import curator import os import json import string, random, tempfile import click from click import testing as clicktest from mock import patch, Mock from . import CuratorTestCase from . import testvars as testvars import logging logger = logging.getLogger(__name__) host, port = os.environ.get('TEST_ES_SERVER', 'localhost:9200').split(':') port = int(port) if port else 9200 class TestCLIOpenClosed(CuratorTestCase): def test_open_closed(self): self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.optionless_proto.format('open')) self.create_index('my_index') self.client.indices.close( index='my_index', ignore_unavailable=True) self.create_index('dummy') test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertNotEqual( 'close', self.client.cluster.state( index='my_index', metric='metadata', )['metadata']['indices']['my_index']['state'] ) self.assertNotEqual( 'close', self.client.cluster.state( index='dummy', metric='metadata', )['metadata']['indices']['dummy']['state'] ) def test_open_opened(self): self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.optionless_proto.format('open')) self.create_index('my_index') self.create_index('dummy') test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertNotEqual( 'close', self.client.cluster.state( index='my_index', metric='metadata', )['metadata']['indices']['my_index']['state'] ) self.assertNotEqual( 'close', self.client.cluster.state( index='dummy', metric='metadata', )['metadata']['indices']['dummy']['state'] ) def test_extra_option(self): self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.bad_option_proto_test.format('open')) self.create_index('my_index') self.client.indices.close( index='my_index', ignore_unavailable=True) self.create_index('dummy') test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertEquals( 'close', self.client.cluster.state( index='my_index', metric='metadata', )['metadata']['indices']['my_index']['state'] ) self.assertNotEqual( 'close', self.client.cluster.state( index='dummy', metric='metadata', )['metadata']['indices']['dummy']['state'] ) self.assertEqual(-1, result.exit_code) curator-4.2.5/test/integration/test_replicas.py000066400000000000000000000045611302700154000217140ustar00rootroot00000000000000import elasticsearch import curator import os import json import string, random, tempfile import click from click import testing as clicktest from mock import patch, Mock from . import CuratorTestCase from . import testvars as testvars import logging logger = logging.getLogger(__name__) host, port = os.environ.get('TEST_ES_SERVER', 'localhost:9200').split(':') port = int(port) if port else 9200 class TestCLIReplicas(CuratorTestCase): def test_increase_count(self): count = 2 idx = 'my_index' self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.replicas_test.format(count)) self.create_index(idx) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertEqual( count, int(self.client.indices.get_settings( index=idx)[idx]['settings']['index']['number_of_replicas']) ) def test_no_count(self): self.create_index('foo') self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.replicas_test.format(' ')) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertEqual(-1, result.exit_code) def test_extra_option(self): self.create_index('foo') self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.bad_option_proto_test.format('replicas')) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertEqual(-1, result.exit_code) curator-4.2.5/test/integration/test_restore.py000066400000000000000000000076061302700154000216000ustar00rootroot00000000000000import elasticsearch import curator import os import json import string, random, tempfile from click import testing as clicktest from mock import patch, Mock from . import CuratorTestCase from . import testvars as testvars import logging logger = logging.getLogger(__name__) host, port = os.environ.get('TEST_ES_SERVER', 'localhost:9200').split(':') port = int(port) if port else 9200 # ' repository: {0}\n' # ' name: {1}\n' # ' indices: {2}\n' # ' include_aliases: {3}\n' # ' ignore_unavailable: {4}\n' # ' include_global_state: {5}\n' # ' partial: {6}\n' # ' rename_pattern: {7}\n' # ' rename_replacement: {8}\n' # ' extra_settings: {9}\n' # ' wait_for_completion: {10}\n' # ' skip_repo_fs_check: {11}\n' # ' timeout_override: {12}\n' class TestCLIRestore(CuratorTestCase): def test_restore(self): indices = [] for i in range(1,4): self.add_docs('my_index{0}'.format(i)) indices.append('my_index{0}'.format(i)) snap_name = 'snapshot1' self.create_snapshot(snap_name, ','.join(indices)) snapshot = curator.get_snapshot( self.client, self.args['repository'], '_all' ) self.assertEqual(1, len(snapshot['snapshots'])) self.client.indices.delete(','.join(indices)) self.assertEqual([], curator.get_indices(self.client)) self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.restore_snapshot_proto.format( self.args['repository'], snap_name, indices, False, False, True, False, ' ', ' ', ' ', True, False, 301 ) ) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) restored_indices = sorted(curator.get_indices(self.client)) self.assertEqual(indices, restored_indices) def test_restore_with_rename(self): indices = [] for i in range(1,4): self.add_docs('my_index{0}'.format(i)) indices.append('my_index{0}'.format(i)) snap_name = 'snapshot1' self.create_snapshot(snap_name, ','.join(indices)) snapshot = curator.get_snapshot( self.client, self.args['repository'], '_all' ) self.assertEqual(1, len(snapshot['snapshots'])) self.client.indices.delete(','.join(indices)) self.assertEqual([], curator.get_indices(self.client)) self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.restore_snapshot_proto.format( self.args['repository'], snap_name, indices, False, False, True, False, 'my_index(.+)', 'new_index$1', ' ', True, False, 301 ) ) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) restored_indices = sorted(curator.get_indices(self.client)) self.assertEqual( ['new_index1', 'new_index2', 'new_index3'], restored_indices ) curator-4.2.5/test/integration/test_snapshot.py000066400000000000000000000104131302700154000217420ustar00rootroot00000000000000import elasticsearch import curator import os import json import string, random, tempfile from click import testing as clicktest from mock import patch, Mock from . import CuratorTestCase from . import testvars as testvars import logging logger = logging.getLogger(__name__) host, port = os.environ.get('TEST_ES_SERVER', 'localhost:9200').split(':') port = int(port) if port else 9200 class TestCLISnapshot(CuratorTestCase): def test_snapshot(self): self.create_indices(5) self.create_repository() snap_name = 'snapshot1' self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.snapshot_test.format(self.args['repository'], snap_name)) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) snapshot = curator.get_snapshot( self.client, self.args['repository'], '_all' ) self.assertEqual(1, len(snapshot['snapshots'])) self.assertEqual(snap_name, snapshot['snapshots'][0]['snapshot']) def test_snapshot_ignore_empty_list(self): self.create_indices(5) self.create_repository() snap_name = 'snapshot1' self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.test_682.format(self.args['repository'], snap_name, True)) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) snapshot = curator.get_snapshot( self.client, self.args['repository'], '_all' ) self.assertEqual(0, len(snapshot['snapshots'])) self.assertEquals(0, len(curator.get_indices(self.client))) def test_snapshot_do_not_ignore_empty_list(self): self.create_indices(5) self.create_repository() snap_name = 'snapshot1' self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.test_682.format(self.args['repository'], snap_name, False)) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) snapshot = curator.get_snapshot( self.client, self.args['repository'], '_all' ) self.assertEqual(0, len(snapshot['snapshots'])) self.assertEquals(5, len(curator.get_indices(self.client))) def test_no_repository(self): self.create_indices(5) self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.snapshot_test.format(' ', 'snap_name')) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertEqual(-1, result.exit_code) def test_extra_option(self): self.create_indices(5) self.write_config( self.args['configfile'], testvars.client_config.format(host, port)) self.write_config(self.args['actionfile'], testvars.bad_option_proto_test.format('snapshot')) test = clicktest.CliRunner() result = test.invoke( curator.cli, [ '--config', self.args['configfile'], self.args['actionfile'] ], ) self.assertEqual(-1, result.exit_code) curator-4.2.5/test/integration/testvars.py000066400000000000000000000317531302700154000207310ustar00rootroot00000000000000import os, sys client_config = ('---\n' 'client:\n' ' hosts: {0}\n' ' port: {1}\n' ' url_prefix:\n' ' use_ssl: False\n' ' certificate:\n' ' client_cert:\n' ' client_key:\n' ' ssl_no_validate: False\n' ' http_auth: \n' ' timeout: 30\n' ' master_only: False\n' '\n' 'logging:\n' ' loglevel: DEBUG\n' ' logfile:\n' ' logformat: default\n' ' blacklist: []\n') client_conf_logfile = ('---\n' 'client:\n' ' hosts: {0}\n' ' port: {1}\n' '\n' 'logging:\n' ' loglevel: DEBUG\n' ' logfile: {2}\n') client_config_envvars = ('---\n' 'client:\n' ' hosts: {0}\n' ' port: {1}\n' ' timeout: {2}\n' ' master_only: False\n' '\n' 'logging:\n' ' loglevel: DEBUG\n' ' logfile:\n' ' logformat: default\n' ' blacklist: []\n') bad_client_config = ('---\n' 'misspelled:\n' ' hosts: {0}\n' ' port: {1}\n' ' url_prefix:\n' ' use_ssl: False\n' ' certificate:\n' ' client_cert:\n' ' client_key:\n' ' ssl_no_validate: False\n' ' http_auth: \n' ' timeout: 30\n' ' master_only: False\n') no_logging_config = ('---\n' 'client:\n' ' hosts: {0}\n' ' port: {1}\n' ' url_prefix:\n' ' use_ssl: False\n' ' certificate:\n' ' client_cert:\n' ' client_key:\n' ' ssl_no_validate: False\n' ' http_auth: \n' ' timeout: 30\n' ' master_only: False\n') none_logging_config = ('---\n' 'client:\n' ' hosts: {0}\n' ' port: {1}\n' ' url_prefix:\n' ' use_ssl: False\n' ' certificate:\n' ' client_cert:\n' ' client_key:\n' ' ssl_no_validate: False\n' ' http_auth: \n' ' timeout: 30\n' ' master_only: False\n' '\n' 'logging: \n' '\n') alias_add_only = ('---\n' 'actions:\n' ' 1:\n' ' description: "Add all indices to specified alias"\n' ' action: alias\n' ' options:\n' ' name: {0}\n' ' continue_if_exception: False\n' ' disable_action: False\n' ' add:\n' ' filters:\n' ' - filtertype: none\n') alias_add_only_with_extra_settings = ('---\n' 'actions:\n' ' 1:\n' ' description: "Add all indices to specified alias"\n' ' action: alias\n' ' options:\n' ' name: {0}\n' ' extra_settings:\n' ' filter:\n' ' term:\n' ' user: kimchy\n' ' continue_if_exception: False\n' ' disable_action: False\n' ' add:\n' ' filters:\n' ' - filtertype: none\n') alias_remove_only = ('---\n' 'actions:\n' ' 1:\n' ' description: "Remove all indices from specified alias"\n' ' action: alias\n' ' options:\n' ' name: {0}\n' ' continue_if_exception: False\n' ' disable_action: False\n' ' remove:\n' ' filters:\n' ' - filtertype: none\n') alias_add_remove = ('---\n' 'actions:\n' ' 1:\n' ' description: "Add/remove specified indices from designated alias"\n' ' action: alias\n' ' options:\n' ' name: {0}\n' ' continue_if_exception: False\n' ' disable_action: False\n' ' remove:\n' ' filters:\n' ' - filtertype: pattern\n' ' kind: prefix\n' ' value: du\n' ' add:\n' ' filters:\n' ' - filtertype: pattern\n' ' kind: prefix\n' ' value: my\n') alias_no_add_remove = ('---\n' 'actions:\n' ' 1:\n' ' description: "No add or remove should raise an exception"\n' ' action: alias\n' ' options:\n' ' name: {0}\n' ' continue_if_exception: False\n' ' disable_action: False\n') alias_no_alias = ('---\n' 'actions:\n' ' 1:\n' ' description: "Removing alias from options should result in an exception"\n' ' action: alias\n' ' options:\n' ' continue_if_exception: False\n' ' disable_action: False\n' ' remove:\n' ' filters:\n' ' - filtertype: pattern\n' ' kind: prefix\n' ' value: du\n' ' add:\n' ' filters:\n' ' - filtertype: pattern\n' ' kind: prefix\n' ' value: my\n') allocation_test = ('---\n' 'actions:\n' ' 1:\n' ' description: "Allocate by key/value/allocation_type"\n' ' action: allocation\n' ' options:\n' ' key: {0}\n' ' value: {1}\n' ' allocation_type: {2}\n' ' continue_if_exception: False\n' ' disable_action: False\n' ' filters:\n' ' - filtertype: pattern\n' ' kind: prefix\n' ' value: my\n') cluster_routing_test = ('---\n' 'actions:\n' ' 1:\n' ' description: "Alter cluster routing by routing_type/value"\n' ' action: cluster_routing\n' ' options:\n' ' routing_type: {0}\n' ' value: {1}\n' ' setting: enable\n') optionless_proto = ('---\n' 'actions:\n' ' 1:\n' ' description: "Act on indices as filtered"\n' ' action: {0}\n' ' options:\n' ' continue_if_exception: False\n' ' disable_action: False\n' ' filters:\n' ' - filtertype: pattern\n' ' kind: prefix\n' ' value: my\n') no_options_proto = ('---\n' 'actions:\n' ' 1:\n' ' description: "Act on indices as filtered"\n' ' action: {0}\n' ' filters:\n' ' - filtertype: pattern\n' ' kind: prefix\n' ' value: my\n') actionless_proto = ('---\n' 'actions:\n' ' 1:\n' ' options:\n' ' continue_if_exception: False\n' ' disable_action: False\n') disabled_proto = ('---\n' 'actions:\n' ' 1:\n' ' description: "Act on indices as filtered"\n' ' action: {0}\n' ' options:\n' ' continue_if_exception: False\n' ' ignore_empty_list: True\n' ' disable_action: True\n' ' filters:\n' ' - filtertype: pattern\n' ' kind: prefix\n' ' value: my\n' ' 2:\n' ' description: "Act on indices as filtered"\n' ' action: {1}\n' ' options:\n' ' continue_if_exception: False\n' ' ignore_empty_list: True\n' ' disable_action: False\n' ' filters:\n' ' - filtertype: pattern\n' ' kind: prefix\n' ' value: log\n') continue_proto = ('---\n' 'actions:\n' ' 1:\n' ' description: "Create named index"\n' ' action: create_index\n' ' options:\n' ' name: {0}\n' ' continue_if_exception: {1}\n' ' disable_action: False\n' ' 2:\n' ' description: "Act on indices as filtered"\n' ' action: {2}\n' ' options:\n' ' continue_if_exception: {3}\n' ' disable_action: False\n' ' filters:\n' ' - filtertype: pattern\n' ' kind: prefix\n' ' value: log\n') close_delete_aliases = ('---\n' 'actions:\n' ' 1:\n' ' description: "Close indices as filtered"\n' ' action: close\n' ' options:\n' ' delete_aliases: True\n' ' continue_if_exception: False\n' ' disable_action: False\n' ' filters:\n' ' - filtertype: pattern\n' ' kind: prefix\n' ' value: my\n') delete_proto = ('---\n' 'actions:\n' ' 1:\n' ' description: "Delete indices as filtered"\n' ' action: delete_indices\n' ' options:\n' ' continue_if_exception: False\n' ' disable_action: False\n' ' filters:\n' ' - filtertype: {0}\n' ' source: {1}\n' ' direction: {2}\n' ' timestring: {3}\n' ' unit: {4}\n' ' unit_count: {5}\n' ' field: {6}\n' ' stats_result: {7}\n' ' epoch: {8}\n') delete_ignore_proto = ('---\n' 'actions:\n' ' 1:\n' ' description: "Delete indices as filtered"\n' ' action: delete_indices\n' ' options:\n' ' ignore_empty_list: True\n' ' continue_if_exception: False\n' ' disable_action: False\n' ' filters:\n' ' - filtertype: {0}\n' ' source: {1}\n' ' direction: {2}\n' ' timestring: {3}\n' ' unit: {4}\n' ' unit_count: {5}\n' ' field: {6}\n' ' stats_result: {7}\n' ' epoch: {8}\n') filter_by_alias = ('---\n' 'actions:\n' ' 1:\n' ' description: "Delete indices as filtered"\n' ' action: delete_indices\n' ' options:\n' ' ignore_empty_list: True\n' ' continue_if_exception: False\n' ' disable_action: False\n' ' filters:\n' ' - filtertype: alias\n' ' aliases: {0}\n' ' exclude: {1}\n') bad_option_proto_test = ('---\n' 'actions:\n' ' 1:\n' ' description: "Should raise exception due to extra option"\n' ' action: {0}\n' ' options:\n' ' invalid: this_should_not_be_here\n' ' continue_if_exception: False\n' ' disable_action: False\n' ' filters:\n' ' - filtertype: none\n') replicas_test = ('---\n' 'actions:\n' ' 1:\n' ' description: "Increase replica count to provided value"\n' ' action: replicas\n' ' options:\n' ' count: {0}\n' ' continue_if_exception: False\n' ' disable_action: False\n' ' filters:\n' ' - filtertype: pattern\n' ' kind: prefix\n' ' value: my\n') forcemerge_test = ('---\n' 'actions:\n' ' 1:\n' ' description: "forceMerge segment count per shard to provided value with optional delay"\n' ' action: forcemerge\n' ' options:\n' ' max_num_segments: {0}\n' ' delay: {1}\n' ' timeout_override: 300\n' ' continue_if_exception: False\n' ' disable_action: False\n' ' filters:\n' ' - filtertype: pattern\n' ' kind: prefix\n' ' value: my\n') snapshot_test = ('---\n' 'actions:\n' ' 1:\n' ' description: "Snapshot selected indices"\n' ' action: snapshot\n' ' options:\n' ' repository: {0}\n' ' name: {1}\n' ' continue_if_exception: False\n' ' disable_action: False\n' ' filters:\n' ' - filtertype: none\n') delete_snap_proto = ('---\n' 'actions:\n' ' 1:\n' ' description: "Delete snapshots as filtered"\n' ' action: delete_snapshots\n' ' options:\n' ' repository: {0}\n' ' continue_if_exception: False\n' ' disable_action: False\n' ' filters:\n' ' - filtertype: {1}\n' ' source: {2}\n' ' direction: {3}\n' ' timestring: {4}\n' ' unit: {5}\n' ' unit_count: {6}\n' ' epoch: {7}\n') create_index = ('---\n' 'actions:\n' ' 1:\n' ' description: "Create index as named"\n' ' action: create_index\n' ' options:\n' ' name: {0}\n' ' continue_if_exception: False\n' ' disable_action: False\n') create_index_with_extra_settings = ('---\n' 'actions:\n' ' 1:\n' ' description: "Create index as named with extra settings"\n' ' action: create_index\n' ' options:\n' ' name: {0}\n' ' extra_settings:\n' ' number_of_shards: 1\n' ' number_of_replicas: 0\n' ' continue_if_exception: False\n' ' disable_action: False\n') restore_snapshot_proto = ('---\n' 'actions:\n' ' 1:\n' ' description: Restore snapshot as configured\n' ' action: restore\n' ' options:\n' ' repository: {0}\n' ' name: {1}\n' ' indices: {2}\n' ' include_aliases: {3}\n' ' ignore_unavailable: {4}\n' ' include_global_state: {5}\n' ' partial: {6}\n' ' rename_pattern: {7}\n' ' rename_replacement: {8}\n' ' extra_settings: {9}\n' ' wait_for_completion: {10}\n' ' skip_repo_fs_check: {11}\n' ' timeout_override: {12}\n' ' continue_if_exception: False\n' ' disable_action: False\n' ' filters:\n' ' - filtertype: none\n') test_687 = ('---\n' 'actions:\n' ' 1:\n' ' action: snapshot\n' ' description: >-\n' ' Create a snapshot with the last week index.\n' ' options:\n' ' repository: {0}\n' ' name: {1}\n' ' ignore_unavailable: False\n' ' include_global_state: True\n' ' partial: False\n' ' wait_for_completion: True\n' ' skip_repo_fs_check: False\n' ' continue_if_exception: False\n' ' disable_action: False\n' ' filters:\n' ' - filtertype: pattern\n' ' kind: prefix\n' ' value: logstash-\n' ' exclude:\n' ' - filtertype: age\n' ' source: creation_date\n' ' direction: younger\n' ' epoch: 1467020729\n' ' unit: weeks\n' ' unit_count: 2\n' ' exclude:\n' ' - filtertype: age\n' ' source: creation_date\n' ' direction: younger\n' ' epoch: 1467020729\n' ' unit: weeks\n' ' unit_count: 1\n' ' exclude: True\n' ' 2:\n' ' action: delete_indices\n' ' description: >-\n' ' Remove indices starting with logstash- older than 5 weeks\n' ' options:\n' ' ignore_empty_list: True\n' ' continue_if_exception: False\n' ' disable_action: False\n' ' filters:\n' ' - filtertype: pattern\n' ' kind: prefix\n' ' value: logstash-\n' ' exclude:\n' ' - filtertype: age\n' ' source: creation_date\n' ' epoch: 1467020729\n' ' direction: older\n' ' unit: weeks\n' ' unit_count: 5\n' ' exclude:\n') test_682 = ('---\n' 'actions:\n' ' 1:\n' ' description: "Snapshot selected indices"\n' ' action: snapshot\n' ' options:\n' ' repository: {0}\n' ' name: {1}\n' ' ignore_empty_list: {2}\n' ' continue_if_exception: False\n' ' disable_action: False\n' ' filters:\n' ' - filtertype: pattern\n' ' kind: prefix\n' ' value: notlogstash-\n' ' exclude:\n' ' 2:\n' ' description: "Delete selected indices"\n' ' action: delete_indices\n' ' options:\n' ' continue_if_exception: False\n' ' disable_action: False\n' ' filters:\n' ' - filtertype: pattern\n' ' kind: prefix\n' ' value: logstash-\n' ' exclude:\n') CRA_all = { u'persistent':{}, u'transient':{u'cluster':{u'routing':{u'allocation':{u'enable':u'all'}}}} } curator-4.2.5/test/run_tests.py000077500000000000000000000013051302700154000165520ustar00rootroot00000000000000#!/usr/bin/env python from __future__ import print_function import sys from os.path import dirname, abspath import nose def run_all(argv=None): sys.exitfunc = lambda: sys.stderr.write('Shutting down....\n') # always insert coverage when running tests through setup.py if argv is None: argv = [ 'nosetests', '--with-xunit', '--logging-format=%(levelname)s %(name)22s %(funcName)22s:%(lineno)-4d %(message)s', '--with-xcoverage', '--cover-package=curator', '--cover-erase', '--verbose', ] nose.run_exit( argv=argv, defaultTest=abspath(dirname(__file__)) ) if __name__ == '__main__': run_all(sys.argv) curator-4.2.5/test/unit/000077500000000000000000000000001302700154000151275ustar00rootroot00000000000000curator-4.2.5/test/unit/__init__.py000066400000000000000000000031401302700154000172360ustar00rootroot00000000000000import os import shutil import tempfile import random import string from unittest import SkipTest, TestCase from mock import Mock from .testvars import * class CLITestCase(TestCase): def setUp(self): super(CLITestCase, self).setUp() self.args = {} dirname = ''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(8)) ymlname = ''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(8)) badyaml = ''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(8)) # This will create a psuedo-random temporary directory on the machine # which runs the unit tests, but NOT on the machine where elasticsearch # is running. This means tests may fail if run against remote instances # unless you explicitly set `self.args['location']` to a proper spot # on the target machine. self.args['tmpdir'] = tempfile.mkdtemp(suffix=dirname) if not os.path.exists(self.args['tmpdir']): os.makedirs(self.args['tmpdir']) self.args['yamlfile'] = os.path.join(self.args['tmpdir'], ymlname) self.args['invalid_yaml'] = os.path.join(self.args['tmpdir'], badyaml) self.args['no_file_here'] = os.path.join(self.args['tmpdir'], 'not_created') with open(self.args['yamlfile'], 'w') as f: f.write(testvars.yamlconfig) with open(self.args['invalid_yaml'], 'w') as f: f.write('gobbledeygook: @failhere\n') def tearDown(self): if os.path.exists(self.args['tmpdir']): shutil.rmtree(self.args['tmpdir']) curator-4.2.5/test/unit/test_action_alias.py000066400000000000000000000153351302700154000211750ustar00rootroot00000000000000from unittest import TestCase from mock import Mock, patch import elasticsearch import curator # Get test variables and constants from a single source from . import testvars as testvars class TestActionAlias(TestCase): def test_init_raise(self): self.assertRaises(curator.MissingArgument, curator.Alias) def test_add_raises_on_missing_parameter(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one ilo = curator.IndexList(client) ao = curator.Alias(name='alias') self.assertRaises(TypeError, ao.add) def test_add_raises_on_invalid_parameter(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one ilo = curator.IndexList(client) ao = curator.Alias(name='alias') self.assertRaises(TypeError, ao.add, []) def test_add_single(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one ilo = curator.IndexList(client) ao = curator.Alias(name='alias') ao.add(ilo) self.assertEqual(testvars.alias_one_add, ao.actions) def test_add_single_with_extra_settings(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one ilo = curator.IndexList(client) esd = { 'filter' : { 'term' : { 'user' : 'kimchy' } } } ao = curator.Alias(name='alias', extra_settings=esd) ao.add(ilo) self.assertEqual(testvars.alias_one_add_with_extras, ao.actions) def test_remove_single(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one ilo = curator.IndexList(client) ao = curator.Alias(name='alias') ao.remove(ilo) self.assertEqual(testvars.alias_one_rm, ao.actions) def test_add_multiple(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two ilo = curator.IndexList(client) ao = curator.Alias(name='alias') ao.add(ilo) cmp = sorted(ao.actions, key=lambda k: k['add']['index']) self.assertEqual(testvars.alias_two_add, cmp) def test_remove_multiple(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two ilo = curator.IndexList(client) ao = curator.Alias(name='alias') ao.remove(ilo) cmp = sorted(ao.actions, key=lambda k: k['remove']['index']) self.assertEqual(testvars.alias_two_rm, cmp) def test_show_body(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one ilo = curator.IndexList(client) ao = curator.Alias(name='alias') ao.remove(ilo) ao.add(ilo) body = ao.body() self.assertEqual( testvars.alias_one_body['actions'][0], body['actions'][0]) self.assertEqual( testvars.alias_one_body['actions'][1], body['actions'][1]) def test_raise_on_empty_body(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one ilo = curator.IndexList(client) ao = curator.Alias(name='alias') self.assertRaises(curator.ActionError, ao.body) def test_do_dry_run(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one client.indices.update_aliases.return_value = testvars.alias_success ilo = curator.IndexList(client) ao = curator.Alias(name='alias') ao.add(ilo) self.assertIsNone(ao.do_dry_run()) def test_do_action(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one client.indices.update_aliases.return_value = testvars.alias_success ilo = curator.IndexList(client) ao = curator.Alias(name='alias') ao.add(ilo) self.assertIsNone(ao.do_action()) def test_do_action_raises_exception(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one client.indices.update_aliases.return_value = testvars.alias_success client.indices.update_aliases.side_effect = testvars.four_oh_one ilo = curator.IndexList(client) ao = curator.Alias(name='alias') ao.add(ilo) self.assertRaises(curator.FailedExecution, ao.do_action) curator-4.2.5/test/unit/test_action_allocation.py000066400000000000000000000120301302700154000222160ustar00rootroot00000000000000from unittest import TestCase from mock import Mock, patch import elasticsearch import curator # Get test variables and constants from a single source from . import testvars as testvars class TestActionAllocation(TestCase): def test_init_raise(self): self.assertRaises(TypeError, curator.Allocation, 'invalid') def test_init(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one ilo = curator.IndexList(client) ao = curator.Allocation(ilo, key='key', value='value') self.assertEqual(ilo, ao.index_list) self.assertEqual(client, ao.client) def test_create_body_no_key(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one ilo = curator.IndexList(client) self.assertRaises(curator.MissingArgument, curator.Allocation, ilo) def test_create_body_no_value(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one ilo = curator.IndexList(client) self.assertRaises(curator.MissingArgument, curator.Allocation, ilo, key='key') def test_create_body_invalid_allocation_type(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one ilo = curator.IndexList(client) self.assertRaises( ValueError, curator.Allocation, ilo, key='key', value='value', allocation_type='invalid' ) def test_create_body_valid(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one ilo = curator.IndexList(client) ao = curator.Allocation(ilo, key='key', value='value') self.assertEqual({'index.routing.allocation.require.key': 'value'}, ao.body) def test_do_action_raise_on_put_settings(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one client.indices.put_settings.return_value = None client.indices.put_settings.side_effect = testvars.fake_fail ilo = curator.IndexList(client) ao = curator.Allocation(ilo, key='key', value='value') self.assertRaises(Exception, ao.do_action) def test_do_dry_run(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one client.indices.put_settings.return_value = None ilo = curator.IndexList(client) ao = curator.Allocation(ilo, key='key', value='value') self.assertIsNone(ao.do_dry_run()) def test_do_action(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one client.indices.put_settings.return_value = None ilo = curator.IndexList(client) ao = curator.Allocation(ilo, key='key', value='value') self.assertIsNone(ao.do_action()) def test_do_action_wait(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one client.indices.put_settings.return_value = None client.cluster.health.return_value = None ilo = curator.IndexList(client) ao = curator.Allocation( ilo, key='key', value='value', wait_for_completion=True) self.assertIsNone(ao.do_action()) curator-4.2.5/test/unit/test_action_close.py000066400000000000000000000074321302700154000212100ustar00rootroot00000000000000from unittest import TestCase from mock import Mock, patch import elasticsearch import curator # Get test variables and constants from a single source from . import testvars as testvars class TestActionClose(TestCase): def test_init_raise(self): self.assertRaises(TypeError, curator.Close, 'invalid') def test_init(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one ilo = curator.IndexList(client) co = curator.Close(ilo) self.assertEqual(ilo, co.index_list) self.assertEqual(client, co.client) def test_do_dry_run(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one client.indices.flush_synced.return_value = testvars.synced_pass client.indices.close.return_value = None ilo = curator.IndexList(client) co = curator.Close(ilo) self.assertIsNone(co.do_dry_run()) def test_do_action(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one client.indices.flush_synced.return_value = testvars.synced_pass client.indices.close.return_value = None ilo = curator.IndexList(client) co = curator.Close(ilo) self.assertIsNone(co.do_action()) def test_do_action_with_delete_aliases(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one client.indices.flush_synced.return_value = testvars.synced_pass client.indices.close.return_value = None ilo = curator.IndexList(client) co = curator.Close(ilo, delete_aliases=True) self.assertIsNone(co.do_action()) def test_do_action_raises_exception(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one client.indices.flush_synced.return_value = testvars.synced_pass client.indices.close.return_value = None client.indices.close.side_effect = testvars.fake_fail ilo = curator.IndexList(client) co = curator.Close(ilo) self.assertRaises(curator.FailedExecution, co.do_action) def test_do_action_delete_aliases_with_exception(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one client.indices.flush_synced.return_value = testvars.synced_pass client.indices.close.return_value = None ilo = curator.IndexList(client) client.indices.delete_alias.side_effect = testvars.fake_fail co = curator.Close(ilo, delete_aliases=True) self.assertIsNone(co.do_action()) curator-4.2.5/test/unit/test_action_clusterrouting.py000066400000000000000000000045211302700154000231700ustar00rootroot00000000000000from unittest import TestCase from mock import Mock, patch import elasticsearch import curator # Get test variables and constants from a single source from . import testvars as testvars class TestActionAllocation(TestCase): def test_bad_client(self): self.assertRaises(TypeError, curator.ClusterRouting, 'invalid') def test_bad_setting(self): client = Mock() self.assertRaises( ValueError, curator.ClusterRouting, client, setting='invalid' ) def test_bad_routing_type(self): client = Mock() self.assertRaises( ValueError, curator.ClusterRouting, client, routing_type='invalid', setting='enable' ) def test_bad_value_with_allocation(self): client = Mock() self.assertRaises( ValueError, curator.ClusterRouting, client, routing_type='allocation', setting='enable', value='invalid' ) def test_bad_value_with_rebalance(self): client = Mock() self.assertRaises( ValueError, curator.ClusterRouting, client, routing_type='rebalance', setting='enable', value='invalid' ) def test_do_dry_run(self): client = Mock() cro = curator.ClusterRouting( client, routing_type='allocation', setting='enable', value='all' ) self.assertIsNone(cro.do_dry_run()) def test_do_action_raise_on_put_settings(self): client = Mock() client.cluster.put_settings.return_value = None client.cluster.put_settings.side_effect = testvars.fake_fail cro = curator.ClusterRouting( client, routing_type='allocation', setting='enable', value='all' ) self.assertRaises(Exception, cro.do_action) def test_do_action_wait(self): client = Mock() client.cluster.put_settings.return_value = None client.cluster.health.return_value = None cro = curator.ClusterRouting( client, routing_type='allocation', setting='enable', value='all', wait_for_completion=True ) self.assertIsNone(cro.do_action()) curator-4.2.5/test/unit/test_action_create_index.py000066400000000000000000000024431302700154000225320ustar00rootroot00000000000000from unittest import TestCase from mock import Mock, patch import elasticsearch import curator # Get test variables and constants from a single source from . import testvars as testvars class TestActionCreate_index(TestCase): def test_init_raise(self): self.assertRaises(TypeError, curator.CreateIndex, 'invalid') def test_init_raise_no_name(self): client = Mock() self.assertRaises(curator.ConfigurationError, curator.CreateIndex, client, None) def test_init(self): client = Mock() co = curator.CreateIndex(client, 'name') self.assertEqual('name', co.name) self.assertEqual(client, co.client) def test_do_dry_run(self): client = Mock() co = curator.CreateIndex(client, 'name') self.assertIsNone(co.do_dry_run()) def test_do_action(self): client = Mock() client.indices.create.return_value = None co = curator.CreateIndex(client, 'name') self.assertIsNone(co.do_action()) def test_do_action_raises_exception(self): client = Mock() client.indices.create.return_value = None client.indices.create.side_effect = testvars.fake_fail co = curator.CreateIndex(client, 'name') self.assertRaises(curator.FailedExecution, co.do_action) curator-4.2.5/test/unit/test_action_delete_indices.py000066400000000000000000000075271302700154000230500ustar00rootroot00000000000000from unittest import TestCase from mock import Mock, patch import elasticsearch import curator # Get test variables and constants from a single source from . import testvars as testvars class TestActionDeleteIndices(TestCase): def test_init_raise(self): self.assertRaises(TypeError, curator.DeleteIndices, 'invalid') def test_init_raise_bad_master_timeout(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one ilo = curator.IndexList(client) self.assertRaises(TypeError, curator.DeleteIndices, ilo, 'invalid') def test_init(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one ilo = curator.IndexList(client) do = curator.DeleteIndices(ilo) self.assertEqual(ilo, do.index_list) self.assertEqual(client, do.client) def test_do_dry_run(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_four client.cluster.state.return_value = testvars.clu_state_four client.indices.stats.return_value = testvars.stats_four client.indices.delete.return_value = None ilo = curator.IndexList(client) do = curator.DeleteIndices(ilo) self.assertIsNone(do.do_dry_run()) def test_do_action(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_four client.cluster.state.return_value = testvars.clu_state_four client.indices.stats.return_value = testvars.stats_four client.indices.delete.return_value = None ilo = curator.IndexList(client) do = curator.DeleteIndices(ilo) self.assertIsNone(do.do_action()) def test_do_action_not_successful(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_four client.cluster.state.return_value = testvars.clu_state_four client.indices.stats.return_value = testvars.stats_four client.indices.delete.return_value = None ilo = curator.IndexList(client) do = curator.DeleteIndices(ilo) self.assertIsNone(do.do_action()) def test_do_action_raises_exception(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_four client.cluster.state.return_value = testvars.clu_state_four client.indices.stats.return_value = testvars.stats_four client.indices.delete.return_value = None client.indices.delete.side_effect = testvars.fake_fail ilo = curator.IndexList(client) do = curator.DeleteIndices(ilo) self.assertRaises(curator.FailedExecution, do.do_action) def test_verify_result_positive(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_four client.cluster.state.return_value = testvars.clu_state_four client.indices.stats.return_value = testvars.stats_four client.indices.delete.return_value = None ilo = curator.IndexList(client) do = curator.DeleteIndices(ilo) self.assertTrue(do._verify_result([],2)) curator-4.2.5/test/unit/test_action_delete_snapshots.py000066400000000000000000000046261302700154000234510ustar00rootroot00000000000000from unittest import TestCase from mock import Mock, patch import elasticsearch import curator # Get test variables and constants from a single source from . import testvars as testvars class TestActionDeleteSnapshots(TestCase): def test_init_raise(self): self.assertRaises(TypeError, curator.DeleteSnapshots, 'invalid') def test_init(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo slo = curator.SnapshotList(client, repository=testvars.repo_name) do = curator.DeleteSnapshots(slo) self.assertEqual(slo, do.snapshot_list) self.assertEqual(client, do.client) def test_do_dry_run(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo client.snapshot.delete.return_value = None slo = curator.SnapshotList(client, repository=testvars.repo_name) do = curator.DeleteSnapshots(slo) self.assertIsNone(do.do_dry_run()) def test_do_action(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo client.snapshot.delete.return_value = None slo = curator.SnapshotList(client, repository=testvars.repo_name) do = curator.DeleteSnapshots(slo) self.assertIsNone(do.do_action()) def test_do_action_raises_exception(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo client.snapshot.delete.return_value = None client.snapshot.delete.side_effect = testvars.fake_fail slo = curator.SnapshotList(client, repository=testvars.repo_name) do = curator.DeleteSnapshots(slo) self.assertRaises(curator.FailedExecution, do.do_action) def test_not_safe_to_snap_raises_exception(self): client = Mock() client.snapshot.get.return_value = testvars.inprogress client.snapshot.get_repository.return_value = testvars.test_repo slo = curator.SnapshotList(client, repository=testvars.repo_name) do = curator.DeleteSnapshots(slo, retry_interval=0, retry_count=1) self.assertRaises(curator.FailedExecution, do.do_action) curator-4.2.5/test/unit/test_action_forcemerge.py000066400000000000000000000114551302700154000222210ustar00rootroot00000000000000from unittest import TestCase from mock import Mock, patch import elasticsearch import curator # Get test variables and constants from a single source from . import testvars as testvars class TestActionForceMerge(TestCase): def test_init_raise_bad_client(self): self.assertRaises( TypeError, curator.ForceMerge, 'invalid', max_num_segments=2) def test_init_raise_no_segment_count(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one client.indices.segments.return_value = testvars.shards ilo = curator.IndexList(client) self.assertRaises( curator.MissingArgument, curator.ForceMerge, ilo) def test_init(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one client.indices.segments.return_value = testvars.shards ilo = curator.IndexList(client) fmo = curator.ForceMerge(ilo, max_num_segments=2) self.assertEqual(ilo, fmo.index_list) self.assertEqual(client, fmo.client) def test_do_dry_run(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one client.indices.segments.return_value = testvars.shards client.indices.forcemerge.return_value = None client.indices.optimize.return_value = None ilo = curator.IndexList(client) fmo = curator.ForceMerge(ilo, max_num_segments=2) self.assertIsNone(fmo.do_dry_run()) def test_do_action_pre5(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one client.indices.segments.return_value = testvars.shards client.info.return_value = {'version': {'number': '2.3.2'} } client.indices.optimize.return_value = None ilo = curator.IndexList(client) fmo = curator.ForceMerge(ilo, max_num_segments=2) self.assertIsNone(fmo.do_action()) def test_do_action(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one client.indices.segments.return_value = testvars.shards client.info.return_value = {'version': {'number': '5.0.0'} } client.indices.forcemerge.return_value = None ilo = curator.IndexList(client) fmo = curator.ForceMerge(ilo, max_num_segments=2) self.assertIsNone(fmo.do_action()) def test_do_action_with_delay(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one client.indices.segments.return_value = testvars.shards client.info.return_value = {'version': {'number': '5.0.0'} } client.indices.forcemerge.return_value = None ilo = curator.IndexList(client) fmo = curator.ForceMerge(ilo, max_num_segments=2, delay=0.050) self.assertIsNone(fmo.do_action()) def test_do_action_raises_exception(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one client.indices.segments.return_value = testvars.shards client.indices.forcemerge.return_value = None client.indices.optimize.return_value = None client.indices.forcemerge.side_effect = testvars.fake_fail client.indices.optimize.side_effect = testvars.fake_fail ilo = curator.IndexList(client) fmo = curator.ForceMerge(ilo, max_num_segments=2) self.assertRaises(curator.FailedExecution, fmo.do_action) curator-4.2.5/test/unit/test_action_open.py000066400000000000000000000047741302700154000210520ustar00rootroot00000000000000from unittest import TestCase from mock import Mock, patch import elasticsearch import curator # Get test variables and constants from a single source from . import testvars as testvars class TestActionOpen(TestCase): def test_init_raise(self): self.assertRaises(TypeError, curator.Open, 'invalid') def test_init(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one ilo = curator.IndexList(client) oo = curator.Open(ilo) self.assertEqual(ilo, oo.index_list) self.assertEqual(client, oo.client) def test_do_dry_run(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_four client.cluster.state.return_value = testvars.clu_state_four client.indices.stats.return_value = testvars.stats_four client.indices.open.return_value = None ilo = curator.IndexList(client) ilo.filter_opened() oo = curator.Open(ilo) self.assertEqual([u'c-2016.03.05'], oo.index_list.indices) self.assertIsNone(oo.do_dry_run()) def test_do_action(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_four client.cluster.state.return_value = testvars.clu_state_four client.indices.stats.return_value = testvars.stats_four client.indices.open.return_value = None ilo = curator.IndexList(client) ilo.filter_opened() oo = curator.Open(ilo) self.assertEqual([u'c-2016.03.05'], oo.index_list.indices) self.assertIsNone(oo.do_action()) def test_do_action_raises_exception(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_four client.cluster.state.return_value = testvars.clu_state_four client.indices.stats.return_value = testvars.stats_four client.indices.open.return_value = None client.indices.open.side_effect = testvars.fake_fail ilo = curator.IndexList(client) oo = curator.Open(ilo) self.assertRaises(curator.FailedExecution, oo.do_action) curator-4.2.5/test/unit/test_action_replicas.py000066400000000000000000000071051302700154000217020ustar00rootroot00000000000000from unittest import TestCase from mock import Mock, patch import elasticsearch import curator # Get test variables and constants from a single source from . import testvars as testvars class TestActionReplicas(TestCase): def test_init_raise_bad_client(self): self.assertRaises( TypeError, curator.Replicas, 'invalid', count=2) def test_init_raise_no_count(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one ilo = curator.IndexList(client) self.assertRaises( curator.MissingArgument, curator.Replicas, ilo) def test_init(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one client.indices.put_settings.return_value = None ilo = curator.IndexList(client) ro = curator.Replicas(ilo, count=2) self.assertEqual(ilo, ro.index_list) self.assertEqual(client, ro.client) def test_do_dry_run(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one client.indices.put_settings.return_value = None ilo = curator.IndexList(client) ro = curator.Replicas(ilo, count=0) self.assertIsNone(ro.do_dry_run()) def test_do_action(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one client.indices.put_settings.return_value = None ilo = curator.IndexList(client) ro = curator.Replicas(ilo, count=0) self.assertIsNone(ro.do_action()) def test_do_action_wait(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one client.indices.put_settings.return_value = None client.cluster.health.return_value = None ilo = curator.IndexList(client) ro = curator.Replicas(ilo, count=1, wait_for_completion=True) self.assertIsNone(ro.do_action()) def test_do_action_raises_exception(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one client.indices.segments.return_value = testvars.shards client.indices.put_settings.return_value = None client.indices.put_settings.side_effect = testvars.fake_fail ilo = curator.IndexList(client) ro = curator.Replicas(ilo, count=2) self.assertRaises(curator.FailedExecution, ro.do_action) curator-4.2.5/test/unit/test_action_restore.py000066400000000000000000000177161302700154000215740ustar00rootroot00000000000000from unittest import TestCase from mock import Mock, patch import elasticsearch import curator # Get test variables and constants from a single source from . import testvars as testvars class TestActionRestore(TestCase): def test_init_raise_bad_snapshot_list(self): self.assertRaises(TypeError, curator.Restore, 'invalid') def test_init_raise_unsuccessful_snapshot_list(self): client = Mock() client.snapshot.get.return_value = testvars.partial client.snapshot.get_repository.return_value = testvars.test_repo slo = curator.SnapshotList(client, repository=testvars.repo_name) self.assertRaises(curator.CuratorException, curator.Restore, slo) def test_snapshot_derived_name(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo slo = curator.SnapshotList(client, repository=testvars.repo_name) ro = curator.Restore(slo) self.assertEqual('snapshot-2015.03.01', ro.name) def test_provided_name(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo slo = curator.SnapshotList(client, repository=testvars.repo_name) ro = curator.Restore(slo, name=testvars.snap_name) self.assertEqual(testvars.snap_name, ro.name) def test_partial_snap(self): client = Mock() client.snapshot.get.return_value = testvars.partial client.snapshot.get_repository.return_value = testvars.test_repo slo = curator.SnapshotList(client, repository=testvars.repo_name) ro = curator.Restore(slo, partial=True) self.assertEqual(testvars.snap_name, ro.name) def test_provided_indices(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo slo = curator.SnapshotList(client, repository=testvars.repo_name) ro = curator.Restore(slo, indices=testvars.named_indices) self.assertEqual('snapshot-2015.03.01', ro.name) def test_extra_settings(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo slo = curator.SnapshotList(client, repository=testvars.repo_name) ro = curator.Restore(slo, extra_settings={'foo':'bar'}) self.assertEqual(ro.body['foo'], 'bar') def test_bad_extra_settings(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo slo = curator.SnapshotList(client, repository=testvars.repo_name) ro = curator.Restore(slo, extra_settings='invalid') self.assertEqual(ro.body, { 'ignore_unavailable': False, 'partial': False, 'include_aliases': False, 'rename_replacement': '', 'rename_pattern': '', 'indices': ['index-2015.01.01', 'index-2015.02.01'], 'include_global_state': True } ) def test_get_expected_output(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo slo = curator.SnapshotList(client, repository=testvars.repo_name) ro = curator.Restore( slo, rename_pattern='(.+)', rename_replacement='new_$1') self.assertEqual( ro.expected_output, ['new_index-2015.01.01', 'new_index-2015.02.01'] ) def test_do_dry_run(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo slo = curator.SnapshotList(client, repository=testvars.repo_name) ro = curator.Restore(slo) self.assertIsNone(ro.do_dry_run()) def test_do_dry_run_with_renames(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo slo = curator.SnapshotList(client, repository=testvars.repo_name) ro = curator.Restore( slo, rename_pattern='(.+)', rename_replacement='new_$1') self.assertIsNone(ro.do_dry_run()) def test_report_state_all(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.snapshot.get.return_value = testvars.snapshot client.snapshot.get_repository.return_value = testvars.test_repo client.indices.get_settings.return_value = testvars.settings_named slo = curator.SnapshotList(client, repository=testvars.repo_name) ro = curator.Restore(slo) self.assertIsNone(ro.report_state()) def test_report_state_not_all(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo client.indices.get_settings.return_value = testvars.settings_one slo = curator.SnapshotList(client, repository=testvars.repo_name) ro = curator.Restore( slo, rename_pattern='(.+)', rename_replacement='new_$1') self.assertIsNone(ro.report_state()) def test_do_action_success(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo client.snapshot.status.return_value = testvars.nosnap_running client.snapshot.verify_repository.return_value = testvars.verified_nodes client.indices.get_settings.return_value = testvars.settings_named slo = curator.SnapshotList(client, repository=testvars.repo_name) ro = curator.Restore(slo) self.assertIsNone(ro.do_action()) def test_do_action_snap_in_progress(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo client.snapshot.status.return_value = testvars.snap_running client.snapshot.verify_repository.return_value = testvars.verified_nodes client.indices.get_settings.return_value = testvars.settings_named slo = curator.SnapshotList(client, repository=testvars.repo_name) ro = curator.Restore(slo) self.assertRaises(curator.SnapshotInProgress, ro.do_action) def test_do_action_success_no_wfc(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo client.snapshot.status.return_value = testvars.nosnap_running client.snapshot.verify_repository.return_value = testvars.verified_nodes client.indices.get_settings.return_value = testvars.settings_named slo = curator.SnapshotList(client, repository=testvars.repo_name) ro = curator.Restore(slo, wait_for_completion=False) self.assertIsNone(ro.do_action()) def test_do_action_report_on_failure(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo client.snapshot.status.return_value = testvars.nosnap_running client.snapshot.verify_repository.return_value = testvars.verified_nodes client.indices.get_settings.return_value = testvars.settings_named client.snapshot.restore.side_effect = testvars.fake_fail slo = curator.SnapshotList(client, repository=testvars.repo_name) ro = curator.Restore(slo) self.assertRaises(curator.FailedExecution, ro.do_action) curator-4.2.5/test/unit/test_action_snapshot.py000066400000000000000000000230271302700154000217400ustar00rootroot00000000000000from unittest import TestCase from mock import Mock, patch import elasticsearch import curator # Get test variables and constants from a single source from . import testvars as testvars class TestActionSnapshot(TestCase): def test_init_raise_bad_index_list(self): self.assertRaises(TypeError, curator.Snapshot, 'invalid') def test_init_no_repo_arg_exception(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one ilo = curator.IndexList(client) self.assertRaises(curator.MissingArgument, curator.Snapshot, ilo) def test_init_no_repo_exception(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one client.snapshot.get_repository.return_value = {'repo':{'foo':'bar'}} ilo = curator.IndexList(client) self.assertRaises( curator.ActionError, curator.Snapshot, ilo, repository='notfound') def test_init_no_name_exception(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one client.snapshot.get_repository.return_value = testvars.test_repo ilo = curator.IndexList(client) self.assertRaises(curator.MissingArgument, curator.Snapshot, ilo, repository=testvars.repo_name) def test_init_success(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one client.snapshot.get_repository.return_value = testvars.test_repo ilo = curator.IndexList(client) so = curator.Snapshot(ilo, repository=testvars.repo_name, name=testvars.snap_name) self.assertEqual(testvars.repo_name, so.repository) self.assertIsNone(so.state) def test_get_state_success(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one client.snapshot.get_repository.return_value = testvars.test_repo client.snapshot.get.return_value = testvars.snapshots ilo = curator.IndexList(client) so = curator.Snapshot(ilo, repository=testvars.repo_name, name=testvars.snap_name) so.get_state() self.assertEqual('SUCCESS', so.state) def test_get_state_fail(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one client.snapshot.get_repository.return_value = testvars.test_repo client.snapshot.get.return_value = {'snapshots':[]} ilo = curator.IndexList(client) so = curator.Snapshot(ilo, repository=testvars.repo_name, name=testvars.snap_name) self.assertRaises(curator.CuratorException, so.get_state) def test_report_state_success(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one client.snapshot.get_repository.return_value = testvars.test_repo client.snapshot.get.return_value = testvars.snapshots ilo = curator.IndexList(client) so = curator.Snapshot(ilo, repository=testvars.repo_name, name=testvars.snap_name) so.report_state() self.assertEqual('SUCCESS', so.state) def test_report_state_other(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one client.snapshot.get_repository.return_value = testvars.test_repo client.snapshot.get.return_value = testvars.highly_unlikely ilo = curator.IndexList(client) so = curator.Snapshot(ilo, repository=testvars.repo_name, name=testvars.snap_name) so.report_state() self.assertEqual('IN_PROGRESS', so.state) def test_do_dry_run(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one client.snapshot.get_repository.return_value = testvars.test_repo client.snapshot.get.return_value = testvars.snapshots client.snapshot.create.return_value = None client.snapshot.status.return_value = testvars.nosnap_running client.snapshot.verify_repository.return_value = testvars.verified_nodes ilo = curator.IndexList(client) so = curator.Snapshot(ilo, repository=testvars.repo_name, name=testvars.snap_name) self.assertIsNone(so.do_dry_run()) def test_do_action_success(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one client.snapshot.get_repository.return_value = testvars.test_repo client.snapshot.get.return_value = testvars.snapshots client.snapshot.create.return_value = None client.snapshot.status.return_value = testvars.nosnap_running client.snapshot.verify_repository.return_value = testvars.verified_nodes ilo = curator.IndexList(client) so = curator.Snapshot(ilo, repository=testvars.repo_name, name=testvars.snap_name) self.assertIsNone(so.do_action()) def test_do_action_raise_snap_in_progress(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one client.snapshot.get_repository.return_value = testvars.test_repo client.snapshot.get.return_value = testvars.snapshots client.snapshot.create.return_value = None client.snapshot.status.return_value = testvars.snap_running client.snapshot.verify_repository.return_value = testvars.verified_nodes ilo = curator.IndexList(client) so = curator.Snapshot(ilo, repository=testvars.repo_name, name=testvars.snap_name) self.assertRaises(curator.SnapshotInProgress, so.do_action) def test_do_action_no_wait_for_completion(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one client.snapshot.get_repository.return_value = testvars.test_repo client.snapshot.get.return_value = testvars.snapshots client.snapshot.create.return_value = None client.snapshot.status.return_value = testvars.nosnap_running client.snapshot.verify_repository.return_value = testvars.verified_nodes ilo = curator.IndexList(client) so = curator.Snapshot(ilo, repository=testvars.repo_name, name=testvars.snap_name, wait_for_completion=False) self.assertIsNone(so.do_action()) def test_do_action_raise_on_failure(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one client.snapshot.get_repository.return_value = testvars.test_repo client.snapshot.get.return_value = testvars.snapshots client.snapshot.create.return_value = None client.snapshot.create.side_effect = testvars.fake_fail client.snapshot.status.return_value = testvars.nosnap_running client.snapshot.verify_repository.return_value = testvars.verified_nodes ilo = curator.IndexList(client) so = curator.Snapshot(ilo, repository=testvars.repo_name, name=testvars.snap_name) self.assertRaises(curator.FailedExecution, so.do_action) curator-4.2.5/test/unit/test_class_index_list.py000066400000000000000000001531441302700154000220770ustar00rootroot00000000000000from unittest import TestCase from mock import Mock, patch import elasticsearch import yaml import curator # Get test variables and constants from a single source from . import testvars as testvars class TestIndexListClientAndInit(TestCase): def test_init_bad_client(self): client = 'not a real client' self.assertRaises(TypeError, curator.IndexList, client) def test_init_get_indices_exception(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.indices.get_settings.side_effect = testvars.fake_fail self.assertRaises(curator.FailedExecution, curator.IndexList, client) def test_init(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two il = curator.IndexList(client) self.assertEqual( testvars.stats_two['indices']['index-2016.03.03']['total']['store']['size_in_bytes'], il.index_info['index-2016.03.03']['size_in_bytes'] ) self.assertEqual( testvars.clu_state_two['metadata']['indices']['index-2016.03.04']['state'], il.index_info['index-2016.03.04']['state'] ) self.assertEqual(['index-2016.03.03','index-2016.03.04'], sorted(il.indices)) def test_for_closed_index(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_2_closed client.cluster.state.return_value = testvars.cs_two_closed client.indices.stats.return_value = testvars.stats_two il = curator.IndexList(client) self.assertEqual('close', il.index_info['index-2016.03.03']['state']) def test_skip_index_without_creation_date(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two_no_cd client.cluster.state.return_value = testvars.clu_state_two_no_cd client.indices.stats.return_value = testvars.stats_two il = curator.IndexList(client) self.assertEqual(['index-2016.03.03'], sorted(il.indices)) class TestIndexListOtherMethods(TestCase): def test_empty_list(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two il = curator.IndexList(client) self.assertEqual(2, len(il.indices)) il.indices = [] self.assertRaises(curator.NoIndices, il.empty_list_check) def test_get_segmentcount(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one client.indices.segments.return_value = testvars.shards il = curator.IndexList(client) il._get_segmentcounts() self.assertEqual(71, il.index_info[testvars.named_index]['segments']) class TestIndexListAgeFilterName(TestCase): def test_get_name_based_ages_match(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two il = curator.IndexList(client) il._get_name_based_ages('%Y.%m.%d') self.assertEqual(1456963200,il.index_info['index-2016.03.03']['age']['name']) def test_get_name_based_ages_no_match(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two nomatch = curator.IndexList(client) nomatch._get_name_based_ages('%Y-%m-%d') self.assertEqual( curator.fix_epoch( testvars.settings_two['index-2016.03.03']['settings']['index']['creation_date'] ), nomatch.index_info['index-2016.03.03']['age']['creation_date'] ) class TestIndexListAgeFilterStatsAPI(TestCase): def test_get_field_stats_dates_success(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two il = curator.IndexList(client) client.field_stats.return_value = testvars.fieldstats_two il._get_field_stats_dates(field='timestamp') self.assertEqual( curator.fix_epoch( testvars.fieldstats_two['indices']['index-2016.03.04']['fields']['timestamp']['min_value'] ), il.index_info['index-2016.03.04']['age']['min_value'] ) def test_get_field_stats_dates_negative(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two il = curator.IndexList(client) client.field_stats.return_value = testvars.fieldstats_two il._get_field_stats_dates(field='timestamp') self.assertNotIn('not_an_index_name', list(il.index_info.keys())) def test_get_field_stats_dates_field_not_found(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two client.field_stats.return_value = testvars.fieldstats_two il = curator.IndexList(client) self.assertRaises( curator.ActionError, il._get_field_stats_dates, field='not_in_index') class TestIndexListRegexFilters(TestCase): def test_filter_by_regex_prefix(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two il = curator.IndexList(client) self.assertEqual( [u'index-2016.03.03', u'index-2016.03.04'], sorted(il.indices) ) il.filter_by_regex(kind='prefix', value='ind') self.assertEqual( [u'index-2016.03.03', u'index-2016.03.04'], sorted(il.indices) ) il.filter_by_regex(kind='prefix', value='ind', exclude=True) self.assertEqual([], il.indices) def test_filter_by_regex_timestring(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two il = curator.IndexList(client) self.assertEqual( [u'index-2016.03.03', u'index-2016.03.04'], sorted(il.indices) ) il.filter_by_regex(kind='timestring', value='%Y.%m.%d') self.assertEqual( [u'index-2016.03.03', u'index-2016.03.04'], sorted(il.indices) ) il.filter_by_regex(kind='timestring', value='%Y.%m.%d', exclude=True) self.assertEqual([], il.indices) def test_filter_by_regex_no_match_exclude(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two il = curator.IndexList(client) self.assertEqual( [u'index-2016.03.03', u'index-2016.03.04'], sorted(il.indices) ) il.filter_by_regex(kind='prefix', value='invalid', exclude=True) # self.assertEqual([], il.indices) self.assertEqual( [u'index-2016.03.03', u'index-2016.03.04'], sorted(il.indices) ) def test_filter_by_regex_no_value(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two il = curator.IndexList(client) self.assertEqual( [u'index-2016.03.03', u'index-2016.03.04'], sorted(il.indices) ) self.assertRaises(ValueError, il.filter_by_regex, kind='prefix', value=None) self.assertEqual( [u'index-2016.03.03', u'index-2016.03.04'], sorted(il.indices) ) il.filter_by_regex(kind='prefix', value=0) self.assertEqual([], il.indices) def test_filter_by_regex_bad_kind(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two il = curator.IndexList(client) self.assertEqual( [u'index-2016.03.03', u'index-2016.03.04'], sorted(il.indices) ) self.assertRaises(ValueError, il.filter_by_regex, kind='invalid', value=None) class TestIndexListFilterByAge(TestCase): def test_missing_direction(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two il = curator.IndexList(client) self.assertRaises(curator.MissingArgument, il.filter_by_age, unit='days', unit_count=1 ) def test_bad_direction(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two il = curator.IndexList(client) self.assertRaises(ValueError, il.filter_by_age, unit='days', unit_count=1, direction="invalid" ) def test_name_no_timestring(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two il = curator.IndexList(client) self.assertRaises(curator.MissingArgument, il.filter_by_age, source='name', unit='days', unit_count=1, direction='older' ) def test_name_older_than_now(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two il = curator.IndexList(client) il.filter_by_age(source='name', direction='older', timestring='%Y.%m.%d', unit='days', unit_count=1 ) self.assertEqual( ['index-2016.03.03','index-2016.03.04'], sorted(il.indices) ) def test_name_older_than_now_exclude(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two il = curator.IndexList(client) il.filter_by_age(source='name', direction='older', timestring='%Y.%m.%d', unit='days', unit_count=1, exclude=True ) self.assertEqual( [], sorted(il.indices) ) def test_name_younger_than_now(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two il = curator.IndexList(client) il.filter_by_age(source='name', direction='younger', timestring='%Y.%m.%d', unit='days', unit_count=1 ) self.assertEqual([], sorted(il.indices)) def test_name_younger_than_now_exclude(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two il = curator.IndexList(client) il.filter_by_age(source='name', direction='younger', timestring='%Y.%m.%d', unit='days', unit_count=1, exclude=True ) self.assertEqual( ['index-2016.03.03','index-2016.03.04'], sorted(il.indices)) def test_name_younger_than_past_date(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two il = curator.IndexList(client) il.filter_by_age(source='name', direction='younger', timestring='%Y.%m.%d', unit='seconds', unit_count=0, epoch=1457049599 ) self.assertEqual(['index-2016.03.04'], sorted(il.indices)) def test_name_older_than_past_date(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two il = curator.IndexList(client) il.filter_by_age(source='name', direction='older', timestring='%Y.%m.%d', unit='seconds', unit_count=0, epoch=1456963201 ) self.assertEqual(['index-2016.03.03'], sorted(il.indices)) def test_creation_date_older_than_now(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two il = curator.IndexList(client) il.filter_by_age(source='creation_date', direction='older', unit='days', unit_count=1 ) self.assertEqual( ['index-2016.03.03','index-2016.03.04'], sorted(il.indices) ) def test_creation_date_older_than_now_raises(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two il = curator.IndexList(client) il.index_info['index-2016.03.03']['age'].pop('creation_date') il.index_info['index-2016.03.04']['age'].pop('creation_date') il.filter_by_age( source='creation_date', direction='older', unit='days', unit_count=1 ) self.assertEqual([], il.indices) def test_creation_date_younger_than_now(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two il = curator.IndexList(client) il.filter_by_age(source='creation_date', direction='younger', unit='days', unit_count=1 ) self.assertEqual([], sorted(il.indices)) def test_creation_date_younger_than_now_raises(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two il = curator.IndexList(client) il.index_info['index-2016.03.03']['age'].pop('creation_date') il.index_info['index-2016.03.04']['age'].pop('creation_date') il.filter_by_age( source='creation_date', direction='younger', unit='days', unit_count=1 ) self.assertEqual([], il.indices) def test_creation_date_younger_than_past_date(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two il = curator.IndexList(client) il.filter_by_age(source='creation_date', direction='younger', unit='seconds', unit_count=0, epoch=1457049599 ) self.assertEqual(['index-2016.03.04'], sorted(il.indices)) def test_creation_date_older_than_past_date(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two il = curator.IndexList(client) il.filter_by_age(source='creation_date', direction='older', unit='seconds', unit_count=0, epoch=1456963201 ) self.assertEqual(['index-2016.03.03'], sorted(il.indices)) def test_field_stats_missing_field(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two il = curator.IndexList(client) self.assertRaises(curator.MissingArgument, il.filter_by_age, source='field_stats', direction='older', unit='days', unit_count=1 ) def test_field_stats_invalid_stats_result(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two il = curator.IndexList(client) self.assertRaises(ValueError, il.filter_by_age, field='timestamp', source='field_stats', direction='older', unit='days', unit_count=1, stats_result='invalid' ) def test_field_stats_invalid_source(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two il = curator.IndexList(client) self.assertRaises(ValueError, il.filter_by_age, source='invalid', direction='older', unit='days', unit_count=1 ) def test_field_stats_older_than_now(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two client.field_stats.return_value = testvars.fieldstats_two il = curator.IndexList(client) il.filter_by_age(source='field_stats', direction='older', field='timestamp', unit='days', unit_count=1 ) self.assertEqual( ['index-2016.03.03','index-2016.03.04'], sorted(il.indices) ) def test_field_stats_younger_than_now(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two client.field_stats.return_value = testvars.fieldstats_two il = curator.IndexList(client) il.filter_by_age(source='field_stats', direction='younger', field='timestamp', unit='days', unit_count=1 ) self.assertEqual([], sorted(il.indices)) def test_field_stats_younger_than_past_date(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two client.field_stats.return_value = testvars.fieldstats_two il = curator.IndexList(client) il.filter_by_age(source='field_stats', direction='younger', field='timestamp', unit='seconds', unit_count=0, epoch=1457049599 ) self.assertEqual(['index-2016.03.04'], sorted(il.indices)) def test_field_stats_older_than_past_date(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two client.field_stats.return_value = testvars.fieldstats_two il = curator.IndexList(client) il.filter_by_age(source='field_stats', direction='older', field='timestamp', unit='seconds', unit_count=0, epoch=1456963207 ) self.assertEqual(['index-2016.03.03'], sorted(il.indices)) def test_field_stats_older_than_now_max(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two client.field_stats.return_value = testvars.fieldstats_two il = curator.IndexList(client) il.filter_by_age(source='field_stats', direction='older', field='timestamp', stats_result='max_value', unit='days', unit_count=0 ) self.assertEqual( ['index-2016.03.03','index-2016.03.04'], sorted(il.indices) ) def test_field_stats_younger_than_now_max(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two client.field_stats.return_value = testvars.fieldstats_two il = curator.IndexList(client) il.filter_by_age(source='field_stats', direction='younger', field='timestamp', stats_result='max_value', unit='days', unit_count=0 ) self.assertEqual([], sorted(il.indices)) def test_field_stats_younger_than_past_date_max(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two client.field_stats.return_value = testvars.fieldstats_two il = curator.IndexList(client) il.filter_by_age(source='field_stats', direction='younger', field='timestamp', stats_result='max_value', unit='seconds', unit_count=0, epoch=1457135998 ) self.assertEqual(['index-2016.03.04'], sorted(il.indices)) def test_field_stats_older_than_past_date_max(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two client.field_stats.return_value = testvars.fieldstats_two il = curator.IndexList(client) il.filter_by_age(source='field_stats', direction='older', field='timestamp', stats_result='max_value', unit='seconds', unit_count=0, epoch=1457049600 ) self.assertEqual(['index-2016.03.03'], sorted(il.indices)) class TestIndexListFilterBySpace(TestCase): def test_missing_disk_space_value(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two client.field_stats.return_value = testvars.fieldstats_two il = curator.IndexList(client) self.assertRaises(curator.MissingArgument, il.filter_by_space) def test_filter_result_by_name(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two client.field_stats.return_value = testvars.fieldstats_two il = curator.IndexList(client) il.filter_by_space(disk_space=1.1) self.assertEqual(['index-2016.03.03'], il.indices) def test_filter_result_by_name_reverse_order(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two client.field_stats.return_value = testvars.fieldstats_two il = curator.IndexList(client) il.filter_by_space(disk_space=1.1, reverse=False) self.assertEqual(['index-2016.03.04'], il.indices) def test_filter_result_by_name_exclude(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two client.field_stats.return_value = testvars.fieldstats_two il = curator.IndexList(client) il.filter_by_space(disk_space=1.1, exclude=True) self.assertEqual(['index-2016.03.04'], il.indices) def test_filter_result_by_date_raise(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_four client.cluster.state.return_value = testvars.clu_state_four client.indices.stats.return_value = testvars.stats_four client.field_stats.return_value = testvars.fieldstats_four il = curator.IndexList(client) self.assertRaises(ValueError, il.filter_by_space, disk_space=2.1, use_age=True, source='invalid' ) def test_filter_result_by_date_timestring_raise(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_four client.cluster.state.return_value = testvars.clu_state_four client.indices.stats.return_value = testvars.stats_four client.field_stats.return_value = testvars.fieldstats_four il = curator.IndexList(client) self.assertRaises(curator.MissingArgument, il.filter_by_space, disk_space=2.1, use_age=True, source='name' ) def test_filter_result_by_date_timestring(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_four client.cluster.state.return_value = testvars.clu_state_four client.indices.stats.return_value = testvars.stats_four client.field_stats.return_value = testvars.fieldstats_four il = curator.IndexList(client) il.filter_by_space( disk_space=2.1, use_age=True, source='name', timestring='%Y.%m.%d' ) self.assertEqual(['a-2016.03.03'], sorted(il.indices)) def test_filter_result_by_date_non_matching_timestring(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_four client.cluster.state.return_value = testvars.clu_state_four client.indices.stats.return_value = testvars.stats_four client.field_stats.return_value = testvars.fieldstats_four il = curator.IndexList(client) il.filter_by_space( disk_space=2.1, use_age=True, source='name', timestring='%Y.%m.%d.%H' ) self.assertEqual([], sorted(il.indices)) def test_filter_result_by_date_field_stats_raise(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_four client.cluster.state.return_value = testvars.clu_state_four client.indices.stats.return_value = testvars.stats_four client.field_stats.return_value = testvars.fieldstats_four il = curator.IndexList(client) self.assertRaises(ValueError, il.filter_by_space, disk_space=2.1, use_age=True, source='min_value' ) def test_filter_result_by_date_no_field_raise(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_four client.cluster.state.return_value = testvars.clu_state_four client.indices.stats.return_value = testvars.stats_four client.field_stats.return_value = testvars.fieldstats_four il = curator.IndexList(client) self.assertRaises(curator.MissingArgument, il.filter_by_space, disk_space=2.1, use_age=True, source='field_stats' ) def test_filter_result_by_date_invalid_stats_result_raise(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_four client.cluster.state.return_value = testvars.clu_state_four client.indices.stats.return_value = testvars.stats_four client.field_stats.return_value = testvars.fieldstats_four il = curator.IndexList(client) self.assertRaises(ValueError, il.filter_by_space, disk_space=2.1, use_age=True, source='field_stats', field='timestamp', stats_result='invalid' ) def test_filter_result_by_date_field_stats(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_four client.cluster.state.return_value = testvars.clu_state_four client.indices.stats.return_value = testvars.stats_four client.field_stats.return_value = testvars.fieldstats_four il = curator.IndexList(client) il.filter_by_space( disk_space=2.1, use_age=True, source='field_stats', field='timestamp' ) self.assertEqual(['a-2016.03.03'], il.indices) def test_filter_result_by_creation_date(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_four client.cluster.state.return_value = testvars.clu_state_four client.indices.stats.return_value = testvars.stats_four client.field_stats.return_value = testvars.fieldstats_four il = curator.IndexList(client) il.filter_by_space(disk_space=2.1, use_age=True) self.assertEqual(['a-2016.03.03'], il.indices) class TestIndexListFilterKibana(TestCase): def test_filter_kibana_positive(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two client.field_stats.return_value = testvars.fieldstats_two il = curator.IndexList(client) # Establish the object per requirements, then overwrite il.indices = ['.kibana', '.marvel-kibana', 'kibana-int', '.marvel-es-data', 'dummy'] il.filter_kibana() self.assertEqual(['dummy'], il.indices) def test_filter_kibana_positive_exclude(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two client.field_stats.return_value = testvars.fieldstats_two il = curator.IndexList(client) # Establish the object per requirements, then overwrite kibana_indices = [ '.kibana', '.marvel-kibana', 'kibana-int', '.marvel-es-data'] il.indices = kibana_indices il.indices.append('dummy') il.filter_kibana(exclude=True) self.assertEqual(kibana_indices, il.indices) def test_filter_kibana_negative(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two client.field_stats.return_value = testvars.fieldstats_two il = curator.IndexList(client) # Establish the object per requirements, then overwrite il.indices = ['kibana', 'marvel-kibana', 'cabana-int', 'marvel-es-data', 'dummy'] il.filter_kibana() self.assertEqual( ['kibana', 'marvel-kibana', 'cabana-int', 'marvel-es-data', 'dummy'], il.indices ) class TestIndexListFilterForceMerged(TestCase): def test_filter_forcemerge_raise(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one client.indices.segments.return_value = testvars.shards il = curator.IndexList(client) self.assertRaises(curator.MissingArgument, il.filter_forceMerged) def test_filter_forcemerge_positive(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one client.indices.segments.return_value = testvars.shards il = curator.IndexList(client) il.filter_forceMerged(max_num_segments=2) self.assertEqual([testvars.named_index], il.indices) def test_filter_forcemerge_negative(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one client.indices.segments.return_value = testvars.fm_shards il = curator.IndexList(client) il.filter_forceMerged(max_num_segments=2) self.assertEqual([], il.indices) class TestIndexListFilterOpened(TestCase): def test_filter_opened(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_four client.cluster.state.return_value = testvars.clu_state_four client.indices.stats.return_value = testvars.stats_four client.field_stats.return_value = testvars.fieldstats_four il = curator.IndexList(client) il.filter_opened() self.assertEqual(['c-2016.03.05'], il.indices) class TestIndexListFilterAllocated(TestCase): def test_missing_key(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two il = curator.IndexList(client) self.assertRaises( curator.MissingArgument, il.filter_allocated, value='foo', allocation_type='invalid' ) def test_missing_value(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two il = curator.IndexList(client) self.assertRaises( curator.MissingArgument, il.filter_allocated, key='tag', allocation_type='invalid' ) def test_invalid_allocation_type(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two il = curator.IndexList(client) self.assertRaises( ValueError, il.filter_allocated, key='tag', value='foo', allocation_type='invalid' ) def test_success(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two il = curator.IndexList(client) il.filter_allocated(key='tag', value='foo', allocation_type='include') self.assertEqual(['index-2016.03.04'], il.indices) def test_invalid_tag(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two il = curator.IndexList(client) il.filter_allocated( key='invalid', value='foo', allocation_type='include') self.assertEqual( ['index-2016.03.03','index-2016.03.04'], sorted(il.indices)) class TestIterateFiltersIndex(TestCase): def test_no_filters(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_four client.cluster.state.return_value = testvars.clu_state_four client.indices.stats.return_value = testvars.stats_four ilo = curator.IndexList(client) ilo.iterate_filters({}) self.assertEqual( ['a-2016.03.03', 'b-2016.03.04', 'c-2016.03.05', 'd-2016.03.06'], sorted(ilo.indices) ) def test_no_filtertype(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_four client.cluster.state.return_value = testvars.clu_state_four client.indices.stats.return_value = testvars.stats_four ilo = curator.IndexList(client) config = {'filters': [{'no_filtertype':'fail'}]} self.assertRaises( curator.ConfigurationError, ilo.iterate_filters, config) def test_invalid_filtertype(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_four client.cluster.state.return_value = testvars.clu_state_four client.indices.stats.return_value = testvars.stats_four ilo = curator.IndexList(client) config = {'filters': [{'filtertype':12345.6789}]} self.assertRaises( curator.ConfigurationError, ilo.iterate_filters, config) def test_pattern_filtertype(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_four client.cluster.state.return_value = testvars.clu_state_four client.indices.stats.return_value = testvars.stats_four ilo = curator.IndexList(client) config = yaml.load(testvars.pattern_ft)['actions'][1] ilo.iterate_filters(config) self.assertEqual(['a-2016.03.03'], ilo.indices) def test_age_filtertype(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two ilo = curator.IndexList(client) config = yaml.load(testvars.age_ft)['actions'][1] ilo.iterate_filters(config) self.assertEqual(['index-2016.03.03'], ilo.indices) def test_space_filtertype(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_four client.cluster.state.return_value = testvars.clu_state_four client.indices.stats.return_value = testvars.stats_four client.field_stats.return_value = testvars.fieldstats_four ilo = curator.IndexList(client) config = yaml.load(testvars.space_ft)['actions'][1] ilo.iterate_filters(config) self.assertEqual(['a-2016.03.03'], ilo.indices) def test_forcemerge_filtertype(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one client.indices.segments.return_value = testvars.shards ilo = curator.IndexList(client) config = yaml.load(testvars.forcemerge_ft)['actions'][1] ilo.iterate_filters(config) self.assertEqual([testvars.named_index], ilo.indices) def test_allocated_filtertype(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two ilo = curator.IndexList(client) config = yaml.load(testvars.allocated_ft)['actions'][1] ilo.iterate_filters(config) self.assertEqual(['index-2016.03.04'], ilo.indices) def test_kibana_filtertype(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two client.field_stats.return_value = testvars.fieldstats_two ilo = curator.IndexList(client) # Establish the object per requirements, then overwrite ilo.indices = [ '.kibana', '.marvel-kibana', 'kibana-int', '.marvel-es-data', 'dummy' ] config = yaml.load(testvars.kibana_ft)['actions'][1] ilo.iterate_filters(config) self.assertEqual(['dummy'], ilo.indices) def test_opened_filtertype(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_four client.cluster.state.return_value = testvars.clu_state_four client.indices.stats.return_value = testvars.stats_four client.field_stats.return_value = testvars.fieldstats_four ilo = curator.IndexList(client) config = yaml.load(testvars.opened_ft)['actions'][1] ilo.iterate_filters(config) self.assertEqual(['c-2016.03.05'], ilo.indices) def test_closed_filtertype(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_four client.cluster.state.return_value = testvars.clu_state_four client.indices.stats.return_value = testvars.stats_four client.field_stats.return_value = testvars.fieldstats_four ilo = curator.IndexList(client) config = yaml.load(testvars.closed_ft)['actions'][1] ilo.iterate_filters(config) self.assertEqual( ['a-2016.03.03','b-2016.03.04','d-2016.03.06'], sorted(ilo.indices)) def test_none_filtertype(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two ilo = curator.IndexList(client) config = yaml.load(testvars.none_ft)['actions'][1] ilo.iterate_filters(config) self.assertEqual( ['index-2016.03.03', 'index-2016.03.04'], sorted(ilo.indices)) def test_unknown_filtertype_raises(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two ilo = curator.IndexList(client) config = yaml.load(testvars.invalid_ft)['actions'][1] self.assertRaises( curator.ConfigurationError, ilo.iterate_filters, config ) class TestIndexListFilterAlias(TestCase): def test_raise(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one il = curator.IndexList(client) self.assertRaises(curator.MissingArgument, il.filter_by_alias) def test_positive(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two client.indices.get_alias.return_value = testvars.settings_2_get_aliases il = curator.IndexList(client) il.filter_by_alias(aliases=['my_alias']) self.assertEqual( sorted(list(testvars.settings_two.keys())), sorted(il.indices)) def test_negative(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two client.indices.get_alias.return_value = {} il = curator.IndexList(client) il.filter_by_alias(aliases=['not_my_alias']) self.assertEqual( sorted([]), sorted(il.indices)) def test_get_alias_raises(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two client.indices.get_alias.side_effect = testvars.get_alias_fail client.indices.get_alias.return_value = testvars.settings_2_get_aliases il = curator.IndexList(client) il.filter_by_alias(aliases=['my_alias']) self.assertEqual( sorted([]), sorted(il.indices)) class TestIndexListFilterCount(TestCase): def test_raise(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_one client.cluster.state.return_value = testvars.clu_state_one client.indices.stats.return_value = testvars.stats_one il = curator.IndexList(client) self.assertRaises(curator.MissingArgument, il.filter_by_count) def test_without_age(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two client.indices.get_alias.return_value = testvars.settings_2_get_aliases il = curator.IndexList(client) il.filter_by_count(count=1) self.assertEqual([u'index-2016.03.03'], il.indices) def test_without_age_reversed(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two client.indices.get_alias.return_value = testvars.settings_2_get_aliases il = curator.IndexList(client) il.filter_by_count(count=1, reverse=False) self.assertEqual([u'index-2016.03.04'], il.indices) def test_with_age(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two client.indices.get_alias.return_value = testvars.settings_2_get_aliases il = curator.IndexList(client) il.filter_by_count( count=1, use_age=True, source='name', timestring='%Y.%m.%d' ) self.assertEqual([u'index-2016.03.03'], il.indices) def test_with_age_creation_date(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two client.indices.get_alias.return_value = testvars.settings_2_get_aliases il = curator.IndexList(client) il.filter_by_count(count=1, use_age=True) self.assertEqual([u'index-2016.03.03'], il.indices) def test_with_age_reversed(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two client.indices.get_alias.return_value = testvars.settings_2_get_aliases il = curator.IndexList(client) il.filter_by_count( count=1, use_age=True, source='name', timestring='%Y.%m.%d', reverse=False ) self.assertEqual([u'index-2016.03.04'], il.indices) curator-4.2.5/test/unit/test_class_snapshot_list.py000066400000000000000000000444141302700154000226260ustar00rootroot00000000000000from unittest import TestCase from mock import Mock, patch import elasticsearch import yaml import curator # Get test variables and constants from a single source from . import testvars as testvars class TestSnapshotListClientAndInit(TestCase): def test_init_bad_client(self): client = 'not a real client' self.assertRaises(TypeError, curator.SnapshotList, client) def test_init_no_repo_exception(self): client = Mock() self.assertRaises(curator.MissingArgument, curator.SnapshotList, client) def test_init_get_snapshots_exception(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get.side_effect = testvars.fake_fail client.snapshot.get_repository.return_value = {} self.assertRaises( curator.FailedExecution, curator.SnapshotList, client, repository=testvars.repo_name ) def test_init(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo sl = curator.SnapshotList(client, repository=testvars.repo_name) self.assertEqual(testvars.snapshots['snapshots'],sl.all_snapshots) self.assertEqual( ['snap_name','snapshot-2015.03.01'], sorted(sl.snapshots) ) class TestSnapshotListOtherMethods(TestCase): def test_empty_list(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo sl = curator.SnapshotList(client, repository=testvars.repo_name) self.assertEqual(2, len(sl.snapshots)) sl.snapshots = [] self.assertRaises(curator.NoSnapshots, sl.empty_list_check) def test_working_list(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo sl = curator.SnapshotList(client, repository=testvars.repo_name) self.assertEqual(['snap_name', 'snapshot-2015.03.01'], sl.working_list()) class TestSnapshotListAgeFilterName(TestCase): def test_get_name_based_ages_match(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo sl = curator.SnapshotList(client, repository=testvars.repo_name) sl._get_name_based_ages('%Y.%m.%d') self.assertEqual(1425168000, sl.snapshot_info['snapshot-2015.03.01']['age_by_name'] ) def test_get_name_based_ages_no_match(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo sl = curator.SnapshotList(client, repository=testvars.repo_name) sl._get_name_based_ages('%Y.%m.%d') self.assertIsNone(sl.snapshot_info['snap_name']['age_by_name']) class TestSnapshotListStateFilter(TestCase): def test_success_inclusive(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo sl = curator.SnapshotList(client, repository=testvars.repo_name) sl.filter_by_state(state='SUCCESS') self.assertEqual( [u'snap_name', u'snapshot-2015.03.01'], sorted(sl.snapshots) ) def test_success_exclusive(self): client = Mock() client.snapshot.get.return_value = testvars.inprogress client.snapshot.get_repository.return_value = testvars.test_repo sl = curator.SnapshotList(client, repository=testvars.repo_name) sl.filter_by_state(state='SUCCESS', exclude=True) self.assertEqual([u'snapshot-2015.03.01'], sorted(sl.snapshots)) def test_invalid_state(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo sl = curator.SnapshotList(client, repository=testvars.repo_name) self.assertRaises(ValueError, sl.filter_by_state, state='invalid') class TestSnapshotListRegexFilters(TestCase): def test_filter_by_regex_prefix(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo sl = curator.SnapshotList(client, repository=testvars.repo_name) self.assertEqual( [u'snap_name', u'snapshot-2015.03.01'], sorted(sl.snapshots) ) sl.filter_by_regex(kind='prefix', value='sna') self.assertEqual( [u'snap_name', u'snapshot-2015.03.01'], sorted(sl.snapshots) ) sl.filter_by_regex(kind='prefix', value='sna', exclude=True) self.assertEqual([], sl.snapshots) def test_filter_by_regex_prefix_exclude(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo sl = curator.SnapshotList(client, repository=testvars.repo_name) self.assertEqual( [u'snap_name', u'snapshot-2015.03.01'], sorted(sl.snapshots) ) sl.filter_by_regex(kind='prefix', value='snap_', exclude=True) self.assertEqual([u'snapshot-2015.03.01'], sl.snapshots) def test_filter_by_regex_timestring(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo sl = curator.SnapshotList(client, repository=testvars.repo_name) self.assertEqual( [u'snap_name', u'snapshot-2015.03.01'], sorted(sl.snapshots) ) sl.filter_by_regex(kind='timestring', value='%Y.%m.%d') self.assertEqual( [u'snapshot-2015.03.01'], sorted(sl.snapshots) ) sl.filter_by_regex(kind='timestring', value='%Y.%m.%d', exclude=True) self.assertEqual([], sl.snapshots) def test_filter_by_regex_no_value(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo sl = curator.SnapshotList(client, repository=testvars.repo_name) self.assertEqual( [u'snap_name', u'snapshot-2015.03.01'], sorted(sl.snapshots) ) self.assertRaises(ValueError, sl.filter_by_regex, kind='prefix', value=None) self.assertEqual( [u'snap_name', u'snapshot-2015.03.01'], sorted(sl.snapshots) ) sl.filter_by_regex(kind='prefix', value=0) self.assertEqual([], sl.snapshots) def test_filter_by_regex_bad_kind(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo sl = curator.SnapshotList(client, repository=testvars.repo_name) self.assertEqual( [u'snap_name', u'snapshot-2015.03.01'], sorted(sl.snapshots) ) self.assertRaises( ValueError, sl.filter_by_regex, kind='invalid', value=None) class TestSnapshotListFilterByAge(TestCase): def test_filter_by_age_missing_direction(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo sl = curator.SnapshotList(client, repository=testvars.repo_name) self.assertRaises(curator.MissingArgument, sl.filter_by_age, unit='days', unit_count=1 ) def test_filter_by_age_bad_direction(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo sl = curator.SnapshotList(client, repository=testvars.repo_name) self.assertRaises(ValueError, sl.filter_by_age, unit='days', unit_count=1, direction="invalid" ) def test_filter_by_age_invalid_source(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo sl = curator.SnapshotList(client, repository=testvars.repo_name) self.assertRaises(ValueError, sl.filter_by_age, unit='days', source='invalid', unit_count=1, direction="older" ) def test_filter_by_age__name_no_timestring(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo sl = curator.SnapshotList(client, repository=testvars.repo_name) self.assertRaises(curator.MissingArgument, sl.filter_by_age, source='name', unit='days', unit_count=1, direction='older' ) def test_filter_by_age__name_older_than_now(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo sl = curator.SnapshotList(client, repository=testvars.repo_name) sl.filter_by_age(source='name', direction='older', timestring='%Y.%m.%d', unit='days', unit_count=1 ) self.assertEqual(['snapshot-2015.03.01'], sl.snapshots) def test_filter_by_age__name_younger_than_now(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo sl = curator.SnapshotList(client, repository=testvars.repo_name) sl.filter_by_age(source='name', direction='younger', timestring='%Y.%m.%d', unit='days', unit_count=1 ) self.assertEqual([], sl.snapshots) def test_filter_by_age__name_younger_than_past_date(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo sl = curator.SnapshotList(client, repository=testvars.repo_name) sl.filter_by_age(source='name', direction='younger', timestring='%Y.%m.%d', unit='seconds', unit_count=0, epoch=1422748800 ) self.assertEqual(['snapshot-2015.03.01'], sl.snapshots) def test_filter_by_age__name_older_than_past_date(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo sl = curator.SnapshotList(client, repository=testvars.repo_name) sl.filter_by_age(source='name', direction='older', timestring='%Y.%m.%d', unit='seconds', unit_count=0, epoch=1456963200 ) self.assertEqual(['snapshot-2015.03.01'], sl.snapshots) def test_filter_by_age__creation_date_older_than_now(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo sl = curator.SnapshotList(client, repository=testvars.repo_name) sl.filter_by_age(direction='older', unit='days', unit_count=1) self.assertEqual( ['snap_name', 'snapshot-2015.03.01'], sorted(sl.snapshots)) def test_filter_by_age__creation_date_younger_than_now(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo sl = curator.SnapshotList(client, repository=testvars.repo_name) sl.filter_by_age(direction='younger', timestring='%Y.%m.%d', unit='days', unit_count=1 ) self.assertEqual([], sl.snapshots) def test_filter_by_age__creation_date_younger_than_past_date(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo sl = curator.SnapshotList(client, repository=testvars.repo_name) sl.filter_by_age(direction='younger', timestring='%Y.%m.%d', unit='seconds', unit_count=0, epoch=1422748801 ) self.assertEqual(['snapshot-2015.03.01'], sl.snapshots) def test_filter_by_age__creation_date_older_than_past_date(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo sl = curator.SnapshotList(client, repository=testvars.repo_name) sl.filter_by_age(direction='older', timestring='%Y.%m.%d', unit='seconds', unit_count=0, epoch=1425168001 ) self.assertEqual(['snap_name'], sl.snapshots) class TestIterateFiltersSnaps(TestCase): def test_no_filters(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo slo = curator.SnapshotList(client, repository=testvars.repo_name) slo.iterate_filters({}) self.assertEqual( ['snap_name', 'snapshot-2015.03.01'], sorted(slo.snapshots) ) def test_no_filtertype(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo slo = curator.SnapshotList(client, repository=testvars.repo_name) config = {'filters': [{'no_filtertype':'fail'}]} self.assertRaises( curator.ConfigurationError, slo.iterate_filters, config) def test_invalid_filtertype_class(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo slo = curator.SnapshotList(client, repository=testvars.repo_name) config = {'filters': [{'filtertype':12345.6789}]} self.assertRaises( curator.ConfigurationError, slo.iterate_filters, config) def test_invalid_filtertype(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo slo = curator.SnapshotList(client, repository=testvars.repo_name) config = yaml.load(testvars.invalid_ft)['actions'][1] self.assertRaises( curator.ConfigurationError, slo.iterate_filters, config ) def test_age_filtertype(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo slo = curator.SnapshotList(client, repository=testvars.repo_name) config = yaml.load(testvars.snap_age_ft)['actions'][1] slo.iterate_filters(config) self.assertEqual( ['snap_name', 'snapshot-2015.03.01'], sorted(slo.snapshots)) def test_pattern_filtertype(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo slo = curator.SnapshotList(client, repository=testvars.repo_name) config = yaml.load(testvars.snap_pattern_ft)['actions'][1] slo.iterate_filters(config) self.assertEqual( ['snap_name', 'snapshot-2015.03.01'], sorted(slo.snapshots)) def test_none_filtertype(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo slo = curator.SnapshotList(client, repository=testvars.repo_name) config = yaml.load(testvars.snap_none_ft)['actions'][1] slo.iterate_filters(config) self.assertEqual( ['snap_name', 'snapshot-2015.03.01'], sorted(slo.snapshots)) class TestSnapshotListFilterCount(TestCase): def test_missing_count(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo slo = curator.SnapshotList(client, repository=testvars.repo_name) self.assertRaises(curator.MissingArgument, slo.filter_by_count) def test_without_age(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo slo = curator.SnapshotList(client, repository=testvars.repo_name) slo.filter_by_count(count=1) self.assertEqual(['snap_name'], slo.snapshots) def test_without_age_reversed(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo slo = curator.SnapshotList(client, repository=testvars.repo_name) slo.filter_by_count(count=1, reverse=False) self.assertEqual(['snapshot-2015.03.01'], slo.snapshots) def test_with_age(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo slo = curator.SnapshotList(client, repository=testvars.repo_name) slo.filter_by_count( count=1, source='creation_date', use_age=True ) self.assertEqual(['snap_name'], slo.snapshots) def test_with_age_reversed(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo slo = curator.SnapshotList(client, repository=testvars.repo_name) slo.filter_by_count( count=1, source='creation_date', use_age=True, reverse=False ) self.assertEqual(['snapshot-2015.03.01'], slo.snapshots) def test_sort_by_age(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo slo = curator.SnapshotList(client, repository=testvars.repo_name) slo._calculate_ages() slo.age_keyfield = 'invalid' snaps = slo.snapshots slo._sort_by_age(snaps) self.assertEqual(['snapshot-2015.03.01'], slo.snapshots) curator-4.2.5/test/unit/test_cli_methods.py000066400000000000000000000050231302700154000210320ustar00rootroot00000000000000import sys import logging from unittest import TestCase from mock import Mock, patch, mock_open import elasticsearch import curator from curator import _version as __version__ from . import CLITestCase # Get test variables and constants from a single source from . import testvars as testvars class TestCLI_A(TestCase): def test_read_file_no_file(self): self.assertRaises(TypeError, curator.read_file) def test_loginfo_defaults(self): loginfo = curator.LogInfo({}) self.assertEqual(20, loginfo.numeric_log_level) self.assertEqual(testvars.default_format, loginfo.format_string) def test_loginfo_debug(self): loginfo = curator.LogInfo({"loglevel": "DEBUG"}) self.assertEqual(10, loginfo.numeric_log_level) self.assertEqual(testvars.debug_format, loginfo.format_string) def test_loginfo_bad_level_raises(self): self.assertRaises( ValueError, curator.LogInfo, {"loglevel": "NOTALOGLEVEL"} ) def test_loginfo_logstash_formatter(self): loginfo = curator.LogInfo({"logformat": "logstash"}) logging.root.addHandler(loginfo.handler) logging.root.setLevel(loginfo.numeric_log_level) logger = logging.getLogger('testing') logger.info('testing') self.assertEqual(20, loginfo.numeric_log_level) def test_client_options_certificate(self): a = {'use_ssl':True, 'certificate':'invalid_path'} self.assertRaises( curator.FailedExecution, curator.test_client_options, a ) def test_client_options_client_cert(self): a = {'use_ssl':True, 'client_cert':'invalid_path'} self.assertRaises( curator.FailedExecution, curator.test_client_options, a ) def test_client_options_client_key(self): a = {'use_ssl':True, 'client_key':'invalid_path'} self.assertRaises( curator.FailedExecution, curator.test_client_options, a ) class TestCLI_B(CLITestCase): def test_read_file_pass(self): cfg = curator.get_yaml(self.args['yamlfile']) self.assertEqual('localhost', cfg['client']['hosts']) self.assertEqual(9200, cfg['client']['port']) def test_read_file_corrupt_fail(self): self.assertRaises(curator.ConfigurationError, curator.get_yaml, self.args['invalid_yaml']) def test_read_file_missing_fail(self): self.assertRaises( curator.FailedExecution, curator.read_file, self.args['no_file_here'] ) curator-4.2.5/test/unit/test_utils.py000066400000000000000000000601621302700154000177050ustar00rootroot00000000000000from datetime import datetime, timedelta from unittest import TestCase from mock import Mock import elasticsearch import yaml from . import testvars as testvars import curator class TestEnsureList(TestCase): def test_ensure_list_returns_lists(self): l = ["a", "b", "c", "d"] e = ["a", "b", "c", "d"] self.assertEqual(e, curator.ensure_list(l)) l = "abcd" e = ["abcd"] self.assertEqual(e, curator.ensure_list(l)) l = [["abcd","defg"], 1, 2, 3] e = [["abcd","defg"], 1, 2, 3] self.assertEqual(e, curator.ensure_list(l)) l = {"a":"b", "c":"d"} e = [{"a":"b", "c":"d"}] self.assertEqual(e, curator.ensure_list(l)) class TestTo_CSV(TestCase): def test_to_csv_will_return_csv(self): l = ["a", "b", "c", "d"] c = "a,b,c,d" self.assertEqual(c, curator.to_csv(l)) def test_to_csv_will_return_single(self): l = ["a"] c = "a" self.assertEqual(c, curator.to_csv(l)) def test_to_csv_will_return_None(self): l = [] self.assertIsNone(curator.to_csv(l)) class TestCheckCSV(TestCase): def test_check_csv_positive(self): c = "1,2,3" self.assertTrue(curator.check_csv(c)) def test_check_csv_negative(self): c = "12345" self.assertFalse(curator.check_csv(c)) def test_check_csv_list(self): l = ["1", "2", "3"] self.assertTrue(curator.check_csv(l)) def test_check_csv_unicode(self): u = u'test' self.assertFalse(curator.check_csv(u)) def test_check_csv_wrong_value(self): v = 123 self.assertRaises(TypeError, curator.check_csv, v ) class TestGetVersion(TestCase): def test_positive(self): client = Mock() client.info.return_value = {'version': {'number': '9.9.9'} } version = curator.get_version(client) self.assertEqual(version, (9,9,9)) def test_negative(self): client = Mock() client.info.return_value = {'version': {'number': '9.9.9'} } version = curator.get_version(client) self.assertNotEqual(version, (8,8,8)) def test_dev_version_4_dots(self): client = Mock() client.info.return_value = {'version': {'number': '9.9.9.dev'} } version = curator.get_version(client) self.assertEqual(version, (9,9,9)) def test_dev_version_with_dash(self): client = Mock() client.info.return_value = {'version': {'number': '9.9.9-dev'} } version = curator.get_version(client) self.assertEqual(version, (9,9,9)) class TestIsMasterNode(TestCase): def test_positive(self): client = Mock() client.nodes.info.return_value = { 'nodes': { "foo" : "bar"} } client.cluster.state.return_value = { "master_node" : "foo" } self.assertTrue(curator.is_master_node(client)) def test_negative(self): client = Mock() client.nodes.info.return_value = { 'nodes': { "bad" : "mojo"} } client.cluster.state.return_value = { "master_node" : "foo" } self.assertFalse(curator.is_master_node(client)) class TestGetIndexTime(TestCase): def test_get_datetime(self): for text, datestring, dt in [ ('2014.01.19', '%Y.%m.%d', datetime(2014, 1, 19)), ('14.01.19', '%y.%m.%d', datetime(2014, 1, 19)), ('2014-01-19', '%Y-%m-%d', datetime(2014, 1, 19)), ('2010-12-29', '%Y-%m-%d', datetime(2010, 12, 29)), ('2012-12', '%Y-%m', datetime(2012, 12, 1)), ('2011.01', '%Y.%m', datetime(2011, 1, 1)), ('2014-28', '%Y-%W', datetime(2014, 7, 14)), ('2014-28', '%Y-%U', datetime(2014, 7, 14)), ('2010.12.29.12', '%Y.%m.%d.%H', datetime(2010, 12, 29, 12)), ('2009101112136', '%Y%m%d%H%M%S', datetime(2009, 10, 11, 12, 13, 6)), ('2016-03-30t16', '%Y-%m-%dt%H', datetime(2016, 3, 30, 16, 0)), ]: self.assertEqual(dt, curator.get_datetime(text, datestring)) class TestGetDateRegex(TestCase): def test_non_escaped(self): self.assertEqual( '\\d{4}\\-\\d{2}\\-\\d{2}t\\d{2}', curator.get_date_regex('%Y-%m-%dt%H') ) class TestFixEpoch(TestCase): def test_fix_epoch(self): for long_epoch, epoch in [ (1459287636999, 1459287636), (1459287636000000, 1459287636), (145928763600000000, 1459287636), (145928763600000001, 1459287636), (1459287636123456789, 1459287636), (1459287636999, 1459287636), ]: self.assertEqual(epoch, curator.fix_epoch(long_epoch)) def test_fix_epoch_raise(self): self.assertRaises(ValueError, curator.fix_epoch, 12345678901) class TestGetPointOfReference(TestCase): def test_get_point_of_reference(self): epoch = 1459288037 for unit, result in [ ('seconds', epoch-1), ('minutes', epoch-60), ('hours', epoch-3600), ('days', epoch-86400), ('weeks', epoch-(86400*7)), ('months', epoch-(86400*30)), ('years', epoch-(86400*365)), ]: self.assertEqual(result, curator.get_point_of_reference(unit, 1, epoch)) def test_get_por_raise(self): self.assertRaises(ValueError, curator.get_point_of_reference, 'invalid', 1) class TestByteSize(TestCase): def test_byte_size(self): size = 3*1024*1024*1024*1024*1024*1024*1024 unit = ['Z','E','P','T','G','M','K',''] for i in range(0,7): self.assertEqual('3.0{0}B'.format(unit[i]), curator.byte_size(size)) size /= 1024 def test_byte_size_yotta(self): size = 3*1024*1024*1024*1024*1024*1024*1024*1024 self.assertEqual('3.0YB', curator.byte_size(size)) def test_raise_invalid(self): self.assertRaises(TypeError, curator.byte_size, 'invalid') class TestChunkIndexList(TestCase): def test_big_list(self): indices = [] for i in range(100,150): indices.append("superlongindexnamebyanystandardyouchoosethisissillyhowbigcanthisgetbeforeitbreaks" + str(i)) self.assertEqual(2, len(curator.chunk_index_list(indices))) def test_small_list(self): self.assertEqual(1, len(curator.chunk_index_list(['short','list','of','indices']))) class TestGetIndices(TestCase): def test_client_exception(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.indices.get_settings.side_effect = testvars.fake_fail self.assertRaises( curator.FailedExecution, curator.get_indices, client) def test_positive(self): client = Mock() client.indices.get_settings.return_value = testvars.settings_two client.info.return_value = {'version': {'number': '2.4.1'} } self.assertEqual( ['index-2016.03.03', 'index-2016.03.04'], sorted(curator.get_indices(client)) ) def test_empty(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = {} self.assertEqual([], curator.get_indices(client)) def test_issue_826(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.2'} } client.indices.get_settings.return_value = testvars.settings_two client.indices.exists.return_value = True self.assertEqual( ['.security', 'index-2016.03.03', 'index-2016.03.04'], sorted(curator.get_indices(client)) ) class TestCheckVersion(TestCase): def test_check_version_(self): client = Mock() client.info.return_value = {'version': {'number': '2.2.0'} } self.assertIsNone(curator.check_version(client)) def test_check_version_less_than(self): client = Mock() client.info.return_value = {'version': {'number': '0.90.3'} } self.assertRaises(curator.CuratorException, curator.check_version, client) def test_check_version_greater_than(self): client = Mock() client.info.return_value = {'version': {'number': '6.0.1'} } self.assertRaises(curator.CuratorException, curator.check_version, client) class TestCheckMaster(TestCase): def test_check_master_positive(self): client = Mock() client.nodes.info.return_value = { 'nodes': { "foo" : "bar"} } client.cluster.state.return_value = { "master_node" : "foo" } self.assertIsNone(curator.check_master(client, master_only=True)) def test_check_master_negative(self): client = Mock() client.nodes.info.return_value = { 'nodes': { "bad" : "mojo"} } client.cluster.state.return_value = { "master_node" : "foo" } with self.assertRaises(SystemExit) as cm: curator.check_master(client, master_only=True) self.assertEqual(cm.exception.code, 0) class TestGetClient(TestCase): # These unit test cases can't really get a client object, so it's more for # code coverage than anything def test_url_prefix_none(self): kwargs = { 'url_prefix': None, 'use_ssl' : True, 'ssl_no_validate' : True } self.assertRaises( elasticsearch.ElasticsearchException, curator.get_client, **kwargs ) def test_url_prefix_none_str(self): kwargs = { 'url_prefix': 'None', 'use_ssl' : True, 'ssl_no_validate' : True } self.assertRaises( elasticsearch.ElasticsearchException, curator.get_client, **kwargs ) def test_master_only_multiple_hosts(self): kwargs = { 'url_prefix': '', 'master_only' : True, 'hosts' : ['127.0.0.1', '127.0.0.1'] } self.assertRaises( curator.ConfigurationError, curator.get_client, **kwargs ) def test_host_with_hosts(self): kwargs = { 'url_prefix': '', 'host' : '127.0.0.1', 'hosts' : ['127.0.0.2'], } self.assertRaises( curator.ConfigurationError, curator.get_client, **kwargs ) def test_certificate_logic(self): kwargs = { 'use_ssl' : True, 'certificate' : 'mycert.pem' } self.assertRaises( elasticsearch.ElasticsearchException, curator.get_client, **kwargs ) def test_client_cert_logic(self): kwargs = { 'use_ssl' : True, 'client_cert' : 'myclientcert.pem' } self.assertRaises( elasticsearch.ElasticsearchException, curator.get_client, **kwargs ) def test_client_key_logic(self): kwargs = { 'use_ssl' : True, 'client_key' : 'myclientkey.pem' } self.assertRaises( elasticsearch.ElasticsearchException, curator.get_client, **kwargs ) def test_certificate_no_verify_logic(self): kwargs = { 'use_ssl' : True, 'ssl_no_validate' : True } self.assertRaises( elasticsearch.ElasticsearchException, curator.get_client, **kwargs ) class TestOverrideTimeout(TestCase): def test_no_change(self): self.assertEqual(30, curator.override_timeout(30, 'delete')) def test_forcemerge_snapshot(self): self.assertEqual(21600, curator.override_timeout(30, 'forcemerge')) self.assertEqual(21600, curator.override_timeout(30, 'snapshot')) def test_sync_flush(self): self.assertEqual(180, curator.override_timeout(30, 'sync_flush')) def test_invalid_action(self): self.assertEqual(30, curator.override_timeout(30, 'invalid')) def test_invalid_timeout(self): self.assertRaises(TypeError, curator.override_timeout('invalid', 'delete')) class TestShowDryRun(TestCase): # For now, since it's a pain to capture logging output, this is just a # simple code coverage run def test_index_list(self): client = Mock() client.info.return_value = {'version': {'number': '2.4.1'} } client.indices.get_settings.return_value = testvars.settings_two client.cluster.state.return_value = testvars.clu_state_two client.indices.stats.return_value = testvars.stats_two client.field_stats.return_value = testvars.fieldstats_two il = curator.IndexList(client) self.assertIsNone(curator.show_dry_run(il, 'test_action')) class TestGetRepository(TestCase): def test_get_repository_missing_arg(self): client = Mock() client.snapshot.get_repository.return_value = {} self.assertEqual({}, curator.get_repository(client)) def test_get_repository_positive(self): client = Mock() client.snapshot.get_repository.return_value = testvars.test_repo self.assertEqual(testvars.test_repo, curator.get_repository(client, repository=testvars.repo_name)) def test_get_repository_transporterror_negative(self): client = Mock() client.snapshot.get_repository.side_effect = elasticsearch.TransportError self.assertFalse(curator.get_repository(client, repository=testvars.repo_name)) def test_get_repository_notfounderror_negative(self): client = Mock() client.snapshot.get_repository.side_effect = elasticsearch.NotFoundError self.assertFalse(curator.get_repository(client, repository=testvars.repo_name)) def test_get_repository__all_positive(self): client = Mock() client.snapshot.get_repository.return_value = testvars.test_repos self.assertEqual(testvars.test_repos, curator.get_repository(client)) class TestGetSnapshot(TestCase): def test_get_snapshot_missing_repository_arg(self): client = Mock() self.assertRaises( curator.MissingArgument, curator.get_snapshot, client, snapshot=testvars.snap_name ) def test_get_snapshot_positive(self): client = Mock() client.snapshot.get.return_value = testvars.snapshot self.assertEqual(testvars.snapshot, curator.get_snapshot(client, repository=testvars.repo_name, snapshot=testvars.snap_name)) def test_get_snapshot_transporterror_negative(self): client = Mock() client.snapshot.get_repository.return_value = testvars.test_repo client.snapshot.get.side_effect = testvars.four_oh_one self.assertRaises( curator.FailedExecution, curator.get_snapshot, client, repository=testvars.repo_name, snapshot=testvars.snap_name ) def test_get_snapshot_notfounderror_negative(self): client = Mock() client.snapshot.get_repository.return_value = testvars.test_repo client.snapshot.get.side_effect = elasticsearch.NotFoundError(404, 'Snapshot not found') self.assertRaises( curator.FailedExecution, curator.get_snapshot, client, repository=testvars.repo_name, snapshot=testvars.snap_name ) class TestGetSnapshotData(TestCase): def test_missing_repo_arg(self): client = Mock() self.assertRaises(curator.MissingArgument, curator.get_snapshot_data, client) def test_return_data(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo self.assertEqual( testvars.snapshots['snapshots'], curator.get_snapshot_data(client, repository=testvars.repo_name) ) def test_raises_exception_onfail(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get.side_effect = testvars.four_oh_one client.snapshot.get_repository.return_value = testvars.test_repo self.assertRaises( curator.FailedExecution, curator.get_snapshot_data, client, repository=testvars.repo_name ) class TestSnapshotInProgress(TestCase): def test_all_snapshots_for_in_progress(self): client = Mock() client.snapshot.get.return_value = testvars.inprogress client.snapshot.get_repository.return_value = testvars.test_repo self.assertEqual( 'snapshot-2015.03.01', curator.snapshot_in_progress(client, repository=testvars.repo_name) ) def test_specified_snapshot_in_progress(self): client = Mock() client.snapshot.get.return_value = testvars.inprogress client.snapshot.get_repository.return_value = testvars.test_repo self.assertEqual( 'snapshot-2015.03.01', curator.snapshot_in_progress( client, repository=testvars.repo_name, snapshot='snapshot-2015.03.01' ) ) def test_specified_snapshot_in_progress_negative(self): client = Mock() client.snapshot.get.return_value = testvars.inprogress client.snapshot.get_repository.return_value = testvars.test_repo self.assertFalse( curator.snapshot_in_progress( client, repository=testvars.repo_name, snapshot=testvars.snap_name ) ) def test_all_snapshots_for_in_progress_negative(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo self.assertFalse( curator.snapshot_in_progress(client, repository=testvars.repo_name) ) def test_for_multiple_in_progress(self): client = Mock() client.snapshot.get.return_value = testvars.highly_unlikely client.snapshot.get_repository.return_value = testvars.test_repo self.assertRaises( curator.CuratorException, curator.snapshot_in_progress, client, repository=testvars.repo_name ) class TestCreateSnapshotBody(TestCase): def test_create_snapshot_body_empty_arg(self): self.assertFalse(curator.create_snapshot_body([])) def test_create_snapshot_body__all_positive(self): self.assertEqual(testvars.snap_body_all, curator.create_snapshot_body('_all')) def test_create_snapshot_body_positive(self): self.assertEqual(testvars.snap_body, curator.create_snapshot_body(testvars.named_indices)) class TestCreateRepoBody(TestCase): def test_missing_repo_type(self): self.assertRaises(curator.MissingArgument, curator.create_repo_body ) def test_s3(self): body = curator.create_repo_body(repo_type='s3') self.assertEqual(body['type'], 's3') class TestCreateRepository(TestCase): def test_missing_arg(self): client = Mock() self.assertRaises(curator.MissingArgument, curator.create_repository, client ) def test_empty_result_call(self): client = Mock() client.snapshot.get_repository.return_value = None self.assertTrue(curator.create_repository(client, repository="repo", repo_type="fs")) def test_repo_not_in_results(self): client = Mock() client.snapshot.get_repository.return_value = {'not_your_repo':{'foo':'bar'}} self.assertTrue(curator.create_repository(client, repository="repo", repo_type="fs")) def test_repo_already_in_results(self): client = Mock() client.snapshot.get_repository.return_value = {'repo':{'foo':'bar'}} self.assertRaises(curator.FailedExecution, curator.create_repository, client, repository="repo", repo_type="fs" ) def test_raises_exception(self): client = Mock() client.snapshot.get_repository.return_value = {'not_your_repo':{'foo':'bar'}} client.snapshot.create_repository.side_effect = elasticsearch.TransportError(500, "Error message", {"message":"Error"}) self.assertRaises(curator.FailedExecution, curator.create_repository, client, repository="repo", repo_type="fs") class TestRepositoryExists(TestCase): def test_missing_arg(self): client = Mock() self.assertRaises(curator.MissingArgument, curator.repository_exists, client ) def test_repository_in_results(self): client = Mock() client.snapshot.get_repository.return_value = {'repo':{'foo':'bar'}} self.assertTrue(curator.repository_exists(client, repository="repo")) def test_repo_not_in_results(self): client = Mock() client.snapshot.get_repository.return_value = {'not_your_repo':{'foo':'bar'}} self.assertFalse(curator.repository_exists(client, repository="repo")) class TestRepositoryFs(TestCase): def test_passing(self): client = Mock() client.snapshot.verify_repository.return_value = testvars.verified_nodes self.assertIsNone( curator.test_repo_fs(client, repository=testvars.repo_name)) def test_raises_404(self): client = Mock() client.snapshot.verify_repository.return_value = testvars.verified_nodes client.snapshot.verify_repository.side_effect = testvars.four_oh_four self.assertRaises(curator.ActionError, curator.test_repo_fs, client, repository=testvars.repo_name) def test_raises_401(self): client = Mock() client.snapshot.verify_repository.return_value = testvars.verified_nodes client.snapshot.verify_repository.side_effect = testvars.four_oh_one self.assertRaises(curator.ActionError, curator.test_repo_fs, client, repository=testvars.repo_name) def test_raises_other(self): client = Mock() client.snapshot.verify_repository.return_value = testvars.verified_nodes client.snapshot.verify_repository.side_effect = testvars.fake_fail self.assertRaises(curator.ActionError, curator.test_repo_fs, client, repository=testvars.repo_name) class TestSafeToSnap(TestCase): def test_missing_arg(self): client = Mock() self.assertRaises(curator.MissingArgument, curator.safe_to_snap, client ) def test_in_progress_fail(self): client = Mock() client.snapshot.get.return_value = testvars.inprogress client.snapshot.get_repository.return_value = testvars.test_repo self.assertFalse( curator.safe_to_snap( client, repository=testvars.repo_name, retry_interval=0, retry_count=1 ) ) def test_in_progress_pass(self): client = Mock() client.snapshot.get.return_value = testvars.snapshots client.snapshot.get_repository.return_value = testvars.test_repo self.assertTrue( curator.safe_to_snap( client, repository=testvars.repo_name, retry_interval=0, retry_count=1 ) ) class TestSnapshotRunning(TestCase): def test_true(self): client = Mock() client.snapshot.status.return_value = testvars.snap_running self.assertTrue(curator.snapshot_running(client)) def test_false(self): client = Mock() client.snapshot.status.return_value = testvars.nosnap_running self.assertFalse(curator.snapshot_running(client)) def test_raises_exception(self): client = Mock() client.snapshot.status.return_value = testvars.nosnap_running client.snapshot.status.side_effect = testvars.fake_fail self.assertRaises( curator.FailedExecution, curator.snapshot_running, client) class TestPruneNones(TestCase): def test_prune_nones_with(self): self.assertEqual({}, curator.prune_nones({'a':None})) def test_prune_nones_without(self): a = {'foo':'bar'} self.assertEqual(a, curator.prune_nones(a)) class TestValidateFilters(TestCase): def test_snapshot_with_index_filter(self): self.assertRaises( curator.ConfigurationError, curator.validate_filters, 'delete_snapshots', [{'filtertype': 'kibana'}] ) def test_index_with_snapshot_filter(self): self.assertRaises( curator.ConfigurationError, curator.validate_filters, 'delete_indices', [{'filtertype': 'state', 'state': 'SUCCESS'}] ) curator-4.2.5/test/unit/test_validators.py000066400000000000000000000151631302700154000207160ustar00rootroot00000000000000import sys import logging from unittest import TestCase from mock import Mock, patch, mock_open from voluptuous import * import curator def shared_result(config, action): return curator.validators.SchemaCheck( config, Schema(curator.validators.filters.Filters(action)), 'filters', 'testing' ).result() class TestFilters(TestCase): def test_single_raises_configuration_error(self): data = {'max_num_segments': 1, 'exclude': True} self.assertRaises( curator.ConfigurationError, curator.validators.filters.single, 'forcemerge', data ) class TestFilterTypes(TestCase): def test_alias(self): action = 'delete_indices' config = [ { 'filtertype' : 'alias', 'aliases' : ['alias1', 'alias2'], 'exclude' : False, } ] self.assertEqual(config, shared_result(config, action)) def test_age(self): action = 'delete_indices' config = [ { 'filtertype' : 'age', 'direction' : 'older', 'unit' : 'days', 'unit_count' : 1, 'source' : 'field_stats', 'field' : '@timestamp', } ] self.assertEqual(config, shared_result(config, action)) def test_age_with_string_unit_count(self): action = 'delete_indices' config = [ { 'filtertype' : 'age', 'direction' : 'older', 'unit' : 'days', 'unit_count' : "1", 'source' : 'field_stats', 'field' : '@timestamp', } ] result = shared_result(config, action) self.assertEqual(1, result[0]['unit_count']) def test_allocated(self): action = 'delete_indices' config = [ { 'filtertype' : 'allocated', 'key' : 'foo', 'value' : 'bar', 'allocation_type' : 'require', 'exclude' : False, } ] self.assertEqual(config, shared_result(config, action)) def test_closed(self): action = 'delete_indices' config = [ { 'filtertype' : 'closed', 'exclude' : False, } ] self.assertEqual(config, shared_result(config, action)) def test_count(self): action = 'delete_indices' config = [ { 'filtertype' : 'count', 'count' : 1, 'reverse' : True, 'exclude' : False, } ] self.assertEqual(config, shared_result(config, action)) def test_forcemerged(self): action = 'delete_indices' config = [ { 'filtertype' : 'forcemerged', 'max_num_segments' : 1, 'exclude' : False, } ] self.assertEqual(config, shared_result(config, action)) def test_kibana(self): action = 'delete_indices' config = [ { 'filtertype' : 'kibana', 'exclude' : False, } ] self.assertEqual(config, shared_result(config, action)) def test_opened(self): action = 'delete_indices' config = [ { 'filtertype' : 'opened', 'exclude' : False, } ] self.assertEqual(config, shared_result(config, action)) def test_space_name_age(self): action = 'delete_indices' config = [ { 'filtertype' : 'space', 'disk_space' : 1, 'use_age' : True, 'exclude' : False, 'source' : 'name', 'timestring' : '%Y.%m.%d', } ] self.assertEqual(config, shared_result(config, action)) def test_space_name_age_string_float(self): action = 'delete_indices' config = [ { 'filtertype' : 'space', 'disk_space' : "1.0", 'use_age' : True, 'exclude' : False, 'source' : 'name', 'timestring' : '%Y.%m.%d', } ] result = shared_result(config, action) self.assertEqual(1.0, result[0]['disk_space']) def test_space_name_age_no_ts(self): action = 'delete_indices' config = [ { 'filtertype' : 'space', 'disk_space' : 1, 'use_age' : True, 'exclude' : False, 'source' : 'name', } ] schema = curator.validators.SchemaCheck( config, Schema(curator.validators.filters.Filters(action)), 'filters', 'testing' ) self.assertRaises(curator.ConfigurationError, schema.result) def test_space_field_stats_age(self): action = 'delete_indices' config = [ { 'filtertype' : 'space', 'disk_space' : 1, 'use_age' : True, 'exclude' : False, 'source' : 'field_stats', 'field' : '@timestamp', } ] self.assertEqual(config, shared_result(config, action)) def test_space_field_stats_age_no_field(self): action = 'delete_indices' config = [ { 'filtertype' : 'space', 'disk_space' : 1, 'use_age' : True, 'exclude' : False, 'source' : 'field_stats', } ] schema = curator.validators.SchemaCheck( config, Schema(curator.validators.filters.Filters(action)), 'filters', 'testing' ) self.assertRaises(curator.ConfigurationError, schema.result) def test_space_creation_date_age(self): action = 'delete_indices' config = [ { 'filtertype' : 'space', 'disk_space' : 1, 'use_age' : True, 'exclude' : False, 'source' : 'creation_date', } ] self.assertEqual(config, shared_result(config, action)) def test_state(self): action = 'delete_snapshots' config = [ { 'filtertype' : 'state', 'state' : 'SUCCESS', 'exclude' : False, } ] self.assertEqual(config, shared_result(config, action)) curator-4.2.5/test/unit/testvars.py000066400000000000000000000656111302700154000173650ustar00rootroot00000000000000import elasticsearch from voluptuous import * fake_fail = Exception('Simulated Failure') four_oh_one = elasticsearch.TransportError(401, "simulated error") four_oh_four = elasticsearch.TransportError(404, "simulated error") get_alias_fail = elasticsearch.NotFoundError(404, "simulated error") named_index = 'index_name' named_indices = [ "index-2015.01.01", "index-2015.02.01" ] open_index = {'metadata': {'indices' : { named_index : {'state' : 'open'}}}} closed_index = {'metadata': {'indices' : { named_index : {'state' : 'close'}}}} cat_open_index = [{'status': 'open'}] cat_closed_index = [{'status': 'close'}] open_indices = { 'metadata': { 'indices' : { 'index1' : { 'state' : 'open' }, 'index2' : { 'state' : 'open' }}}} closed_indices = { 'metadata': { 'indices' : { 'index1' : { 'state' : 'close' }, 'index2' : { 'state' : 'close' }}}} named_alias = 'alias_name' alias_retval = { "pre_aliased_index": { "aliases" : { named_alias : { }}}} aliases_retval = { "index1": { "aliases" : { named_alias : { } } }, "index2": { "aliases" : { named_alias : { } } }, } alias_one_add = [{'add': {'alias': 'alias', 'index': 'index_name'}}] alias_one_add_with_extras = [ { 'add': { 'alias': 'alias', 'index': 'index_name', 'filter' : { 'term' : { 'user' : 'kimchy' }} } }] alias_one_rm = [{'remove': {'alias': 'alias', 'index': 'index_name'}}] alias_one_body = { "actions" : [ {'remove': {'alias': 'alias', 'index': 'index_name'}}, {'add': {'alias': 'alias', 'index': 'index_name'}} ]} alias_two_add = [ {'add': {'alias': 'alias', 'index': 'index-2016.03.03'}}, {'add': {'alias': 'alias', 'index': 'index-2016.03.04'}}, ] alias_two_rm = [ {'remove': {'alias': 'alias', 'index': 'index-2016.03.03'}}, {'remove': {'alias': 'alias', 'index': 'index-2016.03.04'}}, ] alias_success = { "acknowledged": True } allocation_in = {named_index: {'settings': {'index': {'routing': {'allocation': {'require': {'foo': 'bar'}}}}}}} allocation_out = {named_index: {'settings': {'index': {'routing': {'allocation': {'require': {'not': 'foo'}}}}}}} indices_space = { 'indices' : { 'index1' : { 'index' : { 'primary_size_in_bytes': 1083741824 }}, 'index2' : { 'index' : { 'primary_size_in_bytes': 1083741824 }}}} snap_name = 'snap_name' repo_name = 'repo_name' test_repo = {repo_name: {'type': 'fs', 'settings': {'compress': 'true', 'location': '/tmp/repos/repo_name'}}} test_repos = {'TESTING': {'type': 'fs', 'settings': {'compress': 'true', 'location': '/tmp/repos/TESTING'}}, repo_name: {'type': 'fs', 'settings': {'compress': 'true', 'location': '/rmp/repos/repo_name'}}} snap_running = { 'snapshots': ['running'] } nosnap_running = { 'snapshots': [] } snapshot = { 'snapshots': [ { 'duration_in_millis': 60000, 'start_time': '2015-02-01T00:00:00.000Z', 'shards': {'successful': 4, 'failed': 0, 'total': 4}, 'end_time_in_millis': 0, 'state': 'SUCCESS', 'snapshot': snap_name, 'end_time': '2015-02-01T00:00:01.000Z', 'indices': named_indices, 'failures': [], 'start_time_in_millis': 1422748800 }]} partial = { 'snapshots': [ { 'duration_in_millis': 60000, 'start_time': '2015-02-01T00:00:00.000Z', 'shards': {'successful': 4, 'failed': 0, 'total': 4}, 'end_time_in_millis': 0, 'state': 'PARTIAL', 'snapshot': snap_name, 'end_time': '2015-02-01T00:00:01.000Z', 'indices': named_indices, 'failures': [], 'start_time_in_millis': 1422748800 }]} snapshots = { 'snapshots': [ { 'duration_in_millis': 60000, 'start_time': '2015-02-01T00:00:00.000Z', 'shards': {'successful': 4, 'failed': 0, 'total': 4}, 'end_time_in_millis': 0, 'state': 'SUCCESS', 'snapshot': snap_name, 'end_time': '2015-02-01T00:00:01.000Z', 'indices': named_indices, 'failures': [], 'start_time_in_millis': 1422748800 }, { 'duration_in_millis': 60000, 'start_time': '2015-03-01T00:00:02.000Z', 'shards': {'successful': 4, 'failed': 0, 'total': 4}, 'end_time_in_millis': 0, 'state': 'SUCCESS', 'snapshot': 'snapshot-2015.03.01', 'end_time': '2015-03-01T00:00:03.000Z', 'indices': named_indices, 'failures': [], 'start_time_in_millis': 1425168002 }]} inprogress = { 'snapshots': [ { 'duration_in_millis': 60000, 'start_time': '2015-02-01T00:00:00.000Z', 'shards': {'successful': 4, 'failed': 0, 'total': 4}, 'end_time_in_millis': 0, 'state': 'SUCCESS', 'snapshot': snap_name, 'end_time': '2015-02-01T00:00:01.000Z', 'indices': named_indices, 'failures': [], 'start_time_in_millis': 1422748800 }, { 'duration_in_millis': 60000, 'start_time': '2015-03-01T00:00:02.000Z', 'shards': {'successful': 4, 'failed': 0, 'total': 4}, 'end_time_in_millis': 0, 'state': 'IN_PROGRESS', 'snapshot': 'snapshot-2015.03.01', 'end_time': '2015-03-01T00:00:03.000Z', 'indices': named_indices, 'failures': [], 'start_time_in_millis': 1425168002 }]} highly_unlikely = { 'snapshots': [ { 'duration_in_millis': 60000, 'start_time': '2015-02-01T00:00:00.000Z', 'shards': {'successful': 4, 'failed': 0, 'total': 4}, 'end_time_in_millis': 0, 'state': 'IN_PROGRESS', 'snapshot': snap_name, 'end_time': '2015-02-01T00:00:01.000Z', 'indices': named_indices, 'failures': [], 'start_time_in_millis': 1422748800 }, { 'duration_in_millis': 60000, 'start_time': '2015-03-01T00:00:02.000Z', 'shards': {'successful': 4, 'failed': 0, 'total': 4}, 'end_time_in_millis': 0, 'state': 'IN_PROGRESS', 'snapshot': 'snapshot-2015.03.01', 'end_time': '2015-03-01T00:00:03.000Z', 'indices': named_indices, 'failures': [], 'start_time_in_millis': 1425168002 }]} snap_body_all = { "ignore_unavailable": False, "include_global_state": True, "partial": False, "indices" : "_all" } snap_body = { "ignore_unavailable": False, "include_global_state": True, "partial": False, "indices" : "index-2015.01.01,index-2015.02.01" } verified_nodes = {'nodes': {'nodeid1': {'name': 'node1'}, 'nodeid2': {'name': 'node2'}}} synced_pass = { "_shards":{"total":1,"successful":1,"failed":0}, "index_name":{ "total":1,"successful":1,"failed":0, "failures":[], } } synced_fail = { "_shards":{"total":1,"successful":0,"failed":1}, "index_name":{ "total":1,"successful":0,"failed":1, "failures":[ {"shard":0,"reason":"pending operations","routing":{"state":"STARTED","primary":True,"node":"nodeid1","relocating_node":None,"shard":0,"index":"index_name"}}, ] } } sync_conflict = elasticsearch.ConflictError(409, u'{"_shards":{"total":1,"successful":0,"failed":1},"index_name":{"total":1,"successful":0,"failed":1,"failures":[{"shard":0,"reason":"pending operations","routing":{"state":"STARTED","primary":true,"node":"nodeid1","relocating_node":null,"shard":0,"index":"index_name"}}]}})', synced_fail) synced_fails = { "_shards":{"total":2,"successful":1,"failed":1}, "index1":{ "total":1,"successful":0,"failed":1, "failures":[ {"shard":0,"reason":"pending operations","routing":{"state":"STARTED","primary":True,"node":"nodeid1","relocating_node":None,"shard":0,"index":"index_name"}}, ] }, "index2":{ "total":1,"successful":1,"failed":0, "failures":[] }, } settings_one = { named_index: { u'state': u'open', u'aliases': [u'my_alias'], u'mappings': {}, u'settings': { u'index': { u'number_of_replicas': u'1', u'uuid': u'random_uuid_string_here', u'number_of_shards': u'2', u'creation_date': u'1456963200172', u'routing': {u'allocation': {u'include': {u'tag': u'foo'}}}, u'version': {u'created': u'2020099'}, u'refresh_interval': u'5s' } } } } settings_two = { u'index-2016.03.03': { u'state': u'open', u'aliases': [u'my_alias'], u'mappings': {}, u'settings': { u'index': { u'number_of_replicas': u'1', u'uuid': u'random_uuid_string_here', u'number_of_shards': u'5', u'creation_date': u'1456963200172', u'routing': {u'allocation': {u'include': {u'tag': u'foo'}}}, u'version': {u'created': u'2020099'}, u'refresh_interval': u'5s' } } }, u'index-2016.03.04': { u'state': u'open', u'aliases': [u'my_alias'], u'mappings': {}, u'settings': { u'index': { u'number_of_replicas': u'1', u'uuid': u'another_random_uuid_string', u'number_of_shards': u'5', u'creation_date': u'1457049600812', u'routing': {u'allocation': {u'include': {u'tag': u'bar'}}}, u'version': {u'created': u'2020099'}, u'refresh_interval': u'5s' } } } } settings_2_get_aliases = { "index-2016.03.03": { "aliases" : { 'my_alias' : { } } }, "index-2016.03.04": { "aliases" : { 'my_alias' : { } } }, } settings_2_closed = { u'index-2016.03.03': { u'state': u'close', u'aliases': [u'my_alias'], u'mappings': {}, u'settings': { u'index': { u'number_of_replicas': u'1', u'uuid': u'random_uuid_string_here', u'number_of_shards': u'5', u'creation_date': u'1456963200172', u'routing': {u'allocation': {u'include': {u'tag': u'foo'}}}, u'version': {u'created': u'2020099'}, u'refresh_interval': u'5s' } } }, u'index-2016.03.04': { u'state': u'open', u'aliases': [u'my_alias'], u'mappings': {}, u'settings': { u'index': { u'number_of_replicas': u'1', u'uuid': u'another_random_uuid_string', u'number_of_shards': u'5', u'creation_date': u'1457049600812', u'routing': {u'allocation': {u'include': {u'tag': u'bar'}}}, u'version': {u'created': u'2020099'}, u'refresh_interval': u'5s' } } } } settings_two_no_cd = { u'index-2016.03.03': { u'state': u'open', u'aliases': [u'my_alias'], u'mappings': {}, u'settings': { u'index': { u'number_of_replicas': u'1', u'uuid': u'random_uuid_string_here', u'number_of_shards': u'5', u'creation_date': u'1456963200172', u'routing': {u'allocation': {u'include': {u'tag': u'foo'}}}, u'version': {u'created': u'2020099'}, u'refresh_interval': u'5s' } } }, u'index-2016.03.04': { u'state': u'open', u'aliases': [u'my_alias'], u'mappings': {}, u'settings': { u'index': { u'number_of_replicas': u'1', u'uuid': u'another_random_uuid_string', u'number_of_shards': u'5', u'routing': {u'allocation': {u'include': {u'tag': u'bar'}}}, u'version': {u'created': u'2020099'}, u'refresh_interval': u'5s' } } } } settings_four = { u'a-2016.03.03': { u'state': u'open', u'aliases': [u'my_alias'], u'mappings': {}, u'settings': { u'index': { u'number_of_replicas': u'1', u'uuid': u'random_uuid_string_here', u'number_of_shards': u'5', u'creation_date': u'1456963200172', u'routing': {u'allocation': {u'include': {u'tag': u'foo'}}}, u'version': {u'created': u'2020099'}, u'refresh_interval': u'5s' } } }, u'b-2016.03.04': { u'state': u'open', u'aliases': [u'my_alias'], u'mappings': {}, u'settings': { u'index': { u'number_of_replicas': u'1', u'uuid': u'another_random_uuid_string', u'number_of_shards': u'5', u'creation_date': u'1457049600812', u'routing': {u'allocation': {u'include': {u'tag': u'bar'}}}, u'version': {u'created': u'2020099'}, u'refresh_interval': u'5s' } } }, u'c-2016.03.05': { u'state': u'close', u'aliases': [u'my_alias'], u'mappings': {}, u'settings': { u'index': { u'number_of_replicas': u'1', u'uuid': u'random_uuid_string_here', u'number_of_shards': u'5', u'creation_date': u'1457136000933', u'routing': {u'allocation': {u'include': {u'tag': u'foo'}}}, u'version': {u'created': u'2020099'}, u'refresh_interval': u'5s' } } }, u'd-2016.03.06': { u'state': u'open', u'aliases': [u'my_alias'], u'mappings': {}, u'settings': { u'index': { u'number_of_replicas': u'1', u'uuid': u'another_random_uuid_string', u'number_of_shards': u'5', u'creation_date': u'1457222400527', u'routing': {u'allocation': {u'include': {u'tag': u'bar'}}}, u'version': {u'created': u'2020099'}, u'refresh_interval': u'5s' } } } } settings_named = { u'index-2015.01.01': { u'state': u'open', u'aliases': [u'my_alias'], u'mappings': {}, u'settings': { u'index': { u'number_of_replicas': u'1', u'uuid': u'random_uuid_string_here', u'number_of_shards': u'5', u'creation_date': u'1456963200172', u'routing': {u'allocation': {u'include': {u'tag': u'foo'}}}, u'version': {u'created': u'2020099'}, u'refresh_interval': u'5s' } } }, u'index-2015.02.01': { u'state': u'open', u'aliases': [u'my_alias'], u'mappings': {}, u'settings': { u'index': { u'number_of_replicas': u'1', u'uuid': u'another_random_uuid_string', u'number_of_shards': u'5', u'creation_date': u'1457049600812', u'routing': {u'allocation': {u'include': {u'tag': u'bar'}}}, u'version': {u'created': u'2020099'}, u'refresh_interval': u'5s' } } } } clu_state_one = { u'metadata': { u'indices': settings_one } } clu_state_two = { u'metadata': { u'indices': settings_two } } cs_two_closed = { u'metadata': { u'indices': settings_2_closed } } clu_state_two_no_cd = { u'metadata': { u'indices': settings_two_no_cd } } clu_state_four = { u'metadata': { u'indices': settings_four } } stats_one = { u'indices': { named_index : { u'total': { u'docs': {u'count': 6374962, u'deleted': 0}, u'store': {u'size_in_bytes': 1115219663, u'throttle_time_in_millis': 0} }, u'primaries': { u'docs': {u'count': 3187481, u'deleted': 0}, u'store': {u'size_in_bytes': 557951789, u'throttle_time_in_millis': 0} } } } } stats_two = { u'indices': { u'index-2016.03.03': { u'total': { u'docs': {u'count': 6374962, u'deleted': 0}, u'store': {u'size_in_bytes': 1115219663, u'throttle_time_in_millis': 0} }, u'primaries': { u'docs': {u'count': 3187481, u'deleted': 0}, u'store': {u'size_in_bytes': 557951789, u'throttle_time_in_millis': 0} } }, u'index-2016.03.04': { u'total': { u'docs': {u'count': 6377544, u'deleted': 0}, u'store': {u'size_in_bytes': 1120891046, u'throttle_time_in_millis': 0} }, u'primaries': { u'docs': {u'count': 3188772, u'deleted': 0}, u'store': {u'size_in_bytes': 560677114, u'throttle_time_in_millis': 0} } } } } stats_four = { u'indices': { u'a-2016.03.03': { u'total': { u'docs': {u'count': 6374962, u'deleted': 0}, u'store': {u'size_in_bytes': 1115219663, u'throttle_time_in_millis': 0} }, u'primaries': { u'docs': {u'count': 3187481, u'deleted': 0}, u'store': {u'size_in_bytes': 557951789, u'throttle_time_in_millis': 0} } }, u'b-2016.03.04': { u'total': { u'docs': {u'count': 6377544, u'deleted': 0}, u'store': {u'size_in_bytes': 1120891046, u'throttle_time_in_millis': 0} }, u'primaries': { u'docs': {u'count': 3188772, u'deleted': 0}, u'store': {u'size_in_bytes': 560677114, u'throttle_time_in_millis': 0} } }, # CLOSED, ergo, not present # u'c-2016.03.05': { # u'total': { # u'docs': {u'count': 6266434, u'deleted': 0}, # u'store': {u'size_in_bytes': 1120882166, u'throttle_time_in_millis': 0} # }, # u'primaries': { # u'docs': {u'count': 3133217, u'deleted': 0}, # u'store': {u'size_in_bytes': 560441083, u'throttle_time_in_millis': 0} # } # }, u'd-2016.03.06': { u'total': { u'docs': {u'count': 6266436, u'deleted': 0}, u'store': {u'size_in_bytes': 1120882168, u'throttle_time_in_millis': 0} }, u'primaries': { u'docs': {u'count': 3133218, u'deleted': 0}, u'store': {u'size_in_bytes': 560441084, u'throttle_time_in_millis': 0} } } } } fieldstats_one = { u'indices': { named_index : { u'fields': { u'timestamp': { u'density': 100, u'min_value_as_string': u'2016-03-03T00:00:06.189Z', u'max_value': 1457049599152, u'max_doc': 415651, u'min_value': 1456963206189, u'doc_count': 415651, u'max_value_as_string': u'2016-03-03T23:59:59.152Z', u'sum_total_term_freq': -1, u'sum_doc_freq': 1662604}}}} } fieldstats_two = { u'indices': { u'index-2016.03.03': { u'fields': { u'timestamp': { u'density': 100, u'min_value_as_string': u'2016-03-03T00:00:06.189Z', u'max_value': 1457049599152, u'max_doc': 415651, u'min_value': 1456963206189, u'doc_count': 415651, u'max_value_as_string': u'2016-03-03T23:59:59.152Z', u'sum_total_term_freq': -1, u'sum_doc_freq': 1662604}}}, u'index-2016.03.04': { u'fields': { u'timestamp': { u'density': 100, u'min_value_as_string': u'2016-03-04T00:00:00.812Z', u'max_value': 1457135999223, u'max_doc': 426762, u'min_value': 1457049600812, u'doc_count': 426762, u'max_value_as_string': u'2016-03-04T23:59:59.223Z', u'sum_total_term_freq': -1, u'sum_doc_freq': 1673715}}}, } } fieldstats_four = { u'indices': { u'a-2016.03.03': { u'fields': { u'timestamp': { u'density': 100, u'min_value_as_string': u'2016-03-03T00:00:06.189Z', u'max_value': 1457049599152, u'max_doc': 415651, u'min_value': 1456963206189, u'doc_count': 415651, u'max_value_as_string': u'2016-03-03T23:59:59.152Z', u'sum_total_term_freq': -1, u'sum_doc_freq': 1662604}}}, u'b-2016.03.04': { u'fields': { u'timestamp': { u'density': 100, u'min_value_as_string': u'2016-03-04T00:00:00.812Z', u'max_value': 1457135999223, u'max_doc': 426762, u'min_value': 1457049600812, u'doc_count': 426762, u'max_value_as_string': u'2016-03-04T23:59:59.223Z', u'sum_total_term_freq': -1, u'sum_doc_freq': 1673715}}}, u'd-2016.03.06': { u'fields': { u'timestamp': { u'density': 100, u'min_value_as_string': u'2016-03-04T00:00:00.812Z', u'max_value': 1457308799223, u'max_doc': 426762, u'min_value': 1457222400567, u'doc_count': 426762, u'max_value_as_string': u'2016-03-04T23:59:59.223Z', u'sum_total_term_freq': -1, u'sum_doc_freq': 1673715}}}, } } shards = { 'indices': { named_index: { 'shards': { '0': [ { 'num_search_segments' : 15 }, { 'num_search_segments' : 21 } ], '1': [ { 'num_search_segments' : 19 }, { 'num_search_segments' : 16 } ] }}}} fm_shards = { 'indices': { named_index: { 'shards': { '0': [ { 'num_search_segments' : 1 }, { 'num_search_segments' : 1 } ], '1': [ { 'num_search_segments' : 1 }, { 'num_search_segments' : 1 } ] }}}} loginfo = { "loglevel": "INFO", "logfile": None, "logformat": "default" } default_format = '%(asctime)s %(levelname)-9s %(message)s' debug_format = '%(asctime)s %(levelname)-9s %(name)22s %(funcName)22s:%(lineno)-4d %(message)s' yamlconfig = ''' --- # Remember, leave a key empty to use the default value. None will be a string, # not a Python "NoneType" client: hosts: localhost port: 9200 url_prefix: use_ssl: False certificate: client_cert: client_key: ssl_no_validate: False http_auth: timeout: 30 master_only: False options: dry_run: False loglevel: DEBUG logfile: logformat: default quiet: False ''' pattern_ft = ''' --- actions: 1: description: open all matching indices action: open options: continue_if_exception: False disable_action: False filters: - filtertype: pattern kind: prefix value: a exclude: False ''' age_ft = ''' --- actions: 1: description: open all matching indices action: open options: continue_if_exception: False disable_action: False filters: - filtertype: age source: name direction: older timestring: '%Y.%m.%d' unit: seconds unit_count: 0 epoch: 1456963201 ''' space_ft = ''' --- actions: 1: description: open all matching indices action: open options: continue_if_exception: False disable_action: False filters: - filtertype: space disk_space: 2.1 source: name use_age: True timestring: '%Y.%m.%d' ''' forcemerge_ft = ''' --- actions: 1: description: open all matching indices action: open options: continue_if_exception: False disable_action: False filters: - filtertype: forcemerged max_num_segments: 2 ''' allocated_ft = ''' --- actions: 1: description: open all matching indices action: open options: continue_if_exception: False disable_action: False filters: - filtertype: allocated key: tag value: foo allocation_type: include ''' kibana_ft = ''' --- actions: 1: description: open all matching indices action: open options: continue_if_exception: False disable_action: False filters: - filtertype: kibana ''' opened_ft = ''' --- actions: 1: description: open all matching indices action: open options: continue_if_exception: False disable_action: False filters: - filtertype: opened ''' closed_ft = ''' --- actions: 1: description: open all matching indices action: open options: continue_if_exception: False disable_action: False filters: - filtertype: closed ''' none_ft = ''' --- actions: 1: description: open all matching indices action: open options: continue_if_exception: False disable_action: False filters: - filtertype: none ''' invalid_ft = ''' --- actions: 1: description: open all matching indices action: open options: continue_if_exception: False disable_action: False filters: - filtertype: sir_not_appearing_in_this_film ''' snap_age_ft = ''' --- actions: 1: description: test action: delete_snapshots options: continue_if_exception: False disable_action: False filters: - filtertype: age direction: older unit: days unit_count: 1 ''' snap_pattern_ft= ''' --- actions: 1: description: test action: delete_snapshots options: continue_if_exception: False disable_action: False filters: - filtertype: pattern kind: prefix value: sna ''' snap_none_ft = ''' --- actions: 1: description: test action: delete_snapshots options: continue_if_exception: False disable_action: False filters: - filtertype: none ''' curator-4.2.5/travis-run.sh000077500000000000000000000027041302700154000156450ustar00rootroot00000000000000#!/bin/bash set -ex # There's at least 1 expected, skipped test, only with 5.0.0-alpha4 right now expected_skips=1 setup_es() { download_url=$1 curl -sL $download_url > elasticsearch.tar.gz mkdir elasticsearch tar -xzf elasticsearch.tar.gz --strip-components=1 -C ./elasticsearch/. } start_es() { jhome=$1 es_args=$2 export JAVA_HOME=$jhome elasticsearch/bin/elasticsearch $es_args > /tmp/elasticsearch.log & sleep 10 curl http://localhost:9200 && echo "ES is up!" || cat /tmp/elasticsearch.log } if [[ "$ES_VERSION" == 5.* ]]; then setup_es https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-$ES_VERSION.tar.gz else setup_es https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch/$ES_VERSION/elasticsearch-$ES_VERSION.tar.gz fi java_home='/usr/lib/jvm/java-8-oracle' if [[ "$ES_VERSION" == 5.* ]]; then start_es $java_home '-d -Edefault.path.repo=/' else start_es $java_home '-d -Des.path.repo=/' fi python setup.py test result=$(head -1 nosetests.xml | awk '{print $6 " " $7 " " $8}' | awk -F\> '{print $1}' | tr -d '"') echo "Result = $result" errors=$(echo $result | awk '{print $1}' | awk -F\= '{print $2}') failures=$(echo $result | awk '{print $2}' | awk -F\= '{print $2}') skips=$(echo $result | awk '{print $3}' | awk -F\= '{print $2}') if [[ $errors -gt 0 ]]; then exit 1 elif [[ $failures -gt 0 ]]; then exit 1 elif [[ $skips -gt $expected_skips ]]; then exit 1 fi curator-4.2.5/unix_packages/000077500000000000000000000000001302700154000160125ustar00rootroot00000000000000curator-4.2.5/unix_packages/build_official_package.sh000077500000000000000000000117761302700154000227730ustar00rootroot00000000000000#!/bin/bash BASEPATH=$(pwd) PKG_TARGET=/curator_packages WORKDIR=/tmp/curator CX_VER="5.0" CX_FILE="/curator_source/unix_packages/cx_freeze-${CX_VER}.dev.tar.gz" CX_PATH="anthony_tuininga-cx_freeze-71554144c9cc" INPUT_TYPE=python CATEGORY=python VENDOR=Elastic MAINTAINER="'Elastic Developers '" C_POST_INSTALL=${WORKDIR}/es_curator_after_install.sh C_PRE_REMOVE=${WORKDIR}/es_curator_before_removal.sh C_POST_REMOVE=${WORKDIR}/es_curator_after_removal.sh C_PRE_UPGRADE=${WORKDIR}/es_curator_before_upgrade.sh C_POST_UPGRADE=${WORKDIR}/es_curator_after_upgrade.sh # Build our own package pre/post scripts sudo rm -rf ${WORKDIR} /opt/elasticsearch-curator mkdir -p ${WORKDIR} for file in ${C_POST_INSTALL} ${C_PRE_REMOVE} ${C_POST_REMOVE}; do echo '#!/bin/bash' > ${file} echo >> ${file} chmod +x ${file} done echo "ln -s /opt/elasticsearch-curator/curator /usr/bin/curator" >> ${C_POST_INSTALL} echo "ln -s /opt/elasticsearch-curator/curator_cli /usr/bin/curator_cli" >> ${C_POST_INSTALL} echo "ln -s /opt/elasticsearch-curator/es_repo_mgr /usr/bin/es_repo_mgr" >> ${C_POST_INSTALL} echo "ln -s /opt/elasticsearch-curator/curator /usr/bin/curator" >> ${C_POST_UPGRADE} echo "ln -s /opt/elasticsearch-curator/curator_cli /usr/bin/curator_cli" >> ${C_POST_UPGRADE} echo "ln -s /opt/elasticsearch-curator/es_repo_mgr /usr/bin/es_repo_mgr" >> ${C_POST_UPGRADE} echo "rm -f /usr/bin/curator" >> ${C_PRE_REMOVE} echo "rm -f /usr/bin/curator_cli" >> ${C_PRE_REMOVE} echo "rm -f /usr/bin/es_repo_mgr" >> ${C_PRE_REMOVE} echo "rm -f /usr/bin/curator" >> ${C_PRE_UPGRADE} echo "rm -f /usr/bin/curator_cli" >> ${C_PRE_UPGRADE} echo "rm -f /usr/bin/es_repo_mgr" >> ${C_PRE_UPGRADE} echo 'if [ -d "/opt/elasticsearch-curator" ]; then' >> ${C_POST_REMOVE} echo ' rm -rf /opt/elasticsearch-curator' >> ${C_POST_REMOVE} echo 'fi' >> ${C_POST_REMOVE} ID=$(grep ^ID\= /etc/*release | awk -F\= '{print $2}' | tr -d \") VERSION_ID=$(grep ^VERSION_ID\= /etc/*release | awk -F\= '{print $2}' | tr -d \") if [ "${ID}x" == "x" ]; then ID=$(cat /etc/*release | grep -v LSB | uniq | awk '{print $1}' | tr "[:upper:]" "[:lower:]" ) VERSION_ID=$(cat /etc/*release | grep -v LSB | uniq | awk '{print $3}' | awk -F\. '{print $1}') fi # build if [ "${1}x" == "x" ]; then echo "Must provide version number" exit 1 else FILE="v${1}.tar.gz" cd ${WORKDIR} wget https://github.com/elastic/curator/archive/${FILE} fi case "$ID" in ubuntu|debian) PKGTYPE=deb PLATFORM=debian PACKAGEDIR="${PKG_TARGET}/${1}/${PLATFORM}" ;; centos|rhel) PKGTYPE=rpm PLATFORM=centos case "$VERSION_ID" in 6|7) sudo rm -f /etc/yum.repos.d/puppetlabs-pc1.repo sudo yum -y update ;; *) echo "unknown system version: ${VERSION_ID}"; exit 1;; esac PACKAGEDIR="${PKG_TARGET}/${1}/${PLATFORM}/${VERSION_ID}" ;; *) echo "unknown system type: ${ID}"; exit 1;; esac HAS_PY3=$(which python3.5) if [ "${HAS_PY3}x" == "x" ]; then cd /tmp wget https://www.python.org/ftp/python/3.5.2/Python-3.5.2.tgz tar zxf Python-3.5.2.tgz cd Python-3.5.2 ./configure --prefix=/usr/local sudo make altinstall cd /usr/lib sudo ln -s /usr/local/lib/libpython3.5m.a libpython3.5.a cd ${WORKDIR} fi PYVER=3.5 PIPBIN=/usr/local/bin/pip3.5 PYBIN=/usr/local/bin/python3.5 if [ "${CX_VER}" != "$(${PIPBIN} list | grep cx | awk '{print $2}' | tr -d '()')" ]; then cd ${WORKDIR} rm -rf ${CX_PATH} tar zxf ${CX_FILE} cd ${CX_PATH} ${PIPBIN} install -U --user . cd ${WORKDIR} fi if [ -e "/home/vagrant/.rvm/scripts/rvm" ]; then source /home/vagrant/.rvm/scripts/rvm fi HAS_FPM=$(which fpm) if [ "${HAS_FPM}x" == "x" ]; then gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 curl -sSL https://get.rvm.io | bash -s stable source /home/vagrant/.rvm/scripts/rvm rvm install ruby gem install fpm fi tar zxf ${FILE} mkdir -p ${PACKAGEDIR} cd curator-${1} ${PIPBIN} install -U --user setuptools ${PIPBIN} install -U --user requests_aws4auth ${PIPBIN} install -U --user -r requirements.txt ${PYBIN} setup.py build_exe sudo mv build/exe.linux-x86_64-${PYVER} /opt/elasticsearch-curator sudo chown -R root:root /opt/elasticsearch-curator cd .. fpm \ -s dir \ -t ${PKGTYPE} \ -n elasticsearch-curator \ -v ${1} \ --vendor ${VENDOR} \ --maintainer "${MAINTAINER}" \ --license 'Apache-2.0' \ --category tools \ --description 'Have indices in Elasticsearch? This is the tool for you!\n\nLike a museum curator manages the exhibits and collections on display, \nElasticsearch Curator helps you curate, or manage your indices.' \ --after-install ${C_POST_INSTALL} \ --before-remove ${C_PRE_REMOVE} \ --after-remove ${C_POST_REMOVE} \ --before-upgrade ${C_PRE_UPGRADE} \ --after-upgrade ${C_POST_UPGRADE} \ --provides elasticsearch-curator \ --conflicts python-elasticsearch-curator \ --conflicts python3-elasticsearch-curator \ /opt/elasticsearch-curator mv ${WORKDIR}/*.${PKGTYPE} ${PACKAGEDIR} rm ${C_POST_INSTALL} ${C_PRE_REMOVE} ${C_POST_REMOVE} ${C_PRE_UPGRADE} ${C_POST_UPGRADE} # go back to where we started cd ${BASEPATH} curator-4.2.5/unix_packages/build_package_from_source.sh000077500000000000000000000117021302700154000235270ustar00rootroot00000000000000#!/bin/bash BASEPATH=$(pwd) PKG_TARGET=/curator_packages WORKDIR=/tmp/curator CX_VER="5.0" CX_FILE="/curator_source/unix_packages/cx_freeze-${CX_VER}.dev.tar.gz" CX_PATH="anthony_tuininga-cx_freeze-71554144c9cc" INPUT_TYPE=python CATEGORY=python VENDOR=Elastic MAINTAINER="'Elastic Developers '" C_POST_INSTALL=${WORKDIR}/es_curator_after_install.sh C_PRE_REMOVE=${WORKDIR}/es_curator_before_removal.sh C_POST_REMOVE=${WORKDIR}/es_curator_after_removal.sh C_PRE_UPGRADE=${WORKDIR}/es_curator_before_upgrade.sh C_POST_UPGRADE=${WORKDIR}/es_curator_after_upgrade.sh # Build our own package pre/post scripts sudo rm -rf ${WORKDIR} /opt/elasticsearch-curator mkdir -p ${WORKDIR} for file in ${C_POST_INSTALL} ${C_PRE_REMOVE} ${C_POST_REMOVE}; do echo '#!/bin/bash' > ${file} echo >> ${file} chmod +x ${file} done echo "ln -s /opt/elasticsearch-curator/curator /usr/bin/curator" >> ${C_POST_INSTALL} echo "ln -s /opt/elasticsearch-curator/curator_cli /usr/bin/curator_cli" >> ${C_POST_INSTALL} echo "ln -s /opt/elasticsearch-curator/es_repo_mgr /usr/bin/es_repo_mgr" >> ${C_POST_INSTALL} echo "ln -s /opt/elasticsearch-curator/curator /usr/bin/curator" >> ${C_POST_UPGRADE} echo "ln -s /opt/elasticsearch-curator/curator_cli /usr/bin/curator_cli" >> ${C_POST_UPGRADE} echo "ln -s /opt/elasticsearch-curator/es_repo_mgr /usr/bin/es_repo_mgr" >> ${C_POST_UPGRADE} echo "rm -f /usr/bin/curator" >> ${C_PRE_REMOVE} echo "rm -f /usr/bin/curator_cli" >> ${C_PRE_REMOVE} echo "rm -f /usr/bin/es_repo_mgr" >> ${C_PRE_REMOVE} echo "rm -f /usr/bin/curator" >> ${C_PRE_UPGRADE} echo "rm -f /usr/bin/curator_cli" >> ${C_PRE_UPGRADE} echo "rm -f /usr/bin/es_repo_mgr" >> ${C_PRE_UPGRADE} echo 'if [ -d "/opt/elasticsearch-curator" ]; then' >> ${C_POST_REMOVE} echo ' rm -rf /opt/elasticsearch-curator' >> ${C_POST_REMOVE} echo 'fi' >> ${C_POST_REMOVE} ID=$(grep ^ID\= /etc/*release | awk -F\= '{print $2}' | tr -d \") VERSION_ID=$(grep ^VERSION_ID\= /etc/*release | awk -F\= '{print $2}' | tr -d \") if [ "${ID}x" == "x" ]; then ID=$(cat /etc/*release | grep -v LSB | uniq | awk '{print $1}' | tr "[:upper:]" "[:lower:]" ) VERSION_ID=$(cat /etc/*release | grep -v LSB | uniq | awk '{print $3}' | awk -F\. '{print $1}') fi # build if [ "${1}x" == "x" ]; then echo "Must provide version number (can be arbitrary)" exit 1 else cd .. SOURCE_DIR=$(pwd) fi case "$ID" in ubuntu|debian) PKGTYPE=deb PLATFORM=debian PACKAGEDIR="${PKG_TARGET}/${1}/${PLATFORM}" ;; centos|rhel) PKGTYPE=rpm PLATFORM=centos case "$VERSION_ID" in 6|7) sudo rm -f /etc/yum.repos.d/puppetlabs-pc1.repo sudo yum -y update ;; *) echo "unknown system version: ${VERSION_ID}"; exit 1;; esac PACKAGEDIR="${PKG_TARGET}/${1}/${PLATFORM}/${VERSION_ID}" ;; *) echo "unknown system type: ${ID}"; exit 1;; esac HAS_PY3=$(which python3.5) if [ "${HAS_PY3}x" == "x" ]; then cd /tmp wget https://www.python.org/ftp/python/3.5.2/Python-3.5.2.tgz tar zxf Python-3.5.2.tgz cd Python-3.5.2 ./configure --prefix=/usr/local sudo make altinstall cd /usr/lib sudo ln -s /usr/local/lib/libpython3.5m.a libpython3.5.a cd ${WORKDIR} fi PYVER=3.5 PIPBIN=/usr/local/bin/pip3.5 PYBIN=/usr/local/bin/python3.5 if [ "${CX_VER}" != "$(${PIPBIN} list | grep cx | awk '{print $2}' | tr -d '()')" ]; then cd ${WORKDIR} rm -rf ${CX_PATH} tar zxf ${CX_FILE} cd ${CX_PATH} ${PIPBIN} install -U --user . cd ${WORKDIR} fi if [ -e "/home/vagrant/.rvm/scripts/rvm" ]; then source /home/vagrant/.rvm/scripts/rvm fi HAS_FPM=$(which fpm) if [ "${HAS_FPM}x" == "x" ]; then gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 curl -sSL https://get.rvm.io | bash -s stable source /home/vagrant/.rvm/scripts/rvm rvm install ruby gem install fpm fi cd $SOURCE_DIR mkdir -p ${PACKAGEDIR} ${PIPBIN} install -U --user setuptools ${PIPBIN} install -U --user requests_aws4auth ${PIPBIN} install -U --user -r requirements.txt ${PYBIN} setup.py build_exe sudo mv build/exe.linux-x86_64-${PYVER} /opt/elasticsearch-curator sudo chown -R root:root /opt/elasticsearch-curator cd $WORKDIR fpm \ -s dir \ -t ${PKGTYPE} \ -n elasticsearch-curator \ -v ${1} \ --vendor ${VENDOR} \ --maintainer "${MAINTAINER}" \ --license 'Apache-2.0' \ --category tools \ --description 'Have indices in Elasticsearch? This is the tool for you!\n\nLike a museum curator manages the exhibits and collections on display, \nElasticsearch Curator helps you curate, or manage your indices.' \ --after-install ${C_POST_INSTALL} \ --before-remove ${C_PRE_REMOVE} \ --after-remove ${C_POST_REMOVE} \ --before-upgrade ${C_PRE_UPGRADE} \ --after-upgrade ${C_POST_UPGRADE} \ --provides elasticsearch-curator \ --conflicts python-elasticsearch-curator \ --conflicts python3-elasticsearch-curator \ /opt/elasticsearch-curator mv ${WORKDIR}/*.${PKGTYPE} ${PACKAGEDIR} rm ${C_POST_INSTALL} ${C_PRE_REMOVE} ${C_POST_REMOVE} ${C_PRE_UPGRADE} ${C_POST_UPGRADE} # go back to where we started cd ${BASEPATH} curator-4.2.5/unix_packages/cx_freeze-5.0.dev.tar.gz000066400000000000000000002417231302700154000222010ustar00rootroot00000000000000oWk{F(<+#2he+Qd9HJ2=8 J=ߺt7 챞$ }{>?2`=O(%Z ߄}LO+m [{ކwicks{sOƟej8Ehxanћ O60hMD{!OgOnjh{xzscu4o`xh,ygQ݁3n?c؇ylt`~eY ?h]67t7 ڟ(~Ѩ;{|$Y'mgczg:EQ:n;_|͓4uoll>_<σ4uI-C}kUp0[Uчw'Wםd/pu|}/3t~sogþw77??o#mpgnw6Mhd# FƖ=~[o6lnmxgxk/ 6Q4DDou6L'xccfg2|1z3ٍv6Aafg. G;~& v`4v6 D(ܝMP^o￞no턻Im@69osgcooccd&诃N&;ffMC^ol[Qzfћhc<iG G0`lFh/ohuf4Bsnp wgou܉6&^30hc']yƛ`~0 V̈́&Ձs9mlM^OQ:cuf`o0m7;!n,|Q1tjlף׀c;${;o7$n<ލFۣxokggٍm5~-;[a6Cu4؉}ۣ&< Vgw'ڝGxs86AN^ v0n:½&ٰ[[ۯwFFyom?`gGpGQL3|8ga/CqӨa,_P3q/O޾?YUoG]L E}/^R0L`G f$t&I,1Oςx^<쌣 ?2N,?o>|>_\-:K) p{<7YÜ>H7(gDt͆gd LQ0À35D)pĉd% ~7[N;0tΰC%Da>4 T?$DžjwsK4q)0 ,Ou0[bk}+}bm~vGo:ۛh?v>Wn*lNp?b֩nBl|c;Qq.үQ&iV Q݀%{#$ mր/B(^x)>Q7^Wgw_p?`G~Q0mN,4-VEπJ $ϒet2Np6^U|:,Z*=hw[zk Sœ&!mEwQgˑ/]xv,5 KBifmp^,7TӚz U7,nï:\%ߑ~f8._f]w*-e? qS7۷}sH@8 ?xӞ U o!3/ZtBgI0]>Be*Uњv [h p#NhzୟoԮPÝuZ?߲wKQx7qAnܚc> 1EƘֳhA" )Do0E0xU rhRv9ˠUC'j4sBWzu@^= cd:,K\de.i@:%^jy>&<5QWZթryŏ8F {b)c$K&c-]^1&hXuؐW׌zM[C }+Q:c]88=%|$/+KtZ]m&HO7}1^Sh1k  2<d1Ȗ :D{ E=)$F\E`q]$EIg+n${8KЧ(MFufֆt^yocP\0H} At<;+;Bf\~3$<0YiSz;Ro5Q)xVȑduwJvn@.2+jC+^S iZfmP)z]i` 8Ɂe6oyr_6C$ 5tWWV(0%H 0\B7 Ǽ r&6q4Dp'c|KP ]_ݾZ n6sFt|2>VZwSǬG1FJV#g?L96"!I:H@e ǭ7(mlx▶*7r]bj^7Wܻ_Bts*b>3xCWM!67p@4Z;aqkqws{)]Ǎňt̀8f8[0>/:K|iI}+%=7m G I9R/C!idO+m 4 ~K(/:@k- Ro7&/!+Ab%E|,îmQ>%׽Uz(,^) lH(pzZu 8dJ#m'DR[jn *I{=6(<^Rm'XT=.%2˻F 1.@OŲpA-fcͱF>ijDV/BMt(_I\&fn\=]K`ܛa6 i$oi<^(Nt4 G$0\jV;@B=W=Q_Me2#.@ !C68VX4C,VT{,*߶=[uOj"a3#Ia_q2ha]lr>xN͚f_vG{XQ4(3Z8 mZ|TWOTog2,WnIQn2Fzʿ<}.GdwHqY.='~3wNNFڽJ&WzSeNx?2Wib-o2J:؃ Eܥn~<͆eR]/.x~H^ǿ Hr+ XΙBcV>i5*CCim|lh蜟,rkgS 4\8/&OC ւ_,eX!EIJjiA_"֒:)Z. VV-.~A[:(aI.u*oT&i0SeR%꤅.&v8vAr.k5K^66xX%:$]ƃ,_ NCѝDX\xB * Rl])x1[bHTЖU69W5 BG(;@pgU: Wm"4@jgV(ޫ}ckhߥ/z?7?a`n[ XK#qAu}ErN*0J|̟"vˡY$yNgWȹkb,nţmm Z?8 k2Yhyu}tYA˫ݑc#9j7Gg-j{~q {[pl EK E)9 n^G~"O\zkRNj0 U>eDυbXxpԿ(yێrW;i!? xi]\xۛ;޺3@sTm;;E~erA8YFD)}-(m S.;x1A[|O6N;v cp~Hg `PN/c;_F< Egoc:($Y##QAgLᶃ23S<Ƹb;N d20"PNFZsGh6Ӷ}=kD,3<]:8:E>! p>D),W=R*'X|n h$4k=êcyo EfMDx(BhlOӸ9R$LF%BAOH8δ1Oy`8]t.Wb.P>5%b :E}``h`baT_ :lIsn5 {GD./U="^%'5n<5L.F|c1޶ 74?< 9x)&0]EAϘ+XrSՌ Cy nUbP@G2Ēߞ^{7]>&wġaZ>1X..%rx+7`_=YF@I?jQ:rXvt8霏PEl\}xt41~\/G'EI'}Ե ?4X%?vjQ]fen-yfT+F? t+)v}k}hnsAFH4]Eω {H:^1 S"4;~e7\dz-n:u)8XQzsҙ|f]|AaGX׿7ެ:He1|ɝ/̇Hi'X5~>8Y,h*?'FZx=Aх?dѩլɖ}E7h"@=]}a10!Z >Ԝ m/K~ <@={G1WEݝ~dRPU3 *hLPÃ( ;E ʃ:M30exIj 5oͅWK:0nf)ѝQt<Ɖ@9OsR u/IW$4O4shҸ~7 ^=u=$}\N9`A]&hȗ lq}_taa4ۯm'+x^7..l,:a8NFd -~ƣJ6;)Q4&qjb:#(ah7U1<_ Vf 0}X XI3~gٰqa QzCVO@6({* eU3ZΙӡ (=;zLfg9[!d }%F!G [dqP FsPUT-($*fkJrMT=l8F˻0y7/[rkܞRzraa{H7vElPV$JlNHxZ jY)IJL0 lj\Ke;mBܾRwJ1p0#5i IXW1> <&&FuclFG| Q[dݴkxZ]0 AlK0$0R R}Gka YUwonK'W`ډGxaHW$ zHpʺK4 lf__UL%nfl 1DCGd26yv&C mݛhw[}:'N TLtz%R&&<TDT)i2UtlҔ6C[N&Gևd(RH-z}+5.ӺO5`oٴ"Z"R(K/A|c(ԕ)XP*Ud#5tݾ3AxsMȹڸ,5AY,BBg$E "*-} A9vb CN5[?t"%0^Չ'j}լcn$"vj0Kҧ$ly IJ6\4@ZgtwY)Mi@Wt 4{R >@^OI '4 >W ]GB eU8Y VV+qliTor%$qBo#n{:G`C|(u(ȕ#[Ya<$ңԷ(K\tzcOe 0 m wi˅٤fcĪ)>e埦 "AȱZ1kAt{N}I !wI5$IujX4.G!'ҐϜd ^YR eG"@:% j~ ZbZq[,>%ʑT~Ւ&<**Y{A%tZ^ࠋd/н!?*_2Ą[(w5`gxghc&́dPÃu86c+1aPP~:U_>Pڊ ?mvOWa#GBohw߄B詈CAfFI'7( Xʵ* r]<Ȣ҃Wu#cJR 609JYʱUJW5pf O~T"ŵbA-gR]%V+~ VW3Ɔlvu5&+ptE0ۋ# ]χ͚ZhZ|e-7ܿjyɡafcC=O?v-Q]eG (c,D&;sMJlEX^aP}um(.No&y^& lo~+f芃=W"Z+R_bfϻ.\%o lĞPWK]y"PL e)״蛚 D@O9oi3\A+tdk8.iovnd{UKvJh-Py[~h |9^{̔bR yMQV5>g|Rx(RP@Vn|cP4./-\.u Ac'N?.nmz/S:EQ$|nH@"˛P&pe`fP=kW~f{Yt3S-]]`nǩg$> vB\VZqqA&hLV$Q-d'BAC2bàѻ0Qe ͤ dy=@7N`_8hmJ -tdK:۵x"DpvmH™RS JmdCt*p21"< l>M(}~HqN!Dx *EGXDjGFbdCG߽ &]3!] eU[aS`׶A0p9a\Dp.Lx9FL+8&>dXM0w`V]&c"u&qo'G0R#P;0NMh~O聠~TLYr- _N> ~JګݙBVE&H˧4bQL /ʛ fȑt+l- Y$[KA ;V"\,O[[+ŵѤ&:Egp |(ѴF UQ .pTE*aX"^h}Ni*`9~D`vqQ<]Ne{B>yӻ%:b)P`9;_)jٶsb-'[9DDž& L讨,dhrr8_:pOox5(<퀷 ryb?+a6cW$Rvurb@o b쫁Ss=3& MYc0U -mO/š خ3էDU%_"z QH 7[+l۷y}Q$c7ZVF w#!G؈h.Bk΁#ݎ)>Tt啪 OҨWrC/q_tu>r Lq&:45ʚNR*%u4ijn} mFh _6V (ԅOn^3˅""{B6 z[kĀUS0G!lCQSQ_ ggAe Ly0bE 0圁LZTxZ `n?Y%( /14bxRcG*&Z8 ITW_թsdZe<)V *3;sx|>p%#֜XJ2R$+ΐD$ZqdHyQ ^#3g=Ea)K؇R1s!ɘhUdS)\8CHִj}aX7a):e2= ?b^T<y8=Cun3k/Bנ"6EibM]u7 aQ:3nVv0h.*v]ȸE)7\(aE-H*V^==3U1~iƘ UWOWR6ne w"ʄN(DDv ?`a ot)29&:hf NXk#"F:TdDUbe>Խ{3sg*N1`~e޸yST}|9ߪv=^f*.,F؉xv[#*N5$0F|޺[PbR቎v" /e7*YtX򨨉9sEIh{Yqr3ZnsPzh#wֵkZlhz-i:,p:d6ߖ5-c\I0Ŷ%b/Wpm Z/?cpCVm^ Uv6GDžJ@*(U^&q!9F)`"TּPkdHؚjhmMs_]=iW([_uookwϪ`?Qja}|ݧxŌ> W <~1$bĜ9[D!ߢ8fwa|A1))u9K~r}W +p--^JDП06Cyx~-1]DZ Hș CtH'j*#ɋKPh!é,4imk179fW}k"=5ɒK:?)] %͹ -|~D n\*Z1r A:њb.C[3,:y^604 aT,fAiP# 84 < g0.տ҉ jx˄kPiL q.DqᖕyAF9<} 9BJ2/<02-rŽ5byy>-Ҫ:sZdmtvfm-{w1q5GNǓ^$5i۝x:Ʒ7W(~Rk&f7]VwN_0l4(5_K% ~n`hw V&1@񗓏C8r_ ƴdB!\-|[rXB_Է!}MM584|f{ǵWڗWt>o5Q)? ,տVVwtGPGplba|Qd6BC5 m{{zz QFC [XjRG_U'`-hY$A'̵-ٛ.)$KF]T5uEB( y0[hQ۳yk@B{/o=ST,S@qRtU,e`HN3 G6QOU+]$q~ ?z*H W\L= :9jKk;"C)XIZnX7^yic1hs,5G!/Ga6BVdK}CrI+ +(DǽJ6;\۲P5CP"Hr{ rJ%0d,yd-1Ҏ*E [m1Q L5ΣeץA-TS ؕ튊Ԉ\h 3 ?z7N(1wB4#bGI0ُA_Uɮ8ƍ8ނ&ͷ٦ Up4VNRc4DA(_etRI߬C}ztQl|WZW*T m^2KDW˦_>|}J{2p,M3Luд"P~Ddʅ̥S*G"5$S`DL{A r+,xw+tOt W&✝+Q&aw UڌS&"HJOkvҚV8M7݋r TV,ZḦ́D*<ȗ;;׹|-t՘x;Y [IB=~C)1wPy4P1I-UH]m_RbjYXƣ;H*ܯ;k)b(.#&!6BhP%$!! ڂ2aTquoL~U|'## )ycƒ/۶BCF|ЧUЎd݁FXUnܑK K뛷rD߬Pd;X$9bFd$f'V':үbSDC=7e$ (Fy A_gAxJ' u|=s`̹f Z;Im;/1;ϳՀ|vgL#ȇ')#b# sqMc`ܷ+˼]yY8W 2:,JUwMu9~QMXAsTxSbG{iETp(#W.*(*pyyatE~;ݭ1O,8QȽ*mEڅ[-nZo,-p6\;mΛð[ %л=gݐsVpvQ\f)R"yYd3U?N/'PV?.Hq6\*d !ZO-4EZS#;BN栨a͞@ef;b!m i/t.ʢuR$g2r?-<N ໕5|PGJȼ2=,$ncಂuGc&_Wm,sjjdq,xU*Y^kWKsrWє˭JCfUV-NbM*MfPH+^X+)/*,[æ\ drL")Ҟ5+$gpTQޭUmd'(pE^OA.r3zL .% 䭒G*⤨p(/%)Is`oW' 7/gj(M\_cyxo۱iQR=l> :\,rel{&쁷a1ޗx9[dRؿF~sviD37>wR~aUEzaxUj-&Y+ǘ#YCVTW`O_xGNp#mDNI=[uٮSn7rX]9Sqz:-F7BF.#%:XяU X1L+MYƑb7 WAIiF[WQ~e3ҝAfjݥɲ(oGH˴&FKtʷ9F8(\Vt[bWGgce[P9?Fu A*5MFKg,S2P\ѳ+IT,lxpP aO"FQ!&N6W gVpݙp~ 3p/lB\D%nNqNO[h!mܷҥ{էBƮ2r:DI SXcGTJ36^U:1 "#oκ15lymhq*uP`H'jRky oX1)ci\֛NˇRb{I:$`˥t+&PC.42DHȻ6}Z%iP_Loc/H&ZX/ЂyēidS[*X/Y)x)5WP.-+t(J_h–6Di{;U++N_n +j!իqfgؤV_eӊYzy#`G.szgA<}ܫ/0}Iea cxm!nDž~&Fn >y8yK(H0қq4j Q5tjZ_5MTt]v[1ӭJM#XF^ [<*\:J}!4oH<x/߼|k1`::و4.A܈^]Wח%u}/vywMTˆ+ĖirW!t 8TjGD 4T9 ;( y2о'ː zΨ|zi4 Js*վ1f>bʯ f2׸D3̈́bjVQޚYg:(mjPoPu t~BA }G5#kac5 Ǎ9U(c((3nZ*nY(N]Y,P"[fu^T01L/- )٫e^e ®2ZKW6Z2զomdJW9E<BQ_Y˷U^23+7SX8CH*NhSp$Sy:b_ۑ9f}~g#`sx}c{aeB v6(\X$λ%>ن}o6bKM-X6Ohh y>W'w$ oln[6v7?#EkbJ3|` x*3Pl:`_4YF%X.N<2O)C-2 IE]߂ʉnu>B0]ċ,ҝ];qi.]4Y<ېcK"癊lw?[DFSaq|a?F˄}IluJ(y5#ff4yWm i4mEp]kg!`GKqeU: p['w#lq#MM4nrG#͛N53ضYU3ղuD|<FiMRL8~is|YlYv3m7GOS~ A{(W=2ɔ $:EVo3YԔOsЩg]f[9 ϙwG*o}4%! )M`OsoL͊2mAazIמ9h9Q '>*ˣ< -T4#`9L6J }:{uX&0$\:o w&Ėj_ApQޮp*U#'ue}DEۤV[vJ0WW; ~O9JsVPX <-{;Oߟෟ"[ɠG@q+D0? i]рm$3>en".hza]U>uM;v5xm<9z| CBkRt SN,5[㻫[PMQ0 (E $*Q o-Mw79=ϣiC+ɻoo[5t}ƧWۣoNڶ:9~}u#p3@VZw?\459ÏGgMmnjrqy+ۣQ'ڶy]eAL)ob/һfBmÕڸqMx_jysIh7M ?`4[u*9"xA<k|*@kg,i%?" <b#)AP֗")OkV{qԶ/]Er7|-& @AE֏ $ZSD>T5WWgw%i8=G`e(tXYzTlk"4n7ܲVlw\\7A54fkVTlnUl͡}z j,b.kfk-ճMãn1 \3di7)vfrTaSJ街OEbee>5CmZ}Ne<48[=.Gj sn6nx8s1:ԋc@XRR. [ftUgh?'Ukhg*&Σ G| A3nR\@GX;cơ+ R$MagZuH-3Z qButK; ͋Cbǰ rJ2MRySb`藦rZZ5JSYV@)&߃${bɮj|1/2fYV' RԦW&wHFmSnl8f/1-~d׋ d9jP 348 8x-^IinS?;X܏A7æ[-xY,n,\0%QUN 0O:x?$70Z:rN*uM[4iM)@ [kfڅQƳ嬾 Hc>_;A (k[hC멾: g?7~]m$X8{[%"E^ӑ!fz6/5]lD*\zg 7wǕfz2{΀'k69قR-]Z&?{YZ/rv!w- kQxp>j] YZ̆#`ҋ$ˁ9~M*âХ6%Qݵ$B" r`Y,pȝ`(,^\*,2BR1sϵ.f]`Q#]R]u,!{9ڝqZ )t՝e..c5۱:Qv9ǫz+svtߑ(wJqI0ZHJ6X ][D)E+ѲA$eF"91 qXC!Ӈ\oN#ɘXEL%Wv\&0ڻMB ."^h٠1Q *׊7"Hi\`Lc^f8 s6RO[z5KO[װJ.fd,$pLKW必m+B+4h*B.5c*} N4g4/EFH_D S,ÇiUl$@'0dup xp ~8uzf(rlKaax TV ^mYѾ5 W0t,tQ1ߑ  8:?#tvUȭ#}2w)6{[QklObɔps9\LS+q`(9Zo8ep!q MJ`xRDNݐ.W !Ltu}ٷhtbHׄ<aT] ^u*8Ym-VU ŗ嗺=I9n-pRŶLY9Yc!/8~癯 @?L @żS= 5\$cH~9qq@ۨVTDXMby%ɫ5E+/0ޥhf=LYS;*4Mei He@&G`e rPp t?,26C71&oKP傓L^=}J0b n vRK|Pi/y3X,5IdHCirǡXv߇lXgyZ:Mv`=G)^?1)u Qoᵫ?4(`ޥӀwlL-:j+: K d﫜s}U8 T34΀!hOV ߚ+[;1! V$|mK/D3Mi'BO7W1g/.@nZ1FpU覥/LiRC 3 [gĦU6RIS;#C4Bu`R Ij52ٷCQOUGTno..*hw_yF{]ֶǵeay#.%*F' ǸI+b@ׁwN\_?̒.Q"$`(RdXt24xZ\k:v/1am=NrM\Se)tT9t0˓i f֯@X`z;>MψV ^Xjwy_}|\lHߛV:T(60Fc煲 ΁$2HYz88FZqg޴JOxc֦)CcEvFEQ UkPOd Ὗe>VfH>oOWGj/p ]q>{5UNj"y(\p%\Y@(H{*-0t z9R| 0eF/`L=sM6c{Ib*XdRVbݗ]f`UkHx]m~y*#eŶ#jܜ[Y/ Gj}DC:G8 _%oA^c=_1gͺVρtȜ*vNOޛ0$m15h2 N9*1kH}x K 8D7Q݁:U10" 0_m!_qCEĆCѰu-gJI6`lQM3k]kTI:n l uieqfm70jL ĚÖ6̒YduHu9-Y2qvx9?kT z}F{}e${Vn|!\5tީ`YN i2U$PtMxJI#*kJS# оtRarErO^ KSLaSjUW@?ME7'WSiI[ & B` 4e4IEe<{ttƬhyi1z.KO'x WO7qCTMˆEcSU)w$-e@ R!c8TUxPib7Q#6?) Q亊A:2XX T|Kf &Zdg``g:rBc(f|-7 NBp KN'σbh",cV2nP,\[2e1ycFs8eK*/S лNQS?a{ߞ^XCj<):63O1㌨Xv5\ .uE'z%B.@ H~ԁmGa )o>ӕ^_ᮅb9yl3:N@8468@\5?x;2#Öf0Di .RTv[8e),Jy4͍{?ioDۛ@t+3 j-n^+QϥVdQVv7y5k@>Cܩ1Hcz[I0 &y ~Vw""Rb8k?6AX@4/}*"Af2Ǝlݾ0؈'@R K0"C&Q8ʦL[~vË6/EW،^Ŷ ڀ ~88wqUOwG'oW xSSX~K.lsp*;W]v!ϟ.oog_qbELvj)p]Xb #NY+LȂ"Th4G'lqbg bDh90-މD uOTSXͭ AY00xc",|SΩ386*bi"84ṋ_Y8t6Pq 4M[L‰o3X{껈RN+3SRj 3hq.%Ʊ]l(g)l)](ft pG8 |_KL%) 8l9@tF<ȗiXQhfesx ,Vbm`ɸ}֊k!*~\ گ_YH b#,JEA8}>d /R[ X"_ jeZmKr?|Y`R=ǒ`ܫh14|jovks 2/NF d?ymP4v‚˞9"%a&Qa5#\ E1-YmϲXqBSh7ځGݵAu(?% ğ{t}$I-0N(D{7k^k̉c? AXx^Y+ ܽ#Wy;"{ xT<*mvㄈihFIm~l3&WsgTkmeKUUDFvp6FM y.G{dXW[bs?Ӡ9=]EMe8,R|1+rrOѨLX ;ȉơ#nHb2yhwb4ɐ8P4U;F'a`/MRs6«V+`ҼܖqnoT ]K?qlŁPm+TPѰ+1ҽז%+s/fTQEn7]e.jQ~>8q1KghiGdpPL7y^Pg^e*M(}!<5[{Ң3pu>*G$~ XSʜR-5pQ:]Tޙ*A3TPFq.Y3 @ίQ1zd"&@Rf/6e*FOGIb!SF0)Ad!Uٔ,5uXTPyɢ3{I.8:$f~;ZGt,klWx͐g O˼F-&j *۴$4ЅCU.a' M%HB+SӨo #ŠϋȆ;:gN2%7cnqb#z%3Jl9̩V@j 1W/tagnt)2H佩۬C^v{?}ʮHʺѭTFrG%9.! Z27)SLk1;H fAFe8+ŔJ^ <8k|[J39-1S҃Z}S7^l,r gh_7Ob@g(o>3κ: `Ns^lUjm}^:ce(s*ghmYWB;ϴom[k GQ㘒8 ]fmH[ZA7zOapaˈ;-(N ۉR )μn`t 8 nnլ%DfI ܇ WfsefƲdc9X{b1uz /oxȿ)"(xk0J|.TcNyS,v#bsd847T> x/àw$Bm&[ 8n/0J󸒮DŸ2[,dPAޯx ^LU6;'?:>:>9;x}J9q*iⅤ-xɛM 3ClīwzMd39YߐO0b #(;WX_aDmI @!=l=Y2^~tqWMM1"2Jp&%QyRLcs'U0T7 P_6qM$Vl2*GVUOPĪ@SxY1;tޥ2'hd]2+X]T`y oOxxpĪ[l)2Iޞ6E%Ӡ'.q5"2&ds y}7]r(){ƌLh ز$6wզ}ET?D;u% W'deR58[ nu^&ߝgPWZSYMQMy5Rg8LID+){f*5b¾6B+{l_eGI]LJӢwjr П>H XUR G0QRZePOgL JtP(^d4SlWHUki☷H7&mвʒ hޖ.iҢj$MK~J*cRx^;"3%/'L1yB,[;X&QUP<[9oKN&oV]ۮdލF产VNl q8iL2#a(2gTTʑQ1 11kns $T7ّSǨ A7#ex^ͮ`*Y@ݞQQ_(BA*z2Vy)2_eIaGJ-D+ gO|31EGK(p~@H@jc#K_lAR=/K26nSOhXEiLd)t&R>UǍ$գtAu9r^J\ڭ`aYjsDQa2Q*(T,"EWh7iW77V[GwtZ%oU[$>o>'yn8dE&vL83eve-I9U?p>B۩#7eC3Z|_ӟ;*ͱR>$WxW"0=Dp`ǡ4`2k#gD8S8xeE9QnU'1i QȈ&S7o'ĭ:I>תsS=,tƘl+Nڙ }Im4W9K89Piur {߈": rV)'l)[}ϾsPDb+Um^2"XOyc/H5h4Xl㯺޲K5?ߡMlNK?4mWDߌ3Z)̶R&ZYh,oٕ?pM|vS6_77šEy (zrvհ1,l~1cd^İʑ]l^(`ʑ+C*-WQsЀ6vۭUT{ܤ4JA lpZ[,ߋLEL"9 6𳰁84Ʉ=Ssa8BG}0Nj+t6"ժv򰟴~\zp8ļ:BO?_w,:l$)J0kVsss_oN?(#mpx/b_?SQ{̇S܏ݏ+oor x}ս1[>褑UET( WoܰWח뙰m/~ݼ}7A~5[S-=e!i%M TkZI<q=Ԗ'G+X-+.h!19FD$Cf`n 0&/6~q`w簨W -@go?|z6?!Ψҗf6_̘|\Y%mZ6ѮVtN=  ]91y;xT B<-/[|trR|Cu6s9 whZ\g{N jd[L?NvJFt2kN\!@y.CMMkcPϥVF,ZT )g=fY9ųJ]Q&:mG;kP#l8sZL-7C0KX-/}"8+D*?[~c c,LZvટlߌ zȏc5 hTV Xr&9ǔ3,j}!djo*|ws3c5ggV"? xU..ZKJ)8fld677|oqU,GIyEW"0{g?76N~ XG泵s;[kc!}a/!D`cA%&`=?Zk /z /jm _aeO7{oe1?nŮ^i,V.bmݦ2EKGtӋ'`[PbܤW]K|>J⬓d\ 8`m9ˬќC~yӨ&SM"~_G^yKܿxz> > 25=4!ǔhni&Q^fQimqGIq`8Xz7cL j(-lMaKE0{1fǃ0iOLXE tPǧ$> }/#L l $#8Ow~p\e9:E`^5sJ;Xy/&ѰfS˶|T5_i`47v)RHX%͖ucl2ba0s7~{(< NGEr乃ysvi휞%CurNՓzjlwHyv9~=a3 .ѽ:2`ôkHu#W_$jKR~V50czSd={}ltP,mQ2 ^-,67h'/Z =+p XNU+Y/{ó2Sp3Vk%gZ-GR!*"ۂIt1ӲH F0=d`'py8 y8>FF)*y\W˂_Xb7'spNN7!`UY;9ku NRwС'8XeEБ: o9N{G=2N%N;<K#%wQXNf|Z s2'sAÊaRޥ׻xpM𑉲BE`Ě^D` 3J'4=KεN&5k]TqJ4/>tG.tLcټ(2% E_T?? [_$iW z|%i9Qe\1Lcaձdp YbUJ{-_!6m"HGX| A??BBRt~ 8$D͌PȽ}nԆwÒ ~FBD,X.:~f^9ȱ>}lH3cVkw\\jwW A-|5%>]"]beJqهܬƄA"@8,0x%<9 0,5A2m53aّeSP3O#&b?Mor}VF5<~~[O 1TB^!5@tݘE$t1=˖D _Fjt^R`* x~Y{Ƌ?*:3:8d؎E]w29{+dwq!'ϑ~_YX^?) \/RCjz.y>Noլg>/RSS&VW $s]d}2AٚGI鿚<_#/>Y⋱mv~~Z7Gހ:GnJ {1}@kTQeRk.F6a ~.'G6b/̢|=D31=B5A<]nqZWǧ}|tJs$a܋mVpxT<^\TDmv=ΣQ$ψq{ޜ,##j^AWR;Rd§v"5L APRx {AjڍhF sDQj3^_Ӱ ntB匘;[D0d,QђtF>"5{Kq&0K>gj&AM[B6ˣC;ъȡ#/v@^0Plj|o/]m )b)M~8>۫)hPn+Nԍ C_nb{@-hj8t/ޖuΝ%`Y2D=lyyYĆekغJԛty45KFࠎvOj@n ];4f6"˗cPF3=HД Dc_WFf"#UT$ M,wMn^d/в˒߶bQ% ]xpoV#$LDrj)ݿl[~ok2_8 }V,z.+-HTv+JbY^{M;y#fa?c/ohfQ>;Ohw w'8!hZDa LTN>G`NFϦ<{ۮT ne?pVVjY/XH_|[I`ђZXz;AN9T#!oNanTY<+$OUN,a|4 Pp]xx텙|tkGn>V.`WS S#;uH+Fq[F^i5=W짪\E" 5)4EfV`<X]fAto+'aTL*2 6^a/yUjf+8tL+'Wc! &_qD#i*LÐPmT{v#Xv$Ӷn;d @ۋy$ڃS1ʒ81BFʪ,\'NB4ލQ֬#%ȕcO#ww5=Lb# vmL5nD-,K瑢~TVS2Uo'7FQ^:Cߋ)"Mdc{_\$xBITmNߑweߋkJA}CeXٸ !g klQLIX̖AihgC8^*lQ'ބ[Rf~HzziN44LVk!]G.&(«>cd!ROvcCa|U )lpJi$ydI9XZ,Hi暯TO϶ 6@vkO1,)$NCd# Lx1(BL8GޜmX)F7Hsq-`Xp (ӈsz=!2GYkLC]][܌bNlr6"cpF'?0\/ 0?)T]+Q/YPNf  z~w 0o֗= )|&v{Nr@gsqDyeڰ S-8葘r>z7_nkO">;8=Tn`Um2?~?#1,2D;Y/gvY!4kmh!ȓ`:@[`E e㠓8~:,jhV"Evʻwg;[>/;'z="v)4P޽]8Av:#AVCŃu:W:9SԹQj+Xk˭;`z'!2EEr`0FnA/ֳUP#z}xZY-i )Ap#NXEäC[9"8WvDI%ƾQj|V-sI1NKC˙93iG ֮cJ[YmV҉.1/$]xH2_&¬\\J'. x ?"图-XLBXIu4{RE#oA -5JjigBՠ:ڕ:LaTIk3#;.M~{I)]ͻd}_ـ_jO|TL8V7X{h5\zl2{ilaIqF] x=;?U~[&6[ޤ;µ2hvrsI#Z-h'/SԪdVG]X9=N g- sll`澁lG%d;<0ŷ<.^td\ {g-~|׿/0ԴyVޡS1%3u;Yk )7Y=MH:ͯ/an/1lB5y:0j8ꅄ ջnV~S xzLpS3:O߾"[YE/߬b" ;{WET-:8FwיBFH2#rc;s8;Ǘvu=,@Coy?lɚ\%lLwsݴ^I4ŹsJO$TA.]fe&?1Y/̻"hnl6^a#D/򔕫"#NeֿM+Jzˢ1K8݄myQ)yR+NDcO}=a͞q]ۦ"ioz:0p F]"2ea T 9f03ĎJv( }Ơߥ>whD.t|"AQyk &qeHj˫o Qr `wGؕ Sū7f{ 7W#,20hHQ߯1^rmc?1s#СSm?G >oQ_8UP $V3/Q"{:;ExU$鄹 e yF)L} B ȷҙ!bt4^"R`0)LbcƦ;R[AХHӞ_m&KeV+gO4&>IV͏ K*qU ^ Y_Ç낖 _W t.(XtBZSh~)k _ՐIc;UT4ĔP'H3<7a+gTe_aޠpF3o_ݰ_F"^[!DWe VL)O[oJ0:.`2NVJjA {|XgF 2e;|WC3AT@+ +ί(f &Sc1SbQ&(ɆmwGCkGvx\E 3\>ͭl +a0SkM -CJ$w=urqYcf9S~w8fOTyx<"$hT86@1z:ƮBG+W t2bl<4R/('pAMqGU Ջncs{6~V-,Ηg,l kPMӦ\+}XZi\ˈ(ѽcN sL^{4iE1M/==xghx=eV39=<8zy B3bW!cRSL+oz*% >aD#'fy}uGvH9}Y(rtyyeS,TTGoۭt؉w7 ʝDC]^܇Η3m%B ߬%LjDI^h8FԳoGS;Қ]K%g`4?} m7lYҽA%)ŖXћi"Djtey^=NĖWD^NZwiS)a'o i2mSl$e0D5%ӦLRWΥ\Zƚf=ߤk ^[nFpFn=:eTЩTp5,(St\P 9`c?v /CSp2/I.+iҗ+0jWF Pg]B4[ DfiLBC 5w atese%wuZ7_Ʃs}snOͯxrAtFX,'UoOC1dJ<_uF^I Zy5F*pDC7^J"ۍr3)eZGbsЦyfOʃ!?J,ә28i ŽKT P^ ]Xz#]G^43wJt;`jnGƷ b1ں6l~߉&]y~RR+l>W.A\~Orp$þ\TŨ!T O/ F 7Bю:|N5f[߀<{jEWrRp%߀}P7Gut(9kwP8vW.iⴡ=uUSz#a) PN2B oԙL~v W9}B;Ko%-3!^pqτ*ʨ"kXtTFSVTpeJccHˬ>\:#∔bHۍZ@:Ć^'=/QTp釷Z$ ^yy# _9xjy<,b.8esL(b>MlյEOc=V <6G=G sڷmdLZD z?ؕjI6)J&+nܻdW>̇n#ԫA|Pn 6?vV-wZyoI8ߌ_7lѯ*uGf/z.sFc`y5Hf :"FQv0[9rdOkff\,d]G,fqp;A\rm,'ѯRa ƷIPFԃ7 +)mQl Dʓӣ"R2bX˜ RBܫڨEsJ`1[Ј/%$}ߟF?Pc 1|>]:{'j3Qj3= |mW`b\U5RaObu*a d 2T?e`?~bVZX#y2{W47sH^Wkys3[*Ůi/ ॻʲ O>ķP#sviVpɫߖ/w;ƚ|?qsUu;G۶\c`VJ  -u5)#;{a'ݵ9GQA~iju},uxԮێe9'~ڲ/a 2&bGpœ3 U%?}#Ɠ.οsv9{T\.17樔 U1 B|G5?F +=P18'lQ`WgY0u9TCdC:kV5ۗG9Ti|A=K}r)o0=BcHgٵ2բV{OT&f)^lRnAnÊH.-r]˶a-ffC ũ V}/z$hd ̢jG$k7HvD_)ވ>!$u]u`޾iQ ;#ԮpvK/P oQJqg~M ҏB.jzL 6Qh\4B7|QbEʶY MC&;&q.^!OWǯT(j.Ys0Qvw%/ak(b2 $^Z9rpP1S"6 H_ɟǘL̦lvOj&vS,jîبM:@ :G~-9 YF .X#DyX4edabRt=󬽄,„JTP=V|!: (0&ZG4-DEqSёBY lDJN1Nӗ*E^Gs{WM[)b:5ln*⽋\1OŧpOzs)^Ѝ(q@׋EAxQy7׳Ux=U3oٓ XO&_I61kI8K.޾ݲWV[[rkšMQL6D,6LRH oGts&G]bb@ &y=n14s[0^0d;(s)][?WPYS M0ؔ8X@JY=h6j=b Ж8OMMbOpa0v#Zz ?2Ki =Baj R/fk &6I&Aq`r#OT1s3,=MSvZiACMjNaphX+6 1ћ'yc;I|wmW[rt>I}=2 H \wtpKY3V`˟^XZOQ-GQ?TFemYE1fopNd˖Ѡ4ՙeהc}]qK~DsHږ4`O:R jz#&xh-$u JDЍCzi8Hl$zTCqg ʎdm{RķjJ(93'Uk {mLm2)i'&CErbpm Q~X"72) "IF &d`h@z̈uÑ"ՁIaXCu Dsd`B3ƗR׋ݥg.lKP p\peb_Ӥcz40f*m5Y\l%- W!܇ 9b8dDRzTmDS@Dz(QzcJ C* V%ZN:>^ۧPKUQm3`,M$B}R8֨߼3`nx5oPpP|<_+VtL2x(8Z ju)?'[Q49" =J,ci`_E #:Sg$ݎnC趩=_u&0Vom!C5gфDkUX Fvk$|B3OP?ORaFaVL|r{ASzc_ T)ތ/Z)ahpeDz !xʠ}E"|+#iT`92srlz zO3-HϤ%plyCLFST>w&ETqI *֌^?{34tuNB$z?rb7=wUq!zE'Ikr+(}jjz1ɁmqcX-~tW+ I#x% \s!B0$@U /׎/B,kk#GX͈1Б9ix\Rh5qnC{hv~,DnN@ER-ө 7:ųq@zufuT "@Pثy>*5f]z^yu)3Jx=^7[BhST1E&#TAd> kWɀ0R (?e@Bl{/EA ㅈsc03ttMFrm`Ouzebw) mv#`^# Q3cj&9kwe!NHĨ0.(ƩHB2\?E|:hFK0Y!2!wNiF\5,l-tXI%εӞ>Ja~5'44@B0qhm '^8Oa=XE׌&XQH6#XEb;\A^]/x>O* @Mҭ)Dn2(^ܿS͵T8׶0ת\4x1`"|K -w(n΋'ZqKCp؈(E9=dkIUq:ÏE d0ޔUVЬP{Ϲps+u}DNc" &y#\ʖ\k?n@O7|fWՁMF6Yڵ*o*;U$7M17)@ +1n/iր4c0:'4 %e:2n<w]9ڬfIqi[,4t=8t"l8X)#9AE^QØP9J7g/qRͩ Z K3d6+I%~&# >S~+Eea؉IAwauDA{T"z0-G4la 0& mSCL!NA q G1oͨZYvɸR?iY/@0Dß#{*Z]hx <1nCK֍PܩEEmTbo5i3%_ʞ#m]|wa7~!}܊J`oaQ rwmucyq?Yr=P>qZAxiӑI.XRoHN*v $c,16jt=?ꡨ ) )YX="0.(ر3?J GG+J/HILm+L@CNs +RJ[zY8wTSHQ 3=m$C "z /Э]D" Ҥ5O(M‘QN yMjɈIg@Bt > B%G/+n, [;3g+X9ڧ#bs Lڿo8,-'ˉm$dk6ahxP"|^=;k?9W[ 9xRS*> RHQO舳?yI 㷗q<KնWR\pGfBH~ RZ rME3gKDHSv?ڲ,QV[(dkq|eX.u|}ee9=[qD7U&8O#;?#88zU,E n,f/XF"`gH{j jmtYh")JL)9vOa; l,Ƕ@{1hw'Ė Iٍ4l&̥py`;w~Ww ougPRzL1J2B  +STɋ&3UP 4upb}:p045ܾr1σ ߈h2/+(6;T0Y> jn&gdikX:!ЭkXR?Iqy-m=X=K"5ͿA4vSЖ MUxP'c`m*Lj9mJ!ꏠ>ձ9TN@(f} >&PjcB@"5.u}uKUcX.A`DKbXPhfxqp AڰGL+bfpZ/`|HV4Vo)5l`R|TD^۳?H q TRpЍc *LUM( tg]z,MJAn6:I#f!g_d؅ Md %ں7̾uFH"bjZ6l.gbs#w#w0->uxD`.׸1ͥn5uR( 28]T>E=ӯ{IaxtqU5n7AOMUL]Om!bT?| I a7ES`Y.Ccq`alS CDmA`(IxqP9 eG ̴JD:M]@Q|At3,cbDĀ̴6zCuM]9:~D}~d!՚ݺ>ipA:ωKg7WiN8ם_CJ}Ɛ ;>P^=xM7Eќk'g$͉!x(Dg}stEpn~sΆ8N&kYY݌I9b ;RDr/ &ۺi1P4Zα0k݋bLWnōX{&>઼1:Y;c?7ǟ.ʦKgGdASk 0Q_(~6$GC0T~Àx.Yr дj۩$Ԭ3DC{ u0*̜)iw͋IIuI]O>]2%.Sv9g׸C5t>Q`Jjcxf^=QQWl`H4pyjAFD~[tS+J qqC?*5ZFk~4ί=xĦ7oJv0O*YSZV1FQ=FBA9!f3*Xa:qqumR2R-]8k~2qcd٫h7A Y1ޖJ "N(.Ym^,d \kLG`Ξ?Lu3u!_=k] [flI KJ`Z&%vdZ}}!q[]cs|۩tap0ŭ:=kdY# &7UiHS H9{e{~M*s3&mrC~ޒU[gtoq16gGgѓ1k}CךiHVELΒ~Sut|< 0voթb8/3fr gbVQFRp<#WnF$YKqW/.}4jYb&a9d( 0h.#B:%brFZ1Yۛ:T:T^}*fj}ķͱ՝MN9ԗr:{?RK;fȭ<OpjL݇P8 3F Ί3>|KM:#[_D痸,Cp:'<ԁS'qTn4/kh|0]E_NzG;Q1ܿ,w0:tgL tv |E.g3)VxAb'jSU_}R-A^_@ĢfqFmǛ_8#ba֤/Tb\y攣Tީԯt N8y3j۫H)زg{8CORb/=RMj٤3G)T'CyBn%ƊXͧ6&1k>}--::5g5\21xGBű#L>40P4T^e Ƨp·Z~7^5F۽h2Pnn*ƆEQLġ⋺ /;;lhxԷL+j;>:0Gzns92#(Z= KIb+tIx6|n)T_reM?h|z+J%*(+ДD(( A™_"+M ݭOcT S ؚ^$Y  g 4A-̈́dķf)ǺJq;| t-D2 #׽Bb4K*i" f*ն9*\A_A?QPPe )r`.1vAGo㝏 >Sاwo3~2Ù륭4޹,[L2<QWhk9#Z|_wf&j?/hމ0"dSP$P!*ES:fȚX֎ ?f,1c6LtQ5.+W[葞ڀI D,r9V*n/гpJkڔ 9`Ӎ R1bTJCVzނM!L0ܣls3> ڈ, v+NS6~7ty3# ƮMar:N҇ı*LgM_9F2 dա?krQb3`jµ n ltnDp1 2dS^eιJ_b < 1p >[L5avpa..1[xS$]Zcևhܬ___;"qIHa `1`vr?4Ú-ך]ΛyRKnYfͱ"+ Gn:##}vL_/:WFH*/I2좀47ZjK7f2 $ -$lQ&y6׏..?p$j"ԱfH\(sI3()SdkJROyê9͍aWWlohgi[X 6N5It=ڶ7lq.%=hlȤ:{ؐ`*EE UFqʶNukN;>%Qkh%VCc&Aҡ_vjYpӉϕ?`b/ecIhtBW$v<[R5#r0qؐ@ѻnkcNiYتNF?ĺELd*FؤK/kߨ,+ LխQj-$.NTeҾTUgbg.v+.)sZRl'[? x"~\6p0־SٟPlŽ_NNjO1&#iȱ &Z2R,a[r!vEE,.e:8(P7!, )Dz b+(ؓ#A4TI[dp O#鰈ӷ$;)]Ձ|3V47eM8)ej &id$|4fNsnDpl?Xj Qp TJ\_2gCT'܋•}n&.hZx}O:ci&G6cnZzo8F76  @ QIXFK=rh|~{$U*z(7[뫫 UkYjY_Q󢛴mt~-hd7ZX]!:f3xJ qE>nCf?J<Ќ#EЦ) 3$37PUKHf>]Kx EdV#rTܲ@^^̘L9!(( Ob|؆;_+l?DS6qW0jx#=1>z`QC.n{6y"5hz7xasư%pM[5T&PEQÈ: ˇAL F(E5Jtir3!_a!bQ:Q b") ٧œ4vqp+  ?;p{ozR= Zo#R);P#iT"$Ce&s1HB/̀A{wZ7ӴHXn\a|lseGV@g"e WkVr+@g@߱ߌDB|X摁nqE|%1FdN+ŨZy*#=#|ՠTx2p:4OS0z B C**2}n'헊?=%J}r^*M.,k HpaX|!WM0Tuӏs/`JD^*[#@TXAHS_ O AQiApx^3M#b}ߘOI,o6Vʼ]8OxSm %(Oo6A\k;x3YWgýӶzctt 4gmowNOw۴k%Ã='(}^<j5m#l;G{A{p5|)6^;/1| ÝS1Vxsjz ?:/?8zyzp>:{upG1B&k)bFa}v;@)"Uh 6~w_e7{;wߧ9<ﴉ>;x|vzDߠ ,)S>vt%?/흽>I0Z˒wV85*&cq ^%vCՠ3ۤKV,_ooK.[7 $DxIQGl才4E+kxD[M#dC(&{8-ry~2TǞ8p Xn( ciD&왃-R VQս@ҭ/Ǫл٣$'z0F)+jwun|h'[Z Q~s1uJ1c7iN߲s}i޶+f ~ƫ 4`nkzwp͗ Ru|rnS\|1ъIh[yzsE"qF˓ډ_mg?M} Xr[ E&`~O-z7w8'[/K[k>M[Ė0Զm ɅlWgMU*$@@v_)p0l4!3{W;ȣ4LRr ^XTN-4u_b;V{UҪ{R&[05h-tC2aQqfiAn3/#LPq=zʡT)6ф($4JKZX܎(Rg11RStEfA?qD{}HdvN{&xrUio&I/QzaoYnc4vE<Z-4?Uv8D{_ Y{([.ej)E$fwwޘaReqc{kd <;Tjhu|[_SPb|KS^dnJGR/IS䴸+BjtKܳ Z'{=1H)#/Ɯ΃)xOur87> 7/360s^aoQx{K}(1זΔ>Xc91K|V~Gv+$6+rٛeƨaKnA7i:9acR,TD4hTm ";m&PPi \\d:J$2/+G b!F O{y|5ibĊfm Tޫ No0>HR&Q#rUÊbMRާœ*nR+qh-=j)Tdž99p]q>&КI\6ę~ϮAOb|E.Fhe蠋En;)9ma. DehmD2]h*<#p\3"CM/GʴuJE9uk*}Dl tbD~gڵ@T(?|੿uDl'HM?Ul?p#=]}usXt/Ў&U<#*XѪnQFzU0a0pÚ 4nJD;&6d.C#,uu[ӱ Q:]97Z˭?QSTJiF×eYOK M`omFPd%~|Ղ#|+6^t7R8hao(8VݶO=:ObP  2T MKt a4]S޹>O֗A/cIʋ%M(ƓIq@Y_ UhT*%9lG⻇J.- Q>ƻAO#`V5@s8=q ޥ6z٢m݈i81Nu!\\y%jX!f@a04-*G$Ցwd; hH<^=S*Zo# Q0j c`b`6M=/w~: ]l =g3@xDq+gu+]y'R%D7FF"1 1ۢqRdCŴ%n7l^7^+#к8sƔc$*?Cemॸ&KqO:S&F |B Q,RfA!  SaOy;6X+ *=͙ \!`ʺʔ59!./Mi`ʹʔspX0H*ՈOiDv-}#Ct;CL OQo‡ش^ 6 YLas֓ Rn=ʭe7/ j$.)Z`<(%"O7|1*7^55d A&RBB,hqcݥr֚Z5ٟ ӯŒRvHGth "/|+-NSݣR&7ŦBY23prdN}$gvmg"CJ jQ#HG^rҍ(Q壧E1X#l+(=<-.~$֙2ύ$$P7b% ) S$2Ɋp|%OC#A!&7+@b<ڣ#oj:H3D6uq r~D߉sx#'tkctnhޗqd=Ŗ9;&ݔ&WfAl48@3aF!@ښ>h I8Xbh]2{h('ikTDzTpv6+OW LE}ͺҥGILr˖..(TrJ9hw夝儑Y;9'm`;PC$p\IΚb(aeANPېܵ7DXMTv`6Z5TObRLᄼrku Qq<۠K.Ƀ$hw0ighYE C,$YdȁE oSr!o^@=d H)c† GֈV3$#H' Lg#E,tIxQq wn$%{A\&0Jp_[FBE^䎡|h=}dMO -(]s|.$$ !; C`W:\ʡ˵G>骽=dH0xAN ҇ujD y/⽳4{[{zό7xwˌD0>uʗ G[.@1!@!TIČmzlK^%N5 ++O?fueI. UO/銲I5H] pE*iĵpfϒ9K:[-jg 9*ݛcEdfPq w C/)? ȠmxdӹLaa2QfRb^Iݠ|k'yۺIxI+}@](JƔ|66epn]<ŐkS<(ԅE%'ggl V4(ȾSRf>]ysqԼ$ÖnM[̊9χCчe2R)\jst<LN]`I MtM cF*#p@1 DjBLo*llDd @Ǒʀ<2@]cHTPqRabEiaJAR+P|@]Pɋ]!3됉i܈.F"A׫zJ)XA1%2e mq_U1AӔP@+3Z+N\@̨ح%+9ʳ 0x.g7cm yj&NjWܬR'$8EH9(X@Ϣ`ɣ%6mP+ʏAn gjm k,)1MS8qL#[)ͩ(UQq!Yfr%$s)4EaȓO%p 6H[+Š.\,u~o#kEh@i3%P׳i_ffJR11-|p GN TM_׍"hE%uW9-L HAbx׸0Oj-5|:mRɦ;֎amYYzarfῆ- 6&YP fSH{;Gje_W% ?e?b۬GJ\^tt@aj0*u0e[a*sA Rɗ Ĩ |T:1^@ei*0IZ Thkwa팂qe52ѫζaE@e #r!)X?Lbum:099T*02=75< H'WS̾ JliN̏± W[`K8nRWp^#@-a9RQ<E`IvPL}`1obݦe-.#:N5hkgɲdFׄvjn(B<߄I{FR<|+Lm',n VAM09_"Ķe.%iڟ)]ʒŗ9*LV0[Y a+0R8;=;5(*E)p : TeLV7ᜂC43;"c3U meNZ CEyNe׀̻Њϥ*/z>pۑҼ1l)LVSÃ]šXіg-:%ĨZZ@E2'P /2N>]}@|.i~)5Pz2@'Ac)kĀq(ob3dc!\qJ n\G(TKY狞eʼn^b;]\z/ſP(얼0q$,% ) DѰ>a0c r4!<&2BХgWht)-%@ X anij0˜@J_+ ͭ*^eΜ**L7J-)k,>Qm{agbIXLFVD>&~Z"2<J=t@O_{uAݹƐzѣa ܖ J%/΁}#E'XKESS DlJV/VmBKwElH1q~@PR1 .x}r/(-:v‘"1ևYyZ圥9_| !+°--Y J85dl1T:ap/Ibp$O`l2+$<(5B(BzPA?ήg=!"X$1 ȚW8(% * >5V:w+ 7=r(pW,|n2HUo$54(W(ͯxf={WT WjAjυ Yn pRydzjQ #؀ʒ+` GR.I)#`|r@Whڠb}xc$#FdtX}'񙀞I"NSآ{>qJ4GeMPgWlSd ;\WAiEI ?垸SgG:E Nєa]IJYϑOk飸@g=)漅<5AvDn4x7ʹb@d| hj1IiMrfqV(F/Sp21~ܓ1~*N$sz" 0A#KBsar\vvwHRH9Cbn$xӭ۰2_=i*Vݑ4K[`rKdeM; _t 3 ﹰ P YŶZqSRGSXDicU iwETk48@oHxpYݨ!uP6Q$pݭY&-:«0K-+gp~4Sj%Ǵ뾨<4fٯY =.I1S3B>A}Ϗ!D9'!k#K".Q+(G/sAm&6V40R/S]@EĤRa)p{oቹ+NDnqrZTZ-E1 ZBTӔm16ц?AvP$"LђHKkA}$MY0:E=Eo\% OEt\ O8W(X5*Vl[O1;Lk83L_b/C$~()*A="Uɕ i2|Rnc #9UXiP:_ذ[ϻ\AނRl_0h_?h1HPv(GeꉩRk.zbZCܼ 뻂RYnBd -O,^KYpZ$E ri6UY2UrfGbvfVTPm4WTj E󯤋k^0++zd|7R`̢Nn 97CRK]̵, a֯~EV0EbL{4ӼH$4(Rg.?HnL,Պhg80~V+` xuL:An5Nԁ[<-s~1E߉br}ӎIt: &B97 βbG6F-=ǿ%I^@B2(K*O8a6ji oz2IDZ_5- F`J\+>]:~Eg80cU& ;9I/ ]4s`yjy #ۢeAe%'Ȭp0¬GI2) 3 `cl-wl`he]!vd%f'KGty}Wb!WEj9%KЀJ$6%:U#׮" ar\8'ĢX iC8ҵ0!2HCWf}AUxAt m,sH: CU,FP@+TTz)=\Sw*+ Ĥٖ uPH$ҘqI8ņT;~vd'B%$%v9 XD%p!FM^9=&XJ ƄJ_#`0W09[a@y3g6 ap`_ Y}t %Ce _)@l'HB&Nb&7sT`;]ۅ4G#1m$-LlC{6jV阰T>y MvT*3 JŶfxƉJ\N~WJy5a/Fa^vgee&F\(c>&(rrx@7wIPTe X'og}~y7dWJsp+WVti5x("\bC:Px VebNM^?3LZ7?cȫ^"YsJ˸RWed~ ]>T9dGe>_J,"fڦ"@K@sV=3dhwXNU(Bϔ6rV"GDڳű:?wbv;RSUn\$xk%cI:2UT xJd06h7Jr)LvR XU)8%K$a)wYenM_r`Ȟ G27Kzz2uwσl"%Po䲱6TjB@Y=9rtcs x m""q4T)WJ1)i^G;{`r,#fX:^oAUKq.D)t'Gbܔ*jA]4u 2I"띡n: #*ײ8v6w;sA~S.Q{ˏ>}|mH V1@3 v&V✯^HhPl,8 D%4;%,W 9]]'WfbAZ4R :AyTsCe*A^#!g 1ᨦ)i>f smagx0]H{(j/8/V="u7=5dzg.FCr%,4Ady\݋ N~Ƚ뿼Oe{W)-Z0nb]"HO췐"Wlu%NC'j8ȏ܅:yr")nmvj6NyE$$* 4XFy/Zqf9 9%[kِDޑ&Ib.jK(Rd =/Aebn3Em_AIF ]ԏm?3ߕTM@4Ohӎ+ԋ `AMU3WI.﬈Q\RHH y!GFM~Nڗ6FұKqr六ih]k&!O<ͳXIW܁ y dJ;}V̿uë_9zd.מIz_ |KΆ!~HRNqvBtH(=e ep:bGJ]jU*cDT:͉z.vY ȝd9%hGF3FtϤoAg{o6\)']$|uVJ*3Z%՜kљ@xFwk'UK)-,~|q 9oyumNO^(yWaXSn*10 ,HGMJ֤J& AL@2c mިMno^KaNދZ͈iN0<Ǟ|jPg{Ӌ' 2 J 7go( ©kԗ("ՉC>l7E@l^agw=iכMJn>,P#&#';*!a}?`_f7!35[9̳ázՃ47ԏTKV@8XkzK~oQ@?un׀J̠l"wM35fBMp}H OQG m1fAh\E#zs Z`~MƤu0.nEV!-\ d'N|UU7xj>Lal,i8}7~@ LnVot~췇CC&NN <@Z % O!!l0_D@=6KY'2 t>GDžC6gCIf qHv xMA7m+6g.g6gHLſ{ Bx,y=uKJ+׋;s{nu_t>FA-j5j@}t7GkN{kl6w?Z%F? q &?-?\<ٕPum9 O/ZՃbů Pc ԅ8'/ʈbXyv겯9+l9d9,:Qt> f:ok0.m8`LgoD B߶j31lӭ' 6HPuS !:3?[4CZ& Sԧ( z;WV6Gs{F U:}an  ӯyx6l3bΠ R hrG{|9h7W{Һ?n)h(r3^ʫr8`pyerfy$$hnP1ϑPYY8X0x4Dݍz+/3n5|Ln9awuAJpt64=tFgE%=xÁ+;[P9|鎷iD_%|[^58}xjwLyN#XqZX'WяYף2h9,3FVCic{uSY[Ku#!!xyT zV0A=gui?n/l8PqPqsO7n)k .4wݴ@aȮM^֬Í:=skx:gX"VEQ/c/,'\we"mD߮Hxl^]A_LMH qޟmemQ|å=|h?wF5;LDJ (y fYkUM}8$|3xK9=g+DPsJ&Xj6VfE48[ow{?u{Z ^>Q^Rpd=&'/dUYа3vլANC:P*w5Xy^zXy/sbJ״kY>/<ӊ']-Z6PY~m|T!!j>#yEt y۔eJM8=KȥԬN:8_(IJY[ԣXGk x] ?z͆6@;c߿u!ȃM+'ArDR`>IB0zs﫣^hgdžΐrí[BHqf'-J6A37 ֆU%zIxoA/v9{5_͘kInIӒ^`MyY4+8J<0 E~:M#JO0#5 *).4gg.x$V)G`ŹOlXÌL(n4˿}?,ٺ xMLx/wQu1.`l _€@Df \a+Xfr01puxŪ)%#Qen$0 0RPR0փ>R&IB؀I{.<\nDl\N )Ѽ;@6g"sл'\*DbA^0fmG|/ }Es!D~:7s?ht-%IK„<9 6H q7oHUwbI3r X)*.om"W @1i{*2aa[a;J9+` +|G$XpFw1IWLy(7?z;e75_ikssjN܆alvg{|!bVp aW +,J2 _uCԿ \cxcw1$qwTHNo6zSs0f7~QDb ȤBHs?Uo/9' aM󘪨w1* C/dՒZ^\0z>{q)o K#y˹4n4y?ڋ$,os86sV8Ct+Y0jZn{М]?ȑfn <~O'Cy>n ndP&KP`J*edԑɣf.5x]Y6mAm3* 7pCjFVܾ0\ϰN$omMvnK%p3l%ι :{g`DY>(an]X`Ur]&黚v`Ū A!ٿ?3O|E4L|  0=@NB*E>JupMz$vnqj j_{o39ځ4e731N2 @ c:jO52X"`k[qׁ\W-(~x1M.6~CZS9λXlmuШ%N;߉ͪ%oп]c#=gl؛3ZŏWhƿA ѣ(~Ao< w'ēQugh5B6107(LlRv\rAh6_7s`Fk|9 806)c9nyR߷9/&,<6ެVfBx(Wr;B׼ )!fjfW©Rڟj<3йݱRq.b)OYm; 7awG;7x&bޜq:4KL>6\qzk d~_"90r4]/B6UC=V0s1ެn-8 IWA+D^yjV11Un+U 0  )Bw^_i3KL̿oQz+v G~)LRF&=zji4݃Ȧ}pHkXr{r4m@&U+Wtv7YuyfM+\Cr DŽV?` t;s9u; ˭("ݬA(Zry٣!Mxc\`yr(${(7,ǣpQh~aoX0x遟 ׸)<"Ze>I>h4#Ex;U T/jMC) W{j',J` l@%K_ "Γ;nSy seҴ5q%(qsd' t|+]'+pH7H#VpN[iT&p5-܅*Q?{ ͠%EKja`nQG0 SV\kXOsJ7 3?0wx3| HnNMWYoY^ϵ={h6l/rtuܱ04[gxizG8yJ?[v̾rc_Z8z+y 62J==XRC=v%_aC^>}}W[8<8?, 0d?[`f8^R?'2+]X^Tq*H iFIL̸7ys<ڝҗ[0swokg;wtձA2K 3ՄNu`N 1Nٌvю٣ C$5tёUUd%hVd4 b$jb"-cxmJ^{K茒s^Tl'=~Z[[]]wA_x1&`U$%=לl~i?s7s`Wf}~|{iЭc0nY }CnK_iW~wb7{g;L^4ov^tͬ~:?u:׼|zZQ%Æا/KɪݣͿR?-C@ki[{8.vbRjxe.;T$͜?+SQVdu3䗚{YhA) iwNXLSp;HE_`c[v+`^,8z!{ \=Ni+S3M)ô2d,f( N=kjh otd4Ia=5I;xQO^H߫}0JF£B&Tp$Tv\uAW#ب Q < k0g(x8_2?jC}YضW{flӍ Ua};X/ @ `tb(S=4T n6e{BeX&.}p=@*\^t͝K5Z'b3|IJ'UڃK j,B~m ߱^4"6%m̢lskspl&K?C zb=T>G:Snj-;Kiܟ- VExkAe H As:ˍ'㱷`4߽k%'pTېRa #C;*KŨ#atQ+ KMsW6H"65-R, nǬ+wr'`c1Cs! 1-G; bNzL#lID\yb8JGo?tXb[H^F0T~CJ)9͂p̟x oܤ qԂRS-`םVbs`̔\JVRFflpXW DM=FeqVY `вB˼B X/s)hl?btjOG@2K9\"|T(bw)ƳO-&u*lGR~_j3-̏̿3Pko;>A䙇~2_^OOMPQoZr ^e 1ⳑYZz[~sU͹":LےŚ(]X x ΫZ]uEW(W\Yח߸Wr4XڰmP6T;ÐrrzZ(W*<H[YhDׯ8vŠ=pR[496ns5d(L2<AI> "a7Ht))L;{ mJ;euъ~hJbq:UWzn"1ĜIUl\19ћٝduL(=P/È+d0ר\,Y:t6+ Z!NJKiy\g~,#ds7,wnٟqiz ìdr3MD^qdG[p4eGL("VRWPh ϐ/HN=xNc'WJ Ǝ,o!/H.H\eGtJ-oNe:->6]N7E^BNY@-lbͼ;U9XQȕ ("8α l-p<YL 7.3l+廰@J&\MziɄVͩVMM5s%uYL4FYD+d8+USp=8k #YUzY>{<\;_x%C6:TZ%[XSr٪?M MP,,WNƦcUլw߼sKOُ5/;͝>a)[:,nDe~g*RGkwgoj/ҖʻB=Woopw* un(ܽvU+r-+ۭ-s5o6VW=b']5'dl­[0`ȳ^S*`^tA{8iOJ{. >Wnd#2X9 :A 1\ӵ`vT*FӼz2b9ʨQJSiYC7t8IuaJl^p?COn[5e?0릗ggPX):~~4 M@ ?w{MWcs3E5'Rxg!3i*~PB?4GqiO,1. s7`Qm{,QA!]:>A'OBY<۰(<#Pgx7KS߱"teι%&kFrsF [K®g|TLmBflڳދ};0Ű]2j/y0>w\*%! o`^ZCwY 8*}pq, ~瑑̮B6zSf|LĥR1^2;*52%PU[~峬"rCח" I2_Q_˿-O`@_$oiC`9v3w ` y}2^QgD͸<-?Bi,a"?tV:R<)dM].g^|W8q( @#F{wqow{˾Ve@ҩv{=Rmm;pN,^1[rkljMIGМ:)\p) $(ϒWߗWVsWזcqeF Т*F(-A.S@(mniO s w͏oLJ((?ɇ;O?+O,?l_ QQ7pat$KN'KY]߯9`'J{f7QXgky14\Vy&8re?XQJfdnì;zz0!sfb,Ê+rA4ϫ&v.jZKѓg+iq=x/;]{bdۺHʹO"Q`~ ձ 8'k'߅+ʷ lG~.ٻǴ¢g-t9WoDH6$I={b/{JP=zuO`an%Qiw=CS|S05`4X46&k0Q6cA8̀C{AVrpj6l\cvڟc0 /.liձbҘ8Ψ;thn[#,mo:``h:X;͌7טцe>:u#t gig4jݮy7ϗKMfq;;q~=. G}VQsUnܤ$IDy~Z ` 4Z͐F,mȎXh/U^2Jj;;.8F-5'w ?\uN lPƜ_F*H.hHxغRH+lƍF oD;(oh"9B~ -[g%O<4ݷKza0F8ju[Ƿ[%F~yd^qw%y||}c3U7hAw͝6;ͪ|,J -銱fy#8zO cVvޮgIߎ}h?~$ԞPFw +^w7;B2M㮹_"C3h򄓂4ڗ$fQ7g#?.JFܜ)3ß9gs?&iQJGUo Ʃ= D+C6rV[<%i@5Z rx8]Oԝ+374qfhaҋ j8>WH"V)2h%W}R$s=XqawN0Nׇ[Jv>>suVP^-U~|O i h{\9CP "* |H^.~x΅3Ej7U*=S2F6yp:Y"|MRO |t8Wo7Pͱ^XJ3 k? IWm6^!~_Q]$U Ino,#c(mv;`_+DRAu>IWkbi#)O S!ޫ9CvɱzRi]&ml4&yWl2Ot@~Fy/*߷=:2[My8ξ{$t?{j.XӃ? qpFt8ϿؘAIoU<[Bw~חEΩ.4\SIL Bɭ3.@{q޿jV8)oK)͙`hf(c},n^3r:x '7;ǢgQS.G:o)wl6@2ey2'J>UV6+-5?&хpK Am F+Iz4??26oһ4Q?Ng!agwzhn @Wf^YD.4>.TthDDךBP5{gjz:z.< ,ݽ<ٗV|>} ɟFX웹ft^gx<ދ N͏`v.A~n_T1Slpz/,P^ΐ؎N 'K<;dlb[nU×Gݕ;˒r'e/jA +;` zl\YXkk{ŷGT%y'$ARn`;Cj$].21jpn q$8}I\{Ugi{c: XiN܉VZ,EzwU}Uzi4L {&ikԾ,#`IJ{sif E8 } o'ܛMh}^u Ww{&=CYPF[R^9ٕ=K]ÛP۾Wkc43yKyt 幋xC\, j4۹ hI}0(=-)nb R*a_:$&c zx%][Б5k'iתϏׁ<7#yASsΦ'濵9a^qHwI}07sˮF\<07J>ӣwo哆Pe#X`w 0/]r+w7_ߝ<@f#9O<gFp)lqA\o%<^nupC gb 1)Eui~KP+Ic+p7*0GL%6] P( ο1dE|`~r>FUAsm s뿬wyo3VlTޣ\1D4<рE?|w?扁ח?'i~P yaq.?L4,/kql{:hΙkjbI8˟UV! 0 6a4@F׬ d ݾgZ8Z 0r{so惵Z p#üliy9Y/9ξ* O[Nz ̏LdQӍ\mv `j;ٕ&!e&QW%CBt;o1|oV6oFҏgdy>O71~?Mu(#`-bgz]8E0#"c:gt8B dmM3יּ9xUL˭0 Wߴsnr7 쿃e  .ѥ}f ȌDfp1WJ ^@e*,SBbƍAKbMuZwxjg}=$IܱRd2lZ>7+rC@j^kA2^,VY-7/3`a\xWCM4,D}5,'AHJh|dpGˌ41;DepH|EV?3w+s1|87l=2x=Ia&/097eݸZFACi|0\I2!_dsN }Tߧ\jLS]M<.ws}>ws}>ws}>ws}>w?N!ބ@curator-4.2.5/unix_packages/cx_freeze.setup.py.patch000066400000000000000000000032531302700154000225760ustar00rootroot00000000000000--- setup.py 2014-12-26 22:56:49.000000000 +0000 +++ fixed_setup.py 2016-10-04 17:28:03.182647980 +0000 @@ -81,19 +81,18 @@ extraArgs.append("-mwindows") else: vars = distutils.sysconfig.get_config_vars() - if not vars.get("Py_ENABLE_SHARED", 0): - libraryDirs.append(vars["LIBPL"]) - libraries.append("python%s.%s" % sys.version_info[:2]) - if vars["LINKFORSHARED"] and sys.platform != "darwin": - extraArgs.extend(vars["LINKFORSHARED"].split()) - if vars["LIBS"]: - extraArgs.extend(vars["LIBS"].split()) - if vars["LIBM"]: - extraArgs.append(vars["LIBM"]) - if vars["BASEMODLIBS"]: - extraArgs.extend(vars["BASEMODLIBS"].split()) - if vars["LOCALMODLIBS"]: - extraArgs.extend(vars["LOCALMODLIBS"].split()) + libraryDirs.append(vars["LIBPL"]) + libraries.append("python%s.%s" % sys.version_info[:2]) + if vars["LINKFORSHARED"] and sys.platform != "darwin": + extraArgs.extend(vars["LINKFORSHARED"].split()) + if vars["LIBS"]: + extraArgs.extend(vars["LIBS"].split()) + if vars["LIBM"]: + extraArgs.append(vars["LIBM"]) + if vars["BASEMODLIBS"]: + extraArgs.extend(vars["BASEMODLIBS"].split()) + if vars["LOCALMODLIBS"]: + extraArgs.extend(vars["LOCALMODLIBS"].split()) extraArgs.append("-s") self.compiler.link_executable(objects, fullName, libraries = libraries,