parallax-1.0.1/0000755000175000001440000000000012536574645013451 5ustar krigusers00000000000000parallax-1.0.1/README.md0000644000175000001440000000421712471655014014720 0ustar krigusers00000000000000# Parallax SSH Parallax SSH is a fork of [Parallel SSH][pssh] which focuses less on command-line tools and more on providing a flexible and programmable API that can be used by Python application developers to perform SSH operations across multiple machines. ## Installation Parallax intends to be compatible with Python 2.6 and above (including Python 3.1 and greater), but is primarily tested with Python 2.7. Installation requires setuptools or ez_setup.py. The latter can be downloaded [here][ez]. Once those requirements are fulfilled, installation is as simple as: # sudo python setup.py install Packaged versions of Parallax SSH for various distributions can be downloaded from the openSUSE [OBS][obs]. To install via PyPI, use `pip`: # pip install parallax Share and enjoy! ## Usage * `parallax.call(hosts, cmdline, opts)` Executes the given command on a set of hosts, collecting the output. Returns a dict mapping the hostname of each host either to a tuple containing a return code, stdout and stderr, or an `parallax.Error` instance describing the error. * `parallax.copy(hosts, src, dst, opts)` Copies files from `src` on the local machine to `dst` on the remote hosts. Returns a dict mapping the hostname of each host either to a path, or an `parallax.Error` instance describing the error. * `parallax.slurp(hosts, src, dst, opts)` Copies files from `src` on the remote hosts to a local folder for each of the remote hosts. Returns a dict mapping the hostname of each host either to a path, or an `parallax.Error` instance describing the error. ## How it works By default, Parallax SSH uses at most 32 SSH process in parallel to SSH to the nodes. By default, it uses a timeout of one minute to SSH to a node and obtain a result. ## Environment variables * `PARALLAX_HOSTS` * `PARALLAX_USER` * `PARALLAX_PAR` * `PARALLAX_OUTDIR` * `PARALLAX_VERBOSE` * `PARALLAX_OPTIONS` [pssh]: https://code.google.com/p/parallel-ssh/ "parallel-ssh" [ez]: http://peak.telecommunity.com/dist/ez_setup.py "ez_setup.py" [obs]: https://build.opensuse.org/package/show/devel:languages:python/python-parallax "OBS:python-parallax" parallax-1.0.1/setup.py0000644000175000001440000000304712536574176015166 0ustar krigusers00000000000000from distutils.core import setup from parallax import version long_description = """Parallax SSH provides an interface to executing commands on multiple nodes at once using SSH. It also provides commands for sending and receiving files to multiple nodes using SCP.""" setup(name="parallax", version=version.VERSION, author="Kristoffer Gronlund", author_email="krig@koru.se", url="https://github.com/krig/parallax/", description="Execute commands and copy files over SSH to multiple machines at once", long_description=long_description, license="BSD", platforms=['linux'], classifiers=[ "Development Status :: 3 - Alpha", "Intended Audience :: System Administrators", "License :: OSI Approved :: BSD License", "Operating System :: POSIX", "Programming Language :: Python", "Programming Language :: Python :: 2", "Programming Language :: Python :: 2.6", "Programming Language :: Python :: 2.7", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.1", "Programming Language :: Python :: 3.2", "Programming Language :: Python :: 3.3", "Programming Language :: Python :: 3.4", "Topic :: Software Development :: Libraries :: Python Modules", "Topic :: System :: Clustering", "Topic :: System :: Networking", "Topic :: System :: Systems Administration", ], packages=['parallax'], scripts=["bin/parallax-askpass"]) parallax-1.0.1/bin/0000755000175000001440000000000012536574645014221 5ustar krigusers00000000000000parallax-1.0.1/bin/parallax-askpass0000755000175000001440000000042012471655014017376 0ustar krigusers00000000000000#!/usr/bin/env python import os import sys parent, bindir = os.path.split(os.path.dirname(os.path.abspath(sys.argv[0]))) if os.path.exists(os.path.join(parent, 'parallax')): sys.path.insert(0, parent) from parallax.askpass_client import askpass_main askpass_main() parallax-1.0.1/parallax/0000755000175000001440000000000012536574645015255 5ustar krigusers00000000000000parallax-1.0.1/parallax/__init__.py0000644000175000001440000002604312536572525017366 0ustar krigusers00000000000000# Copyright (c) 2013, Kristoffer Gronlund # # Parallax SSH API # # Exposes an API for performing # parallel SSH operations # # Three commands are supplied: # # call(hosts, cmdline, opts) # # copy(hosts, src, dst, opts) # # slurp(hosts, src, dst, opts) # # call returns {host: (rc, stdout, stdin) | error} # copy returns {host: path | error} # slurp returns {host: path | error} # # error is an error object which has an error message (or more) # # opts is bascially command line options # # call: Executes the given command on a set of hosts, collecting the output # copy: Copies files from the local machine to a set of remote hosts # slurp: Copies files from a set of remote hosts to local folders import os import sys DEFAULT_PARALLELISM = 32 DEFAULT_TIMEOUT = 0 # "infinity" by default from parallax.manager import Manager, FatalError from parallax.task import Task try: basestring except NameError: basestring = str class Error(BaseException): """ Returned instead of a result for a host in case of an error during the processing for that host. """ def __init__(self, msg, task): self.msg = msg self.task = task def __str__(self): if self.task and self.task.errorbuffer: return "%s, Error output: %s" % (self.msg, self.task.errorbuffer) return self.msg class Options(object): """ Common options for call, copy and slurp. Note: Setting the inline or inline_stdout options prints the output to stdout unless an alternative manager callback has been set. inline is True by default. This is a change from pssh to parallax. """ limit = DEFAULT_PARALLELISM # Max number of parallel threads timeout = DEFAULT_TIMEOUT # Timeout in seconds askpass = False # Ask for a password outdir = None # Write stdout to a file per host in this directory errdir = None # Write stderr to a file per host in this directory ssh_options = [] # Extra options to pass to SSH ssh_extra = [] # Extra arguments to pass to SSH verbose = False # Warning and diagnostic messages quiet = False # Silence extra output print_out = False # Print output to stdout when received inline = True # Store stdout and stderr in memory buffers inline_stdout = False # Store stdout in memory buffer input_stream = None # Stream to read stdin from default_user = None # User to connect as (unless overridden per host) recursive = True # (copy, slurp only) Copy recursively localdir = None # (slurp only) Local base directory to copy to def _expand_host_port_user(lst): """ Input: list containing hostnames, (host, port)-tuples or (host, port, user)-tuples. Output: list of (host, port, user)-tuples. """ def expand(v): if isinstance(v, basestring): return (v, None, None) elif len(v) == 1: return (v[0], None, None) elif len(v) == 2: return (v[0], v[1], None) else: return v return [expand(x) for x in lst] class _CallOutputBuilder(object): def __init__(self): self.finished_tasks = [] def finished(self, task, n): """Called when Task is complete""" self.finished_tasks.append(task) def result(self, manager): """Called when all Tasks are complete to generate result""" ret = {} for task in self.finished_tasks: if task.failures: ret[task.host] = Error(', '.join(task.failures), task) else: ret[task.host] = (task.exitstatus, task.outputbuffer or manager.outdir, task.errorbuffer or manager.errdir) return ret def _build_call_cmd(host, port, user, cmdline, options, extra): cmd = ['ssh', host, '-o', 'NumberOfPasswordPrompts=1', '-o', 'SendEnv=PARALLAX_NODENUM PARALLAX_HOST'] if options: for opt in options: cmd += ['-o', opt] if user: cmd += ['-l', user] if port: cmd += ['-p', port] if extra: cmd.extend(extra) if cmdline: cmd.append(cmdline) return cmd def call(hosts, cmdline, opts=Options()): """ Executes the given command on a set of hosts, collecting the output Returns {host: (rc, stdout, stdin) | Error} """ if opts.outdir and not os.path.exists(opts.outdir): os.makedirs(opts.outdir) if opts.errdir and not os.path.exists(opts.errdir): os.makedirs(opts.errdir) manager = Manager(limit=opts.limit, timeout=opts.timeout, askpass=opts.askpass, outdir=opts.outdir, errdir=opts.errdir, callbacks=_CallOutputBuilder()) for host, port, user in _expand_host_port_user(hosts): cmd = _build_call_cmd(host, port, user, cmdline, options=opts.ssh_options, extra=opts.ssh_extra) t = Task(host, port, user, cmd, stdin=opts.input_stream, verbose=opts.verbose, quiet=opts.quiet, print_out=opts.print_out, inline=opts.inline, inline_stdout=opts.inline_stdout, default_user=opts.default_user) manager.add_task(t) try: return manager.run() except FatalError: sys.exit(1) class _CopyOutputBuilder(object): def __init__(self): self.finished_tasks = [] def finished(self, task, n): self.finished_tasks.append(task) def result(self, manager): ret = {} for task in self.finished_tasks: if task.failures: ret[task.host] = Error(', '.join(task.failures), task) else: ret[task.host] = (task.exitstatus, task.outputbuffer or manager.outdir, task.errorbuffer or manager.errdir) return ret def _build_copy_cmd(host, port, user, src, dst, opts): cmd = ['scp', '-qC'] if opts.ssh_options: for opt in opts.ssh_options: cmd += ['-o', opt] if port: cmd += ['-P', port] if opts.recursive: cmd.append('-r') if opts.ssh_extra: cmd.extend(opts.ssh_extra) cmd.append(src) if user: cmd.append('%s@%s:%s' % (user, host, dst)) else: cmd.append('%s:%s' % (host, dst)) return cmd def copy(hosts, src, dst, opts=Options()): """ Copies from the local node to a set of remote hosts hosts: [(host, port, user)...] src: local path dst: remote path opts: CopyOptions (optional) Returns {host: (rc, stdout, stdin) | Error} """ if opts.outdir and not os.path.exists(opts.outdir): os.makedirs(opts.outdir) if opts.errdir and not os.path.exists(opts.errdir): os.makedirs(opts.errdir) manager = Manager(limit=opts.limit, timeout=opts.timeout, askpass=opts.askpass, outdir=opts.outdir, errdir=opts.errdir, callbacks=_CopyOutputBuilder()) for host, port, user in _expand_host_port_user(hosts): cmd = _build_copy_cmd(host, port, user, src, dst, opts) t = Task(host, port, user, cmd, stdin=opts.input_stream, verbose=opts.verbose, quiet=opts.quiet, print_out=opts.print_out, inline=opts.inline, inline_stdout=opts.inline_stdout, default_user=opts.default_user) manager.add_task(t) try: return manager.run() except FatalError: sys.exit(1) class _SlurpOutputBuilder(object): def __init__(self, localdirs): self.finished_tasks = [] self.localdirs = localdirs def finished(self, task, n): self.finished_tasks.append(task) def result(self, manager): ret = {} for task in self.finished_tasks: if task.failures: ret[task.host] = Error(', '.join(task.failures), task) else: # TODO: save name of output file in Task ret[task.host] = (task.exitstatus, task.outputbuffer or manager.outdir, task.errorbuffer or manager.errdir, self.localdirs.get(task.host, None) ) return ret def _slurp_make_local_dirs(hosts, dst, opts): if opts.localdir and not os.path.exists(opts.localdir): os.makedirs(opts.localdir) localdirs = {} for host, port, user in _expand_host_port_user(hosts): if opts.localdir: dirname = os.path.join(opts.localdir, host) else: dirname = host if not os.path.exists(dirname): os.makedirs(dirname) localdirs[host] = os.path.join(dirname, dst) return localdirs def _build_slurp_cmd(host, port, user, src, dst, opts): cmd = ['scp', '-qC'] if opts.ssh_options: for opt in opts.ssh_options: cmd += ['-o', opt] if port: cmd += ['-P', port] if opts.recursive: cmd.append('-r') if opts.ssh_extra: cmd.extend(opts.ssh_extra) if user: cmd.append('%s@%s:%s' % (user, host, src)) else: cmd.append('%s:%s' % (host, src)) cmd.append(dst) return cmd def slurp(hosts, src, dst, opts=Options()): """ Copies from the remote node to the local node hosts: [(host, port, user)...] src: remote path dst: local path opts: CopyOptions (optional) Returns {host: (rc, stdout, stdin, localpath) | Error} """ if os.path.isabs(dst): raise ValueError("slurp: Destination must be a relative path") localdirs = _slurp_make_local_dirs(hosts, dst, opts) if opts.outdir and not os.path.exists(opts.outdir): os.makedirs(opts.outdir) if opts.errdir and not os.path.exists(opts.errdir): os.makedirs(opts.errdir) manager = Manager(limit=opts.limit, timeout=opts.timeout, askpass=opts.askpass, outdir=opts.outdir, errdir=opts.errdir, callbacks=_SlurpOutputBuilder(localdirs)) for host, port, user in _expand_host_port_user(hosts): localpath = localdirs[host] cmd = _build_slurp_cmd(host, port, user, src, localpath, opts) t = Task(host, port, user, cmd, stdin=opts.input_stream, verbose=opts.verbose, quiet=opts.quiet, print_out=opts.print_out, inline=opts.inline, inline_stdout=opts.inline_stdout, default_user=opts.default_user) manager.add_task(t) try: return manager.run() except FatalError: sys.exit(1) parallax-1.0.1/parallax/psshutil.py0000644000175000001440000000766212471655014017501 0ustar krigusers00000000000000# Copyright (c) 2009-2012, Andrew McNabb # Copyright (c) 2003-2008, Brent N. Chun import fcntl import sys import subprocess HOST_FORMAT = 'Host format is [user@]host[:port] [user]' def read_host_files(paths, default_user=None, default_port=None): """Reads the given host files. Returns a list of (host, port, user) triples. """ hosts = [] if paths: for path in paths: hosts.extend(read_host_file(path, default_user=default_user)) return hosts def read_host_file(path, default_user=None, default_port=None): """Reads the given host file. Lines are of the form: host[:port] [login]. Returns a list of (host, port, user) triples. """ lines = [] f = open(path) for line in f: lines.append(line.strip()) f.close() hosts = [] for line in lines: # Skip blank lines or lines starting with # line = line.strip() if not line or line.startswith('#'): continue host, port, user = parse_host_entry(line, default_user, default_port) if host: hosts.append((host, port, user)) return hosts # TODO: deprecate the second host field and standardize on the # [user@]host[:port] format. def parse_host_entry(line, default_user, default_port): """Parses a single host entry. This may take either the of the form [user@]host[:port] or host[:port][ user]. Returns a (host, port, user) triple. """ fields = line.split() if len(fields) > 2: sys.stderr.write('Bad line: "%s". Format should be' ' [user@]host[:port] [user]\n' % line) return None, None, None host_field = fields[0] host, port, user = parse_host(host_field, default_port=default_port) if len(fields) == 2: if user is None: user = fields[1] else: sys.stderr.write('User specified twice in line: "%s"\n' % line) return None, None, None if user is None: user = default_user return host, port, user def parse_host_string(host_string, default_user=None, default_port=None): """Parses a whitespace-delimited string of "[user@]host[:port]" entries. Returns a list of (host, port, user) triples. """ hosts = [] entries = host_string.split() for entry in entries: hosts.append(parse_host(entry, default_user, default_port)) return hosts def parse_host(host, default_user=None, default_port=None): """Parses host entries of the form "[user@]host[:port]". Returns a (host, port, user) triple. """ # TODO: when we stop supporting Python 2.4, switch to using str.partition. user = default_user port = default_port if '@' in host: user, host = host.split('@', 1) if ':' in host: host, port = host.rsplit(':', 1) return (host, port, user) def get_pacemaker_nodes(): """Get the list of nodes from crm_node -l. Returns a list of (host, port, user) triples. """ hosts = [] if subprocess.call("which crm_node >/dev/null 2>&1", shell=True) != 0: sys.stderr.write('crm_node not available\n') return hosts cmd = "crm_node -l" p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE) try: outp = p.communicate()[0] p.wait() rc = p.returncode except IOError as msg: sys.stderr.write('%s failed: %s\n' % (cmd, msg)) return hosts if rc != 0: sys.stderr.write('%s failed: exit code %d\n' % (cmd, rc)) return hosts for s in outp.split('\n'): a = s.split() if len(a) < 2: continue hosts.append((a[1], None, None)) return hosts def set_cloexec(filelike): """Sets the underlying filedescriptor to automatically close on exec. If set_cloexec is called for all open files, then subprocess.Popen does not require the close_fds option. """ fcntl.fcntl(filelike.fileno(), fcntl.FD_CLOEXEC, 1) # vim:ts=4:sw=4:et: parallax-1.0.1/parallax/askpass_server.py0000644000175000001440000000561712471655252020663 0ustar krigusers00000000000000# -*- Mode: python -*- # Copyright (c) 2009-2012, Andrew McNabb """Sends the password over a socket to askpass. """ import errno import getpass import os import socket import sys import tempfile import textwrap from parallax import psshutil class PasswordServer(object): """Listens on a UNIX domain socket for password requests.""" def __init__(self): self.sock = None self.tempdir = None self.address = None self.socketmap = {} self.buffermap = {} def start(self, iomap, backlog): """Prompts for the password, creates a socket, and starts listening. The specified backlog should be the max number of clients connecting at once. """ message = ('Warning: do not enter your password if anyone else has' ' superuser privileges or access to your account.') print(textwrap.fill(message)) self.password = getpass.getpass() # Note that according to the docs for mkdtemp, "The directory is # readable, writable, and searchable only by the creating user." self.tempdir = tempfile.mkdtemp(prefix='parallax.') self.address = os.path.join(self.tempdir, 'parallax_askpass_socket') self.sock = socket.socket(socket.AF_UNIX) psshutil.set_cloexec(self.sock) self.sock.bind(self.address) self.sock.listen(backlog) iomap.register_read(self.sock.fileno(), self.handle_listen) def handle_listen(self, fd, iomap): try: conn = self.sock.accept()[0] except socket.error: _, e, _ = sys.exc_info() number = e.args[0] if number == errno.EINTR: return else: # TODO: print an error message here? self.sock.close() self.sock = None fd = conn.fileno() iomap.register_write(fd, self.handle_write) self.socketmap[fd] = conn self.buffermap[fd] = self.password.encode() def handle_write(self, fd, iomap): buffer = self.buffermap[fd] conn = self.socketmap[fd] try: bytes_written = conn.send(buffer) except socket.error: _, e, _ = sys.exc_info() number = e.args[0] if number == errno.EINTR: return else: self.close_socket(fd, iomap) buffer = buffer[bytes_written:] if buffer: self.buffermap[fd] = buffer else: self.close_socket(fd, iomap) def close_socket(self, fd, iomap): iomap.unregister(fd) self.socketmap[fd].close() del self.socketmap[fd] del self.buffermap[fd] def __del__(self): if self.sock: self.sock.close() self.sock = None if self.address: os.remove(self.address) if self.tempdir: os.rmdir(self.tempdir) parallax-1.0.1/parallax/manager.py0000644000175000001440000003500012471655014017223 0ustar krigusers00000000000000# Copyright (c) 2009-2012, Andrew McNabb # Copyright (c) 2013, Kristoffer Gronlund from errno import EINTR import os import select import signal import sys import threading import copy try: import queue except ImportError: import Queue as queue from parallax.askpass_server import PasswordServer from parallax import psshutil from parallax import DEFAULT_PARALLELISM, DEFAULT_TIMEOUT from parallax.callbacks import DefaultCallbacks READ_SIZE = 1 << 16 class FatalError(RuntimeError): """A fatal error in the Parallax SSH Manager.""" pass class Manager(object): """Executes tasks concurrently. Tasks are added with add_task() and executed in parallel with run(). Returns a list of the exit statuses of the processes. Arguments: limit: Maximum number of commands running at once. timeout: Maximum allowed execution time in seconds. """ def __init__(self, limit=DEFAULT_PARALLELISM, timeout=DEFAULT_TIMEOUT, askpass=False, outdir=None, errdir=None, callbacks=DefaultCallbacks()): # Backwards compatibility with old __init__ # format: Only argument is an options dict if not isinstance(limit, int): if hasattr(limit, 'limit'): self.limit = limit.limit elif hasattr(limit, 'par'): self.limit = limit.par else: self.limit = DEFAULT_PARALLELISM if hasattr(limit, 'timeout'): self.timeout = limit.timeout else: self.timeout = DEFAULT_TIMEOUT if hasattr(limit, 'askpass'): self.askpass = limit.askpass else: self.askpass = False if hasattr(limit, 'outdir'): self.outdir = limit.outdir else: self.outdir = None if hasattr(limit, 'errdir'): self.errdir = limit.errdir else: self.errdir = None else: self.limit = limit self.timeout = timeout self.askpass = askpass self.outdir = outdir self.errdir = errdir self.iomap = make_iomap() self.callbacks = callbacks self.taskcount = 0 self.tasks = [] self.save_tasks = [] self.running = [] self.done = [] self.askpass_socket = None def run(self): """Processes tasks previously added with add_task.""" self.save_tasks = copy.copy(self.tasks) try: if self.outdir or self.errdir: writer = Writer(self.outdir, self.errdir) writer.start() else: writer = None if self.askpass: pass_server = PasswordServer() pass_server.start(self.iomap, self.limit) self.askpass_socket = pass_server.address self.set_sigchld_handler() try: self.update_tasks(writer) wait = None while self.running or self.tasks: # Opt for efficiency over subsecond timeout accuracy. if wait is None or wait < 1: wait = 1 self.iomap.poll(wait) self.update_tasks(writer) wait = self.check_timeout() except KeyboardInterrupt: # This exception handler tries to clean things up and prints # out a nice status message for each interrupted host. self.interrupted() except KeyboardInterrupt: # This exception handler doesn't print out any fancy status # information--it just stops. pass if writer: writer.signal_quit() writer.join() return self.callbacks.result(self) def clear_sigchld_handler(self): signal.signal(signal.SIGCHLD, signal.SIG_DFL) def set_sigchld_handler(self): # TODO: find out whether set_wakeup_fd still works if the default # signal handler is used (I'm pretty sure it doesn't work if the # signal is ignored). signal.signal(signal.SIGCHLD, self.handle_sigchld) # This should keep reads and writes from getting EINTR. if hasattr(signal, 'siginterrupt'): signal.siginterrupt(signal.SIGCHLD, False) def handle_sigchld(self, number, frame): """Apparently we need a sigchld handler to make set_wakeup_fd work.""" # Write to the signal pipe (only for Python <2.5, where the # set_wakeup_fd method doesn't exist). if self.iomap.wakeup_writefd: os.write(self.iomap.wakeup_writefd, '\0') for task in self.running: if task.proc: task.proc.poll() # Apparently some UNIX systems automatically reset the SIGCHLD # handler to SIG_DFL. Reset it just in case. self.set_sigchld_handler() def add_task(self, task): """Adds a Task to be processed with run().""" self.tasks.append(task) def update_tasks(self, writer): """Reaps tasks and starts as many new ones as allowed.""" # Mask signals to work around a Python bug: # http://bugs.python.org/issue1068268 # Since sigprocmask isn't in the stdlib, clear the SIGCHLD handler. # Since signals are masked, reap_tasks needs to be called once for # each loop. keep_running = True while keep_running: self.clear_sigchld_handler() self._start_tasks_once(writer) self.set_sigchld_handler() keep_running = self.reap_tasks() def _start_tasks_once(self, writer): """Starts tasks once. Due to http://bugs.python.org/issue1068268, signals must be masked when this method is called. """ while 0 < len(self.tasks) and len(self.running) < self.limit: task = self.tasks.pop(0) self.running.append(task) task.start(self.taskcount, self.iomap, writer, self.askpass_socket) self.taskcount += 1 def reap_tasks(self): """Checks to see if any tasks have terminated. After cleaning up, returns the number of tasks that finished. """ still_running = [] finished_count = 0 for task in self.running: if task.running(): still_running.append(task) else: self.finished(task) finished_count += 1 self.running = still_running return finished_count def check_timeout(self): """Kills timed-out processes and returns the lowest time left.""" if self.timeout <= 0: return None min_timeleft = None for task in self.running: timeleft = self.timeout - task.elapsed() if timeleft <= 0: task.timedout() continue if min_timeleft is None or timeleft < min_timeleft: min_timeleft = timeleft if min_timeleft is None: return 0 else: return max(0, min_timeleft) def interrupted(self): """Cleans up after a keyboard interrupt.""" for task in self.running: task.interrupted() self.finished(task) for task in self.tasks: task.cancel() self.finished(task) def finished(self, task): """Marks a task as complete and reports its status as finished.""" self.done.append(task) n = len(self.done) self.callbacks.finished(task, n) class IOMap(object): """A manager for file descriptors and their associated handlers. The poll method dispatches events to the appropriate handlers. """ def __init__(self): self.readmap = {} self.writemap = {} # Setup the wakeup file descriptor to avoid hanging on lost signals. wakeup_readfd, wakeup_writefd = os.pipe() self.register_read(wakeup_readfd, self.wakeup_handler) # TODO: remove test when we stop supporting Python <2.5 if hasattr(signal, 'set_wakeup_fd'): signal.set_wakeup_fd(wakeup_writefd) self.wakeup_writefd = None else: self.wakeup_writefd = wakeup_writefd def register_read(self, fd, handler): """Registers an IO handler for a file descriptor for reading.""" self.readmap[fd] = handler def register_write(self, fd, handler): """Registers an IO handler for a file descriptor for writing.""" self.writemap[fd] = handler def unregister(self, fd): """Unregisters the given file descriptor.""" if fd in self.readmap: del self.readmap[fd] if fd in self.writemap: del self.writemap[fd] def poll(self, timeout=None): """Performs a poll and dispatches the resulting events.""" if not self.readmap and not self.writemap: return rlist = list(self.readmap) wlist = list(self.writemap) try: rlist, wlist, _ = select.select(rlist, wlist, [], timeout) except select.error: _, e, _ = sys.exc_info() errno = e.args[0] if errno == EINTR: return else: raise for fd in rlist: handler = self.readmap[fd] handler(fd, self) for fd in wlist: handler = self.writemap[fd] handler(fd, self) def wakeup_handler(self, fd, iomap): """Handles read events on the signal wakeup pipe. This ensures that SIGCHLD signals aren't lost. """ try: os.read(fd, READ_SIZE) except (OSError, IOError): _, e, _ = sys.exc_info() errno, message = e.args if errno != EINTR: sys.stderr.write('Fatal error reading from wakeup pipe: %s\n' % message) raise FatalError class PollIOMap(IOMap): """A manager for file descriptors and their associated handlers. The poll method dispatches events to the appropriate handlers. Note that `select.poll` is not available on all operating systems. """ def __init__(self): self._poller = select.poll() super(PollIOMap, self).__init__() def register_read(self, fd, handler): """Registers an IO handler for a file descriptor for reading.""" super(PollIOMap, self).register_read(fd, handler) self._poller.register(fd, select.POLLIN) def register_write(self, fd, handler): """Registers an IO handler for a file descriptor for writing.""" super(PollIOMap, self).register_write(fd, handler) self._poller.register(fd, select.POLLOUT) def unregister(self, fd): """Unregisters the given file descriptor.""" super(PollIOMap, self).unregister(fd) self._poller.unregister(fd) def poll(self, timeout=None): """Performs a poll and dispatches the resulting events.""" if not self.readmap and not self.writemap: return try: event_list = self._poller.poll(timeout) except select.error: _, e, _ = sys.exc_info() errno = e.args[0] if errno == EINTR: return else: raise for fd, event in event_list: if event & (select.POLLIN | select.POLLHUP): handler = self.readmap[fd] handler(fd, self) if event & (select.POLLOUT | select.POLLERR): handler = self.writemap[fd] handler(fd, self) def make_iomap(): """Return a new IOMap or PollIOMap as appropriate. Since `select.poll` is not implemented on all platforms, this ensures that the most appropriate implementation is used. """ if hasattr(select, 'poll'): return PollIOMap() else: return IOMap() class Writer(threading.Thread): """Thread that writes to files by processing requests from a Queue. Until AIO becomes widely available, it is impossible to make a nonblocking write to an ordinary file. The Writer thread processes all writing to ordinary files so that the main thread can work without blocking. """ OPEN = object() EOF = object() ABORT = object() def __init__(self, outdir, errdir): threading.Thread.__init__(self) # A daemon thread automatically dies if the program is terminated. self.setDaemon(True) self.queue = queue.Queue() self.outdir = outdir self.errdir = errdir self.host_counts = {} self.files = {} def run(self): while True: filename, data = self.queue.get() if filename == self.ABORT: return if data == self.OPEN: self.files[filename] = open(filename, 'wb', buffering=1) psshutil.set_cloexec(self.files[filename]) else: dest = self.files[filename] if data == self.EOF: dest.close() else: dest.write(data) dest.flush() def open_files(self, host): """Called from another thread to create files for stdout and stderr. Returns a pair of filenames (outfile, errfile). These filenames are used as handles for future operations. Either or both may be None if outdir or errdir or not set. """ outfile = errfile = None if self.outdir or self.errdir: count = self.host_counts.get(host, 0) self.host_counts[host] = count + 1 if count: filename = "%s.%s" % (host, count) else: filename = host if self.outdir: outfile = os.path.join(self.outdir, filename) self.queue.put((outfile, self.OPEN)) if self.errdir: errfile = os.path.join(self.errdir, filename) self.queue.put((errfile, self.OPEN)) return outfile, errfile def write(self, filename, data): """Called from another thread to enqueue a write.""" self.queue.put((filename, data)) def close(self, filename): """Called from another thread to close the given file.""" self.queue.put((filename, self.EOF)) def signal_quit(self): """Called from another thread to request the Writer to quit.""" self.queue.put((self.ABORT, None)) parallax-1.0.1/parallax/color.py0000644000175000001440000000251012471655014016727 0ustar krigusers00000000000000# Copyright (c) 2009-2012, Andrew McNabb # Copyright (c) 2003-2008, Brent N. Chun def with_color(string, fg, bg=49): '''Given foreground/background ANSI color codes, return a string that, when printed, will format the supplied string using the supplied colors. ''' return "\x1b[%dm\x1b[%dm%s\x1b[39m\x1b[49m" % (fg, bg, string) def B(string): '''Returns a string that, when printed, will display the supplied string in ANSI bold. ''' return "\x1b[1m%s\x1b[22m" % string def r(string): return with_color(string, 31) # Red def g(string): return with_color(string, 32) # Green def y(string): return with_color(string, 33) # Yellow def b(string): return with_color(string, 34) # Blue def m(string): return with_color(string, 35) # Magenta def c(string): return with_color(string, 36) # Cyan def w(string): return with_color(string, 37) # White #following from Python cookbook, #475186 def has_colors(stream): '''Returns boolean indicating whether or not the supplied stream supports ANSI color. ''' if not hasattr(stream, "isatty"): return False if not stream.isatty(): return False # auto color only on TTYs try: import curses curses.setupterm() return curses.tigetnum("colors") > 2 except: # guess false in case of error return False parallax-1.0.1/parallax/callbacks.py0000644000175000001440000000423012536572525017540 0ustar krigusers00000000000000# Copyright (c) 2009-2012, Andrew McNabb # Copyright (c) 2013, Kristoffer Gronlund import sys import time from parallax import color class DefaultCallbacks(object): """ Passed to the Manager and called when events occur. """ def finished(self, task, n): """Pretty prints a status report after the Task completes. task: a Task object n: Index in sequence of completed tasks. """ error = ', '.join(task.failures) tstamp = time.asctime().split()[3] # Current time if color.has_colors(sys.stdout): progress = color.c("[%s]" % color.B(n)) success = color.g("[%s]" % color.B("SUCCESS")) failure = color.r("[%s]" % color.B("FAILURE")) stderr = color.r("Stderr: ") error = color.r(color.B(error)) else: progress = "[%s]" % n success = "[SUCCESS]" failure = "[FAILURE]" stderr = "Stderr: " host = task.pretty_host if not task.quiet: if task.failures: print(' '.join((progress, tstamp, failure, host, error))) else: print(' '.join((progress, tstamp, success, host))) # NOTE: The extra flushes are to ensure that the data is output in # the correct order with the C implementation of io. if task.inline_stdout and task.outputbuffer: sys.stdout.flush() try: sys.stdout.buffer.write(task.outputbuffer) sys.stdout.flush() except AttributeError: sys.stdout.write(task.outputbuffer) if task.inline and task.errorbuffer: sys.stdout.write(stderr) # Flush the TextIOWrapper before writing to the binary buffer. sys.stdout.flush() try: sys.stdout.buffer.write(task.errorbuffer) except AttributeError: sys.stdout.write(task.errorbuffer) def result(self, manager): """ When all Tasks are completed, generate a result to return. """ return [task.exitstatus for task in manager.save_tasks if task in manager.done] parallax-1.0.1/parallax/askpass_client.py0000644000175000001440000000675612471655014020634 0ustar krigusers00000000000000# -*- Mode: python -*- # Copyright (c) 2009-2012, Andrew McNabb """Implementation of SSH_ASKPASS to get a password to ssh from parallax. The password is read from the socket specified by the environment variable PARALLAX_ASKPASS_SOCKET. The other end of this socket is parallax. The ssh man page discusses SSH_ASKPASS as follows: If ssh needs a passphrase, it will read the passphrase from the current terminal if it was run from a terminal. If ssh does not have a terminal associated with it but DISPLAY and SSH_ASKPASS are set, it will execute the program specified by SSH_ASKPASS and open an X11 window to read the passphrase. This is particularly useful when calling ssh from a .xsession or related script. (Note that on some machines it may be necessary to redirect the input from /dev/null to make this work.) """ import os import socket import sys import textwrap bin_dir = os.path.dirname(os.path.abspath(sys.argv[0])) askpass_bin_path = os.path.join(bin_dir, 'parallax-askpass') ASKPASS_PATHS = (askpass_bin_path, '/usr/bin/parallax-askpass', '/usr/libexec/parallax/parallax-askpass', '/usr/local/libexec/parallax/parallax-askpass', '/usr/lib/parallax/parallax-askpass', '/usr/local/lib/parallax/parallax-askpass') _executable_path = None def executable_path(): """Determines the value to use for SSH_ASKPASS. The value is cached since this may be called many times. """ global _executable_path if _executable_path is None: for path in ASKPASS_PATHS: if os.access(path, os.X_OK): _executable_path = path break else: _executable_path = '' sys.stderr.write(textwrap.fill("Warning: could not find an" " executable path for askpass because Parallax SSH was not" " installed correctly. Password prompts will not work.")) sys.stderr.write('\n') return _executable_path def askpass_main(): """Connects to parallax over the socket specified at PARALLAX_ASKPASS_SOCKET.""" verbose = os.getenv('PARALLAX_ASKPASS_VERBOSE') # It's not documented anywhere, as far as I can tell, but ssh may prompt # for a password or ask a yes/no question. The command-line argument # specifies what is needed. if len(sys.argv) > 1: prompt = sys.argv[1] if verbose: sys.stderr.write('parallax-askpass received prompt: "%s"\n' % prompt) if not prompt.strip().lower().endswith('password:'): sys.stderr.write(prompt) sys.stderr.write('\n') sys.exit(1) else: sys.stderr.write('Error: parallax-askpass called without a prompt.\n') sys.exit(1) address = os.getenv('PARALLAX_ASKPASS_SOCKET') if not address: sys.stderr.write(textwrap.fill("parallax error: SSH requested a password." " Please create SSH keys or use the -A option to provide a" " password.")) sys.stderr.write('\n') sys.exit(1) sock = socket.socket(socket.AF_UNIX) try: sock.connect(address) except socket.error: _, e, _ = sys.exc_info() message = e.args[1] sys.stderr.write("Couldn't bind to %s: %s.\n" % (address, message)) sys.exit(2) try: password = sock.makefile().read() except socket.error: sys.stderr.write("Socket error.\n") sys.exit(3) print(password) if __name__ == '__main__': askpass_main() parallax-1.0.1/parallax/version.py0000644000175000001440000000002212536574235017301 0ustar krigusers00000000000000VERSION = '1.0.1' parallax-1.0.1/parallax/task.py0000644000175000001440000002213712536572525016571 0ustar krigusers00000000000000# Copyright (c) 2009-2012, Andrew McNabb # Copyright (c) 2013, Kristoffer Gronlund from errno import EINTR from subprocess import Popen, PIPE import os import signal import sys import time import traceback from parallax import askpass_client BUFFER_SIZE = 1 << 16 try: bytes except NameError: bytes = str class Task(object): """Starts a process and manages its input and output. Upon completion, the `exitstatus` attribute is set to the exit status of the process. """ def __init__(self, host, port, user, cmd, verbose=False, quiet=False, stdin=None, print_out=False, inline=False, inline_stdout=False, default_user=None): # Backwards compatibility: if not isinstance(verbose, bool): opts = verbose verbose = opts.verbose quiet = opts.quiet try: print_out = bool(opts.print_out) except AttributeError: print_out = False try: inline = bool(opts.inline) except AttributeError: inline = False try: inline_stdout = bool(opts.inline_stdout) except AttributeError: inline_stdout = False default_user = opts.user self.exitstatus = None self.host = host self.pretty_host = host self.port = port self.cmd = cmd if user and user != default_user: self.pretty_host = '@'.join((user, self.pretty_host)) if port: self.pretty_host = ':'.join((self.pretty_host, port)) self.proc = None self.writer = None self.timestamp = None self.failures = [] self.killed = False self.inputbuffer = stdin self.byteswritten = 0 self.outputbuffer = bytes() self.errorbuffer = bytes() self.stdin = None self.stdout = None self.stderr = None self.outfile = None self.errfile = None # Set options. self.verbose = verbose self.quiet = quiet self.print_out = print_out self.inline = inline self.inline_stdout = inline or inline_stdout def start(self, nodenum, iomap, writer, askpass_socket=None): """Starts the process and registers files with the IOMap.""" self.writer = writer if writer: self.outfile, self.errfile = writer.open_files(self.pretty_host) # Set up the environment. environ = dict(os.environ) environ['PARALLAX_NODENUM'] = str(nodenum) environ['PARALLAX_HOST'] = self.host # Disable the GNOME pop-up password dialog and allow ssh to use # askpass.py to get a provided password. If the module file is # askpass.pyc, we replace the extension. environ['SSH_ASKPASS'] = askpass_client.executable_path() if askpass_socket: environ['PARALLAX_ASKPASS_SOCKET'] = askpass_socket if self.verbose: environ['PARALLAX_ASKPASS_VERBOSE'] = '1' # Work around a mis-feature in ssh where it won't call SSH_ASKPASS # if DISPLAY is unset. if 'DISPLAY' not in environ: environ['DISPLAY'] = 'parallax-gibberish' # Create the subprocess. Since we carefully call set_cloexec() on # all open files, we specify close_fds=False. self.proc = Popen(self.cmd, stdin=PIPE, stdout=PIPE, stderr=PIPE, close_fds=False, preexec_fn=os.setsid, env=environ) self.timestamp = time.time() if self.inputbuffer: self.stdin = self.proc.stdin iomap.register_write(self.stdin.fileno(), self.handle_stdin) else: self.proc.stdin.close() self.stdout = self.proc.stdout iomap.register_read(self.stdout.fileno(), self.handle_stdout) self.stderr = self.proc.stderr iomap.register_read(self.stderr.fileno(), self.handle_stderr) def _kill(self): """Signals the process to terminate.""" if self.proc: try: os.kill(-self.proc.pid, signal.SIGKILL) except OSError: # If the kill fails, then just assume the process is dead. pass self.killed = True def timedout(self): """Kills the process and registers a timeout error.""" if not self.killed: self._kill() self.failures.append('Timed out') def interrupted(self): """Kills the process and registers an keyboard interrupt error.""" if not self.killed: self._kill() self.failures.append('Interrupted') def cancel(self): """Stops a task that has not started.""" self.failures.append('Cancelled') def elapsed(self): """Finds the time in seconds since the process was started.""" return time.time() - self.timestamp def running(self): """Finds if the process has terminated and saves the return code.""" if self.stdin or self.stdout or self.stderr: return True if self.proc: self.exitstatus = self.proc.poll() if self.exitstatus is None: if self.killed: # Set the exitstatus to what it would be if we waited. self.exitstatus = -signal.SIGKILL return False else: return True else: if self.exitstatus < 0: message = 'Killed by signal %s' % (-self.exitstatus) self.failures.append(message) elif self.exitstatus > 0: message = 'Exited with error code %s' % self.exitstatus self.failures.append(message) self.proc = None return False def handle_stdin(self, fd, iomap): """Called when the process's standard input is ready for writing.""" try: start = self.byteswritten if start < len(self.inputbuffer): chunk = self.inputbuffer[start:start+BUFFER_SIZE] self.byteswritten = start + os.write(fd, chunk) else: self.close_stdin(iomap) except (OSError, IOError): _, e, _ = sys.exc_info() if e.errno != EINTR: self.close_stdin(iomap) self.log_exception(e) def close_stdin(self, iomap): if self.stdin: iomap.unregister(self.stdin.fileno()) self.stdin.close() self.stdin = None def handle_stdout(self, fd, iomap): """Called when the process's standard output is ready for reading.""" try: buf = os.read(fd, BUFFER_SIZE) if buf: if self.inline_stdout: if self.quiet: self.outputbuffer += "%s: %s" % (self.host, buf) else: self.outputbuffer += buf if self.outfile: self.writer.write(self.outfile, buf) if self.print_out: for l in buf.split('\n'): sys.stdout.write('%s: %s\n' % (self.host, l)) else: self.close_stdout(iomap) except (OSError, IOError): _, e, _ = sys.exc_info() if e.errno != EINTR: self.close_stdout(iomap) self.log_exception(e) def close_stdout(self, iomap): if self.stdout: iomap.unregister(self.stdout.fileno()) self.stdout.close() self.stdout = None if self.outfile: self.writer.close(self.outfile) self.outfile = None def handle_stderr(self, fd, iomap): """Called when the process's standard error is ready for reading.""" try: buf = os.read(fd, BUFFER_SIZE) if buf: if self.inline: self.errorbuffer += buf if self.errfile: self.writer.write(self.errfile, buf) else: self.close_stderr(iomap) except (OSError, IOError): _, e, _ = sys.exc_info() if e.errno != EINTR: self.close_stderr(iomap) self.log_exception(e) def close_stderr(self, iomap): if self.stderr: iomap.unregister(self.stderr.fileno()) self.stderr.close() self.stderr = None if self.errfile: self.writer.close(self.errfile) self.errfile = None def log_exception(self, e): """Saves a record of the most recent exception for error reporting.""" if self.verbose: exc_type, exc_value, exc_traceback = sys.exc_info() exc = ("Exception: %s, %s, %s" % (exc_type, exc_value, traceback.format_tb(exc_traceback))) else: exc = str(e) self.failures.append(exc) # vim:ts=4:sw=4:et: parallax-1.0.1/test/0000755000175000001440000000000012536574645014430 5ustar krigusers00000000000000parallax-1.0.1/test/test_api.py0000644000175000001440000000524212471655014016601 0ustar krigusers00000000000000#!/usr/bin/python # Copyright (c) 2013, Kristoffer Gronlund import os import sys import unittest import tempfile import shutil basedir, bin = os.path.split(os.path.dirname(os.path.abspath(sys.argv[0]))) sys.path.insert(0, "%s" % basedir) print basedir import parallax as para if os.getenv("TEST_HOSTS") is None: raise Exception("Must define TEST_HOSTS") g_hosts = os.getenv("TEST_HOSTS").split() if os.getenv("TEST_USER") is None: raise Exception("Must define TEST_USER") g_user = os.getenv("TEST_USER") class CallTest(unittest.TestCase): def testSimpleCall(self): opts = para.Options() opts.default_user = g_user for host, result in para.call(g_hosts, "ls -l /", opts).iteritems(): if isinstance(result, para.Error): raise result rc, out, err = result self.assertEqual(rc, 0) self.assert_(len(out) > 0) def testUptime(self): opts = para.Options() opts.default_user = g_user for host, result in para.call(g_hosts, "uptime", opts).iteritems(): if isinstance(result, para.Error): raise result rc, out, err = result self.assertEqual(rc, 0) self.assert_(out.find("load average") != -1) def testFailingCall(self): opts = para.Options() opts.default_user = g_user for host, result in para.call(g_hosts, "touch /foofoo/barbar/jfikjfdj", opts).iteritems(): self.assert_(isinstance(result, para.Error)) self.assert_(str(result).find('with error code') != -1) class CopySlurpTest(unittest.TestCase): def setUp(self): self.tmpDir = tempfile.mkdtemp() def tearDown(self): shutil.rmtree(self.tmpDir) def testCopyFile(self): opts = para.Options() opts.default_user = g_user opts.localdir = self.tmpDir by_host = para.copy(g_hosts, "/etc/hosts", "/tmp/para.test", opts) for host, result in by_host.iteritems(): if isinstance(result, para.Error): raise result rc, _, _ = result self.assertEqual(rc, 0) by_host = para.slurp(g_hosts, "/tmp/para.test", "para.test", opts) for host, result in by_host.iteritems(): if isinstance(result, para.Error): raise result rc, _, _, path = result self.assertEqual(rc, 0) self.assert_(path.endswith('%s/para.test' % (host))) if __name__ == '__main__': suite = unittest.TestSuite() suite.addTest(unittest.makeSuite(CallTest, "test")) suite.addTest(unittest.makeSuite(CopySlurpTest, "test")) unittest.TextTestRunner().run(suite) parallax-1.0.1/PKG-INFO0000644000175000001440000000243412536574645014551 0ustar krigusers00000000000000Metadata-Version: 1.1 Name: parallax Version: 1.0.1 Summary: Execute commands and copy files over SSH to multiple machines at once Home-page: https://github.com/krig/parallax/ Author: Kristoffer Gronlund Author-email: krig@koru.se License: BSD Description: Parallax SSH provides an interface to executing commands on multiple nodes at once using SSH. It also provides commands for sending and receiving files to multiple nodes using SCP. Platform: linux Classifier: Development Status :: 3 - Alpha Classifier: Intended Audience :: System Administrators Classifier: License :: OSI Approved :: BSD License Classifier: Operating System :: POSIX Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: 2 Classifier: Programming Language :: Python :: 2.6 Classifier: Programming Language :: Python :: 2.7 Classifier: Programming Language :: Python :: 3 Classifier: Programming Language :: Python :: 3.1 Classifier: Programming Language :: Python :: 3.2 Classifier: Programming Language :: Python :: 3.3 Classifier: Programming Language :: Python :: 3.4 Classifier: Topic :: Software Development :: Libraries :: Python Modules Classifier: Topic :: System :: Clustering Classifier: Topic :: System :: Networking Classifier: Topic :: System :: Systems Administration parallax-1.0.1/COPYING0000644000175000001440000000276712471655014014504 0ustar krigusers00000000000000Copyright (c) 2009, Andrew McNabb Copyright (c) 2003-2008, Brent N. Chun All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * The names of its contributors may not be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. parallax-1.0.1/AUTHORS0000644000175000001440000000023012471655014014500 0ustar krigusers00000000000000Andrew McNabb Brent Chun Kristoffer Gronlund Vladislav Bogdanov