magic-wormhole-0.9.1/0000755000076600000240000000000013032243761015056 5ustar warnerstaff00000000000000magic-wormhole-0.9.1/.coveragerc0000644000076600000240000000127112721157076017207 0ustar warnerstaff00000000000000# -*- mode: conf -*- [run] # only record trace data for wormhole.* source = wormhole # and don't trace the test files themselves, or Versioneer's stuff omit = src/wormhole/test/* src/wormhole/_version.py # This allows 'coverage combine' to correlate the tracing data built while # running tests in multiple tox virtualenvs. To take advantage of this # properly, use "coverage erase" before tox, "coverage run --parallel-mode" # inside tox to avoid overwriting the output data (by writing it into # .coverage-XYZ instead of just .coverage), and run "coverage combine" # afterwards. [paths] source = src/ .tox/*/lib/python*/site-packages/ .tox/pypy*/site-packages/ magic-wormhole-0.9.1/docs/0000755000076600000240000000000013032243761016006 5ustar warnerstaff00000000000000magic-wormhole-0.9.1/docs/api.md0000644000076600000240000002201212721763465017112 0ustar warnerstaff00000000000000# Magic-Wormhole This library provides a primitive function to securely transfer small amounts of data between two computers. Both machines must be connected to the internet, but they do not need to have public IP addresses or know how to contact each other ahead of time. Security and connectivity is provided by means of an "invitation code": a short string that is transcribed from one machine to the other by the users at the keyboard. This works in conjunction with a baked-in "rendezvous server" that relays information from one machine to the other. The "Wormhole" object provides a secure record pipe between any two programs that use the same wormhole code (and are configured with the same application ID and rendezvous server). Each side can send multiple messages to the other, but the encrypted data for all messages must pass through (and be temporarily stored on) the rendezvous server, which is a shared resource. For this reason, larger data (including bulk file transfers) should use the Transit class instead. The Wormhole object has a method to create a Transit object for this purpose. ## Modes This library will eventually offer multiple modes. For now, only "transcribe mode" is available. Transcribe mode has two variants. In the "machine-generated" variant, the "initiator" machine creates the invitation code, displays it to the first user, they convey it (somehow) to the second user, who transcribes it into the second ("receiver") machine. In the "human-generated" variant, the two humans come up with the code (possibly without computers), then later transcribe it into both machines. When the initiator machine generates the invitation code, the initiator contacts the rendezvous server and allocates a "channel ID", which is a small integer. The initiator then displays the invitation code, which is the channel-ID plus a few secret words. The user copies the code to the second machine. The receiver machine connects to the rendezvous server, and uses the invitation code to contact the initiator. They agree upon an encryption key, and exchange a small encrypted+authenticated data message. When the humans create an invitation code out-of-band, they are responsible for choosing an unused channel-ID (simply picking a random 3-or-more digit number is probably enough), and some random words. The invitation code uses the same format in either variant: channel-ID, a hyphen, and an arbitrary string. The two machines participating in the wormhole setup are not distinguished: it doesn't matter which one goes first, and both use the same Wormhole class. In the first variant, one side calls `get_code()` while the other calls `set_code()`. In the second variant, both sides call `set_code()`. (Note that this is not true for the "Transit" protocol used for bulk data-transfer: the Transit class currently distinguishes "Sender" from "Receiver", so the programs on each side must have some way to decide ahead of time which is which). Each side can then do an arbitrary number of `send()` and `get()` calls. `send()` writes a message into the channel. `get()` waits for a new message to be available, then returns it. The Wormhole is not meant as a long-term communication channel, but some protocols work better if they can exchange an initial pair of messages (perhaps offering some set of negotiable capabilities), and then follow up with a second pair (to reveal the results of the negotiation). Note: the application developer must be careful to avoid deadlocks (if both sides want to `get()`, somebody has to `send()` first). When both sides are done, they must call `close()`, to flush all pending `send()` calls, deallocate the channel, and close the websocket connection. ## Twisted The Twisted-friendly flow looks like this (note that passing reactor= is how you get a non-blocking Wormhole): ```python from twisted.internet import reactor from wormhole.public_relay import RENDEZVOUS_RELAY from wormhole import wormhole w1 = wormhole(u"appid", RENDEZVOUS_RELAY, reactor) d = w1.get_code() def _got_code(code): print "Invitation Code:", code return w1.send(b"outbound data") d.addCallback(_got_code) d.addCallback(lambda _: w1.get()) def _got(inbound_message): print "Inbound message:", inbound_message d.addCallback(_got) d.addCallback(w1.close) d.addBoth(lambda _: reactor.stop()) reactor.run() ``` On the other side, you call `set_code()` instead of waiting for `get_code()`: ```python w2 = wormhole(u"appid", RENDEZVOUS_RELAY, reactor) w2.set_code(code) d = w2.send(my_message) ... ``` Note that the Twisted-form `close()` accepts (and returns) an optional argument, so you can use `d.addCallback(w.close)` instead of `d.addCallback(lambda _: w.close())`. ## Verifier For extra protection against guessing attacks, Wormhole can provide a "Verifier". This is a moderate-length series of bytes (a SHA256 hash) that is derived from the supposedly-shared session key. If desired, both sides can display this value, and the humans can manually compare them before allowing the rest of the protocol to proceed. If they do not match, then the two programs are not talking to each other (they may both be talking to a man-in-the-middle attacker), and the protocol should be abandoned. To retrieve the verifier, you call `d=w.verify()` before any calls to `send()/get()`. The Deferred will not fire until internal key-confirmation has taken place (meaning the two sides have exchanged their initial PAKE messages, and the wormhole codes matched), so `verify()` is also a good way to detect typos or mistakes entering the code. The Deferred will errback with wormhole.WrongPasswordError if the codes did not match, or it will callback with the verifier bytes if they did match. Once retrieved, you can turn this into hex or Base64 to print it, or render it as ASCII-art, etc. Once the users are convinced that `verify()` from both sides are the same, call `send()/get()` to continue the protocol. If you call `send()/get()` before `verify()`, it will perform the complete protocol without pausing. ## Generating the Invitation Code In most situations, the "sending" or "initiating" side will call `get_code()` to generate the invitation code. This returns a string in the form `NNN-code-words`. The numeric "NNN" prefix is the "channel id", and is a short integer allocated by talking to the rendezvous server. The rest is a randomly-generated selection from the PGP wordlist, providing a default of 16 bits of entropy. The initiating program should display this code to the user, who should transcribe it to the receiving user, who gives it to the Receiver object by calling `set_code()`. The receiving program can also use `input_code()` to use a readline-based input function: this offers tab completion of allocated channel-ids and known codewords. Alternatively, the human users can agree upon an invitation code themselves, and provide it to both programs later (both sides call `set_code()`). They should choose a channel-id that is unlikely to already be in use (3 or more digits are recommended), append a hyphen, and then include randomly-selected words or characters. Dice, coin flips, shuffled cards, or repeated sampling of a high-resolution stopwatch are all useful techniques. Note that the code is a human-readable string (the python "unicode" type in python2, "str" in python3). ## Application Identifier Applications using this library must provide an "application identifier", a simple string that distinguishes one application from another. To ensure uniqueness, use a domain name. To use multiple apps for a single domain, append a URL-like slash and path, like `example.com/app1`. This string must be the same on both clients, otherwise they will not see each other. The invitation codes are scoped to the app-id. Note that the app-id must be unicode, not bytes, so on python2 use `u"appid"`. Distinct app-ids reduce the size of the connection-id numbers. If fewer than ten Wormholes are active for a given app-id, the connection-id will only need to contain a single digit, even if some other app-id is currently using thousands of concurrent sessions. ## Rendezvous Relays The library depends upon a "rendezvous relay", which is a server (with a public IP address) that delivers small encrypted messages from one client to the other. This must be the same for both clients, and is generally baked-in to the application source code or default config. This library includes the URL of a public relay run by the author. Application developers can use this one, or they can run their own (see the `wormhole-server` command and the `src/wormhole/server/` directory) and configure their clients to use it instead. This URL is passed as a unicode string. ## Bytes, Strings, Unicode, and Python 3 All cryptographically-sensitive parameters are passed as bytes ("str" in python2, "bytes" in python3): * verifier string * data in/out * transit records in/out Other (human-facing) values are always unicode ("unicode" in python2, "str" in python3): * wormhole code * relay URL * transit URLs * transit connection hints (e.g. "host:port") * application identifier * derived-key "purpose" string: `w.derive_key(PURPOSE, LENGTH)` magic-wormhole-0.9.1/docs/transit.md0000644000076600000240000002346512661370246020034 0ustar warnerstaff00000000000000= Transit Protocol = The Transit protocol is responsible for establishing an encrypted bidirectional record stream between two programs. It must be given a "transit key" and a set of "hints" which help locate the other end (which are both delivered by Wormhole). The protocol tries hard to create a **direct** connection between the two ends, but if that fails, it uses a centralized relay server to ferry data between two separate TCP streams (one to each client). The current implementation starts with the following: * detect all of the host's IP addresses * listen on a random TCP port * offers the (address,port) pairs as hints The other side will attempt to connect to each of those ports, as well as listening on its own socket. After a few seconds without success, they will both connect to a relay server. == Roles == The Transit protocol has pre-defined "Sender" and "Receiver" roles (unlike Wormhole, which is symmetric/nobody-goes-first). Each connection must have exactly one Sender and exactly one Receiver. The connection itself is bidirectional: either side can send or receive records. However the connection establishment mechanism needs to know who is in charge, and the encryption layer needs a way to produce separate keys for each side.. This may be relaxed in the future, much as Wormhole was. == Records == Transit establishes a **record-pipe**, so the two sides can send and receive whole records, rather than unframed bytes. This is a side-effect of the encryption (which uses the NaCl "secretbox" function). The encryption adds 44 bytes of overhead to each record (4-byte length, 24-byte nonce, 32-byte MAC), so you might want to use slightly larger records for efficiency. The maximum record size is 2^32 bytes (4GiB). The whole record must be held in memory at the same time, plus its ciphertext, so very large ciphertexts are not recommended. Transit provides **confidentiality**, **integrity**, and **ordering** of records. Passive attackers can only do the following: * learn the size and transmission time of each record * learn the sending and destination IP addresses In addition, an active attacker is able to: * delay delivery of individual records, while maintaining ordering (if they delay record #4, they must delay #5 and later as well) * terminate the connection at any time If either side receives a corrupted or out-of-order record, they drop the connection. Attackers cannot modify the contents of a record, or change the order of the records, without being detected and the connection being dropped. If a record is lost (e.g. the receiver observers records #1,#2,#4, but not #3), the connection is dropped when the unexpected sequence number is received. == Handshake == The transit key is used to derive several secondary keys. Two of them are used as a "handshake", to distinguish correct Transit connections from other programs that happen to connect to the Transit sockets by mistake or malice. The handshake is also responsible for choosing exactly one TCP connection to use, even though multiple outbound and inbound connections are being attempted. The SENDER-HANDSHAKE is the string `transit sender %s ready\n\n`, with the `%s` replaced by a hex-encoded 32-byte HKDF derivative of the transit key, using a "context string" of `transit_sender`. The RECEIVER-HANDSHAKE is the same but with `receiver` instead of `sender` (both for the string and the HKDF context). The handshake protocol is like this: * immediately upon socket connection being made, the Sender writes SENDER-HANDSHAKE to the socket (regardless of whether the Sender initiated the TCP connection, or was listening on a socket and just accepted the connection) * likewise the Receiver immediately writes RECEIVER-HANDSHAKE to either kind of socket * if the Sender sees anything other than RECEIVER-HANDSHAKE as the first bytes on the wire, it hangs up * likewise with the Receiver and SENDER-HANDSHAKE * if the Sender sees that this is the first connection to get RECEIVER-HANDSHAKE, it sends `go\n`. If some other connection got there first, it hangs up (or sends `nevermind\n` and then hangs up, but this is mostly for debugging, and implementations should not depend upon it). After sending `go`, it switches to encrypted-record mode. * if the Receiver sees `go\n`, it switches to encrypted-record mode. If the receiver sees anything else, or a disconnected socket, it disconnects. To tolerate the inevitable race conditions created by multiple contending sockets, only the Sender gets to decide which one wins: the first one to make it past negotiation. Hopefully this is correlated with the fastest connection pathway. The protocol ignores any socket that is not somewhat affiliated with the matching Transit instance. Hints will frequently point to local IP addresses (local to the other end) which might be in use by unrelated nearby computers. The handshake helps to ignore these spurious connections. It is still possible for an attacker to cause the connection to fail, by intercepting both connections (to learn the two handshakes), then making new connections to play back the recorded handshakes, but this level of attacker could simply drop the user's packets directly. == Relay == The **Transit Relay** is a host which offers TURN-like services for magic-wormhole instances. It uses a TCP-based protocol with a handshake to determine which connection wants to be connected to which. When connecting to a relay, the Transit client first writes RELAY-HANDSHAKE to the socket, which is `please relay %s\n`, where `%s` is the hex-encoded 32-byte HKDF derivative of the transit key, using `transit_relay_token` as the context. The client then waits for `ok\n`. The relay waits for a second connection that uses the same token. When this happens, the relay sends `ok\n` to both, then wires the connections together, so that everything received after the token on one is written out (after the ok) on the other. When either connection is lost, the other will be closed (the relay does not support "half-close"). When clients use a relay connection, they perform the usual sender/receiver handshake just after the `ok\n` is received: until that point they pretend the connection doesn't even exist. Direct connections are better, since they are faster and less expensive for the relay operator. If there are any potentially-viable direct connection hints available, the Transit instance will wait a few seconds before attempting to use the relay. If it has no viable direct hints, it will start using the relay right away. This prefers direct connections, but doesn't introduce completely unnecessary stalls. == API == First, create a Transit instance, giving it the connection information of the transit relay. The application must know whether it should use a Sender or a Receiver: ```python from wormhole.blocking.transit import TransitSender s = TransitSender("tcp:relayhost.example.org:12345") ``` Next, ask the Transit for its direct and relay hints. This should be delivered to the other side via a Wormhole message (i.e. add them to a dict, serialize it with JSON, send the result as a message with `wormhole.send()`). ```python direct_hints = s.get_direct_hints() relay_hints = s.get_relay_hints() ``` Then, perform the Wormhole exchange, which ought to give you the direct and relay hints of the other side. Tell your Transit instance about their hints. ```python s.add_their_direct_hints(their_direct_hints) s.add_their_relay_hints(their_relay_hints) ``` Then use `wormhole.derive_key()` to obtain a shared key for Transit purposes, and tell your Transit about it. Both sides must use the same derivation string, and this string must not be used for any other purpose, but beyond that it doesn't much matter what the exact string is. ```python key = w.derive_key(application_id + "/transit-key") s.set_transit_key(key) ``` Finally, tell the Transit instance to connect. This will yield a "record pipe" object, on which records can be sent and received. If no connection can be established within a timeout (defaults to 30 seconds), `connect()` will throw an exception instead. The pipe can be closed with `close()`. ```python rp = s.connect() rp.send_record(b"my first record") their_record = rp.receive_record() rp.send_record(b"Greatest Hits) other = rp.receive_record() rp.close() ``` Records can be sent and received arbitrarily (you are not limited to taking turns). However the blocking API does not provide a way to send records while waiting for an inbound record. This *might* work with threads, but it has not been tested. == Twisted API == The same facilities are available in the asynchronous Twisted environment. The difference is that some functions return Deferreds instead of immediate values. The final record-pipe object is a Protocol (TBD: maybe this is a job for Tubes?), which exposes `receive_record()` as a Deferred-returning function that internally holds a queue of inbound records. ```python from twisted.internet.defer import inlineCallbacks from wormhole.twisted.transit import TransitSender @inlineCallbacks def do_transit(): s = TransitSender(relay) my_relay_hints = s.get_relay_hints() my_direct_hints = yield s.get_direct_hints() # (send hints via wormhole) s.add_their_relay_hints(their_relay_hints) s.add_their_direct_hints(their_direct_hints) s.set_transit_key(key) rp = yield s.connect() rp.send_record(b"eponymous") them = yield rp.receive_record() yield rp.close() ``` This object also implements the `IConsumer`/`IProducer` protocols for **bytes**, which means you can transfer a file by wiring up a file reader as a Producer. Each chunk of bytes that the Producer generates will be put into a single record. The Consumer interface works the same way. This enables backpressure and flow-control: if the far end (or the network) cannot keep up with the stream of data, the sender will wait for them to catch up before filling buffers without bound. magic-wormhole-0.9.1/docs/wormhole-server.80000644000076600000240000000150013027040774021236 0ustar warnerstaff00000000000000.TH WORMHOLE-SERVER "8" "July 2016" .SH NAME wormhole-server \- Securely and simply transfer data between computers .SH SYNOPSIS .B wormhole-server [\fI\,OPTIONS\/\fR] \fI\,COMMAND \/\fR[\fI\,ARGS\/\fR]... .SH DESCRIPTION .IP Control a relay server (most users shouldn't need to worry about this and can use the default server). .SH OPTIONS .TP \fB\-\-help\fR Show this message and exit. .SS "Commands:" .TP count\-channels Count active channels .TP count\-events Count events .TP restart Re\-start a relay server .TP start Start a relay server .TP stop Stop a relay server .TP tail\-usage Follow the latest usage .SH SEE ALSO .BR wormhole (1) .SH AUTHORS Brian Warner .PP This manual was written by Jameson Rollins for the Debian project (and may be used by others). magic-wormhole-0.9.1/docs/wormhole.10000644000076600000240000000210113027040774017721 0ustar warnerstaff00000000000000.TH WORMHOLE "1" "July 2016" "" "User Commands" .SH NAME wormhole \- Securely and simply transfer data between computers .SH SYNOPSIS .B wormhole [\fI\,OPTIONS\/\fR] \fI\,COMMAND \/\fR[\fI\,ARGS\/\fR]... .SH DESCRIPTION .IP Create a Magic Wormhole and communicate through it. .IP Wormholes are created by speaking the same magic CODE in two different places at the same time. Wormholes are secure against anyone who doesn't use the same code. .SH OPTIONS .TP \fB\-\-relay\-url\fR URL rendezvous relay to use .TP \fB\-\-transit\-helper\fR tcp:HOST:PORT transit relay to use .TP \fB\-\-dump\-timing\fR FILE.json (debug) write timing data to file .TP \fB\-\-version\fR Show the version and exit. .TP \fB\-\-help\fR Show this message and exit. .SS "Commands:" .TP receive Receive a text message, file, or directory... .TP send Send a text message, file, or directory .SH SEE ALSO .BR wormhole-server (8) .SH AUTHORS Brian Warner .PP This manual was written by Jameson Rollins for the Debian project (and may be used by others). magic-wormhole-0.9.1/LICENSE0000644000076600000240000000207012466753575016105 0ustar warnerstaff00000000000000The MIT License (MIT) Copyright (c) 2015 Brian Warner Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. magic-wormhole-0.9.1/MANIFEST.in0000644000076600000240000000105713031527043016614 0ustar warnerstaff00000000000000include versioneer.py include src/wormhole/_version.py include LICENSE README.md NEWS.md recursive-include docs *.md *.rst include docs/wormhole.1 docs/wormhole-server.8 include .coveragerc tox.ini include misc/windows-build.cmd include misc/*.py misc/web/*.html misc/web/*.js misc/web/*.css include misc/munin/wormhole_active include misc/munin/wormhole_errors include misc/munin/wormhole_event_rate include misc/munin/wormhole_events include misc/munin/wormhole_events_alltime include misc/munin/wormhole_transit include misc/munin/wormhole_transit_alltime magic-wormhole-0.9.1/misc/0000755000076600000240000000000013032243761016011 5ustar warnerstaff00000000000000magic-wormhole-0.9.1/misc/dump-stats.py0000755000076600000240000000064513024532264020474 0ustar warnerstaff00000000000000from __future__ import print_function import time, json # Run this as 'watch python misc/dump-stats.py' against a 'wormhole-server # start --stats-file=stats.json' with open("stats.json") as f: data_s = f.read() now = time.time() data = json.loads(data_s) if now < data["valid_until"]: valid = "valid" else: valid = "EXPIRED" age = now - data["created"] print("age: %d (%s)" % (age, valid)) print(data_s) magic-wormhole-0.9.1/misc/dump-timing.py0000644000076600000240000000374012714205710020617 0ustar warnerstaff00000000000000# To use the web() option, you should do: # * cd misc # * npm install d3 zepto from __future__ import print_function import os, sys, time, json, random streams = sys.argv[1:] if len(streams) != 2: print("run like: python dump-timing.py tx.json rx.json") sys.exit(1) # for now, require sender as first file, receiver as second # later, allow use of only one file. data = {} for i,fn in enumerate(streams): name = ["send", "receive"][i] with open(fn, "rb") as f: events = json.load(f) data[name] = {"fn": os.path.basename(fn), "events": events} from pprint import pprint pprint(data) here = os.path.dirname(__file__) web_root = os.path.join(here, "web") lib_root = os.path.join(here, "node_modules") if not os.path.isdir(lib_root): print("Cannot find 'd3' and 'd3-tip' in misc/node_modules/") print("Please run 'npm install d3 d3-tip zepto' from the misc/ directory.") sys.exit(1) def web(): # set up a server that serves web/ at the root, plus a /data.json built # from {timeline}. Quit when it fetches /done . from twisted.web import resource, static, server from twisted.internet import reactor, endpoints ep = endpoints.serverFromString(reactor, "tcp:8066:interface=127.0.0.1") root = static.File(web_root) root.putChild("data.json", static.Data(json.dumps(data).encode("utf-8"), "application/json")) root.putChild("lib", static.File(lib_root)) class Shutdown(resource.Resource): def render_GET(self, request): if 0: print("timeline ready, server shutting down") reactor.stop() return "shutting down" root.putChild("done", Shutdown()) site = server.Site(root) ep.listen(site) import webbrowser def launch_browser(): webbrowser.open("http://localhost:%d/timeline.html" % 8066) print("browser opened, waiting for shutdown") reactor.callLater(0, launch_browser) reactor.run() web() magic-wormhole-0.9.1/misc/munin/0000755000076600000240000000000013032243761017137 5ustar warnerstaff00000000000000magic-wormhole-0.9.1/misc/munin/wormhole_active0000755000076600000240000000235413024532264022260 0ustar warnerstaff00000000000000#! /usr/bin/env python """ Use the following in /etc/munin/plugin-conf.d/wormhole : [wormhole_*] env.serverdir /path/to/your/wormhole/server """ import os, sys, time, json CONFIG = """\ graph_title Magic-Wormhole Active Channels graph_vlabel Channels graph_category network nameplates.label Nameplates nameplates.draw LINE2 nameplates.type GAUGE mailboxes.label Mailboxes mailboxes.draw LINE2 mailboxes.type GAUGE messages.label Messages messages.draw LINE1 messages.type GAUGE transit_waiting.label Transit Waiting transit_waiting.draw LINE1 transit_waiting.type GAUGE transit_connected.label Transit Connected transit_connected.draw LINE1 transit_connected.type GAUGE """ if len(sys.argv) > 1 and sys.argv[1] == "config": print CONFIG.rstrip() sys.exit(0) serverdir = os.environ["serverdir"] fn = os.path.join(serverdir, "stats.json") with open(fn) as f: data = json.load(f) if time.time() > data["valid_until"]: sys.exit(1) # expired ra = data["rendezvous"]["active"] print "nameplates.value", ra["nameplates_total"] print "mailboxes.value", ra["mailboxes_total"] print "messages.value", ra["messages_total"] ta = data["transit"]["active"] print "transit_waiting.value", ta["waiting"] print "transit_connected.value", ta["connected"] magic-wormhole-0.9.1/misc/munin/wormhole_errors0000755000076600000240000000245213024532264022320 0ustar warnerstaff00000000000000#! /usr/bin/env python """ Use the following in /etc/munin/plugin-conf.d/wormhole : [wormhole_*] env.serverdir /path/to/your/wormhole/server """ import os, sys, time, json CONFIG = """\ graph_title Magic-Wormhole Server Errors graph_vlabel Events Since Reboot graph_category network nameplates.label Nameplate Errors (total) nameplates.draw LINE1 nameplates.type GAUGE mailboxes.label Mailboxes (total) mailboxes.draw LINE1 mailboxes.type GAUGE mailboxes_scary.label Mailboxes (scary) mailboxes_scary.draw LINE1 mailboxes_scary.type GAUGE transit.label Transit transit.draw LINE1 transit.type GAUGE """ if len(sys.argv) > 1 and sys.argv[1] == "config": print CONFIG.rstrip() sys.exit(0) serverdir = os.environ["serverdir"] fn = os.path.join(serverdir, "stats.json") with open(fn) as f: data = json.load(f) if time.time() > data["valid_until"]: sys.exit(1) # expired r = data["rendezvous"]["since_reboot"] print "nameplates.value", (r["nameplates_total"] - r["nameplate_moods"].get("happy", 0)) print "mailboxes.value", (r["mailboxes_total"] - r["mailbox_moods"].get("happy", 0)) print "mailboxes_scary.value", r["mailbox_moods"].get("scary", 0) t = data["transit"]["since_reboot"] print "transit.value", (t["total"] - t["moods"].get("happy", 0)) magic-wormhole-0.9.1/misc/munin/wormhole_event_rate0000755000076600000240000000213313024532264023134 0ustar warnerstaff00000000000000#! /usr/bin/env python """ Use the following in /etc/munin/plugin-conf.d/wormhole : [wormhole_*] env.serverdir /path/to/your/wormhole/server """ import os, sys, time, json CONFIG = """\ graph_title Magic-Wormhole Server Events graph_vlabel Events per Hour graph_category network happy.label Happy happy.draw LINE happy.type DERIVE happy.min 0 happy.max 60 happy.cdef happy,3600,* incomplete.label Incomplete incomplete.draw LINE incomplete.type DERIVE incomplete.min 0 incomplete.max 60 incomplete.cdef happy,3600,* scary.label Scary scary.draw LINE scary.type DERIVE scary.min 0 scary.max 60 scary.cdef happy,3600,* """ if len(sys.argv) > 1 and sys.argv[1] == "config": print CONFIG.rstrip() sys.exit(0) serverdir = os.environ["serverdir"] fn = os.path.join(serverdir, "stats.json") with open(fn) as f: data = json.load(f) if time.time() > data["valid_until"]: sys.exit(1) # expired atm = data["rendezvous"]["all_time"]["mailbox_moods"] print "happy.value", atm.get("happy", 0) print "incomplete.value", (atm.get("pruney", 0) + atm.get("lonely", 0)) print "scary.value", atm.get("scary", 0) magic-wormhole-0.9.1/misc/munin/wormhole_events0000755000076600000240000000240013024532264022301 0ustar warnerstaff00000000000000#! /usr/bin/env python """ Use the following in /etc/munin/plugin-conf.d/wormhole : [wormhole_*] env.serverdir /path/to/your/wormhole/server """ import os, sys, time, json CONFIG = """\ graph_title Magic-Wormhole Mailbox Events (since reboot) graph_vlabel Events Since Reboot graph_category network happy.label Happy happy.draw LINE2 happy.type GAUGE total.label Total total.draw LINE1 total.type GAUGE scary.label Scary scary.draw LINE2 scary.type GAUGE pruney.label Pruney pruney.draw LINE1 pruney.type GAUGE lonely.label Lonely lonely.draw LINE2 lonely.type GAUGE errory.label Errory errory.draw LINE1 errory.type GAUGE """ if len(sys.argv) > 1 and sys.argv[1] == "config": print CONFIG.rstrip() sys.exit(0) serverdir = os.environ["serverdir"] fn = os.path.join(serverdir, "stats.json") with open(fn) as f: data = json.load(f) if time.time() > data["valid_until"]: sys.exit(1) # expired r = data["rendezvous"]["since_reboot"] print "happy.value", r["mailbox_moods"].get("happy", 0) print "total.value", r["mailboxes_total"] print "scary.value", r["mailbox_moods"].get("scary", 0) print "pruney.value", r["mailbox_moods"].get("pruney", 0) print "lonely.value", r["mailbox_moods"].get("lonely", 0) print "errory.value", r["mailbox_moods"].get("errory", 0) magic-wormhole-0.9.1/misc/munin/wormhole_events_alltime0000644000076600000240000000237513024532264024020 0ustar warnerstaff00000000000000#! /usr/bin/env python """ Use the following in /etc/munin/plugin-conf.d/wormhole : [wormhole_*] env.serverdir /path/to/your/wormhole/server """ import os, sys, time, json CONFIG = """\ graph_title Magic-Wormhole Mailbox Events (all time) graph_vlabel Events Since DB Creation graph_category network happy.label Happy happy.draw LINE2 happy.type GAUGE total.label Total total.draw LINE1 total.type GAUGE scary.label Scary scary.draw LINE2 scary.type GAUGE pruney.label Pruney pruney.draw LINE1 pruney.type GAUGE lonely.label Lonely lonely.draw LINE2 lonely.type GAUGE errory.label Errory errory.draw LINE1 errory.type GAUGE """ if len(sys.argv) > 1 and sys.argv[1] == "config": print CONFIG.rstrip() sys.exit(0) serverdir = os.environ["serverdir"] fn = os.path.join(serverdir, "stats.json") with open(fn) as f: data = json.load(f) if time.time() > data["valid_until"]: sys.exit(1) # expired r = data["rendezvous"]["all_time"] print "happy.value", r["mailbox_moods"].get("happy", 0) print "total.value", r["mailboxes_total"] print "scary.value", r["mailbox_moods"].get("scary", 0) print "pruney.value", r["mailbox_moods"].get("pruney", 0) print "lonely.value", r["mailbox_moods"].get("lonely", 0) print "errory.value", r["mailbox_moods"].get("errory", 0) magic-wormhole-0.9.1/misc/munin/wormhole_transit0000755000076600000240000000131313024532264022463 0ustar warnerstaff00000000000000#! /usr/bin/env python """ Use the following in /etc/munin/plugin-conf.d/wormhole : [wormhole_*] env.serverdir /path/to/your/wormhole/server """ import os, sys, time, json CONFIG = """\ graph_title Magic-Wormhole Transit Usage (since reboot) graph_vlabel Bytes Since Reboot graph_category network bytes.label Transit Bytes bytes.draw LINE1 bytes.type GAUGE """ if len(sys.argv) > 1 and sys.argv[1] == "config": print CONFIG.rstrip() sys.exit(0) serverdir = os.environ["serverdir"] fn = os.path.join(serverdir, "stats.json") with open(fn) as f: data = json.load(f) if time.time() > data["valid_until"]: sys.exit(1) # expired t = data["transit"]["since_reboot"] print "bytes.value", t["bytes"] magic-wormhole-0.9.1/misc/munin/wormhole_transit_alltime0000644000076600000240000000131013024532264024164 0ustar warnerstaff00000000000000#! /usr/bin/env python """ Use the following in /etc/munin/plugin-conf.d/wormhole : [wormhole_*] env.serverdir /path/to/your/wormhole/server """ import os, sys, time, json CONFIG = """\ graph_title Magic-Wormhole Transit Usage (all time) graph_vlabel Bytes Since DB Creation graph_category network bytes.label Transit Bytes bytes.draw LINE1 bytes.type GAUGE """ if len(sys.argv) > 1 and sys.argv[1] == "config": print CONFIG.rstrip() sys.exit(0) serverdir = os.environ["serverdir"] fn = os.path.join(serverdir, "stats.json") with open(fn) as f: data = json.load(f) if time.time() > data["valid_until"]: sys.exit(1) # expired t = data["transit"]["all_time"] print "bytes.value", t["bytes"] magic-wormhole-0.9.1/misc/web/0000755000076600000240000000000013032243761016566 5ustar warnerstaff00000000000000magic-wormhole-0.9.1/misc/web/timeline.css0000644000076600000240000000231513024532264021107 0ustar warnerstaff00000000000000 line.client_tx { stroke: red; stroke-dasharray: 5,5; } line.client_rx { stroke: blue; stroke-dasharray: 5,5; } line.c2c_column { stroke: black; stroke-dasharray: 1,5; } line.c2c { stroke-width: 2.0; } line.c2c.active { stroke-width: 4.0; } line.y_axis { stroke: gray; } /* putting these in a .css file doesn't work, for some reason I have to add the markers as a style= attribute directly. */ line.circle-arrow-circle { /*marker-start: url(#markerCircle);*/ marker-mid: url(#markerArrow); /*marker-end: url(#markerCircle);*/ } rect.wait-crypto { stroke: black; fill: #ccc; } rect.wait-user { stroke: #00f; fill: #bbe; } text.wait-text-user { fill: #00f; } rect.proc-span-import { fill: #fcc; } rect.proc-span-websocket { fill: #cfc; } rect.api { fill: #cfc; } rect.bar { stroke: black; } .lane-0 { fill: #fcc; } .lane-1 { fill: #cfc; } .lane-2 { fill: #ccf; } .lane-3 { fill: #ccf; } .lane-4 { fill: #ccf; } .lane-5 { fill: #ccf; } .lane-6 { fill: #ccf; } .vis-item .vis-item-overflow { overflow: visible; } .d3-tip { margin: 4px; padding: 2px; background: #111; color: #fff; } magic-wormhole-0.9.1/misc/web/timeline.html0000644000076600000240000000073112711707617021273 0ustar warnerstaff00000000000000 Timeline Visualizer

Wormhole Timeline


magic-wormhole-0.9.1/misc/web/timeline.js0000644000076600000240000011364413024532264020743 0ustar warnerstaff00000000000000var d3; // hush var container = d3.select("#viz"); var data; var items; var globals = {}; var server_time_offset=0, rx_time_offset=0; // in seconds, relative to tx var zoom = d3.behavior.zoom().scaleExtent([1, Infinity]); function zoomin() { //var w = Number(container.style("width").slice(0,-2)); //console.log("zoomin", w); //zoom.center([w/2, 20]); // doesn't work yet zoom.scale(zoom.scale() * 2); globals.redraw(); } function zoomout() { zoom.scale(zoom.scale() * 0.5); globals.redraw(); } function is_span(ev, category) { if (ev.category === category && !!ev.stop) return true; return false; } function is_event(ev, category) { if (ev.category === category && !ev.stop) return true; return false; } const server_message_color = { "welcome": 0, // receive "bind": 0, // send "allocate": 1, // send "allocated": 1, // receive "list": 2, // send "channelids": 2, // receive "claim": 3, // send "watch": 4, // send "deallocate": 5, // send "deallocated": 5, // receive "error": 6, // receive //"add": 8, // send (client message) //"message": 8, // receive (client message) "ping": 7, // send "pong": 7 // receive }; const proc_map = { "command dispatch": "dispatch", "open websocket": "websocket", "code established": "code-established", "key established": "key-established", "transit connected": "transit-connected", "exit": "exit", "transit connect": "transit-connect", "import": "import" }; const TX_COLUMN = 14; const RX_COLUMN = 18; const SERVER_COLUMN0 = 20; const SERVER_COLUMNS = [20,21,22,23,24,25]; const NUM_SERVER_COLUMNS = 6; const MAX_COLUMN = 45; function x_offset(offset, side_name) { if (side_name === "send") return offset; return MAX_COLUMN - offset; } function side_text_anchor(side_name) { if (side_name === "send") return "end"; return "start"; } function side_text_dx(side_name) { if (side_name === "send") return "-5px"; return "5px"; } d3.json("data.json", function(d) { data = d; // data is {send,receive}{fn,events} // each event has: {name, start, [stop], [server_rx], [server_tx], // [id], details={} } // Display all timestamps relative to the sender's startup event. If all // clocks are in sync, this will be the same as first_timestamp, but in // case they aren't, I'd rather use the sender as the reference point. var first = data.send.events[0].start; // The X axis is divided up into 50 slots, and then scaled to the screen // later. The left portion represents the "wormhole send" side, the // middle is the rendezvous server, and the right portion is the // "wormhole receive" side. // // 0: time axis, tick marks // 3: sender process events: import, dispatch, exit // 4: sender major application-level events: code/key establishment, // transit-connect // 8: sender stalls: waiting for user, waiting for permission // 10: sender websocket transmits originate from here // 15: sender websocket receives terminate here // 20-25: rendezvous-server message lanes // 30: receiver websocket receives // 35: receiver websocket transmits // 37: receiver stalls // 41: receiver app-level events // 42: receiver process events var first_timestamp = Infinity; var last_timestamp = 0; function prepare_data(e, side_name) { var rel_e = {side_name: side_name, // send or receive name: e.name, start: e.start - first, details: e.details }; if (e.stop) rel_e.stop = e.stop - first; if (side_name == "receive") { rel_e.start -= rx_time_offset; if (e.stop) rel_e.stop -= rx_time_offset; } if (rel_e.details.message) { if (rel_e.details.message.server_rx) rel_e.details.message.server_rx -= server_time_offset; if (rel_e.details.message.server_tx) rel_e.details.message.server_tx -= server_time_offset; } // sort events into categories, assign X coordinates to some if (proc_map[e.name]) { rel_e.category = "proc"; rel_e.x = x_offset(3, side_name); if (e.name === "open websocket") rel_e.x = x_offset(4, side_name); rel_e.text = proc_map[e.name]; if (e.name === "import") rel_e.text += " " + e.details.which; } if (e.details.waiting) { rel_e.category = "wait"; var off = 8; if (e.details.waiting === "user") off += 0.5; rel_e.x = x_offset(off, side_name); } // also, calculate the overall time domain while we're at it [rel_e.start, rel_e.stop].forEach(v => { if (v) { if (v > last_timestamp) last_timestamp = v; if (v < first_timestamp) first_timestamp = v; } }); return rel_e; } var events = data.send.events.map(e => prepare_data(e, "send")); events = events.concat(data.receive.events.map(e => prepare_data(e, "receive"))); /* "Client messages" are ones that go all the way from one client to the other, through the rendezvous channel (and get echoed back to the sender too). We can correlate three websocket messages for each (the send, the local receive, and the remote receive) by comparing their "id" strings. Scan for all client messages, to build a list of central columns. For each message, we'll have tx/server_rx/server_tx/rx for the sending side, and server_rx/server_tx/rx for the receiving side. The "add" event contributes tx, the sender's echo contributes and the "message" event contributes server_rx, server_tx, and rx. */ var side_map = new Map(); // side -> "send"/"receive" var c2c = new Map(); // msgid => {send,receive}{tx,server_rx,server_tx,rx} events.forEach(ev => { var id, phase; if (ev.name === "ws_send") { if (ev.details.type !== "add") return; id = ev.details.id; phase = ev.details.phase; side_map.set(ev.details._side, ev.side_name); } else if (ev.name === "ws_receive") { if (ev.details.message.type !== "message") return; id = ev.details.message.id; phase = ev.details.message.phase; } else return; if (!c2c.has(id)) { c2c.set(id, {phase: phase, side_id: ev.details._side, //tx_side_name: assigned when we see 'add' id: id, arrivals: [] //col, server_x: assigned later //server_rx: assigned when we see 'message' }); } var cm = c2c.get(id); if (ev.name === "ws_send") { // add cm.tx = ev.start; cm.tx_x = x_offset(TX_COLUMN, ev.side_name); cm.tx_side_name = ev.side_name; } else { // message cm.server_rx = ev.details.message.server_rx - first; cm.arrivals.push({server_tx: ev.details.message.server_tx - first, rx: ev.start, rx_x: x_offset(RX_COLUMN, ev.side_name)}); } }); // sort c2c messages by initial sending time var client_messages = Array.from(c2c.values()); client_messages.sort( (a,b) => (a.tx - b.tx) ); // assign columns // TODO: identify overlaps between the c2c messages, share columns // between messages which don't overlap client_messages.forEach((cm,index) => { cm.col = index % 6; cm.server_x = 20 + cm.col; }); console.log("client_messages", client_messages); console.log(side_map); console.log(first_timestamp, last_timestamp); /* "Server messages" are ones that stop or originate at the rendezvous server. These are of types other than "add" or "message". Although many of these provoke responses, we do not attempt to correlate these with any other message. For outbound ws_send messages, we know the send timestamp, but not the server receipt timestamp. For inbound ws_receive messages, we know both. */ var outbound_sm = new Map(); globals.outbound_sm = outbound_sm; events .filter(ev => ev.name === "ws_send") .forEach(ev => { // we don't know the server receipt time, so draw a horizontal // line by setting stop_timestamp=start_timestamp var sm = {side_name: ev.side_name, start_timestamp: ev.start, stop_timestamp: ev.start, start_x: x_offset(TX_COLUMN, ev.side_name), end_x: x_offset(20, ev.side_name), text_x: x_offset(TX_COLUMN, ev.side_name), text_timestamp: ev.start, text_dy: "-5px", type: ev.details.type, tip: ev.details.type, ev: ev }; outbound_sm.set(ev.details.id, sm); }); events .filter(ev => ev.name === "ws_receive") .filter(ev => ev.details.message.type === "ack") .forEach(ev => { var id = ev.details.message.id; var server_tx = ev.details.message.server_tx; var sm = outbound_sm.get(id); sm.stop_timestamp = server_tx - first; }); var server_messages = []; events .filter(ev => ev.name === "ws_receive") .filter(ev => ev.details.message.type !== "message") .filter(ev => ev.details.message.type !== "ack") .forEach(ev => { var sm = {side_name: ev.side_name, start_timestamp: ev.details.message.server_tx - first, stop_timestamp: ev.start, start_x: x_offset(20, ev.side_name), end_x: x_offset(RX_COLUMN, ev.side_name), text_x: x_offset(RX_COLUMN, ev.side_name), text_timestamp: ev.start, text_dy: "8px", type: ev.details.message.type, tip: ev.details.message.type, ev: ev }; server_messages.push(sm); }); server_messages = server_messages.concat( Array.from(outbound_sm.values()) .filter(sm => sm.type !== "add")); console.log("server_messages", server_messages); // TODO: this goes off the edge of the screen, use the viewport instead var container_width = Number(container.style("width").slice(0,-2)); var container_height = Number(container.style("height").slice(0,-2)); container_height = 700; // no contents, so no height is allocated yet // scale the X axis to the full width of our container var x = d3.scale.linear().domain([0, 50]).range([0, container_width]); // scale the Y axis later var y = d3.scale.linear().domain([first_timestamp, last_timestamp]) .range([0, container_height]); zoom.y(y); zoom.on("zoom", redraw); var tip = d3.tip() .attr("class", "d3-tip") .html(function(d) { return "" + d + ""; }) .direction("s") ; var chart = container.append("svg:svg") .attr("id", "outer_chart") .attr("width", container_width) .attr("height", container_height) .attr("pointer-events", "all") .call(zoom) .call(tip) ; var defs = chart.append("svg:defs"); defs.append("svg:marker") .attr("id", "markerCircle") .attr("markerWidth", 8) .attr("markerHeight", 8) .attr("refX", 5) .attr("refY", 5) .append("circle") .attr("cx", 5) .attr("cy", 5) .attr("r", 3) .attr("style", "stroke: none; fill: #000000;") ; defs.append("svg:marker") .attr("id", "markerArrow") .attr("markerWidth", 26) .attr("markerHeight", 26) .attr("refX", 26) .attr("refY", 12) .attr("orient", "auto") .attr("markerUnits", "userSpaceOnUse") // don't scale to stroke-width .append("path") .attr("d", "M8,20 L20,12 L8,4") .attr("style", "stroke: #000000; fill: none") ; chart.append("svg:line") .attr("x1", x(0.5)).attr("y1", 0) .attr("x2", x(0.5)).attr("y2", container_height) .attr("class", "y_axis") ; chart.append("svg:g") .attr("class", "seconds_g") .attr("transform", "translate("+(x(0.5)+5)+","+(container_height-10)+")") .append("svg:text") .text("seconds") ; chart.append("svg:line") .attr("x1", x(TX_COLUMN)).attr("y1", y(first_timestamp)) .attr("x2", x(TX_COLUMN)).attr("y2", y(last_timestamp)) .attr("class", "client_tx") ; chart.append("svg:text") .attr("x", x(TX_COLUMN)).attr("y", 10) .attr("text-anchor", "middle") .text("sender tx"); chart.append("svg:line") .attr("x1", x(RX_COLUMN)).attr("y1", y(first_timestamp)) .attr("x2", x(RX_COLUMN)).attr("y2", y(last_timestamp)) .attr("class", "client_rx") ; chart.append("svg:text") .attr("x", x(RX_COLUMN)).attr("y", 10) .attr("text-anchor", "middle") .text("sender rx"); chart.selectAll("line.c2c_column").data(SERVER_COLUMNS) .enter().append("svg:line") .attr("class", "c2c_column") .attr("x1", d => x(d)).attr("y1", y(first_timestamp)) .attr("x2", d => x(d)).attr("y2", y(last_timestamp)) ; chart.append("svg:line") .attr("x1", x(MAX_COLUMN-RX_COLUMN)).attr("y1", y(first_timestamp)) .attr("x2", x(MAX_COLUMN-RX_COLUMN)).attr("y2", y(last_timestamp)) .attr("class", "client_rx") ; chart.append("svg:text") .attr("x", x(MAX_COLUMN-RX_COLUMN)).attr("y", 10) .attr("text-anchor", "middle") .text("receiver rx"); chart.append("svg:line") .attr("x1", x(MAX_COLUMN-TX_COLUMN)).attr("y1", y(first_timestamp)) .attr("x2", x(MAX_COLUMN-TX_COLUMN)).attr("y2", y(last_timestamp)) .attr("class", "client_tx") ; chart.append("svg:text") .attr("x", x(MAX_COLUMN-TX_COLUMN)).attr("y", 10) .attr("text-anchor", "middle") .text("receiver tx"); // produces list of {p_from, p_to, col, add_arrow, tip} function cm_line(cm) { // We draw a bunch of two-point lines var lines = []; function push(p_from, p_to, add_arrow) { lines.push({p_from: p_from, p_to: p_to, col: cm.col, tip: cm.tip, add_arrow: add_arrow}); } // the first goes from the sender to the server_rx, if we know it // TODO: tolerate not knowing it var sender_point = [cm.tx_x, cm.tx]; var server_rx_point = [cm.server_x, cm.server_rx]; push(sender_point, server_rx_point, true); // the second goes from the server_rx to the last server_tx var last_server_tx = Math.max.apply(null, cm.arrivals.map(a => a.server_tx)); var last_server_tx_point = [cm.server_x, last_server_tx]; push(server_rx_point, last_server_tx_point, false); cm.arrivals.forEach(ar => { var delivery_tx_point = [cm.server_x, ar.server_tx]; var delivery_rx_point = [ar.rx_x, ar.rx]; push(delivery_tx_point, delivery_rx_point, true); }); return lines; } var all_cm_lines = []; client_messages.forEach(v => { all_cm_lines = all_cm_lines.concat(cm_line(v)); }); console.log(all_cm_lines); var cm_colors = d3.scale.category10(); chart.selectAll("line.c2c").data(all_cm_lines) .enter() .append("svg:line") .attr("class", "c2c") // circle-arrow-circle") .attr("stroke", ls => cm_colors(ls.col)) .attr("style", ls => { if (ls.add_arrow) return "marker-end: url(#markerArrow);"; return ""; }) .on("mouseover", ls => { if (ls.tip) tip.show(ls.tip); chart.selectAll("circle.c2c").filter(d => d.col == ls.col) .attr("r", 10); chart.selectAll("line.c2c") .classed("active", d => d.col == ls.col); }) .on("mouseout", ls => { tip.hide(ls); chart.selectAll("circle.c2c") .attr("r", 5); chart.selectAll("line.c2c") .classed("active", false); }) ; chart.selectAll("g.c2c").data(client_messages) .enter() .append("svg:g") .attr("class", "c2c") .append("svg:text") .attr("class", "c2c") .attr("text-anchor", cm => side_text_anchor(cm.tx_side_name)) .attr("dx", cm => side_text_dx(cm.tx_side_name)) .attr("dy", "10px") .attr("fill", cm => cm_colors(cm.col)) .text(cm => cm.phase); function cm_dot(cm) { var dots = []; var color = cm_colors(cm.col); var tip = cm.phase; function push(x,y) { dots.push({x: x, y: y, col: cm.col, color: color, tip: tip}); } push(cm.tx_x, cm.tx); cm.arrivals.forEach(ar => push(ar.rx_x, ar.rx)); return dots; } var all_cm_dots = []; client_messages.forEach(cm => { all_cm_dots = all_cm_dots.concat(cm_dot(cm)); }); chart.selectAll("circle.c2c").data(all_cm_dots) .enter() .append("svg:circle") .attr("class", "c2c") .attr("r", 5) .attr("fill", dot => dot.color) .on("mouseover", dot => { if (dot.tip) tip.show(dot.tip); chart.selectAll("circle.c2c").filter(d => d.col == dot.col) .attr("r", 10); chart.selectAll("line.c2c") .classed("active", d => d.col == dot.col); }) .on("mouseout", dot => { tip.hide(dot); chart.selectAll("circle.c2c") .attr("r", 5); chart.selectAll("line.c2c") .classed("active", false); }) ; // server messages chart.selectAll("line.server-message").data(server_messages) .enter() .append("svg:line") .attr("class", "server-message") .attr("stroke", sm => cm_colors(server_message_color[sm.type] || 0)) .attr("style", "marker-end: url(#markerArrow)") .on("mouseover", sm => { if (sm.tip) tip.show(sm.tip); }) .on("mouseout", sm => { tip.hide(sm); }) ; chart.selectAll("g.server-message").data(server_messages) .enter() .append("svg:g") .attr("class", "server-message") .append("svg:text") .attr("class", "server-message") .attr("text-anchor", sm => side_text_anchor(sm.side_name)) .attr("dx", sm => side_text_dx(sm.side_name)) .attr("dy", sm => sm.text_dy) .attr("fill", sm => cm_colors(server_message_color[sm.type] || 0)) .text(sm => sm.type); // TODO: add dots on the known send/receive time points var w = chart.selectAll("g.wait") .data(events.filter(ev => ev.category === "wait")) .enter().append("svg:g") .attr("class", "wait"); w.append("svg:rect") .attr("class", ev => "wait wait-"+ev.details.waiting) .attr("width", 10); var wt = chart.selectAll("g.wait-text") .data(events.filter(ev => ev.category === "wait")) .enter().append("svg:g") .attr("class", "wait-text"); wt.append("svg:text") .attr("class", ev => "wait-text wait-text-"+ev.details.waiting) .attr("text-anchor", ev => ev.side_name === "send" ? "end" : "start") .attr("dx", ev => ev.side_name === "send" ? "-5px" : "15px") .attr("dy", "5px") .text(v => v.name+" ("+v.details.waiting+")"); // process-related events var pe = chart.selectAll("g.proc-event") .data(events.filter(ev => is_event(ev, "proc"))) .enter().append("svg:g") .attr("class", "proc-event"); pe.append("svg:circle") .attr("class", ev => "proc-event proc-event-"+proc_map[ev.name]) .attr("cx", ev => ev.side_name === "send" ? "12px" : "-2px") .attr("r", 5) .attr("fill", "red") .attr("width", 10); pe.append("svg:text") .attr("class", ev => "proc-event proc-event-"+proc_map[ev.name]) .attr("text-anchor", ev => ev.side_name === "send" ? "start" : "end") .attr("dx", ev => ev.side_name === "send" ? "15px" : "-5px") .attr("dy", "5px") .attr("transform", "rotate(-30)") .text(ev => proc_map[ev.name]); // process-related spans var ps = chart.selectAll("g.proc-span") .data(events.filter(ev => is_span(ev, "proc"))) .enter().append("svg:g") .attr("class", "proc-span"); ps.append("svg:rect") .attr("class", ev => "proc-span proc-span-"+proc_map[ev.name]) .attr("width", 10); var pst = chart.selectAll("g.proc-span-text") .data(events.filter(ev => is_span(ev, "proc"))) .enter().append("svg:g") .attr("class", "proc-span-text"); pst.append("svg:text") .attr("class", ev => "proc-span-text proc-span-text-"+proc_map[ev.name]) .attr("text-anchor", ev => ev.side_name === "send" ? "start" : "end") .attr("dx", ev => ev.side_name === "send" ? "15px" : "-5px") .attr("dy", "5px") .text(ev => ev.text); function ty(d) { return "translate(0,"+y(d)+")"; } function redraw() { chart.selectAll("line.c2c") .attr("x1", ls => x(ls.p_from[0])) .attr("y1", ls => y(ls.p_from[1])) .attr("x2", ls => x(ls.p_to[0])) .attr("y2", ls => y(ls.p_to[1])) ; chart.selectAll("g.c2c") .attr("transform", cm => "translate("+x(cm.tx_x)+","+y(cm.tx)+")") ; chart.selectAll("circle.c2c") .attr("cx", d => x(d.x)) .attr("cy", d => y(d.y)) ; chart.selectAll("line.server-message") .attr("x1", sm => x(sm.start_x)) .attr("y1", sm => y(sm.start_timestamp)) .attr("x2", sm => x(sm.end_x)) .attr("y2", sm => y(sm.stop_timestamp)); chart.selectAll("g.server-message") .attr("transform", sm => { return "translate("+x(sm.text_x)+","+y(sm.text_timestamp)+")"; }) ; chart.selectAll("g.wait") .attr("transform", ev => { return "translate("+x(ev.x)+","+y(ev.start)+")"; }); chart.selectAll("rect.wait") .attr("height", ev => y(ev.stop)-y(ev.start)); chart.selectAll("g.wait-text") .attr("transform", ev => { return "translate("+x(ev.x)+","+y((ev.start+ev.stop)/2)+")"; }); chart.selectAll("g.proc-event") .attr("transform", ev => { return "translate("+x(ev.x)+","+y(ev.start)+")"; }) ; chart.selectAll("g.proc-span") .attr("transform", ev => { return "translate("+x(ev.x)+","+y(ev.start)+")"; }) ; chart.selectAll("rect.proc-span") .attr("height", ev => y(ev.stop)-y(ev.start)); chart.selectAll("g.proc-span-text") .attr("transform", ev => { return "translate("+x(ev.x)+","+y((ev.start+ev.stop)/2)+")"; }); // vertical scale markers: horizontal tick lines at rational // timestamps // TODO: clicking on a dot should set the new zero time var rules = chart.selectAll("g.rule") .data(y.ticks(10)) .attr("transform", ty); rules.select("text") .text(t => y.tickFormat(10, "s")(t)+"s"); var newrules = rules.enter().insert("svg:g") .attr("class", "rule") .attr("transform", ty) ; newrules.append("svg:line") .attr("class", "rule-tick") .attr("stroke", "black"); chart.selectAll("line.rule-tick") .attr("x1", x(0.5)-5) .attr("x2", x(0.5)); newrules.append("svg:line") .attr("class", "rule-red") .attr("stroke", "red") .attr("stroke-opacity", .3); chart.selectAll("line.rule-red") .attr("x1", x(0.5)) .attr("x2", x(MAX_COLUMN)); newrules.append("svg:text") .attr("class", "rule-text") .attr("dx", ".1em") .attr("dy", "-0.2em") .attr("text-anchor", "start") .attr("fill", "black") .text(t => y.tickFormat(10, "s")(t)+"s"); chart.selectAll("text.rule-text") .attr("x", 6 + 9); rules.exit().remove(); } redraw(); }); /* TODO * identify the largest gaps in the timeline (biggest is probably waiting for the recipient to start the program, followed by waiting for recipient to type in code, followed by waiting for recipient to approve transfer, with the time of actual transfer being anywhere among the others). * identify groups of events that are separated by those gaps * put a [1 2 3 4 all] set of buttons at the top of the page * clicking on each button will zoom the display to 10% beyond the span of events in the given group, or reset the zoom to include all events */ function OFF() { /* leftover code from an older implementation, retained since there might still be some useful pieces here */ function y_off(d) { return (LANE_HEIGHT * (d.side*(data.lanes.length+1) + d.lane) + d.wiggle); } var bottom_rule_y = LANE_HEIGHT * data.sides.length * (data.lanes.length+1); var bottom_y = bottom_rule_y + 45; //var chart_g = chart.append("svg:g"); // this "backboard" rect lets us catch mouse events anywhere in the // chart, even between the bars. Without it, we only see events on solid // objects like bars and text, but not in the gaps between. chart.append("svg:rect") .attr("id", "outer_rect") .attr("width", w).attr("height", bottom_y).attr("fill", "none"); // but the stuff we put inside it should have some room w = w-50; chart.selectAll("text.sides-label").data(data.sides).enter() .append("svg:text") .attr("class", "sides-label") .attr("x", "0px") .attr("y", function(d,idx) { return y_off({side: idx, lane: data.lanes.length/2, wiggle: 0}) ;}) .attr("text-anchor", "start") // anchor at top-left .attr("dy", ".71em") .attr("fill", "black") .text(function(d) { return d; }) ; var lanes_by_sides = []; data.sides.forEach(function(side, side_index) { data.lanes.forEach(function(lane, lane_index) { lanes_by_sides.push({side: side, side_index: side_index, lane: lane, lane_index: lane_index}); }); }); chart.selectAll("text.lanes-label").data(lanes_by_sides).enter() .append("svg:text") .attr("class", "lanes-label") .attr("x", "50px") .attr("y", function(d) { return y_off({side: d.side_index, lane: d.lane_index, wiggle: 0}) ;}) .attr("text-anchor", "start") // anchor at top-left .attr("dy", ".91em") .attr("fill", "#f88") .text(function(d) { return d.lane; }) ; chart.append("svg:text") .attr("class", "seconds-label") //.attr("x", w/2).attr("y", y + 35) .attr("text-anchor", "middle") .attr("fill", "black") .text("seconds"); d3.select("#outer_chart").attr("height", bottom_y); d3.select("#outer_rect").attr("height", bottom_y); d3.select("#zoom").attr("transform", "translate("+(w-10)+","+10+")"); function reltime(t) {return t-data.bounds.min;} var last = data.bounds.max - data.bounds.min; //last = reltime(d3.max(data.dyhb, function(d){return d.finish_time;})); last = last * 1.05; // long downloads are likely to have too much info, start small if (last > 10.0) last = 10.0; // d3.time.scale() has no support for ms or us. var xOFF = d3.time.scale().domain([data.bounds.min, data.bounds.max]) .range([0,w]); var x = d3.scale.linear().domain([-last*0.05, last]) .range([0,w]); zoom.x(x); function tx(d) { return "translate(" +x(d) + ",0)"; } function left(d) { return x(reltime(d.start_time)); } function left_server(d) { return x(reltime(d.server_sent)); } function right(d) { return d.finish_time ? x(reltime(d.finish_time)) : "1px"; } function width(d) { return d.finish_time ? x(reltime(d.finish_time))-x(reltime(d.start_time)) : "1px"; } function halfwidth(d) { if (d.finish_time) return (x(reltime(d.finish_time))-x(reltime(d.start_time)))/2; return "1px"; } function middle(d) { if (d.finish_time) return (x(reltime(d.start_time))+x(reltime(d.finish_time)))/2; else return x(reltime(d.start_time)) + 1; } function color(d) { return data.server_info[d.serverid].color; } function servername(d) { return data.server_info[d.serverid].short; } function timeformat(duration) { // TODO: trim to microseconds, maybe humanize return duration; } function oldredraw() { // at this point zoom/pan must be fixed var min = data.bounds.min + x.domain()[0]; var max = data.bounds.min + x.domain()[1]; function inside(d) { var finish_time = d.finish_time || d.start_time; if (Math.max(d.start_time, min) <= Math.min(finish_time, max)) return true; return false; } // from the data, build a list of bars, dots, and lines var clipped = {bars: [], dots: [], lines: []}; data.items.filter(inside).forEach(function(d) { if (!d.finish_time) { clipped.dots.push(d); } else { clipped.bars.push(d); if (!!d.server_sent) { clipped.lines.push(d); } } }); globals.clipped = clipped; //chart.select(".dyhb-label") // .attr("x", x(0))//"20px") // .attr("y", y); // Panning and zooming will re-run this function multiple times, and // bars will come and go, so we must process all three selections // (including enter() and exit()). // TODO: add dots for events that have only start, not finish. Add // the server-sent bar (a vertical line, half height, centered // vertically) for events that have server-sent as well as finish. // This probably requires creating a dot for everything, but making // it invisible if finished is non-null, likewise for the server-sent // bar. // each item gets an SVG group (g.bars), translated left and down // to match the start time and side/lane of the event var bars = chart.selectAll("g.bars") .data(clipped.bars, function(d) { return d.start_time; }) .attr("transform", function(d) { return "translate("+left(d)+","+y_off(d)+")"; }) ; // update the variable parts of each bar, which depends upon the // current pan/zoom values bars.select("rect") .attr("width", width); bars.select("text") .attr("x", halfwidth); bars.exit().remove(); var new_bars = bars.enter() .append("svg:g") .attr("class", "bars") .attr("transform", function(d) { return "translate("+left(d)+","+y_off(d)+")"; }) ; // inside the group, we have a rect with a width for the duration of // the event, and a fixed height. The fill and stroke color depend // upon the event, and the title has the details. We append the rects // first, so the text is drawn on top (higher z-order) //y += 30*(1+d3.max(data.bars, function(d){return d.row;})); new_bars.append("svg:rect") .attr("width", width) .attr("height", RECT_HEIGHT) .attr("class", function(d) { var c = ["bar", "lane-" + d.lane]; if (d.details.waiting) c.push("wait-" + d.details.waiting); return c.join(" "); }) .on("mouseover", function(d) {if (d.details_str) tip.show(d);}) .on("mouseout", tip.hide) //.attr("title", function(d) {return d.details_str;}) ; // each group also has a text, with 'x' set to place it in the middle // of the rect, and text contents that are drawn in the rect new_bars.append("svg:text") .attr("x", halfwidth) .attr("text-anchor", "middle") .attr("dy", "0.9em") //.attr("fill", "black") .text((d) => d.what) .on("mouseover", function(d) {if (d.details_str) tip.show(d);}) .on("mouseout", tip.hide) ; // dots: events that have a single timestamp, rather than a range. // These get an SVG group, and a circle and some text. var dots = chart.selectAll("g.dots") .data(clipped.dots, (d) => d.start_time) .attr("transform", (d) => "translate("+left(d)+","+(y_off(d)+LANE_HEIGHT/3)+")") ; dots.exit().remove(); var new_dots = dots.enter() .append("svg:g") .attr("class", "dots") .attr("transform", (d) => "translate("+left(d)+","+(y_off(d)+LANE_HEIGHT/3)+")") ; new_dots.append("svg:circle") .attr("r", "5") .attr("class", (d) => "dot lane-"+d.lane) .attr("fill", "#888") .attr("stroke", "black") .on("mouseover", function(d) {if (d.details_str) tip.show(d);}) .on("mouseout", tip.hide) ; new_dots.append("svg:text") .attr("x", "5px") .attr("text-anchor", "start") .attr("dy", "0.2em") .text((d) => d.what) .on("mouseover", function(d) {if (d.details_str) tip.show(d);}) .on("mouseout", tip.hide) ; // lines: these represent the time at which the server sent a message // which finished a bar. These get an SVG group, and a line var linedata = clipped.lines.map(d => [ [d.server_sent, 0], [d.server_sent, LANE_HEIGHT], [d.finish_time, 0], ]); function lineshape(d) { var l = d3.svg.line() .x(d => x(d[0])) .y(d => y_off(d) + 12345); } function update_line(sel) { sel.attr("d", lineshape) .attr("class", d => "line lane-"+d.lane) ; } var lines = chart.selectAll("polyline.lines") .data(linedata) .attr("transform", (d) => "translate("+left(d)+","+y_off(d)+")") ; lines.exit().remove(); var new_lines = lines.enter() .append("svg:g") .attr("class", "lines") .attr("transform", (d) => "translate("+left_server(d)+","+(y_off(d))+")") ; new_lines.append("svg:line") .attr("x1", 0) .attr("y1", -5) .attr("x2", "0") .attr("y2", LANE_HEIGHT) .attr("class", (d) => "line lane-"+d.lane) .attr("stroke", "red") ; new_lines.append("svg:line") .attr("x1", 0).attr("y1", -5) .attr("x2", (d) => x(d.finish_time - d.server_sent)) .attr("y2", 0) .attr("class", (d) => "line lane-"+d.lane) .attr("stroke", "red") ; // horizontal scale markers: vertical lines at rational timestamps var rules = chart.selectAll("g.rule") .data(x.ticks(10)) .attr("transform", tx); rules.select("text").text(x.tickFormat(10)); var newrules = rules.enter().insert("svg:g") .attr("class", "rule") .attr("transform", tx) ; newrules.append("svg:line") .attr("class", "rule-tick") .attr("stroke", "black"); chart.selectAll("line.rule-tick") .attr("y1", bottom_rule_y) .attr("y2", bottom_rule_y + 6); newrules.append("svg:line") .attr("class", "rule-red") .attr("stroke", "red") .attr("stroke-opacity", .3); chart.selectAll("line.rule-red") .attr("y1", 0) .attr("y2", bottom_rule_y); newrules.append("svg:text") .attr("class", "rule-text") .attr("dy", ".71em") .attr("text-anchor", "middle") .attr("fill", "black") .text(x.tickFormat(10)); chart.selectAll("text.rule-text") .attr("y", bottom_rule_y + 9); rules.exit().remove(); chart.select(".seconds-label") .attr("x", w/2) .attr("y", bottom_rule_y + 35); } globals.x = x; globals.redraw = redraw; zoom.on("zoom", redraw); d3.select("#zoom_in_button").on("click", zoomin); d3.select("#zoom_out_button").on("click", zoomout); d3.select("#reset_button").on("click", function() { x.domain([-last*0.05, last]).range([0,w]); redraw(); }); redraw(); $.get("done", function(_) {}); } magic-wormhole-0.9.1/misc/windows-build.cmd0000644000076600000240000000150612664633034021275 0ustar warnerstaff00000000000000@echo off :: To build extensions for 64 bit Python 3, we need to configure environment :: variables to use the MSVC 2010 C++ compilers from GRMSDKX_EN_DVD.iso of: :: MS Windows SDK for Windows 7 and .NET Framework 4 :: :: More details at: :: https://github.com/cython/cython/wiki/64BitCythonExtensionsOnWindows IF "%DISTUTILS_USE_SDK%"=="1" ( ECHO Configuring environment to build with MSVC on a 64bit architecture ECHO Using Windows SDK 7.1 "C:\Program Files\Microsoft SDKs\Windows\v7.1\Setup\WindowsSdkVer.exe" -q -version:v7.1 CALL "C:\Program Files\Microsoft SDKs\Windows\v7.1\Bin\SetEnv.cmd" /x64 /release SET MSSdk=1 REM Need the following to allow tox to see the SDK compiler SET TOX_TESTENV_PASSENV=DISTUTILS_USE_SDK MSSdk INCLUDE LIB ) ELSE ( ECHO Using default MSVC build environment ) CALL %* magic-wormhole-0.9.1/NEWS.md0000644000076600000240000003464313032243376016170 0ustar warnerstaff00000000000000 User-visible changes in "magic-wormhole": ## Release 0.9.1 (01-Jan-2017) The `wormhole` client's `--transit-helper=` argument can now include a "relay priority" via a numerical `priority=` field, e.g. `--transit-helper tcp:example.org:12345:priority=2.5`. Clients exchange transit relay suggestions, then try to use the highest-priority relay first, falling back to others after a few seconds if necessary. Direct connections are always preferred to a relay. Clients running 0.9.0 or earlier will ignore priorities, and unmarked relay arguments have an implicit priority of 0. (#103) Other changes: * clients now tolerate duplicate peer messages: in the future, this will help clients recover from intermittent rendezvous connections (#121) * rendezvous server: ensure release() and close() are idempotent (from different connections), also for lost-connection recovery (#118) * transit server: respect --blur-usage= by not logging connections * README: note py3.6 compatibility Thanks to xloem, kneufeld, and meejah for their help this cycle. ## Release 0.9.0 (24-Dec-2016) This release fixes an important "Transit Relay" bug that would have prevented future versions from using non-default relay servers. It is now easier to run `wormhole` as a subprocess beneath some other program (the long term goal is to provide a nice API, but even with one, there will be programs written in languages without Wormhole bindings that may find it most convenient to use a subprocess). * fix `--transit-helper=`: Older versions had a bug that broke file/directory transfers when the two sides offered different transit-relay servers. This was fixed by deduplicating relay hints and adding a new kind of relay handshake. Clients running 0.9.0 or higher now require a transit-relay server running 0.9.0 or higher. (#115) * `wormhole receive`: reject transfers when the target does not appear to have enough space (not available on windows) (#91) * CLI: emit pacifier message when key-verification is slow (#29) * add `--appid=` so wrapping scripts can use a distinct value (#113) * `wormhole send`: flush output after displaying code, for use in scripts (#108) * CLI: print progress messages to stderr, not stdout (#99) * add basic man(1) pages (#69) Many thanks to patch submitters for this release: Joey Hess, Jared Anderson, Antoine Beaupré, and to everyone testing and filing issues on Github. ## Release 0.8.2 (08-Dec-2016) * CLI: add new "wormhole ssh invite" and "wormhole ssh accept" commands, to facilitate appending your `~/.ssh/id_*.pub` key into a suitably-permissioned remote `~/.ssh/authorized_keys` file. These commands are experimental: the syntax might be changed in the future, or they might be removed altogether. * CLI: "wormhole recv" and "wormhole recieve" are now accepted as aliases for "wormhole receive", to help bad spelers :) * CLI: improve display of abbreviated file sizes * CLI: don't print traceback upon "normal" errors * CLI: when target file already exists, don't reveal that fact to the sender, just say "transfer rejected" * magic-wormhole now depends upon `Twisted[tls]`, which will cause pyOpenSSL and the `cryptography` package to be installed. This should prevent a warning about the "service_identity" module not being available. * other smaller internal changes Thanks to everyone who submitted patches in this release cycle: anarcat, Ofekmeister, Tom Lowenthal, meejah, dreid, and dkg. And thanks to the many bug reporters on Github! ## Release 0.8.1 (27-Jul-2016) This release contains mostly minor changes. The most noticeable is that long-lived wormholes should be more reliable now. Previously, if you run `wormhole send` but your peer doesn't run their `receive` for several hours, a NAT/firewall box on either side could stop forwarding traffic for the idle connection (without sending a FIN or RST to properly close the socket), causing both sides to hang forever and never actually connect. Now both sides send periodic keep-alive messages to prevent this. In addition, by switching to "Click" for argument parsing, we now have short command aliases: `wormhole tx` does the same thing as `wormhole send`, and `wormhole rx` is an easier-to-spell equivalent of `wormhole receive`. Other changes: * CLI: move most arguments to be attached to the subcommand (new: `wormhole send --verify`) rather than on the "wormhole" command (old: `wormhole --verify send`). Four arguments remain on the "wormhole" command: `--relay-url=`, `--transit-helper=`, `--dump-timing=`, and `--version`. * docs: add links to PyCon2016 presentation * reject wormhole-codes with spaces with a better error message * magic-wormhole ought to work on windows now * code-input tab-completion should work on stock OS-X python (with libedit) * sending a directory should restore file permissions correctly * server changes: * expire channels after two hours, not 3 days * prune channels more accurately * improve munin plugins for server monitoring Many thanks to the folks who contributed to this release, during the PyCon sprints and afterwards: higs4281, laharah, Chris Wolfe, meejah, wsanchez, Kurt Neufeld, and Francois Marier. ## Release 0.8.0 (28-May-2016) This release is completely incompatible with the previous 0.7.6 release. Clients using 0.7.6 or earlier will not even notice clients using 0.8.0 or later. * Overhaul client-server websocket protocol, client-client PAKE messages, per-message encryption-key derivation, relay-server database schema, SPAKE2 key-derivation, and public relay URLs. Add version fields and unknown-message tolerance to most protocol steps. * Hopefully this will provide forward-compatibility with future protocol changes. I have several on my list, and the version fields should make it possible to add these without a flag day (at worst a "flag month"). * User-visible changes are minimal, although some operations should be faster because we no longer need to wait for ACKs before proceeding. * API changes: `.send_data()/.get_data()` became `.send()/.get()`, neither takes a phase= argument (the Wormhole is now a record pipe) `.get_verifier()` became `.verify()` (and waits to receive the key-confirmation message before firing its Deferred), wormholes are constructed with a function call instead of a class constructor, `close()` always waits for server ack of outbound messages. Note that the API remains unstable until 1.0.0 . * misc/munin/ contains plugins for relay server operators ## Release 0.7.6 (08-May-2016) * Switch to "tqdm" for nicer CLI progress bars. * Fail better when input-code is interrupted (prompt user to hit Return, rather than hanging forever) * Close channel upon error more reliably. * Explain WrongPasswordError better. * (internal): improve --dump-timing instrumentation and rendering. Compatibility: this remains compatible with 0.7.x, and 0.8.x is still expected to break compatibility. ## Release 0.7.5 (20-Apr-2016) * The CLI tools now use the Twisted-based library exclusively. * The blocking-flavor "Transit" library has been removed. Transit is the bulk-transfer protocol used by send-file/send-directory. Upcoming protocol improvements (performance and connectivity) proved too difficult to implement in a blocking fashion, so for now if you want Transit, use Twisted. * The Twisted-flavor "Wormhole" library now uses WebSockets to connect, rather than HTTP. The blocking-flavor library continues to use HTTP. "Wormhole" is the one-message-at-a-time relay-based protocol, and is used to set up Transit for the send-file and send-directory modes of the CLI tool. * Twisted-flavor input_code() now does readline-based code entry, with tab completion. * The package now installs two executables: "wormhole" (for send and receive), and "wormhole-server" (to start and manage the relay servers). These may be re-merged in a future release. Compatibility: * This release remains compatible with the previous ones. The next major release (0.8.x) will probably break compatibility. Packaging: * magic-wormhole now depends upon "Twisted" and "autobahn" (for WebSockets). Autobahn pulls in txaio, but we don't support it yet (a future version of magic-wormhole might). * To work around a bug in autobahn, we also (temporarily) depend upon "pytrie". This dependency will be removed when the next autobahn release is available. ## Release 0.7.0 (28-Mar-2016) * `wormhole send DIRNAME/` used to deal very badly with the trailing slash (sending a directory with an empty name). This is now fixed. * Preliminary Tor support was added. Install `magic-wormhole[tor]`, make sure you have a Tor executable on your $PATH, and run `wormhole --tor send`. This will launch a new Tor process. Do not use this in anger/fear until it has been tested more carefully. This feature is likely to be unstable for a while, and lacks tests. * The relay now prunes unused channels properly. * Added --dump-timing= to record timeline of events, for debugging and performance improvements. You can combine timing data from both sides to see where the delays are happening. The server now returns timestamps in its responses, to measure round-trip delays. A web-based visualization tool was added in `misc/dump-timing.py`. * twisted.transit was not properly handling multiple records received in a single chunk. Some producer/consumer helper methods were added. You can now run e.g. `wormhole --twisted send` to force the use of the Twisted implementation. * The Twisted wormhole now uses a persistent connection for all relay messages, which should be slightly faster. * Add `--no-listen` to prevent Transit from listening for inbound connections (or advertising any addresses): this is only useful for testing. * The tests now collect code coverage information, and upload them to https://codecov.io/github/warner/magic-wormhole?ref=master . ## Release 0.6.3 (29-Feb-2016) Mostly internal changes: * twisted.transit was added, so Twisted-based applications can use it now. This includes Producer/Consumer -based flow control. The Transit protocol and API are documented in docs/transit.md . * The transit relay server can blur filesizes, rounding them to some roughly-logarithmic interval. * Use --relay-helper="" to disable use of the transit relay entirely, limiting the file transfer to direct connections. * The new --hide-progress option disables the progress bar. * Made some windows-compatibility fixes, but all tests do not yet pass. ## Release 0.6.2 (12-Jan-2016) * the server can now "blur" usage information: this turns off HTTP logging, and rounds timestamps to coarse intervals * `wormhole server usage` now shows Transit usage too, not just Rendezvous ## Release 0.6.1 (03-Dec-2015) * `wormhole` can now send/receive entire directories. They are zipped before transport. * Python 3 is now supported for async (Twisted) library use, requiring at least Twisted-15.5.0. * A bug was fixed which prevented py3-based clients from using the relay transit server (not used if the two sides can reach each other directly). * The `--output-file=` argument was finally implemented, which allows the receiver to override the filename that it writes. This may help scripted usage. * Support for Python-2.6 was removed, since the recent Twisted-15.5.0 removed it too. It might still work, but is no longer automatically tested. * The transit relay now implements proper flow control (Producer/Consumer), so it won't buffer the entire file when the sender can push data faster than the receiver can accept it. The sender should now throttle down to the receiver's maximum rate. ## Release 0.6.0 (23-Nov-2015) * Add key-confirmation message so "wormhole send" doesn't hang when the receiver mistypes the code. * Fix `wormhole send --text -` to read the text message from stdin. `wormhole receive >outfile` works, but currently appends an extra newline, which may be removed in a future release. * Arrange for 0.4.0 senders to print an error message when connecting to a current (0.5.0) server, instead of an ugly stack trace. Unfortunately 0.4.0 receivers still display the traceback, since they don't check the welcome message before using a missing API. 0.5.0 and 0.6.0 will do better. * Improve channel deallocation upon error. * Inform the server of our "mood" when the connection closes, so it can track the rate of successful/unsuccessful transfers. The server DB now stores a summary of each transfer (waiting time and reported outcome). * Rename (and deprecate) one server API (the non-EventSource form of "get"), leaving it in place until after the next release. 0.5.0 clients should interoperate with both the 0.6.0 server and 0.6.0 clients, but eventually they'll stop working. ## Release 0.5.0 (07-Oct-2015) * Change the CLI to merge send-file with send-text, and receive-file with receive-text. Add confirmation before accepting a file. * Change the remote server API significantly, breaking compatibility with 0.4.0 peers. Fix EventSource to match W3C spec and real browser behavior. * Add py3 (3.3, 3.4, 3.5) compatibility for blocking calls (but not Twisted). * internals * Introduce Channel and ChannelManager to factor out the HTTP/EventSource technology in use (making room for WebSocket or Tor in the future). * Change app-visible API to allow multiple message phases. * Change most API arguments from bytes to unicode strings (appid, URLs, wormhole code, derive_key purpose string, message phase). Derived keys are bytes, of course. * Add proper unit tests. ## Release 0.4.0 (22-Sep-2015) This changes the protocol (to a symmetric form), breaking compatibility with 0.3.0 peers. Now both blocking-style and Twisted-style use a symmetric protocol, and the two sides do not need to figure out (ahead of time) which one goes first. The internal layout was rearranged, so applications that import wormhole must be updated. ## Release 0.3.0 (24-Jun-2015) Add preliminary Twisted support, only for symmetric endpoints (no initator/receiver distinction). Lacks code-entry tab-completion. May still leave timers lingering. Add test suite (only for Twisted, so far). Use a sqlite database for Relay server state, to survive reboots with less data loss. Add "--advertise-version=" to "wormhole relay start", to override the version we recommend to clients. ## Release 0.2.0 (10-Apr-2015) Initial release: supports blocking/synchronous asymmetric endpoints (Initiator on one side, Receiver on the other). Codes can be generated by Initiator, or created externally and passed into both (as long as they start with digits: NNN-anything). magic-wormhole-0.9.1/PKG-INFO0000644000076600000240000000043013032243761016150 0ustar warnerstaff00000000000000Metadata-Version: 1.0 Name: magic-wormhole Version: 0.9.1 Summary: Securely transfer data between computers Home-page: https://github.com/warner/magic-wormhole Author: Brian Warner Author-email: warner-magic-wormhole@lothar.com License: MIT Description: UNKNOWN Platform: UNKNOWN magic-wormhole-0.9.1/README.md0000644000076600000240000002071213031527221016332 0ustar warnerstaff00000000000000# Magic Wormhole [![Build Status](https://travis-ci.org/warner/magic-wormhole.svg?branch=master)](https://travis-ci.org/warner/magic-wormhole) [![codecov.io](https://codecov.io/github/warner/magic-wormhole/coverage.svg?branch=master)](https://codecov.io/github/warner/magic-wormhole?branch=master) Get things from one computer to another, safely. This package provides a library and a command-line tool named `wormhole`, which makes it possible to get short pieces of text (and arbitrary-sized files and directories) from one computer to another. The two endpoints are identified by using identical "wormhole codes": in general, the sending machine generates and displays the code, which must then be typed into the receiving machine. The codes are short and human-pronounceable, using a phonetically-distinct wordlist. The receiving side offers tab-completion on the codewords, so usually only a few characters must be typed. Wormhole codes are single-use and do not need to be memorized. * PyCon 2016 presentation: [Slides](http://www.lothar.com/~warner/MagicWormhole-PyCon2016.pdf), [Video](https://youtu.be/oFrTqQw0_3c) ## Example Sender: ``` % wormhole send README.md Sending 7924 byte file named 'README.md' On the other computer, please run: wormhole receive Wormhole code is: 7-crossover-clockwork Sending (<-10.0.1.43:58988).. 100%|=========================| 7.92K/7.92K [00:00<00:00, 6.02MB/s] File sent.. waiting for confirmation Confirmation received. Transfer complete. ``` Receiver: ``` % wormhole receive Enter receive wormhole code: 7-crossover-clockwork Receiving file (7924 bytes) into: README.md ok? (y/n): y Receiving (->tcp:10.0.1.43:58986).. 100%|===========================| 7.92K/7.92K [00:00<00:00, 120KB/s] Received file written to README.md ``` ## Installation ```$ pip install magic-wormhole``` On Debian/Ubuntu systems, you may first need `apt-get install python-pip build-essential python-dev libffi-dev libssl-dev`. On Fedora it's `dnf install python-pip python-devel libffi-devel openssl-devel gcc-c++ libtool redhat-rpm-config`. On OS-X, you may need to install `pip` and run `xcode-select --install` to get GCC. On Windows, python2 may work better than python3. On older systems, `pip install --upgrade pip` may be necessary to get a version that can compile all the dependencies. If you get errors like `fatal error: sodium.h: No such file or directory` on Linux, either use `SODIUM_INSTALL=bundled pip install magic-wormhole`, or try installing the `libsodium-dev` / `libsodium-devel` package. These work around a bug in pynacl which gets confused when the libsodium runtime is installed (e.g. `libsodium13`) but not the development package. Developers can clone the source tree and run `tox` to run the unit tests on all supported (and installed) versions of python: 2.7, 3.3, 3.4, 3.5, and 3.6. ## Motivation * Moving a file to a friend's machine, when the humans can speak to each other (directly) but the computers cannot * Delivering a properly-random password to a new user via the phone * Supplying an SSH public key for future login use Copying files onto a USB stick requires physical proximity, and is uncomfortable for transferring long-term secrets because flash memory is hard to erase. Copying files with ssh/scp is fine, but requires previous arrangements and an account on the target machine, and how do you bootstrap the account? Copying files through email first requires transcribing an email address in the opposite direction, and is even worse for secrets, because email is unencrypted. Copying files through encrypted email requires bootstrapping a GPG key as well as an email address. Copying files through Dropbox is not secure against the Dropbox server and results in a large URL that must be transcribed. Using a URL shortener adds an extra step, reveals the full URL to the shortening service, and leaves a short URL that can be guessed by outsiders. Many common use cases start with a human-mediated communication channel, such as IRC, IM, email, a phone call, or a face-to-face conversation. Some of these are basically secret, or are "secret enough" to last until the code is delivered and used. If this does not feel strong enough, users can turn on additional verification that doesn't depend upon the secrecy of the channel. The notion of a "magic wormhole" comes from the image of two distant wizards speaking the same enchanted phrase at the same time, and causing a mystical connection to pop into existence between them. The wizards then throw books into the wormhole and they fall out the other side. Transferring files securely should be that easy. ## Design The `wormhole` tool uses PAKE "Password-Authenticated Key Exchange", a family of cryptographic algorithms that uses a short low-entropy password to establish a strong high-entropy shared key. This key can then be used to encrypt data. `wormhole` uses the SPAKE2 algorithm, due to Abdalla and Pointcheval[1]. PAKE effectively trades off interaction against offline attacks. The only way for a network attacker to learn the shared key is to perform a man-in-the-middle attack during the initial connection attempt, and to correctly guess the code being used by both sides. Their chance of doing this is inversely proportional to the entropy of the wormhole code. The default is to use a 16-bit code (use --code-length= to change this), so for each use of the tool, an attacker gets a 1-in-65536 chance of success. As such, users can expect to see many error messages before the attacker has a reasonable chance of success. ## Timing The program does not have any built-in timeouts, however it is expected that both clients will be run within an hour or so of each other. This makes the tool most useful for people who are having a real-time conversation already, and want to graduate to a secure connection. Both clients must be left running until the transfer has finished. ## Relays The wormhole library requires a "Rendezvous Server": a simple WebSocket-based relay that delivers messages from one client to another. This allows the wormhole codes to omit IP addresses and port numbers. The URL of a public server is baked into the library for use as a default, and will be freely available until volume or abuse makes it infeasible to support. Applications which desire more reliability can easily run their own relay and configure their clients to use it instead. Code for the Rendezvous Server is included in the library. The file-transfer commands also use a "Transit Relay", which is another simple server that glues together two inbound TCP connections and transfers data on each to the other. The `wormhole send` file mode shares the IP addresses of each client with the other (inside the encrypted message), and both clients first attempt to connect directly. If this fails, they fall back to using the transit relay. As before, the host/port of a public server is baked into the library, and should be sufficient to handle moderate traffic. The protocol includes provisions to deliver notices and error messages to clients: if either relay must be shut down, these channels will be used to provide information about alternatives. ## CLI tool * `wormhole send [args] --text TEXT` * `wormhole send [args] FILENAME` * `wormhole send [args] DIRNAME` * `wormhole receive [args]` Both commands accept additional arguments to influence their behavior: * `--code-length WORDS`: use more or fewer than 2 words for the code * `--verify` : print (and ask user to compare) extra verification string ## Library The `wormhole` module makes it possible for other applications to use these code-protected channels. This includes Twisted support, and (in the future) will include blocking/synchronous support too. See docs/api.md for details. The file-transfer tools use a second module named `wormhole.transit`, which provides an encrypted record-pipe. It knows how to use the Transit Relay as well as direct connections, and attempts them all in parallel. `TransitSender` and `TransitReceiver` are distinct, although once the connection is established, data can flow in either direction. All data is encrypted (using nacl/libsodium "secretbox") using a key derived from the PAKE phase. See `src/wormhole/cli/cmd_send.py` for examples. ## License, Compatibility This library is released under the MIT license, see LICENSE for details. This library is compatible with python2.7, 3.3, 3.4, 3.5, and 3.6 . It is probably compatible with py2.6, but the latest Twisted (>=15.5.0) is not. #### footnotes [1]: http://www.di.ens.fr/~pointche/Documents/Papers/2005_rsa.pdf "RSA 2005" magic-wormhole-0.9.1/setup.cfg0000644000076600000240000000036113032243761016677 0ustar warnerstaff00000000000000[wheel] universal = 1 [versioneer] vcs = git versionfile_source = src/wormhole/_version.py versionfile_build = wormhole/_version.py tag_prefix = parentdir_prefix = magic-wormhole [egg_info] tag_build = tag_date = 0 tag_svn_revision = 0 magic-wormhole-0.9.1/setup.py0000644000076600000240000000246413031527242016574 0ustar warnerstaff00000000000000 from setuptools import setup import versioneer commands = versioneer.get_cmdclass() setup(name="magic-wormhole", version=versioneer.get_version(), description="Securely transfer data between computers", author="Brian Warner", author_email="warner-magic-wormhole@lothar.com", license="MIT", url="https://github.com/warner/magic-wormhole", package_dir={"": "src"}, packages=["wormhole", "wormhole.cli", "wormhole.server", "wormhole.test", ], package_data={"wormhole.server": ["db-schemas/*.sql"]}, entry_points={ "console_scripts": [ "wormhole = wormhole.cli.cli:wormhole", "wormhole-server = wormhole.server.cli:server", ] }, install_requires=[ "spake2==0.7", "pynacl", "six", "twisted[tls]", "autobahn[twisted] >= 0.14.1", "hkdf", "tqdm", "click", "humanize", ], extras_require={ ':sys_platform=="win32"': ["pypiwin32"], "tor": ["txtorcon", "ipaddress"], "dev": [ "mock", "tox", "pyflakes", ], }, test_suite="wormhole.test", cmdclass=commands, ) magic-wormhole-0.9.1/src/0000755000076600000240000000000013032243761015645 5ustar warnerstaff00000000000000magic-wormhole-0.9.1/src/magic_wormhole.egg-info/0000755000076600000240000000000013032243761022333 5ustar warnerstaff00000000000000magic-wormhole-0.9.1/src/magic_wormhole.egg-info/dependency_links.txt0000644000076600000240000000000113032243761026401 0ustar warnerstaff00000000000000 magic-wormhole-0.9.1/src/magic_wormhole.egg-info/entry_points.txt0000644000076600000240000000014513032243761025631 0ustar warnerstaff00000000000000[console_scripts] wormhole = wormhole.cli.cli:wormhole wormhole-server = wormhole.server.cli:server magic-wormhole-0.9.1/src/magic_wormhole.egg-info/PKG-INFO0000644000076600000240000000043013032243761023425 0ustar warnerstaff00000000000000Metadata-Version: 1.0 Name: magic-wormhole Version: 0.9.1 Summary: Securely transfer data between computers Home-page: https://github.com/warner/magic-wormhole Author: Brian Warner Author-email: warner-magic-wormhole@lothar.com License: MIT Description: UNKNOWN Platform: UNKNOWN magic-wormhole-0.9.1/src/magic_wormhole.egg-info/requires.txt0000644000076600000240000000026013032243761024731 0ustar warnerstaff00000000000000spake2==0.7 pynacl six twisted[tls] autobahn[twisted] >= 0.14.1 hkdf tqdm click humanize [:sys_platform=="win32"] pypiwin32 [dev] mock tox pyflakes [tor] txtorcon ipaddress magic-wormhole-0.9.1/src/magic_wormhole.egg-info/SOURCES.txt0000644000076600000240000000416713032243761024227 0ustar warnerstaff00000000000000.coveragerc LICENSE MANIFEST.in NEWS.md README.md setup.cfg setup.py tox.ini versioneer.py docs/api.md docs/transit.md docs/wormhole-server.8 docs/wormhole.1 misc/dump-stats.py misc/dump-timing.py misc/windows-build.cmd misc/munin/wormhole_active misc/munin/wormhole_errors misc/munin/wormhole_event_rate misc/munin/wormhole_events misc/munin/wormhole_events_alltime misc/munin/wormhole_transit misc/munin/wormhole_transit_alltime misc/web/timeline.css misc/web/timeline.html misc/web/timeline.js src/magic_wormhole.egg-info/PKG-INFO src/magic_wormhole.egg-info/SOURCES.txt src/magic_wormhole.egg-info/dependency_links.txt src/magic_wormhole.egg-info/entry_points.txt src/magic_wormhole.egg-info/requires.txt src/magic_wormhole.egg-info/top_level.txt src/wormhole/__init__.py src/wormhole/_version.py src/wormhole/channel_monitor.py src/wormhole/codes.py src/wormhole/errors.py src/wormhole/ipaddrs.py src/wormhole/timing.py src/wormhole/tor_manager.py src/wormhole/transit.py src/wormhole/util.py src/wormhole/wordlist.py src/wormhole/wormhole.py src/wormhole/xfer_util.py src/wormhole/cli/__init__.py src/wormhole/cli/cli.py src/wormhole/cli/cmd_receive.py src/wormhole/cli/cmd_send.py src/wormhole/cli/cmd_ssh.py src/wormhole/cli/public_relay.py src/wormhole/server/__init__.py src/wormhole/server/cli.py src/wormhole/server/cmd_server.py src/wormhole/server/cmd_usage.py src/wormhole/server/database.py src/wormhole/server/rendezvous.py src/wormhole/server/rendezvous_websocket.py src/wormhole/server/runner.py src/wormhole/server/server.py src/wormhole/server/transit_server.py src/wormhole/server/db-schemas/upgrade-to-v3.sql src/wormhole/server/db-schemas/v2.sql src/wormhole/server/db-schemas/v3.sql src/wormhole/test/__init__.py src/wormhole/test/common.py src/wormhole/test/run_trial.py src/wormhole/test/test_args.py src/wormhole/test/test_database.py src/wormhole/test/test_hkdf.py src/wormhole/test/test_scripts.py src/wormhole/test/test_server.py src/wormhole/test/test_ssh.py src/wormhole/test/test_transit.py src/wormhole/test/test_transit_server.py src/wormhole/test/test_util.py src/wormhole/test/test_wormhole.py src/wormhole/test/test_xfer_util.pymagic-wormhole-0.9.1/src/magic_wormhole.egg-info/top_level.txt0000644000076600000240000000001113032243761025055 0ustar warnerstaff00000000000000wormhole magic-wormhole-0.9.1/src/wormhole/0000755000076600000240000000000013032243761017501 5ustar warnerstaff00000000000000magic-wormhole-0.9.1/src/wormhole/__init__.py0000644000076600000240000000013512704271263021614 0ustar warnerstaff00000000000000 from ._version import get_versions __version__ = get_versions()['version'] del get_versions magic-wormhole-0.9.1/src/wormhole/_version.py0000644000076600000240000000076113032243761021703 0ustar warnerstaff00000000000000 # This file was generated by 'versioneer.py' (0.17) from # revision-control system data, or from the parent directory name of an # unpacked source archive. Distribution tarballs contain a pre-generated copy # of this file. import json version_json = ''' { "date": "2017-01-01T13:15:04-0500", "dirty": false, "error": null, "full-revisionid": "7393f1e80d81bee95818cec5b7e1fddac191320b", "version": "0.9.1" } ''' # END VERSION_JSON def get_versions(): return json.loads(version_json) magic-wormhole-0.9.1/src/wormhole/channel_monitor.py0000644000076600000240000000073113024532264023233 0ustar warnerstaff00000000000000from __future__ import print_function, unicode_literals import sys from weakref import ref class ChannelMonitor: def __init__(self): self._open_channels = set() def add(self, w): wr = ref(w, self._lost) self._open_channels.add(wr) def _lost(self, wr): print("Error: a Wormhole instance was not closed", file=sys.stderr) def close(self, w): self._open_channels.discard(ref(w)) monitor = ChannelMonitor() # singleton magic-wormhole-0.9.1/src/wormhole/cli/0000755000076600000240000000000013032243761020250 5ustar warnerstaff00000000000000magic-wormhole-0.9.1/src/wormhole/cli/__init__.py0000644000076600000240000000000012706034367022356 0ustar warnerstaff00000000000000magic-wormhole-0.9.1/src/wormhole/cli/cli.py0000644000076600000240000002037613027366362021410 0ustar warnerstaff00000000000000from __future__ import print_function import os import time start = time.time() from textwrap import fill, dedent from sys import stdout, stderr from . import public_relay from .. import __version__ from ..timing import DebugTiming from ..errors import (WrongPasswordError, WelcomeError, KeyFormatError, TransferError) from twisted.internet.defer import inlineCallbacks, maybeDeferred from twisted.python.failure import Failure from twisted.internet.task import react import click top_import_finish = time.time() class Config(object): """ Union of config options that we pass down to (sub) commands. """ def __init__(self): # This only holds attributes which are *not* set by CLI arguments. # Everything else comes from Click decorators, so we can be sure # we're exercising the defaults. self.timing = DebugTiming() self.cwd = os.getcwd() self.stdout = stdout self.stderr = stderr self.tor = False # XXX? def _compose(*decorators): def decorate(f): for d in reversed(decorators): f = d(f) return f return decorate ALIASES = { "tx": "send", "rx": "receive", "recieve": "receive", "recv": "receive", } class AliasedGroup(click.Group): def get_command(self, ctx, cmd_name): cmd_name = ALIASES.get(cmd_name, cmd_name) return click.Group.get_command(self, ctx, cmd_name) # top-level command ("wormhole ...") @click.group(cls=AliasedGroup) @click.option( "--appid", default=None, metavar="APPID", help="appid to use") @click.option( "--relay-url", default=public_relay.RENDEZVOUS_RELAY, metavar="URL", help="rendezvous relay to use", ) @click.option( "--transit-helper", default=public_relay.TRANSIT_RELAY, metavar="tcp:HOST:PORT", help="transit relay to use", ) @click.option( "--dump-timing", type=type(u""), # TODO: hide from --help output default=None, metavar="FILE.json", help="(debug) write timing data to file", ) @click.version_option( message="magic-wormhole %(version)s", version=__version__, ) @click.pass_context def wormhole(context, dump_timing, transit_helper, relay_url, appid): """ Create a Magic Wormhole and communicate through it. Wormholes are created by speaking the same magic CODE in two different places at the same time. Wormholes are secure against anyone who doesn't use the same code. """ context.obj = cfg = Config() cfg.appid = appid cfg.relay_url = relay_url cfg.transit_helper = transit_helper cfg.dump_timing = dump_timing @inlineCallbacks def _dispatch_command(reactor, cfg, command): """ Internal helper. This calls the given command (a no-argument callable) with the Config instance in cfg and interprets any errors for the user. """ cfg.timing.add("command dispatch") cfg.timing.add("import", when=start, which="top").finish(when=top_import_finish) try: yield maybeDeferred(command) except (WrongPasswordError, KeyFormatError) as e: msg = fill("ERROR: " + dedent(e.__doc__)) print(msg, file=stderr) raise SystemExit(1) except WelcomeError as e: msg = fill("ERROR: " + dedent(e.__doc__)) print(msg, file=stderr) print(file=stderr) print(str(e), file=stderr) raise SystemExit(1) except TransferError as e: print("TransferError: %s" % str(e), file=stderr) raise SystemExit(1) except Exception as e: # this prints a proper traceback, whereas # traceback.print_exc() just prints a TB to the "yield" # line above ... Failure().printTraceback(file=stderr) print("ERROR:", e, file=stderr) raise SystemExit(1) cfg.timing.add("exit") if cfg.dump_timing: cfg.timing.write(cfg.dump_timing, stderr) CommonArgs = _compose( click.option("-0", "zeromode", default=False, is_flag=True, help="enable no-code anything-goes mode", ), click.option("-c", "--code-length", default=2, metavar="NUMWORDS", help="length of code (in bytes/words)", ), click.option("-v", "--verify", is_flag=True, default=False, help="display verification string (and wait for approval)", ), click.option("--hide-progress", is_flag=True, default=False, help="supress progress-bar display", ), click.option("--listen/--no-listen", default=True, help="(debug) don't open a listening socket for Transit", ), click.option("--tor", is_flag=True, default=False, help="use Tor when connecting", ), ) # wormhole send (or "wormhole tx") @wormhole.command() @CommonArgs @click.option( "--code", metavar="CODE", help="human-generated code phrase", ) @click.option( "--text", default=None, metavar="MESSAGE", help="text message to send, instead of a file. Use '-' to read from stdin.", ) @click.argument("what", required=False) @click.pass_obj def send(cfg, **kwargs): """Send a text message, file, or directory""" for name, value in kwargs.items(): setattr(cfg, name, value) with cfg.timing.add("import", which="cmd_send"): from . import cmd_send return go(cmd_send.send, cfg) # this intermediate function can be mocked by tests that need to build a # Config object def go(f, cfg): # note: react() does not return return react(_dispatch_command, (cfg, lambda: f(cfg))) # wormhole receive (or "wormhole rx") @wormhole.command() @CommonArgs @click.option( "--only-text", "-t", is_flag=True, help="refuse file transfers, only accept text transfers", ) @click.option( "--accept-file", is_flag=True, help="accept file transfer without asking for confirmation", ) @click.option( "--output-file", "-o", metavar="FILENAME|DIRNAME", help=("The file or directory to create, overriding the name suggested" " by the sender."), ) @click.argument( "code", nargs=-1, default=None, # help=("The magic-wormhole code, from the sender. If omitted, the" # " program will ask for it, using tab-completion."), ) @click.pass_obj def receive(cfg, code, **kwargs): """ Receive a text message, file, or directory (from 'wormhole send') """ for name, value in kwargs.items(): setattr(cfg, name, value) with cfg.timing.add("import", which="cmd_receive"): from . import cmd_receive if len(code) == 1: cfg.code = code[0] elif len(code) > 1: print( "Pass either no code or just one code; you passed" " {}: {}".format(len(code), ', '.join(code)) ) raise SystemExit(1) else: cfg.code = None return go(cmd_receive.receive, cfg) @wormhole.group() def ssh(): """ Facilitate sending/receiving SSH public keys """ pass @ssh.command(name="invite") @click.option( "-c", "--code-length", default=2, metavar="NUMWORDS", help="length of code (in bytes/words)", ) @click.option( "--user", "-u", default=None, metavar="USER", help="Add to USER's ~/.ssh/authorized_keys", ) @click.pass_context def ssh_invite(ctx, code_length, user): """ Add a public-key to a ~/.ssh/authorized_keys file """ from . import cmd_ssh ctx.obj.code_length = code_length ctx.obj.ssh_user = user return go(cmd_ssh.invite, ctx.obj) @ssh.command(name="accept") @click.argument( "code", nargs=1, required=True, ) @click.option( "--key-file", "-F", default=None, type=click.Path(exists=True), ) @click.option( "--yes", "-y", is_flag=True, help="Skip confirmation prompt to send key", ) @click.pass_obj def ssh_accept(cfg, code, key_file, yes): """ Send your SSH public-key In response to a 'wormhole ssh invite' this will send public-key you specify (if there's only one in ~/.ssh/* that will be sent). """ from . import cmd_ssh kind, keyid, pubkey = cmd_ssh.find_public_key(key_file) print("Sending public key type='{}' keyid='{}'".format(kind, keyid)) if yes is not True: click.confirm("Really send public key '{}' ?".format(keyid), abort=True) cfg.public_key = (kind, keyid, pubkey) cfg.code = code return go(cmd_ssh.accept, cfg) magic-wormhole-0.9.1/src/wormhole/cli/cmd_receive.py0000644000076600000240000003314413027402475023077 0ustar warnerstaff00000000000000from __future__ import print_function import os, sys, six, tempfile, zipfile, hashlib from tqdm import tqdm from humanize import naturalsize from twisted.internet import reactor from twisted.internet.defer import inlineCallbacks, returnValue from twisted.python import log from ..wormhole import wormhole from ..transit import TransitReceiver from ..errors import TransferError, WormholeClosedError from ..util import (dict_to_bytes, bytes_to_dict, bytes_to_hexstr, estimate_free_space) APPID = u"lothar.com/wormhole/text-or-file-xfer" VERIFY_TIMER = 1 class RespondError(Exception): def __init__(self, response): self.response = response class TransferRejectedError(RespondError): def __init__(self): RespondError.__init__(self, "transfer rejected") def receive(args, reactor=reactor): """I implement 'wormhole receive'. I return a Deferred that fires with None (for success), or signals one of the following errors: * WrongPasswordError: the two sides didn't use matching passwords * Timeout: something didn't happen fast enough for our tastes * TransferError: the sender rejected the transfer: verifier mismatch * any other error: something unexpected happened """ return TwistedReceiver(args, reactor).go() class TwistedReceiver: def __init__(self, args, reactor=reactor): assert isinstance(args.relay_url, type(u"")) self.args = args self._reactor = reactor self._tor_manager = None self._transit_receiver = None def _msg(self, *args, **kwargs): print(*args, file=self.args.stderr, **kwargs) @inlineCallbacks def go(self): if self.args.tor: with self.args.timing.add("import", which="tor_manager"): from ..tor_manager import TorManager self._tor_manager = TorManager(self._reactor, timing=self.args.timing) # For now, block everything until Tor has started. Soon: launch # tor in parallel with everything else, make sure the TorManager # can lazy-provide an endpoint, and overlap the startup process # with the user handing off the wormhole code yield self._tor_manager.start() w = wormhole(self.args.appid or APPID, self.args.relay_url, self._reactor, self._tor_manager, timing=self.args.timing) # I wanted to do this instead: # # try: # yield self._go(w, tor_manager) # finally: # yield w.close() # # but when _go had a UsageError, the stacktrace was always displayed # as coming from the "yield self._go" line, which wasn't very useful # for tracking it down. d = self._go(w) d.addBoth(w.close) yield d @inlineCallbacks def _go(self, w): yield self._handle_code(w) yield w.establish_key() def on_slow_connection(): print(u"Key established, waiting for confirmation...", file=self.args.stderr) notify = self._reactor.callLater(VERIFY_TIMER, on_slow_connection) try: verifier = yield w.verify() finally: if not notify.called: notify.cancel() self._show_verifier(verifier) want_offer = True done = False while True: try: them_d = yield self._get_data(w) except WormholeClosedError: if done: returnValue(None) raise TransferError("unexpected close") #print("GOT", them_d) recognized = False if u"transit" in them_d: recognized = True yield self._parse_transit(them_d[u"transit"], w) if u"offer" in them_d: recognized = True if not want_offer: raise TransferError("duplicate offer") try: yield self._parse_offer(them_d[u"offer"], w) except RespondError as r: self._send_data({"error": r.response}, w) raise TransferError(r.response) returnValue(None) if not recognized: log.msg("unrecognized message %r" % (them_d,)) def _send_data(self, data, w): data_bytes = dict_to_bytes(data) w.send(data_bytes) @inlineCallbacks def _get_data(self, w): # this may raise WrongPasswordError them_bytes = yield w.get() them_d = bytes_to_dict(them_bytes) if "error" in them_d: raise TransferError(them_d["error"]) returnValue(them_d) @inlineCallbacks def _handle_code(self, w): code = self.args.code if self.args.zeromode: assert not code code = u"0-" if code: w.set_code(code) else: yield w.input_code("Enter receive wormhole code: ", self.args.code_length) def _show_verifier(self, verifier): verifier_hex = bytes_to_hexstr(verifier) if self.args.verify: self._msg(u"Verifier %s." % verifier_hex) @inlineCallbacks def _parse_transit(self, sender_transit, w): if self._transit_receiver: # TODO: accept multiple messages, add the additional hints to the # existing TransitReceiver return yield self._build_transit(w, sender_transit) @inlineCallbacks def _build_transit(self, w, sender_transit): tr = TransitReceiver(self.args.transit_helper, no_listen=(not self.args.listen), tor_manager=self._tor_manager, reactor=self._reactor, timing=self.args.timing) self._transit_receiver = tr transit_key = w.derive_key(APPID+u"/transit-key", tr.TRANSIT_KEY_LENGTH) tr.set_transit_key(transit_key) tr.add_connection_hints(sender_transit.get("hints-v1", [])) receiver_abilities = tr.get_connection_abilities() receiver_hints = yield tr.get_connection_hints() receiver_transit = {"abilities-v1": receiver_abilities, "hints-v1": receiver_hints, } self._send_data({u"transit": receiver_transit}, w) # TODO: send more hints as the TransitReceiver produces them @inlineCallbacks def _parse_offer(self, them_d, w): if "message" in them_d: self._handle_text(them_d, w) returnValue(None) # transit will be created by this point, but not connected if "file" in them_d: f = self._handle_file(them_d) self._send_permission(w) rp = yield self._establish_transit() datahash = yield self._transfer_data(rp, f) self._write_file(f) yield self._close_transit(rp, datahash) elif "directory" in them_d: f = self._handle_directory(them_d) self._send_permission(w) rp = yield self._establish_transit() datahash = yield self._transfer_data(rp, f) self._write_directory(f) yield self._close_transit(rp, datahash) else: self._msg(u"I don't know what they're offering\n") self._msg(u"Offer details: %r" % (them_d,)) raise RespondError("unknown offer type") def _handle_text(self, them_d, w): # we're receiving a text message print(them_d["message"], file=self.args.stdout) self._send_data({"answer": {"message_ack": "ok"}}, w) def _handle_file(self, them_d): file_data = them_d["file"] self.abs_destname = self._decide_destname("file", file_data["filename"]) self.xfersize = file_data["filesize"] free = estimate_free_space(self.abs_destname) if free is not None and free < self.xfersize: self._msg(u"Error: insufficient free space (%sB) for file (%sB)" % (free, self.xfersize)) raise TransferRejectedError() self._msg(u"Receiving file (%s) into: %s" % (naturalsize(self.xfersize), os.path.basename(self.abs_destname))) self._ask_permission() tmp_destname = self.abs_destname + ".tmp" return open(tmp_destname, "wb") def _handle_directory(self, them_d): file_data = them_d["directory"] zipmode = file_data["mode"] if zipmode != "zipfile/deflated": self._msg(u"Error: unknown directory-transfer mode '%s'" % (zipmode,)) raise RespondError("unknown mode") self.abs_destname = self._decide_destname("directory", file_data["dirname"]) self.xfersize = file_data["zipsize"] free = estimate_free_space(self.abs_destname) if free is not None and free < file_data["numbytes"]: self._msg(u"Error: insufficient free space (%sB) for directory (%sB)" % (free, file_data["numbytes"])) raise TransferRejectedError() self._msg(u"Receiving directory (%s) into: %s/" % (naturalsize(self.xfersize), os.path.basename(self.abs_destname))) self._msg(u"%d files, %s (uncompressed)" % (file_data["numfiles"], naturalsize(file_data["numbytes"]))) self._ask_permission() return tempfile.SpooledTemporaryFile() def _decide_destname(self, mode, destname): # the basename() is intended to protect us against # "~/.ssh/authorized_keys" and other attacks destname = os.path.basename(destname) if self.args.output_file: destname = self.args.output_file # override abs_destname = os.path.abspath( os.path.join(self.args.cwd, destname) ) # get confirmation from the user before writing to the local directory if os.path.exists(abs_destname): self._msg(u"Error: refusing to overwrite existing '%s'" % destname) raise TransferRejectedError() return abs_destname def _ask_permission(self): with self.args.timing.add("permission", waiting="user") as t: while True and not self.args.accept_file: ok = six.moves.input("ok? (y/n): ") if ok.lower().startswith("y"): break print(u"transfer rejected", file=sys.stderr) t.detail(answer="no") raise TransferRejectedError() t.detail(answer="yes") def _send_permission(self, w): self._send_data({"answer": { "file_ack": "ok" }}, w) @inlineCallbacks def _establish_transit(self): record_pipe = yield self._transit_receiver.connect() self.args.timing.add("transit connected") returnValue(record_pipe) @inlineCallbacks def _transfer_data(self, record_pipe, f): # now receive the rest of the owl self._msg(u"Receiving (%s).." % record_pipe.describe()) with self.args.timing.add("rx file"): progress = tqdm(file=self.args.stderr, disable=self.args.hide_progress, unit="B", unit_scale=True, total=self.xfersize) hasher = hashlib.sha256() with progress: received = yield record_pipe.writeToFile(f, self.xfersize, progress.update, hasher.update) datahash = hasher.digest() # except TransitError if received < self.xfersize: self._msg() self._msg(u"Connection dropped before full file received") self._msg(u"got %d bytes, wanted %d" % (received, self.xfersize)) raise TransferError("Connection dropped before full file received") assert received == self.xfersize returnValue(datahash) def _write_file(self, f): tmp_name = f.name f.close() os.rename(tmp_name, self.abs_destname) self._msg(u"Received file written to %s" % os.path.basename(self.abs_destname)) def _extract_file(self, zf, info, extract_dir): """ the zipfile module does not restore file permissions so we'll do it manually """ out_path = os.path.join( extract_dir, info.filename ) out_path = os.path.abspath( out_path ) if not out_path.startswith( extract_dir ): raise ValueError( "malicious zipfile, %s outside of extract_dir %s" % (info.filename, extract_dir) ) zf.extract( info.filename, path=extract_dir ) # not sure why zipfiles store the perms 16 bits away but they do perm = info.external_attr >> 16 os.chmod( out_path, perm ) def _write_directory(self, f): self._msg(u"Unpacking zipfile..") with self.args.timing.add("unpack zip"): with zipfile.ZipFile(f, "r", zipfile.ZIP_DEFLATED) as zf: for info in zf.infolist(): self._extract_file( zf, info, self.abs_destname ) self._msg(u"Received files written to %s/" % os.path.basename(self.abs_destname)) f.close() @inlineCallbacks def _close_transit(self, record_pipe, datahash): datahash_hex = bytes_to_hexstr(datahash) ack = {u"ack": u"ok", u"sha256": datahash_hex} ack_bytes = dict_to_bytes(ack) with self.args.timing.add("send ack"): yield record_pipe.send_record(ack_bytes) yield record_pipe.close() magic-wormhole-0.9.1/src/wormhole/cli/cmd_send.py0000644000076600000240000002754713027402475022420 0ustar warnerstaff00000000000000from __future__ import print_function import os, sys, six, tempfile, zipfile, hashlib from tqdm import tqdm from humanize import naturalsize from twisted.python import log from twisted.protocols import basic from twisted.internet import reactor from twisted.internet.defer import inlineCallbacks, returnValue from ..errors import TransferError, WormholeClosedError from ..wormhole import wormhole from ..transit import TransitSender from ..util import dict_to_bytes, bytes_to_dict, bytes_to_hexstr APPID = u"lothar.com/wormhole/text-or-file-xfer" VERIFY_TIMER = 1 def send(args, reactor=reactor): """I implement 'wormhole send'. I return a Deferred that fires with None (for success), or signals one of the following errors: * WrongPasswordError: the two sides didn't use matching passwords * Timeout: something didn't happen fast enough for our tastes * TransferError: the receiver rejected the transfer: verifier mismatch, permission not granted, ack not successful. * any other error: something unexpected happened """ return Sender(args, reactor).go() class Sender: def __init__(self, args, reactor): self._args = args self._reactor = reactor self._tor_manager = None self._timing = args.timing self._fd_to_send = None self._transit_sender = None @inlineCallbacks def go(self): assert isinstance(self._args.relay_url, type(u"")) if self._args.tor: with self._timing.add("import", which="tor_manager"): from ..tor_manager import TorManager self._tor_manager = TorManager(reactor, timing=self._timing) # For now, block everything until Tor has started. Soon: launch # tor in parallel with everything else, make sure the TorManager # can lazy-provide an endpoint, and overlap the startup process # with the user handing off the wormhole code yield self._tor_manager.start() w = wormhole(self._args.appid or APPID, self._args.relay_url, self._reactor, self._tor_manager, timing=self._timing) d = self._go(w) d.addBoth(w.close) # must wait for ack from close() yield d def _send_data(self, data, w): data_bytes = dict_to_bytes(data) w.send(data_bytes) @inlineCallbacks def _go(self, w): # TODO: run the blocking zip-the-directory IO in a thread, let the # wormhole exchange happen in parallel offer, self._fd_to_send = self._build_offer() args = self._args other_cmd = "wormhole receive" if args.verify: other_cmd = "wormhole receive --verify" if args.zeromode: assert not args.code args.code = u"0-" other_cmd += " -0" print(u"On the other computer, please run: %s" % other_cmd, file=args.stderr) if args.code: w.set_code(args.code) code = args.code else: code = yield w.get_code(args.code_length) if not args.zeromode: print(u"Wormhole code is: %s" % code, file=args.stderr) # flush stderr so the code is displayed immediately args.stderr.flush() print(u"", file=args.stderr) yield w.establish_key() def on_slow_connection(): print(u"Key established, waiting for confirmation...", file=args.stderr) notify = self._reactor.callLater(VERIFY_TIMER, on_slow_connection) # TODO: don't stall on w.verify() unless they want it try: verifier_bytes = yield w.verify() # this may raise WrongPasswordError finally: if not notify.called: notify.cancel() if args.verify: verifier = bytes_to_hexstr(verifier_bytes) while True: ok = six.moves.input("Verifier %s. ok? (yes/no): " % verifier) if ok.lower() == "yes": break if ok.lower() == "no": err = "sender rejected verification check, abandoned transfer" reject_data = dict_to_bytes({"error": err}) w.send(reject_data) raise TransferError(err) if self._fd_to_send: ts = TransitSender(args.transit_helper, no_listen=(not args.listen), tor_manager=self._tor_manager, reactor=self._reactor, timing=self._timing) self._transit_sender = ts # for now, send this before the main offer sender_abilities = ts.get_connection_abilities() sender_hints = yield ts.get_connection_hints() sender_transit = {"abilities-v1": sender_abilities, "hints-v1": sender_hints, } self._send_data({u"transit": sender_transit}, w) # TODO: move this down below w.get() transit_key = w.derive_key(APPID+"/transit-key", ts.TRANSIT_KEY_LENGTH) ts.set_transit_key(transit_key) self._send_data({"offer": offer}, w) want_answer = True done = False while True: try: them_d_bytes = yield w.get() except WormholeClosedError: if done: returnValue(None) raise TransferError("unexpected close") # TODO: get() fired, so now it's safe to use w.derive_key() them_d = bytes_to_dict(them_d_bytes) #print("GOT", them_d) recognized = False if u"error" in them_d: raise TransferError("remote error, transfer abandoned: %s" % them_d["error"]) if u"transit" in them_d: recognized = True yield self._handle_transit(them_d[u"transit"]) if u"answer" in them_d: recognized = True if not want_answer: raise TransferError("duplicate answer") yield self._handle_answer(them_d[u"answer"]) done = True returnValue(None) if not recognized: log.msg("unrecognized message %r" % (them_d,)) def _handle_transit(self, receiver_transit): ts = self._transit_sender ts.add_connection_hints(receiver_transit.get("hints-v1", [])) def _build_offer(self): offer = {} args = self._args text = args.text if text == "-": print(u"Reading text message from stdin..", file=args.stderr) text = sys.stdin.read() if not text and not args.what: text = six.moves.input("Text to send: ") if text is not None: print(u"Sending text message (%s)" % naturalsize(len(text)), file=args.stderr) offer = { "message": text } fd_to_send = None return offer, fd_to_send what = os.path.join(args.cwd, args.what) what = what.rstrip(os.sep) if not os.path.exists(what): raise TransferError("Cannot send: no file/directory named '%s'" % args.what) basename = os.path.basename(what) if os.path.isfile(what): # we're sending a file filesize = os.stat(what).st_size offer["file"] = { "filename": basename, "filesize": filesize, } print(u"Sending %s file named '%s'" % (naturalsize(filesize), basename), file=args.stderr) fd_to_send = open(what, "rb") return offer, fd_to_send if os.path.isdir(what): print(u"Building zipfile..", file=args.stderr) # We're sending a directory. Create a zipfile in a tempdir and # send that. fd_to_send = tempfile.SpooledTemporaryFile() num_files = 0 num_bytes = 0 tostrip = len(what.split(os.sep)) with zipfile.ZipFile(fd_to_send, "w", zipfile.ZIP_DEFLATED) as zf: for path,dirs,files in os.walk(what): # path always starts with args.what, then sometimes might # have "/subdir" appended. We want the zipfile to contain # "" or "subdir" localpath = list(path.split(os.sep)[tostrip:]) for fn in files: archivename = os.path.join(*tuple(localpath+[fn])) localfilename = os.path.join(path, fn) zf.write(localfilename, archivename) num_bytes += os.stat(localfilename).st_size num_files += 1 fd_to_send.seek(0,2) filesize = fd_to_send.tell() fd_to_send.seek(0,0) offer["directory"] = { "mode": "zipfile/deflated", "dirname": basename, "zipsize": filesize, "numbytes": num_bytes, "numfiles": num_files, } print(u"Sending directory (%s compressed) named '%s'" % (naturalsize(filesize), basename), file=args.stderr) return offer, fd_to_send raise TypeError("'%s' is neither file nor directory" % args.what) @inlineCallbacks def _handle_answer(self, them_answer): if self._fd_to_send is None: if them_answer["message_ack"] == "ok": print(u"text message sent", file=self._args.stderr) returnValue(None) # terminates this function raise TransferError("error sending text: %r" % (them_answer,)) if them_answer.get("file_ack") != "ok": raise TransferError("ambiguous response from remote, " "transfer abandoned: %s" % (them_answer,)) yield self._send_file() @inlineCallbacks def _send_file(self): ts = self._transit_sender self._fd_to_send.seek(0,2) filesize = self._fd_to_send.tell() self._fd_to_send.seek(0,0) record_pipe = yield ts.connect() self._timing.add("transit connected") # record_pipe should implement IConsumer, chunks are just records stderr = self._args.stderr print(u"Sending (%s).." % record_pipe.describe(), file=stderr) hasher = hashlib.sha256() progress = tqdm(file=stderr, disable=self._args.hide_progress, unit="B", unit_scale=True, total=filesize) def _count_and_hash(data): hasher.update(data) progress.update(len(data)) return data fs = basic.FileSender() with self._timing.add("tx file"): with progress: yield fs.beginFileTransfer(self._fd_to_send, record_pipe, transform=_count_and_hash) expected_hash = hasher.digest() expected_hex = bytes_to_hexstr(expected_hash) print(u"File sent.. waiting for confirmation", file=stderr) with self._timing.add("get ack") as t: ack_bytes = yield record_pipe.receive_record() record_pipe.close() ack = bytes_to_dict(ack_bytes) ok = ack.get(u"ack", u"") if ok != u"ok": t.detail(ack="failed") raise TransferError("Transfer failed (remote says: %r)" % ack) if u"sha256" in ack: if ack[u"sha256"] != expected_hex: t.detail(datahash="failed") raise TransferError("Transfer failed (bad remote hash)") print(u"Confirmation received. Transfer complete.", file=stderr) t.detail(ack="ok") magic-wormhole-0.9.1/src/wormhole/cli/cmd_ssh.py0000644000076600000240000000720713027366362022257 0ustar warnerstaff00000000000000from __future__ import print_function import os from os.path import expanduser, exists, join from twisted.internet.defer import inlineCallbacks from twisted.internet import reactor import click from .. import xfer_util class PubkeyError(Exception): pass def find_public_key(hint=None): """ This looks for an appropriate SSH key to send, possibly querying the user in the meantime. DO NOT CALL after reactor.run as this (possibly) does blocking stuff like asking the user questions (via click.prompt()) Returns a 3-tuple: kind, keyid, pubkey_data """ if hint is None: hint = expanduser('~/.ssh/') else: if not exists(hint): raise PubkeyError("Can't find '{}'".format(hint)) pubkeys = [f for f in os.listdir(hint) if f.endswith('.pub')] if len(pubkeys) == 0: raise PubkeyError("No public keys in '{}'".format(hint)) elif len(pubkeys) > 1: got_key = False while not got_key: ans = click.prompt( "Multiple public-keys found:\n" + \ "\n".join([" {}: {}".format(a, b) for a, b in enumerate(pubkeys)]) + \ "\nSend which one?" ) try: ans = int(ans) if ans < 0 or ans >= len(pubkeys): ans = None else: got_key = True with open(join(hint, pubkeys[ans]), 'r') as f: pubkey = f.read() except Exception: got_key = False else: with open(join(hint, pubkeys[0]), 'r') as f: pubkey = f.read() parts = pubkey.strip().split() kind = parts[0] keyid = 'unknown' if len(parts) <= 2 else parts[2] return kind, keyid, pubkey @inlineCallbacks def accept(cfg, reactor=reactor): yield xfer_util.send( reactor, cfg.appid or u"lothar.com/wormhole/ssh-add", cfg.relay_url, data=cfg.public_key[2], code=cfg.code, use_tor=cfg.tor, ) print("Key sent.") @inlineCallbacks def invite(cfg, reactor=reactor): def on_code_created(code): print("Now tell the other user to run:") print() print("wormhole ssh accept {}".format(code)) print() if cfg.ssh_user is None: ssh_path = expanduser('~/.ssh/'.format(cfg.ssh_user)) else: ssh_path = expanduser('~{}/.ssh/'.format(cfg.ssh_user)) auth_key_path = join(ssh_path, 'authorized_keys') if not exists(auth_key_path): print("Note: '{}' not found; will be created".format(auth_key_path)) if not exists(ssh_path): print(" '{}' doesn't exist either".format(ssh_path)) else: try: open(auth_key_path, 'a').close() except OSError: print("No write permission on '{}'".format(auth_key_path)) return try: os.listdir(ssh_path) except OSError: print("Can't read '{}'".format(ssh_path)) return pubkey = yield xfer_util.receive( reactor, cfg.appid or u"lothar.com/wormhole/ssh-add", cfg.relay_url, None, # allocate a code for us use_tor=cfg.tor, on_code=on_code_created, ) parts = pubkey.split() kind = parts[0] keyid = 'unknown' if len(parts) <= 2 else parts[2] if not exists(auth_key_path): if not exists(ssh_path): os.mkdir(ssh_path, mode=0o700) with open(auth_key_path, 'a', 0o600) as f: f.write('{}\n'.format(pubkey.strip())) print("Appended key type='{kind}' id='{key_id}' to '{auth_file}'".format( kind=kind, key_id=keyid, auth_file=auth_key_path)) magic-wormhole-0.9.1/src/wormhole/cli/public_relay.py0000644000076600000240000000033212722447014023274 0ustar warnerstaff00000000000000 # This is a relay I run on a personal server. If it gets too expensive to # run, I'll shut it down. RENDEZVOUS_RELAY = u"ws://wormhole-relay.lothar.com:4000/v1" TRANSIT_RELAY = u"tcp:wormhole-transit.lothar.com:4001" magic-wormhole-0.9.1/src/wormhole/codes.py0000644000076600000240000001003413024532264021146 0ustar warnerstaff00000000000000from __future__ import print_function, unicode_literals import os, six from .wordlist import (byte_to_even_word, byte_to_odd_word, even_words_lowercase, odd_words_lowercase) def make_code(channel_id, code_length): assert isinstance(channel_id, type("")), type(channel_id) words = [] for i in range(code_length): # we start with an "odd word" if i % 2 == 0: words.append(byte_to_odd_word[os.urandom(1)].lower()) else: words.append(byte_to_even_word[os.urandom(1)].lower()) return "%s-%s" % (channel_id, "-".join(words)) def extract_channel_id(code): channel_id = int(code.split("-")[0]) return channel_id class CodeInputter: def __init__(self, initial_channelids, get_channel_ids, code_length): self._initial_channelids = initial_channelids self._get_channel_ids = get_channel_ids self.code_length = code_length self.last_text = None # memoize for a speedup self.last_matches = None def get_current_channel_ids(self): if self._initial_channelids is not None: channelids = self._initial_channelids self._initial_channelids = None return channelids return self._get_channel_ids() def wrap_completer(self, text, state): try: return self.completer(text, state) except Exception as e: # completer exceptions are normally silently discarded, which # makes debugging challenging print("completer exception: %s" % e) raise e def completer(self, text, state): #if state == 0: # print("", file=sys.stderr) #print("completer: '%s' %d '%d'" % (text, state, # readline.get_completion_type()), # file=sys.stderr) #sys.stderr.flush() pieces = text.split("-") last = pieces[-1].lower() if text == self.last_text and len(pieces) >= 2: # if len(pieces) == 1, skip the cache, so we can re-fetch the # channel_id list matches = self.last_matches #print(" old matches", len(matches), file=sys.stderr) else: if len(pieces) <= 1: channel_ids = self.get_current_channel_ids() matches = [str(channel_id) for channel_id in channel_ids if str(channel_id).startswith(last)] else: if len(pieces) % 2 == 0: words = odd_words_lowercase else: words = even_words_lowercase so_far = "-".join(pieces[:-1]) + "-" matches = sorted([so_far+word for word in words if word.startswith(last)]) self.last_text = text self.last_matches = matches #print(" new matches:", matches, file=sys.stderr) if state >= len(matches): return None match = matches[state] if len(pieces) < 1+self.code_length: match += "-" #print(" match: '%s'" % match, file=sys.stderr) #sys.stderr.flush() return match def input_code_with_completion(prompt, initial_channelids, get_channel_ids, code_length): try: import readline c = CodeInputter(initial_channelids, get_channel_ids, code_length) if "libedit" in readline.__doc__: readline.parse_and_bind("bind ^I rl_complete") else: readline.parse_and_bind("tab: complete") readline.set_completer(c.wrap_completer) readline.set_completer_delims("") except ImportError: pass code = six.moves.input(prompt) # Code is str(bytes) on py2, and str(unicode) on py3. We want unicode. if isinstance(code, bytes): code = code.decode("utf-8") return code if __name__ == "__main__": code = input_code_with_completion("Enter wormhole code: ", [], lambda: [], 2) print("code is:", code) magic-wormhole-0.9.1/src/wormhole/errors.py0000644000076600000240000000327513024532264021376 0ustar warnerstaff00000000000000from __future__ import unicode_literals import functools class ServerError(Exception): def __init__(self, message, relay): self.message = message self.relay = relay def __str__(self): return self.message def handle_server_error(func): @functools.wraps(func) def _wrap(*args, **kwargs): try: return func(*args, **kwargs) except ServerError as e: print("Server error (from %s):\n%s" % (e.relay, e.message)) return 1 return _wrap class Timeout(Exception): pass class WelcomeError(Exception): """ The relay server told us to signal an error, probably because our version is too old to possibly work. The server said:""" pass class WrongPasswordError(Exception): """ Key confirmation failed. Either you or your correspondent typed the code wrong, or a would-be man-in-the-middle attacker guessed incorrectly. You could try again, giving both your correspondent and the attacker another chance. """ # or the data blob was corrupted, and that's why decrypt failed pass class KeyFormatError(Exception): """ The key you entered contains spaces. Magic-wormhole expects keys to be separated by dashes. Please reenter the key you were given separating the words with dashes. """ class ReflectionAttack(Exception): """An attacker (or bug) reflected our outgoing message back to us.""" class InternalError(Exception): """The programmer did something wrong.""" class WormholeClosedError(InternalError): """API calls may not be made after close() is called.""" class TransferError(Exception): """Something bad happened and the transfer failed.""" magic-wormhole-0.9.1/src/wormhole/ipaddrs.py0000644000076600000240000000570313024532264021506 0ustar warnerstaff00000000000000# no unicode_literals # Find all of our ip addresses. From tahoe's src/allmydata/util/iputil.py import os, re, subprocess, errno from sys import platform from twisted.python.procutils import which # Wow, I'm really amazed at home much mileage we've gotten out of calling # the external route.exe program on windows... It appears to work on all # versions so far. Still, the real system calls would much be preferred... # ... thus wrote Greg Smith in time immemorial... _win32_re = re.compile(r'^\s*\d+\.\d+\.\d+\.\d+\s.+\s(?P
\d+\.\d+\.\d+\.\d+)\s+(?P\d+)\s*$', flags=re.M|re.I|re.S) _win32_commands = (('route.exe', ('print',), _win32_re),) # These work in most Unices. _addr_re = re.compile(r'^\s*inet [a-zA-Z]*:?(?P
\d+\.\d+\.\d+\.\d+)[\s/].+$', flags=re.M|re.I|re.S) _unix_commands = (('/bin/ip', ('addr',), _addr_re), ('/sbin/ip', ('addr',), _addr_re), ('/sbin/ifconfig', ('-a',), _addr_re), ('/usr/sbin/ifconfig', ('-a',), _addr_re), ('/usr/etc/ifconfig', ('-a',), _addr_re), ('ifconfig', ('-a',), _addr_re), ('/sbin/ifconfig', (), _addr_re), ) def find_addresses(): # originally by Greg Smith, hacked by Zooko and then Daira # We don't reach here for cygwin. if platform == 'win32': commands = _win32_commands else: commands = _unix_commands for (pathtotool, args, regex) in commands: # If pathtotool is a fully qualified path then we just try that. # If it is merely an executable name then we use Twisted's # "which()" utility and try each executable in turn until one # gives us something that resembles a dotted-quad IPv4 address. if os.path.isabs(pathtotool): exes_to_try = [pathtotool] else: exes_to_try = which(pathtotool) for exe in exes_to_try: try: addresses = _query(exe, args, regex) except Exception: addresses = [] if addresses: return addresses return ["127.0.0.1"] def _query(path, args, regex): env = {'LANG': 'en_US.UTF-8'} TRIES = 5 for trial in range(TRIES): try: p = subprocess.Popen([path] + list(args), stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env, universal_newlines=True) (output, err) = p.communicate() break except OSError as e: if e.errno == errno.EINTR and trial < TRIES-1: continue raise addresses = [] outputsplit = output.split('\n') for outline in outputsplit: m = regex.match(outline) if m: addr = m.group('address') if addr not in addresses: addresses.append(addr) return addresses magic-wormhole-0.9.1/src/wormhole/server/0000755000076600000240000000000013032243761021007 5ustar warnerstaff00000000000000magic-wormhole-0.9.1/src/wormhole/server/__init__.py0000644000076600000240000000000012706034367023115 0ustar warnerstaff00000000000000magic-wormhole-0.9.1/src/wormhole/server/cli.py0000644000076600000240000000766613024532264022147 0ustar warnerstaff00000000000000from __future__ import print_function import click # can put this back in to get this command as "wormhole server" # instead #from ..cli.cli import wormhole #@wormhole.group() @click.group() @click.pass_context def server(ctx): """ Control a relay server (most users shouldn't need to worry about this and can use the default server). """ # just leaving this pointing to wormhole.cli.cli.Config for now, # but if we want to keep wormhole-server as a separate command # should probably have our own Config without all the options the # server commands don't use from ..cli.cli import Config ctx.obj = Config() @server.command() @click.option( "--rendezvous", default="tcp:4000", metavar="tcp:PORT", help="endpoint specification for the rendezvous port", ) @click.option( "--transit", default="tcp:4001", metavar="tcp:PORT", help="endpoint specification for the transit-relay port", ) @click.option( "--advertise-version", metavar="VERSION", help="version to recommend to clients", ) @click.option( "--blur-usage", default=None, type=int, metavar="SECONDS", help="round logged access times to improve privacy", ) @click.option( "--no-daemon", "-n", is_flag=True, help="Run in the foreground", ) @click.option( "--signal-error", is_flag=True, help="force all clients to fail with a message", ) @click.pass_obj def start(cfg, signal_error, no_daemon, blur_usage, advertise_version, transit, rendezvous): """ Start a relay server """ from wormhole.server.cmd_server import start_server cfg.no_daemon = no_daemon cfg.blur_usage = blur_usage cfg.advertise_version = advertise_version cfg.transit = str(transit) cfg.rendezvous = str(rendezvous) cfg.signal_error = signal_error start_server(cfg) # XXX it would be nice to reduce the duplication between 'restart' and # 'start' options... @server.command() @click.option( "--rendezvous", default="tcp:4000", metavar="tcp:PORT", help="endpoint specification for the rendezvous port", ) @click.option( "--transit", default="tcp:4001", metavar="tcp:PORT", help="endpoint specification for the transit-relay port", ) @click.option( "--advertise-version", metavar="VERSION", help="version to recommend to clients", ) @click.option( "--blur-usage", default=None, type=int, metavar="SECONDS", help="round logged access times to improve privacy", ) @click.option( "--no-daemon", "-n", is_flag=True, help="Run in the foreground", ) @click.option( "--signal-error", is_flag=True, help="force all clients to fail with a message", ) @click.pass_obj def restart(cfg, signal_error, no_daemon, blur_usage, advertise_version, transit, rendezvous): """ Re-start a relay server """ from wormhole.server.cmd_server import restart_server cfg.no_daemon = no_daemon cfg.blur_usage = blur_usage cfg.advertise_version = advertise_version cfg.transit = str(transit) cfg.rendezvous = str(rendezvous) cfg.signal_error = signal_error restart_server(cfg) @server.command() @click.pass_obj def stop(cfg): """ Stop a relay server """ from wormhole.server.cmd_server import stop_server stop_server(cfg) @server.command(name="tail-usage") @click.pass_obj def tail_usage(cfg): """ Follow the latest usage """ from wormhole.server.cmd_usage import tail_usage tail_usage(cfg) @server.command(name='count-channels') @click.option( "--json", is_flag=True, ) @click.pass_obj def count_channels(cfg, json): """ Count active channels """ from wormhole.server.cmd_usage import count_channels cfg.json = json count_channels(cfg) @server.command(name='count-events') @click.option( "--json", is_flag=True, ) @click.pass_obj def count_events(cfg, json): """ Count events """ from wormhole.server.cmd_usage import count_events cfg.json = json count_events(cfg) magic-wormhole-0.9.1/src/wormhole/server/cmd_server.py0000644000076600000240000000410413024532264023511 0ustar warnerstaff00000000000000from __future__ import print_function, unicode_literals import os, time from twisted.python import usage from twisted.scripts import twistd class MyPlugin: tapname = "xyznode" def __init__(self, args): self.args = args def makeService(self, so): # delay this import as late as possible, to allow twistd's code to # accept --reactor= selection from .server import RelayServer return RelayServer(self.args.rendezvous, self.args.transit, self.args.advertise_version, "relay.sqlite", self.args.blur_usage, signal_error=self.args.signal_error, stats_file="stats.json", ) class MyTwistdConfig(twistd.ServerOptions): subCommands = [("XYZ", None, usage.Options, "node")] def start_server(args): c = MyTwistdConfig() #twistd_args = tuple(args.twistd_args) + ("XYZ",) base_args = [] if args.no_daemon: base_args.append("--nodaemon") twistd_args = base_args + ["XYZ"] c.parseOptions(tuple(twistd_args)) c.loadedPlugins = {"XYZ": MyPlugin(args)} print("starting wormhole relay server") # this forks and never comes back. The parent calls os._exit(0) twistd.runApp(c) def kill_server(): try: f = open("twistd.pid", "r") except EnvironmentError: print("Unable to find twistd.pid: is this really a server directory?") print("oh well, ignoring 'stop'") return pid = int(f.read().strip()) f.close() os.kill(pid, 15) print("server process %d sent SIGTERM" % pid) return def stop_server(args): kill_server() def restart_server(args): kill_server() time.sleep(0.1) timeout = 0 while os.path.exists("twistd.pid") and timeout < 10: if timeout == 0: print(" waiting for shutdown..") timeout += 1 time.sleep(1) if os.path.exists("twistd.pid"): print("error: unable to shut down old server") return 1 print(" old server shut down") start_server(args) magic-wormhole-0.9.1/src/wormhole/server/cmd_usage.py0000644000076600000240000002171613024532264023317 0ustar warnerstaff00000000000000from __future__ import print_function, unicode_literals import os, time, json from collections import defaultdict import click from humanize import naturalsize from .database import get_db def abbrev(t): if t is None: return "-" if t > 1.0: return "%.3fs" % t if t > 1e-3: return "%.1fms" % (t*1e3) return "%.1fus" % (t*1e6) def print_event(event): event_type, started, result, total_bytes, waiting_time, total_time = event followthrough = None if waiting_time and total_time: followthrough = total_time - waiting_time print("%17s: total=%7s wait=%7s ft=%7s size=%s (%s)" % ("%s-%s" % (event_type, result), abbrev(total_time), abbrev(waiting_time), abbrev(followthrough), naturalsize(total_bytes), time.ctime(started), )) def show_usage(args): print("closed for renovation") return 0 if not os.path.exists("relay.sqlite"): raise click.UsageError( "cannot find relay.sqlite, please run from the server directory" ) oldest = None newest = None rendezvous_counters = defaultdict(int) transit_counters = defaultdict(int) total_transit_bytes = 0 db = get_db("relay.sqlite") c = db.execute("SELECT * FROM `usage`" " ORDER BY `started` ASC LIMIT ?", (args.n,)) for row in c.fetchall(): if row["type"] == "rendezvous": counters = rendezvous_counters elif row["type"] == "transit": counters = transit_counters total_transit_bytes += row["total_bytes"] else: continue counters["total"] += 1 counters[row["result"]] += 1 if oldest is None or row["started"] < oldest: oldest = row["started"] if newest is None or row["started"] > newest: newest = row["started"] event = (row["type"], row["started"], row["result"], row["total_bytes"], row["waiting_time"], row["total_time"]) print_event(event) if rendezvous_counters["total"] or transit_counters["total"]: print("---") print("(most recent started %s ago)" % abbrev(time.time() - newest)) if rendezvous_counters["total"]: print("rendezvous events:") counters = rendezvous_counters elapsed = time.time() - oldest total = counters["total"] print(" %d events in %s (%.2f per hour)" % (total, abbrev(elapsed), (3600 * total / elapsed))) print("", ", ".join(["%s=%d (%d%%)" % (k, counters[k], (100.0 * counters[k] / total)) for k in sorted(counters) if k != "total"])) if transit_counters["total"]: print("transit events:") counters = transit_counters elapsed = time.time() - oldest total = counters["total"] print(" %d events in %s (%.2f per hour)" % (total, abbrev(elapsed), (3600 * total / elapsed))) rate = total_transit_bytes / elapsed print(" %s total bytes, %sps" % (naturalsize(total_transit_bytes), naturalsize(rate))) print("", ", ".join(["%s=%d (%d%%)" % (k, counters[k], (100.0 * counters[k] / total)) for k in sorted(counters) if k != "total"])) return 0 def tail_usage(args): if not os.path.exists("relay.sqlite"): raise click.UsageError( "cannot find relay.sqlite, please run from the server directory" ) db = get_db("relay.sqlite") # we don't seem to have unique row IDs, so this is an inaccurate and # inefficient hack seen = set() try: while True: old = time.time() - 2*60*60 c = db.execute("SELECT * FROM `usage`" " WHERE `started` > ?" " ORDER BY `started` ASC", (old,)) for row in c.fetchall(): event = (row["type"], row["started"], row["result"], row["total_bytes"], row["waiting_time"], row["total_time"]) if event not in seen: print_event(event) seen.add(event) time.sleep(2) except KeyboardInterrupt: return 0 return 0 def count_channels(args): if not os.path.exists("relay.sqlite"): raise click.UsageError( "cannot find relay.sqlite, please run from the server directory" ) db = get_db("relay.sqlite") c_list = [] c_dict = {} def add(key, value): c_list.append((key, value)) c_dict[key] = value OLD = time.time() - 10*60 def q(query, values=()): return list(db.execute(query, values).fetchone().values())[0] add("apps", q("SELECT COUNT(DISTINCT(`app_id`)) FROM `nameplates`")) add("total nameplates", q("SELECT COUNT() FROM `nameplates`")) add("waiting nameplates", q("SELECT COUNT() FROM `nameplates`" " WHERE `second` is null")) add("connected nameplates", q("SELECT COUNT() FROM `nameplates`" " WHERE `second` is not null")) add("stale nameplates", q("SELECT COUNT() FROM `nameplates`" " where `updated` < ?", (OLD,))) add("total mailboxes", q("SELECT COUNT() FROM `mailboxes`")) add("waiting mailboxes", q("SELECT COUNT() FROM `mailboxes`" " WHERE `second` is null")) add("connected mailboxes", q("SELECT COUNT() FROM `mailboxes`" " WHERE `second` is not null")) stale_mailboxes = 0 for mbox_row in db.execute("SELECT * FROM `mailboxes`").fetchall(): newest = db.execute("SELECT `server_rx` FROM `messages`" " WHERE `app_id`=? AND `mailbox_id`=?" " ORDER BY `server_rx` DESC LIMIT 1", (mbox_row["app_id"], mbox_row["id"])).fetchone() if newest and newest[0] < OLD: stale_mailboxes += 1 add("stale mailboxes", stale_mailboxes) add("messages", q("SELECT COUNT() FROM `messages`")) if args.json: print(json.dumps(c_dict)) else: for (key, value) in c_list: print(key, value) return 0 def count_events(args): if not os.path.exists("relay.sqlite"): raise click.UsageError( "cannot find relay.sqlite, please run from the server directory" ) db = get_db("relay.sqlite") c_list = [] c_dict = {} def add(key, value): c_list.append((key, value)) c_dict[key] = value def q(query, values=()): return list(db.execute(query, values).fetchone().values())[0] add("apps", q("SELECT COUNT(DISTINCT(`app_id`)) FROM `nameplate_usage`")) add("total nameplates", q("SELECT COUNT() FROM `nameplate_usage`")) add("happy nameplates", q("SELECT COUNT() FROM `nameplate_usage`" " WHERE `result`='happy'")) add("lonely nameplates", q("SELECT COUNT() FROM `nameplate_usage`" " WHERE `result`='lonely'")) add("pruney nameplates", q("SELECT COUNT() FROM `nameplate_usage`" " WHERE `result`='pruney'")) add("crowded nameplates", q("SELECT COUNT() FROM `nameplate_usage`" " WHERE `result`='crowded'")) add("total mailboxes", q("SELECT COUNT() FROM `mailbox_usage`")) add("happy mailboxes", q("SELECT COUNT() FROM `mailbox_usage`" " WHERE `result`='happy'")) add("scary mailboxes", q("SELECT COUNT() FROM `mailbox_usage`" " WHERE `result`='scary'")) add("lonely mailboxes", q("SELECT COUNT() FROM `mailbox_usage`" " WHERE `result`='lonely'")) add("errory mailboxes", q("SELECT COUNT() FROM `mailbox_usage`" " WHERE `result`='errory'")) add("pruney mailboxes", q("SELECT COUNT() FROM `mailbox_usage`" " WHERE `result`='pruney'")) add("crowded mailboxes", q("SELECT COUNT() FROM `mailbox_usage`" " WHERE `result`='crowded'")) add("total transit", q("SELECT COUNT() FROM `transit_usage`")) add("happy transit", q("SELECT COUNT() FROM `transit_usage`" " WHERE `result`='happy'")) add("lonely transit", q("SELECT COUNT() FROM `transit_usage`" " WHERE `result`='lonely'")) add("errory transit", q("SELECT COUNT() FROM `transit_usage`" " WHERE `result`='errory'")) add("transit bytes", q("SELECT SUM(`total_bytes`) FROM `transit_usage`")) if args.json: print(json.dumps(c_dict)) else: for (key, value) in c_list: print(key, value) return 0 magic-wormhole-0.9.1/src/wormhole/server/database.py0000644000076600000240000000563213027366340023136 0ustar warnerstaff00000000000000from __future__ import unicode_literals import os import sqlite3 from pkg_resources import resource_string from twisted.python import log class DBError(Exception): pass def get_schema(version): schema_bytes = resource_string("wormhole.server", "db-schemas/v%d.sql" % version) return schema_bytes.decode("utf-8") def get_upgrader(new_version): schema_bytes = resource_string("wormhole.server", "db-schemas/upgrade-to-v%d.sql" % new_version) return schema_bytes.decode("utf-8") TARGET_VERSION = 3 def dict_factory(cursor, row): d = {} for idx, col in enumerate(cursor.description): d[col[0]] = row[idx] return d def get_db(dbfile, target_version=TARGET_VERSION): """Open or create the given db file. The parent directory must exist. Returns the db connection object, or raises DBError. """ must_create = (dbfile == ":memory:") or not os.path.exists(dbfile) try: db = sqlite3.connect(dbfile) except (EnvironmentError, sqlite3.OperationalError) as e: raise DBError("Unable to create/open db file %s: %s" % (dbfile, e)) db.row_factory = dict_factory db.execute("PRAGMA foreign_keys = ON") problems = db.execute("PRAGMA foreign_key_check").fetchall() if problems: raise DBError("failed foreign key check: %s" % (problems,)) if must_create: log.msg("populating new database with schema v%s" % target_version) schema = get_schema(target_version) db.executescript(schema) db.execute("INSERT INTO version (version) VALUES (?)", (target_version,)) db.commit() try: version = db.execute("SELECT version FROM version").fetchone()["version"] except sqlite3.DatabaseError as e: # this indicates that the file is not a compatible database format. # Perhaps it was created with an old version, or it might be junk. raise DBError("db file is unusable: %s" % e) while version < target_version: log.msg(" need to upgrade from %s to %s" % (version, target_version)) try: upgrader = get_upgrader(version+1) except ValueError: # ResourceError?? log.msg(" unable to upgrade %s to %s" % (version, version+1)) raise DBError("Unable to upgrade %s to version %s, left at %s" % (dbfile, version+1, version)) log.msg(" executing upgrader v%s->v%s" % (version, version+1)) db.executescript(upgrader) db.commit() version = version+1 if version != target_version: raise DBError("Unable to handle db version %s" % version) return db def dump_db(db): # to let _iterdump work, we need to restore the original row factory orig = db.row_factory try: db.row_factory = sqlite3.Row return "".join(db.iterdump()) finally: db.row_factory = orig magic-wormhole-0.9.1/src/wormhole/server/db-schemas/0000755000076600000240000000000013032243761023015 5ustar warnerstaff00000000000000magic-wormhole-0.9.1/src/wormhole/server/db-schemas/upgrade-to-v3.sql0000644000076600000240000000424713024532264026142 0ustar warnerstaff00000000000000DROP TABLE `nameplates`; DROP TABLE `messages`; DROP TABLE `mailboxes`; -- Wormhole codes use a "nameplate": a short name which is only used to -- reference a specific (long-named) mailbox. The codes only use numeric -- nameplates, but the protocol and server allow can use arbitrary strings. CREATE TABLE `nameplates` ( `id` INTEGER PRIMARY KEY AUTOINCREMENT, `app_id` VARCHAR, `name` VARCHAR, `mailbox_id` VARCHAR REFERENCES `mailboxes`(`id`), `request_id` VARCHAR -- from 'allocate' message, for future deduplication ); CREATE INDEX `nameplates_idx` ON `nameplates` (`app_id`, `name`); CREATE INDEX `nameplates_mailbox_idx` ON `nameplates` (`app_id`, `mailbox_id`); CREATE INDEX `nameplates_request_idx` ON `nameplates` (`app_id`, `request_id`); CREATE TABLE `nameplate_sides` ( `nameplates_id` REFERENCES `nameplates`(`id`), `claimed` BOOLEAN, -- True after claim(), False after release() `side` VARCHAR, `added` INTEGER -- time when this side first claimed the nameplate ); -- Clients exchange messages through a "mailbox", which has a long (randomly -- unique) identifier and a queue of messages. -- `id` is randomly-generated and unique across all apps. CREATE TABLE `mailboxes` ( `app_id` VARCHAR, `id` VARCHAR PRIMARY KEY, `updated` INTEGER, -- time of last activity, used for pruning `for_nameplate` BOOLEAN -- allocated for a nameplate, not standalone ); CREATE INDEX `mailboxes_idx` ON `mailboxes` (`app_id`, `id`); CREATE TABLE `mailbox_sides` ( `mailbox_id` REFERENCES `mailboxes`(`id`), `opened` BOOLEAN, -- True after open(), False after close() `side` VARCHAR, `added` INTEGER, -- time when this side first opened the mailbox `mood` VARCHAR ); CREATE TABLE `messages` ( `app_id` VARCHAR, `mailbox_id` VARCHAR, `side` VARCHAR, `phase` VARCHAR, -- numeric or string `body` VARCHAR, `server_rx` INTEGER, `msg_id` VARCHAR ); CREATE INDEX `messages_idx` ON `messages` (`app_id`, `mailbox_id`); ALTER TABLE `mailbox_usage` ADD COLUMN `for_nameplate` BOOLEAN; CREATE INDEX `mailbox_usage_result_idx` ON `mailbox_usage` (`result`); CREATE INDEX `transit_usage_result_idx` ON `transit_usage` (`result`); DELETE FROM `version`; INSERT INTO `version` (`version`) VALUES (3); magic-wormhole-0.9.1/src/wormhole/server/db-schemas/v2.sql0000644000076600000240000000777212732104714024102 0ustar warnerstaff00000000000000 -- note: anything which isn't an boolean, integer, or human-readable unicode -- string, (i.e. binary strings) will be stored as hex CREATE TABLE `version` ( `version` INTEGER -- contains one row, set to 2 ); -- Wormhole codes use a "nameplate": a short identifier which is only used to -- reference a specific (long-named) mailbox. The codes only use numeric -- nameplates, but the protocol and server allow can use arbitrary strings. CREATE TABLE `nameplates` ( `app_id` VARCHAR, `id` VARCHAR, `mailbox_id` VARCHAR, -- really a foreign key `side1` VARCHAR, -- side name, or NULL `side2` VARCHAR, -- side name, or NULL `request_id` VARCHAR, -- from 'allocate' message, for future deduplication `crowded` BOOLEAN, -- at some point, three or more sides were involved `updated` INTEGER, -- time of last activity, used for pruning -- timing data `started` INTEGER, -- time when nameplace was opened `second` INTEGER -- time when second side opened ); CREATE INDEX `nameplates_idx` ON `nameplates` (`app_id`, `id`); CREATE INDEX `nameplates_updated_idx` ON `nameplates` (`app_id`, `updated`); CREATE INDEX `nameplates_mailbox_idx` ON `nameplates` (`app_id`, `mailbox_id`); CREATE INDEX `nameplates_request_idx` ON `nameplates` (`app_id`, `request_id`); -- Clients exchange messages through a "mailbox", which has a long (randomly -- unique) identifier and a queue of messages. CREATE TABLE `mailboxes` ( `app_id` VARCHAR, `id` VARCHAR, `side1` VARCHAR, -- side name, or NULL `side2` VARCHAR, -- side name, or NULL `crowded` BOOLEAN, -- at some point, three or more sides were involved `first_mood` VARCHAR, -- timing data for the mailbox itself `started` INTEGER, -- time when opened `second` INTEGER -- time when second side opened ); CREATE INDEX `mailboxes_idx` ON `mailboxes` (`app_id`, `id`); CREATE TABLE `messages` ( `app_id` VARCHAR, `mailbox_id` VARCHAR, `side` VARCHAR, `phase` VARCHAR, -- numeric or string `body` VARCHAR, `server_rx` INTEGER, `msg_id` VARCHAR ); CREATE INDEX `messages_idx` ON `messages` (`app_id`, `mailbox_id`); CREATE TABLE `nameplate_usage` ( `app_id` VARCHAR, `started` INTEGER, -- seconds since epoch, rounded to "blur time" `waiting_time` INTEGER, -- seconds from start to 2nd side appearing, or None `total_time` INTEGER, -- seconds from open to last close/prune `result` VARCHAR -- happy, lonely, pruney, crowded -- nameplate moods: -- "happy": two sides open and close -- "lonely": one side opens and closes (no response from 2nd side) -- "pruney": channels which get pruned for inactivity -- "crowded": three or more sides were involved ); CREATE INDEX `nameplate_usage_idx` ON `nameplate_usage` (`app_id`, `started`); CREATE TABLE `mailbox_usage` ( `app_id` VARCHAR, `started` INTEGER, -- seconds since epoch, rounded to "blur time" `total_time` INTEGER, -- seconds from open to last close `waiting_time` INTEGER, -- seconds from start to 2nd side appearing, or None `result` VARCHAR -- happy, scary, lonely, errory, pruney -- rendezvous moods: -- "happy": both sides close with mood=happy -- "scary": any side closes with mood=scary (bad MAC, probably wrong pw) -- "lonely": any side closes with mood=lonely (no response from 2nd side) -- "errory": any side closes with mood=errory (other errors) -- "pruney": channels which get pruned for inactivity -- "crowded": three or more sides were involved ); CREATE INDEX `mailbox_usage_idx` ON `mailbox_usage` (`app_id`, `started`); CREATE TABLE `transit_usage` ( `started` INTEGER, -- seconds since epoch, rounded to "blur time" `total_time` INTEGER, -- seconds from open to last close `waiting_time` INTEGER, -- seconds from start to 2nd side appearing, or None `total_bytes` INTEGER, -- total bytes relayed (both directions) `result` VARCHAR -- happy, scary, lonely, errory, pruney -- transit moods: -- "errory": one side gave the wrong handshake -- "lonely": good handshake, but the other side never showed up -- "happy": both sides gave correct handshake ); CREATE INDEX `transit_usage_idx` ON `transit_usage` (`started`); magic-wormhole-0.9.1/src/wormhole/server/db-schemas/v3.sql0000644000076600000240000001032413024532264024066 0ustar warnerstaff00000000000000 -- note: anything which isn't an boolean, integer, or human-readable unicode -- string, (i.e. binary strings) will be stored as hex CREATE TABLE `version` ( `version` INTEGER -- contains one row, set to 3 ); -- Wormhole codes use a "nameplate": a short name which is only used to -- reference a specific (long-named) mailbox. The codes only use numeric -- nameplates, but the protocol and server allow can use arbitrary strings. CREATE TABLE `nameplates` ( `id` INTEGER PRIMARY KEY AUTOINCREMENT, `app_id` VARCHAR, `name` VARCHAR, `mailbox_id` VARCHAR REFERENCES `mailboxes`(`id`), `request_id` VARCHAR -- from 'allocate' message, for future deduplication ); CREATE INDEX `nameplates_idx` ON `nameplates` (`app_id`, `name`); CREATE INDEX `nameplates_mailbox_idx` ON `nameplates` (`app_id`, `mailbox_id`); CREATE INDEX `nameplates_request_idx` ON `nameplates` (`app_id`, `request_id`); CREATE TABLE `nameplate_sides` ( `nameplates_id` REFERENCES `nameplates`(`id`), `claimed` BOOLEAN, -- True after claim(), False after release() `side` VARCHAR, `added` INTEGER -- time when this side first claimed the nameplate ); -- Clients exchange messages through a "mailbox", which has a long (randomly -- unique) identifier and a queue of messages. -- `id` is randomly-generated and unique across all apps. CREATE TABLE `mailboxes` ( `app_id` VARCHAR, `id` VARCHAR PRIMARY KEY, `updated` INTEGER, -- time of last activity, used for pruning `for_nameplate` BOOLEAN -- allocated for a nameplate, not standalone ); CREATE INDEX `mailboxes_idx` ON `mailboxes` (`app_id`, `id`); CREATE TABLE `mailbox_sides` ( `mailbox_id` REFERENCES `mailboxes`(`id`), `opened` BOOLEAN, -- True after open(), False after close() `side` VARCHAR, `added` INTEGER, -- time when this side first opened the mailbox `mood` VARCHAR ); CREATE TABLE `messages` ( `app_id` VARCHAR, `mailbox_id` VARCHAR, `side` VARCHAR, `phase` VARCHAR, -- numeric or string `body` VARCHAR, `server_rx` INTEGER, `msg_id` VARCHAR ); CREATE INDEX `messages_idx` ON `messages` (`app_id`, `mailbox_id`); CREATE TABLE `nameplate_usage` ( `app_id` VARCHAR, `started` INTEGER, -- seconds since epoch, rounded to "blur time" `waiting_time` INTEGER, -- seconds from start to 2nd side appearing, or None `total_time` INTEGER, -- seconds from open to last close/prune `result` VARCHAR -- happy, lonely, pruney, crowded -- nameplate moods: -- "happy": two sides open and close -- "lonely": one side opens and closes (no response from 2nd side) -- "pruney": channels which get pruned for inactivity -- "crowded": three or more sides were involved ); CREATE INDEX `nameplate_usage_idx` ON `nameplate_usage` (`app_id`, `started`); CREATE TABLE `mailbox_usage` ( `app_id` VARCHAR, `for_nameplate` BOOLEAN, -- allocated for a nameplate, not standalone `started` INTEGER, -- seconds since epoch, rounded to "blur time" `total_time` INTEGER, -- seconds from open to last close `waiting_time` INTEGER, -- seconds from start to 2nd side appearing, or None `result` VARCHAR -- happy, scary, lonely, errory, pruney -- rendezvous moods: -- "happy": both sides close with mood=happy -- "scary": any side closes with mood=scary (bad MAC, probably wrong pw) -- "lonely": any side closes with mood=lonely (no response from 2nd side) -- "errory": any side closes with mood=errory (other errors) -- "pruney": channels which get pruned for inactivity -- "crowded": three or more sides were involved ); CREATE INDEX `mailbox_usage_idx` ON `mailbox_usage` (`app_id`, `started`); CREATE INDEX `mailbox_usage_result_idx` ON `mailbox_usage` (`result`); CREATE TABLE `transit_usage` ( `started` INTEGER, -- seconds since epoch, rounded to "blur time" `total_time` INTEGER, -- seconds from open to last close `waiting_time` INTEGER, -- seconds from start to 2nd side appearing, or None `total_bytes` INTEGER, -- total bytes relayed (both directions) `result` VARCHAR -- happy, scary, lonely, errory, pruney -- transit moods: -- "errory": one side gave the wrong handshake -- "lonely": good handshake, but the other side never showed up -- "happy": both sides gave correct handshake ); CREATE INDEX `transit_usage_idx` ON `transit_usage` (`started`); CREATE INDEX `transit_usage_result_idx` ON `transit_usage` (`result`); magic-wormhole-0.9.1/src/wormhole/server/rendezvous.py0000644000076600000240000006733013031527043023573 0ustar warnerstaff00000000000000from __future__ import print_function, unicode_literals import os, random, base64, collections from collections import namedtuple from twisted.python import log from twisted.application import service def generate_mailbox_id(): return base64.b32encode(os.urandom(8)).lower().strip(b"=").decode("ascii") class CrowdedError(Exception): pass class ReclaimedError(Exception): pass Usage = namedtuple("Usage", ["started", "waiting_time", "total_time", "result"]) TransitUsage = namedtuple("TransitUsage", ["started", "waiting_time", "total_time", "total_bytes", "result"]) SidedMessage = namedtuple("SidedMessage", ["side", "phase", "body", "server_rx", "msg_id"]) class Mailbox: def __init__(self, app, db, app_id, mailbox_id): self._app = app self._db = db self._app_id = app_id self._mailbox_id = mailbox_id self._listeners = {} # handle -> (send_f, stop_f) # "handle" is a hashable object, for deregistration # send_f() takes a JSONable object, stop_f() has no args def open(self, side, when): # requires caller to db.commit() assert isinstance(side, type("")), type(side) db = self._db already = db.execute("SELECT * FROM `mailbox_sides`" " WHERE `mailbox_id`=? AND `side`=?", (self._mailbox_id, side)).fetchone() if not already: db.execute("INSERT INTO `mailbox_sides`" " (`mailbox_id`, `opened`, `side`, `added`)" " VALUES(?,?,?,?)", (self._mailbox_id, True, side, when)) # We accept re-opening a mailbox which a side previously closed, # unlike claim_nameplate(), which forbids any side from re-claiming a # nameplate which they previously released. (Nameplates forbid this # because the act of claiming a nameplate for the first time causes a # new mailbox to be created, which should only happen once). # Mailboxes have their own distinct objects (to manage # subscriptions), so closing one which was already closed requires # making a new object, which works by calling open() just before # close(). We really do want to support re-closing closed mailboxes, # because this enables intermittently-connected clients, who remember # sending a 'close' but aren't sure whether it was received or not, # then get shut down. Those clients will wake up and re-send the # 'close', until they receive the 'closed' ack message. self._touch(when) db.commit() # XXX: reconcile the need for this with the comment above def _touch(self, when): self._db.execute("UPDATE `mailboxes` SET `updated`=? WHERE `id`=?", (when, self._mailbox_id)) def get_messages(self): messages = [] db = self._db for row in db.execute("SELECT * FROM `messages`" " WHERE `app_id`=? AND `mailbox_id`=?" " ORDER BY `server_rx` ASC", (self._app_id, self._mailbox_id)).fetchall(): sm = SidedMessage(side=row["side"], phase=row["phase"], body=row["body"], server_rx=row["server_rx"], msg_id=row["msg_id"]) messages.append(sm) return messages def add_listener(self, handle, send_f, stop_f): #log.msg("add_listener", self._mailbox_id, handle) self._listeners[handle] = (send_f, stop_f) #log.msg(" added", len(self._listeners)) return self.get_messages() def remove_listener(self, handle): #log.msg("remove_listener", self._mailbox_id, handle) self._listeners.pop(handle, None) #log.msg(" removed", len(self._listeners)) def has_listeners(self): return bool(self._listeners) def broadcast_message(self, sm): for (send_f, stop_f) in self._listeners.values(): send_f(sm) def _add_message(self, sm): self._db.execute("INSERT INTO `messages`" " (`app_id`, `mailbox_id`, `side`, `phase`, `body`," " `server_rx`, `msg_id`)" " VALUES (?,?,?,?,?, ?,?)", (self._app_id, self._mailbox_id, sm.side, sm.phase, sm.body, sm.server_rx, sm.msg_id)) self._touch(sm.server_rx) self._db.commit() def add_message(self, sm): assert isinstance(sm, SidedMessage) self._add_message(sm) self.broadcast_message(sm) def close(self, side, mood, when): assert isinstance(side, type("")), type(side) db = self._db row = db.execute("SELECT * FROM `mailboxes`" " WHERE `app_id`=? AND `id`=?", (self._app_id, self._mailbox_id)).fetchone() if not row: return for_nameplate = row["for_nameplate"] row = db.execute("SELECT * FROM `mailbox_sides`" " WHERE `mailbox_id`=? AND `side`=?", (self._mailbox_id, side)).fetchone() if not row: return db.execute("UPDATE `mailbox_sides` SET `opened`=?, `mood`=?" " WHERE `mailbox_id`=? AND `side`=?", (False, mood, self._mailbox_id, side)) db.commit() # are any sides still open? side_rows = db.execute("SELECT * FROM `mailbox_sides`" " WHERE `mailbox_id`=?", (self._mailbox_id,)).fetchall() if any([sr["opened"] for sr in side_rows]): return # nope. delete and summarize db.execute("DELETE FROM `messages` WHERE `mailbox_id`=?", (self._mailbox_id,)) db.execute("DELETE FROM `mailbox_sides` WHERE `mailbox_id`=?", (self._mailbox_id,)) db.execute("DELETE FROM `mailboxes` WHERE `id`=?", (self._mailbox_id,)) self._app._summarize_mailbox_and_store(for_nameplate, side_rows, when, pruned=False) db.commit() # Shut down any listeners, just in case they're still lingering # around. for (send_f, stop_f) in self._listeners.values(): stop_f() self._listeners = {} self._app.free_mailbox(self._mailbox_id) def _shutdown(self): # used at test shutdown to accelerate client disconnects for (send_f, stop_f) in self._listeners.values(): stop_f() self._listeners = {} class AppNamespace: def __init__(self, db, blur_usage, log_requests, app_id): self._db = db self._blur_usage = blur_usage self._log_requests = log_requests self._app_id = app_id self._mailboxes = {} self._nameplate_counts = collections.defaultdict(int) self._mailbox_counts = collections.defaultdict(int) def get_nameplate_ids(self): db = self._db # TODO: filter this to numeric ids? c = db.execute("SELECT DISTINCT `name` FROM `nameplates`" " WHERE `app_id`=?", (self._app_id,)) return set([row["name"] for row in c.fetchall()]) def _find_available_nameplate_id(self): claimed = self.get_nameplate_ids() for size in range(1,4): # stick to 1-999 for now available = set() for id_int in range(10**(size-1), 10**size): id = "%d" % id_int if id not in claimed: available.add(id) if available: return random.choice(list(available)) # ouch, 999 currently claimed. Try random ones for a while. for tries in range(1000): id_int = random.randrange(1000, 1000*1000) id = "%d" % id_int if id not in claimed: return id raise ValueError("unable to find a free nameplate-id") def allocate_nameplate(self, side, when): nameplate_id = self._find_available_nameplate_id() mailbox_id = self.claim_nameplate(nameplate_id, side, when) del mailbox_id # ignored, they'll learn it from claim() return nameplate_id def claim_nameplate(self, name, side, when): # when we're done: # * there will be one row for the nameplate # * there will be one 'side' attached to it, with claimed=True # * a mailbox id and mailbox row will be created # * a mailbox 'side' will be attached, with opened=True assert isinstance(name, type("")), type(name) assert isinstance(side, type("")), type(side) db = self._db row = db.execute("SELECT * FROM `nameplates`" " WHERE `app_id`=? AND `name`=?", (self._app_id, name)).fetchone() if not row: if self._log_requests: log.msg("creating nameplate#%s for app_id %s" % (name, self._app_id)) mailbox_id = generate_mailbox_id() self._add_mailbox(mailbox_id, True, side, when) # ensure row exists sql = ("INSERT INTO `nameplates`" " (`app_id`, `name`, `mailbox_id`)" " VALUES(?,?,?)") npid = db.execute(sql, (self._app_id, name, mailbox_id) ).lastrowid else: npid = row["id"] mailbox_id = row["mailbox_id"] row = db.execute("SELECT * FROM `nameplate_sides`" " WHERE `nameplates_id`=? AND `side`=?", (npid, side)).fetchone() if not row: db.execute("INSERT INTO `nameplate_sides`" " (`nameplates_id`, `claimed`, `side`, `added`)" " VALUES(?,?,?,?)", (npid, True, side, when)) else: if not row["claimed"]: raise ReclaimedError("you cannot re-claim a nameplate that your side previously released") # since that might cause a new mailbox to be allocated db.commit() self.open_mailbox(mailbox_id, side, when) # may raise CrowdedError rows = db.execute("SELECT * FROM `nameplate_sides`" " WHERE `nameplates_id`=?", (npid,)).fetchall() if len(rows) > 2: # this line will probably never get hit: any crowding is noticed # on mailbox_sides first, inside open_mailbox() raise CrowdedError("too many sides have claimed this nameplate") return mailbox_id def release_nameplate(self, name, side, when): # when we're done: # * the 'claimed' flag will be cleared on the nameplate_sides row # * if the nameplate is now unused (no claimed sides): # * a usage record will be added # * the nameplate row will be removed # * the nameplate sides will be removed assert isinstance(name, type("")), type(name) assert isinstance(side, type("")), type(side) db = self._db np_row = db.execute("SELECT * FROM `nameplates`" " WHERE `app_id`=? AND `name`=?", (self._app_id, name)).fetchone() if not np_row: return npid = np_row["id"] row = db.execute("SELECT * FROM `nameplate_sides`" " WHERE `nameplates_id`=? AND `side`=?", (npid, side)).fetchone() if not row: return db.execute("UPDATE `nameplate_sides` SET `claimed`=?" " WHERE `nameplates_id`=? AND `side`=?", (False, npid, side)) db.commit() # now, are there any remaining claims? side_rows = db.execute("SELECT * FROM `nameplate_sides`" " WHERE `nameplates_id`=?", (npid,)).fetchall() claims = [1 for sr in side_rows if sr["claimed"]] if claims: return # delete and summarize db.execute("DELETE FROM `nameplate_sides` WHERE `nameplates_id`=?", (npid,)) db.execute("DELETE FROM `nameplates` WHERE `id`=?", (npid,)) self._summarize_nameplate_and_store(side_rows, when, pruned=False) db.commit() def _summarize_nameplate_and_store(self, side_rows, delete_time, pruned): # requires caller to db.commit() u = self._summarize_nameplate_usage(side_rows, delete_time, pruned) self._db.execute("INSERT INTO `nameplate_usage`" " (`app_id`," " `started`, `total_time`, `waiting_time`, `result`)" " VALUES (?, ?,?,?,?)", (self._app_id, u.started, u.total_time, u.waiting_time, u.result)) self._nameplate_counts[u.result] += 1 def _summarize_nameplate_usage(self, side_rows, delete_time, pruned): times = sorted([row["added"] for row in side_rows]) started = times[0] if self._blur_usage: started = self._blur_usage * (started // self._blur_usage) waiting_time = None if len(times) > 1: waiting_time = times[1] - times[0] total_time = delete_time - times[0] result = "lonely" if len(times) == 2: result = "happy" if pruned: result = "pruney" if len(times) > 2: result = "crowded" return Usage(started=started, waiting_time=waiting_time, total_time=total_time, result=result) def _add_mailbox(self, mailbox_id, for_nameplate, side, when): assert isinstance(mailbox_id, type("")), type(mailbox_id) db = self._db row = db.execute("SELECT * FROM `mailboxes`" " WHERE `app_id`=? AND `id`=?", (self._app_id, mailbox_id)).fetchone() if not row: self._db.execute("INSERT INTO `mailboxes`" " (`app_id`, `id`, `for_nameplate`, `updated`)" " VALUES(?,?,?,?)", (self._app_id, mailbox_id, for_nameplate, when)) # we don't need a commit here, because mailbox.open() only # does SELECT FROM `mailbox_sides`, not from `mailboxes` def open_mailbox(self, mailbox_id, side, when): assert isinstance(mailbox_id, type("")), type(mailbox_id) self._add_mailbox(mailbox_id, False, side, when) # ensure row exists db = self._db if not mailbox_id in self._mailboxes: # ensure Mailbox object exists if self._log_requests: log.msg("spawning #%s for app_id %s" % (mailbox_id, self._app_id)) self._mailboxes[mailbox_id] = Mailbox(self, self._db, self._app_id, mailbox_id) mailbox = self._mailboxes[mailbox_id] # delegate to mailbox.open() to add a row to mailbox_sides, and # update the mailbox.updated timestamp mailbox.open(side, when) db.commit() rows = db.execute("SELECT * FROM `mailbox_sides`" " WHERE `mailbox_id`=?", (mailbox_id,)).fetchall() if len(rows) > 2: raise CrowdedError("too many sides have opened this mailbox") return mailbox def free_mailbox(self, mailbox_id): # called from Mailbox.delete_and_summarize(), which deletes any # messages if mailbox_id in self._mailboxes: self._mailboxes.pop(mailbox_id) #if self._log_requests: # log.msg("freed+killed #%s, now have %d DB mailboxes, %d live" % # (mailbox_id, len(self.get_claimed()), len(self._mailboxes))) def _summarize_mailbox_and_store(self, for_nameplate, side_rows, delete_time, pruned): db = self._db u = self._summarize_mailbox(side_rows, delete_time, pruned) db.execute("INSERT INTO `mailbox_usage`" " (`app_id`, `for_nameplate`," " `started`, `total_time`, `waiting_time`, `result`)" " VALUES (?,?, ?,?,?,?)", (self._app_id, for_nameplate, u.started, u.total_time, u.waiting_time, u.result)) self._mailbox_counts[u.result] += 1 def _summarize_mailbox(self, side_rows, delete_time, pruned): times = sorted([row["added"] for row in side_rows]) started = times[0] if self._blur_usage: started = self._blur_usage * (started // self._blur_usage) waiting_time = None if len(times) > 1: waiting_time = times[1] - times[0] total_time = delete_time - times[0] num_sides = len(times) if num_sides == 0: result = "quiet" elif num_sides == 1: result = "lonely" else: result = "happy" # "mood" is only recorded at close() moods = [row["mood"] for row in side_rows if row.get("mood")] if "lonely" in moods: result = "lonely" if "errory" in moods: result = "errory" if "scary" in moods: result = "scary" if pruned: result = "pruney" if num_sides > 2: result = "crowded" return Usage(started=started, waiting_time=waiting_time, total_time=total_time, result=result) def prune(self, now, old): # The pruning check runs every 10 minutes, and "old" is defined to be # 11 minutes ago (unit tests can use different values). The client is # allowed to disconnect for up to 9 minutes without losing the # channel (nameplate, mailbox, and messages). # Each time a client does something, the mailbox.updated field is # updated with the current timestamp. If a client is subscribed to # the mailbox when pruning check runs, the "updated" field is also # updated. After that check, if the "updated" field is "old", the # channel is deleted. # For now, pruning is logged even if log_requests is False, to debug # the pruning process, and since pruning is triggered by a timer # instead of by user action. It does reveal which mailboxes were # present when the pruning process began, though, so in the log run # it should do less logging. log.msg(" prune begins (%s)" % self._app_id) db = self._db modified = False for mailbox in self._mailboxes.values(): if mailbox.has_listeners(): log.msg("touch %s because listeners" % mailbox._mailbox_id) mailbox._touch(now) db.commit() # make sure the updates are visible below new_mailboxes = set() old_mailboxes = set() for row in db.execute("SELECT * FROM `mailboxes` WHERE `app_id`=?", (self._app_id,)).fetchall(): mailbox_id = row["id"] log.msg(" 1: age=%s, old=%s, %s" % (now - row["updated"], now - old, mailbox_id)) if row["updated"] > old: new_mailboxes.add(mailbox_id) else: old_mailboxes.add(mailbox_id) log.msg(" 2: mailboxes:", new_mailboxes, old_mailboxes) old_nameplates = set() for row in db.execute("SELECT * FROM `nameplates` WHERE `app_id`=?", (self._app_id,)).fetchall(): npid = row["id"] mailbox_id = row["mailbox_id"] if mailbox_id in old_mailboxes: old_nameplates.add(npid) log.msg(" 3: old_nameplates", old_nameplates) for npid in old_nameplates: log.msg(" deleting nameplate", npid) side_rows = db.execute("SELECT * FROM `nameplate_sides`" " WHERE `nameplates_id`=?", (npid,)).fetchall() db.execute("DELETE FROM `nameplate_sides` WHERE `nameplates_id`=?", (npid,)) db.execute("DELETE FROM `nameplates` WHERE `id`=?", (npid,)) self._summarize_nameplate_and_store(side_rows, now, pruned=True) modified = True # delete all messages for old mailboxes # delete all old mailboxes for mailbox_id in old_mailboxes: log.msg(" deleting mailbox", mailbox_id) row = db.execute("SELECT * FROM `mailboxes`" " WHERE `id`=?", (mailbox_id,)).fetchone() for_nameplate = row["for_nameplate"] side_rows = db.execute("SELECT * FROM `mailbox_sides`" " WHERE `mailbox_id`=?", (mailbox_id,)).fetchall() db.execute("DELETE FROM `messages` WHERE `mailbox_id`=?", (mailbox_id,)) db.execute("DELETE FROM `mailbox_sides` WHERE `mailbox_id`=?", (mailbox_id,)) db.execute("DELETE FROM `mailboxes` WHERE `id`=?", (mailbox_id,)) self._summarize_mailbox_and_store(for_nameplate, side_rows, now, pruned=True) modified = True if modified: db.commit() log.msg(" prune complete, modified:", modified) def get_counts(self): return (self._nameplate_counts, self._mailbox_counts) def _shutdown(self): for channel in self._mailboxes.values(): channel._shutdown() class Rendezvous(service.MultiService): def __init__(self, db, welcome, blur_usage): service.MultiService.__init__(self) self._db = db self._welcome = welcome self._blur_usage = blur_usage log_requests = blur_usage is None self._log_requests = log_requests self._apps = {} def get_welcome(self): return self._welcome def get_log_requests(self): return self._log_requests def get_app(self, app_id): assert isinstance(app_id, type("")) if not app_id in self._apps: if self._log_requests: log.msg("spawning app_id %s" % (app_id,)) self._apps[app_id] = AppNamespace(self._db, self._blur_usage, self._log_requests, app_id) return self._apps[app_id] def get_all_apps(self): apps = set() for row in self._db.execute("SELECT DISTINCT `app_id`" " FROM `nameplates`").fetchall(): apps.add(row["app_id"]) for row in self._db.execute("SELECT DISTINCT `app_id`" " FROM `mailboxes`").fetchall(): apps.add(row["app_id"]) for row in self._db.execute("SELECT DISTINCT `app_id`" " FROM `messages`").fetchall(): apps.add(row["app_id"]) return apps def prune_all_apps(self, now, old): # As with AppNamespace.prune_old_mailboxes, we log for now. log.msg("beginning app prune") for app_id in sorted(self.get_all_apps()): log.msg(" app prune checking %r" % (app_id,)) app = self.get_app(app_id) app.prune(now, old) log.msg("app prune ends, %d apps" % len(self._apps)) def get_stats(self): stats = {} # current status: expected to be zero most of the time c = stats["active"] = {} c["apps"] = len(self.get_all_apps()) def q(query, values=()): row = self._db.execute(query, values).fetchone() return list(row.values())[0] c["nameplates_total"] = q("SELECT COUNT() FROM `nameplates`") # TODO: nameplates with only one side (most of them) # TODO: nameplates with two sides (very fleeting) # TODO: nameplates with three or more sides (crowded, unlikely) c["mailboxes_total"] = q("SELECT COUNT() FROM `mailboxes`") # TODO: mailboxes with only one side (most of them) # TODO: mailboxes with two sides (somewhat fleeting, in-transit) # TODO: mailboxes with three or more sides (unlikely) c["messages_total"] = q("SELECT COUNT() FROM `messages`") # usage since last reboot nameplate_counts = collections.defaultdict(int) mailbox_counts = collections.defaultdict(int) for app in self._apps.values(): nc, mc = app.get_counts() for result, count in nc.items(): nameplate_counts[result] += count for result, count in mc.items(): mailbox_counts[result] += count urb = stats["since_reboot"] = {} urb["nameplate_moods"] = {} for result, count in nameplate_counts.items(): urb["nameplate_moods"][result] = count urb["nameplates_total"] = sum(nameplate_counts.values()) urb["mailbox_moods"] = {} for result, count in mailbox_counts.items(): urb["mailbox_moods"][result] = count urb["mailboxes_total"] = sum(mailbox_counts.values()) # historical usage (all-time) u = stats["all_time"] = {} un = u["nameplate_moods"] = {} # TODO: there's probably a single SQL query for all this un["happy"] = q("SELECT COUNT() FROM `nameplate_usage`" " WHERE `result`='happy'") un["lonely"] = q("SELECT COUNT() FROM `nameplate_usage`" " WHERE `result`='lonely'") un["pruney"] = q("SELECT COUNT() FROM `nameplate_usage`" " WHERE `result`='pruney'") un["crowded"] = q("SELECT COUNT() FROM `nameplate_usage`" " WHERE `result`='crowded'") u["nameplates_total"] = q("SELECT COUNT() FROM `nameplate_usage`") um = u["mailbox_moods"] = {} um["happy"] = q("SELECT COUNT() FROM `mailbox_usage`" " WHERE `result`='happy'") um["scary"] = q("SELECT COUNT() FROM `mailbox_usage`" " WHERE `result`='scary'") um["lonely"] = q("SELECT COUNT() FROM `mailbox_usage`" " WHERE `result`='lonely'") um["quiet"] = q("SELECT COUNT() FROM `mailbox_usage`" " WHERE `result`='quiet'") um["errory"] = q("SELECT COUNT() FROM `mailbox_usage`" " WHERE `result`='errory'") um["pruney"] = q("SELECT COUNT() FROM `mailbox_usage`" " WHERE `result`='pruney'") um["crowded"] = q("SELECT COUNT() FROM `mailbox_usage`" " WHERE `result`='crowded'") u["mailboxes_total"] = q("SELECT COUNT() FROM `mailbox_usage`") u["mailboxes_standalone"] = q("SELECT COUNT() FROM `mailbox_usage`" " WHERE `for_nameplate`=0") # recent timings (last 100 operations) # TODO: median/etc of nameplate.total_time # TODO: median/etc of mailbox.waiting_time (should be the same) # TODO: median/etc of mailbox.total_time # other # TODO: mailboxes without nameplates (needs new DB schema) return stats def stopService(self): # This forcibly boots any clients that are still connected, which # helps with unit tests that use threads for both clients. One client # hits an exception, which terminates the test (and .tearDown calls # stopService on the relay), but the other client (in its thread) is # still waiting for a message. By killing off all connections, that # other client gets an error, and exits promptly. for app in self._apps.values(): app._shutdown() return service.MultiService.stopService(self) magic-wormhole-0.9.1/src/wormhole/server/rendezvous_websocket.py0000644000076600000240000002760113031527043025636 0ustar warnerstaff00000000000000from __future__ import unicode_literals import time from twisted.internet import reactor from twisted.python import log from autobahn.twisted import websocket from .rendezvous import CrowdedError, ReclaimedError, SidedMessage from ..util import dict_to_bytes, bytes_to_dict # The WebSocket allows the client to send "commands" to the server, and the # server to send "responses" to the client. Note that commands and responses # are not necessarily one-to-one. All commands provoke an "ack" response # (with a copy of the original message) for timing, testing, and # synchronization purposes. All commands and responses are JSON-encoded. # Each WebSocket connection is bound to one "appid" and one "side", which are # set by the "bind" command (which must be the first command on the # connection), and must be set before any other command will be accepted. # Each connection can be bound to a single "mailbox" (a two-sided # store-and-forward queue, identified by the "mailbox id": a long, randomly # unique string identifier) by using the "open" command. This protects the # mailbox from idle closure, enables the "add" command (to put new messages # in the queue), and triggers delivery of past and future messages via the # "message" response. The "close" command removes the binding (but note that # it does not enable the subsequent binding of a second mailbox). When the # last side closes a mailbox, its contents are deleted. # Additionally, the connection can be bound a single "nameplate", which is # short identifier that makes up the first component of a wormhole code. Each # nameplate points to a single long-id "mailbox". The "allocate" message # determines the shortest available numeric nameplate, reserves it, and # returns the nameplate id. "list" returns a list of all numeric nameplates # which currently have only one side active (i.e. they are waiting for a # partner). The "claim" message reserves an arbitrary nameplate id (perhaps # the receiver of a wormhole connection typed in a code they got from the # sender, or perhaps the two sides agreed upon a code offline and are both # typing it in), and the "release" message releases it. When every side that # has claimed the nameplate has also released it, the nameplate is # deallocated (but they will probably keep the underlying mailbox open). # Inbound (client to server) commands are marked as "->" below. Unrecognized # inbound keys will be ignored. Outbound (server to client) responses use # "<-". There is no guaranteed correlation between requests and responses. In # this list, "A -> B" means that some time after A is received, at least one # message of type B will be sent out (probably). # All responses include a "server_tx" key, which is a float (seconds since # epoch) with the server clock just before the outbound response was written # to the socket. # connection -> welcome # <- {type: "welcome", welcome: {}} # .welcome keys are all optional: # current_cli_version: out-of-date clients display a warning # motd: all clients display message, then continue normally # error: all clients display mesage, then terminate with error # -> {type: "bind", appid:, side:} # # -> {type: "list"} -> nameplates # <- {type: "nameplates", nameplates: [{id: str,..},..]} # -> {type: "allocate"} -> nameplate, mailbox # <- {type: "allocated", nameplate: str} # -> {type: "claim", nameplate: str} -> mailbox # <- {type: "claimed", mailbox: str} # -> {type: "release"} # .nameplate is optional, but must match previous claim() # <- {type: "released"} # # -> {type: "open", mailbox: str} -> message # sends old messages now, and subscribes to deliver future messages # <- {type: "message", side:, phase:, body:, msg_id:}} # body is hex # -> {type: "add", phase: str, body: hex} # will send echo in a "message" # # -> {type: "close", mood: str} -> closed # .mailbox is optional, but must match previous open() # <- {type: "closed"} # # <- {type: "error", error: str, orig: {}} # in response to malformed msgs # for tests that need to know when a message has been processed: # -> {type: "ping", ping: int} -> pong (does not require bind/claim) # <- {type: "pong", pong: int} class Error(Exception): def __init__(self, explain): self._explain = explain class WebSocketRendezvous(websocket.WebSocketServerProtocol): def __init__(self): websocket.WebSocketServerProtocol.__init__(self) self._app = None self._side = None self._did_allocate = False # only one allocate() per websocket self._listening = False self._did_claim = False self._nameplate_id = None self._did_release = False self._did_open = False self._mailbox = None self._mailbox_id = None self._did_close = False def onConnect(self, request): rv = self.factory.rendezvous if rv.get_log_requests(): log.msg("ws client connecting: %s" % (request.peer,)) self._reactor = self.factory.reactor def onOpen(self): rv = self.factory.rendezvous self.send("welcome", welcome=rv.get_welcome()) def onMessage(self, payload, isBinary): server_rx = time.time() msg = bytes_to_dict(payload) try: if "type" not in msg: raise Error("missing 'type'") self.send("ack", id=msg.get("id")) mtype = msg["type"] if mtype == "ping": return self.handle_ping(msg) if mtype == "bind": return self.handle_bind(msg) if not self._app: raise Error("must bind first") if mtype == "list": return self.handle_list() if mtype == "allocate": return self.handle_allocate(server_rx) if mtype == "claim": return self.handle_claim(msg, server_rx) if mtype == "release": return self.handle_release(msg, server_rx) if mtype == "open": return self.handle_open(msg, server_rx) if mtype == "add": return self.handle_add(msg, server_rx) if mtype == "close": return self.handle_close(msg, server_rx) raise Error("unknown type") except Error as e: self.send("error", error=e._explain, orig=msg) def handle_ping(self, msg): if "ping" not in msg: raise Error("ping requires 'ping'") self.send("pong", pong=msg["ping"]) def handle_bind(self, msg): if self._app or self._side: raise Error("already bound") if "appid" not in msg: raise Error("bind requires 'appid'") if "side" not in msg: raise Error("bind requires 'side'") self._app = self.factory.rendezvous.get_app(msg["appid"]) self._side = msg["side"] def handle_list(self): nameplate_ids = sorted(self._app.get_nameplate_ids()) # provide room to add nameplate attributes later (like which wordlist # is used for each, maybe how many words) nameplates = [{"id": nid} for nid in nameplate_ids] self.send("nameplates", nameplates=nameplates) def handle_allocate(self, server_rx): if self._did_allocate: raise Error("you already allocated one, don't be greedy") nameplate_id = self._app.allocate_nameplate(self._side, server_rx) assert isinstance(nameplate_id, type("")) self._did_allocate = True self.send("allocated", nameplate=nameplate_id) def handle_claim(self, msg, server_rx): if "nameplate" not in msg: raise Error("claim requires 'nameplate'") if self._did_claim: raise Error("only one claim per connection") self._did_claim = True nameplate_id = msg["nameplate"] assert isinstance(nameplate_id, type("")), type(nameplate_id) self._nameplate_id = nameplate_id try: mailbox_id = self._app.claim_nameplate(nameplate_id, self._side, server_rx) except CrowdedError: raise Error("crowded") except ReclaimedError: raise Error("reclaimed") self.send("claimed", mailbox=mailbox_id) def handle_release(self, msg, server_rx): if self._did_release: raise Error("only one release per connection") if "nameplate" in msg: if self._nameplate_id is not None: if msg["nameplate"] != self._nameplate_id: raise Error("release and claim must use same nameplate") nameplate_id = msg["nameplate"] else: if self._nameplate_id is None: raise Error("release without nameplate must follow claim") nameplate_id = self._nameplate_id assert nameplate_id is not None self._did_release = True self._app.release_nameplate(nameplate_id, self._side, server_rx) self.send("released") def handle_open(self, msg, server_rx): if self._mailbox: raise Error("only one open per connection") if "mailbox" not in msg: raise Error("open requires 'mailbox'") mailbox_id = msg["mailbox"] assert isinstance(mailbox_id, type("")) self._mailbox_id = mailbox_id self._mailbox = self._app.open_mailbox(mailbox_id, self._side, server_rx) def _send(sm): self.send("message", side=sm.side, phase=sm.phase, body=sm.body, server_rx=sm.server_rx, id=sm.msg_id) def _stop(): pass self._listening = True for old_sm in self._mailbox.add_listener(self, _send, _stop): _send(old_sm) def handle_add(self, msg, server_rx): if not self._mailbox: raise Error("must open mailbox before adding") if "phase" not in msg: raise Error("missing 'phase'") if "body" not in msg: raise Error("missing 'body'") msg_id = msg.get("id") # optional sm = SidedMessage(side=self._side, phase=msg["phase"], body=msg["body"], server_rx=server_rx, msg_id=msg_id) self._mailbox.add_message(sm) def handle_close(self, msg, server_rx): if self._did_close: raise Error("only one close per connection") if "mailbox" in msg: if self._mailbox_id is not None: if msg["mailbox"] != self._mailbox_id: raise Error("open and close must use same mailbox") mailbox_id = msg["mailbox"] else: if self._mailbox_id is None: raise Error("close without mailbox must follow open") mailbox_id = self._mailbox_id if not self._mailbox: self._mailbox = self._app.open_mailbox(mailbox_id, self._side, server_rx) if self._listening: self._mailbox.remove_listener(self) self._listening = False self._did_close = True self._mailbox.close(self._side, msg.get("mood"), server_rx) self._mailbox = None self.send("closed") def send(self, mtype, **kwargs): kwargs["type"] = mtype kwargs["server_tx"] = time.time() payload = dict_to_bytes(kwargs) self.sendMessage(payload, False) def onClose(self, wasClean, code, reason): #log.msg("onClose", self, self._mailbox, self._listening) if self._mailbox and self._listening: self._mailbox.remove_listener(self) class WebSocketRendezvousFactory(websocket.WebSocketServerFactory): protocol = WebSocketRendezvous def __init__(self, url, rendezvous): websocket.WebSocketServerFactory.__init__(self, url) self.setProtocolOptions(autoPingInterval=60, autoPingTimeout=600) self.rendezvous = rendezvous self.reactor = reactor # for tests to control magic-wormhole-0.9.1/src/wormhole/server/runner.py0000644000076600000240000000402713024532264022675 0ustar warnerstaff00000000000000from __future__ import print_function, unicode_literals import os, sys from .cli_args import parser def dispatch(args): if args.func == "server/start": from . import cmd_server return cmd_server.start_server(args) if args.func == "server/stop": from . import cmd_server return cmd_server.stop_server(args) if args.func == "server/restart": from . import cmd_server return cmd_server.restart_server(args) if args.func == "usage/usage": from . import cmd_usage return cmd_usage.show_usage(args) if args.func == "usage/tail": from . import cmd_usage return cmd_usage.tail_usage(args) if args.func == "usage/count-channels": from . import cmd_usage return cmd_usage.count_channels(args) if args.func == "usage/count-events": from . import cmd_usage return cmd_usage.count_events(args) raise ValueError("unknown args.func %s" % args.func) def run(args, cwd, stdout, stderr, executable=None): """This is invoked directly by the 'wormhole-server' entry-point script. It can also invoked by entry() below.""" args = parser.parse_args() if not getattr(args, "func", None): # So far this only works on py3. py2 exits with a really terse # "error: too few arguments" during parse_args(). parser.print_help() sys.exit(0) args.cwd = cwd args.stdout = stdout args.stderr = stderr try: rc = dispatch(args) return rc except ImportError as e: print("--- ImportError ---", file=stderr) print(e, file=stderr) print("Please run 'python setup.py build'", file=stderr) raise return 1 def entry(): """This is used by a setuptools entry_point. When invoked this way, setuptools has already put the installed package on sys.path .""" return run(sys.argv[1:], os.getcwd(), sys.stdout, sys.stderr, executable=sys.argv[0]) if __name__ == "__main__": args = parser.parse_args() print(args) magic-wormhole-0.9.1/src/wormhole/server/server.py0000644000076600000240000001227713032241523022672 0ustar warnerstaff00000000000000# NO unicode_literals or static.Data() will break, because it demands # a str on Python 2 from __future__ import print_function import os, time, json from twisted.python import log from twisted.internet import reactor, endpoints from twisted.application import service, internet from twisted.web import server, static, resource from autobahn.twisted.resource import WebSocketResource from .. import __version__ from .database import get_db from .rendezvous import Rendezvous from .rendezvous_websocket import WebSocketRendezvousFactory from .transit_server import Transit SECONDS = 1.0 MINUTE = 60*SECONDS CHANNEL_EXPIRATION_TIME = 11*MINUTE EXPIRATION_CHECK_PERIOD = 10*MINUTE class Root(resource.Resource): # child_FOO is a nevow thing, not a twisted.web.resource thing def __init__(self): resource.Resource.__init__(self) self.putChild(b"", static.Data(b"Wormhole Relay\n", "text/plain")) class PrivacyEnhancedSite(server.Site): logRequests = True def log(self, request): if self.logRequests: return server.Site.log(self, request) class RelayServer(service.MultiService): def __init__(self, rendezvous_web_port, transit_port, advertise_version, db_url=":memory:", blur_usage=None, signal_error=None, stats_file=None): service.MultiService.__init__(self) self._blur_usage = blur_usage db = get_db(db_url) welcome = { # The primary (python CLI) implementation will emit a message if # its version does not match this key. If/when we have # distributions which include older version, but we still expect # them to be compatible, stop sending this key. "current_cli_version": __version__, # adding .motd will cause all clients to display the message, # then keep running normally #"motd": "Welcome to the public relay.\nPlease enjoy this service.", # adding .error will cause all clients to fail, with this message #"error": "This server has been disabled, see URL for details.", } if advertise_version: welcome["current_cli_version"] = advertise_version if signal_error: welcome["error"] = signal_error self._rendezvous = Rendezvous(db, welcome, blur_usage) self._rendezvous.setServiceParent(self) # for the pruning timer root = Root() wsrf = WebSocketRendezvousFactory(None, self._rendezvous) root.putChild(b"v1", WebSocketResource(wsrf)) site = PrivacyEnhancedSite(root) if blur_usage: site.logRequests = False r = endpoints.serverFromString(reactor, rendezvous_web_port) rendezvous_web_service = internet.StreamServerEndpointService(r, site) rendezvous_web_service.setServiceParent(self) if transit_port: transit = Transit(db, blur_usage) transit.setServiceParent(self) # for the timer t = endpoints.serverFromString(reactor, transit_port) transit_service = internet.StreamServerEndpointService(t, transit) transit_service.setServiceParent(self) self._stats_file = stats_file if self._stats_file and os.path.exists(self._stats_file): os.unlink(self._stats_file) # this will be regenerated immediately, but if something goes # wrong in dump_stats(), it's better to have a missing file than # a stale one t = internet.TimerService(EXPIRATION_CHECK_PERIOD, self.timer) t.setServiceParent(self) # make some things accessible for tests self._db = db self._root = root self._rendezvous_web_service = rendezvous_web_service self._rendezvous_websocket = wsrf self._transit = None if transit_port: self._transit = transit self._transit_service = transit_service def startService(self): service.MultiService.startService(self) log.msg("websocket listening on /wormhole-relay/ws") log.msg("Wormhole relay server (Rendezvous and Transit) running") if self._blur_usage: log.msg("blurring access times to %d seconds" % self._blur_usage) log.msg("not logging HTTP requests or Transit connections") else: log.msg("not blurring access times") def timer(self): now = time.time() old = now - CHANNEL_EXPIRATION_TIME self._rendezvous.prune_all_apps(now, old) self.dump_stats(now, validity=EXPIRATION_CHECK_PERIOD+60) def dump_stats(self, now, validity): if not self._stats_file: return tmpfn = self._stats_file + ".tmp" data = {} data["created"] = now data["valid_until"] = now + validity start = time.time() data["rendezvous"] = self._rendezvous.get_stats() data["transit"] = self._transit.get_stats() log.msg("get_stats took:", time.time() - start) with open(tmpfn, "wb") as f: # json.dump(f) has str-vs-unicode issues on py2-vs-py3 f.write(json.dumps(data, indent=1).encode("utf-8")) f.write(b"\n") os.rename(tmpfn, self._stats_file) magic-wormhole-0.9.1/src/wormhole/server/transit_server.py0000644000076600000240000003243213032241074024432 0ustar warnerstaff00000000000000from __future__ import print_function, unicode_literals import re, time, collections from twisted.python import log from twisted.internet import protocol from twisted.application import service SECONDS = 1.0 MINUTE = 60*SECONDS HOUR = 60*MINUTE DAY = 24*HOUR MB = 1000*1000 def round_to(size, coarseness): return int(coarseness*(1+int((size-1)/coarseness))) def blur_size(size): if size == 0: return 0 if size < 1e6: return round_to(size, 10e3) if size < 1e9: return round_to(size, 1e6) return round_to(size, 100e6) class TransitConnection(protocol.Protocol): def __init__(self): self._got_token = False self._got_side = False self._token_buffer = b"" self._sent_ok = False self._buddy = None self._had_buddy = False self._total_sent = 0 def describeToken(self): d = "-" if self._got_token: d = self._got_token[:16].decode("ascii") if self._got_side: d += "-" + self._got_side.decode("ascii") else: d += "-" return d def connectionMade(self): self._started = time.time() self._log_requests = self.factory._log_requests def dataReceived(self, data): if self._sent_ok: # We are an IPushProducer to our buddy's IConsumer, so they'll # throttle us (by calling pauseProducing()) when their outbound # buffer is full (e.g. when their downstream pipe is full). In # practice, this buffers about 10MB per connection, after which # point the sender will only transmit data as fast as the # receiver can handle it. self._total_sent += len(data) self._buddy.transport.write(data) return if self._got_token: # but not yet sent_ok self.transport.write(b"impatient\n") if self._log_requests: log.msg("transit impatience failure") return self.disconnect() # impatience yields failure # else this should be (part of) the token self._token_buffer += data buf = self._token_buffer # old: "please relay {64}\n" # new: "please relay {64} for side {16}\n" (old, handshake_len, token) = self._check_old_handshake(buf) assert old in ("yes", "waiting", "no") if old == "yes": # remember they aren't supposed to send anything past their # handshake until we've said go if len(buf) > handshake_len: self.transport.write(b"impatient\n") if self._log_requests: log.msg("transit impatience failure") return self.disconnect() # impatience yields failure return self._got_handshake(token, None) (new, handshake_len, token, side) = self._check_new_handshake(buf) assert new in ("yes", "waiting", "no") if new == "yes": if len(buf) > handshake_len: self.transport.write(b"impatient\n") if self._log_requests: log.msg("transit impatience failure") return self.disconnect() # impatience yields failure return self._got_handshake(token, side) if (old == "no" and new == "no"): self.transport.write(b"bad handshake\n") if self._log_requests: log.msg("transit handshake failure") return self.disconnect() # incorrectness yields failure # else we'll keep waiting def _check_old_handshake(self, buf): # old: "please relay {64}\n" # return ("yes", handshake, token) if buf contains an old-style handshake # return ("waiting", None, None) if it might eventually contain one # return ("no", None, None) if it could never contain one wanted = len("please relay \n")+32*2 if len(buf) < wanted-1 and b"\n" in buf: return ("no", None, None) if len(buf) < wanted: return ("waiting", None, None) mo = re.search(br"^please relay (\w{64})\n", buf, re.M) if mo: token = mo.group(1) return ("yes", wanted, token) return ("no", None, None) def _check_new_handshake(self, buf): # new: "please relay {64} for side {16}\n" wanted = len("please relay for side \n")+32*2+8*2 if len(buf) < wanted-1 and b"\n" in buf: return ("no", None, None, None) if len(buf) < wanted: return ("waiting", None, None, None) mo = re.search(br"^please relay (\w{64}) for side (\w{16})\n", buf, re.M) if mo: token = mo.group(1) side = mo.group(2) return ("yes", wanted, token, side) return ("no", None, None, None) def _got_handshake(self, token, side): self._got_token = token self._got_side = side self.factory.connection_got_token(token, side, self) def buddy_connected(self, them): self._buddy = them self._had_buddy = True self.transport.write(b"ok\n") self._sent_ok = True # Connect the two as a producer/consumer pair. We use streaming=True, # so this expects the IPushProducer interface, and uses # pauseProducing() to throttle, and resumeProducing() to unthrottle. self._buddy.transport.registerProducer(self.transport, True) # The Transit object calls buddy_connected() on both protocols, so # there will be two producer/consumer pairs. def buddy_disconnected(self): if self._log_requests: log.msg("buddy_disconnected %s" % self.describeToken()) self._buddy = None self.transport.loseConnection() def connectionLost(self, reason): if self._buddy: self._buddy.buddy_disconnected() self.factory.transitFinished(self, self._got_token, self._got_side, self.describeToken()) # Record usage. There are four cases: # * 1: we connected, never had a buddy # * 2: we connected first, we disconnect before the buddy # * 3: we connected first, buddy disconnects first # * 4: buddy connected first, we disconnect before buddy # * 5: buddy connected first, buddy disconnects first # whoever disconnects first gets to write the usage record (1,2,4) finished = time.time() if not self._had_buddy: # 1 total_time = finished - self._started self.factory.recordUsage(self._started, "lonely", 0, total_time, None) if self._had_buddy and self._buddy: # 2,4 total_bytes = self._total_sent + self._buddy._total_sent starts = [self._started, self._buddy._started] total_time = finished - min(starts) waiting_time = max(starts) - min(starts) self.factory.recordUsage(self._started, "happy", total_bytes, total_time, waiting_time) def disconnect(self): self.transport.loseConnection() self.factory.transitFailed(self) finished = time.time() total_time = finished - self._started self.factory.recordUsage(self._started, "errory", 0, total_time, None) class Transit(protocol.ServerFactory, service.MultiService): # I manage pairs of simultaneous connections to a secondary TCP port, # both forwarded to the other. Clients must begin each connection with # "please relay TOKEN for SIDE\n" (or a legacy form without the "for # SIDE"). Two connections match if they use the same TOKEN and have # different SIDEs (the redundant connections are dropped when a match is # made). Legacy connections match any with the same TOKEN, ignoring SIDE # (so two legacy connections will match each other). # I will send "ok\n" when the matching connection is established, or # disconnect if no matching connection is made within MAX_WAIT_TIME # seconds. I will disconnect if you send data before the "ok\n". All data # you get after the "ok\n" will be from the other side. You will not # receive "ok\n" until the other side has also connected and submitted a # matching token (and differing SIDE). # In addition, the connections will be dropped after MAXLENGTH bytes have # been sent by either side, or MAXTIME seconds have elapsed after the # matching connections were established. A future API will reveal these # limits to clients instead of causing mysterious spontaneous failures. # These relay connections are not half-closeable (unlike full TCP # connections, applications will not receive any data after half-closing # their outgoing side). Applications must negotiate shutdown with their # peer and not close the connection until all data has finished # transferring in both directions. Applications which only need to send # data in one direction can use close() as usual. MAX_WAIT_TIME = 30*SECONDS MAXLENGTH = 10*MB MAXTIME = 60*SECONDS protocol = TransitConnection def __init__(self, db, blur_usage): service.MultiService.__init__(self) self._db = db self._blur_usage = blur_usage self._log_requests = blur_usage is None self._pending_requests = {} # token -> set((side, TransitConnection)) self._active_connections = set() # TransitConnection self._counts = collections.defaultdict(int) self._count_bytes = 0 def connection_got_token(self, token, new_side, new_tc): if token not in self._pending_requests: self._pending_requests[token] = set() potentials = self._pending_requests[token] for old in potentials: (old_side, old_tc) = old if ((old_side is None) or (new_side is None) or (old_side != new_side)): # we found a match if self._log_requests: log.msg("transit relay 2: %s" % new_tc.describeToken()) # drop and stop tracking the rest potentials.remove(old) for (_, leftover_tc) in potentials: leftover_tc.disconnect() # TODO: not "errory"? self._pending_requests.pop(token) # glue the two ends together self._active_connections.add(new_tc) self._active_connections.add(old_tc) new_tc.buddy_connected(old_tc) old_tc.buddy_connected(new_tc) return if self._log_requests: log.msg("transit relay 1: %s" % new_tc.describeToken()) potentials.add((new_side, new_tc)) # TODO: timer def recordUsage(self, started, result, total_bytes, total_time, waiting_time): if self._log_requests: log.msg("Transit.recordUsage (%dB)" % total_bytes) if self._blur_usage: started = self._blur_usage * (started // self._blur_usage) total_bytes = blur_size(total_bytes) self._db.execute("INSERT INTO `transit_usage`" " (`started`, `total_time`, `waiting_time`," " `total_bytes`, `result`)" " VALUES (?,?,?, ?,?)", (started, total_time, waiting_time, total_bytes, result)) self._db.commit() self._counts[result] += 1 self._count_bytes += total_bytes def transitFinished(self, tc, token, side, description): if token in self._pending_requests: side_tc = (side, tc) if side_tc in self._pending_requests[token]: self._pending_requests[token].remove(side_tc) if not self._pending_requests[token]: # set is now empty del self._pending_requests[token] if self._log_requests: log.msg("transitFinished %s" % (description,)) self._active_connections.discard(tc) def transitFailed(self, p): if self._log_requests: log.msg("transitFailed %r" % p) pass def get_stats(self): stats = {} def q(query, values=()): row = self._db.execute(query, values).fetchone() return list(row.values())[0] # current status: expected to be zero most of the time c = stats["active"] = {} c["connected"] = len(self._active_connections) / 2 c["waiting"] = len(self._pending_requests) # usage since last reboot rb = stats["since_reboot"] = {} rb["bytes"] = self._count_bytes rb["total"] = sum(self._counts.values(), 0) rbm = rb["moods"] = {} for result, count in self._counts.items(): rbm[result] = count # historical usage (all-time) u = stats["all_time"] = {} u["total"] = q("SELECT COUNT() FROM `transit_usage`") u["bytes"] = q("SELECT SUM(`total_bytes`) FROM `transit_usage`") or 0 um = u["moods"] = {} um["happy"] = q("SELECT COUNT() FROM `transit_usage`" " WHERE `result`='happy'") um["lonely"] = q("SELECT COUNT() FROM `transit_usage`" " WHERE `result`='lonely'") um["errory"] = q("SELECT COUNT() FROM `transit_usage`" " WHERE `result`='errory'") return stats magic-wormhole-0.9.1/src/wormhole/test/0000755000076600000240000000000013032243761020460 5ustar warnerstaff00000000000000magic-wormhole-0.9.1/src/wormhole/test/__init__.py0000644000076600000240000000000012706034367022566 0ustar warnerstaff00000000000000magic-wormhole-0.9.1/src/wormhole/test/common.py0000644000076600000240000000666313024532264022335 0ustar warnerstaff00000000000000# no unicode_literals untill twisted update from twisted.application import service from twisted.internet import defer, task from twisted.python import log from click.testing import CliRunner import mock from ..cli import cli from ..transit import allocate_tcp_port from ..server.server import RelayServer from .. import __version__ class ServerBase: def setUp(self): self._setup_relay(None) def _setup_relay(self, error): self.sp = service.MultiService() self.sp.startService() self.relayport = allocate_tcp_port() self.transitport = allocate_tcp_port() # need to talk to twisted team about only using unicode in # endpoints.serverFromString s = RelayServer("tcp:%d:interface=127.0.0.1" % self.relayport, "tcp:%s:interface=127.0.0.1" % self.transitport, advertise_version=__version__, signal_error=error) s.setServiceParent(self.sp) self._rendezvous = s._rendezvous self._transit_server = s._transit self.relayurl = u"ws://127.0.0.1:%d/v1" % self.relayport self.rdv_ws_port = self.relayport # ws://127.0.0.1:%d/wormhole-relay/ws self.transit = u"tcp:127.0.0.1:%d" % self.transitport def tearDown(self): # Unit tests that spawn a (blocking) client in a thread might still # have threads running at this point, if one is stuck waiting for a # message from a companion which has exited with an error. Our # relay's .stopService() drops all connections, which ought to # encourage those threads to terminate soon. If they don't, print a # warning to ease debugging. # XXX FIXME there's something in _noclobber test that's not # waiting for a close, I think -- was pretty relieably getting # unclean-reactor, but adding a slight pause here stops it... from twisted.internet import reactor tp = reactor.getThreadPool() if not tp.working: d = defer.succeed(None) d.addCallback(lambda _: self.sp.stopService()) d.addCallback(lambda _: task.deferLater(reactor, 0.1, lambda: None)) return d return self.sp.stopService() # disconnect all callers d = defer.maybeDeferred(self.sp.stopService) wait_d = defer.Deferred() # wait a second, then check to see if it worked reactor.callLater(1.0, wait_d.callback, None) def _later(res): if len(tp.working): log.msg("wormhole.test.common.ServerBase.tearDown:" " I was unable to convince all threads to exit.") tp.dumpStats() print("tearDown warning: threads are still active") print("This test will probably hang until one of the" " clients gives up of their own accord.") else: log.msg("wormhole.test.common.ServerBase.tearDown:" " I convinced all threads to exit.") return d wait_d.addCallback(_later) return wait_d def config(*argv): r = CliRunner() with mock.patch("wormhole.cli.cli.go") as go: res = r.invoke(cli.wormhole, argv, catch_exceptions=False) if res.exit_code != 0: print(res.exit_code) print(res.output) print(res) assert 0 cfg = go.call_args[0][1] return cfg magic-wormhole-0.9.1/src/wormhole/test/run_trial.py0000644000076600000240000000115713024532264023035 0ustar warnerstaff00000000000000from __future__ import unicode_literals # This is a tiny helper module, to let "python -m wormhole.test.run_trial # ARGS" does the same thing as running "trial ARGS" (unfortunately # twisted/scripts/trial.py does not have a '__name__=="__main__"' clause). # # This makes it easier to run trial under coverage from tox: # * "coverage run trial ARGS" is how you'd usually do it # * but "trial" must be the one in tox's virtualenv # * "coverage run `which trial` ARGS" works from a shell # * but tox doesn't use a shell # So use: # "coverage run -m wormhole.test.run_trial ARGS" from twisted.scripts.trial import run run() magic-wormhole-0.9.1/src/wormhole/test/test_args.py0000644000076600000240000001171113027366362023035 0ustar warnerstaff00000000000000import sys from twisted.trial import unittest from ..cli.public_relay import RENDEZVOUS_RELAY, TRANSIT_RELAY from .common import config #from pprint import pprint class Send(unittest.TestCase): def test_baseline(self): cfg = config("send", "--text", "hi") #pprint(cfg.__dict__) self.assertEqual(cfg.what, None) self.assertEqual(cfg.code, None) self.assertEqual(cfg.code_length, 2) self.assertEqual(cfg.dump_timing, None) self.assertEqual(cfg.hide_progress, False) self.assertEqual(cfg.listen, True) self.assertEqual(cfg.appid, None) self.assertEqual(cfg.relay_url, RENDEZVOUS_RELAY) self.assertEqual(cfg.transit_helper, TRANSIT_RELAY) self.assertEqual(cfg.text, "hi") self.assertEqual(cfg.tor, False) self.assertEqual(cfg.verify, False) self.assertEqual(cfg.zeromode, False) def test_appid(self): cfg = config("--appid", "xyz", "send", "--text", "hi") self.assertEqual(cfg.appid, "xyz") cfg = config("--appid=xyz", "send", "--text", "hi") self.assertEqual(cfg.appid, "xyz") def test_file(self): cfg = config("send", "fn") #pprint(cfg.__dict__) self.assertEqual(cfg.what, u"fn") self.assertEqual(cfg.text, None) def test_text(self): cfg = config("send", "--text", "hi") self.assertEqual(cfg.what, None) self.assertEqual(cfg.text, u"hi") def test_nolisten(self): cfg = config("send", "--no-listen", "fn") self.assertEqual(cfg.listen, False) def test_code(self): cfg = config("send", "--code", "1-abc", "fn") self.assertEqual(cfg.code, u"1-abc") def test_code_length(self): cfg = config("send", "-c", "3", "fn") self.assertEqual(cfg.code_length, 3) def test_dump_timing(self): cfg = config("--dump-timing", "tx.json", "send", "fn") self.assertEqual(cfg.dump_timing, "tx.json") def test_hide_progress(self): cfg = config("send", "--hide-progress", "fn") self.assertEqual(cfg.hide_progress, True) def test_tor(self): cfg = config("send", "--tor", "fn") self.assertEqual(cfg.tor, True) def test_verify(self): cfg = config("send", "--verify", "fn") self.assertEqual(cfg.verify, True) def test_zeromode(self): cfg = config("send", "-0", "fn") self.assertEqual(cfg.zeromode, True) class Receive(unittest.TestCase): def test_baseline(self): cfg = config("receive") #pprint(cfg.__dict__) self.assertEqual(cfg.accept_file, False) self.assertEqual(cfg.code, None) self.assertEqual(cfg.code_length, 2) self.assertEqual(cfg.dump_timing, None) self.assertEqual(cfg.hide_progress, False) self.assertEqual(cfg.listen, True) self.assertEqual(cfg.only_text, False) self.assertEqual(cfg.output_file, None) self.assertEqual(cfg.appid, None) self.assertEqual(cfg.relay_url, RENDEZVOUS_RELAY) self.assertEqual(cfg.transit_helper, TRANSIT_RELAY) self.assertEqual(cfg.tor, False) self.assertEqual(cfg.verify, False) self.assertEqual(cfg.zeromode, False) def test_appid(self): cfg = config("--appid", "xyz", "receive") self.assertEqual(cfg.appid, "xyz") cfg = config("--appid=xyz", "receive") self.assertEqual(cfg.appid, "xyz") def test_nolisten(self): cfg = config("receive", "--no-listen") self.assertEqual(cfg.listen, False) def test_code(self): cfg = config("receive", "1-abc") self.assertEqual(cfg.code, u"1-abc") def test_code_length(self): cfg = config("receive", "-c", "3") self.assertEqual(cfg.code_length, 3) def test_dump_timing(self): cfg = config("--dump-timing", "tx.json", "receive") self.assertEqual(cfg.dump_timing, "tx.json") def test_hide_progress(self): cfg = config("receive", "--hide-progress") self.assertEqual(cfg.hide_progress, True) def test_tor(self): cfg = config("receive", "--tor") self.assertEqual(cfg.tor, True) def test_verify(self): cfg = config("receive", "--verify") self.assertEqual(cfg.verify, True) def test_zeromode(self): cfg = config("receive", "-0") self.assertEqual(cfg.zeromode, True) def test_only_text(self): cfg = config("receive", "-t") self.assertEqual(cfg.only_text, True) def test_accept_file(self): cfg = config("receive", "--accept-file") self.assertEqual(cfg.accept_file, True) def test_output_file(self): cfg = config("receive", "--output-file", "fn") self.assertEqual(cfg.output_file, u"fn") class Config(unittest.TestCase): def test_send(self): cfg = config("send") self.assertEqual(cfg.stdout, sys.stdout) def test_receive(self): cfg = config("receive") self.assertEqual(cfg.stdout, sys.stdout) magic-wormhole-0.9.1/src/wormhole/test/test_database.py0000644000076600000240000000403513024532264023637 0ustar warnerstaff00000000000000from __future__ import print_function, unicode_literals import os from twisted.trial import unittest from ..server.database import get_db, TARGET_VERSION, dump_db class DB(unittest.TestCase): def test_create_default(self): db_url = ":memory:" db = get_db(db_url) rows = db.execute("SELECT * FROM version").fetchall() self.assertEqual(len(rows), 1) self.assertEqual(rows[0]["version"], TARGET_VERSION) def test_upgrade(self): basedir = self.mktemp() os.mkdir(basedir) fn = os.path.join(basedir, "upgrade.db") self.assertNotEqual(TARGET_VERSION, 2) # create an old-version DB in a file db = get_db(fn, 2) rows = db.execute("SELECT * FROM version").fetchall() self.assertEqual(len(rows), 1) self.assertEqual(rows[0]["version"], 2) del db # then upgrade the file to the latest version dbA = get_db(fn, TARGET_VERSION) rows = dbA.execute("SELECT * FROM version").fetchall() self.assertEqual(len(rows), 1) self.assertEqual(rows[0]["version"], TARGET_VERSION) dbA_text = dump_db(dbA) del dbA # make sure the upgrades got committed to disk dbB = get_db(fn, TARGET_VERSION) dbB_text = dump_db(dbB) del dbB self.assertEqual(dbA_text, dbB_text) # The upgraded schema should be equivalent to that of a new DB. # However a text dump will differ because ALTER TABLE always appends # the new column to the end of a table, whereas our schema puts it # somewhere in the middle (wherever it fits naturally). Also ALTER # TABLE doesn't include comments. if False: latest_db = get_db(":memory:", TARGET_VERSION) latest_text = dump_db(latest_db) with open("up.sql","w") as f: f.write(dbA_text) with open("new.sql","w") as f: f.write(latest_text) # check with "diff -u _trial_temp/up.sql _trial_temp/new.sql" self.assertEqual(dbA_text, latest_text) magic-wormhole-0.9.1/src/wormhole/test/test_hkdf.py0000644000076600000240000000340613024532264023010 0ustar warnerstaff00000000000000from __future__ import print_function, unicode_literals import unittest from binascii import unhexlify #, hexlify from hkdf import Hkdf #def generate_KAT(): # print("KAT = [") # for salt in (b"", b"salt"): # for context in (b"", b"context"): # skm = b"secret" # out = HKDF(skm, 64, XTS=salt, CTXinfo=context) # hexout = " '%s' +\n '%s'" % (hexlify(out[:32]), # hexlify(out[32:])) # print(" (%r, %r, %r,\n%s)," % (salt, context, skm, hexout)) # print("]") KAT = [ ('', '', 'secret', '2f34e5ff91ec85d53ca9b543683174d0cf550b60d5f52b24c97b386cfcf6cbbf' + '9cfd42fd37e1e5a214d15f03058d7fee63dc28f564b7b9fe3da514f80daad4bf'), ('', 'context', 'secret', 'c24c303a1adfb4c3e2b092e6254ed481c41d8955ba8ec3f6a1473493a60c957b' + '31b723018ca75557214d3d5c61c0c7a5315b103b21ff00cb03ebe023dc347a47'), ('salt', '', 'secret', 'f1156507c39b0e326159e778696253122de430899a8df2484040a85a5f95ceb1' + 'dfca555d4cc603bdf7153ed1560de8cbc3234b27a6d2be8e8ca202d90649679a'), ('salt', 'context', 'secret', '61a4f201a867bcc12381ddb180d27074408d03ee9d5750855e5a12d967fa060f' + '10336ead9370927eaabb0d60b259346ee5f57eb7ceba8c72f1ed3f2932b1bf19'), ] class TestKAT(unittest.TestCase): # note: this uses SHA256 def test_kat(self): for (salt, context, skm, expected_hexout) in KAT: expected_out = unhexlify(expected_hexout) for outlen in range(0, len(expected_out)): out = Hkdf(salt.encode("ascii"), skm.encode("ascii")).expand(context.encode("ascii"), outlen) self.assertEqual(out, expected_out[:outlen]) #if __name__ == '__main__': # generate_KAT() magic-wormhole-0.9.1/src/wormhole/test/test_scripts.py0000644000076600000240000007443513027402475023600 0ustar warnerstaff00000000000000from __future__ import print_function, unicode_literals import os, sys, re, io, zipfile, six, stat from humanize import naturalsize import mock from twisted.trial import unittest from twisted.python import procutils, log from twisted.internet.utils import getProcessOutputAndValue from twisted.internet.defer import gatherResults, inlineCallbacks from .. import __version__ from .common import ServerBase, config from ..cli import cmd_send, cmd_receive from ..errors import TransferError, WrongPasswordError, WelcomeError def build_offer(args): s = cmd_send.Sender(args, None) return s._build_offer() class OfferData(unittest.TestCase): def setUp(self): self._things_to_delete = [] self.cfg = cfg = config("send") cfg.stdout = io.StringIO() cfg.stderr = io.StringIO() def tearDown(self): for fn in self._things_to_delete: if os.path.exists(fn): os.unlink(fn) del self.cfg def test_text(self): self.cfg.text = message = "blah blah blah ponies" d, fd_to_send = build_offer(self.cfg) self.assertIn("message", d) self.assertNotIn("file", d) self.assertNotIn("directory", d) self.assertEqual(d["message"], message) self.assertEqual(fd_to_send, None) def test_file(self): self.cfg.what = filename = "my file" message = b"yay ponies\n" send_dir = self.mktemp() os.mkdir(send_dir) abs_filename = os.path.join(send_dir, filename) with open(abs_filename, "wb") as f: f.write(message) self.cfg.cwd = send_dir d, fd_to_send = build_offer(self.cfg) self.assertNotIn("message", d) self.assertIn("file", d) self.assertNotIn("directory", d) self.assertEqual(d["file"]["filesize"], len(message)) self.assertEqual(d["file"]["filename"], filename) self.assertEqual(fd_to_send.tell(), 0) self.assertEqual(fd_to_send.read(), message) def test_missing_file(self): self.cfg.what = filename = "missing" send_dir = self.mktemp() os.mkdir(send_dir) self.cfg.cwd = send_dir e = self.assertRaises(TransferError, build_offer, self.cfg) self.assertEqual(str(e), "Cannot send: no file/directory named '%s'" % filename) def _do_test_directory(self, addslash): parent_dir = self.mktemp() os.mkdir(parent_dir) send_dir = "dirname" os.mkdir(os.path.join(parent_dir, send_dir)) ponies = [str(i) for i in range(5)] for p in ponies: with open(os.path.join(parent_dir, send_dir, p), "wb") as f: f.write(("%s ponies\n" % p).encode("ascii")) send_dir_arg = send_dir if addslash: send_dir_arg += os.sep self.cfg.what = send_dir_arg self.cfg.cwd = parent_dir d, fd_to_send = build_offer(self.cfg) self.assertNotIn("message", d) self.assertNotIn("file", d) self.assertIn("directory", d) self.assertEqual(d["directory"]["dirname"], send_dir) self.assertEqual(d["directory"]["mode"], "zipfile/deflated") self.assertEqual(d["directory"]["numfiles"], 5) self.assertIn("numbytes", d["directory"]) self.assertIsInstance(d["directory"]["numbytes"], six.integer_types) self.assertEqual(fd_to_send.tell(), 0) zdata = fd_to_send.read() self.assertEqual(len(zdata), d["directory"]["zipsize"]) fd_to_send.seek(0, 0) with zipfile.ZipFile(fd_to_send, "r", zipfile.ZIP_DEFLATED) as zf: zipnames = zf.namelist() self.assertEqual(list(sorted(ponies)), list(sorted(zipnames))) for name in zipnames: contents = zf.open(name, "r").read() self.assertEqual(("%s ponies\n" % name).encode("ascii"), contents) def test_directory(self): return self._do_test_directory(addslash=False) def test_directory_addslash(self): return self._do_test_directory(addslash=True) def test_unknown(self): self.cfg.what = filename = "unknown" send_dir = self.mktemp() os.mkdir(send_dir) abs_filename = os.path.abspath(os.path.join(send_dir, filename)) self.cfg.cwd = send_dir try: os.mkfifo(abs_filename) except AttributeError: raise unittest.SkipTest("is mkfifo supported on this platform?") # Delete the named pipe for the sake of users who might run "pip # wheel ." in this directory later. That command wants to copy # everything into a tempdir before building a wheel, and the # shutil.copy_tree() is uses can't handle the named pipe. self._things_to_delete.append(abs_filename) self.assertFalse(os.path.isfile(abs_filename)) self.assertFalse(os.path.isdir(abs_filename)) e = self.assertRaises(TypeError, build_offer, self.cfg) self.assertEqual(str(e), "'%s' is neither file nor directory" % filename) class ScriptsBase: def find_executable(self): # to make sure we're running the right executable (in a virtualenv), # we require that our "wormhole" lives in the same directory as our # "python" locations = procutils.which("wormhole") if not locations: raise unittest.SkipTest("unable to find 'wormhole' in $PATH") wormhole = locations[0] if (os.path.dirname(os.path.abspath(wormhole)) != os.path.dirname(sys.executable)): log.msg("locations: %s" % (locations,)) log.msg("sys.executable: %s" % (sys.executable,)) raise unittest.SkipTest("found the wrong 'wormhole' in $PATH: %s %s" % (wormhole, sys.executable)) return wormhole def is_runnable(self): # One property of Versioneer is that many changes to the source tree # (making a commit, dirtying a previously-clean tree) will change the # version string. Entrypoint scripts frequently insist upon importing # a library version that matches the script version (whatever was # reported when 'pip install' was run), and throw a # DistributionNotFound error when they don't match. This is really # annoying in a workspace created with "pip install -e .", as you # must re-run pip after each commit. # So let's report just one error in this case (from test_version), # and skip the other tests that we know will fail. # Setting LANG/LC_ALL to a unicode-capable locale is necessary to # convince Click to not complain about a forced-ascii locale. My # apologies to folks who want to run tests on a machine that doesn't # have the en_US.UTF-8 locale installed. wormhole = self.find_executable() d = getProcessOutputAndValue(wormhole, ["--version"], env=dict(LC_ALL="en_US.UTF-8", LANG="en_US.UTF-8")) def _check(res): out, err, rc = res if rc != 0: log.msg("wormhole not runnable in this tree:") log.msg("out", out) log.msg("err", err) log.msg("rc", rc) raise unittest.SkipTest("wormhole is not runnable in this tree") d.addCallback(_check) return d class ScriptVersion(ServerBase, ScriptsBase, unittest.TestCase): # we need Twisted to run the server, but we run the sender and receiver # with deferToThread() @inlineCallbacks def test_version(self): # "wormhole" must be on the path, so e.g. "pip install -e ." in a # virtualenv. This guards against an environment where the tests # below might run the wrong executable. self.maxDiff = None wormhole = self.find_executable() # we must pass on the environment so that "something" doesn't # get sad about UTF8 vs. ascii encodings out, err, rc = yield getProcessOutputAndValue(wormhole, ["--version"], env=os.environ) err = err.decode("utf-8") if "DistributionNotFound" in err: log.msg("stderr was %s" % err) last = err.strip().split("\n")[-1] self.fail("wormhole not runnable: %s" % last) ver = out.decode("utf-8") or err self.failUnlessEqual(ver.strip(), "magic-wormhole {}".format(__version__)) self.failUnlessEqual(rc, 0) class PregeneratedCode(ServerBase, ScriptsBase, unittest.TestCase): # we need Twisted to run the server, but we run the sender and receiver # with deferToThread() def setUp(self): d = self.is_runnable() d.addCallback(lambda _: ServerBase.setUp(self)) return d @inlineCallbacks def _do_test(self, as_subprocess=False, mode="text", addslash=False, override_filename=False): assert mode in ("text", "file", "directory", "slow-text") send_cfg = config("send") recv_cfg = config("receive") message = "blah blah blah ponies" for cfg in [send_cfg, recv_cfg]: cfg.hide_progress = True cfg.relay_url = self.relayurl cfg.transit_helper = "" cfg.listen = True cfg.code = "1-abc" cfg.stdout = io.StringIO() cfg.stderr = io.StringIO() send_dir = self.mktemp() os.mkdir(send_dir) receive_dir = self.mktemp() os.mkdir(receive_dir) if mode == "text" or mode == "slow-text": send_cfg.text = message elif mode == "file": send_filename = "testfile" with open(os.path.join(send_dir, send_filename), "w") as f: f.write(message) send_cfg.what = send_filename receive_filename = send_filename recv_cfg.accept_file = True if override_filename: recv_cfg.output_file = receive_filename = "outfile" elif mode == "directory": # $send_dir/ # $send_dir/middle/ # $send_dir/middle/$dirname/ # $send_dir/middle/$dirname/[12345] # cd $send_dir && wormhole send middle/$dirname # cd $receive_dir && wormhole receive # expect: $receive_dir/$dirname/[12345] send_dirname = "testdir" def message(i): return "test message %d\n" % i os.mkdir(os.path.join(send_dir, "middle")) source_dir = os.path.join(send_dir, "middle", send_dirname) os.mkdir(source_dir) modes = {} for i in range(5): path = os.path.join(source_dir, str(i)) with open(path, "w") as f: f.write(message(i)) if i == 3: os.chmod(path, 0o755) modes[i] = stat.S_IMODE(os.stat(path).st_mode) send_dirname_arg = os.path.join("middle", send_dirname) if addslash: send_dirname_arg += os.sep send_cfg.what = send_dirname_arg receive_dirname = send_dirname recv_cfg.accept_file = True if override_filename: recv_cfg.output_file = receive_dirname = "outdir" if as_subprocess: wormhole_bin = self.find_executable() if send_cfg.text: content_args = ['--text', send_cfg.text] elif send_cfg.what: content_args = [send_cfg.what] send_args = [ '--relay-url', self.relayurl, '--transit-helper', '', 'send', '--hide-progress', '--code', send_cfg.code, ] + content_args send_d = getProcessOutputAndValue( wormhole_bin, send_args, path=send_dir, env=dict(LC_ALL="en_US.UTF-8", LANG="en_US.UTF-8"), ) recv_args = [ '--relay-url', self.relayurl, '--transit-helper', '', 'receive', '--hide-progress', '--accept-file', recv_cfg.code, ] if override_filename: recv_args.extend(['-o', receive_filename]) receive_d = getProcessOutputAndValue( wormhole_bin, recv_args, path=receive_dir, env=dict(LC_ALL="en_US.UTF-8", LANG="en_US.UTF-8"), ) (send_res, receive_res) = yield gatherResults([send_d, receive_d], True) send_stdout = send_res[0].decode("utf-8") send_stderr = send_res[1].decode("utf-8") send_rc = send_res[2] receive_stdout = receive_res[0].decode("utf-8") receive_stderr = receive_res[1].decode("utf-8") receive_rc = receive_res[2] NL = os.linesep self.assertEqual((send_rc, receive_rc), (0, 0), (send_res, receive_res)) else: send_cfg.cwd = send_dir send_d = cmd_send.send(send_cfg) recv_cfg.cwd = receive_dir receive_d = cmd_receive.receive(recv_cfg) # The sender might fail, leaving the receiver hanging, or vice # versa. Make sure we don't wait on one side exclusively if mode == "slow-text": with mock.patch.object(cmd_send, "VERIFY_TIMER", 0), \ mock.patch.object(cmd_receive, "VERIFY_TIMER", 0): yield gatherResults([send_d, receive_d], True) else: yield gatherResults([send_d, receive_d], True) send_stdout = send_cfg.stdout.getvalue() send_stderr = send_cfg.stderr.getvalue() receive_stdout = recv_cfg.stdout.getvalue() receive_stderr = recv_cfg.stderr.getvalue() # all output here comes from a StringIO, which uses \n for # newlines, even if we're on windows NL = "\n" self.maxDiff = None # show full output for assertion failures key_established = "" if mode == "slow-text": key_established = "Key established, waiting for confirmation...\n" self.assertEqual(send_stdout, "") # check sender if mode == "text" or mode == "slow-text": expected = ("Sending text message ({bytes:d} Bytes){NL}" "On the other computer, please run: " "wormhole receive{NL}" "Wormhole code is: {code}{NL}{NL}" "{KE}" "text message sent{NL}").format(bytes=len(message), code=send_cfg.code, NL=NL, KE=key_established) self.failUnlessEqual(send_stderr, expected) elif mode == "file": self.failUnlessIn("Sending {size:s} file named '{name}'{NL}" .format(size=naturalsize(len(message)), name=send_filename, NL=NL), send_stderr) self.failUnlessIn("On the other computer, please run: " "wormhole receive{NL}" "Wormhole code is: {code}{NL}{NL}" .format(code=send_cfg.code, NL=NL), send_stderr) self.failUnlessIn("File sent.. waiting for confirmation{NL}" "Confirmation received. Transfer complete.{NL}" .format(NL=NL), send_stderr) elif mode == "directory": self.failUnlessIn("Sending directory", send_stderr) self.failUnlessIn("named 'testdir'", send_stderr) self.failUnlessIn("On the other computer, please run: " "wormhole receive{NL}" "Wormhole code is: {code}{NL}{NL}" .format(code=send_cfg.code, NL=NL), send_stderr) self.failUnlessIn("File sent.. waiting for confirmation{NL}" "Confirmation received. Transfer complete.{NL}" .format(NL=NL), send_stderr) # check receiver if mode == "text" or mode == "slow-text": self.assertEqual(receive_stdout, message+NL) self.assertEqual(receive_stderr, key_established) elif mode == "file": self.failUnlessEqual(receive_stdout, "") self.failUnlessIn("Receiving file ({size:s}) into: {name}" .format(size=naturalsize(len(message)), name=receive_filename), receive_stderr) self.failUnlessIn("Received file written to ", receive_stderr) fn = os.path.join(receive_dir, receive_filename) self.failUnless(os.path.exists(fn)) with open(fn, "r") as f: self.failUnlessEqual(f.read(), message) elif mode == "directory": self.failUnlessEqual(receive_stdout, "") want = (r"Receiving directory \(\d+ \w+\) into: {name}/" .format(name=receive_dirname)) self.failUnless(re.search(want, receive_stderr), (want, receive_stderr)) self.failUnlessIn("Received files written to {name}" .format(name=receive_dirname), receive_stderr) fn = os.path.join(receive_dir, receive_dirname) self.failUnless(os.path.exists(fn), fn) for i in range(5): fn = os.path.join(receive_dir, receive_dirname, str(i)) with open(fn, "r") as f: self.failUnlessEqual(f.read(), message(i)) self.failUnlessEqual(modes[i], stat.S_IMODE(os.stat(fn).st_mode)) # check server stats self._rendezvous.get_stats() def test_text(self): return self._do_test() def test_text_subprocess(self): return self._do_test(as_subprocess=True) def test_file(self): return self._do_test(mode="file") def test_file_override(self): return self._do_test(mode="file", override_filename=True) def test_directory(self): return self._do_test(mode="directory") def test_directory_addslash(self): return self._do_test(mode="directory", addslash=True) def test_directory_override(self): return self._do_test(mode="directory", override_filename=True) def test_slow_text(self): return self._do_test(mode="slow-text") @inlineCallbacks def _do_test_fail(self, mode, failmode): assert mode in ("file", "directory") assert failmode in ("noclobber", "toobig") send_cfg = config("send") recv_cfg = config("receive") for cfg in [send_cfg, recv_cfg]: cfg.hide_progress = True cfg.relay_url = self.relayurl cfg.transit_helper = "" cfg.listen = False cfg.code = "1-abc" cfg.stdout = io.StringIO() cfg.stderr = io.StringIO() send_dir = self.mktemp() os.mkdir(send_dir) receive_dir = self.mktemp() os.mkdir(receive_dir) recv_cfg.accept_file = True # don't ask for permission if mode == "file": message = "test message\n" send_cfg.what = receive_name = send_filename = "testfile" fn = os.path.join(send_dir, send_filename) with open(fn, "w") as f: f.write(message) size = os.stat(fn).st_size elif mode == "directory": # $send_dir/ # $send_dir/$dirname/ # $send_dir/$dirname/[12345] # cd $send_dir && wormhole send $dirname # cd $receive_dir && wormhole receive # expect: $receive_dir/$dirname/[12345] size = 0 send_cfg.what = receive_name = send_dirname = "testdir" os.mkdir(os.path.join(send_dir, send_dirname)) for i in range(5): path = os.path.join(send_dir, send_dirname, str(i)) with open(path, "w") as f: f.write("test message %d\n" % i) size += os.stat(path).st_size if failmode == "noclobber": PRESERVE = "don't clobber me\n" clobberable = os.path.join(receive_dir, receive_name) with open(clobberable, "w") as f: f.write(PRESERVE) send_cfg.cwd = send_dir send_d = cmd_send.send(send_cfg) recv_cfg.cwd = receive_dir receive_d = cmd_receive.receive(recv_cfg) # both sides will fail if failmode == "noclobber": free_space = 10000000 else: free_space = 0 with mock.patch("wormhole.cli.cmd_receive.estimate_free_space", return_value=free_space): f = yield self.assertFailure(send_d, TransferError) self.assertEqual(str(f), "remote error, transfer abandoned: transfer rejected") f = yield self.assertFailure(receive_d, TransferError) self.assertEqual(str(f), "transfer rejected") send_stdout = send_cfg.stdout.getvalue() send_stderr = send_cfg.stderr.getvalue() receive_stdout = recv_cfg.stdout.getvalue() receive_stderr = recv_cfg.stderr.getvalue() # all output here comes from a StringIO, which uses \n for # newlines, even if we're on windows NL = "\n" self.maxDiff = None # show full output for assertion failures self.assertEqual(send_stdout, "") self.assertEqual(receive_stdout, "") # check sender if mode == "file": self.failUnlessIn("Sending {size:s} file named '{name}'{NL}" .format(size=naturalsize(size), name=send_filename, NL=NL), send_stderr) self.failUnlessIn("On the other computer, please run: " "wormhole receive{NL}" "Wormhole code is: {code}{NL}{NL}" .format(code=send_cfg.code, NL=NL), send_stderr) self.failIfIn("File sent.. waiting for confirmation{NL}" "Confirmation received. Transfer complete.{NL}" .format(NL=NL), send_stderr) elif mode == "directory": self.failUnlessIn("Sending directory", send_stderr) self.failUnlessIn("named 'testdir'", send_stderr) self.failUnlessIn("On the other computer, please run: " "wormhole receive{NL}" "Wormhole code is: {code}{NL}{NL}" .format(code=send_cfg.code, NL=NL), send_stderr) self.failIfIn("File sent.. waiting for confirmation{NL}" "Confirmation received. Transfer complete.{NL}" .format(NL=NL), send_stderr) # check receiver if mode == "file": self.failIfIn("Received file written to ", receive_stderr) if failmode == "noclobber": self.failUnlessIn("Error: " "refusing to overwrite existing 'testfile'{NL}" .format(NL=NL), receive_stderr) else: self.failUnlessIn("Error: " "insufficient free space (0B) for file ({size:d}B){NL}" .format(NL=NL, size=size), receive_stderr) elif mode == "directory": self.failIfIn("Received files written to {name}" .format(name=receive_name), receive_stderr) #want = (r"Receiving directory \(\d+ \w+\) into: {name}/" # .format(name=receive_name)) #self.failUnless(re.search(want, receive_stderr), # (want, receive_stderr)) if failmode == "noclobber": self.failUnlessIn("Error: " "refusing to overwrite existing 'testdir'{NL}" .format(NL=NL), receive_stderr) else: self.failUnlessIn("Error: " "insufficient free space (0B) for directory ({size:d}B){NL}" .format(NL=NL, size=size), receive_stderr) if failmode == "noclobber": fn = os.path.join(receive_dir, receive_name) self.failUnless(os.path.exists(fn)) with open(fn, "r") as f: self.failUnlessEqual(f.read(), PRESERVE) # check server stats self._rendezvous.get_stats() def test_fail_file_noclobber(self): return self._do_test_fail("file", "noclobber") def test_fail_directory_noclobber(self): return self._do_test_fail("directory", "noclobber") def test_fail_file_toobig(self): return self._do_test_fail("file", "toobig") def test_fail_directory_toobig(self): return self._do_test_fail("directory", "toobig") class NotWelcome(ServerBase, unittest.TestCase): def setUp(self): self._setup_relay(error="please upgrade XYZ") self.cfg = cfg = config("send") cfg.hide_progress = True cfg.listen = False cfg.relay_url = self.relayurl cfg.transit_helper = "" cfg.stdout = io.StringIO() cfg.stderr = io.StringIO() @inlineCallbacks def test_sender(self): self.cfg.text = "hi" self.cfg.code = "1-abc" send_d = cmd_send.send(self.cfg) f = yield self.assertFailure(send_d, WelcomeError) self.assertEqual(str(f), "please upgrade XYZ") @inlineCallbacks def test_receiver(self): self.cfg.code = "1-abc" receive_d = cmd_receive.receive(self.cfg) f = yield self.assertFailure(receive_d, WelcomeError) self.assertEqual(str(f), "please upgrade XYZ") class Cleanup(ServerBase, unittest.TestCase): def setUp(self): d = super(Cleanup, self).setUp() self.cfg = cfg = config("send") # common options for all tests in this suite cfg.hide_progress = True cfg.relay_url = self.relayurl cfg.transit_helper = "" cfg.stdout = io.StringIO() cfg.stderr = io.StringIO() return d @inlineCallbacks @mock.patch('sys.stdout') def test_text(self, stdout): # the rendezvous channel should be deleted after success self.cfg.text = "hello" self.cfg.code = "1-abc" send_d = cmd_send.send(self.cfg) receive_d = cmd_receive.receive(self.cfg) yield send_d yield receive_d cids = self._rendezvous.get_app(cmd_send.APPID).get_nameplate_ids() self.assertEqual(len(cids), 0) @inlineCallbacks def test_text_wrong_password(self): # if the password was wrong, the rendezvous channel should still be # deleted self.cfg.text = "secret message" self.cfg.code = "1-abc" send_d = cmd_send.send(self.cfg) self.cfg.code = "1-WRONG" receive_d = cmd_receive.receive(self.cfg) # both sides should be capable of detecting the mismatch yield self.assertFailure(send_d, WrongPasswordError) yield self.assertFailure(receive_d, WrongPasswordError) cids = self._rendezvous.get_app(cmd_send.APPID).get_nameplate_ids() self.assertEqual(len(cids), 0) self.flushLoggedErrors(WrongPasswordError) class ExtractFile(unittest.TestCase): def test_filenames(self): args = mock.Mock() args.relay_url = "" ef = cmd_receive.TwistedReceiver(args)._extract_file extract_dir = os.path.abspath(self.mktemp()) zf = mock.Mock() zi = mock.Mock() zi.filename = "ok" zi.external_attr = 5 << 16 expected = os.path.join(extract_dir, "ok") with mock.patch.object(cmd_receive.os, "chmod") as chmod: ef(zf, zi, extract_dir) self.assertEqual(zf.extract.mock_calls, [mock.call(zi.filename, path=extract_dir)]) self.assertEqual(chmod.mock_calls, [mock.call(expected, 5)]) zf = mock.Mock() zi = mock.Mock() zi.filename = "../haha" e = self.assertRaises(ValueError, ef, zf, zi, extract_dir) self.assertIn("malicious zipfile", str(e)) zf = mock.Mock() zi = mock.Mock() zi.filename = "haha//root" # abspath squashes this, hopefully zipfile # does too zi.external_attr = 5 << 16 expected = os.path.join(extract_dir, "haha", "root") with mock.patch.object(cmd_receive.os, "chmod") as chmod: ef(zf, zi, extract_dir) self.assertEqual(zf.extract.mock_calls, [mock.call(zi.filename, path=extract_dir)]) self.assertEqual(chmod.mock_calls, [mock.call(expected, 5)]) zf = mock.Mock() zi = mock.Mock() zi.filename = "/etc/passwd" e = self.assertRaises(ValueError, ef, zf, zi, extract_dir) self.assertIn("malicious zipfile", str(e)) class AppID(ServerBase, unittest.TestCase): def setUp(self): d = super(AppID, self).setUp() self.cfg = cfg = config("send") # common options for all tests in this suite cfg.hide_progress = True cfg.relay_url = self.relayurl cfg.transit_helper = "" cfg.stdout = io.StringIO() cfg.stderr = io.StringIO() return d @inlineCallbacks def test_override(self): # make sure we use the overridden appid, not the default self.cfg.text = "hello" self.cfg.appid = "appid2" self.cfg.code = "1-abc" send_d = cmd_send.send(self.cfg) receive_d = cmd_receive.receive(self.cfg) yield send_d yield receive_d used = self._rendezvous._db.execute("SELECT DISTINCT `app_id`" " FROM `nameplate_usage`" ).fetchall() self.assertEqual(len(used), 1, used) self.assertEqual(used[0]["app_id"], u"appid2") magic-wormhole-0.9.1/src/wormhole/test/test_server.py0000644000076600000240000013763113031527043023407 0ustar warnerstaff00000000000000from __future__ import print_function, unicode_literals import os, json, itertools, time import mock from twisted.trial import unittest from twisted.python import log from twisted.internet import reactor, defer from twisted.internet.defer import inlineCallbacks, returnValue from autobahn.twisted import websocket from .. import __version__ from .common import ServerBase from ..server import server, rendezvous from ..server.rendezvous import Usage, SidedMessage from ..server.database import get_db class _Util: def _nameplate(self, app, name): np_row = app._db.execute("SELECT * FROM `nameplates`" " WHERE `app_id`='appid' AND `name`=?", (name,)).fetchone() if not np_row: return None, None npid = np_row["id"] side_rows = app._db.execute("SELECT * FROM `nameplate_sides`" " WHERE `nameplates_id`=?", (npid,)).fetchall() return np_row, side_rows def _mailbox(self, app, mailbox_id): mb_row = app._db.execute("SELECT * FROM `mailboxes`" " WHERE `app_id`='appid' AND `id`=?", (mailbox_id,)).fetchone() if not mb_row: return None, None side_rows = app._db.execute("SELECT * FROM `mailbox_sides`" " WHERE `mailbox_id`=?", (mailbox_id,)).fetchall() return mb_row, side_rows def _messages(self, app): c = app._db.execute("SELECT * FROM `messages`" " WHERE `app_id`='appid' AND `mailbox_id`='mid'") return c.fetchall() class Server(_Util, ServerBase, unittest.TestCase): def test_apps(self): app1 = self._rendezvous.get_app("appid1") self.assertIdentical(app1, self._rendezvous.get_app("appid1")) app2 = self._rendezvous.get_app("appid2") self.assertNotIdentical(app1, app2) def test_nameplate_allocation(self): app = self._rendezvous.get_app("appid") nids = set() # this takes a second, and claims all the short-numbered nameplates def add(): nameplate_id = app.allocate_nameplate("side1", 0) self.assertEqual(type(nameplate_id), type("")) nid = int(nameplate_id) nids.add(nid) for i in range(9): add() self.assertNotIn(0, nids) self.assertEqual(set(range(1,10)), nids) for i in range(100-10): add() self.assertEqual(len(nids), 99) self.assertEqual(set(range(1,100)), nids) for i in range(1000-100): add() self.assertEqual(len(nids), 999) self.assertEqual(set(range(1,1000)), nids) add() self.assertEqual(len(nids), 1000) biggest = max(nids) self.assert_(1000 <= biggest < 1000000, biggest) def test_nameplate(self): app = self._rendezvous.get_app("appid") name = app.allocate_nameplate("side1", 0) self.assertEqual(type(name), type("")) nid = int(name) self.assert_(0 < nid < 10, nid) self.assertEqual(app.get_nameplate_ids(), set([name])) # allocate also does a claim np_row, side_rows = self._nameplate(app, name) self.assertEqual(len(side_rows), 1) self.assertEqual(side_rows[0]["side"], "side1") self.assertEqual(side_rows[0]["added"], 0) # duplicate claims by the same side are combined mailbox_id = app.claim_nameplate(name, "side1", 1) self.assertEqual(type(mailbox_id), type("")) self.assertEqual(mailbox_id, np_row["mailbox_id"]) np_row, side_rows = self._nameplate(app, name) self.assertEqual(len(side_rows), 1) self.assertEqual(side_rows[0]["added"], 0) self.assertEqual(mailbox_id, np_row["mailbox_id"]) # and they don't updated the 'added' time mailbox_id2 = app.claim_nameplate(name, "side1", 2) self.assertEqual(mailbox_id, mailbox_id2) np_row, side_rows = self._nameplate(app, name) self.assertEqual(len(side_rows), 1) self.assertEqual(side_rows[0]["added"], 0) # claim by the second side is new mailbox_id3 = app.claim_nameplate(name, "side2", 3) self.assertEqual(mailbox_id, mailbox_id3) np_row, side_rows = self._nameplate(app, name) self.assertEqual(len(side_rows), 2) self.assertEqual(sorted([row["side"] for row in side_rows]), sorted(["side1", "side2"])) self.assertIn(("side2", 3), [(row["side"], row["added"]) for row in side_rows]) # a third claim marks the nameplate as "crowded", and adds a third # claim (which must be released later), but leaves the two existing # claims alone self.assertRaises(rendezvous.CrowdedError, app.claim_nameplate, name, "side3", 4) np_row, side_rows = self._nameplate(app, name) self.assertEqual(len(side_rows), 3) # releasing a non-existent nameplate is ignored app.release_nameplate(name+"not", "side4", 0) # releasing a side that never claimed the nameplate is ignored app.release_nameplate(name, "side4", 0) np_row, side_rows = self._nameplate(app, name) self.assertEqual(len(side_rows), 3) # releasing one side leaves the second claim app.release_nameplate(name, "side1", 5) np_row, side_rows = self._nameplate(app, name) claims = [(row["side"], row["claimed"]) for row in side_rows] self.assertIn(("side1", False), claims) self.assertIn(("side2", True), claims) self.assertIn(("side3", True), claims) # releasing one side multiple times is ignored app.release_nameplate(name, "side1", 5) np_row, side_rows = self._nameplate(app, name) claims = [(row["side"], row["claimed"]) for row in side_rows] self.assertIn(("side1", False), claims) self.assertIn(("side2", True), claims) self.assertIn(("side3", True), claims) # release the second side app.release_nameplate(name, "side2", 6) np_row, side_rows = self._nameplate(app, name) claims = [(row["side"], row["claimed"]) for row in side_rows] self.assertIn(("side1", False), claims) self.assertIn(("side2", False), claims) self.assertIn(("side3", True), claims) # releasing the third side frees the nameplate, and adds usage app.release_nameplate(name, "side3", 7) np_row, side_rows = self._nameplate(app, name) self.assertEqual(np_row, None) usage = app._db.execute("SELECT * FROM `nameplate_usage`").fetchone() self.assertEqual(usage["app_id"], "appid") self.assertEqual(usage["started"], 0) self.assertEqual(usage["waiting_time"], 3) self.assertEqual(usage["total_time"], 7) self.assertEqual(usage["result"], "crowded") def test_mailbox(self): app = self._rendezvous.get_app("appid") mailbox_id = "mid" m1 = app.open_mailbox(mailbox_id, "side1", 0) mb_row, side_rows = self._mailbox(app, mailbox_id) self.assertEqual(len(side_rows), 1) self.assertEqual(side_rows[0]["side"], "side1") self.assertEqual(side_rows[0]["added"], 0) # opening the same mailbox twice, by the same side, gets the same # object, and does not update the "added" timestamp self.assertIdentical(m1, app.open_mailbox(mailbox_id, "side1", 1)) mb_row, side_rows = self._mailbox(app, mailbox_id) self.assertEqual(len(side_rows), 1) self.assertEqual(side_rows[0]["side"], "side1") self.assertEqual(side_rows[0]["added"], 0) # opening a second side gets the same object, and adds a new claim self.assertIdentical(m1, app.open_mailbox(mailbox_id, "side2", 2)) mb_row, side_rows = self._mailbox(app, mailbox_id) self.assertEqual(len(side_rows), 2) adds = [(row["side"], row["added"]) for row in side_rows] self.assertIn(("side1", 0), adds) self.assertIn(("side2", 2), adds) # a third open marks it as crowded self.assertRaises(rendezvous.CrowdedError, app.open_mailbox, mailbox_id, "side3", 3) mb_row, side_rows = self._mailbox(app, mailbox_id) self.assertEqual(len(side_rows), 3) m1.close("side3", "company", 4) # closing a side that never claimed the mailbox is ignored m1.close("side4", "mood", 4) mb_row, side_rows = self._mailbox(app, mailbox_id) self.assertEqual(len(side_rows), 3) # closing one side leaves the second claim m1.close("side1", "mood", 5) mb_row, side_rows = self._mailbox(app, mailbox_id) sides = [(row["side"], row["opened"], row["mood"]) for row in side_rows] self.assertIn(("side1", False, "mood"), sides) self.assertIn(("side2", True, None), sides) self.assertIn(("side3", False, "company"), sides) # closing one side multiple times is ignored m1.close("side1", "mood", 6) mb_row, side_rows = self._mailbox(app, mailbox_id) sides = [(row["side"], row["opened"], row["mood"]) for row in side_rows] self.assertIn(("side1", False, "mood"), sides) self.assertIn(("side2", True, None), sides) self.assertIn(("side3", False, "company"), sides) l1 = []; stop1 = []; stop1_f = lambda: stop1.append(True) m1.add_listener("handle1", l1.append, stop1_f) # closing the second side frees the mailbox, and adds usage m1.close("side2", "mood", 7) self.assertEqual(stop1, [True]) mb_row, side_rows = self._mailbox(app, mailbox_id) self.assertEqual(mb_row, None) usage = app._db.execute("SELECT * FROM `mailbox_usage`").fetchone() self.assertEqual(usage["app_id"], "appid") self.assertEqual(usage["started"], 0) self.assertEqual(usage["waiting_time"], 2) self.assertEqual(usage["total_time"], 7) self.assertEqual(usage["result"], "crowded") def test_messages(self): app = self._rendezvous.get_app("appid") mailbox_id = "mid" m1 = app.open_mailbox(mailbox_id, "side1", 0) m1.add_message(SidedMessage(side="side1", phase="phase", body="body", server_rx=1, msg_id="msgid")) msgs = self._messages(app) self.assertEqual(len(msgs), 1) self.assertEqual(msgs[0]["body"], "body") l1 = []; stop1 = []; stop1_f = lambda: stop1.append(True) l2 = []; stop2 = []; stop2_f = lambda: stop2.append(True) old = m1.add_listener("handle1", l1.append, stop1_f) self.assertEqual(len(old), 1) self.assertEqual(old[0].side, "side1") self.assertEqual(old[0].body, "body") m1.add_message(SidedMessage(side="side1", phase="phase2", body="body2", server_rx=1, msg_id="msgid")) self.assertEqual(len(l1), 1) self.assertEqual(l1[0].body, "body2") old = m1.add_listener("handle2", l2.append, stop2_f) self.assertEqual(len(old), 2) m1.add_message(SidedMessage(side="side1", phase="phase3", body="body3", server_rx=1, msg_id="msgid")) self.assertEqual(len(l1), 2) self.assertEqual(l1[-1].body, "body3") self.assertEqual(len(l2), 1) self.assertEqual(l2[-1].body, "body3") m1.remove_listener("handle1") m1.add_message(SidedMessage(side="side1", phase="phase4", body="body4", server_rx=1, msg_id="msgid")) self.assertEqual(len(l1), 2) self.assertEqual(l1[-1].body, "body3") self.assertEqual(len(l2), 2) self.assertEqual(l2[-1].body, "body4") m1._shutdown() self.assertEqual(stop1, []) self.assertEqual(stop2, [True]) # message adds are not idempotent: clients filter duplicates m1.add_message(SidedMessage(side="side1", phase="phase", body="body", server_rx=1, msg_id="msgid")) msgs = self._messages(app) self.assertEqual(len(msgs), 5) self.assertEqual(msgs[-1]["body"], "body") class Prune(unittest.TestCase): def _get_mailbox_updated(self, app, mbox_id): row = app._db.execute("SELECT * FROM `mailboxes` WHERE" " `app_id`=? AND `id`=?", (app._app_id, mbox_id)).fetchone() return row["updated"] def test_update(self): db = get_db(":memory:") rv = rendezvous.Rendezvous(db, None, None) app = rv.get_app("appid") mbox_id = "mbox1" app.open_mailbox(mbox_id, "side1", 1) self.assertEqual(self._get_mailbox_updated(app, mbox_id), 1) mb = app.open_mailbox(mbox_id, "side2", 2) self.assertEqual(self._get_mailbox_updated(app, mbox_id), 2) sm = SidedMessage("side1", "phase", "body", 3, "msgid") mb.add_message(sm) self.assertEqual(self._get_mailbox_updated(app, mbox_id), 3) def test_apps(self): rv = rendezvous.Rendezvous(get_db(":memory:"), None, None) app = rv.get_app("appid") app.allocate_nameplate("side", 121) app.prune = mock.Mock() rv.prune_all_apps(now=123, old=122) self.assertEqual(app.prune.mock_calls, [mock.call(123, 122)]) def test_nameplates(self): db = get_db(":memory:") rv = rendezvous.Rendezvous(db, None, 3600) # timestamps <=50 are "old", >=51 are "new" #OLD = "old"; NEW = "new" #when = {OLD: 1, NEW: 60} new_nameplates = set() APPID = "appid" app = rv.get_app(APPID) # Exercise the first-vs-second newness tests app.claim_nameplate("np-1", "side1", 1) app.claim_nameplate("np-2", "side1", 1) app.claim_nameplate("np-2", "side2", 2) app.claim_nameplate("np-3", "side1", 60) new_nameplates.add("np-3") app.claim_nameplate("np-4", "side1", 1) app.claim_nameplate("np-4", "side2", 60) new_nameplates.add("np-4") app.claim_nameplate("np-5", "side1", 60) app.claim_nameplate("np-5", "side2", 61) new_nameplates.add("np-5") rv.prune_all_apps(now=123, old=50) nameplates = set([row["name"] for row in db.execute("SELECT * FROM `nameplates`").fetchall()]) self.assertEqual(new_nameplates, nameplates) mailboxes = set([row["id"] for row in db.execute("SELECT * FROM `mailboxes`").fetchall()]) self.assertEqual(len(new_nameplates), len(mailboxes)) def test_mailboxes(self): db = get_db(":memory:") rv = rendezvous.Rendezvous(db, None, 3600) # timestamps <=50 are "old", >=51 are "new" #OLD = "old"; NEW = "new" #when = {OLD: 1, NEW: 60} new_mailboxes = set() APPID = "appid" app = rv.get_app(APPID) # Exercise the first-vs-second newness tests app.open_mailbox("mb-11", "side1", 1) app.open_mailbox("mb-12", "side1", 1) app.open_mailbox("mb-12", "side2", 2) app.open_mailbox("mb-13", "side1", 60) new_mailboxes.add("mb-13") app.open_mailbox("mb-14", "side1", 1) app.open_mailbox("mb-14", "side2", 60) new_mailboxes.add("mb-14") app.open_mailbox("mb-15", "side1", 60) app.open_mailbox("mb-15", "side2", 61) new_mailboxes.add("mb-15") rv.prune_all_apps(now=123, old=50) mailboxes = set([row["id"] for row in db.execute("SELECT * FROM `mailboxes`").fetchall()]) self.assertEqual(new_mailboxes, mailboxes) def test_lots(self): OLD = "old"; NEW = "new" for nameplate in [False, True]: for mailbox in [OLD, NEW]: for has_listeners in [False, True]: self.one(nameplate, mailbox, has_listeners) def test_one(self): # to debug specific problems found by test_lots self.one(None, "new", False) def one(self, nameplate, mailbox, has_listeners): desc = ("nameplate=%s, mailbox=%s, has_listeners=%s" % (nameplate, mailbox, has_listeners)) log.msg(desc) db = get_db(":memory:") rv = rendezvous.Rendezvous(db, None, 3600) APPID = "appid" app = rv.get_app(APPID) # timestamps <=50 are "old", >=51 are "new" OLD = "old"; NEW = "new" when = {OLD: 1, NEW: 60} nameplate_survives = False mailbox_survives = False mbid = "mbid" if nameplate: mbid = app.claim_nameplate("npid", "side1", when[mailbox]) mb = app.open_mailbox(mbid, "side1", when[mailbox]) # the pruning algorithm doesn't care about the age of messages, # because mailbox.updated is always updated each time we add a # message sm = SidedMessage("side1", "phase", "body", when[mailbox], "msgid") mb.add_message(sm) if has_listeners: mb.add_listener("handle", None, None) if (mailbox == NEW or has_listeners): if nameplate: nameplate_survives = True mailbox_survives = True messages_survive = mailbox_survives rv.prune_all_apps(now=123, old=50) nameplates = set([row["name"] for row in db.execute("SELECT * FROM `nameplates`").fetchall()]) self.assertEqual(nameplate_survives, bool(nameplates), ("nameplate", nameplate_survives, nameplates, desc)) mailboxes = set([row["id"] for row in db.execute("SELECT * FROM `mailboxes`").fetchall()]) self.assertEqual(mailbox_survives, bool(mailboxes), ("mailbox", mailbox_survives, mailboxes, desc)) messages = set([row["msg_id"] for row in db.execute("SELECT * FROM `messages`").fetchall()]) self.assertEqual(messages_survive, bool(messages), ("messages", messages_survive, messages, desc)) def strip_message(msg): m2 = msg.copy() m2.pop("id", None) m2.pop("server_rx", None) return m2 def strip_messages(messages): return [strip_message(m) for m in messages] class WSClient(websocket.WebSocketClientProtocol): def __init__(self): websocket.WebSocketClientProtocol.__init__(self) self.events = [] self.errors = [] self.d = None self.ping_counter = itertools.count(0) def onOpen(self): self.factory.d.callback(self) def onMessage(self, payload, isBinary): assert not isBinary event = json.loads(payload.decode("utf-8")) if event["type"] == "error": self.errors.append(event) if self.d: assert not self.events d,self.d = self.d,None d.callback(event) return self.events.append(event) def close(self): self.d = defer.Deferred() self.transport.loseConnection() return self.d def onClose(self, wasClean, code, reason): if self.d: self.d.callback((wasClean, code, reason)) def next_event(self): assert not self.d if self.events: event = self.events.pop(0) return defer.succeed(event) self.d = defer.Deferred() return self.d @inlineCallbacks def next_non_ack(self): while True: m = yield self.next_event() if isinstance(m, tuple): print("unexpected onClose", m) raise AssertionError("unexpected onClose") if m["type"] != "ack": returnValue(m) def strip_acks(self): self.events = [e for e in self.events if e["type"] != "ack"] def send(self, mtype, **kwargs): kwargs["type"] = mtype payload = json.dumps(kwargs).encode("utf-8") self.sendMessage(payload, False) def send_notype(self, **kwargs): payload = json.dumps(kwargs).encode("utf-8") self.sendMessage(payload, False) @inlineCallbacks def sync(self): ping = next(self.ping_counter) self.send("ping", ping=ping) # queue all messages until the pong, then put them back old_events = [] while True: ev = yield self.next_event() if ev["type"] == "pong" and ev["pong"] == ping: self.events = old_events + self.events returnValue(None) old_events.append(ev) class WSFactory(websocket.WebSocketClientFactory): protocol = WSClient class WSClientSync(unittest.TestCase): # make sure my 'sync' method actually works @inlineCallbacks def test_sync(self): sent = [] c = WSClient() def _send(mtype, **kwargs): sent.append( (mtype, kwargs) ) c.send = _send def add(mtype, **kwargs): kwargs["type"] = mtype c.onMessage(json.dumps(kwargs).encode("utf-8"), False) # no queued messages sunc = [] d = c.sync() d.addBoth(sunc.append) self.assertEqual(sent, [("ping", {"ping": 0})]) self.assertEqual(sunc, []) add("pong", pong=0) yield d self.assertEqual(c.events, []) # one,two,ping,pong add("one") add("two", two=2) sunc = [] d = c.sync() d.addBoth(sunc.append) add("pong", pong=1) yield d m = yield c.next_non_ack() self.assertEqual(m["type"], "one") m = yield c.next_non_ack() self.assertEqual(m["type"], "two") self.assertEqual(c.events, []) # one,ping,two,pong add("one") sunc = [] d = c.sync() d.addBoth(sunc.append) add("two", two=2) add("pong", pong=2) yield d m = yield c.next_non_ack() self.assertEqual(m["type"], "one") m = yield c.next_non_ack() self.assertEqual(m["type"], "two") self.assertEqual(c.events, []) # ping,one,two,pong sunc = [] d = c.sync() d.addBoth(sunc.append) add("one") add("two", two=2) add("pong", pong=3) yield d m = yield c.next_non_ack() self.assertEqual(m["type"], "one") m = yield c.next_non_ack() self.assertEqual(m["type"], "two") self.assertEqual(c.events, []) class WebSocketAPI(_Util, ServerBase, unittest.TestCase): def setUp(self): self._clients = [] return ServerBase.setUp(self) def tearDown(self): for c in self._clients: c.transport.loseConnection() return ServerBase.tearDown(self) @inlineCallbacks def make_client(self): f = WSFactory(self.relayurl) f.d = defer.Deferred() reactor.connectTCP("127.0.0.1", self.rdv_ws_port, f) c = yield f.d self._clients.append(c) returnValue(c) def check_welcome(self, data): self.failUnlessIn("welcome", data) self.failUnlessEqual(data["welcome"], {"current_cli_version": __version__}) @inlineCallbacks def test_welcome(self): c1 = yield self.make_client() msg = yield c1.next_non_ack() self.check_welcome(msg) self.assertEqual(self._rendezvous._apps, {}) @inlineCallbacks def test_bind(self): c1 = yield self.make_client() yield c1.next_non_ack() c1.send("bind", appid="appid") # missing side= err = yield c1.next_non_ack() self.assertEqual(err["type"], "error") self.assertEqual(err["error"], "bind requires 'side'") c1.send("bind", side="side") # missing appid= err = yield c1.next_non_ack() self.assertEqual(err["type"], "error") self.assertEqual(err["error"], "bind requires 'appid'") c1.send("bind", appid="appid", side="side") yield c1.sync() self.assertEqual(list(self._rendezvous._apps.keys()), ["appid"]) c1.send("bind", appid="appid", side="side") # duplicate err = yield c1.next_non_ack() self.assertEqual(err["type"], "error") self.assertEqual(err["error"], "already bound") c1.send_notype(other="misc") # missing 'type' err = yield c1.next_non_ack() self.assertEqual(err["type"], "error") self.assertEqual(err["error"], "missing 'type'") c1.send("___unknown") # unknown type err = yield c1.next_non_ack() self.assertEqual(err["type"], "error") self.assertEqual(err["error"], "unknown type") c1.send("ping") # missing 'ping' err = yield c1.next_non_ack() self.assertEqual(err["type"], "error") self.assertEqual(err["error"], "ping requires 'ping'") @inlineCallbacks def test_list(self): c1 = yield self.make_client() yield c1.next_non_ack() c1.send("list") # too early, must bind first err = yield c1.next_non_ack() self.assertEqual(err["type"], "error") self.assertEqual(err["error"], "must bind first") c1.send("bind", appid="appid", side="side") c1.send("list") m = yield c1.next_non_ack() self.assertEqual(m["type"], "nameplates") self.assertEqual(m["nameplates"], []) app = self._rendezvous.get_app("appid") nameplate_id1 = app.allocate_nameplate("side", 0) app.claim_nameplate("np2", "side", 0) c1.send("list") m = yield c1.next_non_ack() self.assertEqual(m["type"], "nameplates") nids = set() for n in m["nameplates"]: self.assertEqual(type(n), dict) self.assertEqual(list(n.keys()), ["id"]) nids.add(n["id"]) self.assertEqual(nids, set([nameplate_id1, "np2"])) @inlineCallbacks def test_allocate(self): c1 = yield self.make_client() yield c1.next_non_ack() c1.send("allocate") # too early, must bind first err = yield c1.next_non_ack() self.assertEqual(err["type"], "error") self.assertEqual(err["error"], "must bind first") c1.send("bind", appid="appid", side="side") app = self._rendezvous.get_app("appid") c1.send("allocate") m = yield c1.next_non_ack() self.assertEqual(m["type"], "allocated") name = m["nameplate"] nids = app.get_nameplate_ids() self.assertEqual(len(nids), 1) self.assertEqual(name, list(nids)[0]) c1.send("allocate") err = yield c1.next_non_ack() self.assertEqual(err["type"], "error") self.assertEqual(err["error"], "you already allocated one, don't be greedy") c1.send("claim", nameplate=name) # allocate+claim is ok yield c1.sync() np_row, side_rows = self._nameplate(app, name) self.assertEqual(len(side_rows), 1) self.assertEqual(side_rows[0]["side"], "side") @inlineCallbacks def test_claim(self): c1 = yield self.make_client() yield c1.next_non_ack() c1.send("bind", appid="appid", side="side") app = self._rendezvous.get_app("appid") c1.send("claim") # missing nameplate= err = yield c1.next_non_ack() self.assertEqual(err["type"], "error") self.assertEqual(err["error"], "claim requires 'nameplate'") c1.send("claim", nameplate="np1") m = yield c1.next_non_ack() self.assertEqual(m["type"], "claimed") mailbox_id = m["mailbox"] self.assertEqual(type(mailbox_id), type("")) c1.send("claim", nameplate="np1") err = yield c1.next_non_ack() self.assertEqual(err["type"], "error", err) self.assertEqual(err["error"], "only one claim per connection") nids = app.get_nameplate_ids() self.assertEqual(len(nids), 1) self.assertEqual("np1", list(nids)[0]) np_row, side_rows = self._nameplate(app, "np1") self.assertEqual(len(side_rows), 1) self.assertEqual(side_rows[0]["side"], "side") # claiming a nameplate assigns a random mailbox id and creates the # mailbox row mailboxes = app._db.execute("SELECT * FROM `mailboxes`" " WHERE `app_id`='appid'").fetchall() self.assertEqual(len(mailboxes), 1) @inlineCallbacks def test_claim_crowded(self): c1 = yield self.make_client() yield c1.next_non_ack() c1.send("bind", appid="appid", side="side") app = self._rendezvous.get_app("appid") app.claim_nameplate("np1", "side1", 0) app.claim_nameplate("np1", "side2", 0) # the third claim will signal crowding c1.send("claim", nameplate="np1") err = yield c1.next_non_ack() self.assertEqual(err["type"], "error") self.assertEqual(err["error"], "crowded") @inlineCallbacks def test_release(self): c1 = yield self.make_client() yield c1.next_non_ack() c1.send("bind", appid="appid", side="side") app = self._rendezvous.get_app("appid") app.claim_nameplate("np1", "side2", 0) c1.send("release") # didn't do claim first err = yield c1.next_non_ack() self.assertEqual(err["type"], "error") self.assertEqual(err["error"], "release without nameplate must follow claim") c1.send("claim", nameplate="np1") yield c1.next_non_ack() c1.send("release") m = yield c1.next_non_ack() self.assertEqual(m["type"], "released", m) np_row, side_rows = self._nameplate(app, "np1") claims = [(row["side"], row["claimed"]) for row in side_rows] self.assertIn(("side", False), claims) self.assertIn(("side2", True), claims) c1.send("release") # no longer claimed err = yield c1.next_non_ack() self.assertEqual(err["type"], "error") self.assertEqual(err["error"], "only one release per connection") @inlineCallbacks def test_release_named(self): c1 = yield self.make_client() yield c1.next_non_ack() c1.send("bind", appid="appid", side="side") c1.send("claim", nameplate="np1") yield c1.next_non_ack() c1.send("release", nameplate="np1") m = yield c1.next_non_ack() self.assertEqual(m["type"], "released", m) @inlineCallbacks def test_release_named_ignored(self): c1 = yield self.make_client() yield c1.next_non_ack() c1.send("bind", appid="appid", side="side") c1.send("release", nameplate="np1") # didn't do claim first, ignored m = yield c1.next_non_ack() self.assertEqual(m["type"], "released", m) @inlineCallbacks def test_release_named_mismatch(self): c1 = yield self.make_client() yield c1.next_non_ack() c1.send("bind", appid="appid", side="side") c1.send("claim", nameplate="np1") yield c1.next_non_ack() c1.send("release", nameplate="np2") # mismatching nameplate err = yield c1.next_non_ack() self.assertEqual(err["type"], "error") self.assertEqual(err["error"], "release and claim must use same nameplate") @inlineCallbacks def test_open(self): c1 = yield self.make_client() yield c1.next_non_ack() c1.send("bind", appid="appid", side="side") app = self._rendezvous.get_app("appid") c1.send("open") # missing mailbox= err = yield c1.next_non_ack() self.assertEqual(err["type"], "error") self.assertEqual(err["error"], "open requires 'mailbox'") mb1 = app.open_mailbox("mb1", "side2", 0) mb1.add_message(SidedMessage(side="side2", phase="phase", body="body", server_rx=0, msg_id="msgid")) c1.send("open", mailbox="mb1") m = yield c1.next_non_ack() self.assertEqual(m["type"], "message") self.assertEqual(m["body"], "body") self.assertTrue(mb1.has_listeners()) mb1.add_message(SidedMessage(side="side2", phase="phase2", body="body2", server_rx=0, msg_id="msgid")) m = yield c1.next_non_ack() self.assertEqual(m["type"], "message") self.assertEqual(m["body"], "body2") c1.send("open", mailbox="mb1") err = yield c1.next_non_ack() self.assertEqual(err["type"], "error") self.assertEqual(err["error"], "only one open per connection") @inlineCallbacks def test_add(self): c1 = yield self.make_client() yield c1.next_non_ack() c1.send("bind", appid="appid", side="side") app = self._rendezvous.get_app("appid") mb1 = app.open_mailbox("mb1", "side2", 0) l1 = []; stop1 = []; stop1_f = lambda: stop1.append(True) mb1.add_listener("handle1", l1.append, stop1_f) c1.send("add") # didn't open first err = yield c1.next_non_ack() self.assertEqual(err["type"], "error") self.assertEqual(err["error"], "must open mailbox before adding") c1.send("open", mailbox="mb1") c1.send("add", body="body") # missing phase= err = yield c1.next_non_ack() self.assertEqual(err["type"], "error") self.assertEqual(err["error"], "missing 'phase'") c1.send("add", phase="phase") # missing body= err = yield c1.next_non_ack() self.assertEqual(err["type"], "error") self.assertEqual(err["error"], "missing 'body'") c1.send("add", phase="phase", body="body") m = yield c1.next_non_ack() # echoed back self.assertEqual(m["type"], "message") self.assertEqual(m["body"], "body") self.assertEqual(len(l1), 1) self.assertEqual(l1[0].body, "body") @inlineCallbacks def test_close(self): c1 = yield self.make_client() yield c1.next_non_ack() c1.send("bind", appid="appid", side="side") app = self._rendezvous.get_app("appid") c1.send("close", mood="mood") # must open first err = yield c1.next_non_ack() self.assertEqual(err["type"], "error") self.assertEqual(err["error"], "close without mailbox must follow open") c1.send("open", mailbox="mb1") yield c1.sync() mb1 = app._mailboxes["mb1"] self.assertTrue(mb1.has_listeners()) c1.send("close", mood="mood") m = yield c1.next_non_ack() self.assertEqual(m["type"], "closed") self.assertFalse(mb1.has_listeners()) c1.send("close", mood="mood") # already closed err = yield c1.next_non_ack() self.assertEqual(err["type"], "error", m) self.assertEqual(err["error"], "only one close per connection") @inlineCallbacks def test_close_named(self): c1 = yield self.make_client() yield c1.next_non_ack() c1.send("bind", appid="appid", side="side") c1.send("open", mailbox="mb1") yield c1.sync() c1.send("close", mailbox="mb1", mood="mood") m = yield c1.next_non_ack() self.assertEqual(m["type"], "closed") @inlineCallbacks def test_close_named_ignored(self): c1 = yield self.make_client() yield c1.next_non_ack() c1.send("bind", appid="appid", side="side") c1.send("close", mailbox="mb1", mood="mood") # no open first, ignored m = yield c1.next_non_ack() self.assertEqual(m["type"], "closed") @inlineCallbacks def test_close_named_mismatch(self): c1 = yield self.make_client() yield c1.next_non_ack() c1.send("bind", appid="appid", side="side") c1.send("open", mailbox="mb1") yield c1.sync() c1.send("close", mailbox="mb2", mood="mood") err = yield c1.next_non_ack() self.assertEqual(err["type"], "error") self.assertEqual(err["error"], "open and close must use same mailbox") @inlineCallbacks def test_disconnect(self): c1 = yield self.make_client() yield c1.next_non_ack() c1.send("bind", appid="appid", side="side") app = self._rendezvous.get_app("appid") c1.send("open", mailbox="mb1") yield c1.sync() mb1 = app._mailboxes["mb1"] self.assertTrue(mb1.has_listeners()) yield c1.close() # wait for the server to notice the socket has closed started = time.time() while mb1.has_listeners() and (time.time()-started < 5.0): d = defer.Deferred() reactor.callLater(0.01, d.callback, None) yield d self.assertFalse(mb1.has_listeners()) @inlineCallbacks def test_interrupted_client_nameplate(self): # a client's interactions with the server might be split over # multiple sequential WebSocket connections, e.g. when the server is # bounced and the client reconnects, or vice versa c = yield self.make_client() yield c.next_non_ack() c.send("bind", appid="appid", side="side") app = self._rendezvous.get_app("appid") c.send("claim", nameplate="np1") m = yield c.next_non_ack() self.assertEqual(m["type"], "claimed") mailbox_id = m["mailbox"] self.assertEqual(type(mailbox_id), type("")) np_row, side_rows = self._nameplate(app, "np1") claims = [(row["side"], row["claimed"]) for row in side_rows] self.assertEqual(claims, [("side", True)]) c.close() yield c.d c = yield self.make_client() yield c.next_non_ack() c.send("bind", appid="appid", side="side") c.send("claim", nameplate="np1") # idempotent m = yield c.next_non_ack() self.assertEqual(m["type"], "claimed") self.assertEqual(m["mailbox"], mailbox_id) # mailbox id is stable np_row, side_rows = self._nameplate(app, "np1") claims = [(row["side"], row["claimed"]) for row in side_rows] self.assertEqual(claims, [("side", True)]) c.close() yield c.d c = yield self.make_client() yield c.next_non_ack() c.send("bind", appid="appid", side="side") # we haven't done a claim with this particular connection, but we can # still send a release as long as we include the nameplate c.send("release", nameplate="np1") # release-without-claim m = yield c.next_non_ack() self.assertEqual(m["type"], "released") np_row, side_rows = self._nameplate(app, "np1") self.assertEqual(np_row, None) c.close() yield c.d c = yield self.make_client() yield c.next_non_ack() c.send("bind", appid="appid", side="side") # and the release is idempotent, when done on separate connections c.send("release", nameplate="np1") m = yield c.next_non_ack() self.assertEqual(m["type"], "released") np_row, side_rows = self._nameplate(app, "np1") self.assertEqual(np_row, None) c.close() yield c.d @inlineCallbacks def test_interrupted_client_nameplate_reclaimed(self): c = yield self.make_client() yield c.next_non_ack() c.send("bind", appid="appid", side="side") app = self._rendezvous.get_app("appid") # a new claim on a previously-closed nameplate is forbidden. We make # a new nameplate here and manually open a second claim on it, so the # nameplate stays alive long enough for the code check to happen. c = yield self.make_client() yield c.next_non_ack() c.send("bind", appid="appid", side="side") c.send("claim", nameplate="np2") m = yield c.next_non_ack() self.assertEqual(m["type"], "claimed") app.claim_nameplate("np2", "side2", 0) c.send("release", nameplate="np2") m = yield c.next_non_ack() self.assertEqual(m["type"], "released") np_row, side_rows = self._nameplate(app, "np2") claims = sorted([(row["side"], row["claimed"]) for row in side_rows]) self.assertEqual(claims, [("side", 0), ("side2", 1)]) c.close() yield c.d c = yield self.make_client() yield c.next_non_ack() c.send("bind", appid="appid", side="side") c.send("claim", nameplate="np2") # new claim is forbidden err = yield c.next_non_ack() self.assertEqual(err["type"], "error") self.assertEqual(err["error"], "reclaimed") np_row, side_rows = self._nameplate(app, "np2") claims = sorted([(row["side"], row["claimed"]) for row in side_rows]) self.assertEqual(claims, [("side", 0), ("side2", 1)]) c.close() yield c.d @inlineCallbacks def test_interrupted_client_mailbox(self): # a client's interactions with the server might be split over # multiple sequential WebSocket connections, e.g. when the server is # bounced and the client reconnects, or vice versa c = yield self.make_client() yield c.next_non_ack() c.send("bind", appid="appid", side="side") app = self._rendezvous.get_app("appid") mb1 = app.open_mailbox("mb1", "side2", 0) mb1.add_message(SidedMessage(side="side2", phase="phase", body="body", server_rx=0, msg_id="msgid")) c.send("open", mailbox="mb1") m = yield c.next_non_ack() self.assertEqual(m["type"], "message") self.assertEqual(m["body"], "body") self.assertTrue(mb1.has_listeners()) c.close() yield c.d c = yield self.make_client() yield c.next_non_ack() c.send("bind", appid="appid", side="side") # open should be idempotent c.send("open", mailbox="mb1") m = yield c.next_non_ack() self.assertEqual(m["type"], "message") self.assertEqual(m["body"], "body") mb_row, side_rows = self._mailbox(app, "mb1") openeds = [(row["side"], row["opened"]) for row in side_rows] self.assertIn(("side", 1), openeds) # TODO: why 1, and not True? # close on the same connection as open is ok c.send("close", mailbox="mb1", mood="mood") m = yield c.next_non_ack() self.assertEqual(m["type"], "closed", m) mb_row, side_rows = self._mailbox(app, "mb1") openeds = [(row["side"], row["opened"]) for row in side_rows] self.assertIn(("side", 0), openeds) c.close() yield c.d # close (on a separate connection) is idempotent c = yield self.make_client() yield c.next_non_ack() c.send("bind", appid="appid", side="side") c.send("close", mailbox="mb1", mood="mood") m = yield c.next_non_ack() self.assertEqual(m["type"], "closed", m) mb_row, side_rows = self._mailbox(app, "mb1") openeds = [(row["side"], row["opened"]) for row in side_rows] self.assertIn(("side", 0), openeds) c.close() yield c.d class Summary(unittest.TestCase): def test_mailbox(self): app = rendezvous.AppNamespace(None, None, False, None) # starts at time 1, maybe gets second open at time 3, closes at 5 def s(rows, pruned=False): return app._summarize_mailbox(rows, 5, pruned) rows = [dict(added=1)] self.assertEqual(s(rows), Usage(1, None, 4, "lonely")) rows = [dict(added=1, mood="lonely")] self.assertEqual(s(rows), Usage(1, None, 4, "lonely")) rows = [dict(added=1, mood="errory")] self.assertEqual(s(rows), Usage(1, None, 4, "errory")) rows = [dict(added=1, mood=None)] self.assertEqual(s(rows, pruned=True), Usage(1, None, 4, "pruney")) rows = [dict(added=1, mood="happy")] self.assertEqual(s(rows, pruned=True), Usage(1, None, 4, "pruney")) rows = [dict(added=1, mood="happy"), dict(added=3, mood="happy")] self.assertEqual(s(rows), Usage(1, 2, 4, "happy")) rows = [dict(added=1, mood="errory"), dict(added=3, mood="happy")] self.assertEqual(s(rows), Usage(1, 2, 4, "errory")) rows = [dict(added=1, mood="happy"), dict(added=3, mood="errory")] self.assertEqual(s(rows), Usage(1, 2, 4, "errory")) rows = [dict(added=1, mood="scary"), dict(added=3, mood="happy")] self.assertEqual(s(rows), Usage(1, 2, 4, "scary")) rows = [dict(added=1, mood="scary"), dict(added=3, mood="errory")] self.assertEqual(s(rows), Usage(1, 2, 4, "scary")) rows = [dict(added=1, mood="happy"), dict(added=3, mood=None)] self.assertEqual(s(rows, pruned=True), Usage(1, 2, 4, "pruney")) rows = [dict(added=1, mood="happy"), dict(added=3, mood="happy")] self.assertEqual(s(rows, pruned=True), Usage(1, 2, 4, "pruney")) rows = [dict(added=1), dict(added=3), dict(added=4)] self.assertEqual(s(rows), Usage(1, 2, 4, "crowded")) rows = [dict(added=1), dict(added=3), dict(added=4)] self.assertEqual(s(rows, pruned=True), Usage(1, 2, 4, "crowded")) def test_nameplate(self): a = rendezvous.AppNamespace(None, None, False, None) # starts at time 1, maybe gets second open at time 3, closes at 5 def s(rows, pruned=False): return a._summarize_nameplate_usage(rows, 5, pruned) rows = [dict(added=1)] self.assertEqual(s(rows), Usage(1, None, 4, "lonely")) rows = [dict(added=1), dict(added=3)] self.assertEqual(s(rows), Usage(1, 2, 4, "happy")) rows = [dict(added=1), dict(added=3)] self.assertEqual(s(rows, pruned=True), Usage(1, 2, 4, "pruney")) rows = [dict(added=1), dict(added=3), dict(added=4)] self.assertEqual(s(rows), Usage(1, 2, 4, "crowded")) def test_blur(self): db = get_db(":memory:") rv = rendezvous.Rendezvous(db, None, 3600) APPID = "appid" app = rv.get_app(APPID) app.claim_nameplate("npid", "side1", 10) # start time is 10 rv.prune_all_apps(now=123, old=50) # start time should be rounded to top of the hour (blur_usage=3600) row = db.execute("SELECT * FROM `nameplate_usage`").fetchone() self.assertEqual(row["started"], 0) app = rv.get_app(APPID) app.open_mailbox("mbid", "side1", 20) # start time is 20 rv.prune_all_apps(now=123, old=50) row = db.execute("SELECT * FROM `mailbox_usage`").fetchone() self.assertEqual(row["started"], 0) def test_no_blur(self): db = get_db(":memory:") rv = rendezvous.Rendezvous(db, None, None) APPID = "appid" app = rv.get_app(APPID) app.claim_nameplate("npid", "side1", 10) # start time is 10 rv.prune_all_apps(now=123, old=50) row = db.execute("SELECT * FROM `nameplate_usage`").fetchone() self.assertEqual(row["started"], 10) db.execute("DELETE FROM `mailbox_usage`") db.commit() app = rv.get_app(APPID) app.open_mailbox("mbid", "side1", 20) # start time is 20 rv.prune_all_apps(now=123, old=50) row = db.execute("SELECT * FROM `mailbox_usage`").fetchone() self.assertEqual(row["started"], 20) class DumpStats(unittest.TestCase): def test_nostats(self): rs = server.RelayServer(str("tcp:0"), str("tcp:0"), None) # with no ._stats_file, this should do nothing rs.dump_stats(1, 1) def test_empty(self): basedir = self.mktemp() os.mkdir(basedir) fn = os.path.join(basedir, "stats.json") rs = server.RelayServer(str("tcp:0"), str("tcp:0"), None, stats_file=fn) now = 1234 validity = 500 rs.dump_stats(now, validity) with open(fn, "rb") as f: data_bytes = f.read() data = json.loads(data_bytes.decode("utf-8")) self.assertEqual(data["created"], now) self.assertEqual(data["valid_until"], now+validity) self.assertEqual(data["rendezvous"]["all_time"]["mailboxes_total"], 0) self.assertEqual(data["transit"]["all_time"]["total"], 0) magic-wormhole-0.9.1/src/wormhole/test/test_ssh.py0000644000076600000240000000524113024532264022670 0ustar warnerstaff00000000000000import os, io import mock from twisted.trial import unittest from ..cli import cmd_ssh OTHERS = ["config", "config~", "known_hosts", "known_hosts~"] class FindPubkey(unittest.TestCase): def test_find_one(self): files = OTHERS + ["id_rsa.pub", "id_rsa"] pubkey_data = u"ssh-rsa AAAAkeystuff email@host\n" pubkey_file = io.StringIO(pubkey_data) with mock.patch("wormhole.cli.cmd_ssh.exists", return_value=True): with mock.patch("os.listdir", return_value=files) as ld: with mock.patch("wormhole.cli.cmd_ssh.open", return_value=pubkey_file): res = cmd_ssh.find_public_key() self.assertEqual(ld.mock_calls, [mock.call(os.path.expanduser("~/.ssh/"))]) self.assertEqual(len(res), 3, res) kind, keyid, pubkey = res self.assertEqual(kind, "ssh-rsa") self.assertEqual(keyid, "email@host") self.assertEqual(pubkey, pubkey_data) def test_find_none(self): files = OTHERS # no pubkey with mock.patch("wormhole.cli.cmd_ssh.exists", return_value=True): with mock.patch("os.listdir", return_value=files): e = self.assertRaises(cmd_ssh.PubkeyError, cmd_ssh.find_public_key) dot_ssh = os.path.expanduser("~/.ssh/") self.assertEqual(str(e), "No public keys in '{}'".format(dot_ssh)) def test_bad_hint(self): with mock.patch("wormhole.cli.cmd_ssh.exists", return_value=False): e = self.assertRaises(cmd_ssh.PubkeyError, cmd_ssh.find_public_key, hint="bogus/path") self.assertEqual(str(e), "Can't find 'bogus/path'") def test_find_multiple(self): files = OTHERS + ["id_rsa.pub", "id_rsa", "id_dsa.pub", "id_dsa"] pubkey_data = u"ssh-rsa AAAAkeystuff email@host\n" pubkey_file = io.StringIO(pubkey_data) with mock.patch("wormhole.cli.cmd_ssh.exists", return_value=True): with mock.patch("os.listdir", return_value=files): responses = iter(["frog", "NaN", "-1", "0"]) with mock.patch("click.prompt", side_effect=lambda p: next(responses)): with mock.patch("wormhole.cli.cmd_ssh.open", return_value=pubkey_file): res = cmd_ssh.find_public_key() self.assertEqual(len(res), 3, res) kind, keyid, pubkey = res self.assertEqual(kind, "ssh-rsa") self.assertEqual(keyid, "email@host") self.assertEqual(pubkey, pubkey_data) magic-wormhole-0.9.1/src/wormhole/test/test_transit.py0000644000076600000240000016266713031644470023600 0ustar warnerstaff00000000000000from __future__ import print_function, unicode_literals import io import gc import mock from binascii import hexlify, unhexlify from collections import namedtuple from twisted.trial import unittest from twisted.internet import defer, task, endpoints, protocol, address, error from twisted.internet.defer import gatherResults, inlineCallbacks from twisted.python import log, failure from twisted.test import proto_helpers from ..errors import InternalError from .. import transit from ..server import transit_server from .common import ServerBase from nacl.secret import SecretBox from nacl.exceptions import CryptoError class Highlander(unittest.TestCase): def test_one_winner(self): cancelled = set() contenders = [defer.Deferred(lambda d: cancelled.add(i)) for i in range(4)] result = [] d = transit.there_can_be_only_one(contenders) d.addBoth(result.append) self.assertEqual(result, []) contenders[0].errback(ValueError()) self.assertEqual(result, []) contenders[1].errback(TypeError()) self.assertEqual(result, []) contenders[2].callback("yay") self.assertEqual(result, ["yay"]) self.assertEqual(cancelled, set([3])) def test_there_might_also_be_none(self): cancelled = set() contenders = [defer.Deferred(lambda d: cancelled.add(i)) for i in range(4)] result = [] d = transit.there_can_be_only_one(contenders) d.addBoth(result.append) self.assertEqual(result, []) contenders[0].errback(ValueError()) self.assertEqual(result, []) contenders[1].errback(TypeError()) self.assertEqual(result, []) contenders[2].errback(TypeError()) self.assertEqual(result, []) contenders[3].errback(NameError()) self.assertEqual(len(result), 1) f = result[0] self.assertIsInstance(f.value, ValueError) # first failure is recorded self.assertEqual(cancelled, set()) def test_cancel_early(self): cancelled = set() contenders = [defer.Deferred(lambda d, i=i: cancelled.add(i)) for i in range(4)] result = [] d = transit.there_can_be_only_one(contenders) d.addBoth(result.append) self.assertEqual(result, []) self.assertEqual(cancelled, set()) d.cancel() self.assertEqual(len(result), 1) self.assertIsInstance(result[0].value, defer.CancelledError) self.assertEqual(cancelled, set(range(4))) def test_cancel_after_one_failure(self): cancelled = set() contenders = [defer.Deferred(lambda d, i=i: cancelled.add(i)) for i in range(4)] result = [] d = transit.there_can_be_only_one(contenders) d.addBoth(result.append) self.assertEqual(result, []) self.assertEqual(cancelled, set()) contenders[0].errback(ValueError()) d.cancel() self.assertEqual(len(result), 1) self.assertIsInstance(result[0].value, ValueError) self.assertEqual(cancelled, set([1,2,3])) class Forever(unittest.TestCase): def _forever_setup(self): clock = task.Clock() c = transit.Common("", reactor=clock) cancelled = [] result = [] d0 = defer.Deferred(cancelled.append) d = c._not_forever(1.0, d0) d.addBoth(result.append) return c, clock, d0, d, cancelled, result def test_not_forever_fires(self): c, clock, d0, d, cancelled, result = self._forever_setup() self.assertEqual((result, cancelled), ([], [])) d.callback(1) self.assertEqual((result, cancelled), ([1], [])) self.assertNot(clock.getDelayedCalls()) def test_not_forever_errs(self): c, clock, d0, d, cancelled, result = self._forever_setup() self.assertEqual((result, cancelled), ([], [])) d.errback(ValueError()) self.assertEqual(cancelled, []) self.assertEqual(len(result), 1) self.assertIsInstance(result[0].value, ValueError) self.assertNot(clock.getDelayedCalls()) def test_not_forever_cancel_early(self): c, clock, d0, d, cancelled, result = self._forever_setup() self.assertEqual((result, cancelled), ([], [])) d.cancel() self.assertEqual(cancelled, [d0]) self.assertEqual(len(result), 1) self.assertIsInstance(result[0].value, defer.CancelledError) self.assertNot(clock.getDelayedCalls()) def test_not_forever_timeout(self): c, clock, d0, d, cancelled, result = self._forever_setup() self.assertEqual((result, cancelled), ([], [])) clock.advance(2.0) self.assertEqual(cancelled, [d0]) self.assertEqual(len(result), 1) self.assertIsInstance(result[0].value, defer.CancelledError) self.assertNot(clock.getDelayedCalls()) class Misc(unittest.TestCase): def test_allocate_port(self): portno = transit.allocate_tcp_port() self.assertIsInstance(portno, int) UnknownHint = namedtuple("UnknownHint", ["stuff"]) class Hints(unittest.TestCase): def test_endpoint_from_hint_obj(self): c = transit.Common("") efho = c._endpoint_from_hint_obj self.assertIsInstance(efho(transit.DirectTCPV1Hint("host", 1234, 0.0)), endpoints.HostnameEndpoint) self.assertEqual(efho("unknown:stuff:yowza:pivlor"), None) # c._tor_manager is currently None self.assertEqual(efho(transit.TorTCPV1Hint("host", "port", 0)), None) c._tor_manager = mock.Mock() def tor_ep(hostname, port): if hostname == "non-public": return None return ("tor_ep", hostname, port) c._tor_manager.get_endpoint_for = mock.Mock(side_effect=tor_ep) self.assertEqual(efho(transit.DirectTCPV1Hint("host", 1234, 0.0)), ("tor_ep", "host", 1234)) self.assertEqual(efho(transit.TorTCPV1Hint("host2.onion", 1234, 0.0)), ("tor_ep", "host2.onion", 1234)) self.assertEqual(efho(transit.DirectTCPV1Hint("non-public", 1234, 0.0)), None) self.assertEqual(efho(UnknownHint("foo")), None) def test_comparable(self): h1 = transit.DirectTCPV1Hint("hostname", "port1", 0.0) h1b = transit.DirectTCPV1Hint("hostname", "port1", 0.0) h2 = transit.DirectTCPV1Hint("hostname", "port2", 0.0) r1 = transit.RelayV1Hint(tuple(sorted([h1, h2]))) r2 = transit.RelayV1Hint(tuple(sorted([h2, h1]))) r3 = transit.RelayV1Hint(tuple(sorted([h1b, h2]))) self.assertEqual(r1, r2) self.assertEqual(r2, r3) self.assertEqual(len(set([r1, r2, r3])), 1) def test_parse_tcp_v1_hint(self): c = transit.Common("") p = c._parse_tcp_v1_hint self.assertEqual(p({"type": "unknown"}), None) h = p({"type": "direct-tcp-v1", "hostname": "foo", "port": 1234}) self.assertEqual(h, transit.DirectTCPV1Hint("foo", 1234, 0.0)) h = p({"type": "direct-tcp-v1", "hostname": "foo", "port": 1234, "priority": 2.5}) self.assertEqual(h, transit.DirectTCPV1Hint("foo", 1234, 2.5)) h = p({"type": "tor-tcp-v1", "hostname": "foo", "port": 1234}) self.assertEqual(h, transit.TorTCPV1Hint("foo", 1234, 0.0)) h = p({"type": "tor-tcp-v1", "hostname": "foo", "port": 1234, "priority": 2.5}) self.assertEqual(h, transit.TorTCPV1Hint("foo", 1234, 2.5)) self.assertEqual(p({"type": "direct-tcp-v1"}), None) # missing hostname self.assertEqual(p({"type": "direct-tcp-v1", "hostname": 12}), None) # invalid hostname self.assertEqual(p({"type": "direct-tcp-v1", "hostname": "foo"}), None) # missing port self.assertEqual(p({"type": "direct-tcp-v1", "hostname": "foo", "port": "not a number"}), None) # invalid port def test_parse_hint_argv(self): def p(hint): stderr = io.StringIO() value = transit.parse_hint_argv(hint, stderr=stderr) return value, stderr.getvalue() h,stderr = p("tcp:host:1234") self.assertEqual(h, transit.DirectTCPV1Hint("host", 1234, 0.0)) self.assertEqual(stderr, "") h,stderr = p("tcp:host:1234:priority=2.6") self.assertEqual(h, transit.DirectTCPV1Hint("host", 1234, 2.6)) self.assertEqual(stderr, "") h,stderr = p("$!@#^") self.assertEqual(h, None) self.assertEqual(stderr, "unparseable hint '$!@#^'\n") h,stderr = p("unknown:stuff") self.assertEqual(h, None) self.assertEqual(stderr, "unknown hint type 'unknown' in 'unknown:stuff'\n") h,stderr = p("tcp:just-a-hostname") self.assertEqual(h, None) self.assertEqual(stderr, "unparseable TCP hint (need more colons) 'tcp:just-a-hostname'\n") h,stderr = p("tcp:host:number") self.assertEqual(h, None) self.assertEqual(stderr, "non-numeric port in TCP hint 'tcp:host:number'\n") h,stderr = p("tcp:host:1234:priority=bad") self.assertEqual(h, None) self.assertEqual(stderr, "non-float priority= in TCP hint 'tcp:host:1234:priority=bad'\n") def test_describe_hint_obj(self): d = transit.describe_hint_obj self.assertEqual(d(transit.DirectTCPV1Hint("host", 1234, 0.0)), "tcp:host:1234") self.assertEqual(d(transit.TorTCPV1Hint("host", 1234, 0.0)), "tor:host:1234") self.assertEqual(d(UnknownHint("stuff")), str(UnknownHint("stuff"))) class Basic(unittest.TestCase): @inlineCallbacks def test_relay_hints(self): URL = "tcp:host:1234" c = transit.Common(URL, no_listen=True) hints = yield c.get_connection_hints() self.assertEqual(hints, [{"type": "relay-v1", "hints": [{"type": "direct-tcp-v1", "hostname": "host", "port": 1234, "priority": 0.0}], }]) self.assertRaises(InternalError, transit.Common, 123) @inlineCallbacks def test_no_relay_hints(self): c = transit.Common(None, no_listen=True) hints = yield c.get_connection_hints() self.assertEqual(hints, []) def test_ignore_bad_hints(self): c = transit.Common("") c.add_connection_hints([{"type": "unknown"}]) c.add_connection_hints([{"type": "relay-v1", "hints": [{"type": "unknown"}]}]) self.assertEqual(c._their_direct_hints, []) self.assertEqual(c._our_relay_hints, set()) def test_ignore_localhost_hint(self): # this actually starts the listener c = transit.TransitSender("") results = [] d = c.get_connection_hints() d.addBoth(results.append) hints = results[0] c._stop_listening() # If there are non-localhost hints, then localhost hints should be # removed. But if the only hint is localhost, it should stay. if len(hints) == 1: if hints[0]["hostname"] == "127.0.0.1": return for hint in hints: self.assertFalse(hint["hostname"] == "127.0.0.1") def test_transit_key_wait(self): KEY = b"123" c = transit.Common("") results = [] d = c._get_transit_key() d.addBoth(results.append) self.assertEqual(results, []) c.set_transit_key(KEY) self.assertEqual(results, [KEY]) def test_transit_key_already_set(self): KEY = b"123" c = transit.Common("") c.set_transit_key(KEY) results = [] d = c._get_transit_key() d.addBoth(results.append) self.assertEqual(results, [KEY]) def test_transit_keys(self): KEY = b"123" s = transit.TransitSender("") s.set_transit_key(KEY) r = transit.TransitReceiver("") r.set_transit_key(KEY) self.assertEqual(s._send_this(), b"transit sender 559bdeae4b49fa6a23378d2b68f4c7e69378615d4af049c371c6a26e82391089 ready\n\n") self.assertEqual(s._send_this(), r._expect_this()) self.assertEqual(r._send_this(), b"transit receiver ed447528194bac4c00d0c854b12a97ce51413d89aa74d6304475f516fdc23a1b ready\n\n") self.assertEqual(r._send_this(), s._expect_this()) self.assertEqual(hexlify(s._sender_record_key()), b"5a2fba3a9e524ab2e2823ff53b05f946896f6e4ce4e282ffd8e3ac0e5e9e0cda") self.assertEqual(hexlify(s._sender_record_key()), hexlify(r._receiver_record_key())) self.assertEqual(hexlify(r._sender_record_key()), b"eedb143117249f45b39da324decf6bd9aae33b7ccd58487436de611a3c6b871d") self.assertEqual(hexlify(r._sender_record_key()), hexlify(s._receiver_record_key())) def test_connection_ready(self): s = transit.TransitSender("") self.assertEqual(s.connection_ready("p1"), "go") self.assertEqual(s._winner, "p1") self.assertEqual(s.connection_ready("p2"), "nevermind") self.assertEqual(s._winner, "p1") r = transit.TransitReceiver("") self.assertEqual(r.connection_ready("p1"), "wait-for-decision") self.assertEqual(r.connection_ready("p2"), "wait-for-decision") class Listener(unittest.TestCase): def test_listener(self): c = transit.Common("") hints, ep = c._build_listener() self.assertIsInstance(hints, (list, set)) if hints: self.assertIsInstance(hints[0], transit.DirectTCPV1Hint) self.assertIsInstance(ep, endpoints.TCP4ServerEndpoint) def test_get_direct_hints(self): # this actually starts the listener c = transit.TransitSender("") results = [] d = c.get_connection_hints() d.addBoth(results.append) self.assertEqual(len(results), 1) hints = results[0] # the hints are supposed to be cached, so calling this twice won't # start a second listener self.assert_(c._listener) results = [] d = c.get_connection_hints() d.addBoth(results.append) self.assertEqual(results, [hints]) c._stop_listening() class DummyProtocol(protocol.Protocol): def __init__(self): self.buf = b"" self._count = None self._d2 = None def wait_for(self, count): if len(self.buf) >= count: data = self.buf[:count] self.buf = self.buf[count:] return defer.succeed(data) self._d = defer.Deferred() self._count = count return self._d def dataReceived(self, data): self.buf += data #print("oDR", self._count, len(self.buf)) if self._count is not None and len(self.buf) >= self._count: got = self.buf[:self._count] self.buf = self.buf[self._count:] self._count = None self._d.callback(got) def wait_for_disconnect(self): self._d2 = defer.Deferred() return self._d2 def connectionLost(self, reason): if self._d2: self._d2.callback(None) class FakeTransport: signalConnectionLost = True def __init__(self, p, peeraddr): self.protocol = p self._peeraddr = peeraddr self._buf = b"" self._connected = True def write(self, data): self._buf += data def loseConnection(self): self._connected = False if self.signalConnectionLost: self.protocol.connectionLost() def getPeer(self): return self._peeraddr def read_buf(self): b = self._buf self._buf = b"" return b class RandomError(Exception): pass class MockConnection: def __init__(self, owner, relay_handshake, start, description): self.owner = owner self.relay_handshake = relay_handshake self.start = start self._description = description def cancel(d): self._cancelled = True self._d = defer.Deferred(cancel) self._start_negotiation_called = False self._cancelled = False def startNegotiation(self): self._start_negotiation_called = True return self._d class InboundConnectionFactory(unittest.TestCase): def test_describe(self): f = transit.InboundConnectionFactory(None) addrH = address.HostnameAddress("example.com", 1234) self.assertEqual(f._describePeer(addrH), "<-example.com:1234") addr4 = address.IPv4Address("TCP", "1.2.3.4", 1234) self.assertEqual(f._describePeer(addr4), "<-1.2.3.4:1234") addr6 = address.IPv6Address("TCP", "::1", 1234) self.assertEqual(f._describePeer(addr6), "<-::1:1234") addrU = address.UNIXAddress("/dev/unlikely") self.assertEqual(f._describePeer(addrU), "<-UNIXAddress('/dev/unlikely')") def test_success(self): f = transit.InboundConnectionFactory("owner") f.protocol = MockConnection results = [] d = f.whenDone() d.addBoth(results.append) self.assertEqual(results, []) addr = address.HostnameAddress("example.com", 1234) p = f.buildProtocol(addr) self.assertIsInstance(p, MockConnection) self.assertEqual(p.owner, "owner") self.assertEqual(p.relay_handshake, None) self.assertEqual(p._start_negotiation_called, False) # meh .start # this is normally called from Connection.connectionMade f.connectionWasMade(p) self.assertEqual(p._start_negotiation_called, True) self.assertEqual(results, []) self.assertEqual(p._description, "<-example.com:1234") p._d.callback(p) self.assertEqual(results, [p]) def test_one_fail_one_success(self): f = transit.InboundConnectionFactory("owner") f.protocol = MockConnection results = [] d = f.whenDone() d.addBoth(results.append) self.assertEqual(results, []) addr1 = address.HostnameAddress("example.com", 1234) addr2 = address.HostnameAddress("example.com", 5678) p1 = f.buildProtocol(addr1) p2 = f.buildProtocol(addr2) f.connectionWasMade(p1) f.connectionWasMade(p2) self.assertEqual(results, []) p1._d.errback(transit.BadHandshake("nope")) self.assertEqual(results, []) p2._d.callback(p2) self.assertEqual(results, [p2]) def test_first_success_wins(self): f = transit.InboundConnectionFactory("owner") f.protocol = MockConnection results = [] d = f.whenDone() d.addBoth(results.append) self.assertEqual(results, []) addr1 = address.HostnameAddress("example.com", 1234) addr2 = address.HostnameAddress("example.com", 5678) p1 = f.buildProtocol(addr1) p2 = f.buildProtocol(addr2) f.connectionWasMade(p1) f.connectionWasMade(p2) self.assertEqual(results, []) p1._d.callback(p1) self.assertEqual(results, [p1]) self.assertEqual(p1._cancelled, False) self.assertEqual(p2._cancelled, True) def test_log_other_errors(self): f = transit.InboundConnectionFactory("owner") f.protocol = MockConnection results = [] d = f.whenDone() d.addBoth(results.append) self.assertEqual(results, []) addr = address.HostnameAddress("example.com", 1234) p1 = f.buildProtocol(addr) # if the Connection protocol throws an unexpected error, that should # get logged to the Twisted logs (as an Unhandled Error in Deferred) # so we can diagnose the bug f.connectionWasMade(p1) our_error = RandomError("boom1") p1._d.errback(our_error) self.assertEqual(len(results), 0) log.msg("=== note: the next RandomError is expected ===") # Make sure the Deferred has gone out of scope, so the UnhandledError # happens quickly. We must manually break the gc cycle. del p1._d gc.collect() # make PyPy happy errors = self.flushLoggedErrors(RandomError) self.assertEqual(1, len(errors)) self.assertEqual(our_error, errors[0].value) log.msg("=== note: the preceding RandomError was expected ===") def test_cancel(self): f = transit.InboundConnectionFactory("owner") f.protocol = MockConnection results = [] d = f.whenDone() d.addBoth(results.append) self.assertEqual(results, []) addr1 = address.HostnameAddress("example.com", 1234) addr2 = address.HostnameAddress("example.com", 5678) p1 = f.buildProtocol(addr1) p2 = f.buildProtocol(addr2) f.connectionWasMade(p1) f.connectionWasMade(p2) self.assertEqual(results, []) d.cancel() self.assertEqual(len(results), 1) f = results[0] self.assertIsInstance(f, failure.Failure) self.assertIsInstance(f.value, defer.CancelledError) self.assertEqual(p1._cancelled, True) self.assertEqual(p2._cancelled, True) # XXX check descriptions class OutboundConnectionFactory(unittest.TestCase): def test_success(self): f = transit.OutboundConnectionFactory("owner", "relay_handshake", "description") f.protocol = MockConnection addr = address.HostnameAddress("example.com", 1234) p = f.buildProtocol(addr) self.assertIsInstance(p, MockConnection) self.assertEqual(p.owner, "owner") self.assertEqual(p.relay_handshake, "relay_handshake") self.assertEqual(p._start_negotiation_called, False) # meh .start # this is normally called from Connection.connectionMade f.connectionWasMade(p) # no-op for outbound self.assertEqual(p._start_negotiation_called, False) class MockOwner: _connection_ready_called = False def connection_ready(self, connection): self._connection_ready_called = True self._connection = connection return self._state def _send_this(self): return b"send_this" def _expect_this(self): return b"expect_this" def _sender_record_key(self): return b"s"*32 def _receiver_record_key(self): return b"r"*32 class MockFactory: _connectionWasMade_called = False def connectionWasMade(self, p): self._connectionWasMade_called = True self._p = p class Connection(unittest.TestCase): # exercise the Connection protocol class def test_check_and_remove(self): c = transit.Connection(None, None, None, "description") c.buf = b"" EXP = b"expectation" self.assertFalse(c._check_and_remove(EXP)) self.assertEqual(c.buf, b"") c.buf = b"unexpected" e = self.assertRaises(transit.BadHandshake, c._check_and_remove, EXP) self.assertEqual(str(e), "got %r want %r" % (b'unexpected', b'expectation')) self.assertEqual(c.buf, b"unexpected") c.buf = b"expect" self.assertFalse(c._check_and_remove(EXP)) self.assertEqual(c.buf, b"expect") c.buf = b"expectation" self.assertTrue(c._check_and_remove(EXP)) self.assertEqual(c.buf, b"") c.buf = b"expectation exceeded" self.assertTrue(c._check_and_remove(EXP)) self.assertEqual(c.buf, b" exceeded") def test_sender_accepting(self): relay_handshake = None owner = MockOwner() factory = MockFactory() addr = address.HostnameAddress("example.com", 1234) c = transit.Connection(owner, relay_handshake, None, "description") self.assertEqual(c.state, "too-early") t = c.transport = FakeTransport(c, addr) c.factory = factory c.connectionMade() self.assertEqual(factory._connectionWasMade_called, True) self.assertEqual(factory._p, c) owner._state = "go" d = c.startNegotiation() self.assertEqual(c.state, "handshake") self.assertEqual(t.read_buf(), b"send_this") results = [] d.addBoth(results.append) self.assertEqual(results, []) c.dataReceived(b"expect_this") self.assertEqual(t.read_buf(), b"go\n") self.assertEqual(t._connected, True) self.assertEqual(c.state, "records") self.assertEqual(results, [c]) c.close() self.assertEqual(t._connected, False) def test_sender_rejecting(self): relay_handshake = None owner = MockOwner() factory = MockFactory() addr = address.HostnameAddress("example.com", 1234) c = transit.Connection(owner, relay_handshake, None, "description") self.assertEqual(c.state, "too-early") t = c.transport = FakeTransport(c, addr) c.factory = factory c.connectionMade() self.assertEqual(factory._connectionWasMade_called, True) self.assertEqual(factory._p, c) owner._state = "nevermind" d = c.startNegotiation() self.assertEqual(c.state, "handshake") self.assertEqual(t.read_buf(), b"send_this") results = [] d.addBoth(results.append) self.assertEqual(results, []) c.dataReceived(b"expect_this") self.assertEqual(t.read_buf(), b"nevermind\n") self.assertEqual(t._connected, False) self.assertEqual(c.state, "hung up") self.assertEqual(len(results), 1) f = results[0] self.assertIsInstance(f, failure.Failure) self.assertIsInstance(f.value, transit.BadHandshake) self.assertEqual(str(f.value), "abandoned") def test_handshake_other_error(self): owner = MockOwner() factory = MockFactory() addr = address.HostnameAddress("example.com", 1234) c = transit.Connection(owner, None, None, "description") self.assertEqual(c.state, "too-early") t = c.transport = FakeTransport(c, addr) c.factory = factory c.connectionMade() self.assertEqual(factory._connectionWasMade_called, True) self.assertEqual(factory._p, c) d = c.startNegotiation() self.assertEqual(c.state, "handshake") self.assertEqual(t.read_buf(), b"send_this") results = [] d.addBoth(results.append) self.assertEqual(results, []) c.state = RandomError("boom2") self.assertRaises(RandomError, c.dataReceived, b"surprise!") self.assertEqual(t._connected, False) self.assertEqual(c.state, "hung up") self.assertEqual(len(results), 1) f = results[0] self.assertIsInstance(f, failure.Failure) self.assertIsInstance(f.value, RandomError) def test_relay_handshake(self): relay_handshake = b"relay handshake" owner = MockOwner() factory = MockFactory() addr = address.HostnameAddress("example.com", 1234) c = transit.Connection(owner, relay_handshake, None, "description") self.assertEqual(c.state, "too-early") t = c.transport = FakeTransport(c, addr) c.factory = factory c.connectionMade() self.assertEqual(factory._connectionWasMade_called, True) self.assertEqual(factory._p, c) self.assertEqual(t.read_buf(), b"") # quiet until startNegotiation owner._state = "go" d = c.startNegotiation() self.assertEqual(t.read_buf(), relay_handshake) self.assertEqual(c.state, "relay") # waiting for OK from relay c.dataReceived(b"ok\n") self.assertEqual(t.read_buf(), b"send_this") self.assertEqual(c.state, "handshake") results = [] d.addBoth(results.append) self.assertEqual(results, []) c.dataReceived(b"expect_this") self.assertEqual(c.state, "records") self.assertEqual(results, [c]) self.assertEqual(t.read_buf(), b"go\n") def test_relay_handshake_bad(self): relay_handshake = b"relay handshake" owner = MockOwner() factory = MockFactory() addr = address.HostnameAddress("example.com", 1234) c = transit.Connection(owner, relay_handshake, None, "description") self.assertEqual(c.state, "too-early") t = c.transport = FakeTransport(c, addr) c.factory = factory c.connectionMade() self.assertEqual(factory._connectionWasMade_called, True) self.assertEqual(factory._p, c) self.assertEqual(t.read_buf(), b"") # quiet until startNegotiation owner._state = "go" d = c.startNegotiation() self.assertEqual(t.read_buf(), relay_handshake) self.assertEqual(c.state, "relay") # waiting for OK from relay c.dataReceived(b"not ok\n") self.assertEqual(t._connected, False) self.assertEqual(c.state, "hung up") results = [] d.addBoth(results.append) self.assertEqual(len(results), 1) f = results[0] self.assertIsInstance(f, failure.Failure) self.assertIsInstance(f.value, transit.BadHandshake) self.assertEqual(str(f.value), "got %r want %r" % (b"not ok\n", b"ok\n")) def test_receiver_accepted(self): # we're on the receiving side, so we wait for the sender to decide owner = MockOwner() factory = MockFactory() addr = address.HostnameAddress("example.com", 1234) c = transit.Connection(owner, None, None, "description") self.assertEqual(c.state, "too-early") t = c.transport = FakeTransport(c, addr) c.factory = factory c.connectionMade() self.assertEqual(factory._connectionWasMade_called, True) self.assertEqual(factory._p, c) owner._state = "wait-for-decision" d = c.startNegotiation() self.assertEqual(c.state, "handshake") self.assertEqual(t.read_buf(), b"send_this") results = [] d.addBoth(results.append) self.assertEqual(results, []) c.dataReceived(b"expect_this") self.assertEqual(c.state, "wait-for-decision") self.assertEqual(results, []) c.dataReceived(b"go\n") self.assertEqual(c.state, "records") self.assertEqual(results, [c]) def test_receiver_rejected_politely(self): # we're on the receiving side, so we wait for the sender to decide owner = MockOwner() factory = MockFactory() addr = address.HostnameAddress("example.com", 1234) c = transit.Connection(owner, None, None, "description") self.assertEqual(c.state, "too-early") t = c.transport = FakeTransport(c, addr) c.factory = factory c.connectionMade() self.assertEqual(factory._connectionWasMade_called, True) self.assertEqual(factory._p, c) owner._state = "wait-for-decision" d = c.startNegotiation() self.assertEqual(c.state, "handshake") self.assertEqual(t.read_buf(), b"send_this") results = [] d.addBoth(results.append) self.assertEqual(results, []) c.dataReceived(b"expect_this") self.assertEqual(c.state, "wait-for-decision") self.assertEqual(results, []) c.dataReceived(b"nevermind\n") # polite rejection self.assertEqual(t._connected, False) self.assertEqual(c.state, "hung up") self.assertEqual(len(results), 1) f = results[0] self.assertIsInstance(f, failure.Failure) self.assertIsInstance(f.value, transit.BadHandshake) self.assertEqual(str(f.value), "got %r want %r" % (b"nevermind\n", b"go\n")) def test_receiver_rejected_rudely(self): # we're on the receiving side, so we wait for the sender to decide owner = MockOwner() factory = MockFactory() addr = address.HostnameAddress("example.com", 1234) c = transit.Connection(owner, None, None, "description") self.assertEqual(c.state, "too-early") t = c.transport = FakeTransport(c, addr) c.factory = factory c.connectionMade() self.assertEqual(factory._connectionWasMade_called, True) self.assertEqual(factory._p, c) owner._state = "wait-for-decision" d = c.startNegotiation() self.assertEqual(c.state, "handshake") self.assertEqual(t.read_buf(), b"send_this") results = [] d.addBoth(results.append) self.assertEqual(results, []) c.dataReceived(b"expect_this") self.assertEqual(c.state, "wait-for-decision") self.assertEqual(results, []) t.loseConnection() self.assertEqual(t._connected, False) self.assertEqual(len(results), 1) f = results[0] self.assertIsInstance(f, failure.Failure) self.assertIsInstance(f.value, transit.BadHandshake) self.assertEqual(str(f.value), "connection lost") def test_cancel(self): owner = MockOwner() factory = MockFactory() addr = address.HostnameAddress("example.com", 1234) c = transit.Connection(owner, None, None, "description") self.assertEqual(c.state, "too-early") t = c.transport = FakeTransport(c, addr) c.factory = factory c.connectionMade() d = c.startNegotiation() results = [] d.addBoth(results.append) # while we're waiting for negotiation, we get cancelled d.cancel() self.assertEqual(t._connected, False) self.assertEqual(c.state, "hung up") self.assertEqual(len(results), 1) f = results[0] self.assertIsInstance(f, failure.Failure) self.assertIsInstance(f.value, defer.CancelledError) def test_timeout(self): clock = task.Clock() owner = MockOwner() factory = MockFactory() addr = address.HostnameAddress("example.com", 1234) c = transit.Connection(owner, None, None, "description") def _callLater(period, func): clock.callLater(period, func) c.callLater = _callLater self.assertEqual(c.state, "too-early") t = c.transport = FakeTransport(c, addr) c.factory = factory c.connectionMade() # the timer should now be running d = c.startNegotiation() results = [] d.addBoth(results.append) # while we're waiting for negotiation, the timer expires clock.advance(transit.TIMEOUT + 1.0) self.assertEqual(t._connected, False) self.assertEqual(len(results), 1) f = results[0] self.assertIsInstance(f, failure.Failure) self.assertIsInstance(f.value, transit.BadHandshake) self.assertEqual(str(f.value), "timeout") def make_connection(self): owner = MockOwner() factory = MockFactory() addr = address.HostnameAddress("example.com", 1234) c = transit.Connection(owner, None, None, "description") t = c.transport = FakeTransport(c, addr) c.factory = factory c.connectionMade() owner._state = "go" d = c.startNegotiation() results = [] d.addBoth(results.append) c.dataReceived(b"expect_this") self.assertEqual(results, [c]) t.read_buf() # flush input buffer, prepare for encrypted records return t, c, owner def test_records_good(self): # now make sure that outbound records are encrypted properly t, c, owner = self.make_connection() RECORD1 = b"record" c.send_record(RECORD1) buf = t.read_buf() expected = ("%08x" % (24+len(RECORD1)+16)).encode("ascii") self.assertEqual(hexlify(buf[:4]), expected) encrypted = buf[4:] receive_box = SecretBox(owner._sender_record_key()) nonce_buf = encrypted[:SecretBox.NONCE_SIZE] # assume it's prepended nonce = int(hexlify(nonce_buf), 16) self.assertEqual(nonce, 0) # first message gets nonce 0 decrypted = receive_box.decrypt(encrypted) self.assertEqual(decrypted, RECORD1) # second message gets nonce 1 RECORD2 = b"record2" c.send_record(RECORD2) buf = t.read_buf() expected = ("%08x" % (24+len(RECORD2)+16)).encode("ascii") self.assertEqual(hexlify(buf[:4]), expected) encrypted = buf[4:] receive_box = SecretBox(owner._sender_record_key()) nonce_buf = encrypted[:SecretBox.NONCE_SIZE] # assume it's prepended nonce = int(hexlify(nonce_buf), 16) self.assertEqual(nonce, 1) decrypted = receive_box.decrypt(encrypted) self.assertEqual(decrypted, RECORD2) # and that we can receive records properly inbound_records = [] c.recordReceived = inbound_records.append send_box = SecretBox(owner._receiver_record_key()) RECORD3 = b"record3" nonce_buf = unhexlify("%048x" % 0) # first nonce must be 0 encrypted = send_box.encrypt(RECORD3, nonce_buf) length = unhexlify("%08x" % len(encrypted)) # always 4 bytes long c.dataReceived(length[:2]) c.dataReceived(length[2:]) c.dataReceived(encrypted[:-2]) self.assertEqual(inbound_records, []) c.dataReceived(encrypted[-2:]) self.assertEqual(inbound_records, [RECORD3]) RECORD4 = b"record4" nonce_buf = unhexlify("%048x" % 1) # nonces increment encrypted = send_box.encrypt(RECORD4, nonce_buf) length = unhexlify("%08x" % len(encrypted)) # always 4 bytes long c.dataReceived(length[:2]) c.dataReceived(length[2:]) c.dataReceived(encrypted[:-2]) self.assertEqual(inbound_records, [RECORD3]) c.dataReceived(encrypted[-2:]) self.assertEqual(inbound_records, [RECORD3, RECORD4]) # receiving two records at the same time: deliver both inbound_records[:] = [] RECORD5 = b"record5" nonce_buf = unhexlify("%048x" % 2) # nonces increment encrypted = send_box.encrypt(RECORD5, nonce_buf) length = unhexlify("%08x" % len(encrypted)) # always 4 bytes long r5 = length+encrypted RECORD6 = b"record6" nonce_buf = unhexlify("%048x" % 3) # nonces increment encrypted = send_box.encrypt(RECORD6, nonce_buf) length = unhexlify("%08x" % len(encrypted)) # always 4 bytes long r6 = length+encrypted c.dataReceived(r5+r6) self.assertEqual(inbound_records, [RECORD5, RECORD6]) def corrupt(self, orig): last_byte = orig[-1:] num = int(hexlify(last_byte).decode("ascii"), 16) corrupt_num = 256 - num as_byte = unhexlify("%02x" % corrupt_num) return orig[:-1] + as_byte def test_records_corrupt(self): # corrupt records should be rejected t, c, owner = self.make_connection() inbound_records = [] c.recordReceived = inbound_records.append RECORD = b"record" send_box = SecretBox(owner._receiver_record_key()) nonce_buf = unhexlify("%048x" % 0) # first nonce must be 0 encrypted = self.corrupt(send_box.encrypt(RECORD, nonce_buf)) length = unhexlify("%08x" % len(encrypted)) # always 4 bytes long c.dataReceived(length) c.dataReceived(encrypted[:-2]) self.assertEqual(inbound_records, []) self.assertRaises(CryptoError, c.dataReceived, encrypted[-2:]) self.assertEqual(inbound_records, []) # and the connection should have been dropped self.assertEqual(t._connected, False) def test_out_of_order_nonce(self): # an inbound out-of-order nonce should be rejected t, c, owner = self.make_connection() inbound_records = [] c.recordReceived = inbound_records.append RECORD = b"record" send_box = SecretBox(owner._receiver_record_key()) nonce_buf = unhexlify("%048x" % 1) # first nonce must be 0 encrypted = send_box.encrypt(RECORD, nonce_buf) length = unhexlify("%08x" % len(encrypted)) # always 4 bytes long c.dataReceived(length) c.dataReceived(encrypted[:-2]) self.assertEqual(inbound_records, []) self.assertRaises(transit.BadNonce, c.dataReceived, encrypted[-2:]) self.assertEqual(inbound_records, []) # and the connection should have been dropped self.assertEqual(t._connected, False) # TODO: check that .connectionLost/loseConnection signatures are # consistent: zero args, or one arg? # XXX: if we don't set the transit key before connecting, what # happens? We currently get a type-check assertion from HKDF because # the key is None. def test_receive_queue(self): c = transit.Connection(None, None, None, "description") c.transport = FakeTransport(c, None) c.transport.signalConnectionLost = False results = [[] for i in range(5)] c.recordReceived(b"0") c.recordReceived(b"1") c.recordReceived(b"2") c.receive_record().addBoth(results[0].append) self.assertEqual(results[0], [b"0"]) d1 = c.receive_record() d2 = c.receive_record() # they must fire in order of receipt, not order of addCallback d2.addBoth(results[2].append) self.assertEqual(results[2], [b"2"]) d1.addBoth(results[1].append) self.assertEqual(results[1], [b"1"]) c.receive_record().addBoth(results[3].append) c.receive_record().addBoth(results[4].append) self.assertEqual(results[3], []) self.assertEqual(results[4], []) c.recordReceived(b"3") self.assertEqual(results[3], [b"3"]) self.assertEqual(results[4], []) c.recordReceived(b"4") self.assertEqual(results[3], [b"3"]) self.assertEqual(results[4], [b"4"]) closed = [] c.receive_record().addBoth(closed.append) c.close() self.assertEqual(len(closed), 1) f = closed[0] self.assertIsInstance(f, failure.Failure) self.assertIsInstance(f.value, error.ConnectionClosed) def test_producer(self): # a Transit object (receiving data from the remote peer) produces # data and writes it into a local Consumer c = transit.Connection(None, None, None, "description") c.transport = proto_helpers.StringTransport() c.recordReceived(b"r1.") c.recordReceived(b"r2.") consumer = proto_helpers.StringTransport() rv = c.connectConsumer(consumer) self.assertIs(rv, None) self.assertIs(c._consumer, consumer) self.assertEqual(consumer.value(), b"r1.r2.") self.assertRaises(RuntimeError, c.connectConsumer, consumer) c.recordReceived(b"r3.") self.assertEqual(consumer.value(), b"r1.r2.r3.") c.pauseProducing() self.assertEqual(c.transport.producerState, "paused") c.resumeProducing() self.assertEqual(c.transport.producerState, "producing") c.disconnectConsumer() self.assertEqual(consumer.producer, None) c.connectConsumer(consumer) c.stopProducing() self.assertEqual(c.transport.producerState, "stopped") def test_connectConsumer(self): # connectConsumer() takes an optional number of bytes to expect, and # fires a Deferred when that many have been written c = transit.Connection(None, None, None, "description") c._negotiation_d.addErrback(lambda err: None) # eat it c.transport = proto_helpers.StringTransport() c.recordReceived(b"r1.") consumer = proto_helpers.StringTransport() results = [] d = c.connectConsumer(consumer, expected=10) d.addBoth(results.append) self.assertEqual(consumer.value(), b"r1.") self.assertEqual(results, []) c.recordReceived(b"r2.") self.assertEqual(consumer.value(), b"r1.r2.") self.assertEqual(results, []) c.recordReceived(b"r3.") self.assertEqual(consumer.value(), b"r1.r2.r3.") self.assertEqual(results, []) c.recordReceived(b"!") self.assertEqual(consumer.value(), b"r1.r2.r3.!") self.assertEqual(results, [10]) # that should automatically disconnect the consumer, and subsequent # records should get queued, not delivered self.assertIs(c._consumer, None) c.recordReceived(b"overflow") self.assertEqual(consumer.value(), b"r1.r2.r3.!") # now test that the Deferred errbacks when the connection is lost results = [] d = c.connectConsumer(consumer, expected=10) d.addBoth(results.append) c.connectionLost() self.assertEqual(len(results), 1) f = results[0] self.assertIsInstance(f, failure.Failure) self.assertIsInstance(f.value, error.ConnectionClosed) def test_writeToFile(self): c = transit.Connection(None, None, None, "description") c._negotiation_d.addErrback(lambda err: None) # eat it c.transport = proto_helpers.StringTransport() c.recordReceived(b"r1.") f = io.BytesIO() progress = [] results = [] d = c.writeToFile(f, 10, progress.append) d.addBoth(results.append) self.assertEqual(f.getvalue(), b"r1.") self.assertEqual(progress, [3]) self.assertEqual(results, []) c.recordReceived(b"r2.") self.assertEqual(f.getvalue(), b"r1.r2.") self.assertEqual(progress, [3, 3]) self.assertEqual(results, []) c.recordReceived(b"r3.") self.assertEqual(f.getvalue(), b"r1.r2.r3.") self.assertEqual(progress, [3, 3, 3]) self.assertEqual(results, []) c.recordReceived(b"!") self.assertEqual(f.getvalue(), b"r1.r2.r3.!") self.assertEqual(progress, [3, 3, 3, 1]) self.assertEqual(results, [10]) # that should automatically disconnect the consumer, and subsequent # records should get queued, not delivered self.assertIs(c._consumer, None) c.recordReceived(b"overflow.") self.assertEqual(f.getvalue(), b"r1.r2.r3.!") self.assertEqual(progress, [3, 3, 3, 1]) # test what happens when enough data is queued ahead of time c.recordReceived(b"second.") # now "overflow.second." c.recordReceived(b"third.") # now "overflow.second.third." f = io.BytesIO() results = [] d = c.writeToFile(f, 10) d.addBoth(results.append) self.assertEqual(f.getvalue(), b"overflow.second.") # whole records self.assertEqual(results, [16]) self.assertEqual(list(c._inbound_records), [b"third."]) # now test that the Deferred errbacks when the connection is lost results = [] d = c.writeToFile(f, 10) d.addBoth(results.append) c.connectionLost() self.assertEqual(len(results), 1) f = results[0] self.assertIsInstance(f, failure.Failure) self.assertIsInstance(f.value, error.ConnectionClosed) def test_consumer(self): # a local producer sends data to a consuming Transit object c = transit.Connection(None, None, None, "description") c.transport = proto_helpers.StringTransport() records = [] c.send_record = records.append producer = proto_helpers.StringTransport() c.registerProducer(producer, True) self.assertIs(c.transport.producer, producer) c.write(b"r1.") self.assertEqual(records, [b"r1."]) c.unregisterProducer() self.assertEqual(c.transport.producer, None) class FileConsumer(unittest.TestCase): def test_basic(self): f = io.BytesIO() progress = [] fc = transit.FileConsumer(f, progress.append) self.assertEqual(progress, []) self.assertEqual(f.getvalue(), b"") fc.write(b"."* 99) self.assertEqual(progress, [99]) self.assertEqual(f.getvalue(), b"."*99) fc.write(b"!") self.assertEqual(progress, [99, 1]) self.assertEqual(f.getvalue(), b"."*99+b"!") DIRECT_HINT_JSON = {"type": "direct-tcp-v1", "hostname": "direct", "port": 1234} RELAY_HINT_JSON = {"type": "relay-v1", "hints": [{"type": "direct-tcp-v1", "hostname": "relay", "port": 1234}]} UNRECOGNIZED_HINT_JSON = {"type": "unknown"} UNAVAILABLE_HINT_JSON = {"type": "direct-tcp-v1", # e.g. Tor without txtorcon "hostname": "unavailable", "port": 1234} RELAY_HINT2_JSON = {"type": "relay-v1", "hints": [{"type": "direct-tcp-v1", "hostname": "relay", "port": 1234}, UNRECOGNIZED_HINT_JSON]} UNAVAILABLE_RELAY_HINT_JSON = {"type": "relay-v1", "hints": [UNAVAILABLE_HINT_JSON]} class Transit(unittest.TestCase): def setUp(self): self._connectors = [] self._waiters = [] def _start_connector(self, ep, description, is_relay=False): d = defer.Deferred() self._connectors.append(ep) self._waiters.append(d) return d @inlineCallbacks def test_success_direct(self): clock = task.Clock() s = transit.TransitSender("", reactor=clock) s.set_transit_key(b"key") hints = yield s.get_connection_hints() # start the listener del hints s.add_connection_hints([DIRECT_HINT_JSON, UNRECOGNIZED_HINT_JSON]) s._start_connector = self._start_connector d = s.connect() results = [] d.addBoth(results.append) self.assertEqual(results, []) self.assertEqual(len(self._waiters), 1) self.assertIsInstance(self._waiters[0], defer.Deferred) self._waiters[0].callback("winner") self.assertEqual(results, ["winner"]) def _endpoint_from_hint_obj(self, hint): if isinstance(hint, transit.DirectTCPV1Hint): if hint.hostname == "unavailable": return None return hint.hostname return None @inlineCallbacks def test_wait_for_relay(self): clock = task.Clock() s = transit.TransitSender("", reactor=clock, no_listen=True) s.set_transit_key(b"key") hints = yield s.get_connection_hints() del hints s.add_connection_hints([DIRECT_HINT_JSON, UNRECOGNIZED_HINT_JSON, RELAY_HINT_JSON]) s._endpoint_from_hint_obj = self._endpoint_from_hint_obj s._start_connector = self._start_connector d = s.connect() results = [] d.addBoth(results.append) self.assertEqual(results, []) # the direct connectors are tried right away, but the relay # connectors are stalled for a few seconds self.assertEqual(self._connectors, ["direct"]) clock.advance(s.RELAY_DELAY + 1.0) self.assertEqual(self._connectors, ["direct", "relay"]) self._waiters[0].callback("winner") self.assertEqual(results, ["winner"]) @inlineCallbacks def test_priorities(self): clock = task.Clock() s = transit.TransitSender("", reactor=clock, no_listen=True) s.set_transit_key(b"key") hints = yield s.get_connection_hints() del hints s.add_connection_hints([ {"type": "relay-v1", "hints": [{"type": "direct-tcp-v1", "hostname": "relay", "port": 1234}]}, {"type": "direct-tcp-v1", "hostname": "direct", "port": 1234}, {"type": "relay-v1", "hints": [{"type": "direct-tcp-v1", "priority": 2.0, "hostname": "relay2", "port": 1234}, {"type": "direct-tcp-v1", "priority": 3.0, "hostname": "relay3", "port": 1234}]}, {"type": "relay-v1", "hints": [{"type": "direct-tcp-v1", "priority": 2.0, "hostname": "relay4", "port": 1234}]}, ]) s._endpoint_from_hint_obj = self._endpoint_from_hint_obj s._start_connector = self._start_connector d = s.connect() results = [] d.addBoth(results.append) self.assertEqual(results, []) # direct connector should be used first, then the priority=3.0 relay, # then the two 2.0 relays, then the (default) 0.0 relay self.assertEqual(self._connectors, ["direct"]) clock.advance(s.RELAY_DELAY + 1.0) self.assertEqual(self._connectors, ["direct", "relay3"]) clock.advance(s.RELAY_DELAY) self.assertIn(self._connectors, (["direct", "relay3", "relay2", "relay4"], ["direct", "relay3", "relay4", "relay2"])) clock.advance(s.RELAY_DELAY) self.assertIn(self._connectors, (["direct", "relay3", "relay2", "relay4", "relay"], ["direct", "relay3", "relay4", "relay2", "relay"])) self._waiters[0].callback("winner") self.assertEqual(results, ["winner"]) @inlineCallbacks def test_no_direct_hints(self): clock = task.Clock() s = transit.TransitSender("", reactor=clock, no_listen=True) s.set_transit_key(b"key") hints = yield s.get_connection_hints() # start the listener del hints # include hints that can't be turned into an endpoint at runtime s.add_connection_hints([UNRECOGNIZED_HINT_JSON, UNAVAILABLE_HINT_JSON, RELAY_HINT2_JSON, UNAVAILABLE_RELAY_HINT_JSON]) s._endpoint_from_hint_obj = self._endpoint_from_hint_obj s._start_connector = self._start_connector d = s.connect() results = [] d.addBoth(results.append) self.assertEqual(results, []) # since there are no usable direct hints, the relay connector will # only be stalled for 0 seconds self.assertEqual(self._connectors, []) clock.advance(0) self.assertEqual(self._connectors, ["relay"]) self._waiters[0].callback("winner") self.assertEqual(results, ["winner"]) @inlineCallbacks def test_no_contenders(self): clock = task.Clock() s = transit.TransitSender("", reactor=clock, no_listen=True) s.set_transit_key(b"key") hints = yield s.get_connection_hints() # start the listener del hints s.add_connection_hints([]) # no hints at all s._endpoint_from_hint_obj = self._endpoint_from_hint_obj s._start_connector = self._start_connector d = s.connect() f = self.failureResultOf(d, transit.TransitError) self.assertEqual(str(f.value), "No contenders for connection") class RelayHandshake(unittest.TestCase): def old_build_relay_handshake(self, key): token = transit.HKDF(key, 32, CTXinfo=b"transit_relay_token") return (token, b"please relay "+hexlify(token)+b"\n") def test_old(self): key = b"\x00" token, old_handshake = self.old_build_relay_handshake(key) tc = transit_server.TransitConnection() tc.factory = mock.Mock() tc.factory.connection_got_token = mock.Mock() tc.dataReceived(old_handshake[:-1]) self.assertEqual(tc.factory.connection_got_token.mock_calls, []) tc.dataReceived(old_handshake[-1:]) self.assertEqual(tc.factory.connection_got_token.mock_calls, [mock.call(hexlify(token), None, tc)]) def test_new(self): c = transit.Common(None) c.set_transit_key(b"\x00") new_handshake = c._build_relay_handshake() token, old_handshake = self.old_build_relay_handshake(b"\x00") tc = transit_server.TransitConnection() tc.factory = mock.Mock() tc.factory.connection_got_token = mock.Mock() tc.dataReceived(new_handshake[:-1]) self.assertEqual(tc.factory.connection_got_token.mock_calls, []) tc.dataReceived(new_handshake[-1:]) self.assertEqual(tc.factory.connection_got_token.mock_calls, [mock.call(hexlify(token), c._side.encode("ascii"), tc)]) class Full(ServerBase, unittest.TestCase): def doBoth(self, d1, d2): return gatherResults([d1, d2], True) @inlineCallbacks def test_direct(self): KEY = b"k"*32 s = transit.TransitSender(None) r = transit.TransitReceiver(None) s.set_transit_key(KEY) r.set_transit_key(KEY) shints = yield s.get_connection_hints() rhints = yield r.get_connection_hints() s.add_connection_hints(rhints) r.add_connection_hints(shints) (x,y) = yield self.doBoth(s.connect(), r.connect()) self.assertIsInstance(x, transit.Connection) self.assertIsInstance(y, transit.Connection) d = y.receive_record() x.send_record(b"record1") r = yield d self.assertEqual(r, b"record1") yield x.close() yield y.close() @inlineCallbacks def test_relay(self): KEY = b"k"*32 s = transit.TransitSender(self.transit, no_listen=True) r = transit.TransitReceiver(self.transit, no_listen=True) s.set_transit_key(KEY) r.set_transit_key(KEY) shints = yield s.get_connection_hints() rhints = yield r.get_connection_hints() s.add_connection_hints(rhints) r.add_connection_hints(shints) (x,y) = yield self.doBoth(s.connect(), r.connect()) self.assertIsInstance(x, transit.Connection) self.assertIsInstance(y, transit.Connection) d = y.receive_record() x.send_record(b"record1") r = yield d self.assertEqual(r, b"record1") yield x.close() yield y.close() magic-wormhole-0.9.1/src/wormhole/test/test_transit_server.py0000644000076600000240000002507613031527043025152 0ustar warnerstaff00000000000000from __future__ import print_function, unicode_literals from binascii import hexlify from twisted.trial import unittest from twisted.internet import protocol, reactor, defer from twisted.internet.endpoints import clientFromString, connectProtocol from twisted.web import client from .common import ServerBase from ..server import transit_server class Accumulator(protocol.Protocol): def __init__(self): self.data = b"" self.count = 0 self._wait = None self._disconnect = defer.Deferred() def waitForBytes(self, more): assert self._wait is None self.count = more self._wait = defer.Deferred() self._check_done() return self._wait def dataReceived(self, data): self.data = self.data + data self._check_done() def _check_done(self): if self._wait and len(self.data) >= self.count: d = self._wait self._wait = None d.callback(self) def connectionLost(self, why): if self._wait: self._wait.errback(RuntimeError("closed")) self._disconnect.callback(None) class Transit(ServerBase, unittest.TestCase): def test_blur_size(self): blur = transit_server.blur_size self.failUnlessEqual(blur(0), 0) self.failUnlessEqual(blur(1), 10e3) self.failUnlessEqual(blur(10e3), 10e3) self.failUnlessEqual(blur(10e3+1), 20e3) self.failUnlessEqual(blur(15e3), 20e3) self.failUnlessEqual(blur(20e3), 20e3) self.failUnlessEqual(blur(1e6), 1e6) self.failUnlessEqual(blur(1e6+1), 2e6) self.failUnlessEqual(blur(1.5e6), 2e6) self.failUnlessEqual(blur(2e6), 2e6) self.failUnlessEqual(blur(900e6), 900e6) self.failUnlessEqual(blur(1000e6), 1000e6) self.failUnlessEqual(blur(1050e6), 1100e6) self.failUnlessEqual(blur(1100e6), 1100e6) self.failUnlessEqual(blur(1150e6), 1200e6) @defer.inlineCallbacks def test_web_request(self): resp = yield client.getPage('http://127.0.0.1:{}/'.format(self.relayport).encode('ascii')) self.assertEqual('Wormhole Relay'.encode('ascii'), resp.strip()) @defer.inlineCallbacks def test_register(self): ep = clientFromString(reactor, self.transit) a1 = yield connectProtocol(ep, Accumulator()) token1 = b"\x00"*32 side1 = b"\x01"*8 a1.transport.write(b"please relay " + hexlify(token1) + b" for side " + hexlify(side1) + b"\n") # let that arrive while self.count() == 0: yield self.wait() self.assertEqual(self.count(), 1) a1.transport.loseConnection() # let that get removed while self.count() > 0: yield self.wait() self.assertEqual(self.count(), 0) # the token should be removed too self.assertEqual(len(self._transit_server._pending_requests), 0) @defer.inlineCallbacks def test_both_unsided(self): ep = clientFromString(reactor, self.transit) a1 = yield connectProtocol(ep, Accumulator()) a2 = yield connectProtocol(ep, Accumulator()) token1 = b"\x00"*32 a1.transport.write(b"please relay " + hexlify(token1) + b"\n") a2.transport.write(b"please relay " + hexlify(token1) + b"\n") # a correct handshake yields an ack, after which we can send exp = b"ok\n" yield a1.waitForBytes(len(exp)) self.assertEqual(a1.data, exp) s1 = b"data1" a1.transport.write(s1) exp = b"ok\n" yield a2.waitForBytes(len(exp)) self.assertEqual(a2.data, exp) # all data they sent after the handshake should be given to us exp = b"ok\n"+s1 yield a2.waitForBytes(len(exp)) self.assertEqual(a2.data, exp) a1.transport.loseConnection() a2.transport.loseConnection() @defer.inlineCallbacks def test_sided_unsided(self): ep = clientFromString(reactor, self.transit) a1 = yield connectProtocol(ep, Accumulator()) a2 = yield connectProtocol(ep, Accumulator()) token1 = b"\x00"*32 side1 = b"\x01"*8 a1.transport.write(b"please relay " + hexlify(token1) + b" for side " + hexlify(side1) + b"\n") a2.transport.write(b"please relay " + hexlify(token1) + b"\n") # a correct handshake yields an ack, after which we can send exp = b"ok\n" yield a1.waitForBytes(len(exp)) self.assertEqual(a1.data, exp) s1 = b"data1" a1.transport.write(s1) exp = b"ok\n" yield a2.waitForBytes(len(exp)) self.assertEqual(a2.data, exp) # all data they sent after the handshake should be given to us exp = b"ok\n"+s1 yield a2.waitForBytes(len(exp)) self.assertEqual(a2.data, exp) a1.transport.loseConnection() a2.transport.loseConnection() @defer.inlineCallbacks def test_unsided_sided(self): ep = clientFromString(reactor, self.transit) a1 = yield connectProtocol(ep, Accumulator()) a2 = yield connectProtocol(ep, Accumulator()) token1 = b"\x00"*32 side1 = b"\x01"*8 a1.transport.write(b"please relay " + hexlify(token1) + b"\n") a2.transport.write(b"please relay " + hexlify(token1) + b" for side " + hexlify(side1) + b"\n") # a correct handshake yields an ack, after which we can send exp = b"ok\n" yield a1.waitForBytes(len(exp)) self.assertEqual(a1.data, exp) s1 = b"data1" a1.transport.write(s1) exp = b"ok\n" yield a2.waitForBytes(len(exp)) self.assertEqual(a2.data, exp) # all data they sent after the handshake should be given to us exp = b"ok\n"+s1 yield a2.waitForBytes(len(exp)) self.assertEqual(a2.data, exp) a1.transport.loseConnection() a2.transport.loseConnection() @defer.inlineCallbacks def test_both_sided(self): ep = clientFromString(reactor, self.transit) a1 = yield connectProtocol(ep, Accumulator()) a2 = yield connectProtocol(ep, Accumulator()) token1 = b"\x00"*32 side1 = b"\x01"*8 side2 = b"\x02"*8 a1.transport.write(b"please relay " + hexlify(token1) + b" for side " + hexlify(side1) + b"\n") a2.transport.write(b"please relay " + hexlify(token1) + b" for side " + hexlify(side2) + b"\n") # a correct handshake yields an ack, after which we can send exp = b"ok\n" yield a1.waitForBytes(len(exp)) self.assertEqual(a1.data, exp) s1 = b"data1" a1.transport.write(s1) exp = b"ok\n" yield a2.waitForBytes(len(exp)) self.assertEqual(a2.data, exp) # all data they sent after the handshake should be given to us exp = b"ok\n"+s1 yield a2.waitForBytes(len(exp)) self.assertEqual(a2.data, exp) a1.transport.loseConnection() a2.transport.loseConnection() def count(self): return sum([len(potentials) for potentials in self._transit_server._pending_requests.values()]) def wait(self): d = defer.Deferred() reactor.callLater(0.001, d.callback, None) return d @defer.inlineCallbacks def test_ignore_same_side(self): ep = clientFromString(reactor, self.transit) a1 = yield connectProtocol(ep, Accumulator()) a2 = yield connectProtocol(ep, Accumulator()) token1 = b"\x00"*32 side1 = b"\x01"*8 a1.transport.write(b"please relay " + hexlify(token1) + b" for side " + hexlify(side1) + b"\n") # let that arrive while self.count() == 0: yield self.wait() a2.transport.write(b"please relay " + hexlify(token1) + b" for side " + hexlify(side1) + b"\n") # let that arrive while self.count() == 1: yield self.wait() self.assertEqual(self.count(), 2) # same-side connections don't match a1.transport.loseConnection() a2.transport.loseConnection() @defer.inlineCallbacks def test_bad_handshake(self): ep = clientFromString(reactor, self.transit) a1 = yield connectProtocol(ep, Accumulator()) token1 = b"\x00"*32 # the server waits for the exact number of bytes in the expected # handshake message. to trigger "bad handshake", we must match. a1.transport.write(b"please DELAY " + hexlify(token1) + b"\n") exp = b"bad handshake\n" yield a1.waitForBytes(len(exp)) self.assertEqual(a1.data, exp) a1.transport.loseConnection() @defer.inlineCallbacks def test_binary_handshake(self): ep = clientFromString(reactor, self.transit) a1 = yield connectProtocol(ep, Accumulator()) binary_bad_handshake = b"\x00\x01\xe0\x0f\n\xff" # the embedded \n makes the server trigger early, before the full # expected handshake length has arrived. A non-wormhole client # writing non-ascii junk to the transit port used to trigger a # UnicodeDecodeError when it tried to coerce the incoming handshake # to unicode, due to the ("\n" in buf) check. This was fixed to use # (b"\n" in buf). This exercises the old failure. a1.transport.write(binary_bad_handshake) exp = b"bad handshake\n" yield a1.waitForBytes(len(exp)) self.assertEqual(a1.data, exp) a1.transport.loseConnection() @defer.inlineCallbacks def test_impatience_old(self): ep = clientFromString(reactor, self.transit) a1 = yield connectProtocol(ep, Accumulator()) token1 = b"\x00"*32 # sending too many bytes is impatience. a1.transport.write(b"please relay " + hexlify(token1) + b"\nNOWNOWNOW") exp = b"impatient\n" yield a1.waitForBytes(len(exp)) self.assertEqual(a1.data, exp) a1.transport.loseConnection() @defer.inlineCallbacks def test_impatience_new(self): ep = clientFromString(reactor, self.transit) a1 = yield connectProtocol(ep, Accumulator()) token1 = b"\x00"*32 side1 = b"\x01"*8 # sending too many bytes is impatience. a1.transport.write(b"please relay " + hexlify(token1) + b" for side " + hexlify(side1) + b"\nNOWNOWNOW") exp = b"impatient\n" yield a1.waitForBytes(len(exp)) self.assertEqual(a1.data, exp) a1.transport.loseConnection() magic-wormhole-0.9.1/src/wormhole/test/test_util.py0000644000076600000240000000323513027040774023054 0ustar warnerstaff00000000000000from __future__ import unicode_literals import unicodedata from twisted.trial import unittest from .. import util class Utils(unittest.TestCase): def test_to_bytes(self): b = util.to_bytes("abc") self.assertIsInstance(b, type(b"")) self.assertEqual(b, b"abc") A = unicodedata.lookup("LATIN SMALL LETTER A WITH DIAERESIS") b = util.to_bytes(A + "bc") self.assertIsInstance(b, type(b"")) self.assertEqual(b, b"\xc3\xa4\x62\x63") def test_bytes_to_hexstr(self): b = b"\x00\x45\x91\xfe\xff" hexstr = util.bytes_to_hexstr(b) self.assertIsInstance(hexstr, type("")) self.assertEqual(hexstr, "004591feff") def test_hexstr_to_bytes(self): hexstr = "004591feff" b = util.hexstr_to_bytes(hexstr) hexstr = util.bytes_to_hexstr(b) self.assertIsInstance(b, type(b"")) self.assertEqual(b, b"\x00\x45\x91\xfe\xff") def test_dict_to_bytes(self): d = {"a": "b"} b = util.dict_to_bytes(d) self.assertIsInstance(b, type(b"")) self.assertEqual(b, b'{"a": "b"}') def test_bytes_to_dict(self): b = b'{"a": "b", "c": 2}' d = util.bytes_to_dict(b) self.assertIsInstance(d, dict) self.assertEqual(d, {"a": "b", "c": 2}) class Space(unittest.TestCase): def test_free_space(self): free = util.estimate_free_space(".") self.assert_(isinstance(free, (int, type(None))), repr(free)) # some platforms (I think the VMs used by travis are in this # category) return 0, and windows will return None, so don't assert # anything more specific about the return value magic-wormhole-0.9.1/src/wormhole/test/test_wormhole.py0000644000076600000240000010752313031527221023730 0ustar warnerstaff00000000000000from __future__ import print_function, unicode_literals import os, json, re, gc from binascii import hexlify, unhexlify import mock from twisted.trial import unittest from twisted.internet import reactor from twisted.internet.defer import Deferred, gatherResults, inlineCallbacks from .common import ServerBase from .. import wormhole from ..errors import (WrongPasswordError, WelcomeError, InternalError, KeyFormatError) from spake2 import SPAKE2_Symmetric from ..timing import DebugTiming from ..util import (bytes_to_dict, dict_to_bytes, hexstr_to_bytes, bytes_to_hexstr) from nacl.secret import SecretBox APPID = "appid" class MockWebSocket: def __init__(self): self._payloads = [] def sendMessage(self, payload, is_binary): assert not is_binary self._payloads.append(payload) def outbound(self): out = [] while self._payloads: p = self._payloads.pop(0) out.append(json.loads(p.decode("utf-8"))) return out def response(w, **kwargs): payload = json.dumps(kwargs).encode("utf-8") w._ws_dispatch_response(payload) class Welcome(unittest.TestCase): def test_tolerate_no_current_version(self): w = wormhole._WelcomeHandler("relay_url", "current_cli_version", None) w.handle_welcome({}) def test_print_motd(self): w = wormhole._WelcomeHandler("relay_url", "current_cli_version", None) with mock.patch("sys.stderr") as stderr: w.handle_welcome({"motd": "message of\nthe day"}) self.assertEqual(stderr.method_calls, [mock.call.write("Server (at relay_url) says:\n" " message of\n the day"), mock.call.write("\n")]) # motd can be displayed multiple times with mock.patch("sys.stderr") as stderr2: w.handle_welcome({"motd": "second message"}) self.assertEqual(stderr2.method_calls, [mock.call.write("Server (at relay_url) says:\n" " second message"), mock.call.write("\n")]) def test_current_version(self): w = wormhole._WelcomeHandler("relay_url", "2.0", None) with mock.patch("sys.stderr") as stderr: w.handle_welcome({"current_cli_version": "2.0"}) self.assertEqual(stderr.method_calls, []) with mock.patch("sys.stderr") as stderr: w.handle_welcome({"current_cli_version": "3.0"}) exp1 = ("Warning: errors may occur unless both sides are" " running the same version") exp2 = ("Server claims 3.0 is current, but ours is 2.0") self.assertEqual(stderr.method_calls, [mock.call.write(exp1), mock.call.write("\n"), mock.call.write(exp2), mock.call.write("\n"), ]) # warning is only displayed once with mock.patch("sys.stderr") as stderr: w.handle_welcome({"current_cli_version": "3.0"}) self.assertEqual(stderr.method_calls, []) def test_non_release_version(self): w = wormhole._WelcomeHandler("relay_url", "2.0-dirty", None) with mock.patch("sys.stderr") as stderr: w.handle_welcome({"current_cli_version": "3.0"}) self.assertEqual(stderr.method_calls, []) def test_signal_error(self): se = mock.Mock() w = wormhole._WelcomeHandler("relay_url", "2.0", se) w.handle_welcome({}) self.assertEqual(se.mock_calls, []) w.handle_welcome({"error": "oops"}) self.assertEqual(len(se.mock_calls), 1) self.assertEqual(len(se.mock_calls[0][1]), 2) # posargs we = se.mock_calls[0][1][0] self.assertIsInstance(we, WelcomeError) self.assertEqual(we.args, ("oops",)) mood = se.mock_calls[0][1][1] self.assertEqual(mood, "unwelcome") # alas WelcomeError instances don't compare against each other #self.assertEqual(se.mock_calls, [mock.call(WelcomeError("oops"))]) class InputCode(unittest.TestCase): def test_list(self): send_command = mock.Mock() ic = wormhole._InputCode(None, "prompt", 2, send_command, DebugTiming()) d = ic._list() self.assertNoResult(d) self.assertEqual(send_command.mock_calls, [mock.call("list")]) ic._response_handle_nameplates({"type": "nameplates", "nameplates": [{"id": "123"}]}) res = self.successResultOf(d) self.assertEqual(res, ["123"]) class GetCode(unittest.TestCase): def test_get(self): send_command = mock.Mock() gc = wormhole._GetCode(2, send_command, DebugTiming()) d = gc.go() self.assertNoResult(d) self.assertEqual(send_command.mock_calls, [mock.call("allocate")]) # TODO: nameplate attributes get added and checked here gc._response_handle_allocated({"type": "allocated", "nameplate": "123"}) code = self.successResultOf(d) self.assertIsInstance(code, type("")) self.assert_(code.startswith("123-")) pieces = code.split("-") self.assertEqual(len(pieces), 3) # nameplate plus two words self.assert_(re.search(r'^\d+-\w+-\w+$', code), code) class Basic(unittest.TestCase): def tearDown(self): # flush out any errorful Deferreds left dangling in cycles gc.collect() def check_out(self, out, **kwargs): # Assert that each kwarg is present in the 'out' dict. Ignore other # keys ('msgid' in particular) for key, value in kwargs.items(): self.assertIn(key, out) self.assertEqual(out[key], value, (out, key, value)) def check_outbound(self, ws, types): out = ws.outbound() self.assertEqual(len(out), len(types), (out, types)) for i,t in enumerate(types): self.assertEqual(out[i]["type"], t, (i,t,out)) return out def make_pake(self, code, side, msg1): sp2 = SPAKE2_Symmetric(wormhole.to_bytes(code), idSymmetric=wormhole.to_bytes(APPID)) msg2 = sp2.start() key = sp2.finish(msg1) return key, msg2 def test_create(self): wormhole._Wormhole(APPID, "relay_url", reactor, None, None) def test_basic(self): # We don't call w._start(), so this doesn't create a WebSocket # connection. We provide a mock connection instead. If we wanted to # exercise _connect, we'd mock out WSFactory. # w._connect = lambda self: None # w._event_connected(mock_ws) # w._event_ws_opened() # w._ws_dispatch_response(payload) timing = DebugTiming() with mock.patch("wormhole.wormhole._WelcomeHandler") as wh_c: w = wormhole._Wormhole(APPID, "relay_url", reactor, None, timing) wh = wh_c.return_value self.assertEqual(w._ws_url, "relay_url") self.assertTrue(w._flag_need_nameplate) self.assertTrue(w._flag_need_to_build_msg1) self.assertTrue(w._flag_need_to_send_PAKE) v = w.verify() w._drop_connection = mock.Mock() ws = MockWebSocket() w._event_connected(ws) out = ws.outbound() self.assertEqual(len(out), 0) w._event_ws_opened(None) out = ws.outbound() self.assertEqual(len(out), 1) self.check_out(out[0], type="bind", appid=APPID, side=w._side) self.assertIn("id", out[0]) # WelcomeHandler should get called upon 'welcome' response. Its full # behavior is exercised in 'Welcome' above. WELCOME = {"foo": "bar"} response(w, type="welcome", welcome=WELCOME) self.assertEqual(wh.mock_calls, [mock.call.handle_welcome(WELCOME)]) # because we're connected, setting the code also claims the mailbox CODE = "123-foo-bar" w.set_code(CODE) self.assertFalse(w._flag_need_to_build_msg1) out = ws.outbound() self.assertEqual(len(out), 1) self.check_out(out[0], type="claim", nameplate="123") # the server reveals the linked mailbox response(w, type="claimed", mailbox="mb456") # that triggers event_learned_mailbox, which should send open() and # PAKE self.assertEqual(w._mailbox_state, wormhole.OPEN) out = ws.outbound() self.assertEqual(len(out), 2) self.check_out(out[0], type="open", mailbox="mb456") self.check_out(out[1], type="add", phase="pake") self.assertNoResult(v) # server echoes back all "add" messages response(w, type="message", phase="pake", body=out[1]["body"], side=w._side) self.assertNoResult(v) # extract our outbound PAKE message body = bytes_to_dict(hexstr_to_bytes(out[1]["body"])) msg1 = hexstr_to_bytes(body["pake_v1"]) # next we build the simulated peer's PAKE operation side2 = w._side + "other" key, msg2 = self.make_pake(CODE, side2, msg1) payload = {"pake_v1": bytes_to_hexstr(msg2)} body_hex = bytes_to_hexstr(dict_to_bytes(payload)) response(w, type="message", phase="pake", body=body_hex, side=side2) # hearing the peer's PAKE (msg2) makes us release the nameplate, send # the confirmation message, and sends any queued phase messages. It # doesn't deliver the verifier because we're still waiting on the # confirmation message. self.assertFalse(w._flag_need_to_see_mailbox_used) self.assertEqual(w._key, key) out = ws.outbound() self.assertEqual(len(out), 2, out) self.check_out(out[0], type="release") self.check_out(out[1], type="add", phase="version") self.assertNoResult(v) # hearing a valid confirmation message doesn't throw an error plaintext = json.dumps({}).encode("utf-8") data_key = w._derive_phase_key(side2, "version") confmsg = w._encrypt_data(data_key, plaintext) version2_hex = hexlify(confmsg).decode("ascii") response(w, type="message", phase="version", body=version2_hex, side=side2) # and it releases the verifier verifier = self.successResultOf(v) self.assertEqual(verifier, w.derive_key("wormhole:verifier", SecretBox.KEY_SIZE)) # an outbound message can now be sent immediately w.send(b"phase0-outbound") out = ws.outbound() self.assertEqual(len(out), 1) self.check_out(out[0], type="add", phase="0") # decrypt+check the outbound message p0_outbound = unhexlify(out[0]["body"].encode("ascii")) msgkey0 = w._derive_phase_key(w._side, "0") p0_plaintext = w._decrypt_data(msgkey0, p0_outbound) self.assertEqual(p0_plaintext, b"phase0-outbound") # get() waits for the inbound message to arrive md = w.get() self.assertNoResult(md) self.assertIn("0", w._receive_waiters) self.assertNotIn("0", w._received_messages) msgkey1 = w._derive_phase_key(side2, "0") p0_inbound = w._encrypt_data(msgkey1, b"phase0-inbound") p0_inbound_hex = hexlify(p0_inbound).decode("ascii") response(w, type="message", phase="0", body=p0_inbound_hex, side=side2) p0_in = self.successResultOf(md) self.assertEqual(p0_in, b"phase0-inbound") self.assertNotIn("0", w._receive_waiters) self.assertIn("0", w._received_messages) # receiving an inbound message will queue it until get() is called msgkey2 = w._derive_phase_key(side2, "1") p1_inbound = w._encrypt_data(msgkey2, b"phase1-inbound") p1_inbound_hex = hexlify(p1_inbound).decode("ascii") response(w, type="message", phase="1", body=p1_inbound_hex, side=side2) self.assertIn("1", w._received_messages) self.assertNotIn("1", w._receive_waiters) p1_in = self.successResultOf(w.get()) self.assertEqual(p1_in, b"phase1-inbound") self.assertIn("1", w._received_messages) self.assertNotIn("1", w._receive_waiters) d = w.close() self.assertNoResult(d) out = ws.outbound() self.assertEqual(len(out), 1) self.check_out(out[0], type="close", mood="happy") self.assertEqual(w._drop_connection.mock_calls, []) response(w, type="released") self.assertEqual(w._drop_connection.mock_calls, []) response(w, type="closed") self.assertEqual(w._drop_connection.mock_calls, [mock.call()]) w._ws_closed(True, None, None) self.assertEqual(self.successResultOf(d), None) def test_close_wait_0(self): # Close before the connection is established. The connection still # gets established, but it is then torn down before sending anything. timing = DebugTiming() w = wormhole._Wormhole(APPID, "relay_url", reactor, None, timing) w._drop_connection = mock.Mock() d = w.close() self.assertNoResult(d) ws = MockWebSocket() w._event_connected(ws) w._event_ws_opened(None) self.assertEqual(w._drop_connection.mock_calls, [mock.call()]) self.assertNoResult(d) w._ws_closed(True, None, None) self.successResultOf(d) def test_close_wait_1(self): # close before even claiming the nameplate timing = DebugTiming() w = wormhole._Wormhole(APPID, "relay_url", reactor, None, timing) w._drop_connection = mock.Mock() ws = MockWebSocket() w._event_connected(ws) w._event_ws_opened(None) d = w.close() self.check_outbound(ws, ["bind"]) self.assertNoResult(d) self.assertEqual(w._drop_connection.mock_calls, [mock.call()]) self.assertNoResult(d) w._ws_closed(True, None, None) self.successResultOf(d) def test_close_wait_2(self): # Close after claiming the nameplate, but before opening the mailbox. # The 'claimed' response arrives before we close. timing = DebugTiming() w = wormhole._Wormhole(APPID, "relay_url", reactor, None, timing) w._drop_connection = mock.Mock() ws = MockWebSocket() w._event_connected(ws) w._event_ws_opened(None) CODE = "123-foo-bar" w.set_code(CODE) self.check_outbound(ws, ["bind", "claim"]) response(w, type="claimed", mailbox="mb123") d = w.close() self.check_outbound(ws, ["open", "add", "release", "close"]) self.assertNoResult(d) self.assertEqual(w._drop_connection.mock_calls, []) response(w, type="released") self.assertNoResult(d) self.assertEqual(w._drop_connection.mock_calls, []) response(w, type="closed") self.assertEqual(w._drop_connection.mock_calls, [mock.call()]) self.assertNoResult(d) w._ws_closed(True, None, None) self.successResultOf(d) def test_close_wait_3(self): # close after claiming the nameplate, but before opening the mailbox # The 'claimed' response arrives after we start to close. timing = DebugTiming() w = wormhole._Wormhole(APPID, "relay_url", reactor, None, timing) w._drop_connection = mock.Mock() ws = MockWebSocket() w._event_connected(ws) w._event_ws_opened(None) CODE = "123-foo-bar" w.set_code(CODE) self.check_outbound(ws, ["bind", "claim"]) d = w.close() response(w, type="claimed", mailbox="mb123") self.check_outbound(ws, ["release"]) self.assertNoResult(d) self.assertEqual(w._drop_connection.mock_calls, []) response(w, type="released") self.assertEqual(w._drop_connection.mock_calls, [mock.call()]) self.assertNoResult(d) w._ws_closed(True, None, None) self.successResultOf(d) def test_close_wait_4(self): # close after both claiming the nameplate and opening the mailbox timing = DebugTiming() w = wormhole._Wormhole(APPID, "relay_url", reactor, None, timing) w._drop_connection = mock.Mock() ws = MockWebSocket() w._event_connected(ws) w._event_ws_opened(None) CODE = "123-foo-bar" w.set_code(CODE) response(w, type="claimed", mailbox="mb456") self.check_outbound(ws, ["bind", "claim", "open", "add"]) d = w.close() self.check_outbound(ws, ["release", "close"]) self.assertNoResult(d) self.assertEqual(w._drop_connection.mock_calls, []) response(w, type="released") self.assertNoResult(d) self.assertEqual(w._drop_connection.mock_calls, []) response(w, type="closed") self.assertNoResult(d) self.assertEqual(w._drop_connection.mock_calls, [mock.call()]) w._ws_closed(True, None, None) self.successResultOf(d) def test_close_wait_5(self): # close after claiming the nameplate, opening the mailbox, then # releasing the nameplate timing = DebugTiming() w = wormhole._Wormhole(APPID, "relay_url", reactor, None, timing) w._drop_connection = mock.Mock() ws = MockWebSocket() w._event_connected(ws) w._event_ws_opened(None) CODE = "123-foo-bar" w.set_code(CODE) response(w, type="claimed", mailbox="mb456") w._key = b"" msgkey = w._derive_phase_key("side2", "misc") p1_inbound = w._encrypt_data(msgkey, b"") p1_inbound_hex = hexlify(p1_inbound).decode("ascii") response(w, type="message", phase="misc", side="side2", body=p1_inbound_hex) self.check_outbound(ws, ["bind", "claim", "open", "add", "release"]) d = w.close() self.check_outbound(ws, ["close"]) self.assertNoResult(d) self.assertEqual(w._drop_connection.mock_calls, []) response(w, type="released") self.assertNoResult(d) self.assertEqual(w._drop_connection.mock_calls, []) response(w, type="closed") self.assertNoResult(d) self.assertEqual(w._drop_connection.mock_calls, [mock.call()]) w._ws_closed(True, None, None) self.successResultOf(d) def test_close_errbacks(self): # make sure the Deferreds returned by verify() and get() are properly # errbacked upon close pass def test_get_code_mock(self): timing = DebugTiming() w = wormhole._Wormhole(APPID, "relay_url", reactor, None, timing) ws = MockWebSocket() # TODO: mock w._ws_send_command instead w._event_connected(ws) w._event_ws_opened(None) self.check_outbound(ws, ["bind"]) gc_c = mock.Mock() gc = gc_c.return_value = mock.Mock() gc_d = gc.go.return_value = Deferred() with mock.patch("wormhole.wormhole._GetCode", gc_c): d = w.get_code() self.assertNoResult(d) gc_d.callback("123-foo-bar") code = self.successResultOf(d) self.assertEqual(code, "123-foo-bar") def test_get_code_real(self): timing = DebugTiming() w = wormhole._Wormhole(APPID, "relay_url", reactor, None, timing) ws = MockWebSocket() w._event_connected(ws) w._event_ws_opened(None) self.check_outbound(ws, ["bind"]) d = w.get_code() out = ws.outbound() self.assertEqual(len(out), 1) self.check_out(out[0], type="allocate") # TODO: nameplate attributes go here self.assertNoResult(d) response(w, type="allocated", nameplate="123") code = self.successResultOf(d) self.assertIsInstance(code, type("")) self.assert_(code.startswith("123-")) pieces = code.split("-") self.assertEqual(len(pieces), 3) # nameplate plus two words self.assert_(re.search(r'^\d+-\w+-\w+$', code), code) def _test_establish_key_hook(self, established, before): timing = DebugTiming() w = wormhole._Wormhole(APPID, "relay_url", reactor, None, timing) if before: d = w.establish_key() if established is True: w._key = b"key" elif established is False: w._key = None else: w._key = b"key" w._error = WelcomeError() if not before: d = w.establish_key() else: w._maybe_notify_key() if w._key is not None and established is True: self.successResultOf(d) elif established is False: self.assertNot(d.called) else: self.failureResultOf(d) def test_establish_key_hook(self): for established in (True, False, "error"): for before in (True, False): self._test_establish_key_hook(established, before) def test_establish_key_twice(self): timing = DebugTiming() w = wormhole._Wormhole(APPID, "relay_url", reactor, None, timing) d = w.establish_key() self.assertRaises(InternalError, w.establish_key) del d # make sure verify() can be called both before and after the verifier is # computed def _test_verifier(self, when, order, success): assert when in ("early", "middle", "late") assert order in ("key-then-version", "version-then-key") assert isinstance(success, bool) #print(when, order, success) timing = DebugTiming() w = wormhole._Wormhole(APPID, "relay_url", reactor, None, timing) w._drop_connection = mock.Mock() w._ws_send_command = mock.Mock() w._mailbox_state = wormhole.OPEN side2 = "side2" d = None if success: w._key = b"key" else: w._key = b"wrongkey" plaintext = json.dumps({}).encode("utf-8") data_key = w._derive_phase_key(side2, "version") confmsg = w._encrypt_data(data_key, plaintext) w._key = None if when == "early": d = w.verify() self.assertNoResult(d) if order == "key-then-version": w._key = b"key" w._event_established_key() else: w._event_received_version(side2, confmsg) if when == "middle": d = w.verify() if d: self.assertNoResult(d) # still waiting for other msg if order == "version-then-key": w._key = b"key" w._event_established_key() else: w._event_received_version(side2, confmsg) if when == "late": d = w.verify() if success: self.successResultOf(d) else: self.assertFailure(d, wormhole.WrongPasswordError) self.flushLoggedErrors(WrongPasswordError) def test_verifier(self): for when in ("early", "middle", "late"): for order in ("key-then-version", "version-then-key"): for success in (False, True): self._test_verifier(when, order, success) def test_api_errors(self): # doing things you're not supposed to do pass def test_welcome_error(self): # A welcome message could arrive at any time, with an [error] key # that should make us halt. In practice, though, this gets sent as # soon as the connection is established, which limits the possible # states in which we might see it. timing = DebugTiming() w = wormhole._Wormhole(APPID, "relay_url", reactor, None, timing) w._drop_connection = mock.Mock() ws = MockWebSocket() w._event_connected(ws) w._event_ws_opened(None) self.check_outbound(ws, ["bind"]) d1 = w.get() d2 = w.verify() d3 = w.get_code() # TODO (tricky): test w.input_code self.assertNoResult(d1) self.assertNoResult(d2) self.assertNoResult(d3) w._signal_error(WelcomeError("you are not actually welcome"), "pouty") self.failureResultOf(d1, WelcomeError) self.failureResultOf(d2, WelcomeError) self.failureResultOf(d3, WelcomeError) # once the error is signalled, all API calls should fail self.assertRaises(WelcomeError, w.send, "foo") self.assertRaises(WelcomeError, w.derive_key, "foo", SecretBox.KEY_SIZE) self.failureResultOf(w.get(), WelcomeError) self.failureResultOf(w.verify(), WelcomeError) def test_version_error(self): # we should only receive the "version" message after we receive the # PAKE message, by which point we should know the key. If the # confirmation message doesn't decrypt, we signal an error. timing = DebugTiming() w = wormhole._Wormhole(APPID, "relay_url", reactor, None, timing) w._drop_connection = mock.Mock() ws = MockWebSocket() w._event_connected(ws) w._event_ws_opened(None) w.set_code("123-foo-bar") response(w, type="claimed", mailbox="mb456") d1 = w.get() d2 = w.verify() self.assertNoResult(d1) self.assertNoResult(d2) out = ws.outbound() # ["bind", "claim", "open", "add"] self.assertEqual(len(out), 4) self.assertEqual(out[3]["type"], "add") sp2 = SPAKE2_Symmetric(b"", idSymmetric=wormhole.to_bytes(APPID)) msg2 = sp2.start() payload = {"pake_v1": bytes_to_hexstr(msg2)} body_hex = bytes_to_hexstr(dict_to_bytes(payload)) response(w, type="message", phase="pake", body=body_hex, side="s2") self.assertNoResult(d1) self.assertNoResult(d2) # verify() waits for confirmation # sending a random version message will cause a confirmation error confkey = w.derive_key("WRONG", SecretBox.KEY_SIZE) nonce = os.urandom(wormhole.CONFMSG_NONCE_LENGTH) badversion = wormhole.make_confmsg(confkey, nonce) badversion_hex = hexlify(badversion).decode("ascii") response(w, type="message", phase="version", body=badversion_hex, side="s2") self.failureResultOf(d1, WrongPasswordError) self.failureResultOf(d2, WrongPasswordError) # once the error is signalled, all API calls should fail self.assertRaises(WrongPasswordError, w.send, "foo") self.assertRaises(WrongPasswordError, w.derive_key, "foo", SecretBox.KEY_SIZE) self.failureResultOf(w.get(), WrongPasswordError) self.failureResultOf(w.verify(), WrongPasswordError) # event orderings to exercise: # # * normal sender: set_code, send_phase1, connected, claimed, learn_msg2, # learn_phase1 # * normal receiver (argv[2]=code): set_code, connected, learn_msg1, # learn_phase1, send_phase1, # * normal receiver (readline): connected, input_code # * # * set_code, then connected # * connected, receive_pake, send_phase, set_code class Wormholes(ServerBase, unittest.TestCase): # integration test, with a real server def doBoth(self, d1, d2): return gatherResults([d1, d2], True) @inlineCallbacks def test_basic(self): w1 = wormhole.wormhole(APPID, self.relayurl, reactor) w2 = wormhole.wormhole(APPID, self.relayurl, reactor) code = yield w1.get_code() w2.set_code(code) w1.send(b"data1") w2.send(b"data2") dataX = yield w1.get() dataY = yield w2.get() self.assertEqual(dataX, b"data2") self.assertEqual(dataY, b"data1") yield w1.close() yield w2.close() @inlineCallbacks def test_same_message(self): # the two sides use random nonces for their messages, so it's ok for # both to try and send the same body: they'll result in distinct # encrypted messages w1 = wormhole.wormhole(APPID, self.relayurl, reactor) w2 = wormhole.wormhole(APPID, self.relayurl, reactor) code = yield w1.get_code() w2.set_code(code) w1.send(b"data") w2.send(b"data") dataX = yield w1.get() dataY = yield w2.get() self.assertEqual(dataX, b"data") self.assertEqual(dataY, b"data") yield w1.close() yield w2.close() @inlineCallbacks def test_interleaved(self): w1 = wormhole.wormhole(APPID, self.relayurl, reactor) w2 = wormhole.wormhole(APPID, self.relayurl, reactor) code = yield w1.get_code() w2.set_code(code) w1.send(b"data1") dataY = yield w2.get() self.assertEqual(dataY, b"data1") d = w1.get() w2.send(b"data2") dataX = yield d self.assertEqual(dataX, b"data2") yield w1.close() yield w2.close() @inlineCallbacks def test_unidirectional(self): w1 = wormhole.wormhole(APPID, self.relayurl, reactor) w2 = wormhole.wormhole(APPID, self.relayurl, reactor) code = yield w1.get_code() w2.set_code(code) w1.send(b"data1") dataY = yield w2.get() self.assertEqual(dataY, b"data1") yield w1.close() yield w2.close() @inlineCallbacks def test_early(self): w1 = wormhole.wormhole(APPID, self.relayurl, reactor) w1.send(b"data1") w2 = wormhole.wormhole(APPID, self.relayurl, reactor) d = w2.get() w1.set_code("123-abc-def") w2.set_code("123-abc-def") dataY = yield d self.assertEqual(dataY, b"data1") yield w1.close() yield w2.close() @inlineCallbacks def test_fixed_code(self): w1 = wormhole.wormhole(APPID, self.relayurl, reactor) w2 = wormhole.wormhole(APPID, self.relayurl, reactor) w1.set_code("123-purple-elephant") w2.set_code("123-purple-elephant") w1.send(b"data1"), w2.send(b"data2") dl = yield self.doBoth(w1.get(), w2.get()) (dataX, dataY) = dl self.assertEqual(dataX, b"data2") self.assertEqual(dataY, b"data1") yield w1.close() yield w2.close() @inlineCallbacks def test_multiple_messages(self): w1 = wormhole.wormhole(APPID, self.relayurl, reactor) w2 = wormhole.wormhole(APPID, self.relayurl, reactor) w1.set_code("123-purple-elephant") w2.set_code("123-purple-elephant") w1.send(b"data1"), w2.send(b"data2") w1.send(b"data3"), w2.send(b"data4") dl = yield self.doBoth(w1.get(), w2.get()) (dataX, dataY) = dl self.assertEqual(dataX, b"data2") self.assertEqual(dataY, b"data1") dl = yield self.doBoth(w1.get(), w2.get()) (dataX, dataY) = dl self.assertEqual(dataX, b"data4") self.assertEqual(dataY, b"data3") yield w1.close() yield w2.close() @inlineCallbacks def test_wrong_password(self): w1 = wormhole.wormhole(APPID, self.relayurl, reactor) w2 = wormhole.wormhole(APPID, self.relayurl, reactor) code = yield w1.get_code() w2.set_code(code+"not") # That's enough to allow both sides to discover the mismatch, but # only after the confirmation message gets through. API calls that # don't wait will appear to work until the mismatched confirmation # message arrives. w1.send(b"should still work") w2.send(b"should still work") # API calls that wait (i.e. get) will errback yield self.assertFailure(w2.get(), WrongPasswordError) yield self.assertFailure(w1.get(), WrongPasswordError) yield w1.close() yield w2.close() self.flushLoggedErrors(WrongPasswordError) @inlineCallbacks def test_wrong_password_with_spaces(self): w1 = wormhole.wormhole(APPID, self.relayurl, reactor) w2 = wormhole.wormhole(APPID, self.relayurl, reactor) code = yield w1.get_code() code_no_dashes = code.replace('-', ' ') with self.assertRaises(KeyFormatError) as ex: w2.set_code(code_no_dashes) expected_msg = "code (%s) contains spaces." % (code_no_dashes,) self.assertEqual(expected_msg, str(ex.exception)) yield w1.close() yield w2.close() self.flushLoggedErrors(KeyFormatError) @inlineCallbacks def test_verifier(self): w1 = wormhole.wormhole(APPID, self.relayurl, reactor) w2 = wormhole.wormhole(APPID, self.relayurl, reactor) code = yield w1.get_code() w2.set_code(code) v1 = yield w1.verify() v2 = yield w2.verify() self.failUnlessEqual(type(v1), type(b"")) self.failUnlessEqual(v1, v2) w1.send(b"data1") w2.send(b"data2") dataX = yield w1.get() dataY = yield w2.get() self.assertEqual(dataX, b"data2") self.assertEqual(dataY, b"data1") yield w1.close() yield w2.close() @inlineCallbacks def test_versions(self): # there's no API for this yet, but make sure the internals work w1 = wormhole.wormhole(APPID, self.relayurl, reactor) w1._my_versions = {"w1": 123} w2 = wormhole.wormhole(APPID, self.relayurl, reactor) w2._my_versions = {"w2": 456} code = yield w1.get_code() w2.set_code(code) yield w1.verify() self.assertEqual(w1._their_versions, {"w2": 456}) yield w2.verify() self.assertEqual(w2._their_versions, {"w1": 123}) yield w1.close() yield w2.close() @inlineCallbacks def test_rx_dedup(self): # Future clients will handle losing/reestablishing the Rendezvous # Server connection by retransmitting messages, which will sometimes # cause duplicate messages. Make sure this client can tolerate them. # The first place this would fail was when the second copy of the # incoming PAKE message was received, which would cause # SPAKE2.finish() to be called a second time, which throws an error # (which, being somewhat unexpected, caused a hang rather than a # clear exception). with mock.patch("wormhole.wormhole._Wormhole", MessageDoublingReceiver): w1 = wormhole.wormhole(APPID, self.relayurl, reactor) w2 = wormhole.wormhole(APPID, self.relayurl, reactor) w1.set_code("123-purple-elephant") w2.set_code("123-purple-elephant") w1.send(b"data1"), w2.send(b"data2") dl = yield self.doBoth(w1.get(), w2.get()) (dataX, dataY) = dl self.assertEqual(dataX, b"data2") self.assertEqual(dataY, b"data1") yield w1.close() yield w2.close() class MessageDoublingReceiver(wormhole._Wormhole): # we could double messages on the sending side, but a future server will # strip those duplicates, so to really exercise the receiver, we must # double them on the inbound side instead #def _msg_send(self, phase, body): # wormhole._Wormhole._msg_send(self, phase, body) # self._ws_send_command("add", phase=phase, body=bytes_to_hexstr(body)) def _event_received_peer_message(self, side, phase, body): wormhole._Wormhole._event_received_peer_message(self, side, phase, body) wormhole._Wormhole._event_received_peer_message(self, side, phase, body) class Errors(ServerBase, unittest.TestCase): @inlineCallbacks def test_codes_1(self): w = wormhole.wormhole(APPID, self.relayurl, reactor) # definitely too early self.assertRaises(InternalError, w.derive_key, "purpose", 12) w.set_code("123-purple-elephant") # code can only be set once self.assertRaises(InternalError, w.set_code, "123-nope") yield self.assertFailure(w.get_code(), InternalError) yield self.assertFailure(w.input_code(), InternalError) yield w.close() @inlineCallbacks def test_codes_2(self): w = wormhole.wormhole(APPID, self.relayurl, reactor) yield w.get_code() self.assertRaises(InternalError, w.set_code, "123-nope") yield self.assertFailure(w.get_code(), InternalError) yield self.assertFailure(w.input_code(), InternalError) yield w.close() magic-wormhole-0.9.1/src/wormhole/test/test_xfer_util.py0000644000076600000240000000336413024532264024100 0ustar warnerstaff00000000000000from twisted.trial import unittest from twisted.internet import reactor, defer from twisted.internet.defer import inlineCallbacks from .. import xfer_util from .common import ServerBase APPID = u"appid" class Xfer(ServerBase, unittest.TestCase): @inlineCallbacks def test_xfer(self): code = u"1-code" data = u"data" d1 = xfer_util.send(reactor, APPID, self.relayurl, data, code) d2 = xfer_util.receive(reactor, APPID, self.relayurl, code) send_result = yield d1 receive_result = yield d2 self.assertEqual(send_result, None) self.assertEqual(receive_result, data) @inlineCallbacks def test_on_code(self): code = u"1-code" data = u"data" send_code = [] receive_code = [] d1 = xfer_util.send(reactor, APPID, self.relayurl, data, code, on_code=send_code.append) d2 = xfer_util.receive(reactor, APPID, self.relayurl, code, on_code=receive_code.append) send_result = yield d1 receive_result = yield d2 self.assertEqual(send_code, [code]) self.assertEqual(receive_code, [code]) self.assertEqual(send_result, None) self.assertEqual(receive_result, data) @inlineCallbacks def test_make_code(self): data = u"data" got_code = defer.Deferred() d1 = xfer_util.send(reactor, APPID, self.relayurl, data, code=None, on_code=got_code.callback) code = yield got_code d2 = xfer_util.receive(reactor, APPID, self.relayurl, code) send_result = yield d1 receive_result = yield d2 self.assertEqual(send_result, None) self.assertEqual(receive_result, data) magic-wormhole-0.9.1/src/wormhole/timing.py0000644000076600000240000000371113024532264021344 0ustar warnerstaff00000000000000from __future__ import print_function, absolute_import, unicode_literals import json, time class Event: def __init__(self, name, when, **details): # data fields that will be dumped to JSON later self._name = name self._start = time.time() if when is None else float(when) self._stop = None self._details = details def detail(self, **details): self._details.update(details) def finish(self, when=None, **details): self._stop = time.time() if when is None else float(when) self.detail(**details) def __enter__(self): return self def __exit__(self, exc_type, exc_value, exc_tb): if exc_type: # inlineCallbacks uses a special exception (defer._DefGen_Return) # to deliver returnValue(), so if returnValue is used inside our # with: block, we'll mistakenly think it means something broke. # I've moved all returnValue() calls outside the 'with # timing.add()' blocks to avoid this, but if a new one # accidentally pops up, it'll get marked as an error. I used to # catch-and-release _DefGen_Return to avoid this, but removed it # because it requires referencing defer.py's private class self.finish(exception=str(exc_type)) else: self.finish() class DebugTiming: def __init__(self): self._events = [] def add(self, name, when=None, **details): ev = Event(name, when, **details) self._events.append(ev) return ev def write(self, fn, stderr): with open(fn, "wt") as f: data = [ dict(name=e._name, start=e._start, stop=e._stop, details=e._details, ) for e in self._events ] json.dump(data, f, indent=1) f.write("\n") print("Timing data written to %s" % fn, file=stderr) magic-wormhole-0.9.1/src/wormhole/tor_manager.py0000644000076600000240000001352713024532264022361 0ustar warnerstaff00000000000000from __future__ import print_function, unicode_literals import time from twisted.internet.defer import inlineCallbacks, returnValue from twisted.internet.error import ConnectError import txtorcon import ipaddress from .timing import DebugTiming from .transit import allocate_tcp_port class TorManager: def __init__(self, reactor, tor_socks_port=None, tor_control_port=9051, timing=None): """ If tor_socks_port= is provided, I will assume that it points to a functioning SOCKS server, and will use it for all outbound connections. I will not attempt to establish a control-port connection, and I will not be able to run a server. Otherwise, I will try to connect to an existing Tor process, first on localhost:9051, then /var/run/tor/control. Then I will try to authenticate, by reading a cookie file named by the Tor process. This will succeed if 1: Tor is already running, and 2: the current user can read that file (either they started it, e.g. TorBrowser, or they are in a unix group that's been given access, e.g. debian-tor). If tor_control_port= is provided, I will use it instead of 9051. """ self._reactor = reactor # note: False is int assert isinstance(tor_socks_port, (int, type(None))) assert isinstance(tor_control_port, int) self._tor_socks_port = tor_socks_port self._tor_control_port = tor_control_port self._timing = timing or DebugTiming() @inlineCallbacks def start(self): # Connect to an existing Tor, or create a new one. If we need to # launch an onion service, then we need a working control port (and # authentication cookie). If we're only acting as a client, we don't # need the control port. if self._tor_socks_port is not None: self._can_run_service = False returnValue(True) _start_find = self._timing.add("find tor") # try port 9051, then try /var/run/tor/control . Throws on failure. state = None with self._timing.add("tor localhost"): try: connection = (self._reactor, "127.0.0.1", self._tor_control_port) state = yield txtorcon.build_tor_connection(connection) self._tor_protocol = state.protocol except ConnectError: print("unable to reach Tor on %d" % self._tor_control_port) pass if not state: with self._timing.add("tor unix"): try: connection = (self._reactor, "/var/run/tor/control") # add build_state=False to get back a Protocol object # instead of a State object state = yield txtorcon.build_tor_connection(connection) self._tor_protocol = state.protocol except (ValueError, ConnectError): print("unable to reach Tor on /var/run/tor/control") pass if state: print("connected to pre-existing Tor process") print("state:", state) else: print("launching my own Tor process") yield self._create_my_own_tor() # that sets self._tor_socks_port and self._tor_protocol _start_find.finish() self._can_run_service = True returnValue(True) @inlineCallbacks def _create_my_own_tor(self): with self._timing.add("launch tor"): start = time.time() config = self.config = txtorcon.TorConfig() if 0: # The default is for launch_tor to create a tempdir itself, # and delete it when done. We only need to set a # DataDirectory if we want it to be persistent. import tempfile datadir = tempfile.mkdtemp() config.DataDirectory = datadir #config.ControlPort = allocate_tcp_port() # defaults to 9052 #print("setting config.ControlPort to", config.ControlPort) config.SocksPort = allocate_tcp_port() self._tor_socks_port = config.SocksPort print("setting config.SocksPort to", config.SocksPort) tpp = yield txtorcon.launch_tor(config, self._reactor, #tor_binary= ) # gives a TorProcessProtocol with .tor_protocol self._tor_protocol = tpp.tor_protocol print("tp:", self._tor_protocol) print("elapsed:", time.time() - start) returnValue(True) def is_non_public_numeric_address(self, host): # for numeric hostnames, skip RFC1918 addresses, since no Tor exit # node will be able to reach those. Likewise ignore IPv6 addresses. try: a = ipaddress.ip_address(host) except ValueError: return False # non-numeric, let Tor try it if a.version != 4: return True # IPv6 gets ignored if (a.is_loopback or a.is_multicast or a.is_private or a.is_reserved or a.is_unspecified): return True # too weird, don't connect return False def get_endpoint_for(self, host, port): assert isinstance(port, int) if self.is_non_public_numeric_address(host): print("ignoring non-Tor-able %s" % host) return None # txsocksx doesn't like unicode: it concatenates some binary protocol # bytes with the hostname when talking to the SOCKS server, so the # py2 automatic unicode promotion blows up host = host.encode("ascii") ep = txtorcon.TorClientEndpoint(host, port, socks_hostname="127.0.0.1", socks_port=self._tor_socks_port) return ep magic-wormhole-0.9.1/src/wormhole/transit.py0000644000076600000240000011146213031644470021545 0ustar warnerstaff00000000000000# no unicode_literals, revisit after twisted patch from __future__ import print_function, absolute_import import os, re, sys, time, socket from collections import namedtuple, deque from binascii import hexlify, unhexlify import six from zope.interface import implementer from twisted.python import log from twisted.python.runtime import platformType from twisted.internet import (reactor, interfaces, defer, protocol, endpoints, task, address, error) from twisted.internet.defer import inlineCallbacks, returnValue from twisted.protocols import policies from nacl.secret import SecretBox from hkdf import Hkdf from .errors import InternalError from .timing import DebugTiming from .util import bytes_to_hexstr from . import ipaddrs def HKDF(skm, outlen, salt=None, CTXinfo=b""): return Hkdf(salt, skm).expand(CTXinfo, outlen) class TransitError(Exception): pass class BadHandshake(Exception): pass class TransitClosed(TransitError): pass class BadNonce(TransitError): pass # The beginning of each TCP connection consists of the following handshake # messages. The sender transmits the same text regardless of whether it is on # the initiating/connecting end of the TCP connection, or on the # listening/accepting side. Same for the receiver. # # sender -> receiver: transit sender TXID_HEX ready\n\n # receiver -> sender: transit receiver RXID_HEX ready\n\n # # Any deviations from this result in the socket being closed. The handshake # messages are designed to provoke an invalid response from other sorts of # servers (HTTP, SMTP, echo). # # If the sender is satisfied with the handshake, and this is the first socket # to complete negotiation, the sender does: # # sender -> receiver: go\n # # and the next byte on the wire will be from the application. # # If this is not the first socket, the sender does: # # sender -> receiver: nevermind\n # # and closes the socket. # So the receiver looks for "transit sender TXID_HEX ready\n\ngo\n" and hangs # up upon the first wrong byte. The sender lookgs for "transit receiver # RXID_HEX ready\n\n" and then makes a first/not-first decision about sending # "go\n" or "nevermind\n"+close(). def build_receiver_handshake(key): hexid = HKDF(key, 32, CTXinfo=b"transit_receiver") return b"transit receiver "+hexlify(hexid)+b" ready\n\n" def build_sender_handshake(key): hexid = HKDF(key, 32, CTXinfo=b"transit_sender") return b"transit sender "+hexlify(hexid)+b" ready\n\n" def build_sided_relay_handshake(key, side): assert isinstance(side, type(u"")) assert len(side) == 8*2 token = HKDF(key, 32, CTXinfo=b"transit_relay_token") return b"please relay "+hexlify(token)+b" for side "+side.encode("ascii")+b"\n" # These namedtuples are "hint objects". The JSON-serializable dictionaries # are "hint dicts". # DirectTCPV1Hint and TorTCPV1Hint mean the following protocol: # * make a TCP connection (possibly via Tor) # * send the sender/receiver handshake bytes first # * expect to see the receiver/sender handshake bytes from the other side # * the sender writes "go\n", the receiver waits for "go\n" # * the rest of the connection contains transit data DirectTCPV1Hint = namedtuple("DirectTCPV1Hint", ["hostname", "port", "priority"]) TorTCPV1Hint = namedtuple("TorTCPV1Hint", ["hostname", "port", "priority"]) # RelayV1Hint contains a tuple of DirectTCPV1Hint and TorTCPV1Hint hints (we # use a tuple rather than a list so they'll be hashable into a set). For each # one, make the TCP connection, send the relay handshake, then complete the # rest of the V1 protocol. Only one hint per relay is useful. RelayV1Hint = namedtuple("RelayV1Hint", ["hints"]) def describe_hint_obj(hint): if isinstance(hint, DirectTCPV1Hint): return u"tcp:%s:%d" % (hint.hostname, hint.port) elif isinstance(hint, TorTCPV1Hint): return u"tor:%s:%d" % (hint.hostname, hint.port) else: return str(hint) def parse_hint_argv(hint, stderr=sys.stderr): assert isinstance(hint, type(u"")) # return tuple or None for an unparseable hint priority = 0.0 mo = re.search(r'^([a-zA-Z0-9]+):(.*)$', hint) if not mo: print("unparseable hint '%s'" % (hint,), file=stderr) return None hint_type = mo.group(1) if hint_type != "tcp": print("unknown hint type '%s' in '%s'" % (hint_type, hint), file=stderr) return None hint_value = mo.group(2) pieces = hint_value.split(":") if len(pieces) < 2: print("unparseable TCP hint (need more colons) '%s'" % (hint,), file=stderr) return None mo = re.search(r'^(\d+)$', pieces[1]) if not mo: print("non-numeric port in TCP hint '%s'" % (hint,), file=stderr) return None hint_host = pieces[0] hint_port = int(pieces[1]) for more in pieces[2:]: if more.startswith("priority="): more_pieces = more.split("=") try: priority = float(more_pieces[1]) except ValueError: print("non-float priority= in TCP hint '%s'" % (hint,), file=stderr) return None return DirectTCPV1Hint(hint_host, hint_port, priority) TIMEOUT=15 @implementer(interfaces.IProducer, interfaces.IConsumer) class Connection(protocol.Protocol, policies.TimeoutMixin): def __init__(self, owner, relay_handshake, start, description): self.state = "too-early" self.buf = b"" self.owner = owner self.relay_handshake = relay_handshake self.start = start self._description = description self._negotiation_d = defer.Deferred(self._cancel) self._error = None self._consumer = None self._consumer_bytes_written = 0 self._consumer_bytes_expected = None self._consumer_deferred = None self._inbound_records = deque() self._waiting_reads = deque() def connectionMade(self): self.setTimeout(TIMEOUT) # does timeoutConnection() when it expires self.factory.connectionWasMade(self) def startNegotiation(self): if self.relay_handshake is not None: self.transport.write(self.relay_handshake) self.state = "relay" else: self.state = "start" self.dataReceived(b"") # cycle the state machine return self._negotiation_d def _cancel(self, d): self.state = "hung up" # stop reacting to anything further self._error = defer.CancelledError() self.transport.loseConnection() # if connectionLost isn't called synchronously, then our # self._negotiation_d will have been errbacked by Deferred.cancel # (which is our caller). So if it's still around, clobber it if self._negotiation_d: self._negotiation_d = None def dataReceived(self, data): try: self._dataReceived(data) except Exception as e: self.setTimeout(None) self._error = e self.transport.loseConnection() self.state = "hung up" if not isinstance(e, BadHandshake): raise def _check_and_remove(self, expected): # any divergence is a handshake error if not self.buf.startswith(expected[:len(self.buf)]): raise BadHandshake("got %r want %r" % (self.buf, expected)) if len(self.buf) < len(expected): return False # keep waiting self.buf = self.buf[len(expected):] return True def _dataReceived(self, data): # protocol is: # (maybe: send relay handshake, wait for ok) # send (send|receive)_handshake # wait for (receive|send)_handshake # sender: decide, send "go" or hang up # receiver: wait for "go" self.buf += data assert self.state != "too-early" if self.state == "relay": if not self._check_and_remove(b"ok\n"): return self.state = "start" if self.state == "start": self.transport.write(self.owner._send_this()) self.state = "handshake" if self.state == "handshake": if not self._check_and_remove(self.owner._expect_this()): return self.state = self.owner.connection_ready(self) # If we're the receiver, we'll be moved to state # "wait-for-decision", which means we're waiting for the other # side (the sender) to make a decision. If we're the sender, # we'll either be moved to state "go" (send GO and move directly # to state "records") or state "nevermind" (send NEVERMIND and # hang up). if self.state == "wait-for-decision": if not self._check_and_remove(b"go\n"): return self._negotiationSuccessful() if self.state == "go": GO = b"go\n" self.transport.write(GO) self._negotiationSuccessful() if self.state == "nevermind": self.transport.write(b"nevermind\n") raise BadHandshake("abandoned") if self.state == "records": return self.dataReceivedRECORDS() if isinstance(self.state, Exception): # for tests raise self.state def _negotiationSuccessful(self): self.state = "records" self.setTimeout(None) send_key = self.owner._sender_record_key() self.send_box = SecretBox(send_key) self.send_nonce = 0 receive_key = self.owner._receiver_record_key() self.receive_box = SecretBox(receive_key) self.next_receive_nonce = 0 d, self._negotiation_d = self._negotiation_d, None d.callback(self) def dataReceivedRECORDS(self): while True: if len(self.buf) < 4: return length = int(hexlify(self.buf[:4]), 16) if len(self.buf) < 4+length: return encrypted, self.buf = self.buf[4:4+length], self.buf[4+length:] record = self._decrypt_record(encrypted) self.recordReceived(record) def _decrypt_record(self, encrypted): nonce_buf = encrypted[:SecretBox.NONCE_SIZE] # assume it's prepended nonce = int(hexlify(nonce_buf), 16) if nonce != self.next_receive_nonce: raise BadNonce("received out-of-order record: got %d, expected %d" % (nonce, self.next_receive_nonce)) self.next_receive_nonce += 1 record = self.receive_box.decrypt(encrypted) return record def describe(self): return self._description def send_record(self, record): if not isinstance(record, type(b"")): raise InternalError assert SecretBox.NONCE_SIZE == 24 assert self.send_nonce < 2**(8*24) assert len(record) < 2**(8*4) nonce = unhexlify("%048x" % self.send_nonce) # big-endian self.send_nonce += 1 encrypted = self.send_box.encrypt(record, nonce) length = unhexlify("%08x" % len(encrypted)) # always 4 bytes long self.transport.write(length) self.transport.write(encrypted) def recordReceived(self, record): if self._consumer: self._writeToConsumer(record) return self._inbound_records.append(record) self._deliverRecords() def receive_record(self): d = defer.Deferred() self._waiting_reads.append(d) self._deliverRecords() return d def _deliverRecords(self): while self._inbound_records and self._waiting_reads: r = self._inbound_records.popleft() d = self._waiting_reads.popleft() d.callback(r) def close(self): self.transport.loseConnection() while self._waiting_reads: d = self._waiting_reads.popleft() d.errback(error.ConnectionClosed()) def timeoutConnection(self): self._error = BadHandshake("timeout") self.transport.loseConnection() def connectionLost(self, reason=None): self.setTimeout(None) d, self._negotiation_d = self._negotiation_d, None # the Deferred is only relevant until negotiation finishes, so skip # this if it's alredy been fired if d: # Each call to loseConnection() sets self._error first, so we can # deliver useful information to the Factory that's waiting on # this (although they'll generally ignore the specific error, # except for logging unexpected ones). The possible cases are: # # cancel: defer.CancelledError # far-end disconnect: BadHandshake("connection lost") # handshake error (something we didn't like): BadHandshake(what) # other error: some other Exception # timeout: BadHandshake("timeout") d.errback(self._error or BadHandshake("connection lost")) if self._consumer_deferred: self._consumer_deferred.errback(error.ConnectionClosed()) # IConsumer methods, for outbound flow-control. We pass these through to # the transport. The 'producer' is something like a t.p.basic.FileSender def registerProducer(self, producer, streaming): assert interfaces.IConsumer.providedBy(self.transport) self.transport.registerProducer(producer, streaming) def unregisterProducer(self): self.transport.unregisterProducer() def write(self, data): self.send_record(data) # IProducer methods, for inbound flow-control. We pass these through to # the transport. def stopProducing(self): self.transport.stopProducing() def pauseProducing(self): self.transport.pauseProducing() def resumeProducing(self): self.transport.resumeProducing() # Helper methods def connectConsumer(self, consumer, expected=None): """Helper method to glue an instance of e.g. t.p.ftp.FileConsumer to us. Inbound records will be written as bytes to the consumer. Set 'expected' to an integer to automatically disconnect when at least that number of bytes have been written. This function will then return a Deferred (that fires with the number of bytes actually received). If the connection is lost while this Deferred is outstanding, it will errback. If 'expected' is None, then this function returns None instead of a Deferred, and you must call disconnectConsumer() when you are done.""" if self._consumer: raise RuntimeError("A consumer is already attached: %r" % self._consumer) # be aware of an ordering hazard: when we call the consumer's # .registerProducer method, they are likely to immediately call # self.resumeProducing, which we'll deliver to self.transport, which # might call our .dataReceived, which may cause more records to be # available. By waiting to set self._consumer until *after* we drain # any pending records, we avoid delivering records out of order, # which would be bad. consumer.registerProducer(self, True) # There might be enough data queued to exceed 'expected' before we # leave this function. We must be sure to register the producer # before it gets unregistered. self._consumer = consumer self._consumer_bytes_written = 0 self._consumer_bytes_expected = expected d = None if expected is not None: d = defer.Deferred() self._consumer_deferred = d # drain any pending records while self._consumer and self._inbound_records: r = self._inbound_records.popleft() self._writeToConsumer(r) return d def _writeToConsumer(self, record): self._consumer.write(record) self._consumer_bytes_written += len(record) if self._consumer_bytes_expected is not None: if self._consumer_bytes_written >= self._consumer_bytes_expected: d = self._consumer_deferred self.disconnectConsumer() d.callback(self._consumer_bytes_written) def disconnectConsumer(self): self._consumer.unregisterProducer() self._consumer = None self._consumer_bytes_expected = None self._consumer_deferred = None # Helper method to write a known number of bytes to a file. This has no # flow control: the filehandle cannot push back. 'progress' is an # optional callable which will be called on each write (with the number # of bytes written). Returns a Deferred that fires (with the number of # bytes written) when the count is reached or the RecordPipe is closed. def writeToFile(self, f, expected, progress=None, hasher=None): fc = FileConsumer(f, progress, hasher) return self.connectConsumer(fc, expected) class OutboundConnectionFactory(protocol.ClientFactory): protocol = Connection def __init__(self, owner, relay_handshake, description): self.owner = owner self.relay_handshake = relay_handshake self._description = description self.start = time.time() def buildProtocol(self, addr): p = self.protocol(self.owner, self.relay_handshake, self.start, self._description) p.factory = self return p def connectionWasMade(self, p): # outbound connections are handled via the endpoint pass class InboundConnectionFactory(protocol.ClientFactory): protocol = Connection def __init__(self, owner): self.owner = owner self.start = time.time() self._inbound_d = defer.Deferred(self._cancel) self._pending_connections = set() def whenDone(self): return self._inbound_d def _cancel(self, inbound_d): self._shutdown() # our _inbound_d will be errbacked by Deferred.cancel() def _shutdown(self): for d in list(self._pending_connections): d.cancel() # that fires _remove and _proto_failed def _describePeer(self, addr): if isinstance(addr, address.HostnameAddress): return "<-%s:%d" % (addr.hostname, addr.port) elif isinstance(addr, (address.IPv4Address, address.IPv6Address)): return "<-%s:%d" % (addr.host, addr.port) return "<-%r" % addr def buildProtocol(self, addr): p = self.protocol(self.owner, None, self.start, self._describePeer(addr)) p.factory = self return p def connectionWasMade(self, p): d = p.startNegotiation() self._pending_connections.add(d) d.addBoth(self._remove, d) d.addCallbacks(self._proto_succeeded, self._proto_failed) def _remove(self, res, d): self._pending_connections.remove(d) return res def _proto_succeeded(self, p): self._shutdown() self._inbound_d.callback(p) def _proto_failed(self, f): # ignore these two, let Twisted log everything else f.trap(BadHandshake, defer.CancelledError) pass def allocate_tcp_port(): """Return an (integer) available TCP port on localhost. This briefly listens on the port in question, then closes it right away.""" # We want to bind() the socket but not listen(). Twisted (in # tcp.Port.createInternetSocket) would do several other things: # non-blocking, close-on-exec, and SO_REUSEADDR. We don't need # non-blocking because we never listen on it, and we don't need # close-on-exec because we close it right away. So just add SO_REUSEADDR. s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) if platformType == "posix" and sys.platform != "cygwin": s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) s.bind(("127.0.0.1", 0)) port = s.getsockname()[1] s.close() return port class _ThereCanBeOnlyOne: """Accept a list of contender Deferreds, and return a summary Deferred. When the first contender fires successfully, cancel the rest and fire the summary with the winning contender's result. If all error, errback the summary. status_cb=? """ def __init__(self, contenders): self._remaining = set(contenders) self._winner_d = defer.Deferred(self._cancel) self._first_success = None self._first_failure = None self._have_winner = False self._fired = False def _cancel(self, _): for d in list(self._remaining): d.cancel() # since that will errback everything in _remaining, we'll have hit # _maybe_done() and fired self._winner_d by this point def run(self): for d in list(self._remaining): d.addBoth(self._remove, d) d.addCallbacks(self._succeeded, self._failed) d.addCallback(self._maybe_done) return self._winner_d def _remove(self, res, d): self._remaining.remove(d) return res def _succeeded(self, res): self._have_winner = True self._first_success = res for d in list(self._remaining): d.cancel() def _failed(self, f): if self._first_failure is None: self._first_failure = f def _maybe_done(self, _): if self._remaining: return if self._fired: return self._fired = True if self._have_winner: self._winner_d.callback(self._first_success) else: self._winner_d.errback(self._first_failure) def there_can_be_only_one(contenders): return _ThereCanBeOnlyOne(contenders).run() class Common: RELAY_DELAY = 2.0 TRANSIT_KEY_LENGTH = SecretBox.KEY_SIZE def __init__(self, transit_relay, no_listen=False, tor_manager=None, reactor=reactor, timing=None): self._side = bytes_to_hexstr(os.urandom(8)) # unicode if transit_relay: if not isinstance(transit_relay, type(u"")): raise InternalError # TODO: allow multiple hints for a single relay relay_hint = parse_hint_argv(transit_relay) relay = RelayV1Hint(hints=(relay_hint,)) self._transit_relays = [relay] else: self._transit_relays = [] self._their_direct_hints = [] # hintobjs self._our_relay_hints = set(self._transit_relays) self._tor_manager = tor_manager self._transit_key = None self._no_listen = no_listen self._waiting_for_transit_key = [] self._listener = None self._winner = None self._reactor = reactor self._timing = timing or DebugTiming() self._timing.add("transit") def _build_listener(self): if self._no_listen or self._tor_manager: return ([], None) portnum = allocate_tcp_port() addresses = ipaddrs.find_addresses() non_loopback_addresses = [a for a in addresses if a != "127.0.0.1"] if non_loopback_addresses: # some test hosts, including the appveyor VMs, *only* have # 127.0.0.1, and the tests will hang badly if we remove it. addresses = non_loopback_addresses direct_hints = [DirectTCPV1Hint(six.u(addr), portnum, 0.0) for addr in addresses] ep = endpoints.serverFromString(reactor, "tcp:%d" % portnum) return direct_hints, ep def get_connection_abilities(self): return [{u"type": u"direct-tcp-v1"}, {u"type": u"relay-v1"}, ] @inlineCallbacks def get_connection_hints(self): hints = [] direct_hints = yield self._get_direct_hints() for dh in direct_hints: hints.append({u"type": u"direct-tcp-v1", u"priority": dh.priority, u"hostname": dh.hostname, u"port": dh.port, # integer }) for relay in self._transit_relays: rhint = {u"type": u"relay-v1", u"hints": []} for rh in relay.hints: rhint[u"hints"].append({u"type": u"direct-tcp-v1", u"priority": rh.priority, u"hostname": rh.hostname, u"port": rh.port}) hints.append(rhint) returnValue(hints) def _get_direct_hints(self): if self._listener: return defer.succeed(self._my_direct_hints) # there is a slight race here: if someone calls get_direct_hints() a # second time, before the listener has actually started listening, # then they'll get a Deferred that fires (with the hints) before the # listener starts listening. But most applications won't call this # multiple times, and the race is between 1: the parent Wormhole # protocol getting the connection hints to the other end, and 2: the # listener being ready for connections, and I'm confident that the # listener will win. self._my_direct_hints, self._listener = self._build_listener() if self._listener is None: # don't listen self._listener_d = None return defer.succeed(self._my_direct_hints) # empty # Start the server, so it will be running by the time anyone tries to # connect to the direct hints we return. f = InboundConnectionFactory(self) self._listener_f = f # for tests # XX move to __init__ ? self._listener_d = f.whenDone() d = self._listener.listen(f) def _listening(lp): # lp is an IListeningPort #self._listener_port = lp # for tests def _stop_listening(res): lp.stopListening() return res self._listener_d.addBoth(_stop_listening) return self._my_direct_hints d.addCallback(_listening) return d def _stop_listening(self): # this is for unit tests. The usual control flow (via connect()) # wires the listener's Deferred into a there_can_be_only_one(), which # eats the errback. If we don't ever call connect(), we must catch it # ourselves. self._listener_d.addErrback(lambda f: None) self._listener_d.cancel() def _parse_tcp_v1_hint(self, hint): # hint_struct -> hint_obj hint_type = hint.get(u"type", u"") if hint_type not in [u"direct-tcp-v1", u"tor-tcp-v1"]: log.msg("unknown hint type: %r" % (hint,)) return None if not(u"hostname" in hint and isinstance(hint[u"hostname"], type(u""))): log.msg("invalid hostname in hint: %r" % (hint,)) return None if not(u"port" in hint and isinstance(hint[u"port"], int)): log.msg("invalid port in hint: %r" % (hint,)) return None priority = hint.get(u"priority", 0.0) if hint_type == u"direct-tcp-v1": return DirectTCPV1Hint(hint[u"hostname"], hint[u"port"], priority) else: return TorTCPV1Hint(hint[u"hostname"], hint[u"port"], priority) def add_connection_hints(self, hints): for h in hints: # hint structs hint_type = h.get(u"type", u"") if hint_type in [u"direct-tcp-v1", u"tor-tcp-v1"]: dh = self._parse_tcp_v1_hint(h) if dh: self._their_direct_hints.append(dh) # hint_obj elif hint_type == u"relay-v1": # TODO: each relay-v1 clause describes a different relay, # with a set of equally-valid ways to connect to it. Treat # them as separate relays, instead of merging them all # together like this. relay_hints = [] for rhs in h.get(u"hints", []): h = self._parse_tcp_v1_hint(rhs) if h: relay_hints.append(h) if relay_hints: rh = RelayV1Hint(hints=tuple(sorted(relay_hints))) self._our_relay_hints.add(rh) else: log.msg("unknown hint type: %r" % (h,)) def _send_this(self): assert self._transit_key if self.is_sender: return build_sender_handshake(self._transit_key) else: return build_receiver_handshake(self._transit_key) def _expect_this(self): assert self._transit_key if self.is_sender: return build_receiver_handshake(self._transit_key) else: return build_sender_handshake(self._transit_key)# + b"go\n" def _sender_record_key(self): assert self._transit_key if self.is_sender: return HKDF(self._transit_key, SecretBox.KEY_SIZE, CTXinfo=b"transit_record_sender_key") else: return HKDF(self._transit_key, SecretBox.KEY_SIZE, CTXinfo=b"transit_record_receiver_key") def _receiver_record_key(self): assert self._transit_key if self.is_sender: return HKDF(self._transit_key, SecretBox.KEY_SIZE, CTXinfo=b"transit_record_receiver_key") else: return HKDF(self._transit_key, SecretBox.KEY_SIZE, CTXinfo=b"transit_record_sender_key") def set_transit_key(self, key): assert isinstance(key, type(b"")), type(key) # We use pubsub to protect against the race where the sender knows # the hints and the key, and connects to the receiver's transit # socket before the receiver gets the relay message (and thus the # key). self._transit_key = key waiters = self._waiting_for_transit_key del self._waiting_for_transit_key for d in waiters: # We don't need eventual-send here. It's safer in general, but # set_transit_key() is only called once, and _get_transit_key() # won't touch the subscribers list once the key is set. d.callback(key) def _get_transit_key(self): if self._transit_key: return defer.succeed(self._transit_key) d = defer.Deferred() self._waiting_for_transit_key.append(d) return d @inlineCallbacks def connect(self): with self._timing.add("transit connect"): yield self._get_transit_key() # we want to have the transit key before starting any outbound # connections, so those connections will know what to say when # they connect winner = yield self._connect() returnValue(winner) def _connect(self): # It might be nice to wire this so that a failure in the direct hints # causes the relay hints to be used right away (fast failover). But # none of our current use cases would take advantage of that: if we # have any viable direct hints, then they're either going to succeed # quickly or hang for a long time. contenders = [] if self._listener_d: contenders.append(self._listener_d) relay_delay = 0 for hint_obj in self._their_direct_hints: # Check the hint type to see if we can support it (e.g. skip # onion hints on a non-Tor client). Do not increase relay_delay # unless we have at least one viable hint. ep = self._endpoint_from_hint_obj(hint_obj) if not ep: continue description = "->%s" % describe_hint_obj(hint_obj) d = self._start_connector(ep, description) contenders.append(d) relay_delay = self.RELAY_DELAY # Start trying the relays a few seconds after we start to try the # direct hints. The idea is to prefer direct connections, but not be # afraid of using a relay when we have direct hints that don't # resolve quickly. Many direct hints will be to unused local-network # IP addresses, which won't answer, and would take the full TCP # timeout (30s or more) to fail. prioritized_relays = {} for rh in self._our_relay_hints: for hint_obj in rh.hints: priority = hint_obj.priority if priority not in prioritized_relays: prioritized_relays[priority] = set() prioritized_relays[priority].add(hint_obj) for priority in sorted(prioritized_relays, reverse=True): for hint_obj in prioritized_relays[priority]: ep = self._endpoint_from_hint_obj(hint_obj) if not ep: continue description = "->relay:%s" % describe_hint_obj(hint_obj) d = task.deferLater(self._reactor, relay_delay, self._start_connector, ep, description, is_relay=True) contenders.append(d) relay_delay += self.RELAY_DELAY if not contenders: raise TransitError("No contenders for connection") winner = there_can_be_only_one(contenders) return self._not_forever(2*TIMEOUT, winner) def _not_forever(self, timeout, d): """If the timer fires first, cancel the deferred. If the deferred fires first, cancel the timer.""" t = self._reactor.callLater(timeout, d.cancel) def _done(res): if t.active(): t.cancel() return res d.addBoth(_done) return d def _build_relay_handshake(self): return build_sided_relay_handshake(self._transit_key, self._side) def _start_connector(self, ep, description, is_relay=False): relay_handshake = None if is_relay: assert self._transit_key relay_handshake = self._build_relay_handshake() f = OutboundConnectionFactory(self, relay_handshake, description) d = ep.connect(f) # fires with protocol, or ConnectError d.addCallback(lambda p: p.startNegotiation()) return d def _endpoint_from_hint_obj(self, hint): if self._tor_manager: if isinstance(hint, (DirectTCPV1Hint, TorTCPV1Hint)): # our TorManager will return None for non-public IPv4 # addresses and any IPv6 address return self._tor_manager.get_endpoint_for(hint.hostname, hint.port) return None if isinstance(hint, DirectTCPV1Hint): return endpoints.HostnameEndpoint(self._reactor, hint.hostname, hint.port) return None def connection_ready(self, p): # inbound/outbound Connection protocols call this when they finish # negotiation. The first one wins and gets a "go". Any subsequent # ones lose and get a "nevermind" before being closed. if not self.is_sender: return "wait-for-decision" if self._winner: # we already have a winner, so this one loses return "nevermind" # this one wins! self._winner = p return "go" class TransitSender(Common): is_sender = True class TransitReceiver(Common): is_sender = False # based on twisted.protocols.ftp.FileConsumer, but don't close the filehandle # when done, and add a progress function that gets called with the length of # each write, and a hasher function that gets called with the data. @implementer(interfaces.IConsumer) class FileConsumer: def __init__(self, f, progress=None, hasher=None): self._f = f self._progress = progress self._hasher = hasher self._producer = None def registerProducer(self, producer, streaming): assert not self._producer self._producer = producer assert streaming def write(self, bytes): self._f.write(bytes) if self._progress: self._progress(len(bytes)) if self._hasher: self._hasher(bytes) def unregisterProducer(self): assert self._producer self._producer = None # the TransitSender/Receiver.connect() yields a Connection, on which you can # do send_record(), but what should the receive API be? set a callback for # inbound records? get a Deferred for the next record? The producer/consumer # API is enough for file transfer, but what would other applications want? # how should the Listener be managed? we want to shut it down when the # connect() Deferred is cancelled, as well as terminating any negotiations in # progress. # # the factory should return/manage a deferred, which fires iff an inbound # connection completes negotiation successfully, can be cancelled (which # stops the listener and drops all pending connections), but will never # timeout, and only errbacks if cancelled. # write unit test for _ThereCanBeOnlyOne # check start/finish time-gathering instrumentation # relay URLs are probably mishandled: both sides probably send their URL, # then connect to the *other* side's URL, when they really should connect to # both their own and the other side's. The current implementation probably # only works if the two URLs are the same. magic-wormhole-0.9.1/src/wormhole/util.py0000644000076600000240000000241313027040774021033 0ustar warnerstaff00000000000000# No unicode_literals import os, json, unicodedata from binascii import hexlify, unhexlify def to_bytes(u): return unicodedata.normalize("NFC", u).encode("utf-8") def bytes_to_hexstr(b): assert isinstance(b, type(b"")) hexstr = hexlify(b).decode("ascii") assert isinstance(hexstr, type(u"")) return hexstr def hexstr_to_bytes(hexstr): assert isinstance(hexstr, type(u"")) b = unhexlify(hexstr.encode("ascii")) assert isinstance(b, type(b"")) return b def dict_to_bytes(d): assert isinstance(d, dict) b = json.dumps(d).encode("utf-8") assert isinstance(b, type(b"")) return b def bytes_to_dict(b): assert isinstance(b, type(b"")) d = json.loads(b.decode("utf-8")) assert isinstance(d, dict) return d def estimate_free_space(target): # f_bfree is the blocks available to a root user. It might be more # accurate to use f_bavail (blocks available to non-root user), but we # don't know which user is running us, and a lot of installations don't # bother with reserving extra space for root, so let's just stick to the # basic (larger) estimate. try: s = os.statvfs(os.path.dirname(os.path.abspath(target))) return s.f_frsize * s.f_bfree except AttributeError: return None magic-wormhole-0.9.1/src/wormhole/wordlist.py0000644000076600000240000002200413024532264021720 0ustar warnerstaff00000000000000from __future__ import unicode_literals # The PGP Word List, which maps bytes to phonetically-distinct words. There # are two lists, even and odd, and encodings should alternate between then to # detect dropped words. https://en.wikipedia.org/wiki/PGP_Words # Thanks to Warren Guy for transcribing them: # https://github.com/warrenguy/javascript-pgp-word-list from binascii import unhexlify raw_words = { '00': ['aardvark', 'adroitness'], '01': ['absurd', 'adviser'], '02': ['accrue', 'aftermath'], '03': ['acme', 'aggregate'], '04': ['adrift', 'alkali'], '05': ['adult', 'almighty'], '06': ['afflict', 'amulet'], '07': ['ahead', 'amusement'], '08': ['aimless', 'antenna'], '09': ['Algol', 'applicant'], '0A': ['allow', 'Apollo'], '0B': ['alone', 'armistice'], '0C': ['ammo', 'article'], '0D': ['ancient', 'asteroid'], '0E': ['apple', 'Atlantic'], '0F': ['artist', 'atmosphere'], '10': ['assume', 'autopsy'], '11': ['Athens', 'Babylon'], '12': ['atlas', 'backwater'], '13': ['Aztec', 'barbecue'], '14': ['baboon', 'belowground'], '15': ['backfield', 'bifocals'], '16': ['backward', 'bodyguard'], '17': ['banjo', 'bookseller'], '18': ['beaming', 'borderline'], '19': ['bedlamp', 'bottomless'], '1A': ['beehive', 'Bradbury'], '1B': ['beeswax', 'bravado'], '1C': ['befriend', 'Brazilian'], '1D': ['Belfast', 'breakaway'], '1E': ['berserk', 'Burlington'], '1F': ['billiard', 'businessman'], '20': ['bison', 'butterfat'], '21': ['blackjack', 'Camelot'], '22': ['blockade', 'candidate'], '23': ['blowtorch', 'cannonball'], '24': ['bluebird', 'Capricorn'], '25': ['bombast', 'caravan'], '26': ['bookshelf', 'caretaker'], '27': ['brackish', 'celebrate'], '28': ['breadline', 'cellulose'], '29': ['breakup', 'certify'], '2A': ['brickyard', 'chambermaid'], '2B': ['briefcase', 'Cherokee'], '2C': ['Burbank', 'Chicago'], '2D': ['button', 'clergyman'], '2E': ['buzzard', 'coherence'], '2F': ['cement', 'combustion'], '30': ['chairlift', 'commando'], '31': ['chatter', 'company'], '32': ['checkup', 'component'], '33': ['chisel', 'concurrent'], '34': ['choking', 'confidence'], '35': ['chopper', 'conformist'], '36': ['Christmas', 'congregate'], '37': ['clamshell', 'consensus'], '38': ['classic', 'consulting'], '39': ['classroom', 'corporate'], '3A': ['cleanup', 'corrosion'], '3B': ['clockwork', 'councilman'], '3C': ['cobra', 'crossover'], '3D': ['commence', 'crucifix'], '3E': ['concert', 'cumbersome'], '3F': ['cowbell', 'customer'], '40': ['crackdown', 'Dakota'], '41': ['cranky', 'decadence'], '42': ['crowfoot', 'December'], '43': ['crucial', 'decimal'], '44': ['crumpled', 'designing'], '45': ['crusade', 'detector'], '46': ['cubic', 'detergent'], '47': ['dashboard', 'determine'], '48': ['deadbolt', 'dictator'], '49': ['deckhand', 'dinosaur'], '4A': ['dogsled', 'direction'], '4B': ['dragnet', 'disable'], '4C': ['drainage', 'disbelief'], '4D': ['dreadful', 'disruptive'], '4E': ['drifter', 'distortion'], '4F': ['dropper', 'document'], '50': ['drumbeat', 'embezzle'], '51': ['drunken', 'enchanting'], '52': ['Dupont', 'enrollment'], '53': ['dwelling', 'enterprise'], '54': ['eating', 'equation'], '55': ['edict', 'equipment'], '56': ['egghead', 'escapade'], '57': ['eightball', 'Eskimo'], '58': ['endorse', 'everyday'], '59': ['endow', 'examine'], '5A': ['enlist', 'existence'], '5B': ['erase', 'exodus'], '5C': ['escape', 'fascinate'], '5D': ['exceed', 'filament'], '5E': ['eyeglass', 'finicky'], '5F': ['eyetooth', 'forever'], '60': ['facial', 'fortitude'], '61': ['fallout', 'frequency'], '62': ['flagpole', 'gadgetry'], '63': ['flatfoot', 'Galveston'], '64': ['flytrap', 'getaway'], '65': ['fracture', 'glossary'], '66': ['framework', 'gossamer'], '67': ['freedom', 'graduate'], '68': ['frighten', 'gravity'], '69': ['gazelle', 'guitarist'], '6A': ['Geiger', 'hamburger'], '6B': ['glitter', 'Hamilton'], '6C': ['glucose', 'handiwork'], '6D': ['goggles', 'hazardous'], '6E': ['goldfish', 'headwaters'], '6F': ['gremlin', 'hemisphere'], '70': ['guidance', 'hesitate'], '71': ['hamlet', 'hideaway'], '72': ['highchair', 'holiness'], '73': ['hockey', 'hurricane'], '74': ['indoors', 'hydraulic'], '75': ['indulge', 'impartial'], '76': ['inverse', 'impetus'], '77': ['involve', 'inception'], '78': ['island', 'indigo'], '79': ['jawbone', 'inertia'], '7A': ['keyboard', 'infancy'], '7B': ['kickoff', 'inferno'], '7C': ['kiwi', 'informant'], '7D': ['klaxon', 'insincere'], '7E': ['locale', 'insurgent'], '7F': ['lockup', 'integrate'], '80': ['merit', 'intention'], '81': ['minnow', 'inventive'], '82': ['miser', 'Istanbul'], '83': ['Mohawk', 'Jamaica'], '84': ['mural', 'Jupiter'], '85': ['music', 'leprosy'], '86': ['necklace', 'letterhead'], '87': ['Neptune', 'liberty'], '88': ['newborn', 'maritime'], '89': ['nightbird', 'matchmaker'], '8A': ['Oakland', 'maverick'], '8B': ['obtuse', 'Medusa'], '8C': ['offload', 'megaton'], '8D': ['optic', 'microscope'], '8E': ['orca', 'microwave'], '8F': ['payday', 'midsummer'], '90': ['peachy', 'millionaire'], '91': ['pheasant', 'miracle'], '92': ['physique', 'misnomer'], '93': ['playhouse', 'molasses'], '94': ['Pluto', 'molecule'], '95': ['preclude', 'Montana'], '96': ['prefer', 'monument'], '97': ['preshrunk', 'mosquito'], '98': ['printer', 'narrative'], '99': ['prowler', 'nebula'], '9A': ['pupil', 'newsletter'], '9B': ['puppy', 'Norwegian'], '9C': ['python', 'October'], '9D': ['quadrant', 'Ohio'], '9E': ['quiver', 'onlooker'], '9F': ['quota', 'opulent'], 'A0': ['ragtime', 'Orlando'], 'A1': ['ratchet', 'outfielder'], 'A2': ['rebirth', 'Pacific'], 'A3': ['reform', 'pandemic'], 'A4': ['regain', 'Pandora'], 'A5': ['reindeer', 'paperweight'], 'A6': ['rematch', 'paragon'], 'A7': ['repay', 'paragraph'], 'A8': ['retouch', 'paramount'], 'A9': ['revenge', 'passenger'], 'AA': ['reward', 'pedigree'], 'AB': ['rhythm', 'Pegasus'], 'AC': ['ribcage', 'penetrate'], 'AD': ['ringbolt', 'perceptive'], 'AE': ['robust', 'performance'], 'AF': ['rocker', 'pharmacy'], 'B0': ['ruffled', 'phonetic'], 'B1': ['sailboat', 'photograph'], 'B2': ['sawdust', 'pioneer'], 'B3': ['scallion', 'pocketful'], 'B4': ['scenic', 'politeness'], 'B5': ['scorecard', 'positive'], 'B6': ['Scotland', 'potato'], 'B7': ['seabird', 'processor'], 'B8': ['select', 'provincial'], 'B9': ['sentence', 'proximate'], 'BA': ['shadow', 'puberty'], 'BB': ['shamrock', 'publisher'], 'BC': ['showgirl', 'pyramid'], 'BD': ['skullcap', 'quantity'], 'BE': ['skydive', 'racketeer'], 'BF': ['slingshot', 'rebellion'], 'C0': ['slowdown', 'recipe'], 'C1': ['snapline', 'recover'], 'C2': ['snapshot', 'repellent'], 'C3': ['snowcap', 'replica'], 'C4': ['snowslide', 'reproduce'], 'C5': ['solo', 'resistor'], 'C6': ['southward', 'responsive'], 'C7': ['soybean', 'retraction'], 'C8': ['spaniel', 'retrieval'], 'C9': ['spearhead', 'retrospect'], 'CA': ['spellbind', 'revenue'], 'CB': ['spheroid', 'revival'], 'CC': ['spigot', 'revolver'], 'CD': ['spindle', 'sandalwood'], 'CE': ['spyglass', 'sardonic'], 'CF': ['stagehand', 'Saturday'], 'D0': ['stagnate', 'savagery'], 'D1': ['stairway', 'scavenger'], 'D2': ['standard', 'sensation'], 'D3': ['stapler', 'sociable'], 'D4': ['steamship', 'souvenir'], 'D5': ['sterling', 'specialist'], 'D6': ['stockman', 'speculate'], 'D7': ['stopwatch', 'stethoscope'], 'D8': ['stormy', 'stupendous'], 'D9': ['sugar', 'supportive'], 'DA': ['surmount', 'surrender'], 'DB': ['suspense', 'suspicious'], 'DC': ['sweatband', 'sympathy'], 'DD': ['swelter', 'tambourine'], 'DE': ['tactics', 'telephone'], 'DF': ['talon', 'therapist'], 'E0': ['tapeworm', 'tobacco'], 'E1': ['tempest', 'tolerance'], 'E2': ['tiger', 'tomorrow'], 'E3': ['tissue', 'torpedo'], 'E4': ['tonic', 'tradition'], 'E5': ['topmost', 'travesty'], 'E6': ['tracker', 'trombonist'], 'E7': ['transit', 'truncated'], 'E8': ['trauma', 'typewriter'], 'E9': ['treadmill', 'ultimate'], 'EA': ['Trojan', 'undaunted'], 'EB': ['trouble', 'underfoot'], 'EC': ['tumor', 'unicorn'], 'ED': ['tunnel', 'unify'], 'EE': ['tycoon', 'universe'], 'EF': ['uncut', 'unravel'], 'F0': ['unearth', 'upcoming'], 'F1': ['unwind', 'vacancy'], 'F2': ['uproot', 'vagabond'], 'F3': ['upset', 'vertigo'], 'F4': ['upshot', 'Virginia'], 'F5': ['vapor', 'visitor'], 'F6': ['village', 'vocalist'], 'F7': ['virus', 'voyager'], 'F8': ['Vulcan', 'warranty'], 'F9': ['waffle', 'Waterloo'], 'FA': ['wallet', 'whimsical'], 'FB': ['watchword', 'Wichita'], 'FC': ['wayside', 'Wilmington'], 'FD': ['willow', 'Wyoming'], 'FE': ['woodlark', 'yesteryear'], 'FF': ['Zulu', 'Yucatan'] }; byte_to_even_word = dict([(unhexlify(k.encode("ascii")), both_words[0]) for k,both_words in raw_words.items()]) byte_to_odd_word = dict([(unhexlify(k.encode("ascii")), both_words[1]) for k,both_words in raw_words.items()]) even_words_lowercase, odd_words_lowercase = set(), set() even_words_lowercase_to_byte, odd_words_lowercase_to_byte = dict(), dict() for k,both_words in raw_words.items(): even_word, odd_word = both_words even_words_lowercase.add(even_word.lower()) even_words_lowercase_to_byte[even_word.lower()] = unhexlify(k.encode("ascii")) odd_words_lowercase.add(odd_word.lower()) odd_words_lowercase_to_byte[odd_word.lower()] = unhexlify(k.encode("ascii")) magic-wormhole-0.9.1/src/wormhole/wormhole.py0000644000076600000240000011255013031527242021711 0ustar warnerstaff00000000000000from __future__ import print_function, absolute_import, unicode_literals import os, sys, re from six.moves.urllib_parse import urlparse from twisted.internet import defer, endpoints, error from twisted.internet.threads import deferToThread, blockingCallFromThread from twisted.internet.defer import inlineCallbacks, returnValue from twisted.python import log, failure from autobahn.twisted import websocket from nacl.secret import SecretBox from nacl.exceptions import CryptoError from nacl import utils from spake2 import SPAKE2_Symmetric from hashlib import sha256 from . import __version__ from . import codes #from .errors import ServerError, Timeout from .errors import (WrongPasswordError, InternalError, WelcomeError, WormholeClosedError, KeyFormatError) from .timing import DebugTiming from .util import (to_bytes, bytes_to_hexstr, hexstr_to_bytes, dict_to_bytes, bytes_to_dict) from hkdf import Hkdf def HKDF(skm, outlen, salt=None, CTXinfo=b""): return Hkdf(salt, skm).expand(CTXinfo, outlen) CONFMSG_NONCE_LENGTH = 128//8 CONFMSG_MAC_LENGTH = 256//8 def make_confmsg(confkey, nonce): return nonce+HKDF(confkey, CONFMSG_MAC_LENGTH, nonce) # We send the following messages through the relay server to the far side (by # sending "add" commands to the server, and getting "message" responses): # # phase=setup: # * unauthenticated version strings (but why?) # * early warmup for connection hints ("I can do tor, spin up HS") # * wordlist l10n identifier # phase=pake: just the SPAKE2 'start' message (binary) # phase=version: version data, key verification (HKDF(key, nonce)+nonce) # phase=1,2,3,..: application messages class WSClient(websocket.WebSocketClientProtocol): def onOpen(self): self.wormhole_open = True self.factory.d.callback(self) def onMessage(self, payload, isBinary): assert not isBinary self.wormhole._ws_dispatch_response(payload) def onClose(self, wasClean, code, reason): if self.wormhole_open: self.wormhole._ws_closed(wasClean, code, reason) else: # we closed before establishing a connection (onConnect) or # finishing WebSocket negotiation (onOpen): errback self.factory.d.errback(error.ConnectError(reason)) class WSFactory(websocket.WebSocketClientFactory): protocol = WSClient def buildProtocol(self, addr): proto = websocket.WebSocketClientFactory.buildProtocol(self, addr) proto.wormhole = self.wormhole proto.wormhole_open = False return proto class _GetCode: def __init__(self, code_length, send_command, timing): self._code_length = code_length self._send_command = send_command self._timing = timing self._allocated_d = defer.Deferred() @inlineCallbacks def go(self): with self._timing.add("allocate"): self._send_command("allocate") nameplate_id = yield self._allocated_d code = codes.make_code(nameplate_id, self._code_length) assert isinstance(code, type("")), type(code) returnValue(code) def _response_handle_allocated(self, msg): nid = msg["nameplate"] assert isinstance(nid, type("")), type(nid) self._allocated_d.callback(nid) class _InputCode: def __init__(self, reactor, prompt, code_length, send_command, timing): self._reactor = reactor self._prompt = prompt self._code_length = code_length self._send_command = send_command self._timing = timing @inlineCallbacks def _list(self): self._lister_d = defer.Deferred() self._send_command("list") nameplates = yield self._lister_d self._lister_d = None returnValue(nameplates) def _list_blocking(self): return blockingCallFromThread(self._reactor, self._list) @inlineCallbacks def go(self): # fetch the list of nameplates ahead of time, to give us a chance to # discover the welcome message (and warn the user about an obsolete # client) # # TODO: send the request early, show the prompt right away, hide the # latency in the user's indecision and slow typing. If we're lucky # the answer will come back before they hit TAB. initial_nameplate_ids = yield self._list() with self._timing.add("input code", waiting="user"): t = self._reactor.addSystemEventTrigger("before", "shutdown", self._warn_readline) code = yield deferToThread(codes.input_code_with_completion, self._prompt, initial_nameplate_ids, self._list_blocking, self._code_length) self._reactor.removeSystemEventTrigger(t) returnValue(code) def _response_handle_nameplates(self, msg): nameplates = msg["nameplates"] assert isinstance(nameplates, list), type(nameplates) nids = [] for n in nameplates: assert isinstance(n, dict), type(n) nameplate_id = n["id"] assert isinstance(nameplate_id, type("")), type(nameplate_id) nids.append(nameplate_id) self._lister_d.callback(nids) def _warn_readline(self): # When our process receives a SIGINT, Twisted's SIGINT handler will # stop the reactor and wait for all threads to terminate before the # process exits. However, if we were waiting for # input_code_with_completion() when SIGINT happened, the readline # thread will be blocked waiting for something on stdin. Trick the # user into satisfying the blocking read so we can exit. print("\nCommand interrupted: please press Return to quit", file=sys.stderr) # Other potential approaches to this problem: # * hard-terminate our process with os._exit(1), but make sure the # tty gets reset to a normal mode ("cooked"?) first, so that the # next shell command the user types is echoed correctly # * track down the thread (t.p.threadable.getThreadID from inside the # thread), get a cffi binding to pthread_kill, deliver SIGINT to it # * allocate a pty pair (pty.openpty), replace sys.stdin with the # slave, build a pty bridge that copies bytes (and other PTY # things) from the real stdin to the master, then close the slave # at shutdown, so readline sees EOF # * write tab-completion and basic editing (TTY raw mode, # backspace-is-erase) without readline, probably with curses or # twisted.conch.insults # * write a separate program to get codes (maybe just "wormhole # --internal-get-code"), run it as a subprocess, let it inherit # stdin/stdout, send it SIGINT when we receive SIGINT ourselves. It # needs an RPC mechanism (over some extra file descriptors) to ask # us to fetch the current nameplate_id list. # # Note that hard-terminating our process with os.kill(os.getpid(), # signal.SIGKILL), or SIGTERM, doesn't seem to work: the thread # doesn't see the signal, and we must still wait for stdin to make # readline finish. class _WelcomeHandler: def __init__(self, url, current_version, signal_error): self._ws_url = url self._version_warning_displayed = False self._current_version = current_version self._signal_error = signal_error def handle_welcome(self, welcome): if "motd" in welcome: motd_lines = welcome["motd"].splitlines() motd_formatted = "\n ".join(motd_lines) print("Server (at %s) says:\n %s" % (self._ws_url, motd_formatted), file=sys.stderr) # Only warn if we're running a release version (e.g. 0.0.6, not # 0.0.6-DISTANCE-gHASH). Only warn once. if ("current_cli_version" in welcome and "-" not in self._current_version and not self._version_warning_displayed and welcome["current_cli_version"] != self._current_version): print("Warning: errors may occur unless both sides are running the same version", file=sys.stderr) print("Server claims %s is current, but ours is %s" % (welcome["current_cli_version"], self._current_version), file=sys.stderr) self._version_warning_displayed = True if "error" in welcome: return self._signal_error(WelcomeError(welcome["error"]), "unwelcome") # states for nameplates, mailboxes, and the websocket connection (CLOSED, OPENING, OPEN, CLOSING) = ("closed", "opening", "open", "closing") class _Wormhole: DEBUG = False def __init__(self, appid, relay_url, reactor, tor_manager, timing): self._appid = appid self._ws_url = relay_url self._reactor = reactor self._tor_manager = tor_manager self._timing = timing self._welcomer = _WelcomeHandler(self._ws_url, __version__, self._signal_error) self._side = bytes_to_hexstr(os.urandom(5)) self._connection_state = CLOSED self._connection_waiters = [] self._ws_t = None self._started_get_code = False self._get_code = None self._started_input_code = False self._input_code_waiter = None self._code = None self._nameplate_id = None self._nameplate_state = CLOSED self._mailbox_id = None self._mailbox_state = CLOSED self._flag_need_nameplate = True self._flag_need_to_see_mailbox_used = True self._flag_need_to_build_msg1 = True self._flag_need_to_send_PAKE = True self._establish_key_called = False self._key_waiter = None self._key = None self._version_message = None self._version_checked = False self._get_verifier_called = False self._verifier = None # bytes self._verify_result = None # bytes or a Failure self._verifier_waiter = None self._my_versions = {} # sent self._their_versions = {} # received self._close_called = False # the close() API has been called self._closing = False # we've started shutdown self._disconnect_waiter = defer.Deferred() self._error = None self._next_send_phase = 0 # send() queues plaintext here, waiting for a connection and the key self._plaintext_to_send = [] # (phase, plaintext) self._sent_phases = set() # to detect double-send self._next_receive_phase = 0 self._receive_waiters = {} # phase -> Deferred self._received_messages = {} # phase -> plaintext # API METHODS for applications to call # You must use at least one of these entry points, to establish the # wormhole code. Other APIs will stall or be queued until we have one. # entry point 1: generate a new code. returns a Deferred def get_code(self, code_length=2): # XX rename to allocate_code()? create_? return self._API_get_code(code_length) # entry point 2: interactively type in a code, with completion. returns # Deferred def input_code(self, prompt="Enter wormhole code: ", code_length=2): return self._API_input_code(prompt, code_length) # entry point 3: paste in a fully-formed code. No return value. def set_code(self, code): self._API_set_code(code) # todo: restore-saved-state entry points def establish_key(self): """ returns a Deferred that fires when we've established the shared key. When successful, the Deferred fires with a simple `True`, otherwise it fails. """ return self._API_establish_key() def verify(self): """Returns a Deferred that fires when we've heard back from the other side, and have confirmed that they used the right wormhole code. When successful, the Deferred fires with a "verifier" (a bytestring) which can be compared out-of-band before making additional API calls. If they used the wrong wormhole code, the Deferred errbacks with WrongPasswordError. """ return self._API_verify() def send(self, outbound_data): return self._API_send(outbound_data) def get(self): return self._API_get() def derive_key(self, purpose, length): """Derive a new key from the established wormhole channel for some other purpose. This is a deterministic randomized function of the session key and the 'purpose' string (unicode/py3-string). This cannot be called until verify() or get() has fired. """ return self._API_derive_key(purpose, length) def close(self, res=None): """Collapse the wormhole, freeing up server resources and flushing all pending messages. Returns a Deferred that fires when everything is done. It fires with any argument close() was given, to enable use as a d.addBoth() handler: w = wormhole(...) d = w.get() .. d.addBoth(w.close) return d Another reasonable approach is to use inlineCallbacks: @inlineCallbacks def pair(self, code): w = wormhole(...) try: them = yield w.get() finally: yield w.close() """ return self._API_close(res) # INTERNAL METHODS beyond here def _start(self): d = self._connect() # causes stuff to happen d.addErrback(log.err) return d # fires when connection is established, if you care def _make_endpoint(self, hostname, port): if self._tor_manager: return self._tor_manager.get_endpoint_for(hostname, port) # note: HostnameEndpoints have a default 30s timeout return endpoints.HostnameEndpoint(self._reactor, hostname, port) def _connect(self): # TODO: if we lose the connection, make a new one, re-establish the # state assert self._side self._connection_state = OPENING self._ws_t = self._timing.add("open websocket") p = urlparse(self._ws_url) f = WSFactory(self._ws_url) f.setProtocolOptions(autoPingInterval=60, autoPingTimeout=600) f.wormhole = self f.d = defer.Deferred() # TODO: if hostname="localhost", I get three factories starting # and stopping (maybe 127.0.0.1, ::1, and something else?), and # an error in the factory is masked. ep = self._make_endpoint(p.hostname, p.port or 80) # .connect errbacks if the TCP connection fails d = ep.connect(f) d.addCallback(self._event_connected) # f.d is errbacked if WebSocket negotiation fails, and the WebSocket # drops any data sent before onOpen() fires, so we must wait for it d.addCallback(lambda _: f.d) d.addCallback(self._event_ws_opened) return d def _event_connected(self, ws): self._ws = ws if self._ws_t: self._ws_t.finish() def _event_ws_opened(self, _): self._connection_state = OPEN if self._closing: return self._maybe_finished_closing() self._ws_send_command("bind", appid=self._appid, side=self._side) self._maybe_claim_nameplate() self._maybe_send_pake() waiters, self._connection_waiters = self._connection_waiters, [] for d in waiters: d.callback(None) def _when_connected(self): if self._connection_state == OPEN: return defer.succeed(None) d = defer.Deferred() self._connection_waiters.append(d) return d def _ws_send_command(self, mtype, **kwargs): # msgid is used by misc/dump-timing.py to correlate our sends with # their receives, and vice versa. They are also correlated with the # ACKs we get back from the server (which we otherwise ignore). There # are so few messages, 16 bits is enough to be mostly-unique. if self.DEBUG: print("SEND", mtype) kwargs["id"] = bytes_to_hexstr(os.urandom(2)) kwargs["type"] = mtype payload = dict_to_bytes(kwargs) self._timing.add("ws_send", _side=self._side, **kwargs) self._ws.sendMessage(payload, False) def _ws_dispatch_response(self, payload): msg = bytes_to_dict(payload) if self.DEBUG and msg["type"]!="ack": print("DIS", msg["type"], msg) self._timing.add("ws_receive", _side=self._side, message=msg) mtype = msg["type"] meth = getattr(self, "_response_handle_"+mtype, None) if not meth: # make tests fail, but real application will ignore it log.err(ValueError("Unknown inbound message type %r" % (msg,))) return return meth(msg) def _response_handle_ack(self, msg): pass def _response_handle_welcome(self, msg): self._welcomer.handle_welcome(msg["welcome"]) # entry point 1: generate a new code @inlineCallbacks def _API_get_code(self, code_length): if self._code is not None: raise InternalError if self._started_get_code: raise InternalError self._started_get_code = True with self._timing.add("API get_code"): yield self._when_connected() gc = _GetCode(code_length, self._ws_send_command, self._timing) self._get_code = gc self._response_handle_allocated = gc._response_handle_allocated # TODO: signal_error code = yield gc.go() self._get_code = None self._nameplate_state = OPEN self._event_learned_code(code) returnValue(code) # entry point 2: interactively type in a code, with completion @inlineCallbacks def _API_input_code(self, prompt, code_length): if self._code is not None: raise InternalError if self._started_input_code: raise InternalError self._started_input_code = True with self._timing.add("API input_code"): yield self._when_connected() ic = _InputCode(self._reactor, prompt, code_length, self._ws_send_command, self._timing) self._response_handle_nameplates = ic._response_handle_nameplates # we reveal the Deferred we're waiting on, so _signal_error can # wake us up if something goes wrong (like a welcome error) self._input_code_waiter = ic.go() code = yield self._input_code_waiter self._input_code_waiter = None self._event_learned_code(code) returnValue(None) # entry point 3: paste in a fully-formed code def _API_set_code(self, code): self._timing.add("API set_code") if not isinstance(code, type(u"")): raise TypeError("Unexpected code type '{}'".format(type(code))) if self._code is not None: raise InternalError self._event_learned_code(code) # TODO: entry point 4: restore pre-contact saved state (we haven't heard # from the peer yet, so we still need the nameplate) # TODO: entry point 5: restore post-contact saved state (so we don't need # or use the nameplate, only the mailbox) def _restore_post_contact_state(self, state): # ... self._flag_need_nameplate = False #self._mailbox_id = X(state) self._event_learned_mailbox() def _event_learned_code(self, code): self._timing.add("code established") # bail out early if the password contains spaces... # this should raise a useful error if ' ' in code: raise KeyFormatError("code (%s) contains spaces." % code) self._code = code mo = re.search(r'^(\d+)-', code) if not mo: raise ValueError("code (%s) must start with NN-" % code) nid = mo.group(1) assert isinstance(nid, type("")), type(nid) self._nameplate_id = nid # fire more events self._maybe_build_msg1() self._event_learned_nameplate() def _maybe_build_msg1(self): if not (self._code and self._flag_need_to_build_msg1): return with self._timing.add("pake1", waiting="crypto"): self._sp = SPAKE2_Symmetric(to_bytes(self._code), idSymmetric=to_bytes(self._appid)) self._msg1 = self._sp.start() self._flag_need_to_build_msg1 = False self._event_built_msg1() def _event_built_msg1(self): self._maybe_send_pake() # every _maybe_X starts with a set of conditions # for each such condition Y, every _event_Y must call _maybe_X def _event_learned_nameplate(self): self._maybe_claim_nameplate() def _maybe_claim_nameplate(self): if not (self._nameplate_id and self._connection_state == OPEN): return self._ws_send_command("claim", nameplate=self._nameplate_id) self._nameplate_state = OPEN def _response_handle_claimed(self, msg): mailbox_id = msg["mailbox"] assert isinstance(mailbox_id, type("")), type(mailbox_id) self._mailbox_id = mailbox_id self._event_learned_mailbox() def _event_learned_mailbox(self): if not self._mailbox_id: raise InternalError assert self._mailbox_state == CLOSED, self._mailbox_state if self._closing: return self._ws_send_command("open", mailbox=self._mailbox_id) self._mailbox_state = OPEN # causes old messages to be sent now, and subscribes to new messages self._maybe_send_pake() self._maybe_send_phase_messages() def _maybe_send_pake(self): # TODO: deal with reentrant call if not (self._connection_state == OPEN and self._mailbox_state == OPEN and self._flag_need_to_send_PAKE): return body = {"pake_v1": bytes_to_hexstr(self._msg1)} payload = dict_to_bytes(body) self._msg_send("pake", payload) self._flag_need_to_send_PAKE = False def _event_received_pake(self, pake_msg): payload = bytes_to_dict(pake_msg) msg2 = hexstr_to_bytes(payload["pake_v1"]) with self._timing.add("pake2", waiting="crypto"): self._key = self._sp.finish(msg2) self._event_established_key() def _event_established_key(self): self._timing.add("key established") self._maybe_notify_key() # both sides send different (random) version messages self._send_version_message() verifier = self._derive_key(b"wormhole:verifier") self._event_computed_verifier(verifier) self._maybe_check_version() self._maybe_send_phase_messages() def _API_establish_key(self): if self._error: return defer.fail(self._error) if self._establish_key_called: raise InternalError self._establish_key_called = True if self._key is not None: return defer.succeed(True) self._key_waiter = defer.Deferred() return self._key_waiter def _maybe_notify_key(self): if self._key is None: return if self._error: result = failure.Failure(self._error) else: result = True if self._key_waiter and not self._key_waiter.called: self._key_waiter.callback(result) def _send_version_message(self): # this is encrypted like a normal phase message, and includes a # dictionary of version flags to let the other Wormhole know what # we're capable of (for future expansion) plaintext = dict_to_bytes(self._my_versions) phase = "version" data_key = self._derive_phase_key(self._side, phase) encrypted = self._encrypt_data(data_key, plaintext) self._msg_send(phase, encrypted) def _API_verify(self): if self._error: return defer.fail(self._error) if self._get_verifier_called: raise InternalError self._get_verifier_called = True if self._verify_result: return defer.succeed(self._verify_result) # bytes or Failure self._verifier_waiter = defer.Deferred() return self._verifier_waiter def _event_computed_verifier(self, verifier): self._verifier = verifier self._maybe_notify_verify() def _maybe_notify_verify(self): if not (self._verifier and self._version_checked): return if self._error: self._verify_result = failure.Failure(self._error) else: self._verify_result = self._verifier if self._verifier_waiter and not self._verifier_waiter.called: self._verifier_waiter.callback(self._verify_result) def _event_received_version(self, side, body): # We ought to have the master key by now, because sensible peers # should always send "pake" before sending "version". It might be # nice to relax this requirement, which means storing the received # version message, and having _event_established_key call # _check_version() self._version_message = (side, body) self._maybe_check_version() def _maybe_check_version(self): if not (self._key and self._version_message): return if self._version_checked: return self._version_checked = True side, body = self._version_message data_key = self._derive_phase_key(side, "version") try: plaintext = self._decrypt_data(data_key, body) except CryptoError: # this makes all API calls fail if self.DEBUG: print("CONFIRM FAILED") self._signal_error(WrongPasswordError(), "scary") return msg = bytes_to_dict(plaintext) self._version_received(msg) self._maybe_notify_verify() def _version_received(self, msg): self._their_versions = msg def _API_send(self, outbound_data): if self._error: raise self._error if not isinstance(outbound_data, type(b"")): raise TypeError(type(outbound_data)) phase = self._next_send_phase self._next_send_phase += 1 self._plaintext_to_send.append( (phase, outbound_data) ) with self._timing.add("API send", phase=phase): self._maybe_send_phase_messages() def _derive_phase_key(self, side, phase): assert isinstance(side, type("")), type(side) assert isinstance(phase, type("")), type(phase) side_bytes = side.encode("ascii") phase_bytes = phase.encode("ascii") purpose = (b"wormhole:phase:" + sha256(side_bytes).digest() + sha256(phase_bytes).digest()) return self._derive_key(purpose) def _maybe_send_phase_messages(self): # TODO: deal with reentrant call if not (self._connection_state == OPEN and self._mailbox_state == OPEN and self._key): return plaintexts = self._plaintext_to_send self._plaintext_to_send = [] for pm in plaintexts: (phase_int, plaintext) = pm assert isinstance(phase_int, int), type(phase_int) phase = "%d" % phase_int data_key = self._derive_phase_key(self._side, phase) encrypted = self._encrypt_data(data_key, plaintext) self._msg_send(phase, encrypted) def _encrypt_data(self, key, data): # Without predefined roles, we can't derive predictably unique keys # for each side, so we use the same key for both. We use random # nonces to keep the messages distinct, and we automatically ignore # reflections. # TODO: HKDF(side, nonce, key) ?? include 'side' to prevent # reflections, since we no longer compare messages assert isinstance(key, type(b"")), type(key) assert isinstance(data, type(b"")), type(data) assert len(key) == SecretBox.KEY_SIZE, len(key) box = SecretBox(key) nonce = utils.random(SecretBox.NONCE_SIZE) return box.encrypt(data, nonce) def _msg_send(self, phase, body): if phase in self._sent_phases: raise InternalError assert self._mailbox_state == OPEN, self._mailbox_state self._sent_phases.add(phase) # TODO: retry on failure, with exponential backoff. We're guarding # against the rendezvous server being temporarily offline. self._timing.add("add", phase=phase) self._ws_send_command("add", phase=phase, body=bytes_to_hexstr(body)) def _event_mailbox_used(self): if self.DEBUG: print("_event_mailbox_used") if self._flag_need_to_see_mailbox_used: self._maybe_release_nameplate() self._flag_need_to_see_mailbox_used = False def _API_derive_key(self, purpose, length): if self._error: raise self._error if self._key is None: raise InternalError # call derive_key after get_verifier() or get() if not isinstance(purpose, type("")): raise TypeError(type(purpose)) return self._derive_key(to_bytes(purpose), length) def _derive_key(self, purpose, length=SecretBox.KEY_SIZE): if not isinstance(purpose, type(b"")): raise TypeError(type(purpose)) if self._key is None: raise InternalError # call derive_key after get_verifier() or get() return HKDF(self._key, length, CTXinfo=purpose) def _response_handle_message(self, msg): side = msg["side"] phase = msg["phase"] assert isinstance(phase, type("")), type(phase) body = hexstr_to_bytes(msg["body"]) if side == self._side: return self._event_received_peer_message(side, phase, body) def _event_received_peer_message(self, side, phase, body): # any message in the mailbox means we no longer need the nameplate self._event_mailbox_used() if self._closing: log.msg("received peer message while closing '%s'" % phase) if phase in self._received_messages: log.msg("ignoring duplicate peer message '%s'" % phase) return if phase == "pake": self._received_messages["pake"] = body return self._event_received_pake(body) if phase == "version": self._received_messages["version"] = body return self._event_received_version(side, body) if re.search(r'^\d+$', phase): return self._event_received_phase_message(side, phase, body) # ignore unrecognized phases, for forwards-compatibility log.msg("received unknown phase '%s'" % phase) def _event_received_phase_message(self, side, phase, body): # It's a numbered phase message, aimed at the application above us. # Decrypt and deliver upstairs, notifying anyone waiting on it try: data_key = self._derive_phase_key(side, phase) plaintext = self._decrypt_data(data_key, body) except CryptoError: e = WrongPasswordError() self._signal_error(e, "scary") # flunk all other API calls # make tests fail, if they aren't explicitly catching it if self.DEBUG: print("CryptoError in msg received") log.err(e) if self.DEBUG: print(" did log.err", e) return # ignore this message self._received_messages[phase] = plaintext if phase in self._receive_waiters: d = self._receive_waiters.pop(phase) d.callback(plaintext) def _decrypt_data(self, key, encrypted): assert isinstance(key, type(b"")), type(key) assert isinstance(encrypted, type(b"")), type(encrypted) assert len(key) == SecretBox.KEY_SIZE, len(key) box = SecretBox(key) data = box.decrypt(encrypted) return data def _API_get(self): if self._error: return defer.fail(self._error) phase = "%d" % self._next_receive_phase self._next_receive_phase += 1 with self._timing.add("API get", phase=phase): if phase in self._received_messages: return defer.succeed(self._received_messages[phase]) d = self._receive_waiters[phase] = defer.Deferred() return d def _signal_error(self, error, mood): if self.DEBUG: print("_signal_error", error, mood) if self._error: return self._maybe_close(error, mood) if self.DEBUG: print("_signal_error done") @inlineCallbacks def _API_close(self, res, mood="happy"): if self.DEBUG: print("close") if self._close_called: raise InternalError self._close_called = True self._maybe_close(WormholeClosedError(), mood) if self.DEBUG: print("waiting for disconnect") yield self._disconnect_waiter returnValue(res) def _maybe_close(self, error, mood): if self._closing: return # ordering constraints: # * must wait for nameplate/mailbox acks before closing the websocket # * must mark APIs for failure before errbacking Deferreds # * since we give up control # * must mark self._closing before errbacking Deferreds # * since caller may call close() when we give up control # * and close() will reenter _maybe_close self._error = error # causes new API calls to fail # since we're about to give up control by errbacking any API # Deferreds, set self._closing, to make sure that a new call to # close() isn't going to confuse anything self._closing = True # now errback all API deferreds except close(): get_code, # input_code, verify, get if self._input_code_waiter and not self._input_code_waiter.called: self._input_code_waiter.errback(error) for d in self._connection_waiters: # input_code, get_code (early) if self.DEBUG: print("EB cw") d.errback(error) if self._get_code: # get_code (late) if self.DEBUG: print("EB gc") self._get_code._allocated_d.errback(error) if self._verifier_waiter and not self._verifier_waiter.called: if self.DEBUG: print("EB VW") self._verifier_waiter.errback(error) if self._key_waiter and not self._key_waiter.called: if self.DEBUG: print("EB KW") self._key_waiter.errback(error) for d in self._receive_waiters.values(): if self.DEBUG: print("EB RW") d.errback(error) # Release nameplate and close mailbox, if either was claimed/open. # Since _closing is True when both ACKs come back, the handlers will # close the websocket. When *that* finishes, _disconnect_waiter() # will fire. self._maybe_release_nameplate() self._maybe_close_mailbox(mood) # In the off chance we got closed before we even claimed the # nameplate, give _maybe_finished_closing a chance to run now. self._maybe_finished_closing() def _maybe_release_nameplate(self): if self.DEBUG: print("_maybe_release_nameplate", self._nameplate_state) if self._nameplate_state == OPEN: if self.DEBUG: print(" sending release") self._ws_send_command("release") self._nameplate_state = CLOSING def _response_handle_released(self, msg): self._nameplate_state = CLOSED self._maybe_finished_closing() def _maybe_close_mailbox(self, mood): if self.DEBUG: print("_maybe_close_mailbox", self._mailbox_state) if self._mailbox_state == OPEN: if self.DEBUG: print(" sending close") self._ws_send_command("close", mood=mood) self._mailbox_state = CLOSING def _response_handle_closed(self, msg): self._mailbox_state = CLOSED self._maybe_finished_closing() def _maybe_finished_closing(self): if self.DEBUG: print("_maybe_finished_closing", self._closing, self._nameplate_state, self._mailbox_state, self._connection_state) if not self._closing: return if (self._nameplate_state == CLOSED and self._mailbox_state == CLOSED and self._connection_state == OPEN): self._connection_state = CLOSING self._drop_connection() def _drop_connection(self): # separate method so it can be overridden by tests self._ws.transport.loseConnection() # probably flushes output # calls _ws_closed() when done def _ws_closed(self, wasClean, code, reason): # For now (until we add reconnection), losing the websocket means # losing everything. Make all API callers fail. Help someone waiting # in close() to finish self._connection_state = CLOSED self._disconnect_waiter.callback(None) self._maybe_finished_closing() # what needs to happen when _ws_closed() happens unexpectedly # * errback all API deferreds # * maybe: cause new API calls to fail # * obviously can't release nameplate or close mailbox # * can't re-close websocket # * close(wait=True) callers should fire right away def wormhole(appid, relay_url, reactor, tor_manager=None, timing=None): timing = timing or DebugTiming() w = _Wormhole(appid, relay_url, reactor, tor_manager, timing) w._start() return w #def wormhole_from_serialized(data, reactor, timing=None): # timing = timing or DebugTiming() # w = _Wormhole.from_serialized(data, reactor, timing) # return w magic-wormhole-0.9.1/src/wormhole/xfer_util.py0000644000076600000240000000607313024532264022062 0ustar warnerstaff00000000000000import json from twisted.internet.defer import inlineCallbacks, returnValue from .wormhole import wormhole @inlineCallbacks def receive(reactor, appid, relay_url, code, use_tor=None, on_code=None): """ This is a convenience API which returns a Deferred that callbacks with a single chunk of data from another wormhole (and then closes the wormhole). Under the hood, it's just using an instance returned from :func:`wormhole.wormhole`. This is similar to the `wormhole receive` command. :param unicode appid: our application ID :param unicode relay_url: the relay URL to use :param unicode code: a pre-existing code to use, or None :param bool use_tor: True if we should use Tor, False to not use it (None for default) :param on_code: if not None, this is called when we have a code (even if you passed in one explicitly) :type on_code: single-argument callable """ wh = wormhole(appid, relay_url, reactor, use_tor) if code is None: code = yield wh.get_code() else: wh.set_code(code) # we'll call this no matter what, even if you passed in a code -- # maybe it should be only in the 'if' block above? if on_code: on_code(code) data = yield wh.get() data = json.loads(data.decode("utf-8")) offer = data.get('offer', None) if not offer: raise Exception( "Do not understand response: {}".format(data) ) msg = None if 'message' in offer: msg = offer['message'] wh.send(json.dumps({"answer": {"message_ack": "ok"}}).encode("utf-8")) else: raise Exception( "Unknown offer type: {}".format(offer.keys()) ) yield wh.close() returnValue(msg) @inlineCallbacks def send(reactor, appid, relay_url, data, code, use_tor=None, on_code=None): """ This is a convenience API which returns a Deferred that callbacks after a single chunk of data has been sent to another wormhole. Under the hood, it's just using an instance returned from :func:`wormhole.wormhole`. This is similar to the `wormhole send` command. :param unicode appid: the application ID :param unicode relay_url: the relay URL to use :param unicode code: a pre-existing code to use, or None :param bool use_tor: True if we should use Tor, False to not use it (None for default) :param on_code: if not None, this is called when we have a code (even if you passed in one explicitly) :type on_code: single-argument callable """ wh = wormhole(appid, relay_url, reactor, use_tor) if code is None: code = yield wh.get_code() else: wh.set_code(code) if on_code: on_code(code) wh.send( json.dumps({ "offer": { "message": data } }).encode("utf-8") ) data = yield wh.get() data = json.loads(data.decode("utf-8")) answer = data.get('answer', None) yield wh.close() if answer: returnValue(None) else: raise Exception( "Unknown answer: {}".format(data) ) magic-wormhole-0.9.1/tox.ini0000644000076600000240000000326313031527221016370 0ustar warnerstaff00000000000000# Tox (http://tox.testrun.org/) is a tool for running tests # in multiple virtualenvs. This configuration file will run the # test suite on all supported python versions. To use it, "pip install tox" # and then run "tox" from this directory. [tox] envlist = {py27,py33,py34,py35,py36,pypy} skip_missing_interpreters = True # On windows we need "pypiwin32" installed. It's supposedly possible to make # Twisted do this by depending upon "twisted[windows]" instead of just # "twisted", but when I try this via Appveyor, the extra is ignored. # git+https://github.com/twisted/twisted#egg=twisted{env:TWISTED_EXTRAS:} # #twisted{env:TWISTED_EXTRAS:} # So instead we have the .appveyor.yml set EXTRA_DEPENDENCY=pypiwin32. In # other environments, this variable will be empty, and tox will ignore the # blank value. [testenv] deps = pyflakes >= 1.2.3 mock usedevelop=true install_command = pip install {packages} commands = pyflakes setup.py src wormhole --version python -m wormhole.test.run_trial {posargs:wormhole} # on windows, trial is installed as venv/bin/trial.py, not .exe, but (at # least appveyor) adds .PY to $PATHEXT. So "trial wormhole" might work on # windows, and certainly does on unix. To get "coverage run" to work, we need # a script name (since "python -m twisted.scripts.trial" doesn't have a 'if # __name__ == "__main__": run()' -style clause). The script name will vary on # the platform. [testenv:coverage] deps = pyflakes >= 1.2.3 mock coverage usedevelop=true install_command = pip install {packages} commands = pyflakes setup.py src wormhole --version coverage run --branch -m wormhole.test.run_trial {posargs:wormhole} coverage xml magic-wormhole-0.9.1/versioneer.py0000644000076600000240000020575313024532264017625 0ustar warnerstaff00000000000000 # Version: 0.17 """The Versioneer - like a rocketeer, but for versions. The Versioneer ============== * like a rocketeer, but for versions! * https://github.com/warner/python-versioneer * Brian Warner * License: Public Domain * Compatible With: python2.6, 2.7, 3.2, 3.3, 3.4, 3.5, and pypy * [![Latest Version] (https://pypip.in/version/versioneer/badge.svg?style=flat) ](https://pypi.python.org/pypi/versioneer/) * [![Build Status] (https://travis-ci.org/warner/python-versioneer.png?branch=master) ](https://travis-ci.org/warner/python-versioneer) This is a tool for managing a recorded version number in distutils-based python projects. The goal is to remove the tedious and error-prone "update the embedded version string" step from your release process. Making a new release should be as easy as recording a new tag in your version-control system, and maybe making new tarballs. ## Quick Install * `pip install versioneer` to somewhere to your $PATH * add a `[versioneer]` section to your setup.cfg (see below) * run `versioneer install` in your source tree, commit the results ## Version Identifiers Source trees come from a variety of places: * a version-control system checkout (mostly used by developers) * a nightly tarball, produced by build automation * a snapshot tarball, produced by a web-based VCS browser, like github's "tarball from tag" feature * a release tarball, produced by "setup.py sdist", distributed through PyPI Within each source tree, the version identifier (either a string or a number, this tool is format-agnostic) can come from a variety of places: * ask the VCS tool itself, e.g. "git describe" (for checkouts), which knows about recent "tags" and an absolute revision-id * the name of the directory into which the tarball was unpacked * an expanded VCS keyword ($Id$, etc) * a `_version.py` created by some earlier build step For released software, the version identifier is closely related to a VCS tag. Some projects use tag names that include more than just the version string (e.g. "myproject-1.2" instead of just "1.2"), in which case the tool needs to strip the tag prefix to extract the version identifier. For unreleased software (between tags), the version identifier should provide enough information to help developers recreate the same tree, while also giving them an idea of roughly how old the tree is (after version 1.2, before version 1.3). Many VCS systems can report a description that captures this, for example `git describe --tags --dirty --always` reports things like "0.7-1-g574ab98-dirty" to indicate that the checkout is one revision past the 0.7 tag, has a unique revision id of "574ab98", and is "dirty" (it has uncommitted changes. The version identifier is used for multiple purposes: * to allow the module to self-identify its version: `myproject.__version__` * to choose a name and prefix for a 'setup.py sdist' tarball ## Theory of Operation Versioneer works by adding a special `_version.py` file into your source tree, where your `__init__.py` can import it. This `_version.py` knows how to dynamically ask the VCS tool for version information at import time. `_version.py` also contains `$Revision$` markers, and the installation process marks `_version.py` to have this marker rewritten with a tag name during the `git archive` command. As a result, generated tarballs will contain enough information to get the proper version. To allow `setup.py` to compute a version too, a `versioneer.py` is added to the top level of your source tree, next to `setup.py` and the `setup.cfg` that configures it. This overrides several distutils/setuptools commands to compute the version when invoked, and changes `setup.py build` and `setup.py sdist` to replace `_version.py` with a small static file that contains just the generated version data. ## Installation See [INSTALL.md](./INSTALL.md) for detailed installation instructions. ## Version-String Flavors Code which uses Versioneer can learn about its version string at runtime by importing `_version` from your main `__init__.py` file and running the `get_versions()` function. From the "outside" (e.g. in `setup.py`), you can import the top-level `versioneer.py` and run `get_versions()`. Both functions return a dictionary with different flavors of version information: * `['version']`: A condensed version string, rendered using the selected style. This is the most commonly used value for the project's version string. The default "pep440" style yields strings like `0.11`, `0.11+2.g1076c97`, or `0.11+2.g1076c97.dirty`. See the "Styles" section below for alternative styles. * `['full-revisionid']`: detailed revision identifier. For Git, this is the full SHA1 commit id, e.g. "1076c978a8d3cfc70f408fe5974aa6c092c949ac". * `['date']`: Date and time of the latest `HEAD` commit. For Git, it is the commit date in ISO 8601 format. This will be None if the date is not available. * `['dirty']`: a boolean, True if the tree has uncommitted changes. Note that this is only accurate if run in a VCS checkout, otherwise it is likely to be False or None * `['error']`: if the version string could not be computed, this will be set to a string describing the problem, otherwise it will be None. It may be useful to throw an exception in setup.py if this is set, to avoid e.g. creating tarballs with a version string of "unknown". Some variants are more useful than others. Including `full-revisionid` in a bug report should allow developers to reconstruct the exact code being tested (or indicate the presence of local changes that should be shared with the developers). `version` is suitable for display in an "about" box or a CLI `--version` output: it can be easily compared against release notes and lists of bugs fixed in various releases. The installer adds the following text to your `__init__.py` to place a basic version in `YOURPROJECT.__version__`: from ._version import get_versions __version__ = get_versions()['version'] del get_versions ## Styles The setup.cfg `style=` configuration controls how the VCS information is rendered into a version string. The default style, "pep440", produces a PEP440-compliant string, equal to the un-prefixed tag name for actual releases, and containing an additional "local version" section with more detail for in-between builds. For Git, this is TAG[+DISTANCE.gHEX[.dirty]] , using information from `git describe --tags --dirty --always`. For example "0.11+2.g1076c97.dirty" indicates that the tree is like the "1076c97" commit but has uncommitted changes (".dirty"), and that this commit is two revisions ("+2") beyond the "0.11" tag. For released software (exactly equal to a known tag), the identifier will only contain the stripped tag, e.g. "0.11". Other styles are available. See details.md in the Versioneer source tree for descriptions. ## Debugging Versioneer tries to avoid fatal errors: if something goes wrong, it will tend to return a version of "0+unknown". To investigate the problem, run `setup.py version`, which will run the version-lookup code in a verbose mode, and will display the full contents of `get_versions()` (including the `error` string, which may help identify what went wrong). ## Known Limitations Some situations are known to cause problems for Versioneer. This details the most significant ones. More can be found on Github [issues page](https://github.com/warner/python-versioneer/issues). ### Subprojects Versioneer has limited support for source trees in which `setup.py` is not in the root directory (e.g. `setup.py` and `.git/` are *not* siblings). The are two common reasons why `setup.py` might not be in the root: * Source trees which contain multiple subprojects, such as [Buildbot](https://github.com/buildbot/buildbot), which contains both "master" and "slave" subprojects, each with their own `setup.py`, `setup.cfg`, and `tox.ini`. Projects like these produce multiple PyPI distributions (and upload multiple independently-installable tarballs). * Source trees whose main purpose is to contain a C library, but which also provide bindings to Python (and perhaps other langauges) in subdirectories. Versioneer will look for `.git` in parent directories, and most operations should get the right version string. However `pip` and `setuptools` have bugs and implementation details which frequently cause `pip install .` from a subproject directory to fail to find a correct version string (so it usually defaults to `0+unknown`). `pip install --editable .` should work correctly. `setup.py install` might work too. Pip-8.1.1 is known to have this problem, but hopefully it will get fixed in some later version. [Bug #38](https://github.com/warner/python-versioneer/issues/38) is tracking this issue. The discussion in [PR #61](https://github.com/warner/python-versioneer/pull/61) describes the issue from the Versioneer side in more detail. [pip PR#3176](https://github.com/pypa/pip/pull/3176) and [pip PR#3615](https://github.com/pypa/pip/pull/3615) contain work to improve pip to let Versioneer work correctly. Versioneer-0.16 and earlier only looked for a `.git` directory next to the `setup.cfg`, so subprojects were completely unsupported with those releases. ### Editable installs with setuptools <= 18.5 `setup.py develop` and `pip install --editable .` allow you to install a project into a virtualenv once, then continue editing the source code (and test) without re-installing after every change. "Entry-point scripts" (`setup(entry_points={"console_scripts": ..})`) are a convenient way to specify executable scripts that should be installed along with the python package. These both work as expected when using modern setuptools. When using setuptools-18.5 or earlier, however, certain operations will cause `pkg_resources.DistributionNotFound` errors when running the entrypoint script, which must be resolved by re-installing the package. This happens when the install happens with one version, then the egg_info data is regenerated while a different version is checked out. Many setup.py commands cause egg_info to be rebuilt (including `sdist`, `wheel`, and installing into a different virtualenv), so this can be surprising. [Bug #83](https://github.com/warner/python-versioneer/issues/83) describes this one, but upgrading to a newer version of setuptools should probably resolve it. ### Unicode version strings While Versioneer works (and is continually tested) with both Python 2 and Python 3, it is not entirely consistent with bytes-vs-unicode distinctions. Newer releases probably generate unicode version strings on py2. It's not clear that this is wrong, but it may be surprising for applications when then write these strings to a network connection or include them in bytes-oriented APIs like cryptographic checksums. [Bug #71](https://github.com/warner/python-versioneer/issues/71) investigates this question. ## Updating Versioneer To upgrade your project to a new release of Versioneer, do the following: * install the new Versioneer (`pip install -U versioneer` or equivalent) * edit `setup.cfg`, if necessary, to include any new configuration settings indicated by the release notes. See [UPGRADING](./UPGRADING.md) for details. * re-run `versioneer install` in your source tree, to replace `SRC/_version.py` * commit any changed files ## Future Directions This tool is designed to make it easily extended to other version-control systems: all VCS-specific components are in separate directories like src/git/ . The top-level `versioneer.py` script is assembled from these components by running make-versioneer.py . In the future, make-versioneer.py will take a VCS name as an argument, and will construct a version of `versioneer.py` that is specific to the given VCS. It might also take the configuration arguments that are currently provided manually during installation by editing setup.py . Alternatively, it might go the other direction and include code from all supported VCS systems, reducing the number of intermediate scripts. ## License To make Versioneer easier to embed, all its code is dedicated to the public domain. The `_version.py` that it creates is also in the public domain. Specifically, both are released under the Creative Commons "Public Domain Dedication" license (CC0-1.0), as described in https://creativecommons.org/publicdomain/zero/1.0/ . """ from __future__ import print_function try: import configparser except ImportError: import ConfigParser as configparser import errno import json import os import re import subprocess import sys class VersioneerConfig: """Container for Versioneer configuration parameters.""" def get_root(): """Get the project root directory. We require that all commands are run from the project root, i.e. the directory that contains setup.py, setup.cfg, and versioneer.py . """ root = os.path.realpath(os.path.abspath(os.getcwd())) setup_py = os.path.join(root, "setup.py") versioneer_py = os.path.join(root, "versioneer.py") if not (os.path.exists(setup_py) or os.path.exists(versioneer_py)): # allow 'python path/to/setup.py COMMAND' root = os.path.dirname(os.path.realpath(os.path.abspath(sys.argv[0]))) setup_py = os.path.join(root, "setup.py") versioneer_py = os.path.join(root, "versioneer.py") if not (os.path.exists(setup_py) or os.path.exists(versioneer_py)): err = ("Versioneer was unable to run the project root directory. " "Versioneer requires setup.py to be executed from " "its immediate directory (like 'python setup.py COMMAND'), " "or in a way that lets it use sys.argv[0] to find the root " "(like 'python path/to/setup.py COMMAND').") raise VersioneerBadRootError(err) try: # Certain runtime workflows (setup.py install/develop in a setuptools # tree) execute all dependencies in a single python process, so # "versioneer" may be imported multiple times, and python's shared # module-import table will cache the first one. So we can't use # os.path.dirname(__file__), as that will find whichever # versioneer.py was first imported, even in later projects. me = os.path.realpath(os.path.abspath(__file__)) me_dir = os.path.normcase(os.path.splitext(me)[0]) vsr_dir = os.path.normcase(os.path.splitext(versioneer_py)[0]) if me_dir != vsr_dir: print("Warning: build in %s is using versioneer.py from %s" % (os.path.dirname(me), versioneer_py)) except NameError: pass return root def get_config_from_root(root): """Read the project setup.cfg file to determine Versioneer config.""" # This might raise EnvironmentError (if setup.cfg is missing), or # configparser.NoSectionError (if it lacks a [versioneer] section), or # configparser.NoOptionError (if it lacks "VCS="). See the docstring at # the top of versioneer.py for instructions on writing your setup.cfg . setup_cfg = os.path.join(root, "setup.cfg") parser = configparser.SafeConfigParser() with open(setup_cfg, "r") as f: parser.readfp(f) VCS = parser.get("versioneer", "VCS") # mandatory def get(parser, name): if parser.has_option("versioneer", name): return parser.get("versioneer", name) return None cfg = VersioneerConfig() cfg.VCS = VCS cfg.style = get(parser, "style") or "" cfg.versionfile_source = get(parser, "versionfile_source") cfg.versionfile_build = get(parser, "versionfile_build") cfg.tag_prefix = get(parser, "tag_prefix") if cfg.tag_prefix in ("''", '""'): cfg.tag_prefix = "" cfg.parentdir_prefix = get(parser, "parentdir_prefix") cfg.verbose = get(parser, "verbose") return cfg class NotThisMethod(Exception): """Exception raised if a method is not valid for the current scenario.""" # these dictionaries contain VCS-specific tools LONG_VERSION_PY = {} HANDLERS = {} def register_vcs_handler(vcs, method): # decorator """Decorator to mark a method as the handler for a particular VCS.""" def decorate(f): """Store f in HANDLERS[vcs][method].""" if vcs not in HANDLERS: HANDLERS[vcs] = {} HANDLERS[vcs][method] = f return f return decorate def run_command(commands, args, cwd=None, verbose=False, hide_stderr=False, env=None): """Call the given command(s).""" assert isinstance(commands, list) p = None for c in commands: try: dispcmd = str([c] + args) # remember shell=False, so use git.cmd on windows, not just git p = subprocess.Popen([c] + args, cwd=cwd, env=env, stdout=subprocess.PIPE, stderr=(subprocess.PIPE if hide_stderr else None)) break except EnvironmentError: e = sys.exc_info()[1] if e.errno == errno.ENOENT: continue if verbose: print("unable to run %s" % dispcmd) print(e) return None, None else: if verbose: print("unable to find command, tried %s" % (commands,)) return None, None stdout = p.communicate()[0].strip() if sys.version_info[0] >= 3: stdout = stdout.decode() if p.returncode != 0: if verbose: print("unable to run %s (error)" % dispcmd) print("stdout was %s" % stdout) return None, p.returncode return stdout, p.returncode LONG_VERSION_PY['git'] = ''' # This file helps to compute a version number in source trees obtained from # git-archive tarball (such as those provided by githubs download-from-tag # feature). Distribution tarballs (built by setup.py sdist) and build # directories (produced by setup.py build) will contain a much shorter file # that just contains the computed version number. # This file is released into the public domain. Generated by # versioneer-0.17 (https://github.com/warner/python-versioneer) """Git implementation of _version.py.""" import errno import os import re import subprocess import sys def get_keywords(): """Get the keywords needed to look up the version information.""" # these strings will be replaced by git during git-archive. # setup.py/versioneer.py will grep for the variable names, so they must # each be defined on a line of their own. _version.py will just call # get_keywords(). git_refnames = "%(DOLLAR)sFormat:%%d%(DOLLAR)s" git_full = "%(DOLLAR)sFormat:%%H%(DOLLAR)s" git_date = "%(DOLLAR)sFormat:%%ci%(DOLLAR)s" keywords = {"refnames": git_refnames, "full": git_full, "date": git_date} return keywords class VersioneerConfig: """Container for Versioneer configuration parameters.""" def get_config(): """Create, populate and return the VersioneerConfig() object.""" # these strings are filled in when 'setup.py versioneer' creates # _version.py cfg = VersioneerConfig() cfg.VCS = "git" cfg.style = "%(STYLE)s" cfg.tag_prefix = "%(TAG_PREFIX)s" cfg.parentdir_prefix = "%(PARENTDIR_PREFIX)s" cfg.versionfile_source = "%(VERSIONFILE_SOURCE)s" cfg.verbose = False return cfg class NotThisMethod(Exception): """Exception raised if a method is not valid for the current scenario.""" LONG_VERSION_PY = {} HANDLERS = {} def register_vcs_handler(vcs, method): # decorator """Decorator to mark a method as the handler for a particular VCS.""" def decorate(f): """Store f in HANDLERS[vcs][method].""" if vcs not in HANDLERS: HANDLERS[vcs] = {} HANDLERS[vcs][method] = f return f return decorate def run_command(commands, args, cwd=None, verbose=False, hide_stderr=False, env=None): """Call the given command(s).""" assert isinstance(commands, list) p = None for c in commands: try: dispcmd = str([c] + args) # remember shell=False, so use git.cmd on windows, not just git p = subprocess.Popen([c] + args, cwd=cwd, env=env, stdout=subprocess.PIPE, stderr=(subprocess.PIPE if hide_stderr else None)) break except EnvironmentError: e = sys.exc_info()[1] if e.errno == errno.ENOENT: continue if verbose: print("unable to run %%s" %% dispcmd) print(e) return None, None else: if verbose: print("unable to find command, tried %%s" %% (commands,)) return None, None stdout = p.communicate()[0].strip() if sys.version_info[0] >= 3: stdout = stdout.decode() if p.returncode != 0: if verbose: print("unable to run %%s (error)" %% dispcmd) print("stdout was %%s" %% stdout) return None, p.returncode return stdout, p.returncode def versions_from_parentdir(parentdir_prefix, root, verbose): """Try to determine the version from the parent directory name. Source tarballs conventionally unpack into a directory that includes both the project name and a version string. We will also support searching up two directory levels for an appropriately named parent directory """ rootdirs = [] for i in range(3): dirname = os.path.basename(root) if dirname.startswith(parentdir_prefix): return {"version": dirname[len(parentdir_prefix):], "full-revisionid": None, "dirty": False, "error": None, "date": None} else: rootdirs.append(root) root = os.path.dirname(root) # up a level if verbose: print("Tried directories %%s but none started with prefix %%s" %% (str(rootdirs), parentdir_prefix)) raise NotThisMethod("rootdir doesn't start with parentdir_prefix") @register_vcs_handler("git", "get_keywords") def git_get_keywords(versionfile_abs): """Extract version information from the given file.""" # the code embedded in _version.py can just fetch the value of these # keywords. When used from setup.py, we don't want to import _version.py, # so we do it with a regexp instead. This function is not used from # _version.py. keywords = {} try: f = open(versionfile_abs, "r") for line in f.readlines(): if line.strip().startswith("git_refnames ="): mo = re.search(r'=\s*"(.*)"', line) if mo: keywords["refnames"] = mo.group(1) if line.strip().startswith("git_full ="): mo = re.search(r'=\s*"(.*)"', line) if mo: keywords["full"] = mo.group(1) if line.strip().startswith("git_date ="): mo = re.search(r'=\s*"(.*)"', line) if mo: keywords["date"] = mo.group(1) f.close() except EnvironmentError: pass return keywords @register_vcs_handler("git", "keywords") def git_versions_from_keywords(keywords, tag_prefix, verbose): """Get version information from git keywords.""" if not keywords: raise NotThisMethod("no keywords at all, weird") date = keywords.get("date") if date is not None: # git-2.2.0 added "%%cI", which expands to an ISO-8601 -compliant # datestamp. However we prefer "%%ci" (which expands to an "ISO-8601 # -like" string, which we must then edit to make compliant), because # it's been around since git-1.5.3, and it's too difficult to # discover which version we're using, or to work around using an # older one. date = date.strip().replace(" ", "T", 1).replace(" ", "", 1) refnames = keywords["refnames"].strip() if refnames.startswith("$Format"): if verbose: print("keywords are unexpanded, not using") raise NotThisMethod("unexpanded keywords, not a git-archive tarball") refs = set([r.strip() for r in refnames.strip("()").split(",")]) # starting in git-1.8.3, tags are listed as "tag: foo-1.0" instead of # just "foo-1.0". If we see a "tag: " prefix, prefer those. TAG = "tag: " tags = set([r[len(TAG):] for r in refs if r.startswith(TAG)]) if not tags: # Either we're using git < 1.8.3, or there really are no tags. We use # a heuristic: assume all version tags have a digit. The old git %%d # expansion behaves like git log --decorate=short and strips out the # refs/heads/ and refs/tags/ prefixes that would let us distinguish # between branches and tags. By ignoring refnames without digits, we # filter out many common branch names like "release" and # "stabilization", as well as "HEAD" and "master". tags = set([r for r in refs if re.search(r'\d', r)]) if verbose: print("discarding '%%s', no digits" %% ",".join(refs - tags)) if verbose: print("likely tags: %%s" %% ",".join(sorted(tags))) for ref in sorted(tags): # sorting will prefer e.g. "2.0" over "2.0rc1" if ref.startswith(tag_prefix): r = ref[len(tag_prefix):] if verbose: print("picking %%s" %% r) return {"version": r, "full-revisionid": keywords["full"].strip(), "dirty": False, "error": None, "date": date} # no suitable tags, so version is "0+unknown", but full hex is still there if verbose: print("no suitable tags, using unknown + full revision id") return {"version": "0+unknown", "full-revisionid": keywords["full"].strip(), "dirty": False, "error": "no suitable tags", "date": None} @register_vcs_handler("git", "pieces_from_vcs") def git_pieces_from_vcs(tag_prefix, root, verbose, run_command=run_command): """Get version from 'git describe' in the root of the source tree. This only gets called if the git-archive 'subst' keywords were *not* expanded, and _version.py hasn't already been rewritten with a short version string, meaning we're inside a checked out source tree. """ GITS = ["git"] if sys.platform == "win32": GITS = ["git.cmd", "git.exe"] out, rc = run_command(GITS, ["rev-parse", "--git-dir"], cwd=root, hide_stderr=True) if rc != 0: if verbose: print("Directory %%s not under git control" %% root) raise NotThisMethod("'git rev-parse --git-dir' returned error") # if there is a tag matching tag_prefix, this yields TAG-NUM-gHEX[-dirty] # if there isn't one, this yields HEX[-dirty] (no NUM) describe_out, rc = run_command(GITS, ["describe", "--tags", "--dirty", "--always", "--long", "--match", "%%s*" %% tag_prefix], cwd=root) # --long was added in git-1.5.5 if describe_out is None: raise NotThisMethod("'git describe' failed") describe_out = describe_out.strip() full_out, rc = run_command(GITS, ["rev-parse", "HEAD"], cwd=root) if full_out is None: raise NotThisMethod("'git rev-parse' failed") full_out = full_out.strip() pieces = {} pieces["long"] = full_out pieces["short"] = full_out[:7] # maybe improved later pieces["error"] = None # parse describe_out. It will be like TAG-NUM-gHEX[-dirty] or HEX[-dirty] # TAG might have hyphens. git_describe = describe_out # look for -dirty suffix dirty = git_describe.endswith("-dirty") pieces["dirty"] = dirty if dirty: git_describe = git_describe[:git_describe.rindex("-dirty")] # now we have TAG-NUM-gHEX or HEX if "-" in git_describe: # TAG-NUM-gHEX mo = re.search(r'^(.+)-(\d+)-g([0-9a-f]+)$', git_describe) if not mo: # unparseable. Maybe git-describe is misbehaving? pieces["error"] = ("unable to parse git-describe output: '%%s'" %% describe_out) return pieces # tag full_tag = mo.group(1) if not full_tag.startswith(tag_prefix): if verbose: fmt = "tag '%%s' doesn't start with prefix '%%s'" print(fmt %% (full_tag, tag_prefix)) pieces["error"] = ("tag '%%s' doesn't start with prefix '%%s'" %% (full_tag, tag_prefix)) return pieces pieces["closest-tag"] = full_tag[len(tag_prefix):] # distance: number of commits since tag pieces["distance"] = int(mo.group(2)) # commit: short hex revision ID pieces["short"] = mo.group(3) else: # HEX: no tags pieces["closest-tag"] = None count_out, rc = run_command(GITS, ["rev-list", "HEAD", "--count"], cwd=root) pieces["distance"] = int(count_out) # total number of commits # commit date: see ISO-8601 comment in git_versions_from_keywords() date = run_command(GITS, ["show", "-s", "--format=%%ci", "HEAD"], cwd=root)[0].strip() pieces["date"] = date.strip().replace(" ", "T", 1).replace(" ", "", 1) return pieces def plus_or_dot(pieces): """Return a + if we don't already have one, else return a .""" if "+" in pieces.get("closest-tag", ""): return "." return "+" def render_pep440(pieces): """Build up version string, with post-release "local version identifier". Our goal: TAG[+DISTANCE.gHEX[.dirty]] . Note that if you get a tagged build and then dirty it, you'll get TAG+0.gHEX.dirty Exceptions: 1: no tags. git_describe was just HEX. 0+untagged.DISTANCE.gHEX[.dirty] """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"] or pieces["dirty"]: rendered += plus_or_dot(pieces) rendered += "%%d.g%%s" %% (pieces["distance"], pieces["short"]) if pieces["dirty"]: rendered += ".dirty" else: # exception #1 rendered = "0+untagged.%%d.g%%s" %% (pieces["distance"], pieces["short"]) if pieces["dirty"]: rendered += ".dirty" return rendered def render_pep440_pre(pieces): """TAG[.post.devDISTANCE] -- No -dirty. Exceptions: 1: no tags. 0.post.devDISTANCE """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"]: rendered += ".post.dev%%d" %% pieces["distance"] else: # exception #1 rendered = "0.post.dev%%d" %% pieces["distance"] return rendered def render_pep440_post(pieces): """TAG[.postDISTANCE[.dev0]+gHEX] . The ".dev0" means dirty. Note that .dev0 sorts backwards (a dirty tree will appear "older" than the corresponding clean one), but you shouldn't be releasing software with -dirty anyways. Exceptions: 1: no tags. 0.postDISTANCE[.dev0] """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"] or pieces["dirty"]: rendered += ".post%%d" %% pieces["distance"] if pieces["dirty"]: rendered += ".dev0" rendered += plus_or_dot(pieces) rendered += "g%%s" %% pieces["short"] else: # exception #1 rendered = "0.post%%d" %% pieces["distance"] if pieces["dirty"]: rendered += ".dev0" rendered += "+g%%s" %% pieces["short"] return rendered def render_pep440_old(pieces): """TAG[.postDISTANCE[.dev0]] . The ".dev0" means dirty. Eexceptions: 1: no tags. 0.postDISTANCE[.dev0] """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"] or pieces["dirty"]: rendered += ".post%%d" %% pieces["distance"] if pieces["dirty"]: rendered += ".dev0" else: # exception #1 rendered = "0.post%%d" %% pieces["distance"] if pieces["dirty"]: rendered += ".dev0" return rendered def render_git_describe(pieces): """TAG[-DISTANCE-gHEX][-dirty]. Like 'git describe --tags --dirty --always'. Exceptions: 1: no tags. HEX[-dirty] (note: no 'g' prefix) """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"]: rendered += "-%%d-g%%s" %% (pieces["distance"], pieces["short"]) else: # exception #1 rendered = pieces["short"] if pieces["dirty"]: rendered += "-dirty" return rendered def render_git_describe_long(pieces): """TAG-DISTANCE-gHEX[-dirty]. Like 'git describe --tags --dirty --always -long'. The distance/hash is unconditional. Exceptions: 1: no tags. HEX[-dirty] (note: no 'g' prefix) """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] rendered += "-%%d-g%%s" %% (pieces["distance"], pieces["short"]) else: # exception #1 rendered = pieces["short"] if pieces["dirty"]: rendered += "-dirty" return rendered def render(pieces, style): """Render the given version pieces into the requested style.""" if pieces["error"]: return {"version": "unknown", "full-revisionid": pieces.get("long"), "dirty": None, "error": pieces["error"], "date": None} if not style or style == "default": style = "pep440" # the default if style == "pep440": rendered = render_pep440(pieces) elif style == "pep440-pre": rendered = render_pep440_pre(pieces) elif style == "pep440-post": rendered = render_pep440_post(pieces) elif style == "pep440-old": rendered = render_pep440_old(pieces) elif style == "git-describe": rendered = render_git_describe(pieces) elif style == "git-describe-long": rendered = render_git_describe_long(pieces) else: raise ValueError("unknown style '%%s'" %% style) return {"version": rendered, "full-revisionid": pieces["long"], "dirty": pieces["dirty"], "error": None, "date": pieces.get("date")} def get_versions(): """Get version information or return default if unable to do so.""" # I am in _version.py, which lives at ROOT/VERSIONFILE_SOURCE. If we have # __file__, we can work backwards from there to the root. Some # py2exe/bbfreeze/non-CPython implementations don't do __file__, in which # case we can only use expanded keywords. cfg = get_config() verbose = cfg.verbose try: return git_versions_from_keywords(get_keywords(), cfg.tag_prefix, verbose) except NotThisMethod: pass try: root = os.path.realpath(__file__) # versionfile_source is the relative path from the top of the source # tree (where the .git directory might live) to this file. Invert # this to find the root from __file__. for i in cfg.versionfile_source.split('/'): root = os.path.dirname(root) except NameError: return {"version": "0+unknown", "full-revisionid": None, "dirty": None, "error": "unable to find root of source tree", "date": None} try: pieces = git_pieces_from_vcs(cfg.tag_prefix, root, verbose) return render(pieces, cfg.style) except NotThisMethod: pass try: if cfg.parentdir_prefix: return versions_from_parentdir(cfg.parentdir_prefix, root, verbose) except NotThisMethod: pass return {"version": "0+unknown", "full-revisionid": None, "dirty": None, "error": "unable to compute version", "date": None} ''' @register_vcs_handler("git", "get_keywords") def git_get_keywords(versionfile_abs): """Extract version information from the given file.""" # the code embedded in _version.py can just fetch the value of these # keywords. When used from setup.py, we don't want to import _version.py, # so we do it with a regexp instead. This function is not used from # _version.py. keywords = {} try: f = open(versionfile_abs, "r") for line in f.readlines(): if line.strip().startswith("git_refnames ="): mo = re.search(r'=\s*"(.*)"', line) if mo: keywords["refnames"] = mo.group(1) if line.strip().startswith("git_full ="): mo = re.search(r'=\s*"(.*)"', line) if mo: keywords["full"] = mo.group(1) if line.strip().startswith("git_date ="): mo = re.search(r'=\s*"(.*)"', line) if mo: keywords["date"] = mo.group(1) f.close() except EnvironmentError: pass return keywords @register_vcs_handler("git", "keywords") def git_versions_from_keywords(keywords, tag_prefix, verbose): """Get version information from git keywords.""" if not keywords: raise NotThisMethod("no keywords at all, weird") date = keywords.get("date") if date is not None: # git-2.2.0 added "%cI", which expands to an ISO-8601 -compliant # datestamp. However we prefer "%ci" (which expands to an "ISO-8601 # -like" string, which we must then edit to make compliant), because # it's been around since git-1.5.3, and it's too difficult to # discover which version we're using, or to work around using an # older one. date = date.strip().replace(" ", "T", 1).replace(" ", "", 1) refnames = keywords["refnames"].strip() if refnames.startswith("$Format"): if verbose: print("keywords are unexpanded, not using") raise NotThisMethod("unexpanded keywords, not a git-archive tarball") refs = set([r.strip() for r in refnames.strip("()").split(",")]) # starting in git-1.8.3, tags are listed as "tag: foo-1.0" instead of # just "foo-1.0". If we see a "tag: " prefix, prefer those. TAG = "tag: " tags = set([r[len(TAG):] for r in refs if r.startswith(TAG)]) if not tags: # Either we're using git < 1.8.3, or there really are no tags. We use # a heuristic: assume all version tags have a digit. The old git %d # expansion behaves like git log --decorate=short and strips out the # refs/heads/ and refs/tags/ prefixes that would let us distinguish # between branches and tags. By ignoring refnames without digits, we # filter out many common branch names like "release" and # "stabilization", as well as "HEAD" and "master". tags = set([r for r in refs if re.search(r'\d', r)]) if verbose: print("discarding '%s', no digits" % ",".join(refs - tags)) if verbose: print("likely tags: %s" % ",".join(sorted(tags))) for ref in sorted(tags): # sorting will prefer e.g. "2.0" over "2.0rc1" if ref.startswith(tag_prefix): r = ref[len(tag_prefix):] if verbose: print("picking %s" % r) return {"version": r, "full-revisionid": keywords["full"].strip(), "dirty": False, "error": None, "date": date} # no suitable tags, so version is "0+unknown", but full hex is still there if verbose: print("no suitable tags, using unknown + full revision id") return {"version": "0+unknown", "full-revisionid": keywords["full"].strip(), "dirty": False, "error": "no suitable tags", "date": None} @register_vcs_handler("git", "pieces_from_vcs") def git_pieces_from_vcs(tag_prefix, root, verbose, run_command=run_command): """Get version from 'git describe' in the root of the source tree. This only gets called if the git-archive 'subst' keywords were *not* expanded, and _version.py hasn't already been rewritten with a short version string, meaning we're inside a checked out source tree. """ GITS = ["git"] if sys.platform == "win32": GITS = ["git.cmd", "git.exe"] out, rc = run_command(GITS, ["rev-parse", "--git-dir"], cwd=root, hide_stderr=True) if rc != 0: if verbose: print("Directory %s not under git control" % root) raise NotThisMethod("'git rev-parse --git-dir' returned error") # if there is a tag matching tag_prefix, this yields TAG-NUM-gHEX[-dirty] # if there isn't one, this yields HEX[-dirty] (no NUM) describe_out, rc = run_command(GITS, ["describe", "--tags", "--dirty", "--always", "--long", "--match", "%s*" % tag_prefix], cwd=root) # --long was added in git-1.5.5 if describe_out is None: raise NotThisMethod("'git describe' failed") describe_out = describe_out.strip() full_out, rc = run_command(GITS, ["rev-parse", "HEAD"], cwd=root) if full_out is None: raise NotThisMethod("'git rev-parse' failed") full_out = full_out.strip() pieces = {} pieces["long"] = full_out pieces["short"] = full_out[:7] # maybe improved later pieces["error"] = None # parse describe_out. It will be like TAG-NUM-gHEX[-dirty] or HEX[-dirty] # TAG might have hyphens. git_describe = describe_out # look for -dirty suffix dirty = git_describe.endswith("-dirty") pieces["dirty"] = dirty if dirty: git_describe = git_describe[:git_describe.rindex("-dirty")] # now we have TAG-NUM-gHEX or HEX if "-" in git_describe: # TAG-NUM-gHEX mo = re.search(r'^(.+)-(\d+)-g([0-9a-f]+)$', git_describe) if not mo: # unparseable. Maybe git-describe is misbehaving? pieces["error"] = ("unable to parse git-describe output: '%s'" % describe_out) return pieces # tag full_tag = mo.group(1) if not full_tag.startswith(tag_prefix): if verbose: fmt = "tag '%s' doesn't start with prefix '%s'" print(fmt % (full_tag, tag_prefix)) pieces["error"] = ("tag '%s' doesn't start with prefix '%s'" % (full_tag, tag_prefix)) return pieces pieces["closest-tag"] = full_tag[len(tag_prefix):] # distance: number of commits since tag pieces["distance"] = int(mo.group(2)) # commit: short hex revision ID pieces["short"] = mo.group(3) else: # HEX: no tags pieces["closest-tag"] = None count_out, rc = run_command(GITS, ["rev-list", "HEAD", "--count"], cwd=root) pieces["distance"] = int(count_out) # total number of commits # commit date: see ISO-8601 comment in git_versions_from_keywords() date = run_command(GITS, ["show", "-s", "--format=%ci", "HEAD"], cwd=root)[0].strip() pieces["date"] = date.strip().replace(" ", "T", 1).replace(" ", "", 1) return pieces def do_vcs_install(manifest_in, versionfile_source, ipy): """Git-specific installation logic for Versioneer. For Git, this means creating/changing .gitattributes to mark _version.py for export-subst keyword substitution. """ GITS = ["git"] if sys.platform == "win32": GITS = ["git.cmd", "git.exe"] files = [manifest_in, versionfile_source] if ipy: files.append(ipy) try: me = __file__ if me.endswith(".pyc") or me.endswith(".pyo"): me = os.path.splitext(me)[0] + ".py" versioneer_file = os.path.relpath(me) except NameError: versioneer_file = "versioneer.py" files.append(versioneer_file) present = False try: f = open(".gitattributes", "r") for line in f.readlines(): if line.strip().startswith(versionfile_source): if "export-subst" in line.strip().split()[1:]: present = True f.close() except EnvironmentError: pass if not present: f = open(".gitattributes", "a+") f.write("%s export-subst\n" % versionfile_source) f.close() files.append(".gitattributes") run_command(GITS, ["add", "--"] + files) def versions_from_parentdir(parentdir_prefix, root, verbose): """Try to determine the version from the parent directory name. Source tarballs conventionally unpack into a directory that includes both the project name and a version string. We will also support searching up two directory levels for an appropriately named parent directory """ rootdirs = [] for i in range(3): dirname = os.path.basename(root) if dirname.startswith(parentdir_prefix): return {"version": dirname[len(parentdir_prefix):], "full-revisionid": None, "dirty": False, "error": None, "date": None} else: rootdirs.append(root) root = os.path.dirname(root) # up a level if verbose: print("Tried directories %s but none started with prefix %s" % (str(rootdirs), parentdir_prefix)) raise NotThisMethod("rootdir doesn't start with parentdir_prefix") SHORT_VERSION_PY = """ # This file was generated by 'versioneer.py' (0.17) from # revision-control system data, or from the parent directory name of an # unpacked source archive. Distribution tarballs contain a pre-generated copy # of this file. import json version_json = ''' %s ''' # END VERSION_JSON def get_versions(): return json.loads(version_json) """ def versions_from_file(filename): """Try to determine the version from _version.py if present.""" try: with open(filename) as f: contents = f.read() except EnvironmentError: raise NotThisMethod("unable to read _version.py") mo = re.search(r"version_json = '''\n(.*)''' # END VERSION_JSON", contents, re.M | re.S) if not mo: mo = re.search(r"version_json = '''\r\n(.*)''' # END VERSION_JSON", contents, re.M | re.S) if not mo: raise NotThisMethod("no version_json in _version.py") return json.loads(mo.group(1)) def write_to_version_file(filename, versions): """Write the given version number to the given _version.py file.""" os.unlink(filename) contents = json.dumps(versions, sort_keys=True, indent=1, separators=(",", ": ")) with open(filename, "w") as f: f.write(SHORT_VERSION_PY % contents) print("set %s to '%s'" % (filename, versions["version"])) def plus_or_dot(pieces): """Return a + if we don't already have one, else return a .""" if "+" in pieces.get("closest-tag", ""): return "." return "+" def render_pep440(pieces): """Build up version string, with post-release "local version identifier". Our goal: TAG[+DISTANCE.gHEX[.dirty]] . Note that if you get a tagged build and then dirty it, you'll get TAG+0.gHEX.dirty Exceptions: 1: no tags. git_describe was just HEX. 0+untagged.DISTANCE.gHEX[.dirty] """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"] or pieces["dirty"]: rendered += plus_or_dot(pieces) rendered += "%d.g%s" % (pieces["distance"], pieces["short"]) if pieces["dirty"]: rendered += ".dirty" else: # exception #1 rendered = "0+untagged.%d.g%s" % (pieces["distance"], pieces["short"]) if pieces["dirty"]: rendered += ".dirty" return rendered def render_pep440_pre(pieces): """TAG[.post.devDISTANCE] -- No -dirty. Exceptions: 1: no tags. 0.post.devDISTANCE """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"]: rendered += ".post.dev%d" % pieces["distance"] else: # exception #1 rendered = "0.post.dev%d" % pieces["distance"] return rendered def render_pep440_post(pieces): """TAG[.postDISTANCE[.dev0]+gHEX] . The ".dev0" means dirty. Note that .dev0 sorts backwards (a dirty tree will appear "older" than the corresponding clean one), but you shouldn't be releasing software with -dirty anyways. Exceptions: 1: no tags. 0.postDISTANCE[.dev0] """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"] or pieces["dirty"]: rendered += ".post%d" % pieces["distance"] if pieces["dirty"]: rendered += ".dev0" rendered += plus_or_dot(pieces) rendered += "g%s" % pieces["short"] else: # exception #1 rendered = "0.post%d" % pieces["distance"] if pieces["dirty"]: rendered += ".dev0" rendered += "+g%s" % pieces["short"] return rendered def render_pep440_old(pieces): """TAG[.postDISTANCE[.dev0]] . The ".dev0" means dirty. Eexceptions: 1: no tags. 0.postDISTANCE[.dev0] """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"] or pieces["dirty"]: rendered += ".post%d" % pieces["distance"] if pieces["dirty"]: rendered += ".dev0" else: # exception #1 rendered = "0.post%d" % pieces["distance"] if pieces["dirty"]: rendered += ".dev0" return rendered def render_git_describe(pieces): """TAG[-DISTANCE-gHEX][-dirty]. Like 'git describe --tags --dirty --always'. Exceptions: 1: no tags. HEX[-dirty] (note: no 'g' prefix) """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"]: rendered += "-%d-g%s" % (pieces["distance"], pieces["short"]) else: # exception #1 rendered = pieces["short"] if pieces["dirty"]: rendered += "-dirty" return rendered def render_git_describe_long(pieces): """TAG-DISTANCE-gHEX[-dirty]. Like 'git describe --tags --dirty --always -long'. The distance/hash is unconditional. Exceptions: 1: no tags. HEX[-dirty] (note: no 'g' prefix) """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] rendered += "-%d-g%s" % (pieces["distance"], pieces["short"]) else: # exception #1 rendered = pieces["short"] if pieces["dirty"]: rendered += "-dirty" return rendered def render(pieces, style): """Render the given version pieces into the requested style.""" if pieces["error"]: return {"version": "unknown", "full-revisionid": pieces.get("long"), "dirty": None, "error": pieces["error"], "date": None} if not style or style == "default": style = "pep440" # the default if style == "pep440": rendered = render_pep440(pieces) elif style == "pep440-pre": rendered = render_pep440_pre(pieces) elif style == "pep440-post": rendered = render_pep440_post(pieces) elif style == "pep440-old": rendered = render_pep440_old(pieces) elif style == "git-describe": rendered = render_git_describe(pieces) elif style == "git-describe-long": rendered = render_git_describe_long(pieces) else: raise ValueError("unknown style '%s'" % style) return {"version": rendered, "full-revisionid": pieces["long"], "dirty": pieces["dirty"], "error": None, "date": pieces.get("date")} class VersioneerBadRootError(Exception): """The project root directory is unknown or missing key files.""" def get_versions(verbose=False): """Get the project version from whatever source is available. Returns dict with two keys: 'version' and 'full'. """ if "versioneer" in sys.modules: # see the discussion in cmdclass.py:get_cmdclass() del sys.modules["versioneer"] root = get_root() cfg = get_config_from_root(root) assert cfg.VCS is not None, "please set [versioneer]VCS= in setup.cfg" handlers = HANDLERS.get(cfg.VCS) assert handlers, "unrecognized VCS '%s'" % cfg.VCS verbose = verbose or cfg.verbose assert cfg.versionfile_source is not None, \ "please set versioneer.versionfile_source" assert cfg.tag_prefix is not None, "please set versioneer.tag_prefix" versionfile_abs = os.path.join(root, cfg.versionfile_source) # extract version from first of: _version.py, VCS command (e.g. 'git # describe'), parentdir. This is meant to work for developers using a # source checkout, for users of a tarball created by 'setup.py sdist', # and for users of a tarball/zipball created by 'git archive' or github's # download-from-tag feature or the equivalent in other VCSes. get_keywords_f = handlers.get("get_keywords") from_keywords_f = handlers.get("keywords") if get_keywords_f and from_keywords_f: try: keywords = get_keywords_f(versionfile_abs) ver = from_keywords_f(keywords, cfg.tag_prefix, verbose) if verbose: print("got version from expanded keyword %s" % ver) return ver except NotThisMethod: pass try: ver = versions_from_file(versionfile_abs) if verbose: print("got version from file %s %s" % (versionfile_abs, ver)) return ver except NotThisMethod: pass from_vcs_f = handlers.get("pieces_from_vcs") if from_vcs_f: try: pieces = from_vcs_f(cfg.tag_prefix, root, verbose) ver = render(pieces, cfg.style) if verbose: print("got version from VCS %s" % ver) return ver except NotThisMethod: pass try: if cfg.parentdir_prefix: ver = versions_from_parentdir(cfg.parentdir_prefix, root, verbose) if verbose: print("got version from parentdir %s" % ver) return ver except NotThisMethod: pass if verbose: print("unable to compute version") return {"version": "0+unknown", "full-revisionid": None, "dirty": None, "error": "unable to compute version", "date": None} def get_version(): """Get the short version string for this project.""" return get_versions()["version"] def get_cmdclass(): """Get the custom setuptools/distutils subclasses used by Versioneer.""" if "versioneer" in sys.modules: del sys.modules["versioneer"] # this fixes the "python setup.py develop" case (also 'install' and # 'easy_install .'), in which subdependencies of the main project are # built (using setup.py bdist_egg) in the same python process. Assume # a main project A and a dependency B, which use different versions # of Versioneer. A's setup.py imports A's Versioneer, leaving it in # sys.modules by the time B's setup.py is executed, causing B to run # with the wrong versioneer. Setuptools wraps the sub-dep builds in a # sandbox that restores sys.modules to it's pre-build state, so the # parent is protected against the child's "import versioneer". By # removing ourselves from sys.modules here, before the child build # happens, we protect the child from the parent's versioneer too. # Also see https://github.com/warner/python-versioneer/issues/52 cmds = {} # we add "version" to both distutils and setuptools from distutils.core import Command class cmd_version(Command): description = "report generated version string" user_options = [] boolean_options = [] def initialize_options(self): pass def finalize_options(self): pass def run(self): vers = get_versions(verbose=True) print("Version: %s" % vers["version"]) print(" full-revisionid: %s" % vers.get("full-revisionid")) print(" dirty: %s" % vers.get("dirty")) print(" date: %s" % vers.get("date")) if vers["error"]: print(" error: %s" % vers["error"]) cmds["version"] = cmd_version # we override "build_py" in both distutils and setuptools # # most invocation pathways end up running build_py: # distutils/build -> build_py # distutils/install -> distutils/build ->.. # setuptools/bdist_wheel -> distutils/install ->.. # setuptools/bdist_egg -> distutils/install_lib -> build_py # setuptools/install -> bdist_egg ->.. # setuptools/develop -> ? # pip install: # copies source tree to a tempdir before running egg_info/etc # if .git isn't copied too, 'git describe' will fail # then does setup.py bdist_wheel, or sometimes setup.py install # setup.py egg_info -> ? # we override different "build_py" commands for both environments if "setuptools" in sys.modules: from setuptools.command.build_py import build_py as _build_py else: from distutils.command.build_py import build_py as _build_py class cmd_build_py(_build_py): def run(self): root = get_root() cfg = get_config_from_root(root) versions = get_versions() _build_py.run(self) # now locate _version.py in the new build/ directory and replace # it with an updated value if cfg.versionfile_build: target_versionfile = os.path.join(self.build_lib, cfg.versionfile_build) print("UPDATING %s" % target_versionfile) write_to_version_file(target_versionfile, versions) cmds["build_py"] = cmd_build_py if "cx_Freeze" in sys.modules: # cx_freeze enabled? from cx_Freeze.dist import build_exe as _build_exe # nczeczulin reports that py2exe won't like the pep440-style string # as FILEVERSION, but it can be used for PRODUCTVERSION, e.g. # setup(console=[{ # "version": versioneer.get_version().split("+", 1)[0], # FILEVERSION # "product_version": versioneer.get_version(), # ... class cmd_build_exe(_build_exe): def run(self): root = get_root() cfg = get_config_from_root(root) versions = get_versions() target_versionfile = cfg.versionfile_source print("UPDATING %s" % target_versionfile) write_to_version_file(target_versionfile, versions) _build_exe.run(self) os.unlink(target_versionfile) with open(cfg.versionfile_source, "w") as f: LONG = LONG_VERSION_PY[cfg.VCS] f.write(LONG % {"DOLLAR": "$", "STYLE": cfg.style, "TAG_PREFIX": cfg.tag_prefix, "PARENTDIR_PREFIX": cfg.parentdir_prefix, "VERSIONFILE_SOURCE": cfg.versionfile_source, }) cmds["build_exe"] = cmd_build_exe del cmds["build_py"] if 'py2exe' in sys.modules: # py2exe enabled? try: from py2exe.distutils_buildexe import py2exe as _py2exe # py3 except ImportError: from py2exe.build_exe import py2exe as _py2exe # py2 class cmd_py2exe(_py2exe): def run(self): root = get_root() cfg = get_config_from_root(root) versions = get_versions() target_versionfile = cfg.versionfile_source print("UPDATING %s" % target_versionfile) write_to_version_file(target_versionfile, versions) _py2exe.run(self) os.unlink(target_versionfile) with open(cfg.versionfile_source, "w") as f: LONG = LONG_VERSION_PY[cfg.VCS] f.write(LONG % {"DOLLAR": "$", "STYLE": cfg.style, "TAG_PREFIX": cfg.tag_prefix, "PARENTDIR_PREFIX": cfg.parentdir_prefix, "VERSIONFILE_SOURCE": cfg.versionfile_source, }) cmds["py2exe"] = cmd_py2exe # we override different "sdist" commands for both environments if "setuptools" in sys.modules: from setuptools.command.sdist import sdist as _sdist else: from distutils.command.sdist import sdist as _sdist class cmd_sdist(_sdist): def run(self): versions = get_versions() self._versioneer_generated_versions = versions # unless we update this, the command will keep using the old # version self.distribution.metadata.version = versions["version"] return _sdist.run(self) def make_release_tree(self, base_dir, files): root = get_root() cfg = get_config_from_root(root) _sdist.make_release_tree(self, base_dir, files) # now locate _version.py in the new base_dir directory # (remembering that it may be a hardlink) and replace it with an # updated value target_versionfile = os.path.join(base_dir, cfg.versionfile_source) print("UPDATING %s" % target_versionfile) write_to_version_file(target_versionfile, self._versioneer_generated_versions) cmds["sdist"] = cmd_sdist return cmds CONFIG_ERROR = """ setup.cfg is missing the necessary Versioneer configuration. You need a section like: [versioneer] VCS = git style = pep440 versionfile_source = src/myproject/_version.py versionfile_build = myproject/_version.py tag_prefix = parentdir_prefix = myproject- You will also need to edit your setup.py to use the results: import versioneer setup(version=versioneer.get_version(), cmdclass=versioneer.get_cmdclass(), ...) Please read the docstring in ./versioneer.py for configuration instructions, edit setup.cfg, and re-run the installer or 'python versioneer.py setup'. """ SAMPLE_CONFIG = """ # See the docstring in versioneer.py for instructions. Note that you must # re-run 'versioneer.py setup' after changing this section, and commit the # resulting files. [versioneer] #VCS = git #style = pep440 #versionfile_source = #versionfile_build = #tag_prefix = #parentdir_prefix = """ INIT_PY_SNIPPET = """ from ._version import get_versions __version__ = get_versions()['version'] del get_versions """ def do_setup(): """Main VCS-independent setup function for installing Versioneer.""" root = get_root() try: cfg = get_config_from_root(root) except (EnvironmentError, configparser.NoSectionError, configparser.NoOptionError) as e: if isinstance(e, (EnvironmentError, configparser.NoSectionError)): print("Adding sample versioneer config to setup.cfg", file=sys.stderr) with open(os.path.join(root, "setup.cfg"), "a") as f: f.write(SAMPLE_CONFIG) print(CONFIG_ERROR, file=sys.stderr) return 1 print(" creating %s" % cfg.versionfile_source) with open(cfg.versionfile_source, "w") as f: LONG = LONG_VERSION_PY[cfg.VCS] f.write(LONG % {"DOLLAR": "$", "STYLE": cfg.style, "TAG_PREFIX": cfg.tag_prefix, "PARENTDIR_PREFIX": cfg.parentdir_prefix, "VERSIONFILE_SOURCE": cfg.versionfile_source, }) ipy = os.path.join(os.path.dirname(cfg.versionfile_source), "__init__.py") if os.path.exists(ipy): try: with open(ipy, "r") as f: old = f.read() except EnvironmentError: old = "" if INIT_PY_SNIPPET not in old: print(" appending to %s" % ipy) with open(ipy, "a") as f: f.write(INIT_PY_SNIPPET) else: print(" %s unmodified" % ipy) else: print(" %s doesn't exist, ok" % ipy) ipy = None # Make sure both the top-level "versioneer.py" and versionfile_source # (PKG/_version.py, used by runtime code) are in MANIFEST.in, so # they'll be copied into source distributions. Pip won't be able to # install the package without this. manifest_in = os.path.join(root, "MANIFEST.in") simple_includes = set() try: with open(manifest_in, "r") as f: for line in f: if line.startswith("include "): for include in line.split()[1:]: simple_includes.add(include) except EnvironmentError: pass # That doesn't cover everything MANIFEST.in can do # (http://docs.python.org/2/distutils/sourcedist.html#commands), so # it might give some false negatives. Appending redundant 'include' # lines is safe, though. if "versioneer.py" not in simple_includes: print(" appending 'versioneer.py' to MANIFEST.in") with open(manifest_in, "a") as f: f.write("include versioneer.py\n") else: print(" 'versioneer.py' already in MANIFEST.in") if cfg.versionfile_source not in simple_includes: print(" appending versionfile_source ('%s') to MANIFEST.in" % cfg.versionfile_source) with open(manifest_in, "a") as f: f.write("include %s\n" % cfg.versionfile_source) else: print(" versionfile_source already in MANIFEST.in") # Make VCS-specific changes. For git, this means creating/changing # .gitattributes to mark _version.py for export-subst keyword # substitution. do_vcs_install(manifest_in, cfg.versionfile_source, ipy) return 0 def scan_setup_py(): """Validate the contents of setup.py against Versioneer's expectations.""" found = set() setters = False errors = 0 with open("setup.py", "r") as f: for line in f.readlines(): if "import versioneer" in line: found.add("import") if "versioneer.get_cmdclass()" in line: found.add("cmdclass") if "versioneer.get_version()" in line: found.add("get_version") if "versioneer.VCS" in line: setters = True if "versioneer.versionfile_source" in line: setters = True if len(found) != 3: print("") print("Your setup.py appears to be missing some important items") print("(but I might be wrong). Please make sure it has something") print("roughly like the following:") print("") print(" import versioneer") print(" setup( version=versioneer.get_version(),") print(" cmdclass=versioneer.get_cmdclass(), ...)") print("") errors += 1 if setters: print("You should remove lines like 'versioneer.VCS = ' and") print("'versioneer.versionfile_source = ' . This configuration") print("now lives in setup.cfg, and should be removed from setup.py") print("") errors += 1 return errors if __name__ == "__main__": cmd = sys.argv[1] if cmd == "setup": errors = do_setup() errors += scan_setup_py() if errors: sys.exit(1)