pax_global_header00006660000000000000000000000064124171254620014516gustar00rootroot0000000000000052 comment=6c3dabf4632e1b385e21ce13c4ea93bb134e80c7 node-fd-slicer-0.3.2/000077500000000000000000000000001241712546200143135ustar00rootroot00000000000000node-fd-slicer-0.3.2/.gitignore000066400000000000000000000000161241712546200163000ustar00rootroot00000000000000/node_modules node-fd-slicer-0.3.2/CHANGELOG.md000066400000000000000000000011311241712546200161200ustar00rootroot00000000000000### 0.3.2 * fix write stream and read stream destroy behavior ### 0.3.1 * write stream: fix end option behavior ### 0.3.0 * write stream emits 'progress' events * write stream supports 'end' option which causes the stream to emit an error if a maximum size is exceeded * improve documentation ### 0.2.1 * Update pend dependency to latest bugfix version. ### 0.2.0 * Add read and write functions ### 0.1.0 * Add `autoClose` option and `ref()` and `unref()`. ### 0.0.2 * Add API documentation * read stream: create buffer at last possible moment ### 0.0.1 * Initial release node-fd-slicer-0.3.2/LICENSE000066400000000000000000000020411241712546200153150ustar00rootroot00000000000000Copyright (c) 2014 Andrew Kelley Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. node-fd-slicer-0.3.2/README.md000066400000000000000000000122151241712546200155730ustar00rootroot00000000000000# fd-slicer Safe `fs.ReadStream` and `fs.WriteStream` using the same fd. Let's say that you want to perform a parallel upload of a file to a remote server. To do this, we want to create multiple read streams. The first thing you might think of is to use the `{start: 0, end: 0}` API of `fs.createReadStream`. This gives you two choices: 0. Use the same file descriptor for all `fs.ReadStream` objects. 0. Open the file multiple times, resulting in a separate file descriptor for each read stream. Neither of these are acceptable options. The first one is a severe bug, because the API docs for `fs.write` state: > Note that it is unsafe to use `fs.write` multiple times on the same file > without waiting for the callback. For this scenario, `fs.createWriteStream` > is strongly recommended. `fs.createWriteStream` will solve the problem if you only create one of them for the file descriptor, but it will exhibit this unsafety if you create multiple write streams per file descriptor. The second option suffers from a race condition. For each additional time the file is opened after the first, it is possible that the file is modified. So in our parallel uploading example, we might upload a corrupt file that never existed on the client's computer. This module solves this problem by providing `createReadStream` and `createWriteStream` that operate on a shared file descriptor and provides the convenient stream API while still allowing slicing and dicing. This module also gives you some additional power that the builtin `fs.createWriteStream` do not give you. These features are: * Emitting a 'progress' event on write. * Ability to set a maximum size and emit an error if this size is exceeded. ## Usage ```js var FdSlicer = require('fd-slicer'); var fs = require('fs'); fs.open("file.txt", 'r', function(err, fd) { if (err) throw err; var fdSlicer = new FdSlicer(fd); var firstPart = fdSlicer.createReadStream({start: 0, end: 100}); var secondPart = fdSlicer.createReadStream({start: 100}); var firstOut = fs.createWriteStream("first.txt"); var secondOut = fs.createWriteStream("second.txt"); firstPart.pipe(firstOut); secondPart.pipe(secondOut); }); ``` ## API Documentation ### FdSlicer(fd, [options]) ```js var FdSlicer = require('fd-slicer'); fs.open("file.txt", 'r', function(err, fd) { if (err) throw err; var fdSlicer = new FdSlicer(fd); // ... }); ``` Make sure `fd` is a properly initialized file descriptor. If you want to use `createReadStream` make sure you open it for reading and if you want to use `createWriteStream` make sure you open it for writing. `options` is an optional object which can contain: * `autoClose` - if set to `true`, the file descriptor will be automatically closed once the last stream that references it is closed. Defaults to `false`. `ref()` and `unref()` can be used to increase or decrease the reference count, respectively. #### Properties ##### fd The file descriptor passed in. #### Methods ##### createReadStream(options) Available `options`: * `start` - Number. The offset into the file to start reading from. Defaults to 0. * `end` - Number. Exclusive upper bound offset into the file to stop reading from. * `highWaterMark` - Number. The maximum number of bytes to store in the internal buffer before ceasing to read from the underlying resource. Defaults to 16 KB. * `encoding` - String. If specified, then buffers will be decoded to strings using the specified encoding. Defaults to `null`. The ReadableStream that this returns has these additional methods: * `destroy(err)` - stop streaming. `err` is optional and is the error that will be emitted in order to cause the streaming to stop. Defaults to `new Error("stream destroyed")`. ##### createWriteStream(options) Available `options`: * `start` - Number. The offset into the file to start writing to. Defaults to 0. * `end` - Number. Exclusive upper bound offset into the file. If this offset is reached, the write stream will emit an 'error' event and stop functioning. In this situation, `err.code === 'ETOOBIG'`. Defaults to `Infinity`. * `highWaterMark` - Number. Buffer level when `write()` starts returning false. Defaults to 16KB. * `decodeStrings` - Boolean. Whether or not to decode strings into Buffers before passing them to` _write()`. Defaults to `true`. The WritableStream that this returns has these additional methods: * `destroy()` - stop streaming And these additional properties: * `bytesWritten` - number of bytes written to the stream And these additional events: * 'progress' - emitted when `bytesWritten` changes. ##### read(buffer, offset, length, position, callback) Equivalent to `fs.read`, but with concurrency protection. `callback` must be defined. ##### write(buffer, offset, length, position, callback) Equivalent to `fs.write`, but with concurrency protection. `callback` must be defined. ##### ref() Increase the `autoClose` reference count by 1. ##### unref() Decrease the `autoClose` reference count by 1. #### Events ##### 'error' Emitted if `fs.close` returns an error when auto closing. ##### 'close' Emitted when fd-slicer closes the file descriptor due to `autoClose`. node-fd-slicer-0.3.2/index.js000066400000000000000000000103321241712546200157570ustar00rootroot00000000000000var fs = require('fs'); var util = require('util'); var stream = require('stream'); var Readable = stream.Readable; var Writable = stream.Writable; var Pend = require('pend'); var EventEmitter = require('events').EventEmitter; module.exports = FdSlicer; util.inherits(FdSlicer, EventEmitter); function FdSlicer(fd, options) { options = options || {}; EventEmitter.call(this); this.fd = fd; this.pend = new Pend(); this.pend.max = 1; this.refCount = 0; this.autoClose = !!options.autoClose; } FdSlicer.prototype.read = function(buffer, offset, length, position, callback) { var self = this; self.pend.go(function(cb) { fs.read(self.fd, buffer, offset, length, position, function(err, bytesRead, buffer) { cb(); callback(err, bytesRead, buffer); }); }); }; FdSlicer.prototype.write = function(buffer, offset, length, position, callback) { var self = this; self.pend.go(function(cb) { fs.write(self.fd, buffer, offset, length, position, function(err, written, buffer) { cb(); callback(err, written, buffer); }); }); }; FdSlicer.prototype.createReadStream = function(options) { return new ReadStream(this, options); }; FdSlicer.prototype.createWriteStream = function(options) { return new WriteStream(this, options); }; FdSlicer.prototype.ref = function() { this.refCount += 1; }; FdSlicer.prototype.unref = function() { var self = this; self.refCount -= 1; if (self.refCount > 0) return; if (self.refCount < 0) throw new Error("invalid unref"); if (self.autoClose) { fs.close(self.fd, onCloseDone); } function onCloseDone(err) { if (err) { self.emit('error', err); } else { self.emit('close'); } } }; util.inherits(ReadStream, Readable); function ReadStream(context, options) { options = options || {}; Readable.call(this, options); this.context = context; this.context.ref(); this.start = options.start || 0; this.end = options.end; this.pos = this.start; this.destroyed = false; } ReadStream.prototype._read = function(n) { var self = this; if (self.destroyed) return; var toRead = Math.min(self._readableState.highWaterMark, n); if (self.end != null) { toRead = Math.min(toRead, self.end - self.pos); } if (toRead <= 0) { self.destroyed = true; self.push(null); self.context.unref(); return; } self.context.pend.go(function(cb) { if (self.destroyed) return cb(); var buffer = new Buffer(toRead); fs.read(self.context.fd, buffer, 0, toRead, self.pos, function(err, bytesRead) { if (err) { self.destroy(err); } else if (bytesRead === 0) { self.destroyed = true; self.push(null); self.context.unref(); } else { self.pos += bytesRead; self.push(buffer.slice(0, bytesRead)); } cb(); }); }); }; ReadStream.prototype.destroy = function(err) { if (this.destroyed) return; err = err || new Error("stream destroyed"); this.destroyed = true; this.emit('error', err); this.context.unref(); }; util.inherits(WriteStream, Writable); function WriteStream(context, options) { options = options || {}; Writable.call(this, options); this.context = context; this.context.ref(); this.start = options.start || 0; this.endOffset = options.end || Infinity; this.bytesWritten = 0; this.pos = this.start; this.destroyed = false; this.on('finish', this.destroy.bind(this)); } WriteStream.prototype._write = function(buffer, encoding, callback) { var self = this; if (self.destroyed) return; if (self.pos + buffer.length > self.endOffset) { var err = new Error("maximum file length exceeded"); err.code = 'ETOOBIG'; self.destroy(); callback(err); return; } self.context.pend.go(function(cb) { if (self.destroyed) return cb(); fs.write(self.context.fd, buffer, 0, buffer.length, self.pos, function(err, bytes) { if (err) { self.destroy(); cb(); callback(err); } else { self.bytesWritten += bytes; self.pos += bytes; self.emit('progress'); cb(); callback(); } }); }); }; WriteStream.prototype.destroy = function() { if (this.destroyed) return; this.destroyed = true; this.context.unref(); }; node-fd-slicer-0.3.2/package.json000066400000000000000000000012171241712546200166020ustar00rootroot00000000000000{ "name": "fd-slicer", "version": "0.3.2", "description": "safely create multiple ReadStream or WriteStream objects from the same file descriptor", "main": "index.js", "scripts": { "test": "mocha --reporter spec" }, "author": "Andrew Kelley ", "license": "MIT", "devDependencies": { "mocha": "~1.21.5", "stream-equal": "~0.1.5" }, "dependencies": { "pend": "~1.1.3" }, "directories": { "test": "test" }, "repository": { "type": "git", "url": "git://github.com/andrewrk/node-fd-slicer.git" }, "bugs": { "url": "https://github.com/andrewrk/node-fd-slicer/issues" } } node-fd-slicer-0.3.2/test/000077500000000000000000000000001241712546200152725ustar00rootroot00000000000000node-fd-slicer-0.3.2/test/test.js000066400000000000000000000213731241712546200166150ustar00rootroot00000000000000var FdSlicer = require('../'); var fs = require('fs'); var crypto = require('crypto'); var path = require('path'); var streamEqual = require('stream-equal'); var assert = require('assert'); var Pend = require('pend'); var describe = global.describe; var it = global.it; var before = global.before; var beforeEach = global.beforeEach; var after = global.after; var testBlobFile = path.join(__dirname, "test-blob.bin"); var testBlobFileSize = 20 * 1024 * 1024; var testOutBlobFile = path.join(__dirname, "test-blob-out.bin"); describe("FdSlicer", function() { before(function(done) { var out = fs.createWriteStream(testBlobFile); for (var i = 0; i < testBlobFileSize / 1024; i += 1) { out.write(crypto.pseudoRandomBytes(1024)); } out.end(); out.on('close', done); }); beforeEach(function() { try { fs.unlinkSync(testOutBlobFile); } catch (err) { } }); after(function() { try { fs.unlinkSync(testBlobFile); fs.unlinkSync(testOutBlobFile); } catch (err) { } }); it("reads a 20MB file (autoClose on)", function(done) { fs.open(testBlobFile, 'r', function(err, fd) { if (err) return done(err); var fdSlicer = new FdSlicer(fd, {autoClose: true}); var actualStream = fdSlicer.createReadStream(); var expectedStream = fs.createReadStream(testBlobFile); var pend = new Pend(); pend.go(function(cb) { fdSlicer.on('close', cb); }); pend.go(function(cb) { streamEqual(expectedStream, actualStream, function(err, equal) { if (err) return done(err); assert.ok(equal); cb(); }); }); pend.wait(done); }); }); it("reads 4 chunks simultaneously", function(done) { fs.open(testBlobFile, 'r', function(err, fd) { if (err) return done(err); var fdSlicer = new FdSlicer(fd); var actualPart1 = fdSlicer.createReadStream({start: testBlobFileSize * 0/4, end: testBlobFileSize * 1/4}); var actualPart2 = fdSlicer.createReadStream({start: testBlobFileSize * 1/4, end: testBlobFileSize * 2/4}); var actualPart3 = fdSlicer.createReadStream({start: testBlobFileSize * 2/4, end: testBlobFileSize * 3/4}); var actualPart4 = fdSlicer.createReadStream({start: testBlobFileSize * 3/4, end: testBlobFileSize * 4/4}); var expectedPart1 = fdSlicer.createReadStream({start: testBlobFileSize * 0/4, end: testBlobFileSize * 1/4}); var expectedPart2 = fdSlicer.createReadStream({start: testBlobFileSize * 1/4, end: testBlobFileSize * 2/4}); var expectedPart3 = fdSlicer.createReadStream({start: testBlobFileSize * 2/4, end: testBlobFileSize * 3/4}); var expectedPart4 = fdSlicer.createReadStream({start: testBlobFileSize * 3/4, end: testBlobFileSize * 4/4}); var pend = new Pend(); pend.go(function(cb) { streamEqual(expectedPart1, actualPart1, function(err, equal) { assert.ok(equal); cb(err); }); }); pend.go(function(cb) { streamEqual(expectedPart2, actualPart2, function(err, equal) { assert.ok(equal); cb(err); }); }); pend.go(function(cb) { streamEqual(expectedPart3, actualPart3, function(err, equal) { assert.ok(equal); cb(err); }); }); pend.go(function(cb) { streamEqual(expectedPart4, actualPart4, function(err, equal) { assert.ok(equal); cb(err); }); }); pend.wait(function(err) { if (err) return done(err); fs.close(fd, done); }); }); }); it("writes a 20MB file (autoClose on)", function(done) { fs.open(testOutBlobFile, 'w', function(err, fd) { if (err) return done(err); var fdSlicer = new FdSlicer(fd, {autoClose: true}); var actualStream = fdSlicer.createWriteStream(); var inStream = fs.createReadStream(testBlobFile); fdSlicer.on('close', function() { var expected = fs.createReadStream(testBlobFile); var actual = fs.createReadStream(testOutBlobFile); streamEqual(expected, actual, function(err, equal) { if (err) return done(err); assert.ok(equal); done(); }); }); inStream.pipe(actualStream); }); }); it("writes 4 chunks simultaneously", function(done) { fs.open(testOutBlobFile, 'w', function(err, fd) { if (err) return done(err); var fdSlicer = new FdSlicer(fd); var actualPart1 = fdSlicer.createWriteStream({start: testBlobFileSize * 0/4}); var actualPart2 = fdSlicer.createWriteStream({start: testBlobFileSize * 1/4}); var actualPart3 = fdSlicer.createWriteStream({start: testBlobFileSize * 2/4}); var actualPart4 = fdSlicer.createWriteStream({start: testBlobFileSize * 3/4}); var in1 = fs.createReadStream(testBlobFile, {start: testBlobFileSize * 0/4, end: testBlobFileSize * 1/4}); var in2 = fs.createReadStream(testBlobFile, {start: testBlobFileSize * 1/4, end: testBlobFileSize * 2/4}); var in3 = fs.createReadStream(testBlobFile, {start: testBlobFileSize * 2/4, end: testBlobFileSize * 3/4}); var in4 = fs.createReadStream(testBlobFile, {start: testBlobFileSize * 3/4, end: testBlobFileSize * 4/4}); var pend = new Pend(); pend.go(function(cb) { actualPart1.on('finish', cb); }); pend.go(function(cb) { actualPart2.on('finish', cb); }); pend.go(function(cb) { actualPart3.on('finish', cb); }); pend.go(function(cb) { actualPart4.on('finish', cb); }); in1.pipe(actualPart1); in2.pipe(actualPart2); in3.pipe(actualPart3); in4.pipe(actualPart4); pend.wait(function() { fs.close(fd, function(err) { if (err) return done(err); var expected = fs.createReadStream(testBlobFile); var actual = fs.createReadStream(testOutBlobFile); streamEqual(expected, actual, function(err, equal) { if (err) return done(err); assert.ok(equal); done(); }); }); }); }); }); it("write stream emits error when max size exceeded", function(done) { fs.open(testOutBlobFile, 'w', function(err, fd) { if (err) return done(err); var fdSlicer = new FdSlicer(fd, {autoClose: true}); var ws = fdSlicer.createWriteStream({start: 0, end: 1000}); ws.on('error', function(err) { assert.strictEqual(err.code, 'ETOOBIG'); fdSlicer.on('close', done); }); ws.end(new Buffer(1001)); }); }); it("write stream does not emit error when max size not exceeded", function(done) { fs.open(testOutBlobFile, 'w', function(err, fd) { if (err) return done(err); var fdSlicer = new FdSlicer(fd, {autoClose: true}); var ws = fdSlicer.createWriteStream({end: 1000}); fdSlicer.on('close', done); ws.end(new Buffer(1000)); }); }); it("write stream start and end work together", function(done) { fs.open(testOutBlobFile, 'w', function(err, fd) { if (err) return done(err); var fdSlicer = new FdSlicer(fd, {autoClose: true}); var ws = fdSlicer.createWriteStream({start: 1, end: 1000}); ws.on('error', function(err) { assert.strictEqual(err.code, 'ETOOBIG'); fdSlicer.on('close', done); }); ws.end(new Buffer(1000)); }); }); it("write stream emits progress events", function(done) { fs.open(testOutBlobFile, 'w', function(err, fd) { if (err) return done(err); var fdSlicer = new FdSlicer(fd, {autoClose: true}); var ws = fdSlicer.createWriteStream(); var progressEventCount = 0; var prevBytesWritten = 0; ws.on('progress', function() { progressEventCount += 1; assert.ok(ws.bytesWritten > prevBytesWritten); prevBytesWritten = ws.bytesWritten; }); fdSlicer.on('close', function() { assert.ok(progressEventCount > 5); done(); }); for (var i = 0; i < 10; i += 1) { ws.write(new Buffer(16 * 1024 * 2)); } ws.end(); }); }); it("write stream unrefs when destroyed", function(done) { fs.open(testOutBlobFile, 'w', function(err, fd) { if (err) return done(err); var fdSlicer = new FdSlicer(fd, {autoClose: true}); var ws = fdSlicer.createWriteStream(); fdSlicer.on('close', done); ws.write(new Buffer(1000)); ws.destroy(); }); }); it("read stream unrefs when destroyed", function(done) { fs.open(testBlobFile, 'r', function(err, fd) { if (err) return done(err); var fdSlicer = new FdSlicer(fd, {autoClose: true}); var rs = fdSlicer.createReadStream(); rs.on('error', function(err) { assert.strictEqual(err.message, "stream destroyed"); fdSlicer.on('close', done); }); rs.destroy(); }); }); });