mongo-tools-r3.2.11/000077500000000000000000000000001276133741700142505ustar00rootroot00000000000000mongo-tools-r3.2.11/.gitattributes000066400000000000000000000001061276133741700171400ustar00rootroot00000000000000# Force gpm to always have lf endings even on Windows gpm text eol=lf mongo-tools-r3.2.11/.gitignore000066400000000000000000000004451276133741700162430ustar00rootroot00000000000000*.swp *.exe *.pyc bin/ vendor/pkg bsondump/main/bsondump mongodump/main/mongodump mongoexport/main/mongoexport mongofiles/main/mongofiles mongoimport/main/mongoimport mongooplog/main/mongooplog mongorestore/main/mongorestore mongotop/main/mongotop mongostat/main/mongostat .gopath .godeps mongo-tools-r3.2.11/CONTRIBUTING.md000066400000000000000000000053461276133741700165110ustar00rootroot00000000000000Contributing to the MongoDB Tools Project =================================== Pull requests are always welcome, and the MongoDB engineering team appreciates any help the community can give to make the MongoDB tools better. For any particular improvement you want to make, you can begin a discussion on the [MongoDB Developers Forum](https://groups.google.com/forum/?fromgroups#!forum/mongodb-dev). This is the best place to discuss your proposed improvement (and its implementation) with the core development team. If you're interested in contributing, we have a list of some suggested tickets that are easy enough to get started on here: https://jira.mongodb.org/issues/?jql=project%20%3D%20TOOLS%20AND%20labels%20%3D%20community%20and%20status%20%3D%20open Getting Started --------------- 1. Create a [MongoDB JIRA account](https://jira.mongodb.org/secure/Signup!default.jspa). 2. Create a [Github account](https://github.com/signup/free). 3. [Fork](https://help.github.com/articles/fork-a-repo/) the repository on Github at https://github.com/mongodb/mongo-tools. 4. For more details see http://www.mongodb.org/about/contributors/. JIRA Tickets ------------ 1. File a JIRA ticket in the [TOOLS project](https://jira.mongodb.org/browse/TOOLS). 2. All commit messages to the MongoDB Tools repository must be prefaced with the relevant JIRA ticket number e.g. "TOOLS-XXX: add support for xyz". In filing JIRA tickets for bugs, please clearly describe the issue you are resolving, including the platforms on which the issue is present and clear steps to reproduce. For improvements or feature requests, be sure to explain the goal or use case, and the approach your solution will take. Style Guide ----------- All commits to the MongoDB Tools repository must pass golint: ```go run vendor/src/github.com/3rf/mongo-lint/golint/golint.go mongo* bson* common/*``` _We use a modified version of [golint](https://github.com/golang/lint)_ Testing ------- To run unit and integration tests: ``` go test -v -test.types=unit,integration ``` This should be run in all package directories - common, mongorestore, mongoexport, etc. The `test.types` flag indicates what kinds of tests to run. Integration tests require a `mongod` (running on port 33333) while unit tests do not. To run the quality assurance tests, you need to have the latest stable version of the rebuilt tools, `mongod`, `mongos`, and `mongo` in your current working directory. ``` cd test/qa-tests python buildscripts/smoke.py bson export files import oplog restore stat top ``` _Some tests require older binaries that are named accordingly (e.g. `mongod-2.4`, `mongod-2.6`, etc). You can use [setup_multiversion_mongodb.py](test/qa-tests/buildscripts/setup_multiversion_mongodb.py) to download those binaries_ mongo-tools-r3.2.11/Godeps000066400000000000000000000014511276133741700154150ustar00rootroot00000000000000gopkg.in/mgo.v2 1e52f6152a9b262873f831bb5a94bcd29ef38c38 github.com/10gen/openssl gopkg.in/tomb.v2 14b3d72120e8d10ea6e6b7f87f7175734b1faab8 github.com/go-tomb/tomb github.com/jacobsa/oglematchers 3ecefc49db07722beca986d9bb71ddd026b133f0 github.com/smartystreets/goconvey 995f5b2e021c69b8b028ba6d0b05c1dd500783db github.com/jessevdk/go-flags 5e118789801496c93ba210d34ef1f2ce5a9173bd github.com/3rf/mongo-lint f6cf4f8a7d07167375b035d186a1e8b3ebf28aa3 github.com/spacemonkeygo/openssl 688903e99b30b3f3a54c03f069085a246bf300b1 github.com/10gen/openssl github.com/howeyc/gopass 2c70fa70727c953c51695f800f25d6b44abb368e golang.org/x/crypto c57d4a71915a248dbad846d60825145062b4c18e mongo-tools-r3.2.11/LICENSE.md000066400000000000000000000010521276133741700156520ustar00rootroot00000000000000Copyright 2014 MongoDB, Inc. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. mongo-tools-r3.2.11/README.md000066400000000000000000000034371276133741700155360ustar00rootroot00000000000000MongoDB Tools =================================== - **bsondump** - _display BSON files in a human-readable format_ - **mongoimport** - _Convert data from JSON, TSV or CSV and insert them into a collection_ - **mongoexport** - _Write an existing collection to CSV or JSON format_ - **mongodump/mongorestore** - _Dump MongoDB backups to disk in .BSON format, or restore them to a live database_ - **mongostat** - _Monitor live MongoDB servers, replica sets, or sharded clusters_ - **mongofiles** - _Read, write, delete, or update files in [GridFS](http://docs.mongodb.org/manual/core/gridfs/)_ - **mongooplog** - _Replay oplog entries between MongoDB servers_ - **mongotop** - _Monitor read/write activity on a mongo server_ Report any bugs, improvements, or new feature requests at https://jira.mongodb.org/browse/TOOLS Setup --------------- Clone the repo and run `. ./set_gopath.sh` (`set_gopath.bat` on Windows) to setup your GOPATH: ``` git clone https://github.com/mongodb/mongo-tools cd mongo-tools . ./set_gopath.sh ``` Building Tools --------------- To build the tools, you need to have Go version 1.3 and up. An additional flag, `-tags`, can be passed to the `go build` command in order to build the tools with support for SSL and/or SASL. For example: ``` mkdir bin go build -o bin/mongoimport mongoimport/main/mongoimport.go # build mongoimport go build -o bin/mongoimport -tags ssl mongoimport/main/mongoimport.go # build mongoimport with SSL support enabled go build -o bin/mongoimport -tags "ssl sasl" mongoimport/main/mongoimport.go # build mongoimport with SSL and SASL support enabled ``` Contributing --------------- See our [Contributor's Guide](CONTRIBUTING.md). Documentation --------------- See the MongoDB packages [documentation](http://docs.mongodb.org/master/reference/program/). mongo-tools-r3.2.11/THIRD-PARTY-NOTICES000066400000000000000000001160441276133741700170520ustar00rootroot00000000000000MongoDB uses third-party libraries or other resources that may be distributed under licenses different than the MongoDB software. In the event that we accidentally failed to list a required notice, please bring it to our attention through any of the ways detailed here : mongodb-dev@googlegroups.com The attached notices are provided for information only. 1) License notice for github.com/nsf/gocode ------------------------------------------------ Copyright (C) 2010 nsf Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 2) License notice for github.com/spacemonkeygo/spacelog ------------------------------------------------ Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: You must give any other recipients of the Work or Derivative Works a copy of this License; and You must cause any modified files to carry prominent notices stating that You changed the files; and You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. 3) License notice for gopkg.in/mgo.v2 ------------------------------------------------ mgo - MongoDB driver for Go Copyright (c) 2010-2013 - Gustavo Niemeyer All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 4) License notice for gopkg.in/mgo.v2/bson ------------------------------------------------ BSON library for Go Copyright (c) 2010-2012 - Gustavo Niemeyer All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 5) License notice for github.com/jacobsa/oglematchers ------------------------------------------------ Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. 6) License notice for gopkg.in/tomb.v2 ------------------------------------------------ tomb - support for clean goroutine termination in Go. Copyright (c) 2010-2011 - Gustavo Niemeyer All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 7) License notice for github.com/jessevdk/go-flags ------------------------------------------------ Copyright (c) 2012 Jesse van den Kieboom. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of Google Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 8) License notice for github.com/spacemonkeygo/openssl ------------------------------------------------ Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: You must give any other recipients of the Work or Derivative Works a copy of this License; and You must cause any modified files to carry prominent notices stating that You changed the files; and You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. mongo-tools-r3.2.11/bsondump/000077500000000000000000000000001276133741700160775ustar00rootroot00000000000000mongo-tools-r3.2.11/bsondump/bsondump.go000066400000000000000000000125261276133741700202630ustar00rootroot00000000000000// Package bsondump converts BSON files into human-readable formats such as JSON. package bsondump import ( "bytes" "fmt" "github.com/mongodb/mongo-tools/common/bsonutil" "github.com/mongodb/mongo-tools/common/db" "github.com/mongodb/mongo-tools/common/json" "github.com/mongodb/mongo-tools/common/log" "github.com/mongodb/mongo-tools/common/options" "gopkg.in/mgo.v2/bson" "io" "os" "strings" ) // BSONDump is a container for the user-specified options and // internal state used for running bsondump. type BSONDump struct { // generic mongo tool options ToolOptions *options.ToolOptions // BSONDumpOptions defines options used to control how BSON data is displayed BSONDumpOptions *BSONDumpOptions // Path to the BSON file FileName string // Handle to where the BSON data should be displayed Out io.Writer bsonSource *db.BSONSource } // Open opens the relevant file for reading. It returns a // non-nil error if it is unable to open the file. func (bd *BSONDump) Open() error { file, err := os.Open(bd.FileName) if err != nil { return fmt.Errorf("couldn't open BSON file: %v", err) } bd.bsonSource = db.NewBSONSource(file) return nil } func printJSON(doc *bson.Raw, out io.Writer, pretty bool) error { decodedDoc := bson.D{} err := bson.Unmarshal(doc.Data, &decodedDoc) if err != nil { return err } extendedDoc, err := bsonutil.ConvertBSONValueToJSON(decodedDoc) if err != nil { return fmt.Errorf("error converting BSON to extended JSON: %v", err) } jsonBytes, err := json.Marshal(extendedDoc) if pretty { var jsonFormatted bytes.Buffer json.Indent(&jsonFormatted, jsonBytes, "", "\t") jsonBytes = jsonFormatted.Bytes() } if err != nil { return fmt.Errorf("error converting doc to JSON: %v", err) } _, err = out.Write(jsonBytes) return err } // JSON iterates through the BSON file and for each document it finds, // recursively descends into objects and arrays and prints the human readable // JSON representation. // It returns the number of documents processed and a non-nil error if one is // encountered before the end of the file is reached. func (bd *BSONDump) JSON() (int, error) { numFound := 0 if bd.bsonSource == nil { panic("Tried to call JSON() before opening file") } decodedStream := db.NewDecodedBSONSource(bd.bsonSource) defer decodedStream.Close() var result bson.Raw for decodedStream.Next(&result) { if err := printJSON(&result, bd.Out, bd.BSONDumpOptions.Pretty); err != nil { log.Logf(log.Always, "unable to dump document %v: %v", numFound+1, err) //if objcheck is turned on, stop now. otherwise keep on dumpin' if bd.BSONDumpOptions.ObjCheck { return numFound, err } } else { _, err := bd.Out.Write([]byte("\n")) if err != nil { return numFound, err } } numFound++ } if err := decodedStream.Err(); err != nil { return numFound, err } return numFound, nil } // Debug iterates through the BSON file and for each document it finds, // recursively descends into objects and arrays and prints a human readable // BSON representation containing the type and size of each field. // It returns the number of documents processed and a non-nil error if one is // encountered before the end of the file is reached. func (bd *BSONDump) Debug() (int, error) { numFound := 0 if bd.bsonSource == nil { panic("Tried to call Debug() before opening file") } defer bd.bsonSource.Close() var result bson.Raw for { doc := bd.bsonSource.LoadNext() if doc == nil { break } result.Data = doc if bd.BSONDumpOptions.ObjCheck { validated := bson.M{} err := bson.Unmarshal(result.Data, &validated) if err != nil { // ObjCheck is turned on and we hit an error, so short-circuit now. return numFound, fmt.Errorf("failed to validate bson during objcheck: %v", err) } } err := printBSON(result, 0, bd.Out) if err != nil { log.Logf(log.Always, "encountered error debugging BSON data: %v", err) } numFound++ } if err := bd.bsonSource.Err(); err != nil { // This error indicates the BSON document header is corrupted; // either the 4-byte header couldn't be read in full, or // the size in the header would require reading more bytes // than the file has left return numFound, err } return numFound, nil } func printBSON(raw bson.Raw, indentLevel int, out io.Writer) error { indent := strings.Repeat("\t", indentLevel) fmt.Fprintf(out, "%v--- new object ---\n", indent) fmt.Fprintf(out, "%v\tsize : %v\n", indent, len(raw.Data)) //Convert raw into an array of RawD we can iterate over. var rawD bson.RawD err := bson.Unmarshal(raw.Data, &rawD) if err != nil { return err } for _, rawElem := range rawD { fmt.Fprintf(out, "%v\t\t%v\n", indent, rawElem.Name) // the size of an element is the combined size of the following: // 1. 1 byte for the BSON type // 2. 'e_name' : the BSON key, which is a null-terminated cstring // 3. The BSON value // So size == 1 [size of type byte] + 1 [null byte for cstring key] + len(bson key) + len(bson value) // see http://bsonspec.org/spec.html for more details fmt.Fprintf(out, "%v\t\t\ttype: %4v size: %v\n", indent, int8(rawElem.Value.Kind), 2+len(rawElem.Name)+len(rawElem.Value.Data)) //For nested objects or arrays, recurse. if rawElem.Value.Kind == 0x03 || rawElem.Value.Kind == 0x04 { err = printBSON(rawElem.Value, indentLevel+3, out) if err != nil { return err } } } return nil } mongo-tools-r3.2.11/bsondump/main/000077500000000000000000000000001276133741700170235ustar00rootroot00000000000000mongo-tools-r3.2.11/bsondump/main/bsondump.go000066400000000000000000000041661276133741700212100ustar00rootroot00000000000000// Main package for the bsondump tool. package main import ( "github.com/mongodb/mongo-tools/bsondump" "github.com/mongodb/mongo-tools/common/log" "github.com/mongodb/mongo-tools/common/options" "github.com/mongodb/mongo-tools/common/signals" "github.com/mongodb/mongo-tools/common/util" "os" ) func main() { go signals.Handle() // initialize command-line opts opts := options.New("bsondump", bsondump.Usage, options.EnabledOptions{}) bsonDumpOpts := &bsondump.BSONDumpOptions{} opts.AddOptions(bsonDumpOpts) args, err := opts.Parse() if err != nil { log.Logf(log.Always, "error parsing command line options: %v", err) log.Logf(log.Always, "try 'bsondump --help' for more information") os.Exit(util.ExitBadOptions) } // print help, if specified if opts.PrintHelp(false) { return } // print version, if specified if opts.PrintVersion() { return } log.SetVerbosity(opts.Verbosity) // pull out the filename if len(args) == 0 { log.Logf(log.Always, "must provide a filename") log.Logf(log.Always, "try 'bsondump --help' for more information") os.Exit(util.ExitBadOptions) } else if len(args) > 1 { log.Logf(log.Always, "too many positional arguments: %v", args) log.Logf(log.Always, "try 'bsondump --help' for more information") os.Exit(util.ExitBadOptions) } dumper := bsondump.BSONDump{ ToolOptions: opts, BSONDumpOptions: bsonDumpOpts, FileName: args[0], Out: os.Stdout, } log.Logf(log.DebugLow, "running bsondump with --objcheck: %v", bsonDumpOpts.ObjCheck) if len(bsonDumpOpts.Type) != 0 && bsonDumpOpts.Type != "debug" && bsonDumpOpts.Type != "json" { log.Logf(log.Always, "Unsupported output type '%v'. Must be either 'debug' or 'json'", bsonDumpOpts.Type) os.Exit(util.ExitBadOptions) } err = dumper.Open() if err != nil { log.Logf(log.Always, "Failed: %v", err) os.Exit(util.ExitError) } var numFound int if bsonDumpOpts.Type == "debug" { numFound, err = dumper.Debug() } else { numFound, err = dumper.JSON() } log.Logf(log.Always, "%v objects found", numFound) if err != nil { log.Log(log.Always, err.Error()) os.Exit(util.ExitError) } } mongo-tools-r3.2.11/bsondump/options.go000066400000000000000000000014531276133741700201240ustar00rootroot00000000000000package bsondump var Usage = ` View and debug .bson files. See http://docs.mongodb.org/manual/reference/program/bsondump/ for more information.` type BSONDumpOptions struct { // Format to display the BSON data file Type string `long:"type" value-name:"" default:"json" default-mask:"-" description:"type of output: debug, json (default 'json')"` // Validate each BSON document before displaying ObjCheck bool `long:"objcheck" description:"validate BSON during processing"` // Display JSON data with indents Pretty bool `long:"pretty" description:"output JSON formatted to be human-readable"` } func (_ *BSONDumpOptions) Name() string { return "output" } func (_ *BSONDumpOptions) PostParse() error { return nil } func (_ *BSONDumpOptions) Validate() error { return nil } mongo-tools-r3.2.11/build.bat000066400000000000000000000005511276133741700160400ustar00rootroot00000000000000@echo off REM This is not handling tags if exist "%cd%\vendor\pkg" rd /s /q "%cd%\vendor\pkg" call set_gopath.bat if not exist "%cd%\bin" mkdir "%cd%\bin" for %%i in (bsondump, mongostat, mongofiles, mongoexport, mongoimport, mongorestore, mongodump, mongotop, mongooplog) do ( echo Building %%i go build -o "%cd%\bin\%%i.exe" "%cd%\%%i\main\%%i.go" ) mongo-tools-r3.2.11/build.sh000077500000000000000000000013521276133741700157070ustar00rootroot00000000000000#!/bin/sh set -o errexit tags="" if [ ! -z "$1" ] then tags="$@" fi # make sure we're in the directory where the script lives SCRIPT_DIR="$(cd "$(dirname ${BASH_SOURCE[0]})" && pwd)" cd $SCRIPT_DIR sed -i.bak -e "s/built-without-version-string/$(git describe)/" \ -e "s/built-without-git-spec/$(git rev-parse HEAD)/" \ common/options/options.go # remove stale packages rm -rf vendor/pkg . ./set_gopath.sh mkdir -p bin for i in bsondump mongostat mongofiles mongoexport mongoimport mongorestore mongodump mongotop mongooplog; do echo "Building ${i}..." go build -o "bin/$i" -tags "$tags" "$i/main/$i.go" ./bin/$i --version done mv -f common/options/options.go.bak common/options/options.go mongo-tools-r3.2.11/common.yml000066400000000000000000001426671276133741700163030ustar00rootroot00000000000000####################################### # Tools Driver Config for MCI # ####################################### # default command type command_type: system # run the same task in the previous revision if the current task fails stepback: true disable_cleanup: true mongo_tools_variables: ## Common download URLs (merge in as hashes) mongo_download_urls: osx_1010: &mongo_url_osx1010 mongo_os: "osx" mongo_url: "https://fastdl.mongodb.org/osx/mongodb-osx-x86_64-latest.tgz" mongo32_url: "https://fastdl.mongodb.org/osx/mongodb-osx-x86_64-3.2.8.tgz" mongo30_url: "https://fastdl.mongodb.org/osx/mongodb-osx-x86_64-3.0.7.tgz" mongo26_url: "https://fastdl.mongodb.org/osx/mongodb-osx-x86_64-2.6.5.tgz" mongo24_url: "https://fastdl.mongodb.org/osx/mongodb-osx-x86_64-2.4.12.tgz" solaris: &mongo_url_solaris mongo_url: "https://fastdl.mongodb.org/sunos5/mongodb-sunos5-x86_64-latest.tgz" mongo32_url: "https://fastdl.mongodb.org/sunos5/mongodb-sunos5-x86_64-3.2.8.tgz" mongo30_url: "https://fastdl.mongodb.org/sunos5/mongodb-sunos5-x86_64-3.0.7.tgz" mongo26_url: "https://fastdl.mongodb.org/sunos5/mongodb-sunos5-x86_64-2.6.5.tgz" mongo24_url: "https://fastdl.mongodb.org/sunos5/mongodb-sunos5-x86_64-2.4.12.tgz" linux_64_bit: &mongo_url_ubuntu1204 mongo_os: "ubuntu1204" mongo_url: "http://downloads.10gen.com/linux/mongodb-linux-x86_64-enterprise-ubuntu1204-latest.tgz" mongo32_url: "http://downloads.10gen.com/linux/mongodb-linux-x86_64-enterprise-ubuntu1204-3.2.8.tgz" mongo30_url: "http://downloads.10gen.com/linux/mongodb-linux-x86_64-enterprise-ubuntu1204-3.0.7.tgz" mongo26_url: "http://downloads.10gen.com/linux/mongodb-linux-x86_64-enterprise-ubuntu1204-2.6.5.tgz" mongo24_url: "http://downloads.10gen.com/linux/mongodb-linux-x86_64-subscription-ubuntu1204-2.4.12.tgz" linux_64_bit_enterprise: &mongo_url_ubuntu1204_enterprise mongo_os: "ubuntu1204" mongo_url: "http://downloads.10gen.com/linux/mongodb-linux-x86_64-enterprise-ubuntu1204-latest.tgz" mongo32_url: "http://downloads.10gen.com/linux/mongodb-linux-x86_64-enterprise-ubuntu1204-3.2.8.tgz" mongo30_url: "http://downloads.10gen.com/linux/mongodb-linux-x86_64-enterprise-ubuntu1204-3.0.7.tgz" mongo26_url: "http://downloads.10gen.com/linux/mongodb-linux-x86_64-enterprise-ubuntu1204-2.6.5.tgz" mongo24_url: "http://downloads.10gen.com/linux/mongodb-linux-x86_64-subscription-ubuntu1204-2.4.12.tgz" windows_32_bit: &mongo_url_windows32 mongo_url: "https://fastdl.mongodb.org/win32/mongodb-win32-i386-latest.zip" mongo32_url: "https://fastdl.mongodb.org/win32/mongodb-win32-i386-3.2.8.zip" mongo30_url: "https://fastdl.mongodb.org/win32/mongodb-win32-i386-3.0.7.zip" mongo26_url: "https://fastdl.mongodb.org/win32/mongodb-win32-i386-2.6.5.zip" mongo24_url: "https://fastdl.mongodb.org/win32/mongodb-win32-i386-2.4.12.zip" windows_64_bit: &mongo_url_windows64 mongo_os: "windows-64" mongo_url: "https://fastdl.mongodb.org/win32/mongodb-win32-x86_64-2008plus-latest.zip" mongo32_url: "https://fastdl.mongodb.org/win32/mongodb-win32-x86_64-2008plus-3.2.8.zip" mongo30_url: "https://fastdl.mongodb.org/win32/mongodb-win32-x86_64-2008plus-3.0.7.zip" mongo26_url: "https://fastdl.mongodb.org/win32/mongodb-win32-x86_64-2008plus-2.6.5.zip" mongo24_url: "https://fastdl.mongodb.org/win32/mongodb-win32-x86_64-2008plus-2.4.12.zip" windows_64_bit_enterprise: &mongo_url_windows64_enterprise mongo_os: "windows-64" mongo_url: "http://downloads.10gen.com/win32/mongodb-win32-x86_64-enterprise-windows-64-latest.zip" mongo32_url: "http://downloads.10gen.com/win32/mongodb-win32-x86_64-enterprise-windows-64-3.2.8.zip" mongo30_url: "http://downloads.10gen.com/win32/mongodb-win32-x86_64-enterprise-windows-64-3.0.7.zip" mongo26_url: "http://downloads.10gen.com/win32/mongodb-win32-x86_64-enterprise-windows-64-2.6.5.zip" ## List of tests to run on each buildvariant mongo_tools_task_lists: osx_1010_task_list: &osx_1010_tasks - name: db - name: dist - name: integration - name: integration-auth - name: legacy28 - name: legacy26 - name: legacy24 - name: qa-tests - name: qa-dump-restore-archiving - name: qa-dump-restore-gzip - name: unit osx_1010_ssl_task_list: &osx_1010_ssl_tasks - name: dist - name: qa-tests solaris_task_list: &solaris_tasks - name: db - name: dist - name: integration - name: integration-auth - name: legacy28 - name: legacy26 - name: legacy24 ubuntu1204_task_list: &ubuntu1204_tasks - name: bsonutil - name: db - name: dist - name: integration - name: integration-auth - name: json - name: legacy28 - name: legacy28-wt - name: legacy26 - name: legacy24 - name: lint - name: log - name: progress - name: qa-tests - name: qa-tests-15k - name: qa-tests-unstable - name: qa-dump-restore-archiving - name: qa-dump-restore-gzip - name: qa-tests-wt - name: text - name: unit - name: util - name: vet ubuntu1204_ssl_task_list: &ubuntu1204_ssl_tasks - name: dist - name: integration - name: integration-auth - name: legacy28 - name: qa-tests - name: qa-tests-15k - name: qa-tests-unstable ubuntu1204_enterprise_task_list: &ubuntu1204_enterprise_tasks - name: bsonutil - name: db - name: dist - name: integration - name: integration-auth - name: json - name: kerberos - name: legacy28 - name: lint - name: log - name: progress - name: qa-tests - name: qa-tests-15k - name: text - name: util ubuntu1204_race_task_list: &ubuntu1204_race_tasks - name: bsonutil - name: db - name: dist - name: integration - name: integration-auth - name: json - name: legacy28 - name: legacy28-wt - name: legacy26 - name: legacy24 - name: log - name: qa-tests - name: qa-tests-wt - name: text - name: unit - name: util windows_32_task_list: &windows_32_tasks - name: db distros: - windows-64-vs2013-test - name: dist - name: integration - name: integration-auth - name: legacy28 distros: - windows-64-vs2013-test - name: legacy26 distros: - windows-64-vs2013-test - name: legacy24 distros: - windows-64-vs2013-test - name: unit windows_64_task_list: &windows_64_tasks - name: db - name: dist - name: integration - name: integration-auth - name: legacy28 distros: - windows-64-vs2013-test - name: legacy26 distros: - windows-64-vs2013-test - name: legacy24 distros: - windows-64-vs2013-test - name: qa-tests distros: - windows-64-vs2013-test - name: qa-dump-restore-archiving distros: - windows-64-vs2013-test - name: qa-dump-restore-gzip distros: - windows-64-vs2013-test - name: unit windows_64_ssl_task_list: &windows_64_ssl_tasks - name: dist - name: integration - name: integration-auth - name: legacy28 - name: qa-tests windows_64_enterprise_task_list: &windows_64_enterprise_tasks - name: db - name: dist - name: integration - name: integration-auth - name: kerberos - name: legacy28 distros: - windows-64-vs2013-test - name: qa-tests distros: - windows-64-vs2013-test - name: unit ## Common mongodb arguments mongod_arguments: default: &mongod_default_startup_args mongod_args: &mongod_default_startup_args_string "--port 33333" ssl: &mongod_ssl_startup_args mongod_args: "--port 33333 --sslMode requireSSL --sslCAFile common/db/openssl/testdata/ca.pem --sslPEMKeyFile common/db/openssl/testdata/server.pem" # Set storage engine as mmapv1 for 32 bit variants because WiredTiger requires 64 bit support. win32: &mongod_win32_startup_args mongod_args: "--port 33333 --storageEngine=mmapv1" mongo_arguments: default: &mongo_default_startup_args mongo_args: &mongo_default_startup_args_string "--port 33333" ssl: &mongo_ssl_startup_args mongo_args: "--port 33333 --ssl --sslCAFile ./common/db/openssl/testdata/ca.pem --sslPEMKeyFile ./common/db/openssl/testdata/server.pem --sslAllowInvalidCertificates" functions: "build tool": command: shell.exec type: test params: working_dir: src script: | echo "Building ${tool}..." if [ "Windows_NT" = "$OS" ]; then set -o igncr fi; if [ '${library_path}' != '' ]; then export ${library_path} fi # In RHEL 5.5, /usr/bin/ld can't handle --build-id parameters, so # use a wrapper if it's present on the system # if [ -d /opt/ldwrapper/bin ] then export PATH=/opt/ldwrapper/bin:$PATH fi sed -i.bak "s/built-without-version-string/$(git describe)/" common/options/options.go sed -i.bak "s/built-without-git-spec/$(git rev-parse HEAD)/" common/options/options.go . ./set_gopath.sh ${gorootvars} go build ${args} ${build_tags} -o bin/${tool} ${tool}/main/${tool}.go ./bin/${tool} --version "download mongod": command: shell.exec params: working_dir: src script: | set -o verbose rm -rf mongodb mkdir mongodb cd mongodb curl -s ${mongo_url} --output mongodb.tgz ${decompress} mongodb.tgz chmod +x ./mongodb-*/bin/* mv ./mongodb-*/bin/* . "download & copy shell": command: shell.exec params: working_dir: src script: | set -o verbose rm -rf temp mkdir temp cd temp curl -s ${mongo_url} --output mongodb.tgz ${decompress} mongodb.tgz chmod +x ./mongodb-*/bin/* mv ./mongodb-*/bin/mongo${extension} ../mongodb/ "fetch tool" : command: s3.get params: bucket: mciuploads aws_key: ${aws_key} aws_secret: ${aws_secret} local_file: src/${tool}${extension} remote_file: mongo-tools/binaries/${build_id}/${edition|community}/${tool}${extension} "generate coverage html + text": command: shell.exec type: test params: working_dir: src script: | set -o verbose if [ "${coverage}" = "true" ]; then if [ "Windows_NT" = "$OS" ]; then set -o igncr fi; . ./set_gopath.sh go tool cover -html=coverage.out -o coverage.html go tool cover -func=coverage.out -o coverage.txt fi; "get buildnumber": command: keyval.inc params: key: "${build_variant}_tools" destination: "builder_num" "move coverage data": command: shell.exec params: working_dir: src script: | set -o verbose if [ "${coverage}" = "true" ]; then mv ${package}/coverage.out . fi "run unit test": command: shell.exec type: test params: working_dir: src script: | if [ "Windows_NT" = "$OS" ]; then set -o igncr fi; . ./set_gopath.sh if [ '${library_path}' != '' ]; then export ${library_path} fi export MONGODB_KERBEROS_PASSWORD=${kerberos_password} export basedir=`pwd` cd ${package} go test ${coverage_args} ${args} ${build_tags} -test.v > unit.suite export exitcode=$? cat unit.suite cp unit.suite $basedir/. exit $exitcode "setup integration test": command: shell.exec params: working_dir: src # Set up Kerberos stuff: run kinit if necessary, and add KDC to registry # on Windows (see https://wiki.mongodb.com/display/DH/Testing+Kerberos) script: | if [ '${run_kinit}' = 'true' ] then echo "${kerberos_password}" | kinit -p drivers@LDAPTEST.10GEN.CC; fi; if [ "Windows_NT" = "$OS" ]; then cmd /c "REG ADD HKLM\SYSTEM\ControlSet001\Control\Lsa\Kerberos\Domains\LDAPTEST.10GEN.CC /v KdcNames /d ldaptest.10gen.cc /t REG_MULTI_SZ /f" fi; "setup permissions": command: shell.exec params: working_dir: src script: | chmod 700 set_gopath.sh "run tool unit tests": command: shell.exec type: test params: working_dir: src script: | if [ "Windows_NT" = "$OS" ]; then set -o igncr fi; . ./set_gopath.sh # TODO bsondump needs tests # TODO mongotop needs tests for i in mongoimport mongoexport mongostat mongooplog mongorestore mongodump mongofiles; do cd $i COVERAGE_ARGS="" if [ "${run_coverage}" ]; then COVERAGE_ARGS="-coverprofile=coverage_$i.out" fi; ${library_path} go test $COVERAGE_ARGS ${args} ${build_tags} -test.v > $i.suite if [ $? -ne 0 ]; then echo "suite failed, will exit with error on finish" export exitcode=1 fi cat $i.suite cp $i.suite ../. cd .. done exit $exitcode "run tool integration tests": command: shell.exec type: test params: working_dir: src script: | if [ "Windows_NT" = "$OS" ]; then set -o igncr export MONGODB_KERBEROS_PASSWORD=${kerberos_password} fi; . ./set_gopath.sh if [ "${library_path}" != "" ]; then export ${library_path} fi; if [ "${create_mongod_users_command}" != "" ]; then export AUTH_USERNAME=${auth_username} export AUTH_PASSWORD=${auth_password} echo "${create_mongod_users_command}" | ./mongodb/mongo${extension} ${mongo_args} admin fi; for i in mongoimport mongoexport mongostat mongooplog mongorestore mongodump mongofiles; do cd $i COVERAGE_ARGS="" if [ "${run_coverage}" ]; then COVERAGE_ARGS="-coverprofile=coverage_$i.out" fi; ${library_path} go test $COVERAGE_ARGS -test.v ${args} ${build_tags} > $i.suite if [ $? -ne 0 ]; then echo "suite failed, will exit with error on finish" export exitcode=1 fi cat $i.suite cp $i.suite ../. cd .. done exit $exitcode "create coverage reports": command: shell.exec type: test params: working_dir: src script: | if [ "Windows_NT" = "$OS" ]; then set -o igncr fi; . ./set_gopath.sh for i in mongoimport mongoexport mongostat mongooplog mongorestore mongodump mongofiles; do cd $i perl -pe 's/.*src/github.com\/mongodb\/mongo-tools/' coverage_$i.out > coverage_$i_rewrite.out ${library_path} go tool cover -html=coverage_$i_rewrite.out -o coverage_$i.html ${library_path} go tool cover -func=coverage_$i_rewrite.out -o coverage_$i.txt cd .. done "upload html coverage": command: s3.put params: aws_key: ${aws_key} aws_secret: ${aws_secret} local_file: src/${coverage_pkg}/coverage_${coverage_pkg}.html remote_file: mongo-tools/coverage/${coverage_pkg}/${task_id}.html bucket: mciuploads permissions: public-read content_type: text/html build_variants: ["ubuntu", "windows-64"] display_name: ${coverage_pkg}-html "upload text coverage": command: s3.put params: aws_key: ${aws_key} aws_secret: ${aws_secret} local_file: src/${coverage_pkg}/coverage_${coverage_pkg}.txt remote_file: mongo-tools/coverage/${coverage_pkg}/${task_id}.txt bucket: mciuploads permissions: public-read content_type: text/plain build_variants: ["ubuntu", "windows-64"] display_name: ${coverage_pkg}-text "setup credentials" : command: shell.exec params: working_dir: src silent: true script: | cat > mci.buildlogger </dev/null >/dev/null if [ "$?" = "0" ]; then echo "mongod ready"; exit 0 else SECONDS=`expr $SECONDS + 1` if [ $SECONDS -gt 20 ]; then echo "mongod not ready after 20 seconds" exit 1 fi echo "waiting for mongod to be ready..." ; sleep 1 ; fi done "upload tool": command: s3.put params: aws_key: ${aws_key} aws_secret: ${aws_secret} local_file: src/bin/${tool} remote_file: mongo-tools/binaries/${build_id}/${edition|community}/${tool}${extension} bucket: mciuploads permissions: public-read content_type: application/octet-stream display_name: ${tool} pre: - command: expansions.fetch params: keys: - local_key: "aws_key" remote_key: "project_aws_key" - local_key: "aws_secret" remote_key: "project_aws_secret" - local_key: "slave" remote_key: "slave" - local_key: "passwd" remote_key: "passwd" - local_key: "kerberos_password" remote_key: "kerberos_password" - command: shell.exec params: silent: true script: | ${killall_mci|pkill -9 mongo; pkill -9 mongodump; pkill -9 mongoexport; pkill -9 mongoimport; pkill -9 mongofiles; pkill -9 mongooplog; pkill -9 mongorestore; pkill -9 mongostat; pkill -9 mongotop; pkill -9 mongod; pkill -9 mongos; pkill -f buildlogger.py; pkill -f smoke.py} >/dev/null 2>&1 rm -rf src /data/db/* exit 0 post: - command: attach.results params: file_location: src/report.json - command: shell.exec params: silent: true script: | ${killall_mci|pkill -9 mongo; pkill -9 mongodump; pkill -9 mongoexport; pkill -9 mongoimport; pkill -9 mongofiles; pkill -9 mongooplog; pkill -9 mongorestore; pkill -9 mongostat; pkill -9 mongotop; pkill -9 mongod; pkill -9 mongos; pkill -f buildlogger.py; pkill -f smoke.py} >/dev/null 2>&1 exit 0 - command: gotest.parse_files params: files: ["src/*.suite"] - command: shell.exec params: script: | rm -rf /data/db/* exit 0 tasks: - name: bsonutil commands: - command: git.get_project params: directory: src - command: git.apply_patch params: directory: src - command: expansions.update params: updates: - key: "package" value: "common/bsonutil" - func: "run unit test" - name: db commands: - command: git.get_project params: directory: src - command: git.apply_patch params: directory: src - command: expansions.update params: updates: - key: "package" value: "common/db" - key: "args" value: "-test.types=db" - func: "download mongod" vars: mongo_url: "${mongo32_url}" - func: "start mongod" - func: "wait for mongod to be ready" - func: "setup integration test" - func: "run unit test" - name: dist depends_on: commands: - command: git.get_project params: directory: src - command: git.apply_patch params: directory: src - command: shell.exec params: working_dir: src script: | rm -rf bin/ mkdir bin # bsondump - func: "build tool" vars: tool: bsondump - func: "upload tool" vars: tool: bsondump # mongodump - func: "build tool" vars: tool: mongodump - func: "upload tool" vars: tool: mongodump # mongoexport - func: "build tool" vars: tool: mongoexport - func: "upload tool" vars: tool: mongoexport # mongofiles - func: "build tool" vars: tool: mongofiles - func: "upload tool" vars: tool: mongofiles # mongoimport - func: "build tool" vars: tool: mongoimport - func: "upload tool" vars: tool: mongoimport # mongooplog - func: "build tool" vars: tool: mongooplog - func: "upload tool" vars: tool: mongooplog # mongorestore - func: "build tool" vars: tool: mongorestore - func: "upload tool" vars: tool: mongorestore # mongostat - func: "build tool" vars: tool: mongostat - func: "upload tool" vars: tool: mongostat # mongotop - func: "build tool" vars: tool: mongotop - func: "upload tool" vars: tool: mongotop - name: integration commands: - command: git.get_project params: directory: src - command: git.apply_patch params: directory: src - command: expansions.update params: updates: - key: "args" value: "-test.types=${integration_test_args}" - func: "download mongod" vars: mongo_url: "${mongo32_url}" - func: "start mongod" - func: "wait for mongod to be ready" - func: "run tool integration tests" - name: integration-auth commands: - command: git.get_project params: directory: src - command: git.apply_patch params: directory: src # Concat auth args - command: expansions.update params: updates: - key: "args" value: "-test.types=${integration_test_args},auth" - key: "mongod_args" concat: " --auth" - key: "auth_username" value: "passwordIsTaco" - key: "auth_password" value: "Taco" - key: "create_mongod_users_command" value: "db.createUser({ user: '${auth_username}', pwd: '${auth_password}', roles: [{ role: '__system', db: 'admin' }] });" - func: "download mongod" vars: mongo_url: "${mongo32_url}" - func: "start mongod" - func: "wait for mongod to be ready" - func: "run tool integration tests" - name: json commands: - command: git.get_project params: directory: src - command: git.apply_patch params: directory: src - command: expansions.update params: updates: - key: "package" value: "common/json" - func: "run unit test" - name: kerberos commands: - command: git.get_project params: directory: src - command: git.apply_patch params: directory: src # Explicitly run ONLY Kerberos tests - command: expansions.update params: updates: - key: "args" value: "-test.types=kerberos" - func: "setup integration test" - func: "run tool integration tests" - name: legacy28 depends_on: - name: dist commands: - command: git.get_project params: directory: src - command: git.apply_patch params: directory: src - func: "get buildnumber" - func: "setup credentials" - func: "download mongod" vars: mongo_url: "${mongo30_url}" - func: "fetch tool" vars: tool: mongoimport - func: "fetch tool" vars: tool: mongoexport - func: "fetch tool" vars: tool: mongodump - func: "fetch tool" vars: tool: mongostat - func: "fetch tool" vars: tool: mongorestore - func: "fetch tool" vars: tool: mongooplog - func: "fetch tool" vars: tool: mongofiles - command: shell.exec type: test params: working_dir: src script: | mv ./mongodb/mongod${extension} . mv ./mongodb/mongo${extension} . mv test/legacy28/* . chmod +x mongo* python buildscripts/smoke.py ${smoke_use_ssl} --nopreallocj --with-cleanbb --mongod ./mongod --mongo ./mongo --report-file report.json --continue-on-failure --buildlogger-builder MCI_${build_variant} --buildlogger-buildnum ${builder_num|} --buildlogger-credentials ./mci.buildlogger --buildlogger-phase ${task_name}_${execution} --authMechanism SCRAM-SHA-1 tool --buildlogger-url https://logkeeper.mongodb.org - name: legacy28-wt depends_on: - name: dist commands: - command: git.get_project params: directory: src - command: git.apply_patch params: directory: src - func: "get buildnumber" - func: "setup credentials" - func: "download mongod" vars: mongo_url: "${mongo30_url}" - func: "fetch tool" vars: tool: mongoimport - func: "fetch tool" vars: tool: mongoexport - func: "fetch tool" vars: tool: mongodump - func: "fetch tool" vars: tool: mongostat - func: "fetch tool" vars: tool: mongorestore - func: "fetch tool" vars: tool: mongooplog - func: "fetch tool" vars: tool: mongofiles - command: shell.exec type: test params: working_dir: src script: | mv ./mongodb/mongod${extension} . mv ./mongodb/mongo${extension} . mv test/legacy28/* . chmod +x mongo* python buildscripts/smoke.py ${smoke_use_ssl} --nopreallocj --with-cleanbb --mongod ./mongod --mongo ./mongo --report-file report.json --continue-on-failure --buildlogger-builder MCI_${build_variant} --buildlogger-buildnum ${builder_num|} --buildlogger-credentials ./mci.buildlogger --buildlogger-phase ${task_name}_${execution} --authMechanism SCRAM-SHA-1 --storageEngine=wiredTiger tool --buildlogger-url https://logkeeper.mongodb.org - name: legacy26 depends_on: - name: dist commands: - command: git.get_project params: directory: src - command: git.apply_patch params: directory: src - func: "get buildnumber" - func: "setup credentials" - func: "download mongod" vars: mongo_url: "${mongo26_url}" - func: "fetch tool" vars: tool: mongoimport - func: "fetch tool" vars: tool: mongoexport - func: "fetch tool" vars: tool: mongodump - func: "fetch tool" vars: tool: mongostat - func: "fetch tool" vars: tool: mongorestore - func: "fetch tool" vars: tool: mongooplog - func: "fetch tool" vars: tool: mongofiles - command: shell.exec type: test params: working_dir: src script: | mv ./mongodb/mongod${extension} . mv ./mongodb/mongo${extension} . mv test/legacy26/* . chmod +x mongo* python buildscripts/smoke.py --nopreallocj --with-cleanbb --mongod ./mongod --mongo ./mongo --report-file report.json --continue-on-failure --buildlogger-builder MCI_${build_variant} --buildlogger-buildnum ${builder_num|} --buildlogger-credentials ./mci.buildlogger --buildlogger-phase ${task_name}_${execution} tool --buildlogger-url https://logkeeper.mongodb.org - name: legacy24 depends_on: - name: dist commands: - command: git.get_project params: directory: src - command: git.apply_patch params: directory: src - func: "get buildnumber" - func: "setup credentials" - func: "download mongod" vars: mongo_url: "${mongo24_url}" - func: "download & copy shell" vars: mongo_url: "${mongo26_url}" - func: "fetch tool" vars: tool: mongoimport - func: "fetch tool" vars: tool: mongoexport - func: "fetch tool" vars: tool: mongodump - func: "fetch tool" vars: tool: mongostat - func: "fetch tool" vars: tool: mongorestore - func: "fetch tool" vars: tool: mongooplog - func: "fetch tool" vars: tool: mongofiles - command: shell.exec type: test params: working_dir: src script: | mv ./mongodb/mongod${extension} . mv ./mongodb/mongo${extension} . mv test/legacy24/* . chmod +x mongo* python buildscripts/smoke.py --nopreallocj --with-cleanbb --mongod ./mongod --mongo ./mongo --report-file report.json --continue-on-failure --buildlogger-builder MCI_${build_variant} --buildlogger-buildnum ${builder_num|} --buildlogger-credentials ./mci.buildlogger --buildlogger-phase ${task_name}_${execution} tool --buildlogger-url https://logkeeper.mongodb.org - name: lint commands: - command: git.get_project params: directory: src - command: git.apply_patch params: directory: src - command: shell.exec type: test params: working_dir: src script: | set -o errexit set -o verbose retVal=$(. ./set_gopath.sh && go run vendor/src/github.com/3rf/mongo-lint/golint/golint.go mongo* bson* common/*); if [ "$retVal" = "" ]; then exit 0; else echo $retVal; exit 1; fi; - name: log commands: - command: git.get_project params: directory: src - command: git.apply_patch params: directory: src - command: expansions.update params: updates: - key: "package" value: "common/log" - func: "run unit test" - name: progress commands: - command: git.get_project params: directory: src - command: git.apply_patch params: directory: src - command: expansions.update params: updates: - key: "package" value: "common/progress" - func: "run unit test" - name: qa-tests-unstable depends_on: - name: dist commands: - command: git.get_project params: directory: src - command: git.apply_patch params: directory: src - func: "get buildnumber" - func: "setup credentials" - func: "download mongod" vars: mongo_url: "${mongo_url}" - func: "fetch tool" vars: tool: mongoimport - func: "fetch tool" vars: tool: mongoexport - func: "fetch tool" vars: tool: mongodump - func: "fetch tool" vars: tool: mongorestore - func: "fetch tool" vars: tool: mongostat - func: "fetch tool" vars: tool: mongotop - func: "fetch tool" vars: tool: mongooplog - func: "fetch tool" vars: tool: mongofiles - func: "fetch tool" vars: tool: bsondump - command: shell.exec type: test params: working_dir: src script: | set -e mv ./mongodb/mongod${extension} . mv ./mongodb/mongo${extension} . mv ./mongodb/mongos${extension} . mv test/qa-tests/* . chmod +x mongo* chmod +x bsondump* rm -rf /data/install /data/multiversion python buildscripts/setup_multiversion_mongodb.py /data/install /data/multiversion ${arch} "2.6" "2.4" --latest ${smoke_use_ssl} --os="${mongo_os}" || true chmod 400 jstests/libs/key* PATH=$PATH:/data/multiversion python buildscripts/smoke.py ${smoke_use_ssl} --nopreallocj --with-cleanbb --mongod ./mongod --mongo ./mongo --report-file report.json --continue-on-failure --buildlogger-builder MCI_${build_variant} --buildlogger-buildnum ${builder_num|} --buildlogger-credentials ./mci.buildlogger --buildlogger-phase ${task_name}_${execution} --authMechanism SCRAM-SHA-1 top oplog dump files restore stat bson export import ssl unstable --buildlogger-url https://logkeeper.mongodb.org - name: qa-tests-15k depends_on: - name: dist commands: - command: git.get_project params: directory: src - command: git.apply_patch params: directory: src - func: "get buildnumber" - func: "setup credentials" - func: "download mongod" vars: mongo_url: "${mongo32_url}" - func: "fetch tool" vars: tool: mongodump - func: "fetch tool" vars: tool: mongorestore - command: shell.exec type: test params: working_dir: src script: | set -e mv ./mongodb/mongod${extension} . mv ./mongodb/mongo${extension} . mv test/qa-tests/* . chmod +x mongo* chmod 400 jstests/libs/key* python buildscripts/smoke.py ${smoke_use_ssl} --nopreallocj --with-cleanbb --mongod ./mongod --mongo ./mongo --report-file report.json --continue-on-failure --buildlogger-builder MCI_${build_variant} --buildlogger-buildnum ${builder_num|} --buildlogger-credentials ./mci.buildlogger --buildlogger-phase ${task_name}_${execution} --authMechanism SCRAM-SHA-1 restore-15k --buildlogger-url https://logkeeper.mongodb.org - name: qa-tests depends_on: - name: dist commands: - command: git.get_project params: directory: src - command: git.apply_patch params: directory: src - func: "get buildnumber" - func: "setup credentials" - func: "download mongod" vars: mongo_url: "${mongo32_url}" - func: "fetch tool" vars: tool: mongoimport - func: "fetch tool" vars: tool: mongoexport - func: "fetch tool" vars: tool: mongodump - func: "fetch tool" vars: tool: mongorestore - func: "fetch tool" vars: tool: mongostat - func: "fetch tool" vars: tool: mongotop - func: "fetch tool" vars: tool: mongooplog - func: "fetch tool" vars: tool: mongofiles - func: "fetch tool" vars: tool: bsondump - command: shell.exec type: test params: working_dir: src script: | set -e mv ./mongodb/mongod${extension} . mv ./mongodb/mongo${extension} . mv ./mongodb/mongos${extension} . mv test/qa-tests/* . chmod +x mongo* chmod +x bsondump* rm -rf /data/install /data/multiversion python buildscripts/setup_multiversion_mongodb.py /data/install /data/multiversion ${arch} "2.6" "2.4" --latest ${smoke_use_ssl} --os="${mongo_os}" || true chmod 400 jstests/libs/key* PATH=$PATH:/data/multiversion python buildscripts/smoke.py ${smoke_use_ssl} --nopreallocj --with-cleanbb --mongod ./mongod --mongo ./mongo --report-file report.json --continue-on-failure --buildlogger-builder MCI_${build_variant} --buildlogger-buildnum ${builder_num|} --buildlogger-credentials ./mci.buildlogger --buildlogger-phase ${task_name}_${execution} --authMechanism SCRAM-SHA-1 top oplog dump files restore stat bson export import ssl --buildlogger-url https://logkeeper.mongodb.org - name: qa-dump-restore-archiving depends_on: - name: dist commands: - command: git.get_project params: directory: src - command: git.apply_patch params: directory: src - func: "get buildnumber" - func: "setup credentials" - func: "download mongod" vars: mongo_url: "${mongo32_url}" - func: "fetch tool" vars: tool: mongodump - func: "fetch tool" vars: tool: mongorestore - command: shell.exec type: test params: working_dir: src script: | set -e mv ./mongodb/mongod${extension} . mv ./mongodb/mongo${extension} . mv ./mongodb/mongos${extension} . mv test/qa-tests/* . chmod +x mongo* rm -rf /data/install /data/multiversion python buildscripts/setup_multiversion_mongodb.py /data/install /data/multiversion ${arch} "2.6" "2.4" --latest ${smoke_use_ssl} --os="${mongo_os}" || true chmod 400 jstests/libs/key* SMOKE_EVAL="load('jstests/configs/archive_targets.js')" PATH=$PATH:/data/multiversion python buildscripts/smoke.py ${smoke_use_ssl} --nopreallocj --with-cleanbb --mongod ./mongod --mongo ./mongo --report-file report.json --continue-on-failure --buildlogger-builder MCI_${build_variant} --buildlogger-buildnum ${builder_num|} --buildlogger-credentials ./mci.buildlogger --buildlogger-phase ${task_name}_${execution} --authMechanism SCRAM-SHA-1 dump restore --buildlogger-url https://logkeeper.mongodb.org - name: qa-dump-restore-gzip depends_on: - name: dist commands: - command: git.get_project params: directory: src - command: git.apply_patch params: directory: src - func: "get buildnumber" - func: "setup credentials" - func: "download mongod" vars: mongo_url: "${mongo32_url}" - func: "fetch tool" vars: tool: mongodump - func: "fetch tool" vars: tool: mongorestore - command: shell.exec type: test params: working_dir: src script: | set -e mv ./mongodb/mongod${extension} . mv ./mongodb/mongo${extension} . mv ./mongodb/mongos${extension} . mv test/qa-tests/* . chmod +x mongo* rm -rf /data/install /data/multiversion python buildscripts/setup_multiversion_mongodb.py /data/install /data/multiversion ${arch} "2.6" "2.4" --latest ${smoke_use_ssl} --os="${mongo_os}" || true chmod 400 jstests/libs/key* SMOKE_EVAL="load('jstests/configs/gzip_targets.js')" PATH=$PATH:/data/multiversion python buildscripts/smoke.py ${smoke_use_ssl} --nopreallocj --with-cleanbb --mongod ./mongod --mongo ./mongo --report-file report.json --continue-on-failure --buildlogger-builder MCI_${build_variant} --buildlogger-buildnum ${builder_num|} --buildlogger-credentials ./mci.buildlogger --buildlogger-phase ${task_name}_${execution} --authMechanism SCRAM-SHA-1 dump restore --buildlogger-url https://logkeeper.mongodb.org - name: qa-tests-wt depends_on: - name: dist commands: - command: git.get_project params: directory: src - command: git.apply_patch params: directory: src - func: "get buildnumber" - func: "setup credentials" - func: "download mongod" vars: mongo_url: "${mongo32_url}" - func: "fetch tool" vars: tool: mongoimport - func: "fetch tool" vars: tool: mongoexport - func: "fetch tool" vars: tool: mongodump - func: "fetch tool" vars: tool: mongorestore - func: "fetch tool" vars: tool: mongostat - func: "fetch tool" vars: tool: mongotop - func: "fetch tool" vars: tool: mongooplog - func: "fetch tool" vars: tool: mongofiles - func: "fetch tool" vars: tool: bsondump - command: shell.exec type: test params: working_dir: src script: | mv ./mongodb/mongod${extension} . mv ./mongodb/mongo${extension} . mv ./mongodb/mongos${extension} . mv test/qa-tests/* . chmod +x mongo* chmod +x bsondump* rm -rf /data/install /data/multiversion python buildscripts/setup_multiversion_mongodb.py /data/install /data/multiversion ${arch} "2.6" "2.4" --latest ${smoke_use_ssl} --os="${mongo_os}" || true chmod 400 jstests/libs/key* PATH=$PATH:/data/multiversion python buildscripts/smoke.py ${smoke_use_ssl} --nopreallocj --with-cleanbb --mongod ./mongod --mongo ./mongo --report-file report.json --continue-on-failure --buildlogger-builder MCI_${build_variant} --buildlogger-buildnum ${builder_num|} --buildlogger-credentials ./mci.buildlogger --buildlogger-phase ${task_name}_${execution} --authMechanism SCRAM-SHA-1 --storageEngine=wiredTiger top dump oplog files restore stat bson export import --buildlogger-url https://logkeeper.mongodb.org - name: text commands: - command: git.get_project params: directory: src - command: git.apply_patch params: directory: src - command: expansions.update params: updates: - key: "package" value: "common/text" - func: "run unit test" - name: unit commands: - command: expansions.update params: updates: - key: "run_coverage" value: "true" - command: git.get_project params: directory: src - command: git.apply_patch params: directory: src - func: "run tool unit tests" - func: "create coverage reports" - command: expansions.update params: updates: - key: "coverage_pkg" value: "mongoimport" - func: "upload html coverage" - func: "upload text coverage" - command: expansions.update params: updates: - key: "coverage_pkg" value: "mongoexport" - func: "upload html coverage" - func: "upload text coverage" - command: expansions.update params: updates: - key: "coverage_pkg" value: "mongostat" - func: "upload html coverage" - func: "upload text coverage" - command: expansions.update params: updates: - key: "coverage_pkg" value: "mongooplog" - func: "upload html coverage" - func: "upload text coverage" - command: expansions.update params: updates: - key: "coverage_pkg" value: "mongodump" - func: "upload html coverage" - func: "upload text coverage" - command: expansions.update params: updates: - key: "coverage_pkg" value: "mongorestore" - func: "upload html coverage" - func: "upload text coverage" - name: util commands: - command: git.get_project params: directory: src - command: git.apply_patch params: directory: src - command: expansions.update params: updates: - key: "package" value: "common/util" - func: "run unit test" - name: vet commands: - command: git.get_project params: directory: src - command: git.apply_patch params: directory: src - command: shell.exec type: test params: working_dir: src script: | set -o verbose . ./set_gopath.sh go tool vet . buildvariants: ####################################### # OSX Buildvariant # ####################################### - name: osx-1010 display_name: OSX 10.10 64-bit run_on: - osx-1010 expansions: <<: *mongo_url_osx1010 <<: *mongod_default_startup_args <<: *mongo_default_startup_args arch: "osx/x86_64" build_tags: -tags "ssl" tasks: *osx_1010_tasks - name: osx-1010-ssl display_name: OSX 10.10 64-bit SSL run_on: - osx-1010 expansions: <<: *mongo_url_osx1010 <<: *mongod_ssl_startup_args <<: *mongo_ssl_startup_args arch: "osx/x86_64" build_tags: -tags "ssl" edition: ssl tasks: *osx_1010_ssl_tasks ####################################### # Ubuntu Buildvariants # ####################################### - name: ubuntu display_name: Linux 64-bit run_on: - ubuntu1204-test expansions: <<: *mongo_url_ubuntu1204 <<: *mongod_default_startup_args <<: *mongo_default_startup_args build_tags: -tags "ssl" arch: "linux/x86_64" integration_test_args: integration tasks: *ubuntu1204_tasks - name: ubuntu-ssl display_name: Linux 64-bit SSL run_on: - ubuntu1204-test expansions: <<: *mongo_url_ubuntu1204_enterprise <<: *mongod_ssl_startup_args <<: *mongo_ssl_startup_args build_tags: -tags "ssl" edition: ssl arch: "linux/x86_64" smoke_use_ssl: --use-ssl integration_test_args: "integration,ssl" tasks: *ubuntu1204_ssl_tasks - name: ubuntu-enterprise display_name: Linux 64-bit Enterprise run_on: - ubuntu1204-test expansions: <<: *mongo_url_ubuntu1204_enterprise <<: *mongod_default_startup_args <<: *mongo_default_startup_args build_tags: -tags "ssl sasl" smoke_use_ssl: --use-ssl arch: "linux/x86_64" edition: enterprise run_kinit: true integration_test_args: integration tasks: *ubuntu1204_enterprise_tasks ####################################### # Solaris Buildvariant # ####################################### - name: solaris display_name: Solaris 64-bit run_on: - solaris expansions: <<: *mongo_url_solaris <<: *mongod_default_startup_args <<: *mongo_default_startup_args library_path: PATH="/opt/mongodbtoolchain/v2/bin/:$PATH" build_tags: -gccgoflags "-lsocket -lnsl" tasks: *solaris_tasks ####################################### # Windows Buildvariants # ####################################### - name: windows-32 display_name: Windows 32-bit run_on: - windows-32 expansions: <<: *mongo_url_windows32 <<: *mongod_win32_startup_args <<: *mongo_default_startup_args extension: .exe preproc_gpm: "perl -pi -e 's/\\r\\n/\\n/g' " integration_test_args: "integration" tasks: *windows_32_tasks - name: windows-64 display_name: Windows 64-bit run_on: - windows-64-vs2013-test expansions: <<: *mongo_url_windows64 <<: *mongod_default_startup_args <<: *mongo_default_startup_args extension: .exe arch: "win32/x86_64" preproc_gpm: "perl -pi -e 's/\\r\\n/\\n/g' " integration_test_args: "integration" tasks: *windows_64_tasks - name: windows-64-ssl display_name: Windows 64-bit SSL run_on: - windows-64-vs2013-compile expansions: <<: *mongo_url_windows64_enterprise <<: *mongod_ssl_startup_args <<: *mongo_ssl_startup_args build_tags: -tags "ssl" edition: ssl smoke_use_ssl: --use-ssl extension: .exe arch: "win32/x86_64" library_path: PATH="/cygdrive/c/mingw-w64/x86_64-4.9.1-posix-seh-rt_v3-rev1/mingw64/bin:/cygdrive/c/sasl/:$PATH" preproc_gpm: "perl -pi -e 's/\\r\\n/\\n/g' " integration_test_args: "integration,ssl" tasks: *windows_64_ssl_tasks - name: windows-64-enterprise display_name: Windows 64-bit Enterprise run_on: - windows-64-vs2013-compile expansions: <<: *mongo_url_windows64_enterprise <<: *mongod_default_startup_args <<: *mongo_default_startup_args build_tags: -tags "sasl ssl" smoke_use_ssl: --use-ssl edition: enterprise extension: .exe arch: "win32/x86_64" library_path: PATH="/cygdrive/c/mingw-w64/x86_64-4.9.1-posix-seh-rt_v3-rev1/mingw64/bin:/cygdrive/c/sasl/:$PATH" preproc_gpm: "perl -pi -e 's/\\r\\n/\\n/g' " integration_test_args: "integration" tasks: *windows_64_enterprise_tasks ####################################### # Experimental Buildvariants # ####################################### - name: ubuntu-race stepback: false batchtime: 1440 # daily display_name: z Race Detector Linux 64-bit run_on: - ubuntu1204-test expansions: <<: *mongo_url_ubuntu1204 <<: *mongod_default_startup_args <<: *mongo_default_startup_args build_tags: -tags "ssl" arch: "linux/x86_64" args: "-race" integration_test_args: integration tasks: *ubuntu1204_race_tasks ####################################### # Dist only Buildvariants # ####################################### - name: suse11 display_name: SUSE 11 SSL run_on: - suse11-test expansions: <<: *mongo_url_ubuntu1204 build_tags: -tags "sasl ssl" arch: "linux/x86_64" tasks: - name: dist - name: suse12 display_name: SUSE 12 SSL run_on: - suse12-test expansions: <<: *mongo_url_ubuntu1204 build_tags: -tags "sasl ssl" arch: "linux/x86_64" tasks: - name: dist - name: rhel55 display_name: RHEL 5.5 SSL run_on: - rhel55-test expansions: <<: *mongo_url_ubuntu1204 gorootvars: PATH=/opt/mongodbtoolchain/v2/bin:$PATH build_tags: -gccgoflags "$(pkg-config --libs --cflags libssl)" -tags "sasl ssl" arch: "linux/x86_64" tasks: - name: dist - name: rhel62 display_name: RHEL 6.2 SSL run_on: - rhel62-test expansions: <<: *mongo_url_ubuntu1204 gorootvars: GOROOT=/opt/go PATH="/opt/go/bin:$PATH" build_tags: -tags "sasl ssl" arch: "linux/x86_64" tasks: - name: dist - name: rhel70 display_name: RHEL 7.0 SSL run_on: - rhel70 expansions: <<: *mongo_url_ubuntu1204 gorootvars: GOROOT=/opt/go PATH="/opt/go/bin:$PATH" build_tags: -tags "sasl ssl" arch: "linux/x86_64" tasks: - name: dist - name: ubuntu1404 display_name: Ubuntu 14.04 SSL run_on: - ubuntu1404-test expansions: <<: *mongo_url_ubuntu1204 build_tags: -tags "sasl ssl" arch: "linux/x86_64" tasks: - name: dist - name: debian71 display_name: Debian 7.1 SSL run_on: - debian71-test expansions: <<: *mongo_url_ubuntu1204 gorootvars: GOROOT=/opt/go PATH="/opt/go/bin:$PATH" build_tags: -tags "sasl ssl" arch: "linux/x86_64" tasks: - name: dist mongo-tools-r3.2.11/common/000077500000000000000000000000001276133741700155405ustar00rootroot00000000000000mongo-tools-r3.2.11/common/archive/000077500000000000000000000000001276133741700171615ustar00rootroot00000000000000mongo-tools-r3.2.11/common/archive/archive.go000066400000000000000000000040311276133741700211270ustar00rootroot00000000000000package archive import ( "io" ) // NamespaceHeader is a data structure that, as BSON, is found in archives where it indicates // that either the subsequent stream of BSON belongs to this new namespace, or that the // indicated namespace will have no more documents (EOF) type NamespaceHeader struct { Database string `bson:"db"` Collection string `bson:"collection"` EOF bool `bson:"EOF",omitempty` CRC int64 `bson:"CRC",omitempty` } // CollectionMetadata is a data structure that, as BSON, is found in the prelude of the archive. // There is one CollectionMetadata per collection that will be in the archive. type CollectionMetadata struct { Database string `bson:"db"` Collection string `bson:"collection"` Metadata string `bson:"metadata"` Size int `bson:"size"` } // Header is a data structure that, as BSON, is found immediately after the magic // number in the archive, before any CollectionMetadatas. It is the home of any archive level information type Header struct { ConcurrentCollections int32 `bson:"concurrent_collections",omitempty` FormatVersion string `bson:"version"` ServerVersion string `bson:"server_version"` ToolVersion string `bson:"tool_version"` } const minBSONSize = 4 + 1 // an empty BSON document should be exactly five bytes long var terminator int32 = -1 var terminatorBytes = []byte{0xFF, 0xFF, 0xFF, 0xFF} // TODO, rectify this with terminator // MagicNumber is four bytes that are found at the beginning of the archive that indicate that // the byte stream is an archive, as opposed to anything else, including a stream of BSON documents const MagicNumber uint32 = 0x8199e26d const archiveFormatVersion = "0.1" // Writer is the top level object to contain information about archives in mongodump type Writer struct { Out io.WriteCloser Prelude *Prelude Mux *Multiplexer } // Reader is the top level object to contain information about archives in mongorestore type Reader struct { In io.ReadCloser Demux *Demultiplexer Prelude *Prelude } mongo-tools-r3.2.11/common/archive/demultiplexer.go000066400000000000000000000330631276133741700224000ustar00rootroot00000000000000package archive import ( "bytes" "fmt" "github.com/mongodb/mongo-tools/common/db" "github.com/mongodb/mongo-tools/common/intents" "github.com/mongodb/mongo-tools/common/log" "gopkg.in/mgo.v2/bson" "hash" "hash/crc64" "io" "sync/atomic" ) // DemuxOut is a Demultiplexer output consumer // The Write() and Close() occur in the same thread as the Demultiplexer runs in. type DemuxOut interface { Write([]byte) (int, error) Close() error } // Demultiplexer implements Parser. type Demultiplexer struct { In io.Reader //TODO wrap up these three into a structure outs map[string]DemuxOut hashes map[string]hash.Hash64 lengths map[string]int64 currentNamespace string buf [db.MaxBSONSize]byte NamespaceChan chan string NamespaceErrorChan chan error } // Run creates and runs a parser with the Demultiplexer as a consumer func (demux *Demultiplexer) Run() error { parser := Parser{In: demux.In} err := parser.ReadAllBlocks(demux) if len(demux.outs) > 0 { log.Logf(log.Always, "demux finishing when there are still outs (%v)", len(demux.outs)) } log.Logf(log.DebugLow, "demux finishing (err:%v)", err) return err } type demuxError struct { Err error Msg string } // Error is part of the Error interface. It formats a demuxError for human readability. func (pe *demuxError) Error() string { err := fmt.Sprintf("error demultiplexing archive; %v", pe.Msg) if pe.Err != nil { err = fmt.Sprintf("%v ( %v )", err, pe.Err) } return err } // newError creates a demuxError with just a message func newError(msg string) error { return &demuxError{ Msg: msg, } } // newWrappedError creates a demuxError with a message as well as an underlying cause error func newWrappedError(msg string, err error) error { return &demuxError{ Err: err, Msg: msg, } } // HeaderBSON is part of the ParserConsumer interface and receives headers from parser. // Its main role is to implement opens and EOFs of the embedded stream. func (demux *Demultiplexer) HeaderBSON(buf []byte) error { colHeader := NamespaceHeader{} err := bson.Unmarshal(buf, &colHeader) if err != nil { return newWrappedError("header bson doesn't unmarshal as a collection header", err) } log.Logf(log.DebugHigh, "demux namespaceHeader: %v", colHeader) if colHeader.Collection == "" { return newError("collection header is missing a Collection") } demux.currentNamespace = colHeader.Database + "." + colHeader.Collection if _, ok := demux.outs[demux.currentNamespace]; !ok { if demux.NamespaceChan != nil { demux.NamespaceChan <- demux.currentNamespace err := <-demux.NamespaceErrorChan if err == io.EOF { // if the Prioritizer sends us back an io.EOF then it's telling us that // it's finishing and doesn't need any more namespace announcements. close(demux.NamespaceChan) demux.NamespaceChan = nil return nil } if err != nil { return newWrappedError("failed arranging a consumer for new namespace", err) } } } if colHeader.EOF { crc := int64(demux.hashes[demux.currentNamespace].Sum64()) length := int64(demux.lengths[demux.currentNamespace]) if crc != colHeader.CRC { return fmt.Errorf("CRC mismatch for namespace %v, %v!=%v", demux.currentNamespace, crc, colHeader.CRC, ) } log.Logf(log.DebugHigh, "demux checksum for namespace %v is correct (%v), %v bytes", demux.currentNamespace, crc, length) demux.outs[demux.currentNamespace].Close() delete(demux.outs, demux.currentNamespace) delete(demux.hashes, demux.currentNamespace) delete(demux.lengths, demux.currentNamespace) // in case we get a BSONBody with this block, // we want to ensure that that causes an error demux.currentNamespace = "" } return nil } // End is part of the ParserConsumer interface and receives the end of archive notification. func (demux *Demultiplexer) End() error { log.Logf(log.DebugHigh, "demux End") if len(demux.outs) != 0 { openNss := []string{} for ns := range demux.outs { openNss = append(openNss, ns) } return newError(fmt.Sprintf("archive finished but contained files were unfinished (%v)", openNss)) } if demux.NamespaceChan != nil { close(demux.NamespaceChan) } return nil } // BodyBSON is part of the ParserConsumer interface and receives BSON bodies from the parser. // Its main role is to dispatch the body to the Read() function of the current DemuxOut. func (demux *Demultiplexer) BodyBSON(buf []byte) error { if demux.currentNamespace == "" { return newError("collection data without a collection header") } hash, ok := demux.hashes[demux.currentNamespace] if !ok { return newError("no checksum for current namespace " + demux.currentNamespace) } hash.Write(buf) demux.lengths[demux.currentNamespace] += int64(len(buf)) out, ok := demux.outs[demux.currentNamespace] if !ok { return newError("no demux consumer currently consuming namespace " + demux.currentNamespace) } _, err := out.Write(buf) return err } // Open installs the DemuxOut as the handler for data for the namespace ns func (demux *Demultiplexer) Open(ns string, out DemuxOut) { // In the current implementation where this is either called before the demultiplexing is running // or while the demutiplexer is inside of the NamespaceChan NamespaceErrorChan conversation // I think that we don't need to lock outs, but I suspect that if the implementation changes // we may need to lock when outs is accessed log.Logf(log.DebugHigh, "demux Open") if demux.outs == nil { demux.outs = make(map[string]DemuxOut) demux.hashes = make(map[string]hash.Hash64) demux.lengths = make(map[string]int64) } demux.outs[ns] = out demux.hashes[ns] = crc64.New(crc64.MakeTable(crc64.ECMA)) demux.lengths[ns] = 0 } // RegularCollectionReceiver implements the intents.file interface. // RegularCollectionReceivers get paired with RegularCollectionSenders. type RegularCollectionReceiver struct { readLenChan <-chan int readBufChan chan<- []byte Intent *intents.Intent Demux *Demultiplexer partialReadArray []byte partialReadBuf []byte isOpen bool pos int64 } func (receiver *RegularCollectionReceiver) Read(r []byte) (int, error) { if receiver.partialReadBuf != nil && len(receiver.partialReadBuf) > 0 { wLen := len(receiver.partialReadBuf) copyLen := copy(r, receiver.partialReadBuf) if wLen == copyLen { receiver.partialReadBuf = nil } else { receiver.partialReadBuf = receiver.partialReadBuf[copyLen:] } atomic.AddInt64(&receiver.pos, int64(copyLen)) return copyLen, nil } // Since we're the "reader" here, not the "writer" we need to start with a read, in case the chan is closed wLen, ok := <-receiver.readLenChan if !ok { close(receiver.readBufChan) return 0, io.EOF } if wLen > db.MaxBSONSize { return 0, fmt.Errorf("incomming buffer size is too big %v", wLen) } rLen := len(r) if wLen > rLen { // if the incomming write size is larger then the incomming read buffer then we need to accept // the write in a larger buffer, fill the read buffer, then cache the remainder receiver.partialReadBuf = receiver.partialReadArray[:wLen] receiver.readBufChan <- receiver.partialReadBuf writtenLength := <-receiver.readLenChan if wLen != writtenLength { return 0, fmt.Errorf("regularCollectionSender didn't send what it said it would") } copy(r, receiver.partialReadBuf) receiver.partialReadBuf = receiver.partialReadBuf[rLen:] atomic.AddInt64(&receiver.pos, int64(rLen)) return rLen, nil } // Send the read buff to the BodyBSON ParserConsumer to fill receiver.readBufChan <- r // Receiver the wLen of data written wLen = <-receiver.readLenChan atomic.AddInt64(&receiver.pos, int64(wLen)) return wLen, nil } func (receiver *RegularCollectionReceiver) Pos() int64 { return atomic.LoadInt64(&receiver.pos) } // Close is part of the intents.file interface. It currently does nothing. We can't close the // regularCollectionSender before the embedded stream reaches EOF. If this needs to be // implemented, then we need to swap out the regularCollectionSender with a null writer func (receiver *RegularCollectionReceiver) Close() error { return nil } // Open is part of the intents.file interface. It creates the chan's in the // RegularCollectionReceiver and adds the RegularCollectionReceiver to the set of // RegularCollectonReceivers in the demultiplexer func (receiver *RegularCollectionReceiver) Open() error { // TODO move this implementation to some non intents.file method, to be called from prioritizer.Get // So that we don't have to enable this double open stuff. // Currently the open needs to finish before the prioritizer.Get finishes, so we open the intents.file // in prioritizer.Get even though it's going to get opened again in DumpIntent. if receiver.isOpen { return nil } readLenChan := make(chan int) readBufChan := make(chan []byte) receiver.readLenChan = readLenChan receiver.readBufChan = readBufChan sender := ®ularCollectionSender{readLenChan: readLenChan, readBufChan: readBufChan} receiver.Demux.Open(receiver.Intent.Namespace(), sender) receiver.isOpen = true return nil } // Write is part of the intents.file interface. // It does nothing, and only exists so that RegularCollectionReceiver fulfills the interface func (receiver *RegularCollectionReceiver) Write([]byte) (int, error) { return 0, nil } func (receiver *RegularCollectionReceiver) TakeIOBuffer(ioBuf []byte) { receiver.partialReadArray = ioBuf } func (receiver *RegularCollectionReceiver) ReleaseIOBuffer() { receiver.partialReadArray = nil } // regularCollectionSender implements DemuxOut type regularCollectionSender struct { readLenChan chan<- int readBufChan <-chan []byte } // Write is part of the DemuxOut interface. func (sender *regularCollectionSender) Write(buf []byte) (int, error) { // As a writer, we need to write first, so that the reader can properly detect EOF // Additionally, the reader needs to know the write size, so that it can give us a // properly sized buffer. Sending the incomming buffersize fills both of these needs. sender.readLenChan <- len(buf) // Receive from the reader a buffer to put the bytes into readBuf := <-sender.readBufChan if len(readBuf) < len(buf) { return 0, fmt.Errorf("readbuf is not large enough for incoming BodyBSON (%v<%v)", len(readBuf), len(buf)) } copy(readBuf, buf) // Send back the length of the data copied in to the buffer sender.readLenChan <- len(buf) return len(buf), nil } // Close is part of the DemuxOut interface. It only closes the readLenChan, as that is what will // cause the RegularCollectionReceiver.Read() to receive EOF func (sender *regularCollectionSender) Close() error { close(sender.readLenChan) return nil } // SpecialCollectionCache implemnts both DemuxOut as well as intents.file type SpecialCollectionCache struct { Intent *intents.Intent Demux *Demultiplexer bytes.Buffer pos int64 } // Open is part of the both interfaces, and it does nothing func (cache *SpecialCollectionCache) Open() error { return nil } // Close is part of the both interfaces, and it does nothing func (cache *SpecialCollectionCache) Close() error { cache.Intent.Size = int64(cache.Buffer.Len()) return nil } func (cache *SpecialCollectionCache) Read(p []byte) (int, error) { n, err := cache.Buffer.Read(p) atomic.AddInt64(&cache.pos, int64(n)) return n, err } func (cache *SpecialCollectionCache) Pos() int64 { return atomic.LoadInt64(&cache.pos) } // MutedCollection implements both DemuxOut as well as intents.file. It serves as a way to // let the demutiplexer ignore certain embedded streams type MutedCollection struct { Intent *intents.Intent Demux *Demultiplexer } // Read is part of the intents.file interface, and does nothing func (*MutedCollection) Read([]byte) (int, error) { // Read is part of the intents.file interface, and does nothing return 0, io.EOF } // Write is part of the intents.file interface, and does nothing func (*MutedCollection) Write(b []byte) (int, error) { return len(b), nil } // Close is part of the intents.file interface, and does nothing func (*MutedCollection) Close() error { return nil } // Open is part of the intents.file interface, and does nothing func (*MutedCollection) Open() error { return nil } //===== Archive Manager Prioritizer ===== // NewPrioritizer careates a new Prioritizer and hooks up its Namespace channels to the ones in demux func (demux *Demultiplexer) NewPrioritizer(mgr *intents.Manager) *Prioritizer { return &Prioritizer{ NamespaceChan: demux.NamespaceChan, NamespaceErrorChan: demux.NamespaceErrorChan, mgr: mgr, } } // Prioritizer is a completely reactive prioritizer // Intents are handed out as they arrive in the archive type Prioritizer struct { NamespaceChan <-chan string NamespaceErrorChan chan<- error mgr *intents.Manager } // Get waits for a new namespace from the NamespaceChan, and returns a Intent found for it func (prioritizer *Prioritizer) Get() *intents.Intent { namespace, ok := <-prioritizer.NamespaceChan if !ok { return nil } intent := prioritizer.mgr.IntentForNamespace(namespace) if intent == nil { prioritizer.NamespaceErrorChan <- fmt.Errorf("no intent for namespace %v", namespace) } else { if intent.BSONFile != nil { intent.BSONFile.Open() } if intent.IsOplog() { // once we see the oplog we // cause the RestoreIntents to finish because we don't // want RestoreIntents to restore the oplog prioritizer.NamespaceErrorChan <- io.EOF return nil } prioritizer.NamespaceErrorChan <- nil } return intent } // Finish is part of the IntentPrioritizer interface, and does nothing func (prioritizer *Prioritizer) Finish(*intents.Intent) { // no-op return } mongo-tools-r3.2.11/common/archive/multiplexer.go000066400000000000000000000216401276133741700220650ustar00rootroot00000000000000package archive import ( "fmt" "github.com/mongodb/mongo-tools/common/db" "github.com/mongodb/mongo-tools/common/intents" "github.com/mongodb/mongo-tools/common/log" "gopkg.in/mgo.v2/bson" "hash" "hash/crc64" "io" "reflect" ) // bufferSize enables or disables the MuxIn buffering // TODO: remove this constant and the non-buffered MuxIn implementations const bufferWrites = true const bufferSize = db.MaxBSONSize // Multiplexer is what one uses to create interleaved intents in an archive type Multiplexer struct { Out io.WriteCloser Control chan *MuxIn Completed chan error // shutdownInputs allows the mux to tell the intent dumping worker // go routines to shutdown, so that we can shutdown shutdownInputs notifier // ins and selectCases are correlating slices ins []*MuxIn selectCases []reflect.SelectCase currentNamespace string } type notifier interface { Notify() } // NewMultiplexer creates a Multiplexer and populates its Control/Completed chans // it takes a WriteCloser, which is where in imputs will get multiplexed on to, // and it takes a notifier, which should allow the multiplexer to ask for the shutdown // of the inputs. func NewMultiplexer(out io.WriteCloser, shutdownInputs notifier) *Multiplexer { mux := &Multiplexer{ Out: out, Control: make(chan *MuxIn), Completed: make(chan error), shutdownInputs: shutdownInputs, ins: []*MuxIn{ nil, // There is no MuxIn for the Control case }, } mux.selectCases = []reflect.SelectCase{ reflect.SelectCase{ Dir: reflect.SelectRecv, Chan: reflect.ValueOf(mux.Control), Send: reflect.Value{}, }, } return mux } // Run multiplexes until it receives an EOF on its Control chan. func (mux *Multiplexer) Run() { var err, completionErr error for { index, value, notEOF := reflect.Select(mux.selectCases) EOF := !notEOF if index == 0 { //Control index if EOF { log.Logf(log.DebugLow, "Mux finish") mux.Out.Close() if completionErr != nil { mux.Completed <- completionErr } else if len(mux.selectCases) != 1 { mux.Completed <- fmt.Errorf("Mux ending but selectCases still open %v", len(mux.selectCases)) } else { mux.Completed <- nil } return } muxIn, ok := value.Interface().(*MuxIn) if !ok { mux.Completed <- fmt.Errorf("non MuxIn received on Control chan") // one for the MuxIn.Open return } log.Logf(log.DebugLow, "Mux open namespace %v", muxIn.Intent.Namespace()) mux.selectCases = append(mux.selectCases, reflect.SelectCase{ Dir: reflect.SelectRecv, Chan: reflect.ValueOf(muxIn.writeChan), Send: reflect.Value{}, }) mux.ins = append(mux.ins, muxIn) } else { if EOF { // We need to let the MuxIn know that we've // noticed this close. This fixes a race where // the intent processing threads finish, then the main // thread closes the mux's control chan and the mux // processes the close on the control chan before it processes // the close on the MuxIn chan mux.ins[index].writeCloseFinishedChan <- struct{}{} err = mux.formatEOF(index, mux.ins[index]) if err != nil { mux.shutdownInputs.Notify() mux.Out = &nopCloseNopWriter{} completionErr = err } log.Logf(log.DebugLow, "Mux close namespace %v", mux.ins[index].Intent.Namespace()) mux.currentNamespace = "" mux.selectCases = append(mux.selectCases[:index], mux.selectCases[index+1:]...) mux.ins = append(mux.ins[:index], mux.ins[index+1:]...) } else { bsonBytes, ok := value.Interface().([]byte) if !ok { mux.Completed <- fmt.Errorf("multiplexer received a value that wasn't a []byte") return } err = mux.formatBody(mux.ins[index], bsonBytes) if err != nil { mux.shutdownInputs.Notify() mux.Out = &nopCloseNopWriter{} completionErr = err } } } } } type nopCloseNopWriter struct{} func (*nopCloseNopWriter) Close() error { return nil } func (*nopCloseNopWriter) Write(p []byte) (int, error) { return len(p), nil } // formatBody writes the BSON in to the archive, potentially writing a new header // if the document belongs to a different namespace from the last header. func (mux *Multiplexer) formatBody(in *MuxIn, bsonBytes []byte) error { var err error var length int defer func() { in.writeLenChan <- length }() if in.Intent.Namespace() != mux.currentNamespace { // Handle the change of which DB/Collection we're writing docs for // If mux.currentNamespace then we need to terminate the current block if mux.currentNamespace != "" { l, err := mux.Out.Write(terminatorBytes) if err != nil { return err } if l != len(terminatorBytes) { return io.ErrShortWrite } } header, err := bson.Marshal(NamespaceHeader{ Database: in.Intent.DB, Collection: in.Intent.C, }) if err != nil { return err } l, err := mux.Out.Write(header) if err != nil { return err } if l != len(header) { return io.ErrShortWrite } } mux.currentNamespace = in.Intent.Namespace() length, err = mux.Out.Write(bsonBytes) if err != nil { return err } return nil } // formatEOF writes the EOF header in to the archive func (mux *Multiplexer) formatEOF(index int, in *MuxIn) error { var err error if mux.currentNamespace != "" { l, err := mux.Out.Write(terminatorBytes) if err != nil { return err } if l != len(terminatorBytes) { return io.ErrShortWrite } } eofHeader, err := bson.Marshal(NamespaceHeader{ Database: in.Intent.DB, Collection: in.Intent.C, EOF: true, CRC: int64(in.hash.Sum64()), }) if err != nil { return err } l, err := mux.Out.Write(eofHeader) if err != nil { return err } if l != len(eofHeader) { return io.ErrShortWrite } l, err = mux.Out.Write(terminatorBytes) if err != nil { return err } if l != len(terminatorBytes) { return io.ErrShortWrite } return nil } // MuxIn is an implementation of the intents.file interface. // They live in the intents, and are potentially owned by different threads than // the thread owning the Multiplexer. // They are out the intents write data to the multiplexer type MuxIn struct { writeChan chan []byte writeLenChan chan int writeCloseFinishedChan chan struct{} buf []byte hash hash.Hash64 Intent *intents.Intent Mux *Multiplexer } // Read does nothing for MuxIns func (muxIn *MuxIn) Read([]byte) (int, error) { return 0, nil } func (muxIn *MuxIn) Pos() int64 { return 0 } // Close closes the chans in the MuxIn. // Ultimately the multiplexer will detect that they are closed and cause a // formatEOF to occur. func (muxIn *MuxIn) Close() error { // the mux side of this gets closed in the mux when it gets an eof on the read log.Logf(log.DebugHigh, "MuxIn close %v", muxIn.Intent.Namespace()) if bufferWrites { muxIn.writeChan <- muxIn.buf length := <-muxIn.writeLenChan if length != len(muxIn.buf) { return io.ErrShortWrite } muxIn.buf = nil } close(muxIn.writeChan) close(muxIn.writeLenChan) // We need to wait for the close on the writeChan to be processed before proceeding // Otherwise we might assume that all work is finished and exit the program before // the mux finishes writing the end of the archive <-muxIn.writeCloseFinishedChan return nil } // Open is implemented in Mux.open, but in short, it creates chans and a select case // and adds the SelectCase and the MuxIn in to the Multiplexer. func (muxIn *MuxIn) Open() error { log.Logf(log.DebugHigh, "MuxIn open %v", muxIn.Intent.Namespace()) muxIn.writeChan = make(chan []byte) muxIn.writeLenChan = make(chan int) muxIn.writeCloseFinishedChan = make(chan struct{}) muxIn.buf = make([]byte, 0, bufferSize) muxIn.hash = crc64.New(crc64.MakeTable(crc64.ECMA)) if bufferWrites { muxIn.buf = make([]byte, 0, db.MaxBSONSize) } muxIn.Mux.Control <- muxIn return nil } // Write hands a buffer to the Multiplexer and receives a written length from the multiplexer // after the length is received, the buffer is free to be reused. func (muxIn *MuxIn) Write(buf []byte) (int, error) { size := int( (uint32(buf[0]) << 0) | (uint32(buf[1]) << 8) | (uint32(buf[2]) << 16) | (uint32(buf[3]) << 24), ) // TODO remove these checks, they're for debugging if len(buf) < size { panic(fmt.Errorf("corrupt bson in MuxIn.Write (size %v/%v)", size, len(buf))) } if buf[size-1] != 0 { panic(fmt.Errorf("corrupt bson in MuxIn.Write bson has no-zero terminator %v, (size %v/%v)", buf[size-1], size, len(buf))) } if bufferWrites { if len(muxIn.buf)+len(buf) > cap(muxIn.buf) { muxIn.writeChan <- muxIn.buf length := <-muxIn.writeLenChan if length != len(muxIn.buf) { return 0, io.ErrShortWrite } muxIn.buf = muxIn.buf[:0] } muxIn.buf = append(muxIn.buf, buf...) } else { muxIn.writeChan <- buf length := <-muxIn.writeLenChan if length != len(buf) { return 0, io.ErrShortWrite } } muxIn.hash.Write(buf) return len(buf), nil } mongo-tools-r3.2.11/common/archive/multiplexer_roundtrip_test.go000066400000000000000000000143421276133741700252330ustar00rootroot00000000000000package archive import ( "bytes" "fmt" "github.com/mongodb/mongo-tools/common/db" "github.com/mongodb/mongo-tools/common/intents" . "github.com/smartystreets/goconvey/convey" "gopkg.in/mgo.v2/bson" "hash" "hash/crc32" "io" "os" "testing" ) var testIntents = []*intents.Intent{ &intents.Intent{ DB: "foo", C: "bar", Location: "foo.bar", }, &intents.Intent{ DB: "ding", C: "bats", Location: "ding.bats", }, &intents.Intent{ DB: "flim", C: "flam.fooey", Location: "flim.flam.fooey", }, &intents.Intent{ DB: "crow", C: "bar", Location: "crow.bar", }, } type testDoc struct { Bar int Baz string } type closingBuffer struct { bytes.Buffer } func (*closingBuffer) Close() error { return nil } func TestBasicMux(t *testing.T) { var err error Convey("with 10000 docs in each of five collections", t, func() { buf := &closingBuffer{bytes.Buffer{}} mux := NewMultiplexer(buf) muxIns := map[string]*MuxIn{} inChecksum := map[string]hash.Hash{} inLength := map[string]int{} outChecksum := map[string]hash.Hash{} outLength := map[string]int{} // To confirm that what we multiplex is the same as what we demultiplex, we // create input and output hashes for each namespace. After we finish // multiplexing and demultiplexing we will compare all of the CRCs for each // namespace for _, dbc := range testIntents { inChecksum[dbc.Namespace()] = crc32.NewIEEE() muxIns[dbc.Namespace()] = &MuxIn{Intent: dbc, Mux: mux} } errChan := make(chan error) for index, dbc := range testIntents { closeDbc := dbc go func() { err = muxIns[closeDbc.Namespace()].Open() if err != nil { errChan <- err return } staticBSONBuf := make([]byte, db.MaxBSONSize) for i := 0; i < 10000; i++ { bsonBytes, _ := bson.Marshal(testDoc{Bar: index * i, Baz: closeDbc.Namespace()}) bsonBuf := staticBSONBuf[:len(bsonBytes)] copy(bsonBuf, bsonBytes) muxIns[closeDbc.Namespace()].Write(bsonBuf) inChecksum[closeDbc.Namespace()].Write(bsonBuf) inLength[closeDbc.Namespace()] += len(bsonBuf) } err = muxIns[closeDbc.Namespace()].Close() errChan <- err }() } Convey("each document should be multiplexed", func() { fmt.Fprintf(os.Stderr, "About to mux\n") go mux.Run() for _ = range testIntents { err := <-errChan So(err, ShouldBeNil) } close(mux.Control) err = <-mux.Completed So(err, ShouldBeNil) demux := &Demultiplexer{In: buf} demuxOuts := map[string]*RegularCollectionReceiver{} for _, dbc := range testIntents { outChecksum[dbc.Namespace()] = crc32.NewIEEE() demuxOuts[dbc.Namespace()] = &RegularCollectionReceiver{Intent: dbc, Demux: demux} demuxOuts[dbc.Namespace()].Open() } errChan := make(chan error) for _, dbc := range testIntents { closeDbc := dbc go func() { bs := make([]byte, db.MaxBSONSize) var readErr error //var length int var i int for { i++ var length int length, readErr = demuxOuts[closeDbc.Namespace()].Read(bs) if readErr != nil { break } outChecksum[closeDbc.Namespace()].Write(bs[:length]) outLength[closeDbc.Namespace()] += len(bs[:length]) } if readErr == io.EOF { readErr = nil } errChan <- readErr }() } Convey("and demultiplexed successfully", func() { demux.Run() So(err, ShouldBeNil) for _ = range testIntents { err := <-errChan So(err, ShouldBeNil) } for _, dbc := range testIntents { So(inLength[dbc.Namespace()], ShouldEqual, outLength[dbc.Namespace()]) So(inChecksum[dbc.Namespace()].Sum([]byte{}), ShouldResemble, outChecksum[dbc.Namespace()].Sum([]byte{})) } }) }) }) return } func TestParallelMux(t *testing.T) { Convey("parllel mux/demux over a pipe", t, func() { readPipe, writePipe, err := os.Pipe() So(err, ShouldBeNil) mux := NewMultiplexer(writePipe) muxIns := map[string]*MuxIn{} demux := &Demultiplexer{In: readPipe} demuxOuts := map[string]*RegularCollectionReceiver{} inChecksum := map[string]hash.Hash{} inLength := map[string]int{} outChecksum := map[string]hash.Hash{} outLength := map[string]int{} for _, dbc := range testIntents { inChecksum[dbc.Namespace()] = crc32.NewIEEE() outChecksum[dbc.Namespace()] = crc32.NewIEEE() muxIns[dbc.Namespace()] = &MuxIn{Intent: dbc, Mux: mux} demuxOuts[dbc.Namespace()] = &RegularCollectionReceiver{Intent: dbc, Demux: demux} } writeErrChan := make(chan error) readErrChan := make(chan error) for index, dbc := range testIntents { closeDbc := dbc go func() { err = muxIns[closeDbc.Namespace()].Open() if err != nil { writeErrChan <- nil return } staticBSONBuf := make([]byte, db.MaxBSONSize) for i := 0; i < 10000; i++ { bsonBytes, _ := bson.Marshal(testDoc{Bar: index * i, Baz: closeDbc.Namespace()}) bsonBuf := staticBSONBuf[:len(bsonBytes)] copy(bsonBuf, bsonBytes) muxIns[closeDbc.Namespace()].Write(bsonBuf) inChecksum[closeDbc.Namespace()].Write(bsonBuf) inLength[closeDbc.Namespace()] += len(bsonBuf) } err = muxIns[closeDbc.Namespace()].Close() writeErrChan <- err }() } for _, dbc := range testIntents { closeDbc := dbc go func() { demuxOuts[closeDbc.Namespace()].Open() bs := make([]byte, db.MaxBSONSize) var readErr error //var length int var i int for { i++ var length int length, readErr = demuxOuts[closeDbc.Namespace()].Read(bs) if readErr != nil { break } outChecksum[closeDbc.Namespace()].Write(bs[:length]) outLength[closeDbc.Namespace()] += len(bs[:length]) } if readErr == io.EOF { readErr = nil } readErrChan <- readErr }() } go demux.Run() go mux.Run() for _ = range testIntents { err := <-writeErrChan So(err, ShouldBeNil) err = <-readErrChan So(err, ShouldBeNil) } close(mux.Control) muxErr := <-mux.Completed So(muxErr, ShouldBeNil) for _, dbc := range testIntents { So(inLength[dbc.Namespace()], ShouldEqual, outLength[dbc.Namespace()]) So(inChecksum[dbc.Namespace()].Sum([]byte{}), ShouldResemble, outChecksum[dbc.Namespace()].Sum([]byte{})) } }) return } mongo-tools-r3.2.11/common/archive/parser.go000066400000000000000000000112541276133741700210070ustar00rootroot00000000000000package archive import ( "fmt" "github.com/mongodb/mongo-tools/common/db" "io" ) // parser.go implements the parsing of the low-level archive format // The low level archive format is defined as zero or more blocks // where each block is defined as: // a header BSON document // zero or more body BSON documents // a four byte terminator (0xFFFFFFFF) // ParserConsumer is the interface that one needs to implement to consume data from the Parser type ParserConsumer interface { HeaderBSON([]byte) error BodyBSON([]byte) error End() error } // Parser encapsulates the small amount of state that the parser needs to keep type Parser struct { In io.Reader buf [db.MaxBSONSize]byte length int } type parserError struct { Err error Msg string } // Error is part of the Error interface. It formats a parserError for human readability. func (pe *parserError) Error() string { err := fmt.Sprintf("corruption found in archive; %v", pe.Msg) if pe.Err != nil { err = fmt.Sprintf("%v ( %v )", err, pe.Err) } return err } // newParserError creates a parserError with just a message func newParserError(msg string) error { return &parserError{ Msg: msg, } } // newParserWrappedError creates a parserError with a message as well as an underlying cause error func newParserWrappedError(msg string, err error) error { return &parserError{ Err: err, Msg: msg, } } // readBSONOrTerminator reads at least four bytes, determines // if the first four bytes are a terminator, a bson lenght, or something else. // If they are a terminator, true,nil are returned. If they are a BSON length, // then the remainder of the BSON document are read in to the parser, otherwise // an error is returned. func (parse *Parser) readBSONOrTerminator() (isTerminator bool, err error) { parse.length = 0 _, err = io.ReadFull(parse.In, parse.buf[0:4]) if err == io.EOF { return false, err } if err != nil { return false, newParserWrappedError("I/O error reading length or terminator", err) } size := int32( (uint32(parse.buf[0]) << 0) | (uint32(parse.buf[1]) << 8) | (uint32(parse.buf[2]) << 16) | (uint32(parse.buf[3]) << 24), ) if size == terminator { return true, nil } if size < minBSONSize || size > db.MaxBSONSize { return false, newParserError(fmt.Sprintf("%v is neither a valid bson length nor a archive terminator", size)) } // TODO Because we're reusing this same buffer for all of our IO, we are basically guaranteeing that we'll // copy the bytes twice. At some point we should fix this. It's slightly complex, because we'll need consumer // methods closing one buffer and acquiring another _, err = io.ReadFull(parse.In, parse.buf[4:size]) if err != nil { // any error, including EOF is an error so we wrap it up return false, newParserWrappedError("read bson", err) } if parse.buf[size-1] != 0x00 { return false, newParserError(fmt.Sprintf("bson (size: %v, byte: %d) doesn't end with a null byte", size, parse.buf[size-1])) } parse.length = int(size) return false, nil } // ReadAllBlocks calls ReadBlock() until it returns an error. // If the error is EOF, then nil is returned, otherwise it returns the error func (parse *Parser) ReadAllBlocks(consumer ParserConsumer) (err error) { for err == nil { err = parse.ReadBlock(consumer) } if err == io.EOF { return nil } return err } // ReadBlock reads one archive block ( header + body* + terminator ) // calling consumer.HeaderBSON() on the header, consumer.BodyBSON() on each piece of body, // and consumer.EOF() when EOF is encountered before any data was read. // It returns nil if a whole block was read, io.EOF if nothing was read, // and a parserError if there was any io error in the middle of the block, // if either of the consumer methods return error, or if there was any sort of // parsing failure. func (parse *Parser) ReadBlock(consumer ParserConsumer) (err error) { isTerminator, err := parse.readBSONOrTerminator() if err == io.EOF { handlerErr := consumer.End() if handlerErr != nil { return newParserWrappedError("ParserConsumer.End", handlerErr) } return err } if err != nil { return err } if isTerminator { return newParserError("consecutive terminators / headerless blocks are not allowed") } err = consumer.HeaderBSON(parse.buf[:parse.length]) if err != nil { return newParserWrappedError("ParserConsumer.HeaderBSON()", err) } for { isTerminator, err = parse.readBSONOrTerminator() if err != nil { // all errors, including EOF are errors here return newParserWrappedError("ParserConsumer.BodyBSON()", err) } if isTerminator { return nil } err = consumer.BodyBSON(parse.buf[:parse.length]) if err != nil { return newParserWrappedError("ParserConsumer.BodyBSON()", err) } } } mongo-tools-r3.2.11/common/archive/parser_test.go000066400000000000000000000074331276133741700220520ustar00rootroot00000000000000package archive import ( "bytes" "fmt" . "github.com/smartystreets/goconvey/convey" "gopkg.in/mgo.v2/bson" "io" "testing" ) type testConsumer struct { headers []string // header data bodies []string // body data eof bool } func (tc *testConsumer) HeaderBSON(b []byte) error { ss := strStruct{} err := bson.Unmarshal(b, &ss) tc.headers = append(tc.headers, ss.Str) return err } func (tc *testConsumer) BodyBSON(b []byte) error { ss := strStruct{} err := bson.Unmarshal(b, &ss) tc.bodies = append(tc.bodies, ss.Str) return err } func (tc *testConsumer) End() (err error) { if tc.eof { err = fmt.Errorf("double end") } tc.eof = true return err } type strStruct struct { Str string } var term = []byte{0xFF, 0xFF, 0xFF, 0xFF} var notTerm = []byte{0xFF, 0xFF, 0xFF, 0xFE} func TestParsing(t *testing.T) { Convey("With a parser with a simple parser consumer", t, func() { tc := &testConsumer{} parser := Parser{} Convey("a well formed header and body data parse correctly", func() { buf := bytes.Buffer{} b, _ := bson.Marshal(strStruct{"header"}) buf.Write(b) b, _ = bson.Marshal(strStruct{"body"}) buf.Write(b) buf.Write(term) parser.In = &buf err := parser.ReadBlock(tc) So(err, ShouldBeNil) So(tc.eof, ShouldBeFalse) So(tc.headers[0], ShouldEqual, "header") So(tc.bodies[0], ShouldEqual, "body") err = parser.ReadBlock(tc) So(err, ShouldEqual, io.EOF) So(tc.eof, ShouldBeTrue) }) Convey("a well formed header and multiple body datas parse correctly", func() { buf := bytes.Buffer{} b, _ := bson.Marshal(strStruct{"header"}) buf.Write(b) b, _ = bson.Marshal(strStruct{"body0"}) buf.Write(b) b, _ = bson.Marshal(strStruct{"body1"}) buf.Write(b) b, _ = bson.Marshal(strStruct{"body2"}) buf.Write(b) buf.Write(term) parser.In = &buf err := parser.ReadBlock(tc) So(err, ShouldBeNil) So(tc.eof, ShouldBeFalse) So(tc.headers[0], ShouldEqual, "header") So(tc.bodies[0], ShouldEqual, "body0") So(tc.bodies[1], ShouldEqual, "body1") So(tc.bodies[2], ShouldEqual, "body2") err = parser.ReadBlock(tc) So(err, ShouldEqual, io.EOF) So(tc.eof, ShouldBeTrue) }) Convey("an incorrect terminator should cause an error", func() { buf := bytes.Buffer{} b, _ := bson.Marshal(strStruct{"header"}) buf.Write(b) b, _ = bson.Marshal(strStruct{"body"}) buf.Write(b) buf.Write(notTerm) parser.In = &buf err := parser.ReadBlock(tc) So(err, ShouldNotBeNil) }) Convey("an empty block should result in EOF", func() { buf := bytes.Buffer{} parser.In = &buf err := parser.ReadBlock(tc) So(err, ShouldEqual, io.EOF) So(tc.eof, ShouldBeTrue) }) Convey("an error comming from the consumer should propigate through the parser", func() { tc.eof = true buf := bytes.Buffer{} parser.In = &buf err := parser.ReadBlock(tc) So(err.Error(), ShouldContainSubstring, "double end") }) Convey("a partial block should result in a non-EOF error", func() { buf := bytes.Buffer{} b, _ := bson.Marshal(strStruct{"header"}) buf.Write(b) b, _ = bson.Marshal(strStruct{"body"}) buf.Write(b) parser.In = &buf err := parser.ReadBlock(tc) So(err, ShouldNotBeNil) So(tc.eof, ShouldBeFalse) So(tc.headers[0], ShouldEqual, "header") So(tc.bodies[0], ShouldEqual, "body") }) Convey("a block with a missing terminator shoud result in a non-EOF error", func() { buf := bytes.Buffer{} b, _ := bson.Marshal(strStruct{"header"}) buf.Write(b) b, _ = bson.Marshal(strStruct{"body"}) buf.Write(b[:len(b)-1]) buf.Write([]byte{0x01}) buf.Write(notTerm) parser.In = &buf err := parser.ReadBlock(tc) So(err, ShouldNotBeNil) So(tc.eof, ShouldBeFalse) So(tc.headers[0], ShouldEqual, "header") So(tc.bodies, ShouldBeNil) }) }) return } mongo-tools-r3.2.11/common/archive/prelude.go000066400000000000000000000261461276133741700211610ustar00rootroot00000000000000package archive import ( "bytes" "fmt" "github.com/mongodb/mongo-tools/common/intents" "github.com/mongodb/mongo-tools/common/log" "github.com/mongodb/mongo-tools/common/options" "gopkg.in/mgo.v2/bson" "io" "path/filepath" "sync/atomic" ) //MetadataFile implements intents.file type MetadataFile struct { *bytes.Buffer Intent *intents.Intent pos int64 } func (md *MetadataFile) Open() error { return nil } func (md *MetadataFile) Close() error { return nil } func (md *MetadataFile) Read(p []byte) (int, error) { n, err := md.Buffer.Read(p) atomic.AddInt64(&md.pos, int64(n)) return n, err } func (md *MetadataFile) Pos() int64 { return atomic.LoadInt64(&md.pos) } // DirLike represents the group of methods done on directories and files in dump directories, // or in archives, when mongorestore is figuring out what intents to create. type DirLike interface { Name() string Path() string Size() int64 IsDir() bool Stat() (DirLike, error) ReadDir() ([]DirLike, error) Parent() DirLike } // Prelude represents the knowledge gleaned from reading the prelude out of the archive. type Prelude struct { Header *Header DBS []string NamespaceMetadatas []*CollectionMetadata NamespaceMetadatasByDB map[string][]*CollectionMetadata } // Read consumes and checks the magic number at the beginning of the archive, // then it runs the parser with a Prelude as its consumer. func (prelude *Prelude) Read(in io.Reader) error { readMagicNumberBuf := make([]byte, 4) _, err := io.ReadAtLeast(in, readMagicNumberBuf, 4) if err != nil { return fmt.Errorf("I/O failure reading beginning of archive: %v", err) } readMagicNumber := uint32( (uint32(readMagicNumberBuf[0]) << 0) | (uint32(readMagicNumberBuf[1]) << 8) | (uint32(readMagicNumberBuf[2]) << 16) | (uint32(readMagicNumberBuf[3]) << 24), ) if readMagicNumber != MagicNumber { return fmt.Errorf("stream or file does not appear to be a mongodump archive") } if prelude.NamespaceMetadatasByDB != nil { prelude.NamespaceMetadatasByDB = make(map[string][]*CollectionMetadata, 0) } parser := Parser{In: in} parserConsumer := &preludeParserConsumer{prelude: prelude} return parser.ReadBlock(parserConsumer) } // NewPrelude generates a Prelude using the contents of an intent.Manager. func NewPrelude(manager *intents.Manager, concurrentColls int, serverVersion string) (*Prelude, error) { prelude := Prelude{ Header: &Header{ FormatVersion: archiveFormatVersion, ServerVersion: serverVersion, ToolVersion: options.VersionStr, ConcurrentCollections: int32(concurrentColls), }, NamespaceMetadatasByDB: make(map[string][]*CollectionMetadata, 0), } allIntents := manager.Intents() for _, intent := range allIntents { if intent.MetadataFile != nil { archiveMetadata, ok := intent.MetadataFile.(*MetadataFile) if !ok { return nil, fmt.Errorf("MetadataFile is not an archive.Metadata") } prelude.AddMetadata(&CollectionMetadata{ Database: intent.DB, Collection: intent.C, Metadata: archiveMetadata.Buffer.String(), }) } else { prelude.AddMetadata(&CollectionMetadata{ Database: intent.DB, Collection: intent.C, }) } } return &prelude, nil } // AddMetadata adds a metadata data structure to a prelude and does the required bookkeeping. func (prelude *Prelude) AddMetadata(cm *CollectionMetadata) { prelude.NamespaceMetadatas = append(prelude.NamespaceMetadatas, cm) if prelude.NamespaceMetadatasByDB == nil { prelude.NamespaceMetadatasByDB = make(map[string][]*CollectionMetadata) } _, ok := prelude.NamespaceMetadatasByDB[cm.Database] if !ok { prelude.DBS = append(prelude.DBS, cm.Database) } prelude.NamespaceMetadatasByDB[cm.Database] = append(prelude.NamespaceMetadatasByDB[cm.Database], cm) log.Logf(log.Info, "archive prelude %v.%v", cm.Database, cm.Collection) } // Write writes the archive header. func (prelude *Prelude) Write(out io.Writer) error { magicNumberBytes := make([]byte, 4) for i := range magicNumberBytes { magicNumberBytes[i] = byte(uint32(MagicNumber) >> uint(i*8)) } _, err := out.Write(magicNumberBytes) if err != nil { return err } buf, err := bson.Marshal(prelude.Header) if err != nil { return err } _, err = out.Write(buf) if err != nil { return err } for _, cm := range prelude.NamespaceMetadatas { buf, err = bson.Marshal(cm) if err != nil { return err } _, err = out.Write(buf) if err != nil { return err } } _, err = out.Write(terminatorBytes) if err != nil { return err } return nil } // preludeParserConsumer wraps a Prelude, and implements ParserConsumer. type preludeParserConsumer struct { prelude *Prelude } // HeaderBSON is part of the ParserConsumer interface, it unmarshals archive Headers. func (hpc *preludeParserConsumer) HeaderBSON(data []byte) error { hpc.prelude.Header = &Header{} err := bson.Unmarshal(data, hpc.prelude.Header) if err != nil { return err } return nil } // BodyBSON is part of the ParserConsumer interface, it unmarshals CollectionMetadata's. func (hpc *preludeParserConsumer) BodyBSON(data []byte) error { cm := &CollectionMetadata{} err := bson.Unmarshal(data, cm) if err != nil { return err } hpc.prelude.AddMetadata(cm) return nil } // BodyBSON is part of the ParserConsumer interface. func (hpc *preludeParserConsumer) End() error { return nil } // PreludeExplorer implements DirLike. PreludeExplorer represent the databases, collections, // and their metadata json files, of an archive, in such a way that they can be explored like a filesystem. type PreludeExplorer struct { prelude *Prelude database string collection string isMetadata bool } // NewPreludeExplorer creates a PreludeExplorer from a Prelude. func (prelude *Prelude) NewPreludeExplorer() (*PreludeExplorer, error) { pe := &PreludeExplorer{ prelude: prelude, } return pe, nil } // Name is part of the DirLike interface. It synthesizes a filename for the given "location" the prelude. func (pe *PreludeExplorer) Name() string { if pe.collection == "" { return pe.database } if pe.isMetadata { return pe.collection + ".metadata.json" } return pe.collection + ".bson" } // Path is part of the DirLike interface. It creates the full path for the "location" in the prelude. func (pe *PreludeExplorer) Path() string { if pe.collection == "" { return pe.database } if pe.database == "" { return pe.Name() } return filepath.Join(pe.database, pe.Name()) } // Size is part of the DirLike interface. It returns the size from the metadata // of the prelude, if the "location" is a collection. func (pe *PreludeExplorer) Size() int64 { if pe.IsDir() { return 0 } for _, ns := range pe.prelude.NamespaceMetadatas { if ns.Database == pe.database && ns.Collection == pe.collection { return int64(ns.Size) } } return 0 } // IsDir is part of the DirLike interface. All pes that are not collections are Dirs. func (pe *PreludeExplorer) IsDir() bool { return pe.collection == "" } // Stat is part of the DirLike interface. os.Stat returns a FileInfo, and since // DirLike is similar to FileInfo, we just return the pe, here. func (pe *PreludeExplorer) Stat() (DirLike, error) { return pe, nil } // ReadDir is part of the DirLIke interface. ReadDir generates a list of PreludeExplorers // whose "locations" are encapsulated by the current pes "location". // // "dump/oplog.bson" => &PreludeExplorer{ database: "", collection: "oplog.bson" } // "dump/test/" => &PreludeExplorer{ database: "test", collection: "foo.bson" } // "dump/test/foo.bson" => &PreludeExplorer{ database: "test", collection: "" } // "dump/test/foo.json" => &PreludeExplorer{ database: "test", collection: "foo", isMetadata: true } // func (pe *PreludeExplorer) ReadDir() ([]DirLike, error) { if !pe.IsDir() { return nil, fmt.Errorf("not a directory") } pes := []DirLike{} if pe.database == "" { // when reading the top level of the archive, we need return all of the // collections that are not bound to a database, aka, the oplog, and then all of // the databases the prelude stores all top-level collections as collections in // the "" database topLevelNamespaceMetadatas, ok := pe.prelude.NamespaceMetadatasByDB[""] if ok { for _, topLevelNamespaceMetadata := range topLevelNamespaceMetadatas { pes = append(pes, &PreludeExplorer{ prelude: pe.prelude, collection: topLevelNamespaceMetadata.Collection, }) if topLevelNamespaceMetadata.Metadata != "" { pes = append(pes, &PreludeExplorer{ prelude: pe.prelude, collection: topLevelNamespaceMetadata.Collection, isMetadata: true, }) } } } for _, db := range pe.prelude.DBS { pes = append(pes, &PreludeExplorer{ prelude: pe.prelude, database: db, }) } } else { // when reading the contents of a database directory, we just return all of the bson and // json files for all of the collections bound to that database namespaceMetadatas, ok := pe.prelude.NamespaceMetadatasByDB[pe.database] if !ok { return nil, fmt.Errorf("no such directory") //TODO: replace with real ERRNOs? } for _, namespaceMetadata := range namespaceMetadatas { pes = append(pes, &PreludeExplorer{ prelude: pe.prelude, database: pe.database, collection: namespaceMetadata.Collection, }) if namespaceMetadata.Metadata != "" { pes = append(pes, &PreludeExplorer{ prelude: pe.prelude, database: pe.database, collection: namespaceMetadata.Collection, isMetadata: true, }) } } } return pes, nil } // Parent is part of the DirLike interface. It returns a pe without a collection, if there is one, // otherwise, without a database. func (pe *PreludeExplorer) Parent() DirLike { if pe.collection != "" { return &PreludeExplorer{ prelude: pe.prelude, database: pe.database, } } return &PreludeExplorer{ prelude: pe.prelude, } } // MetadataPreludeFile is part of the intents.file. It allows the metadata contained in the prelude to be opened and read type MetadataPreludeFile struct { Intent *intents.Intent Prelude *Prelude *bytes.Buffer pos int64 } // Open is part of the intents.file interface, it finds the metadata in the prelude and creates a bytes.Buffer from it. func (mpf *MetadataPreludeFile) Open() error { if mpf.Intent.C == "" { return fmt.Errorf("so such file") // what's the errno that occurs when one tries to open a directory } dbMetadatas, ok := mpf.Prelude.NamespaceMetadatasByDB[mpf.Intent.DB] if !ok { return fmt.Errorf("so such file") // what's the errno that occurs when one tries to open a directory } for _, metadata := range dbMetadatas { if metadata.Collection == mpf.Intent.C { mpf.Buffer = bytes.NewBufferString(metadata.Metadata) return nil } } return fmt.Errorf("so such file") // what's the errno that occurs when one tries to open a directory } // Close is part of the intents.file interface. func (mpf *MetadataPreludeFile) Close() error { mpf.Buffer = nil return nil } func (mpf *MetadataPreludeFile) Read(p []byte) (int, error) { n, err := mpf.Buffer.Read(p) atomic.AddInt64(&mpf.pos, int64(n)) return n, err } func (mpf *MetadataPreludeFile) Pos() int64 { return atomic.LoadInt64(&mpf.pos) } mongo-tools-r3.2.11/common/archive/prelude_test.go000066400000000000000000000023521276133741700222110ustar00rootroot00000000000000package archive import ( "bytes" . "github.com/smartystreets/goconvey/convey" // "gopkg.in/mgo.v2/bson" "testing" ) func TestPrelude(t *testing.T) { var err error Convey("WritePrelude/ReadPrelude roundtrip", t, func() { cm1 := &CollectionMetadata{ Database: "db1", Collection: "c1", Metadata: "m1", } cm2 := &CollectionMetadata{ Database: "db1", Collection: "c2", Metadata: "m2", } cm3 := &CollectionMetadata{ Database: "db2", Collection: "c3", Metadata: "m3", } cm4 := &CollectionMetadata{ Database: "db3", Collection: "c4", Metadata: "m4", } archivePrelude := &Prelude{ Header: &Header{ FormatVersion: "version-foo", }, NamespaceMetadatas: []*CollectionMetadata{cm1, cm2, cm3, cm4}, DBS: []string{"db1", "db2", "db3"}, NamespaceMetadatasByDB: map[string][]*CollectionMetadata{ "db1": []*CollectionMetadata{cm1, cm2}, "db2": []*CollectionMetadata{cm3}, "db3": []*CollectionMetadata{cm4}, }, } buf := &bytes.Buffer{} err = archivePrelude.Write(buf) So(err, ShouldBeNil) archivePrelude2 := &Prelude{} err := archivePrelude2.Read(buf) So(err, ShouldBeNil) So(archivePrelude2, ShouldResemble, archivePrelude) }) } mongo-tools-r3.2.11/common/auth/000077500000000000000000000000001276133741700165015ustar00rootroot00000000000000mongo-tools-r3.2.11/common/auth/auth_info.go000066400000000000000000000036061276133741700210110ustar00rootroot00000000000000// Package auth provides utilities for performing tasks related to authentication. package auth import ( "fmt" "github.com/mongodb/mongo-tools/common/db" "gopkg.in/mgo.v2/bson" "strings" ) // GetAuthVersion gets the authentication schema version of the connected server // and returns that value as an integer along with any error that occurred. func GetAuthVersion(commander db.CommandRunner) (int, error) { results := bson.M{} err := commander.Run( bson.D{ {"getParameter", 1}, {"authSchemaVersion", 1}, }, &results, "admin", ) if err != nil { errMessage := err.Error() // as a necessary hack, if the error message takes a certain form, // we can infer version 1. This is because early versions of mongodb // had no concept of an "auth schema version", so asking for the // authSchemaVersion value will return a "no option found" or "no such cmd" if errMessage == "no option found to get" || strings.HasPrefix(errMessage, "no such cmd") { return 1, nil } // otherwise it's a connection error, so bubble it up return 0, err } version, ok := results["authSchemaVersion"].(int) if !ok { // very unlikely this will ever happen return 0, fmt.Errorf( "getParameter command returned non-numeric result: %v", results["authSchemaVersion"]) } return version, nil } // VerifySystemAuthVersion returns an error if authentication is not set up for // the given server. func VerifySystemAuthVersion(sessionProvider *db.SessionProvider) error { session, err := sessionProvider.GetSession() if err != nil { return fmt.Errorf("error getting session from server: %v", err) } defer session.Close() versionEntries := session.DB("admin").C("system.version") if count, err := versionEntries.Count(); err != nil { return fmt.Errorf("error checking pressence of auth version: %v", err) } else if count == 0 { return fmt.Errorf("found no auth version") } return nil } mongo-tools-r3.2.11/common/bsonutil/000077500000000000000000000000001276133741700173775ustar00rootroot00000000000000mongo-tools-r3.2.11/common/bsonutil/bsonutil.go000066400000000000000000000246621276133741700215770ustar00rootroot00000000000000// Package bsonutil provides utilities for processing BSON data. package bsonutil import ( "encoding/base64" "encoding/hex" "errors" "fmt" "github.com/mongodb/mongo-tools/common/json" "github.com/mongodb/mongo-tools/common/util" "gopkg.in/mgo.v2" "gopkg.in/mgo.v2/bson" "strconv" "time" ) var ErrNoSuchField = errors.New("no such field") // ConvertJSONDocumentToBSON iterates through the document map and converts JSON // values to their corresponding BSON values. It also replaces any extended JSON // type value (e.g. $date) with the corresponding BSON type. func ConvertJSONDocumentToBSON(doc map[string]interface{}) error { for key, jsonValue := range doc { var bsonValue interface{} var err error switch v := jsonValue.(type) { case map[string]interface{}, bson.D: // subdocument bsonValue, err = ParseSpecialKeys(v) default: bsonValue, err = ConvertJSONValueToBSON(v) } if err != nil { return err } doc[key] = bsonValue } return nil } // GetExtendedBsonD iterates through the document and returns a bson.D that adds type // information for each key in document. func GetExtendedBsonD(doc bson.D) (bson.D, error) { var err error var bsonDoc bson.D for _, docElem := range doc { var bsonValue interface{} switch v := docElem.Value.(type) { case map[string]interface{}, bson.D: // subdocument bsonValue, err = ParseSpecialKeys(v) default: bsonValue, err = ConvertJSONValueToBSON(v) } if err != nil { return nil, err } bsonDoc = append(bsonDoc, bson.DocElem{docElem.Name, bsonValue}) } return bsonDoc, nil } // FindValueByKey returns the value of keyName in document. If keyName is not found // in the top-level of the document, ErrNoSuchField is returned as the error. func FindValueByKey(keyName string, document *bson.D) (interface{}, error) { for _, key := range *document { if key.Name == keyName { return key.Value, nil } } return nil, ErrNoSuchField } // ParseSpecialKeys takes a JSON document and inspects it for any extended JSON // type (e.g $numberLong) and replaces any such values with the corresponding // BSON type. func ParseSpecialKeys(special interface{}) (interface{}, error) { // first ensure we are using a correct document type var doc map[string]interface{} switch v := special.(type) { case bson.D: doc = v.Map() case map[string]interface{}: doc = v default: return nil, fmt.Errorf("%v (type %T) is not valid input to ParseSpecialKeys", special, special) } // check document to see if it is special switch len(doc) { case 1: // document has a single field if jsonValue, ok := doc["$date"]; ok { switch v := jsonValue.(type) { case string: return util.FormatDate(v) case bson.D: asMap := v.Map() if jsonValue, ok := asMap["$numberLong"]; ok { n, err := parseNumberLongField(jsonValue) if err != nil { return nil, err } return time.Unix(n/1e3, n%1e3*1e6), err } return nil, errors.New("expected $numberLong field in $date") case map[string]interface{}: if jsonValue, ok := v["$numberLong"]; ok { n, err := parseNumberLongField(jsonValue) if err != nil { return nil, err } return time.Unix(n/1e3, n%1e3*1e6), err } return nil, errors.New("expected $numberLong field in $date") case json.Number: n, err := v.Int64() return time.Unix(n/1e3, n%1e3*1e6), err case float64: n := int64(v) return time.Unix(n/1e3, n%1e3*1e6), nil case int32: n := int64(v) return time.Unix(n/1e3, n%1e3*1e6), nil case int64: return time.Unix(v/1e3, v%1e3*1e6), nil case json.ISODate: return v, nil default: return nil, errors.New("invalid type for $date field") } } if jsonValue, ok := doc["$code"]; ok { switch v := jsonValue.(type) { case string: return bson.JavaScript{Code: v}, nil default: return nil, errors.New("expected $code field to have string value") } } if jsonValue, ok := doc["$oid"]; ok { switch v := jsonValue.(type) { case string: if !bson.IsObjectIdHex(v) { return nil, errors.New("expected $oid field to contain 24 hexadecimal character") } return bson.ObjectIdHex(v), nil default: return nil, errors.New("expected $oid field to have string value") } } if jsonValue, ok := doc["$numberLong"]; ok { return parseNumberLongField(jsonValue) } if jsonValue, ok := doc["$numberInt"]; ok { switch v := jsonValue.(type) { case string: // all of decimal, hex, and octal are supported here n, err := strconv.ParseInt(v, 0, 32) return int32(n), err default: return nil, errors.New("expected $numberInt field to have string value") } } if jsonValue, ok := doc["$timestamp"]; ok { ts := json.Timestamp{} var tsDoc map[string]interface{} switch internalDoc := jsonValue.(type) { case map[string]interface{}: tsDoc = internalDoc case bson.D: tsDoc = internalDoc.Map() default: return nil, errors.New("expected $timestamp key to have internal document") } if seconds, ok := tsDoc["t"]; ok { if asUint32, err := util.ToUInt32(seconds); err == nil { ts.Seconds = asUint32 } else { return nil, errors.New("expected $timestamp 't' field to be a numeric type") } } else { return nil, errors.New("expected $timestamp to have 't' field") } if inc, ok := tsDoc["i"]; ok { if asUint32, err := util.ToUInt32(inc); err == nil { ts.Increment = asUint32 } else { return nil, errors.New("expected $timestamp 'i' field to be a numeric type") } } else { return nil, errors.New("expected $timestamp to have 'i' field") } // see BSON spec for details on the bit fiddling here return bson.MongoTimestamp(int64(ts.Seconds)<<32 | int64(ts.Increment)), nil } if _, ok := doc["$undefined"]; ok { return bson.Undefined, nil } if _, ok := doc["$maxKey"]; ok { return bson.MaxKey, nil } if _, ok := doc["$minKey"]; ok { return bson.MinKey, nil } case 2: // document has two fields if jsonValue, ok := doc["$code"]; ok { code := bson.JavaScript{} switch v := jsonValue.(type) { case string: code.Code = v default: return nil, errors.New("expected $code field to have string value") } if jsonValue, ok = doc["$scope"]; ok { switch v2 := jsonValue.(type) { case map[string]interface{}, bson.D: x, err := ParseSpecialKeys(v2) if err != nil { return nil, err } code.Scope = x return code, nil default: return nil, errors.New("expected $scope field to contain map") } } else { return nil, errors.New("expected $scope field with $code field") } } if jsonValue, ok := doc["$regex"]; ok { regex := bson.RegEx{} switch pattern := jsonValue.(type) { case string: regex.Pattern = pattern default: return nil, errors.New("expected $regex field to have string value") } if jsonValue, ok = doc["$options"]; !ok { return nil, errors.New("expected $options field with $regex field") } switch options := jsonValue.(type) { case string: regex.Options = options default: return nil, errors.New("expected $options field to have string value") } // Validate regular expression options for i := range regex.Options { switch o := regex.Options[i]; o { default: return nil, fmt.Errorf("invalid regular expression option '%v'", o) case 'g', 'i', 'm', 's': // allowed } } return regex, nil } if jsonValue, ok := doc["$binary"]; ok { binary := bson.Binary{} switch data := jsonValue.(type) { case string: bytes, err := base64.StdEncoding.DecodeString(data) if err != nil { return nil, err } binary.Data = bytes default: return nil, errors.New("expected $binary field to have string value") } if jsonValue, ok = doc["$type"]; !ok { return nil, errors.New("expected $type field with $binary field") } switch typ := jsonValue.(type) { case string: kind, err := hex.DecodeString(typ) if err != nil { return nil, err } else if len(kind) != 1 { return nil, errors.New("expected single byte (as hexadecimal string) for $type field") } binary.Kind = kind[0] default: return nil, errors.New("expected $type field to have string value") } return binary, nil } if jsonValue, ok := doc["$ref"]; ok { dbRef := mgo.DBRef{} switch data := jsonValue.(type) { case string: dbRef.Collection = data default: return nil, errors.New("expected string for $ref field") } if jsonValue, ok = doc["$id"]; ok { switch v2 := jsonValue.(type) { case map[string]interface{}, bson.D: x, err := ParseSpecialKeys(v2) if err != nil { return nil, fmt.Errorf("error parsing $id field: %v", err) } dbRef.Id = x default: dbRef.Id = v2 } return dbRef, nil } } case 3: if jsonValue, ok := doc["$ref"]; ok { dbRef := mgo.DBRef{} switch data := jsonValue.(type) { case string: dbRef.Collection = data default: return nil, errors.New("expected string for $ref field") } if jsonValue, ok = doc["$id"]; ok { switch v2 := jsonValue.(type) { case map[string]interface{}, bson.D: x, err := ParseSpecialKeys(v2) if err != nil { return nil, fmt.Errorf("error parsing $id field: %v", err) } dbRef.Id = x default: dbRef.Id = v2 } if dbValue, ok := doc["$db"]; ok { switch v3 := dbValue.(type) { case string: dbRef.Database = v3 default: return nil, errors.New("expected string for $db field") } return dbRef, nil } } } } // nothing matched, so we recurse deeper switch v := special.(type) { case bson.D: return GetExtendedBsonD(v) case map[string]interface{}: return ConvertJSONValueToBSON(v) default: return nil, fmt.Errorf("%v (type %T) is not valid input to ParseSpecialKeys", special, special) } } // ParseJSONValue takes any value generated by the json package and returns a // BSON version of that value. func ParseJSONValue(jsonValue interface{}) (interface{}, error) { switch v := jsonValue.(type) { case map[string]interface{}, bson.D: // subdocument return ParseSpecialKeys(v) default: return ConvertJSONValueToBSON(v) } } func parseNumberLongField(jsonValue interface{}) (int64, error) { switch v := jsonValue.(type) { case string: // all of decimal, hex, and octal are supported here return strconv.ParseInt(v, 0, 64) default: return 0, errors.New("expected $numberLong field to have string value") } } mongo-tools-r3.2.11/common/bsonutil/converter.go000066400000000000000000000207431276133741700217430ustar00rootroot00000000000000package bsonutil import ( "encoding/base64" "errors" "fmt" "github.com/mongodb/mongo-tools/common/json" "github.com/mongodb/mongo-tools/common/util" "gopkg.in/mgo.v2" "gopkg.in/mgo.v2/bson" "time" ) // ConvertJSONValueToBSON walks through a document or an array and // replaces any extended JSON value with its corresponding BSON type. func ConvertJSONValueToBSON(x interface{}) (interface{}, error) { switch v := x.(type) { case nil: return nil, nil case bool: return v, nil case map[string]interface{}: // document for key, jsonValue := range v { bsonValue, err := ParseJSONValue(jsonValue) if err != nil { return nil, err } v[key] = bsonValue } return v, nil case bson.D: for i := range v { var err error v[i].Value, err = ParseJSONValue(v[i].Value) if err != nil { return nil, err } } return v, nil case []interface{}: // array for i, jsonValue := range v { bsonValue, err := ParseJSONValue(jsonValue) if err != nil { return nil, err } v[i] = bsonValue } return v, nil case string, float64, int32, int64: return v, nil // require no conversion case json.ObjectId: // ObjectId s := string(v) if !bson.IsObjectIdHex(s) { return nil, errors.New("expected ObjectId to contain 24 hexadecimal characters") } return bson.ObjectIdHex(s), nil case json.Date: // Date n := int64(v) return time.Unix(n/1e3, n%1e3*1e6), nil case json.ISODate: // ISODate n := string(v) return util.FormatDate(n) case json.NumberLong: // NumberLong return int64(v), nil case json.NumberInt: // NumberInt return int32(v), nil case json.NumberFloat: // NumberFloat return float64(v), nil case json.BinData: // BinData data, err := base64.StdEncoding.DecodeString(v.Base64) if err != nil { return nil, err } return bson.Binary{v.Type, data}, nil case json.DBRef: // DBRef var err error v.Id, err = ParseJSONValue(v.Id) if err != nil { return nil, err } return mgo.DBRef{v.Collection, v.Id, v.Database}, nil case json.DBPointer: // DBPointer, for backwards compatibility return bson.DBPointer{v.Namespace, v.Id}, nil case json.RegExp: // RegExp return bson.RegEx{v.Pattern, v.Options}, nil case json.Timestamp: // Timestamp ts := (int64(v.Seconds) << 32) | int64(v.Increment) return bson.MongoTimestamp(ts), nil case json.JavaScript: // Javascript return bson.JavaScript{v.Code, v.Scope}, nil case json.MinKey: // MinKey return bson.MinKey, nil case json.MaxKey: // MaxKey return bson.MaxKey, nil case json.Undefined: // undefined return bson.Undefined, nil default: return nil, fmt.Errorf("conversion of JSON value '%v' of type '%T' not supported", v, v) } } func convertKeys(v bson.M) (bson.M, error) { for key, value := range v { jsonValue, err := ConvertBSONValueToJSON(value) if err != nil { return nil, err } v[key] = jsonValue } return v, nil } func getConvertedKeys(v bson.M) (bson.M, error) { out := bson.M{} for key, value := range v { jsonValue, err := GetBSONValueAsJSON(value) if err != nil { return nil, err } out[key] = jsonValue } return out, nil } // ConvertBSONValueToJSON walks through a document or an array and // converts any BSON value to its corresponding extended JSON type. // It returns the converted JSON document and any error encountered. func ConvertBSONValueToJSON(x interface{}) (interface{}, error) { switch v := x.(type) { case nil: return nil, nil case bool: return v, nil case *bson.M: // document doc, err := convertKeys(*v) if err != nil { return nil, err } return doc, err case bson.M: // document return convertKeys(v) case map[string]interface{}: return convertKeys(v) case bson.D: for i, value := range v { jsonValue, err := ConvertBSONValueToJSON(value.Value) if err != nil { return nil, err } v[i].Value = jsonValue } return MarshalD(v), nil case MarshalD: return v, nil case []interface{}: // array for i, value := range v { jsonValue, err := ConvertBSONValueToJSON(value) if err != nil { return nil, err } v[i] = jsonValue } return v, nil case string: return v, nil // require no conversion case int: return json.NumberInt(v), nil case bson.ObjectId: // ObjectId return json.ObjectId(v.Hex()), nil case time.Time: // Date return json.Date(v.Unix()*1000 + int64(v.Nanosecond()/1e6)), nil case int64: // NumberLong return json.NumberLong(v), nil case int32: // NumberInt return json.NumberInt(v), nil case float64: return json.NumberFloat(v), nil case float32: return json.NumberFloat(float64(v)), nil case []byte: // BinData (with generic type) data := base64.StdEncoding.EncodeToString(v) return json.BinData{0x00, data}, nil case bson.Binary: // BinData data := base64.StdEncoding.EncodeToString(v.Data) return json.BinData{v.Kind, data}, nil case mgo.DBRef: // DBRef return json.DBRef{v.Collection, v.Id, v.Database}, nil case bson.DBPointer: // DBPointer return json.DBPointer{v.Namespace, v.Id}, nil case bson.RegEx: // RegExp return json.RegExp{v.Pattern, v.Options}, nil case bson.MongoTimestamp: // Timestamp timestamp := int64(v) return json.Timestamp{ Seconds: uint32(timestamp >> 32), Increment: uint32(timestamp), }, nil case bson.JavaScript: // JavaScript var scope interface{} var err error if v.Scope != nil { scope, err = ConvertBSONValueToJSON(v.Scope) if err != nil { return nil, err } } return json.JavaScript{v.Code, scope}, nil default: switch x { case bson.MinKey: // MinKey return json.MinKey{}, nil case bson.MaxKey: // MaxKey return json.MaxKey{}, nil case bson.Undefined: // undefined return json.Undefined{}, nil } } return nil, fmt.Errorf("conversion of BSON value '%v' of type '%T' not supported", x, x) } // GetBSONValueAsJSON is equivalent to ConvertBSONValueToJSON, but does not mutate its argument. func GetBSONValueAsJSON(x interface{}) (interface{}, error) { switch v := x.(type) { case nil: return nil, nil case bool: return v, nil case *bson.M: // document doc, err := getConvertedKeys(*v) if err != nil { return nil, err } return doc, err case bson.M: // document return getConvertedKeys(v) case map[string]interface{}: return getConvertedKeys(v) case bson.D: out := bson.D{} for _, value := range v { jsonValue, err := GetBSONValueAsJSON(value.Value) if err != nil { return nil, err } out = append(out, bson.DocElem{value.Name, jsonValue}) } return MarshalD(out), nil case MarshalD: out, err := GetBSONValueAsJSON(bson.D(v)) if err != nil { return nil, err } return MarshalD(out.(bson.D)), nil case []interface{}: // array out := []interface{}{} for _, value := range v { jsonValue, err := GetBSONValueAsJSON(value) if err != nil { return nil, err } out = append(out, jsonValue) } return out, nil case string: return v, nil // require no conversion case int: return json.NumberInt(v), nil case bson.ObjectId: // ObjectId return json.ObjectId(v.Hex()), nil case time.Time: // Date return json.Date(v.Unix()*1000 + int64(v.Nanosecond()/1e6)), nil case int64: // NumberLong return json.NumberLong(v), nil case int32: // NumberInt return json.NumberInt(v), nil case float64: return json.NumberFloat(v), nil case float32: return json.NumberFloat(float64(v)), nil case []byte: // BinData (with generic type) data := base64.StdEncoding.EncodeToString(v) return json.BinData{0x00, data}, nil case bson.Binary: // BinData data := base64.StdEncoding.EncodeToString(v.Data) return json.BinData{v.Kind, data}, nil case mgo.DBRef: // DBRef return json.DBRef{v.Collection, v.Id, v.Database}, nil case bson.DBPointer: // DBPointer return json.DBPointer{v.Namespace, v.Id}, nil case bson.RegEx: // RegExp return json.RegExp{v.Pattern, v.Options}, nil case bson.MongoTimestamp: // Timestamp timestamp := int64(v) return json.Timestamp{ Seconds: uint32(timestamp >> 32), Increment: uint32(timestamp), }, nil case bson.JavaScript: // JavaScript var scope interface{} var err error if v.Scope != nil { scope, err = GetBSONValueAsJSON(v.Scope) if err != nil { return nil, err } } return json.JavaScript{v.Code, scope}, nil default: switch x { case bson.MinKey: // MinKey return json.MinKey{}, nil case bson.MaxKey: // MaxKey return json.MaxKey{}, nil case bson.Undefined: // undefined return json.Undefined{}, nil } } return nil, fmt.Errorf("conversion of BSON value '%v' of type '%T' not supported", x, x) } mongo-tools-r3.2.11/common/bsonutil/converter_test.go000066400000000000000000000240521276133741700227770ustar00rootroot00000000000000package bsonutil import ( "encoding/base64" "fmt" "github.com/mongodb/mongo-tools/common/json" "github.com/mongodb/mongo-tools/common/testutil" . "github.com/smartystreets/goconvey/convey" "gopkg.in/mgo.v2" "gopkg.in/mgo.v2/bson" "testing" "time" ) func TestObjectIdBSONToJSON(t *testing.T) { testutil.VerifyTestType(t, testutil.UnitTestType) Convey("Converting a BSON ObjectId", t, func() { Convey("that is valid to JSON should produce a json.ObjectId", func() { bsonObjId := bson.NewObjectId() jsonObjId := json.ObjectId(bsonObjId.Hex()) _jObjId, err := ConvertBSONValueToJSON(bsonObjId) So(err, ShouldBeNil) jObjId, ok := _jObjId.(json.ObjectId) So(ok, ShouldBeTrue) So(jObjId, ShouldNotEqual, bsonObjId) So(jObjId, ShouldEqual, jsonObjId) }) }) } func TestArraysBSONToJSON(t *testing.T) { testutil.VerifyTestType(t, testutil.UnitTestType) Convey("Converting BSON arrays to JSON arrays", t, func() { Convey("should work for empty arrays", func() { jArr, err := ConvertBSONValueToJSON([]interface{}{}) So(err, ShouldBeNil) So(jArr, ShouldResemble, []interface{}{}) }) Convey("should work for one-level deep arrays", func() { objId := bson.NewObjectId() bsonArr := []interface{}{objId, 28, 0.999, "plain"} _jArr, err := ConvertBSONValueToJSON(bsonArr) So(err, ShouldBeNil) jArr, ok := _jArr.([]interface{}) So(ok, ShouldBeTrue) So(len(jArr), ShouldEqual, 4) So(jArr[0], ShouldEqual, json.ObjectId(objId.Hex())) So(jArr[1], ShouldEqual, 28) So(jArr[2], ShouldEqual, 0.999) So(jArr[3], ShouldEqual, "plain") }) Convey("should work for arrays with embedded objects", func() { bsonObj := []interface{}{80, bson.M{"a": int64(20), "b": bson.M{"c": bson.RegEx{"hi", "i"}}}} __jObj, err := ConvertBSONValueToJSON(bsonObj) So(err, ShouldBeNil) _jObj, ok := __jObj.([]interface{}) So(ok, ShouldBeTrue) jObj, ok := _jObj[1].(bson.M) So(ok, ShouldBeTrue) So(len(jObj), ShouldEqual, 2) So(jObj["a"], ShouldEqual, json.NumberLong(20)) jjObj, ok := jObj["b"].(bson.M) So(ok, ShouldBeTrue) So(jjObj["c"], ShouldResemble, json.RegExp{"hi", "i"}) So(jjObj["c"], ShouldNotResemble, json.RegExp{"i", "hi"}) }) }) } func TestDateBSONToJSON(t *testing.T) { testutil.VerifyTestType(t, testutil.UnitTestType) timeNow := time.Now() secs := int64(timeNow.Unix()) nanosecs := timeNow.Nanosecond() millis := int64(nanosecs / 1e6) timeNowSecs := time.Unix(secs, int64(0)) timeNowMillis := time.Unix(secs, int64(millis*1e6)) Convey("Converting BSON time.Time 's dates to JSON", t, func() { // json.Date is stored as an int64 representing the number of milliseconds since the epoch Convey(fmt.Sprintf("should work with second granularity: %v", timeNowSecs), func() { _jObj, err := ConvertBSONValueToJSON(timeNowSecs) So(err, ShouldBeNil) jObj, ok := _jObj.(json.Date) So(ok, ShouldBeTrue) So(int64(jObj), ShouldEqual, secs*1e3) }) Convey(fmt.Sprintf("should work with millisecond granularity: %v", timeNowMillis), func() { _jObj, err := ConvertBSONValueToJSON(timeNowMillis) So(err, ShouldBeNil) jObj, ok := _jObj.(json.Date) So(ok, ShouldBeTrue) So(int64(jObj), ShouldEqual, secs*1e3+millis) }) Convey(fmt.Sprintf("should work with nanosecond granularity: %v", timeNow), func() { _jObj, err := ConvertBSONValueToJSON(timeNow) So(err, ShouldBeNil) jObj, ok := _jObj.(json.Date) So(ok, ShouldBeTrue) // we lose nanosecond precision So(int64(jObj), ShouldEqual, secs*1e3+millis) }) }) } func TestMaxKeyBSONToJSON(t *testing.T) { testutil.VerifyTestType(t, testutil.UnitTestType) Convey("Converting a BSON Maxkey to JSON", t, func() { Convey("should produce a json.MaxKey", func() { _jObj, err := ConvertBSONValueToJSON(bson.MaxKey) So(err, ShouldBeNil) jObj, ok := _jObj.(json.MaxKey) So(ok, ShouldBeTrue) So(jObj, ShouldResemble, json.MaxKey{}) }) }) } func TestMinKeyBSONToJSON(t *testing.T) { testutil.VerifyTestType(t, testutil.UnitTestType) Convey("Converting a BSON Maxkey to JSON", t, func() { Convey("should produce a json.MinKey", func() { _jObj, err := ConvertBSONValueToJSON(bson.MinKey) So(err, ShouldBeNil) jObj, ok := _jObj.(json.MinKey) So(ok, ShouldBeTrue) So(jObj, ShouldResemble, json.MinKey{}) }) }) } func Test64BitIntBSONToJSON(t *testing.T) { testutil.VerifyTestType(t, testutil.UnitTestType) Convey("Converting a BSON int64 to JSON", t, func() { Convey("should produce a json.NumberLong", func() { _jObj, err := ConvertBSONValueToJSON(int32(243)) So(err, ShouldBeNil) jObj, ok := _jObj.(json.NumberInt) So(ok, ShouldBeTrue) So(jObj, ShouldEqual, json.NumberInt(243)) }) }) } func Test32BitIntBSONToJSON(t *testing.T) { testutil.VerifyTestType(t, testutil.UnitTestType) Convey("Converting a BSON int32 integer to JSON", t, func() { Convey("should produce a json.NumberInt", func() { _jObj, err := ConvertBSONValueToJSON(int64(888234334343)) So(err, ShouldBeNil) jObj, ok := _jObj.(json.NumberLong) So(ok, ShouldBeTrue) So(jObj, ShouldEqual, json.NumberLong(888234334343)) }) }) } func TestRegExBSONToJSON(t *testing.T) { testutil.VerifyTestType(t, testutil.UnitTestType) Convey("Converting a BSON Regular Expression (= /decision/gi) to JSON", t, func() { Convey("should produce a json.RegExp", func() { _jObj, err := ConvertBSONValueToJSON(bson.RegEx{"decision", "gi"}) So(err, ShouldBeNil) jObj, ok := _jObj.(json.RegExp) So(ok, ShouldBeTrue) So(jObj, ShouldResemble, json.RegExp{"decision", "gi"}) }) }) } func TestUndefinedValueBSONToJSON(t *testing.T) { testutil.VerifyTestType(t, testutil.UnitTestType) Convey("Converting a BSON Undefined type to JSON", t, func() { Convey("should produce a json.Undefined", func() { _jObj, err := ConvertBSONValueToJSON(bson.Undefined) So(err, ShouldBeNil) jObj, ok := _jObj.(json.Undefined) So(ok, ShouldBeTrue) So(jObj, ShouldResemble, json.Undefined{}) }) }) } func TestDBRefBSONToJSON(t *testing.T) { testutil.VerifyTestType(t, testutil.UnitTestType) Convey("Converting BSON DBRef to JSON", t, func() { Convey("should produce a json.DBRef", func() { _jObj, err := ConvertBSONValueToJSON(mgo.DBRef{"coll1", "some_id", "test"}) So(err, ShouldBeNil) jObj, ok := _jObj.(json.DBRef) So(ok, ShouldBeTrue) So(jObj, ShouldResemble, json.DBRef{"coll1", "some_id", "test"}) So(jObj, ShouldNotResemble, json.DBRef{"coll1", "test", "some_id"}) }) }) } func TestTimestampBSONToJSON(t *testing.T) { testutil.VerifyTestType(t, testutil.UnitTestType) Convey("Converting a BSON Timestamp to JSON", t, func() { Convey("should produce a json.Timestamp", func() { // {t:803434343, i:9} == bson.MongoTimestamp(803434343*2**32 + 9) _jObj, err := ConvertBSONValueToJSON(bson.MongoTimestamp(uint64(803434343<<32) | uint64(9))) So(err, ShouldBeNil) jObj, ok := _jObj.(json.Timestamp) So(ok, ShouldBeTrue) So(jObj, ShouldResemble, json.Timestamp{Seconds: 803434343, Increment: 9}) So(jObj, ShouldNotResemble, json.Timestamp{Seconds: 803434343, Increment: 8}) }) }) } func TestBinaryBSONToJSON(t *testing.T) { testutil.VerifyTestType(t, testutil.UnitTestType) Convey("Converting BSON Binary data to JSON", t, func() { Convey("should produce a json.BinData", func() { _jObj, err := ConvertBSONValueToJSON(bson.Binary{'\x01', []byte("\x05\x20\x02\xae\xf7")}) So(err, ShouldBeNil) jObj, ok := _jObj.(json.BinData) So(ok, ShouldBeTrue) base64data1 := base64.StdEncoding.EncodeToString([]byte("\x05\x20\x02\xae\xf7")) base64data2 := base64.StdEncoding.EncodeToString([]byte("\x05\x20\x02\xaf\xf7")) So(jObj, ShouldResemble, json.BinData{'\x01', base64data1}) So(jObj, ShouldNotResemble, json.BinData{'\x01', base64data2}) }) }) } func TestGenericBytesBSONToJSON(t *testing.T) { testutil.VerifyTestType(t, testutil.UnitTestType) Convey("Converting Go bytes to JSON", t, func() { Convey("should produce a json.BinData with Type=0x00 (Generic)", func() { _jObj, err := ConvertBSONValueToJSON([]byte("this is something that's cool")) So(err, ShouldBeNil) jObj, ok := _jObj.(json.BinData) So(ok, ShouldBeTrue) base64data := base64.StdEncoding.EncodeToString([]byte("this is something that's cool")) So(jObj, ShouldResemble, json.BinData{0x00, base64data}) So(jObj, ShouldNotResemble, json.BinData{0x01, base64data}) }) }) } func TestUnknownBSONTypeToJSON(t *testing.T) { testutil.VerifyTestType(t, testutil.UnitTestType) Convey("Converting an unknown BSON type to JSON", t, func() { Convey("should produce an error", func() { _, err := ConvertBSONValueToJSON(func() {}) So(err, ShouldNotBeNil) }) }) } func TestDBPointerBSONToJSON(t *testing.T) { testutil.VerifyTestType(t, testutil.UnitTestType) Convey("Converting BSON DBPointer to JSON", t, func() { Convey("should produce a json.DBPointer", func() { objId := bson.NewObjectId() _jObj, err := ConvertBSONValueToJSON(bson.DBPointer{"dbrefnamespace", objId}) So(err, ShouldBeNil) jObj, ok := _jObj.(json.DBPointer) So(ok, ShouldBeTrue) So(jObj, ShouldResemble, json.DBPointer{"dbrefnamespace", objId}) }) }) } func TestJSCodeBSONToJSON(t *testing.T) { testutil.VerifyTestType(t, testutil.UnitTestType) Convey("Converting BSON Javascript code to JSON", t, func() { Convey("should produce a json.Javascript", func() { Convey("without scope if the scope for the BSON Javascript code is nil", func() { _jObj, err := ConvertBSONValueToJSON(bson.JavaScript{"function() { return null; }", nil}) So(err, ShouldBeNil) jObj, ok := _jObj.(json.JavaScript) So(ok, ShouldBeTrue) So(jObj, ShouldResemble, json.JavaScript{"function() { return null; }", nil}) }) Convey("with scope if the scope for the BSON Javascript code is non-nil", func() { _jObj, err := ConvertBSONValueToJSON(bson.JavaScript{"function() { return x; }", bson.M{"x": 2}}) So(err, ShouldBeNil) jObj, ok := _jObj.(json.JavaScript) So(ok, ShouldBeTrue) So(jObj.Scope.(bson.M)["x"], ShouldEqual, 2) So(jObj.Code, ShouldEqual, "function() { return x; }") }) }) }) } mongo-tools-r3.2.11/common/bsonutil/date_test.go000066400000000000000000000102241276133741700217010ustar00rootroot00000000000000package bsonutil import ( "fmt" "github.com/mongodb/mongo-tools/common/json" . "github.com/smartystreets/goconvey/convey" "testing" "time" ) func TestDateValue(t *testing.T) { Convey("When converting JSON with Date values", t, func() { Convey("works for Date object", func() { key := "key" jsonMap := map[string]interface{}{ key: json.Date(100), } err := ConvertJSONDocumentToBSON(jsonMap) So(err, ShouldBeNil) jsonValue, ok := jsonMap[key].(time.Time) So(ok, ShouldBeTrue) So(jsonValue.Equal(time.Unix(0, int64(100*time.Millisecond))), ShouldBeTrue) }) Convey("works for Date document", func() { dates := []string{ "2006-01-02T15:04:05.000Z", "2006-01-02T15:04:05.000-0700", "2006-01-02T15:04:05Z", "2006-01-02T15:04:05-0700", "2006-01-02T15:04Z", "2006-01-02T15:04-0700", } for _, dateString := range dates { example := fmt.Sprintf(`{ "$date": "%v" }`, dateString) Convey(fmt.Sprintf("of string ('%v')", example), func() { key := "key" jsonMap := map[string]interface{}{ key: map[string]interface{}{ "$date": dateString, }, } err := ConvertJSONDocumentToBSON(jsonMap) So(err, ShouldBeNil) // dateString is a valid time format string date, err := time.Parse(dateString, dateString) So(err, ShouldBeNil) jsonValue, ok := jsonMap[key].(time.Time) So(ok, ShouldBeTrue) So(jsonValue.Equal(date), ShouldBeTrue) }) } date := time.Unix(0, int64(time.Duration(1136214245000)*time.Millisecond)) Convey(`of $numberLong ('{ "$date": { "$numberLong": "1136214245000" } }')`, func() { key := "key" jsonMap := map[string]interface{}{ key: map[string]interface{}{ "$date": map[string]interface{}{ "$numberLong": "1136214245000", }, }, } err := ConvertJSONDocumentToBSON(jsonMap) So(err, ShouldBeNil) jsonValue, ok := jsonMap[key].(time.Time) So(ok, ShouldBeTrue) So(jsonValue.Equal(date), ShouldBeTrue) }) Convey(`of json.Number ('{ "$date": 1136214245000 }')`, func() { key := "key" jsonMap := map[string]interface{}{ key: map[string]interface{}{ "$date": json.Number("1136214245000"), }, } err := ConvertJSONDocumentToBSON(jsonMap) So(err, ShouldBeNil) jsonValue, ok := jsonMap[key].(time.Time) So(ok, ShouldBeTrue) So(jsonValue.Equal(date), ShouldBeTrue) }) Convey(`of numeric int64 ('{ "$date": 1136214245000 }')`, func() { key := "key" jsonMap := map[string]interface{}{ key: map[string]interface{}{ "$date": int64(1136214245000), }, } err := ConvertJSONDocumentToBSON(jsonMap) So(err, ShouldBeNil) jsonValue, ok := jsonMap[key].(time.Time) So(ok, ShouldBeTrue) So(jsonValue.Equal(date), ShouldBeTrue) }) Convey(`of numeric float64 ('{ "$date": 1136214245000 }')`, func() { key := "key" jsonMap := map[string]interface{}{ key: map[string]interface{}{ "$date": float64(1136214245000), }, } err := ConvertJSONDocumentToBSON(jsonMap) So(err, ShouldBeNil) jsonValue, ok := jsonMap[key].(time.Time) So(ok, ShouldBeTrue) So(jsonValue.Equal(date), ShouldBeTrue) }) Convey(`of numeric int32 ('{ "$date": 2136800000 }')`, func() { key := "key" date = time.Unix(0, int64(time.Duration(2136800000)*time.Millisecond)) jsonMap := map[string]interface{}{ key: map[string]interface{}{ "$date": int32(2136800000), }, } err := ConvertJSONDocumentToBSON(jsonMap) So(err, ShouldBeNil) jsonValue, ok := jsonMap[key].(time.Time) So(ok, ShouldBeTrue) So(jsonValue.Equal(date), ShouldBeTrue) }) Convey(`of negative numeric int32 ('{ "$date": -2136800000 }')`, func() { key := "key" date = time.Unix(0, int64(time.Duration(-2136800000)*time.Millisecond)) jsonMap := map[string]interface{}{ key: map[string]interface{}{ "$date": int32(-2136800000), }, } err := ConvertJSONDocumentToBSON(jsonMap) So(err, ShouldBeNil) jsonValue, ok := jsonMap[key].(time.Time) So(ok, ShouldBeTrue) So(jsonValue.Equal(date), ShouldBeTrue) }) }) }) } mongo-tools-r3.2.11/common/bsonutil/marshal_d.go000066400000000000000000000030221276133741700216550ustar00rootroot00000000000000package bsonutil import ( "bytes" "fmt" "github.com/mongodb/mongo-tools/common/json" "github.com/mongodb/mongo-tools/common/util" "gopkg.in/mgo.v2/bson" ) // MarshalD is a wrapper for bson.D that allows unmarshalling // of bson.D with preserved order. Necessary for printing // certain database commands. type MarshalD bson.D // MarshalJSON makes the MarshalD type usable by // the encoding/json package. func (md MarshalD) MarshalJSON() ([]byte, error) { var buff bytes.Buffer buff.WriteString("{") for i, item := range md { key, err := json.Marshal(item.Name) if err != nil { return nil, fmt.Errorf("cannot marshal key %v: %v", item.Name, err) } val, err := json.Marshal(item.Value) if err != nil { return nil, fmt.Errorf("cannot marshal value %v: %v", item.Value, err) } buff.Write(key) buff.WriteString(":") buff.Write(val) if i != len(md)-1 { buff.WriteString(",") } } buff.WriteString("}") return buff.Bytes(), nil } // MakeSortString takes a bson.D object and converts it to a slice of strings // that can be used as the input args to mgo's .Sort(...) function. // For example: // {a:1, b:-1} -> ["+a", "-b"] func MakeSortString(sortObj bson.D) ([]string, error) { sortStrs := make([]string, 0, len(sortObj)) for _, docElem := range sortObj { valueAsNumber, err := util.ToFloat64(docElem.Value) if err != nil { return nil, err } prefix := "+" if valueAsNumber < 0 { prefix = "-" } sortStrs = append(sortStrs, fmt.Sprintf("%v%v", prefix, docElem.Name)) } return sortStrs, nil } mongo-tools-r3.2.11/common/bsonutil/marshal_d_test.go000066400000000000000000000061761276133741700227310ustar00rootroot00000000000000package bsonutil import ( "encoding/json" . "github.com/smartystreets/goconvey/convey" "gopkg.in/mgo.v2/bson" "strings" "testing" ) func TestMarshalDMarshalJSON(t *testing.T) { Convey("With a valid bson.D", t, func() { testD := bson.D{ {"cool", "rad"}, {"aaa", 543.2}, {"I", 0}, {"E", 0}, {"map", bson.M{"1": 1, "2": "two"}}, } Convey("wrapping with MarshalD should allow json.Marshal to work", func() { asJSON, err := json.Marshal(MarshalD(testD)) So(err, ShouldBeNil) strJSON := string(asJSON) Convey("with order preserved", func() { So(strings.Index(strJSON, "cool"), ShouldBeLessThan, strings.Index(strJSON, "aaa")) So(strings.Index(strJSON, "aaa"), ShouldBeLessThan, strings.Index(strJSON, "I")) So(strings.Index(strJSON, "I"), ShouldBeLessThan, strings.Index(strJSON, "E")) So(strings.Index(strJSON, "E"), ShouldBeLessThan, strings.Index(strJSON, "map")) So(strings.Count(strJSON, ","), ShouldEqual, 5) // 4 + 1 from internal map }) Convey("but still usable by the json parser", func() { var asMap bson.M err := json.Unmarshal(asJSON, &asMap) So(err, ShouldBeNil) Convey("with types & values preserved", func() { So(asMap["cool"], ShouldEqual, "rad") So(asMap["aaa"], ShouldEqual, 543.2) So(asMap["I"], ShouldEqual, 0) So(asMap["E"], ShouldEqual, 0) So(asMap["map"].(map[string]interface{})["1"], ShouldEqual, 1) So(asMap["map"].(map[string]interface{})["2"], ShouldEqual, "two") }) }) Convey("putting it inside another map should still be usable by json.Marshal", func() { _, err := json.Marshal(bson.M{"x": 0, "y": MarshalD(testD)}) So(err, ShouldBeNil) }) }) }) Convey("With en empty bson.D", t, func() { testD := bson.D{} Convey("wrapping with MarshalD should allow json.Marshal to work", func() { asJSON, err := json.Marshal(MarshalD(testD)) So(err, ShouldBeNil) strJSON := string(asJSON) So(strJSON, ShouldEqual, "{}") Convey("but still usable by the json parser", func() { var asInterface interface{} err := json.Unmarshal(asJSON, &asInterface) So(err, ShouldBeNil) asMap, ok := asInterface.(map[string]interface{}) So(ok, ShouldBeTrue) So(len(asMap), ShouldEqual, 0) }) }) }) } func TestFindValueByKey(t *testing.T) { Convey("Given a bson.D document and a specific key", t, func() { subDocument := &bson.D{ bson.DocElem{"field4", "c"}, } document := &bson.D{ bson.DocElem{"field1", "a"}, bson.DocElem{"field2", "b"}, bson.DocElem{"field3", subDocument}, } Convey("the corresponding value top-level keys should be returned", func() { value, err := FindValueByKey("field1", document) So(value, ShouldEqual, "a") So(err, ShouldBeNil) }) Convey("the corresponding value top-level keys with sub-document values should be returned", func() { value, err := FindValueByKey("field3", document) So(value, ShouldEqual, subDocument) So(err, ShouldBeNil) }) Convey("for non-existent keys nil and an error should be returned", func() { value, err := FindValueByKey("field4", document) So(value, ShouldBeNil) So(err, ShouldNotBeNil) }) }) } mongo-tools-r3.2.11/common/bsonutil/maxkey_test.go000066400000000000000000000014751276133741700222720ustar00rootroot00000000000000package bsonutil import ( "github.com/mongodb/mongo-tools/common/json" . "github.com/smartystreets/goconvey/convey" "gopkg.in/mgo.v2/bson" "testing" ) func TestMaxKeyValue(t *testing.T) { Convey("When converting JSON with MaxKey values", t, func() { Convey("works for MaxKey literal", func() { key := "key" jsonMap := map[string]interface{}{ key: json.MaxKey{}, } err := ConvertJSONDocumentToBSON(jsonMap) So(err, ShouldBeNil) So(jsonMap[key], ShouldResemble, bson.MaxKey) }) Convey(`works for MaxKey document ('{ "$maxKey": 1 }')`, func() { key := "key" jsonMap := map[string]interface{}{ key: map[string]interface{}{ "$maxKey": 1, }, } err := ConvertJSONDocumentToBSON(jsonMap) So(err, ShouldBeNil) So(jsonMap[key], ShouldResemble, bson.MaxKey) }) }) } mongo-tools-r3.2.11/common/bsonutil/minkey_test.go000066400000000000000000000014751276133741700222700ustar00rootroot00000000000000package bsonutil import ( "github.com/mongodb/mongo-tools/common/json" . "github.com/smartystreets/goconvey/convey" "gopkg.in/mgo.v2/bson" "testing" ) func TestMinKeyValue(t *testing.T) { Convey("When converting JSON with MinKey values", t, func() { Convey("works for MinKey literal", func() { key := "key" jsonMap := map[string]interface{}{ key: json.MinKey{}, } err := ConvertJSONDocumentToBSON(jsonMap) So(err, ShouldBeNil) So(jsonMap[key], ShouldResemble, bson.MinKey) }) Convey(`works for MinKey document ('{ "$minKey": 1 }')`, func() { key := "key" jsonMap := map[string]interface{}{ key: map[string]interface{}{ "$minKey": 1, }, } err := ConvertJSONDocumentToBSON(jsonMap) So(err, ShouldBeNil) So(jsonMap[key], ShouldResemble, bson.MinKey) }) }) } mongo-tools-r3.2.11/common/bsonutil/number.go000066400000000000000000000005451276133741700212220ustar00rootroot00000000000000package bsonutil import ( "fmt" "reflect" ) var floatType = reflect.TypeOf(float64(0)) func getFloat(unk interface{}) (float64, error) { v := reflect.ValueOf(unk) v = reflect.Indirect(v) if !v.Type().ConvertibleTo(floatType) { return 0, fmt.Errorf("cannot convert %v to float64", v.Type()) } fv := v.Convert(floatType) return fv.Float(), nil } mongo-tools-r3.2.11/common/bsonutil/numberint_test.go000066400000000000000000000014741276133741700227760ustar00rootroot00000000000000package bsonutil import ( "github.com/mongodb/mongo-tools/common/json" . "github.com/smartystreets/goconvey/convey" "testing" ) func TestNumberIntValue(t *testing.T) { Convey("When converting JSON with NumberInt values", t, func() { Convey("works for NumberInt constructor", func() { key := "key" jsonMap := map[string]interface{}{ key: json.NumberInt(42), } err := ConvertJSONDocumentToBSON(jsonMap) So(err, ShouldBeNil) So(jsonMap[key], ShouldEqual, int32(42)) }) Convey(`works for NumberInt document ('{ "$numberInt": "42" }')`, func() { key := "key" jsonMap := map[string]interface{}{ key: map[string]interface{}{ "$numberInt": "42", }, } err := ConvertJSONDocumentToBSON(jsonMap) So(err, ShouldBeNil) So(jsonMap[key], ShouldEqual, int32(42)) }) }) } mongo-tools-r3.2.11/common/bsonutil/numberlong_test.go000066400000000000000000000015031276133741700231340ustar00rootroot00000000000000package bsonutil import ( "github.com/mongodb/mongo-tools/common/json" . "github.com/smartystreets/goconvey/convey" "testing" ) func TestNumberLongValue(t *testing.T) { Convey("When converting JSON with NumberLong values", t, func() { Convey("works for NumberLong constructor", func() { key := "key" jsonMap := map[string]interface{}{ key: json.NumberLong(42), } err := ConvertJSONDocumentToBSON(jsonMap) So(err, ShouldBeNil) So(jsonMap[key], ShouldEqual, int64(42)) }) Convey(`works for NumberLong document ('{ "$numberLong": "42" }')`, func() { key := "key" jsonMap := map[string]interface{}{ key: map[string]interface{}{ "$numberLong": "42", }, } err := ConvertJSONDocumentToBSON(jsonMap) So(err, ShouldBeNil) So(jsonMap[key], ShouldEqual, int64(42)) }) }) } mongo-tools-r3.2.11/common/bsonutil/objectid_test.go000066400000000000000000000017151276133741700225540ustar00rootroot00000000000000package bsonutil import ( "github.com/mongodb/mongo-tools/common/json" . "github.com/smartystreets/goconvey/convey" "gopkg.in/mgo.v2/bson" "testing" ) func TestObjectIdValue(t *testing.T) { Convey("When converting JSON with ObjectId values", t, func() { Convey("works for ObjectId constructor", func() { key := "key" jsonMap := map[string]interface{}{ key: json.ObjectId("0123456789abcdef01234567"), } err := ConvertJSONDocumentToBSON(jsonMap) So(err, ShouldBeNil) So(jsonMap[key], ShouldEqual, bson.ObjectIdHex("0123456789abcdef01234567")) }) Convey(`works for ObjectId document ('{ "$oid": "0123456789abcdef01234567" }')`, func() { key := "key" jsonMap := map[string]interface{}{ key: map[string]interface{}{ "$oid": "0123456789abcdef01234567", }, } err := ConvertJSONDocumentToBSON(jsonMap) So(err, ShouldBeNil) So(jsonMap[key], ShouldEqual, bson.ObjectIdHex("0123456789abcdef01234567")) }) }) } mongo-tools-r3.2.11/common/bsonutil/regexp_test.go000066400000000000000000000031031276133741700222540ustar00rootroot00000000000000package bsonutil import ( "github.com/mongodb/mongo-tools/common/json" . "github.com/smartystreets/goconvey/convey" "gopkg.in/mgo.v2/bson" "testing" ) func TestRegExpValue(t *testing.T) { Convey("When converting JSON with RegExp values", t, func() { Convey("works for RegExp constructor", func() { key := "key" jsonMap := map[string]interface{}{ key: json.RegExp{"foo", "i"}, } err := ConvertJSONDocumentToBSON(jsonMap) So(err, ShouldBeNil) So(jsonMap[key], ShouldResemble, bson.RegEx{"foo", "i"}) }) Convey(`works for RegExp document ('{ "$regex": "foo", "$options": "i" }')`, func() { key := "key" jsonMap := map[string]interface{}{ key: map[string]interface{}{ "$regex": "foo", "$options": "i", }, } err := ConvertJSONDocumentToBSON(jsonMap) So(err, ShouldBeNil) So(jsonMap[key], ShouldResemble, bson.RegEx{"foo", "i"}) }) Convey(`can use multiple options ('{ "$regex": "bar", "$options": "gims" }')`, func() { key := "key" jsonMap := map[string]interface{}{ key: map[string]interface{}{ "$regex": "bar", "$options": "gims", }, } err := ConvertJSONDocumentToBSON(jsonMap) So(err, ShouldBeNil) So(jsonMap[key], ShouldResemble, bson.RegEx{"bar", "gims"}) }) Convey(`fails for an invalid option ('{ "$regex": "baz", "$options": "y" }')`, func() { key := "key" jsonMap := map[string]interface{}{ key: map[string]interface{}{ "$regex": "baz", "$options": "y", }, } err := ConvertJSONDocumentToBSON(jsonMap) So(err, ShouldNotBeNil) }) }) } mongo-tools-r3.2.11/common/bsonutil/timestamp_test.go000066400000000000000000000017761276133741700230030ustar00rootroot00000000000000package bsonutil import ( "github.com/mongodb/mongo-tools/common/json" . "github.com/smartystreets/goconvey/convey" "gopkg.in/mgo.v2/bson" "testing" ) func TestTimestampValue(t *testing.T) { Convey("When converting JSON with Timestamp values", t, func() { testTS := bson.MongoTimestamp(123456<<32 | 55) Convey("works for Timestamp literal", func() { jsonMap := map[string]interface{}{ "ts": json.Timestamp{123456, 55}, } err := ConvertJSONDocumentToBSON(jsonMap) So(err, ShouldBeNil) So(jsonMap["ts"], ShouldEqual, testTS) }) Convey(`works for Timestamp document`, func() { Convey(`{"ts":{"$timestamp":{"t":123456, "i":55}}}`, func() { jsonMap := map[string]interface{}{ "ts": map[string]interface{}{ "$timestamp": map[string]interface{}{ "t": 123456.0, "i": 55.0, }, }, } bsonMap, err := ConvertJSONValueToBSON(jsonMap) So(err, ShouldBeNil) So(bsonMap.(map[string]interface{})["ts"], ShouldEqual, testTS) }) }) }) } mongo-tools-r3.2.11/common/bsonutil/undefined_test.go000066400000000000000000000015361276133741700227330ustar00rootroot00000000000000package bsonutil import ( "github.com/mongodb/mongo-tools/common/json" . "github.com/smartystreets/goconvey/convey" "gopkg.in/mgo.v2/bson" "testing" ) func TestUndefinedValue(t *testing.T) { Convey("When converting JSON with undefined values", t, func() { Convey("works for undefined literal", func() { key := "key" jsonMap := map[string]interface{}{ key: json.Undefined{}, } err := ConvertJSONDocumentToBSON(jsonMap) So(err, ShouldBeNil) So(jsonMap[key], ShouldResemble, bson.Undefined) }) Convey(`works for undefined document ('{ "$undefined": true }')`, func() { key := "key" jsonMap := map[string]interface{}{ key: map[string]interface{}{ "$undefined": true, }, } err := ConvertJSONDocumentToBSON(jsonMap) So(err, ShouldBeNil) So(jsonMap[key], ShouldResemble, bson.Undefined) }) }) } mongo-tools-r3.2.11/common/common.go000066400000000000000000000001421276133741700173540ustar00rootroot00000000000000// Package common contains subpackages that are shared amongst the mongo // tools. package common mongo-tools-r3.2.11/common/db/000077500000000000000000000000001276133741700161255ustar00rootroot00000000000000mongo-tools-r3.2.11/common/db/bson_stream.go000066400000000000000000000070661276133741700210010ustar00rootroot00000000000000package db import ( "fmt" "gopkg.in/mgo.v2/bson" "io" ) // BSONSource reads documents from the underlying io.ReadCloser, Stream which // wraps a stream of BSON documents. type BSONSource struct { reusableBuf []byte Stream io.ReadCloser err error } // DecodedBSONSource reads documents from the underlying io.ReadCloser, Stream which // wraps a stream of BSON documents. type DecodedBSONSource struct { RawDocSource err error } // RawDocSource wraps basic functions for reading a BSON source file. type RawDocSource interface { LoadNext() []byte Close() error Err() error } // NewBSONSource creates a BSONSource with a reusable I/O buffer func NewBSONSource(in io.ReadCloser) *BSONSource { return &BSONSource{make([]byte, MaxBSONSize), in, nil} } // NewBufferlessBSONSource creates a BSONSource without a reusable I/O buffer func NewBufferlessBSONSource(in io.ReadCloser) *BSONSource { return &BSONSource{nil, in, nil} } // Close closes the BSONSource, rendering it unusable for I/O. // It returns an error, if any. func (bs *BSONSource) Close() error { return bs.Stream.Close() } func NewDecodedBSONSource(ds RawDocSource) *DecodedBSONSource { return &DecodedBSONSource{ds, nil} } // Err returns any error in the DecodedBSONSource or its RawDocSource. func (dbs *DecodedBSONSource) Err() error { if dbs.err != nil { return dbs.err } return dbs.RawDocSource.Err() } // Next unmarshals the next BSON document into result. Returns true if no errors // are encountered and false otherwise. func (dbs *DecodedBSONSource) Next(result interface{}) bool { doc := dbs.LoadNext() if doc == nil { return false } if err := bson.Unmarshal(doc, result); err != nil { dbs.err = err return false } dbs.err = nil return true } // LoadNext reads and returns the next BSON document in the stream. If the // BSONSource was created with NewBSONSource then each returned []byte will be // a slice of a single reused I/O buffer. If the BSONSource was created with // NewBufferlessBSONSource then each returend []byte will be individually // allocated func (bs *BSONSource) LoadNext() []byte { var into []byte if bs.reusableBuf == nil { into = make([]byte, 4) } else { into = bs.reusableBuf } // read the bson object size (a 4 byte integer) _, err := io.ReadAtLeast(bs.Stream, into[0:4], 4) if err != nil { if err != io.EOF { bs.err = err return nil } // we hit EOF right away, so we're at the end of the stream. bs.err = nil return nil } bsonSize := int32( (uint32(into[0]) << 0) | (uint32(into[1]) << 8) | (uint32(into[2]) << 16) | (uint32(into[3]) << 24), ) // Verify that the size of the BSON object we are about to read can // actually fit into the buffer that was provided. If not, either the BSON is // invalid, or the buffer passed in is too small. // Verify that we do not have an invalid BSON document with size < 5. if bsonSize > MaxBSONSize || bsonSize < 5 { bs.err = fmt.Errorf("invalid BSONSize: %v bytes", bsonSize) return nil } if int(bsonSize) > cap(into) { bigInto := make([]byte, bsonSize) copy(bigInto, into) into = bigInto if bs.reusableBuf != nil { bs.reusableBuf = bigInto } } into = into[:int(bsonSize)] _, err = io.ReadAtLeast(bs.Stream, into[4:], int(bsonSize-4)) if err != nil { if err != io.EOF { bs.err = err return nil } // this case means we hit EOF but read a partial document, // so there's a broken doc in the stream. Treat this as error. bs.err = fmt.Errorf("invalid bson: %v", err) return nil } bs.err = nil return into } func (bs *BSONSource) Err() error { return bs.err } mongo-tools-r3.2.11/common/db/bson_stream_test.go000066400000000000000000000021061276133741700220260ustar00rootroot00000000000000package db import ( "bytes" . "github.com/smartystreets/goconvey/convey" "gopkg.in/mgo.v2/bson" "io/ioutil" "testing" ) func TestBufferlessBSONSource(t *testing.T) { var testValues = []bson.M{ {"_": bson.Binary{0x80, []byte("apples")}}, {"_": bson.Binary{0x80, []byte("bananas")}}, {"_": bson.Binary{0x80, []byte("cherries")}}, } Convey("with a buffer containing several bson documents with binary fields", t, func() { writeBuf := bytes.NewBuffer(make([]byte, 0, 1024)) for _, tv := range testValues { data, err := bson.Marshal(&tv) So(err, ShouldBeNil) _, err = writeBuf.Write(data) So(err, ShouldBeNil) } Convey("that we parse correctly with a BufferlessBSONSource", func() { bsonSource := NewDecodedBSONSource( NewBufferlessBSONSource(ioutil.NopCloser(writeBuf))) docs := []bson.M{} count := 0 doc := &bson.M{} for bsonSource.Next(doc) { count++ docs = append(docs, *doc) doc = &bson.M{} } So(bsonSource.Err(), ShouldBeNil) So(count, ShouldEqual, len(testValues)) So(docs, ShouldResemble, testValues) }) }) } mongo-tools-r3.2.11/common/db/buffered_bulk.go000066400000000000000000000037731276133741700212650ustar00rootroot00000000000000package db import ( "fmt" "gopkg.in/mgo.v2" "gopkg.in/mgo.v2/bson" ) // BufferedBulkInserter implements a bufio.Writer-like design for queuing up // documents and inserting them in bulk when the given doc limit (or max // message size) is reached. Must be flushed at the end to ensure that all // documents are written. type BufferedBulkInserter struct { bulk *mgo.Bulk collection *mgo.Collection continueOnError bool docLimit int byteCount int docCount int unordered bool } // NewBufferedBulkInserter returns an initialized BufferedBulkInserter // for writing. func NewBufferedBulkInserter(collection *mgo.Collection, docLimit int, continueOnError bool) *BufferedBulkInserter { bb := &BufferedBulkInserter{ collection: collection, continueOnError: continueOnError, docLimit: docLimit, } bb.resetBulk() return bb } func (bb *BufferedBulkInserter) Unordered() { bb.unordered = true bb.bulk.Unordered() } // throw away the old bulk and init a new one func (bb *BufferedBulkInserter) resetBulk() { bb.bulk = bb.collection.Bulk() if bb.continueOnError || bb.unordered { bb.bulk.Unordered() } bb.byteCount = 0 bb.docCount = 0 } // Insert adds a document to the buffer for bulk insertion. If the buffer is // full, the bulk insert is made, returning any error that occurs. func (bb *BufferedBulkInserter) Insert(doc interface{}) error { rawBytes, err := bson.Marshal(doc) if err != nil { return fmt.Errorf("bson encoding error: %v", err) } // flush if we are full if bb.docCount >= bb.docLimit || bb.byteCount+len(rawBytes) > MaxBSONSize { err = bb.Flush() } // buffer the document bb.docCount++ bb.byteCount += len(rawBytes) bb.bulk.Insert(bson.Raw{Data: rawBytes}) return err } // Flush writes all buffered documents in one bulk insert then resets the buffer. func (bb *BufferedBulkInserter) Flush() error { if bb.docCount == 0 { return nil } defer bb.resetBulk() if _, err := bb.bulk.Run(); err != nil { return err } return nil } mongo-tools-r3.2.11/common/db/buffered_bulk_test.go000066400000000000000000000057451276133741700223250ustar00rootroot00000000000000package db import ( "github.com/mongodb/mongo-tools/common/options" "github.com/mongodb/mongo-tools/common/testutil" . "github.com/smartystreets/goconvey/convey" "gopkg.in/mgo.v2/bson" "testing" ) func TestBufferedBulkInserterInserts(t *testing.T) { var bufBulk *BufferedBulkInserter testutil.VerifyTestType(t, "db") Convey("With a valid session", t, func() { opts := options.ToolOptions{ Connection: &options.Connection{ Port: DefaultTestPort, }, SSL: &options.SSL{}, Auth: &options.Auth{}, } provider, err := NewSessionProvider(opts) session, err := provider.GetSession() So(session, ShouldNotBeNil) So(err, ShouldBeNil) Convey("using a test collection and a doc limit of 3", func() { testCol := session.DB("tools-test").C("bulk1") bufBulk = NewBufferedBulkInserter(testCol, 3, false) So(bufBulk, ShouldNotBeNil) Convey("inserting 10 documents into the BufferedBulkInserter", func() { flushCount := 0 for i := 0; i < 10; i++ { So(bufBulk.Insert(bson.D{}), ShouldBeNil) if bufBulk.docCount%3 == 0 { flushCount++ } } Convey("should have flushed 3 times with one doc still buffered", func() { So(flushCount, ShouldEqual, 3) So(bufBulk.byteCount, ShouldBeGreaterThan, 0) So(bufBulk.docCount, ShouldEqual, 1) }) }) }) Convey("using a test collection and a doc limit of 1", func() { testCol := session.DB("tools-test").C("bulk2") bufBulk = NewBufferedBulkInserter(testCol, 1, false) So(bufBulk, ShouldNotBeNil) Convey("inserting 10 documents into the BufferedBulkInserter and flushing", func() { for i := 0; i < 10; i++ { So(bufBulk.Insert(bson.D{}), ShouldBeNil) } So(bufBulk.Flush(), ShouldBeNil) Convey("should have no docs buffered", func() { So(bufBulk.docCount, ShouldEqual, 0) So(bufBulk.byteCount, ShouldEqual, 0) }) }) }) Convey("using a test collection and a doc limit of 1000", func() { testCol := session.DB("tools-test").C("bulk3") bufBulk = NewBufferedBulkInserter(testCol, 100, false) So(bufBulk, ShouldNotBeNil) Convey("inserting 1,000,000 documents into the BufferedBulkInserter and flushing", func() { session.SetSocketTimeout(0) for i := 0; i < 1000000; i++ { bufBulk.Insert(bson.M{"_id": i}) } So(bufBulk.Flush(), ShouldBeNil) Convey("should have inserted all of the documents", func() { count, err := testCol.Count() So(err, ShouldBeNil) So(count, ShouldEqual, 1000000) // test values testDoc := bson.M{} err = testCol.Find(bson.M{"_id": 477232}).One(&testDoc) So(err, ShouldBeNil) So(testDoc["_id"], ShouldEqual, 477232) err = testCol.Find(bson.M{"_id": 999999}).One(&testDoc) So(err, ShouldBeNil) So(testDoc["_id"], ShouldEqual, 999999) err = testCol.Find(bson.M{"_id": 1}).One(&testDoc) So(err, ShouldBeNil) So(testDoc["_id"], ShouldEqual, 1) }) }) }) Reset(func() { session.DB("tools-test").DropDatabase() }) }) } mongo-tools-r3.2.11/common/db/command.go000066400000000000000000000142071276133741700200760ustar00rootroot00000000000000package db import ( "fmt" "gopkg.in/mgo.v2" "gopkg.in/mgo.v2/bson" "strings" ) // Query flags const ( Snapshot = 1 << iota LogReplay Prefetch ) type NodeType string const ( Mongos NodeType = "mongos" Standalone = "standalone" ReplSet = "replset" Unknown = "unknown" ) // CommandRunner exposes functions that can be run against a server type CommandRunner interface { Run(command interface{}, out interface{}, database string) error FindOne(db, collection string, skip int, query interface{}, sort []string, into interface{}, opts int) error Remove(db, collection string, query interface{}) error DatabaseNames() ([]string, error) CollectionNames(db string) ([]string, error) } // Remove removes all documents matched by query q in the db database and c collection. func (sp *SessionProvider) Remove(db, c string, q interface{}) error { session, err := sp.GetSession() if err != nil { return err } defer session.Close() _, err = session.DB(db).C(c).RemoveAll(q) return err } // Run issues the provided command on the db database and unmarshals its result // into out. func (sp *SessionProvider) Run(command interface{}, out interface{}, db string) error { session, err := sp.GetSession() if err != nil { return err } defer session.Close() return session.DB(db).Run(command, out) } // DatabaseNames returns a slice containing the names of all the databases on the // connected server. func (sp *SessionProvider) DatabaseNames() ([]string, error) { session, err := sp.GetSession() if err != nil { return nil, err } session.SetSocketTimeout(0) defer session.Close() return session.DatabaseNames() } // CollectionNames returns the names of all the collections in the dbName database. func (sp *SessionProvider) CollectionNames(dbName string) ([]string, error) { session, err := sp.GetSession() if err != nil { return nil, err } defer session.Close() session.SetSocketTimeout(0) return session.DB(dbName).CollectionNames() } // GetNodeType checks if the connected SessionProvider is a mongos, standalone, or replset, // by looking at the result of calling isMaster. func (sp *SessionProvider) GetNodeType() (NodeType, error) { session, err := sp.GetSession() if err != nil { return Unknown, err } session.SetSocketTimeout(0) defer session.Close() masterDoc := struct { SetName interface{} `bson:"setName"` Hosts interface{} `bson:"hosts"` Msg string `bson:"msg"` }{} err = session.Run("isMaster", &masterDoc) if err != nil { return Unknown, err } if masterDoc.SetName != nil || masterDoc.Hosts != nil { return ReplSet, nil } else if masterDoc.Msg == "isdbgrid" { // isdbgrid is always the msg value when calling isMaster on a mongos // see http://docs.mongodb.org/manual/core/sharded-cluster-query-router/ return Mongos, nil } return Standalone, nil } // IsReplicaSet returns a boolean which is true if the connected server is part // of a replica set. func (sp *SessionProvider) IsReplicaSet() (bool, error) { nodeType, err := sp.GetNodeType() if err != nil { return false, err } return nodeType == ReplSet, nil } // IsMongos returns true if the connected server is a mongos. func (sp *SessionProvider) IsMongos() (bool, error) { nodeType, err := sp.GetNodeType() if err != nil { return false, err } return nodeType == Mongos, nil } // SupportsRepairCursor takes in an example db and collection name and // returns true if the connected server supports the repairCursor command. // It returns false and the error that occurred if it is not supported. func (sp *SessionProvider) SupportsRepairCursor(db, collection string) (bool, error) { session, err := sp.GetSession() if err != nil { return false, err } session.SetSocketTimeout(0) defer session.Close() // This check is slightly hacky, but necessary to allow users to run repair without // permissions to all collections. There are multiple reasons a repair command could fail, // but we are only interested in the ones that imply that the repair command is not // usable by the connected server. If we do not get one of these specific error messages, // we will let the error happen again later. repairIter := session.DB(db).C(collection).Repair() repairIter.Next(bson.D{}) err = repairIter.Err() if err == nil { return true, nil } if strings.Index(err.Error(), "no such cmd: repairCursor") > -1 { // return a helpful error message for early server versions return false, fmt.Errorf("--repair flag cannot be used on mongodb versions before 2.7.8") } if strings.Index(err.Error(), "repair iterator not supported") > -1 { // helpful error message if the storage engine does not support repair (WiredTiger) return false, fmt.Errorf("--repair is not supported by the connected storage engine") } return true, nil } // SupportsWriteCommands returns true if the connected server supports write // commands, returns false otherwise. func (sp *SessionProvider) SupportsWriteCommands() (bool, error) { session, err := sp.GetSession() if err != nil { return false, err } session.SetSocketTimeout(0) defer session.Close() masterDoc := struct { Ok int `bson:"ok"` MaxWire int `bson:"maxWireVersion"` }{} err = session.Run("isMaster", &masterDoc) if err != nil { return false, err } // the connected server supports write commands if // the maxWriteVersion field is present return (masterDoc.Ok == 1 && masterDoc.MaxWire >= 2), nil } // FindOne retuns the first document in the collection and database that matches // the query after skip, sort and query flags are applied. func (sp *SessionProvider) FindOne(db, collection string, skip int, query interface{}, sort []string, into interface{}, flags int) error { session, err := sp.GetSession() if err != nil { return err } defer session.Close() q := session.DB(db).C(collection).Find(query).Sort(sort...).Skip(skip) q = ApplyFlags(q, session, flags) return q.One(into) } // ApplyFlags applies flags to the given query session. func ApplyFlags(q *mgo.Query, session *mgo.Session, flags int) *mgo.Query { if flags&Snapshot > 0 { q = q.Snapshot() } if flags&LogReplay > 0 { q = q.LogReplay() } if flags&Prefetch > 0 { session.SetPrefetch(1.0) } return q } mongo-tools-r3.2.11/common/db/connector.go000066400000000000000000000034461276133741700204550ustar00rootroot00000000000000package db import ( "time" "github.com/mongodb/mongo-tools/common/db/kerberos" "github.com/mongodb/mongo-tools/common/options" "github.com/mongodb/mongo-tools/common/util" "gopkg.in/mgo.v2" ) // Interface type for connecting to the database. type DBConnector interface { // configure, based on the options passed in Configure(options.ToolOptions) error // dial the database and get a fresh new session GetNewSession() (*mgo.Session, error) } // Basic connector for dialing the database, with no authentication. type VanillaDBConnector struct { dialInfo *mgo.DialInfo } // Configure sets up the db connector using the options in opts. It parses the // connection string and then sets up the dial information using the default // dial timeout. func (self *VanillaDBConnector) Configure(opts options.ToolOptions) error { // create the addresses to be used to connect connectionAddrs := util.CreateConnectionAddrs(opts.Host, opts.Port) timeout := time.Duration(options.DefaultDialTimeoutSeconds) * time.Second if opts.HiddenOptions != nil && opts.HiddenOptions.DialTimeoutSeconds != nil { timeout = time.Duration(*opts.HiddenOptions.DialTimeoutSeconds) * time.Second } // set up the dial info self.dialInfo = &mgo.DialInfo{ Addrs: connectionAddrs, Timeout: timeout, Direct: opts.Direct, ReplicaSetName: opts.ReplicaSetName, Username: opts.Auth.Username, Password: opts.Auth.Password, Source: opts.GetAuthenticationDatabase(), Mechanism: opts.Auth.Mechanism, } kerberos.AddKerberosOpts(opts, self.dialInfo) return nil } // GetNewSession connects to the server and returns the established session and any // error encountered. func (self *VanillaDBConnector) GetNewSession() (*mgo.Session, error) { return mgo.DialWithInfo(self.dialInfo) } mongo-tools-r3.2.11/common/db/connector_sasl_test.go000066400000000000000000000025471276133741700225370ustar00rootroot00000000000000// +build sasl package db // This file runs Kerberos tests if build with sasl is enabled import ( "fmt" "os" "runtime" "testing" "github.com/mongodb/mongo-tools/common/options" "github.com/mongodb/mongo-tools/common/testutil" . "github.com/smartystreets/goconvey/convey" "gopkg.in/mgo.v2/bson" ) var ( KERBEROS_HOST = "ldaptest.10gen.cc" KERBEROS_USER = "drivers@LDAPTEST.10GEN.CC" ) func TestKerberosAuthMechanism(t *testing.T) { Convey("should be able to successfully connect", t, func() { connector := &VanillaDBConnector{} opts := options.ToolOptions{ Connection: &options.Connection{ Host: KERBEROS_HOST, Port: "27017", }, Auth: &options.Auth{ Username: KERBEROS_USER, Mechanism: "GSSAPI", }, Kerberos: &options.Kerberos{ Service: "mongodb", ServiceHost: KERBEROS_HOST, }, } if runtime.GOOS == "windows" { opts.Auth.Password = os.Getenv(testutil.WinKerberosPwdEnv) if opts.Auth.Password == "" { panic(fmt.Sprintf("Need to set %v environment variable to run kerberos tests on windows", testutil.WinKerberosPwdEnv)) } } So(connector.Configure(opts), ShouldBeNil) session, err := connector.GetNewSession() So(err, ShouldBeNil) So(session, ShouldNotBeNil) n, err := session.DB("kerberos").C("test").Find(bson.M{}).Count() So(err, ShouldBeNil) So(n, ShouldEqual, 1) }) } mongo-tools-r3.2.11/common/db/connector_test.go000066400000000000000000000060141276133741700215060ustar00rootroot00000000000000package db import ( "github.com/mongodb/mongo-tools/common/options" "github.com/mongodb/mongo-tools/common/testutil" . "github.com/smartystreets/goconvey/convey" "gopkg.in/mgo.v2" "testing" "time" ) func TestVanillaDBConnector(t *testing.T) { testutil.VerifyTestType(t, "db") Convey("With a vanilla db connector", t, func() { var connector *VanillaDBConnector Convey("calling Configure should populate the addrs and dial timeout"+ " appropriately with no error", func() { connector = &VanillaDBConnector{} opts := options.ToolOptions{ Connection: &options.Connection{ Host: "host1,host2", Port: "20000", }, Auth: &options.Auth{}, } So(connector.Configure(opts), ShouldBeNil) So(connector.dialInfo.Addrs, ShouldResemble, []string{"host1:20000", "host2:20000"}) So(connector.dialInfo.Timeout, ShouldResemble, time.Duration(options.DefaultDialTimeoutSeconds)*time.Second) }) Convey("calling GetNewSession with a running mongod should connect"+ " successfully", func() { connector = &VanillaDBConnector{} opts := options.ToolOptions{ Connection: &options.Connection{ Host: "localhost", Port: DefaultTestPort, }, Auth: &options.Auth{}, } So(connector.Configure(opts), ShouldBeNil) session, err := connector.GetNewSession() So(err, ShouldBeNil) So(session, ShouldNotBeNil) session.Close() }) }) } func TestVanillaDBConnectorWithAuth(t *testing.T) { testutil.VerifyTestType(t, "auth") session, err := mgo.Dial("localhost:33333") if err != nil { t.Fatalf("error dialing server: %v", err) } err = testutil.CreateUserAdmin(session) So(err, ShouldBeNil) err = testutil.CreateUserWithRole(session, "cAdmin", "password", mgo.RoleClusterAdmin, true) So(err, ShouldBeNil) session.Close() Convey("With a vanilla db connector and a mongod running with"+ " auth", t, func() { var connector *VanillaDBConnector Convey("connecting without authentication should not be able"+ " to run commands", func() { connector = &VanillaDBConnector{} opts := options.ToolOptions{ Connection: &options.Connection{ Host: "localhost", Port: DefaultTestPort, }, Auth: &options.Auth{}, } So(connector.Configure(opts), ShouldBeNil) session, err := connector.GetNewSession() So(err, ShouldBeNil) So(session, ShouldNotBeNil) So(session.DB("admin").Run("top", &struct{}{}), ShouldNotBeNil) session.Close() }) Convey("connecting with authentication should succeed and"+ " authenticate properly", func() { connector = &VanillaDBConnector{} opts := options.ToolOptions{ Connection: &options.Connection{ Host: "localhost", Port: DefaultTestPort, }, Auth: &options.Auth{ Username: "cAdmin", Password: "password", }, } So(connector.Configure(opts), ShouldBeNil) session, err := connector.GetNewSession() So(err, ShouldBeNil) So(session, ShouldNotBeNil) So(session.DB("admin").Run("top", &struct{}{}), ShouldBeNil) session.Close() }) }) } mongo-tools-r3.2.11/common/db/db.go000066400000000000000000000155241276133741700170500ustar00rootroot00000000000000// Package db implements generic connection to MongoDB, and contains // subpackages for specific methods of connection. package db import ( "github.com/mongodb/mongo-tools/common/options" "github.com/mongodb/mongo-tools/common/password" "gopkg.in/mgo.v2" "gopkg.in/mgo.v2/bson" "fmt" "io" "strings" "sync" ) type ( sessionFlag uint32 // Used to get appropriate the DBConnector(s) based on opts GetConnectorFunc func(opts options.ToolOptions) DBConnector ) // Session flags. const ( None sessionFlag = 0 Monotonic sessionFlag = 1 << iota DisableSocketTimeout ) // MongoDB enforced limits. const ( MaxBSONSize = 16 * 1024 * 1024 // 16MB - maximum BSON document size ) // Default port for integration tests const ( DefaultTestPort = "33333" ) const ( ErrLostConnection = "lost connection to server" ErrNoReachableServers = "no reachable servers" ErrNsNotFound = "ns not found" // replication errors list the replset name if we are talking to a mongos, // so we can only check for this universal prefix ErrReplTimeoutPrefix = "waiting for replication timed out" ErrCouldNotContactPrimaryPrefix = "could not contact primary for replica set" ErrCouldNotFindPrimaryPrefix = `could not find host matching read preference { mode: "primary"` ErrUnableToTargetPrefix = "unable to target" ErrNotMaster = "not master" ErrConnectionRefusedSuffix = "Connection refused" ) var ( GetConnectorFuncs = []GetConnectorFunc{} ) // Used to manage database sessions type SessionProvider struct { // For connecting to the database connector DBConnector // used to avoid a race condition around creating the master session masterSessionLock sync.Mutex // the master session to use for connection pooling masterSession *mgo.Session // flags for generating the master session bypassDocumentValidation bool flags sessionFlag readPreference mgo.Mode tags bson.D } // ApplyOpsResponse represents the response from an 'applyOps' command. type ApplyOpsResponse struct { Ok bool `bson:"ok"` ErrMsg string `bson:"errmsg"` } // Oplog represents a MongoDB oplog document. type Oplog struct { Timestamp bson.MongoTimestamp `bson:"ts"` HistoryID int64 `bson:"h"` Version int `bson:"v"` Operation string `bson:"op"` Namespace string `bson:"ns"` Object bson.D `bson:"o"` Query bson.D `bson:"o2"` } // Returns a session connected to the database server for which the // session provider is configured. func (self *SessionProvider) GetSession() (*mgo.Session, error) { self.masterSessionLock.Lock() defer self.masterSessionLock.Unlock() // The master session is initialized if self.masterSession != nil { return self.masterSession.Copy(), nil } // initialize the provider's master session var err error self.masterSession, err = self.connector.GetNewSession() if err != nil { return nil, fmt.Errorf("error connecting to db server: %v", err) } // update masterSession based on flags self.refresh() // copy the provider's master session, for connection pooling return self.masterSession.Copy(), nil } // refresh is a helper for modifying the session based on the // session provider flags passed in with SetFlags. // This helper assumes a lock is already taken. func (self *SessionProvider) refresh() { // handle bypassDocumentValidation self.masterSession.SetBypassValidation(self.bypassDocumentValidation) // handle readPreference self.masterSession.SetMode(self.readPreference, true) // disable timeouts if (self.flags & DisableSocketTimeout) > 0 { self.masterSession.SetSocketTimeout(0) } if self.tags != nil { self.masterSession.SelectServers(self.tags) } } // SetFlags allows certain modifications to the masterSession after initial creation. func (self *SessionProvider) SetFlags(flagBits sessionFlag) { self.masterSessionLock.Lock() defer self.masterSessionLock.Unlock() self.flags = flagBits // make sure we update the master session if one already exists if self.masterSession != nil { self.refresh() } } // SetReadPreference sets the read preference mode in the SessionProvider // and eventually in the masterSession func (self *SessionProvider) SetReadPreference(pref mgo.Mode) { self.masterSessionLock.Lock() defer self.masterSessionLock.Unlock() self.readPreference = pref if self.masterSession != nil { self.refresh() } } // SetBypassDocumentValidation sets whether to bypass document validation in the SessionProvider // and eventually in the masterSession func (self *SessionProvider) SetBypassDocumentValidation(bypassDocumentValidation bool) { self.masterSessionLock.Lock() defer self.masterSessionLock.Unlock() self.bypassDocumentValidation = bypassDocumentValidation if self.masterSession != nil { self.refresh() } } // SetTags sets the server selection tags in the SessionProvider // and eventually in the masterSession func (self *SessionProvider) SetTags(tags bson.D) { self.masterSessionLock.Lock() defer self.masterSessionLock.Unlock() self.tags = tags if self.masterSession != nil { self.refresh() } } // NewSessionProvider constructs a session provider but does not attempt to // create the initial session. func NewSessionProvider(opts options.ToolOptions) (*SessionProvider, error) { // create the provider provider := &SessionProvider{ readPreference: mgo.Primary, bypassDocumentValidation: false, } // finalize auth options, filling in missing passwords if opts.Auth.ShouldAskForPassword() { opts.Auth.Password = password.Prompt() } // create the connector for dialing the database provider.connector = getConnector(opts) // configure the connector err := provider.connector.Configure(opts) if err != nil { return nil, fmt.Errorf("error configuring the connector: %v", err) } return provider, nil } // IsConnectionError returns a boolean indicating if a given error is due to // an error in an underlying DB connection (as opposed to some other write // failure such as a duplicate key error) func IsConnectionError(err error) bool { if err == nil { return false } if err.Error() == ErrNoReachableServers || err.Error() == io.EOF.Error() || strings.HasPrefix(err.Error(), ErrReplTimeoutPrefix) || strings.HasPrefix(err.Error(), ErrCouldNotContactPrimaryPrefix) || strings.HasPrefix(err.Error(), ErrCouldNotFindPrimaryPrefix) || strings.HasPrefix(err.Error(), ErrUnableToTargetPrefix) || err.Error() == ErrNotMaster || strings.HasSuffix(err.Error(), ErrConnectionRefusedSuffix) { return true } return false } // Get the right type of connector, based on the options func getConnector(opts options.ToolOptions) DBConnector { for _, getConnectorFunc := range GetConnectorFuncs { if connector := getConnectorFunc(opts); connector != nil { return connector } } return &VanillaDBConnector{} } mongo-tools-r3.2.11/common/db/db_ssl.go000066400000000000000000000006501276133741700177230ustar00rootroot00000000000000// +build ssl package db import ( "github.com/mongodb/mongo-tools/common/db/openssl" "github.com/mongodb/mongo-tools/common/options" ) func init() { GetConnectorFuncs = append(GetConnectorFuncs, getSSLConnector) } // return the SSL DB connector if using SSL, otherwise, return nil. func getSSLConnector(opts options.ToolOptions) DBConnector { if opts.SSL.UseSSL { return &openssl.SSLDBConnector{} } return nil } mongo-tools-r3.2.11/common/db/db_test.go000066400000000000000000000027611276133741700201060ustar00rootroot00000000000000package db import ( "github.com/mongodb/mongo-tools/common/options" "github.com/mongodb/mongo-tools/common/testutil" . "github.com/smartystreets/goconvey/convey" "reflect" "testing" ) func TestNewSessionProvider(t *testing.T) { testutil.VerifyTestType(t, "db") Convey("When initializing a session provider", t, func() { Convey("with the standard options, a provider with a standard"+ " connector should be returned", func() { opts := options.ToolOptions{ Connection: &options.Connection{ Port: DefaultTestPort, }, SSL: &options.SSL{}, Auth: &options.Auth{}, } provider, err := NewSessionProvider(opts) So(err, ShouldBeNil) So(reflect.TypeOf(provider.connector), ShouldEqual, reflect.TypeOf(&VanillaDBConnector{})) }) Convey("the master session should be successfully "+ " initialized", func() { opts := options.ToolOptions{ Connection: &options.Connection{ Port: DefaultTestPort, }, SSL: &options.SSL{}, Auth: &options.Auth{}, } provider, err := NewSessionProvider(opts) So(err, ShouldBeNil) So(provider.masterSession, ShouldBeNil) session, err := provider.GetSession() So(err, ShouldBeNil) So(session, ShouldNotBeNil) So(provider.masterSession, ShouldNotBeNil) }) }) } type listDatabasesCommand struct { Databases []map[string]interface{} `json:"databases"` Ok bool `json:"ok"` } func (self *listDatabasesCommand) AsRunnable() interface{} { return "listDatabases" } mongo-tools-r3.2.11/common/db/kerberos/000077500000000000000000000000001276133741700177415ustar00rootroot00000000000000mongo-tools-r3.2.11/common/db/kerberos/gssapi.go000066400000000000000000000013521276133741700215570ustar00rootroot00000000000000// Package kerberos implements authentication to MongoDB using kerberos package kerberos // #cgo windows CFLAGS: -Ic:/sasl/include // #cgo windows LDFLAGS: -Lc:/sasl/lib import ( "github.com/mongodb/mongo-tools/common/options" "gopkg.in/mgo.v2" ) const authMechanism = "GSSAPI" func AddKerberosOpts(opts options.ToolOptions, dialInfo *mgo.DialInfo) { if dialInfo == nil { return } if opts.Kerberos == nil || opts.Kerberos.Service == "" || opts.Kerberos.ServiceHost == "" { return } if opts.Auth == nil || (opts.Auth.Mechanism != authMechanism && dialInfo.Mechanism != authMechanism) { return } dialInfo.Service = opts.Kerberos.Service dialInfo.ServiceHost = opts.Kerberos.ServiceHost dialInfo.Mechanism = authMechanism } mongo-tools-r3.2.11/common/db/namespaces.go000066400000000000000000000110701276133741700205720ustar00rootroot00000000000000package db import ( "fmt" "github.com/mongodb/mongo-tools/common/bsonutil" "github.com/mongodb/mongo-tools/common/log" "gopkg.in/mgo.v2" "gopkg.in/mgo.v2/bson" "strings" ) // IsNoCmd reeturns true if err indicates a query command is not supported, // otherwise, returns false. func IsNoCmd(err error) bool { e, ok := err.(*mgo.QueryError) return ok && strings.HasPrefix(e.Message, "no such cmd:") } // IsNoCollection returns true if err indicates a query resulted in a "no collection" error // otherwise, returns false. func IsNoCollection(err error) bool { e, ok := err.(*mgo.QueryError) return ok && e.Message == "no collection" } // buildBsonArray takes a cursor iterator and returns an array of // all of its documents as bson.D objects. func buildBsonArray(iter *mgo.Iter) ([]bson.D, error) { ret := make([]bson.D, 0, 0) index := new(bson.D) for iter.Next(index) { ret = append(ret, *index) index = new(bson.D) } if iter.Err() != nil { return nil, iter.Err() } return ret, nil } // GetIndexes returns an iterator to thethe raw index info for a collection by // using the listIndexes command if available, or by falling back to querying // against system.indexes (pre-3.0 systems). nil is returned if the collection // does not exist. func GetIndexes(coll *mgo.Collection) (*mgo.Iter, error) { var cmdResult struct { Cursor struct { FirstBatch []bson.Raw "firstBatch" NS string Id int64 } } err := coll.Database.Run(bson.D{{"listIndexes", coll.Name}, {"cursor", bson.M{}}}, &cmdResult) switch { case err == nil: ns := strings.SplitN(cmdResult.Cursor.NS, ".", 2) if len(ns) < 2 { return nil, fmt.Errorf("server returned invalid cursor.ns `%v` on listIndexes for `%v`: %v", cmdResult.Cursor.NS, coll.FullName, err) } ses := coll.Database.Session return ses.DB(ns[0]).C(ns[1]).NewIter(ses, cmdResult.Cursor.FirstBatch, cmdResult.Cursor.Id, nil), nil case IsNoCmd(err): log.Logf(log.DebugLow, "No support for listIndexes command, falling back to querying system.indexes") return getIndexesPre28(coll) case IsNoCollection(err): return nil, nil default: return nil, fmt.Errorf("error running `listIndexes`. Collection: `%v` Err: %v", coll.FullName, err) } } func getIndexesPre28(coll *mgo.Collection) (*mgo.Iter, error) { indexColl := coll.Database.C("system.indexes") iter := indexColl.Find(&bson.M{"ns": coll.FullName}).Iter() return iter, nil } func GetCollections(database *mgo.Database, name string) (*mgo.Iter, bool, error) { var cmdResult struct { Cursor struct { FirstBatch []bson.Raw "firstBatch" NS string Id int64 } } command := bson.D{{"listCollections", 1}, {"cursor", bson.M{}}} if len(name) > 0 { command = bson.D{{"listCollections", 1}, {"filter", bson.M{"name": name}}, {"cursor", bson.M{}}} } err := database.Run(command, &cmdResult) switch { case err == nil: ns := strings.SplitN(cmdResult.Cursor.NS, ".", 2) if len(ns) < 2 { return nil, false, fmt.Errorf("server returned invalid cursor.ns `%v` on listCollections for `%v`: %v", cmdResult.Cursor.NS, database.Name, err) } return database.Session.DB(ns[0]).C(ns[1]).NewIter(database.Session, cmdResult.Cursor.FirstBatch, cmdResult.Cursor.Id, nil), false, nil case IsNoCmd(err): log.Logf(log.DebugLow, "No support for listCollections command, falling back to querying system.namespaces") iter, err := getCollectionsPre28(database, name) return iter, true, err default: return nil, false, fmt.Errorf("error running `listCollections`. Database: `%v` Err: %v", database.Name, err) } } func getCollectionsPre28(database *mgo.Database, name string) (*mgo.Iter, error) { indexColl := database.C("system.namespaces") selector := bson.M{} if len(name) > 0 { selector["name"] = database.Name + "." + name } iter := indexColl.Find(selector).Iter() return iter, nil } func GetCollectionOptions(coll *mgo.Collection) (*bson.D, error) { iter, useFullName, err := GetCollections(coll.Database, coll.Name) if err != nil { return nil, err } comparisonName := coll.Name if useFullName { comparisonName = coll.FullName } collInfo := &bson.D{} for iter.Next(collInfo) { name, err := bsonutil.FindValueByKey("name", collInfo) if err != nil { collInfo = nil continue } if nameStr, ok := name.(string); ok { if nameStr == comparisonName { // we've found the collection we're looking for return collInfo, nil } } else { collInfo = nil continue } } err = iter.Err() if err != nil { return nil, err } // The given collection was not found, but no error encountered. return nil, nil } mongo-tools-r3.2.11/common/db/openssl/000077500000000000000000000000001276133741700176105ustar00rootroot00000000000000mongo-tools-r3.2.11/common/db/openssl/openssl.go000066400000000000000000000123521276133741700216250ustar00rootroot00000000000000// Package openssl implements connection to MongoDB over ssl. package openssl import ( "fmt" "net" "time" "github.com/mongodb/mongo-tools/common/db/kerberos" "github.com/mongodb/mongo-tools/common/options" "github.com/mongodb/mongo-tools/common/util" "github.com/spacemonkeygo/openssl" "gopkg.in/mgo.v2" ) // For connecting to the database over ssl type SSLDBConnector struct { dialInfo *mgo.DialInfo dialError error ctx *openssl.Ctx } // Configure the connector to connect to the server over ssl. Parses the // connection string, and sets up the correct function to dial the server // based on the ssl options passed in. func (self *SSLDBConnector) Configure(opts options.ToolOptions) error { // create the addresses to be used to connect connectionAddrs := util.CreateConnectionAddrs(opts.Host, opts.Port) var err error self.ctx, err = setupCtx(opts) if err != nil { return fmt.Errorf("openssl configuration: %v", err) } var flags openssl.DialFlags flags = 0 if opts.SSLAllowInvalidCert || opts.SSLAllowInvalidHost || opts.SSLCAFile == "" { flags = openssl.InsecureSkipHostVerification } // create the dialer func that will be used to connect dialer := func(addr *mgo.ServerAddr) (net.Conn, error) { conn, err := openssl.Dial("tcp", addr.String(), self.ctx, flags) self.dialError = err return conn, err } timeout := time.Duration(options.DefaultDialTimeoutSeconds) * time.Second if opts.HiddenOptions != nil && opts.HiddenOptions.DialTimeoutSeconds != nil { timeout = time.Duration(*opts.HiddenOptions.DialTimeoutSeconds) * time.Second } // set up the dial info self.dialInfo = &mgo.DialInfo{ Addrs: connectionAddrs, Timeout: timeout, Direct: opts.Direct, ReplicaSetName: opts.ReplicaSetName, DialServer: dialer, Username: opts.Auth.Username, Password: opts.Auth.Password, Source: opts.GetAuthenticationDatabase(), Mechanism: opts.Auth.Mechanism, } kerberos.AddKerberosOpts(opts, self.dialInfo) return nil } // Dial the server. func (self *SSLDBConnector) GetNewSession() (*mgo.Session, error) { session, err := mgo.DialWithInfo(self.dialInfo) if err != nil && self.dialError != nil { return nil, fmt.Errorf("%v, openssl error: %v", err, self.dialError) } return session, err } // To be handed to mgo.DialInfo for connecting to the server. type dialerFunc func(addr *mgo.ServerAddr) (net.Conn, error) // Handle optionally compiled SSL initialization functions (fips mode set) type sslInitializationFunction func(options.ToolOptions) error var sslInitializationFunctions []sslInitializationFunction // Creates and configures an openssl.Ctx func setupCtx(opts options.ToolOptions) (*openssl.Ctx, error) { var ctx *openssl.Ctx var err error for _, sslInitFunc := range sslInitializationFunctions { sslInitFunc(opts) } if ctx, err = openssl.NewCtxWithVersion(openssl.AnyVersion); err != nil { return nil, fmt.Errorf("failure creating new openssl context with "+ "NewCtxWithVersion(AnyVersion): %v", err) } // OpAll - Activate all bug workaround options, to support buggy client SSL's. // NoSSLv2 - Disable SSL v2 support ctx.SetOptions(openssl.OpAll | openssl.NoSSLv2) // HIGH - Enable strong ciphers // !EXPORT - Disable export ciphers (40/56 bit) // !aNULL - Disable anonymous auth ciphers // @STRENGTH - Sort ciphers based on strength ctx.SetCipherList("HIGH:!EXPORT:!aNULL@STRENGTH") // add the PEM key file with the cert and private key, if specified if opts.SSLPEMKeyFile != "" { if err = ctx.UseCertificateChainFile(opts.SSLPEMKeyFile); err != nil { return nil, fmt.Errorf("UseCertificateChainFile: %v", err) } if opts.SSLPEMKeyPassword != "" { if err = ctx.UsePrivateKeyFileWithPassword( opts.SSLPEMKeyFile, openssl.FiletypePEM, opts.SSLPEMKeyPassword); err != nil { return nil, fmt.Errorf("UsePrivateKeyFile: %v", err) } } else { if err = ctx.UsePrivateKeyFile(opts.SSLPEMKeyFile, openssl.FiletypePEM); err != nil { return nil, fmt.Errorf("UsePrivateKeyFile: %v", err) } } // Verify that the certificate and the key go together. if err = ctx.CheckPrivateKey(); err != nil { return nil, fmt.Errorf("CheckPrivateKey: %v", err) } } // If renegotiation is needed, don't return from recv() or send() until it's successful. // Note: this is for blocking sockets only. ctx.SetMode(openssl.AutoRetry) // Disable session caching (see SERVER-10261) ctx.SetSessionCacheMode(openssl.SessionCacheOff) if opts.SSLCAFile != "" { calist, err := openssl.LoadClientCAFile(opts.SSLCAFile) if err != nil { return nil, fmt.Errorf("LoadClientCAFile: %v", err) } ctx.SetClientCAList(calist) if err = ctx.LoadVerifyLocations(opts.SSLCAFile, ""); err != nil { return nil, fmt.Errorf("LoadVerifyLocations: %v", err) } var verifyOption openssl.VerifyOptions if opts.SSLAllowInvalidCert { verifyOption = openssl.VerifyNone } else { verifyOption = openssl.VerifyPeer } ctx.SetVerify(verifyOption, nil) } if opts.SSLCRLFile != "" { store := ctx.GetCertificateStore() store.SetFlags(openssl.CRLCheck) lookup, err := store.AddLookup(openssl.X509LookupFile()) if err != nil { return nil, fmt.Errorf("AddLookup(X509LookupFile()): %v", err) } lookup.LoadCRLFile(opts.SSLCRLFile) } return ctx, nil } mongo-tools-r3.2.11/common/db/openssl/openssl_fips.go000066400000000000000000000006121276133741700226420ustar00rootroot00000000000000// +build ssl // +build -darwin package openssl import "github.com/spacemonkeygo/openssl" func init() { sslInitializationFunctions = append(sslInitializationFunctions, SetUpFIPSMode) } func SetUpFIPSMode(opts *ToolOptions) error { if err := openssl.FIPSModeSet(opts.SSLFipsMode); err != nil { return fmt.Errorf("couldn't set FIPS mode to %v: %v", opts.SSLFipsMode, err) } return nil } mongo-tools-r3.2.11/common/db/openssl/testdata/000077500000000000000000000000001276133741700214215ustar00rootroot00000000000000mongo-tools-r3.2.11/common/db/openssl/testdata/ca.pem000066400000000000000000000036621276133741700225160ustar00rootroot00000000000000-----BEGIN PRIVATE KEY----- MIICdgIBADANBgkqhkiG9w0BAQEFAASCAmAwggJcAgEAAoGBAMbN8D5Au+xWdY+s GpUuSFSbHGzYfHmw0yajA9J8PiwDePRMl71OMMsByNsykjzXEr0BBOn4PNO6KW7K HdDicRavuC/iFucVpILUiJoLOUCPKb/EyAHUk0r2fdr3Ypd2ZXkD1EXmM9WTQnyW PEWqr1T7MmM9PhsD0r8ZbQVu8R49AgMBAAECgYBbC+mguQjXfektOUabV6zsgnUM LEElgiPRqAqSFTBr+9MjHwjHO84Ayvpv2MM8dcsxIAxeEr/Yv4NGJ+5rwajESir6 /7UzqzhXmj6ylqTfbMRJCRsqnwvSfNwpsxtMSYieCxtdYqTLaJLAItBjuZPAYL8W 9Tf/NMc4AjLLHx7PyQJBAOyOcIS/i23td6ZX+QtppXL1fF/JMiKooE9m/npAT5K/ hQEaAatdLyQ669id181KY9F0JR1TEbzb0A1yo73soRsCQQDXJSG4ID8lfR9SXnEE y/RqYv0eKneER+V7e1Cy7bYHvJxZK0sWXYzIZhTl8PABh3PCoLdxjY0IM7UNWlwU dAuHAkAOUaTv9CQ9eDVY5VRW44M3TTLFHYmiXXCuvb5Dqibm7B7h7TASrmZPHB3w k8VfUNRv9kbU2pVlSCz0026j7XHnAkEAk/qZP8EGTe3K3mfRCsCSA57EhLwm6phd ElrWPcvc2WN0kqyBgAembqwwEZxwKE0XZTYQFw2KhKq0DFQrY3IR/wJAIAnLtabL aF819WI/VYlMmwb3GAO2w5KQilGhYl7tv1BghH+Qmg7HZEcIRmSwPKEQveT3YpCH nCu38jgPXhhqdg== -----END PRIVATE KEY----- -----BEGIN CERTIFICATE----- MIIC3DCCAkWgAwIBAgIJAKwksc/otf2iMA0GCSqGSIb3DQEBCwUAMIGGMQswCQYD VQQGEwJVUzERMA8GA1UECAwITmV3IFlvcmsxFjAUBgNVBAcMDU5ldyBZb3JrIENp dHkxHTAbBgNVBAoMFE1vbmdvREIgS2VybmVsIFRvb2xzMRkwFwYDVQQLDBBUb29s cyBUZXN0aW5nIENBMRIwEAYDVQQDDAlsb2NhbGhvc3QwHhcNMTUwNjA1MTU1MTQ1 WhcNMzUwNjA0MTU1MTQ1WjCBhjELMAkGA1UEBhMCVVMxETAPBgNVBAgMCE5ldyBZ b3JrMRYwFAYDVQQHDA1OZXcgWW9yayBDaXR5MR0wGwYDVQQKDBRNb25nb0RCIEtl cm5lbCBUb29sczEZMBcGA1UECwwQVG9vbHMgVGVzdGluZyBDQTESMBAGA1UEAwwJ bG9jYWxob3N0MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDGzfA+QLvsVnWP rBqVLkhUmxxs2Hx5sNMmowPSfD4sA3j0TJe9TjDLAcjbMpI81xK9AQTp+DzTuilu yh3Q4nEWr7gv4hbnFaSC1IiaCzlAjym/xMgB1JNK9n3a92KXdmV5A9RF5jPVk0J8 ljxFqq9U+zJjPT4bA9K/GW0FbvEePQIDAQABo1AwTjAdBgNVHQ4EFgQU+QOiCHTF 8At8aMOBvHF6wWZpcZUwHwYDVR0jBBgwFoAU+QOiCHTF8At8aMOBvHF6wWZpcZUw DAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOBgQCbbIAjdV+M8RR3ZF1WMBYD 8aMr55kgtnCWn4mTCDdombCYgtbaPq5sy8Hb/2wLQ9Zl4UuFL5wKWcx3kOLo3cw/ boj8jnUDnwrsBd2nN7sYdjF+M7FLp6U1AxrE5ejijtg2KCl+p4b7jJgJBSFIQD45 7CAJVjIrajY4LlJj3x+caQ== -----END CERTIFICATE----- mongo-tools-r3.2.11/common/db/openssl/testdata/server.pem000066400000000000000000000034341276133741700234360ustar00rootroot00000000000000-----BEGIN PRIVATE KEY----- MIICeAIBADANBgkqhkiG9w0BAQEFAASCAmIwggJeAgEAAoGBALOkdwU9Qx4FRn+z coBkeYYpVRg0pknPMDo4Q50TqZPfVhroTynx2Or+cjl5csd5hMKxWQpdzGq8JzH9 2BCLcDz/51vG3tPrpLIB50ABqa0wRGGDOO+XN0h+VkdqJvKReWOsNRoMT3s0Lh78 BqvRUomYXnbc1RBaxwWa+UoLCFgnAgMBAAECgYBd9XmjLeW6//tds5gB+4tsVpYB cRhAprOM3/zNXYlmpHu+2x78y1gvoSJRWWplVvPPeT8fIuxWL0844JJwJN5wyCwN nnrA28l6+Tcde+NlzCxwED+QDjAH20BRxCs0BLvnx3WAXRDmUbWAjOl/qPn9H6m1 nmUQ7H/f6dxZ0vVMQQJBAOl3xeVLyZZ828P/p3PvYkaeIxxVK1QDGOWi/3vC0DrY WK8xAoopjj0RHHZ1fL5bG31G3OR9Vc/rfk4a5XPIlRECQQDE+teCTiwV5Wwzdpg3 r440qOLCmpMXwJr/Jlh+C4c8ebnIQ9P5sSe4wQNHyeEZ2t7SGvPfjr7glpPhAkXy JTm3AkEAvNPgvVoUy6Bk5xuJRl2hMNiKMUo5ZxOyOVkiJeklHdMJt3h+Q1zk7ENA sBbKM/PgQezkj/FHTIl9eJKMbp8W4QJBAL4aXHyslw12wisUrKkpa7PUviwT5BvL TYsrZcIXvCeYTr1BAMX8vBopZNIWuoEqY1sgmfZKnFrB1+wTNpAQbxcCQQCHbtvQ 1U2p5Pz5XYyaoK2OEZhPMuLnOBMpzjSxRLxKyhb4k+ssIA0IeAiT4RIECtHJ8DJX 4aZK/qg9WmBH+zbO -----END PRIVATE KEY----- -----BEGIN CERTIFICATE----- MIICbzCCAdgCAQEwDQYJKoZIhvcNAQEFBQAwgYYxCzAJBgNVBAYTAlVTMREwDwYD VQQIDAhOZXcgWW9yazEWMBQGA1UEBwwNTmV3IFlvcmsgQ2l0eTEdMBsGA1UECgwU TW9uZ29EQiBLZXJuZWwgVG9vbHMxGTAXBgNVBAsMEFRvb2xzIFRlc3RpbmcgQ0Ex EjAQBgNVBAMMCWxvY2FsaG9zdDAeFw0xNTA2MDUxNTUxNDVaFw0zNTA2MDQxNTUx NDVaMHkxCzAJBgNVBAYTAlVTMREwDwYDVQQIDAhOZXcgWW9yazEWMBQGA1UEBwwN TmV3IFlvcmsgQ2l0eTEUMBIGA1UECgwLTW9uZ29EQiBJbmMxFTATBgNVBAsMDEtl cm5lbCBUb29sczESMBAGA1UEAwwJbG9jYWxob3N0MIGfMA0GCSqGSIb3DQEBAQUA A4GNADCBiQKBgQCzpHcFPUMeBUZ/s3KAZHmGKVUYNKZJzzA6OEOdE6mT31Ya6E8p 8djq/nI5eXLHeYTCsVkKXcxqvCcx/dgQi3A8/+dbxt7T66SyAedAAamtMERhgzjv lzdIflZHaibykXljrDUaDE97NC4e/Aar0VKJmF523NUQWscFmvlKCwhYJwIDAQAB MA0GCSqGSIb3DQEBBQUAA4GBACJiTnC3nksZsmMyD88+DuV8IA1DHSby4X/qtDYT eSuNbxRKnihXkm2KE+MGn7YeKg4a7FaYiH3ejk0ZBlY3TZXK3I1uh/zIhC9aMnSL z0z4OLcqp46F8PpYF7ARtXXWQuOEWe6k+VKy5XP1NX60sEJ0KwGBQjUw3Ys41JE8 iigw -----END CERTIFICATE----- mongo-tools-r3.2.11/common/db/read_preferences.go000066400000000000000000000021571276133741700217550ustar00rootroot00000000000000package db import ( "fmt" "github.com/mongodb/mongo-tools/common/json" "gopkg.in/mgo.v2" "gopkg.in/mgo.v2/bson" ) type readPrefDoc struct { Mode string Tags bson.D } const ( WarningNonPrimaryMongosConnection = "Warning: using a non-primary readPreference with a " + "connection to mongos may produce inconsistent duplicates or miss some documents." ) func ParseReadPreference(rp string) (mgo.Mode, bson.D, error) { var mode string var tags bson.D if rp == "" { return mgo.Nearest, nil, nil } if rp[0] != '{' { mode = rp } else { var doc readPrefDoc err := json.Unmarshal([]byte(rp), &doc) if err != nil { return 0, nil, fmt.Errorf("invalid --ReadPreferences json object: %v", err) } tags = doc.Tags mode = doc.Mode } switch mode { case "primary": return mgo.Primary, tags, nil case "primaryPreferred": return mgo.PrimaryPreferred, tags, nil case "secondary": return mgo.Secondary, tags, nil case "secondaryPreferred": return mgo.SecondaryPreferred, tags, nil case "nearest": return mgo.Nearest, tags, nil } return 0, nil, fmt.Errorf("invalid readPreference mode '%v'", mode) } mongo-tools-r3.2.11/common/db/testdata/000077500000000000000000000000001276133741700177365ustar00rootroot00000000000000mongo-tools-r3.2.11/common/db/testdata/testdata.bson000066400000000000000000000034101276133741700224300ustar00rootroot00000000000000_id_idð?_id@_id@_id@_id@_id@_id@_id @_id"@_id$@_id&@_id(@_id*@_id,@_id.@_id0@_id1@_id2@_id3@_id4@_id5@_id6@_id7@_id8@_id9@_id:@_id;@_id<@_id=@_id>@_id?@_id@@_id€@@_idA@_id€A@_idB@_id€B@_idC@_id€C@_idD@_id€D@_idE@_id€E@_idF@_id€F@_idG@_id€G@_idH@_id€H@_idI@_id€I@_idJ@_id€J@_idK@_id€K@_idL@_id€L@_idM@_id€M@_idN@_id€N@_idO@_id€O@_idP@_id@P@_id€P@_idÀP@_idQ@_id@Q@_id€Q@_idÀQ@_idR@_id@R@_id€R@_idÀR@_idS@_id@S@_id€S@_idÀS@_idT@_id@T@_id€T@_idÀT@_idU@_id@U@_id€U@_idÀU@_idV@_id@V@_id€V@_idÀV@_idW@_id@W@_id€W@_idÀW@_idX@_id@X@_id€X@_idÀX@mongo-tools-r3.2.11/common/db/write_concern.go000066400000000000000000000073771276133741700213330ustar00rootroot00000000000000package db import ( "fmt" "github.com/mongodb/mongo-tools/common/json" "github.com/mongodb/mongo-tools/common/log" "github.com/mongodb/mongo-tools/common/util" "gopkg.in/mgo.v2" "strconv" ) // write concern fields const ( j = "j" w = "w" fSync = "fsync" wTimeout = "wtimeout" ) // constructWCObject takes in a write concern and attempts to construct an // mgo.Safe object from it. It returns an error if it is unable to parse the // string or if a parsed write concern field value is invalid. func constructWCObject(writeConcern string) (sessionSafety *mgo.Safe, err error) { sessionSafety = &mgo.Safe{} defer func() { // If the user passes a w value of 0, we set the session to use the // unacknowledged write concern but only if journal commit acknowledgment, // is not required. If commit acknowledgment is required, it prevails, // and the server will require that mongod acknowledge the write operation if sessionSafety.WMode == "" && sessionSafety.W == 0 && !sessionSafety.J { sessionSafety = nil } }() jsonWriteConcern := map[string]interface{}{} if err = json.Unmarshal([]byte(writeConcern), &jsonWriteConcern); err != nil { // if the writeConcern string can not be unmarshaled into JSON, this // allows a default to the old behavior wherein the entire argument // passed in is assigned to the 'w' field - thus allowing users pass // a write concern that looks like: "majority", 0, "4", etc. wValue, err := strconv.Atoi(writeConcern) if err != nil { sessionSafety.WMode = writeConcern } else { sessionSafety.W = wValue if wValue < 0 { return sessionSafety, fmt.Errorf("invalid '%v' argument: %v", w, wValue) } } return sessionSafety, nil } if jVal, ok := jsonWriteConcern[j]; ok && util.IsTruthy(jVal) { sessionSafety.J = true } if fsyncVal, ok := jsonWriteConcern[fSync]; ok && util.IsTruthy(fsyncVal) { sessionSafety.FSync = true } if wtimeout, ok := jsonWriteConcern[wTimeout]; ok { wtimeoutValue, err := util.ToInt(wtimeout) if err != nil { return sessionSafety, fmt.Errorf("invalid '%v' argument: %v", wTimeout, wtimeout) } sessionSafety.WTimeout = wtimeoutValue } if wInterface, ok := jsonWriteConcern[w]; ok { wValue, err := util.ToInt(wInterface) if err != nil { // if the argument is neither a string nor int, error out wStrVal, ok := wInterface.(string) if !ok { return sessionSafety, fmt.Errorf("invalid '%v' argument: %v", w, wInterface) } sessionSafety.WMode = wStrVal } else { sessionSafety.W = wValue if wValue < 0 { return sessionSafety, fmt.Errorf("invalid '%v' argument: %v", w, wValue) } } } return sessionSafety, nil } // BuildWriteConcern takes a string and a NodeType indicating the type of node the write concern // is intended to be used against, and converts the write concern string argument into an // mgo.Safe object that's usable on sessions for that node type. func BuildWriteConcern(writeConcern string, nodeType NodeType) (*mgo.Safe, error) { sessionSafety, err := constructWCObject(writeConcern) if err != nil { return nil, err } if sessionSafety == nil { log.Logf(log.DebugLow, "using unacknowledged write concern") return nil, nil } // for standalone mongods, set the default write concern to 1 if nodeType == Standalone { log.Logf(log.DebugLow, "standalone server: setting write concern %v to 1", w) sessionSafety.W = 1 sessionSafety.WMode = "" } var writeConcernStr interface{} if sessionSafety.WMode != "" { writeConcernStr = sessionSafety.WMode } else { writeConcernStr = sessionSafety.W } log.Logf(log.Info, "using write concern: %v='%v', %v=%v, %v=%v, %v=%v", w, writeConcernStr, j, sessionSafety.J, fSync, sessionSafety.FSync, wTimeout, sessionSafety.WTimeout, ) return sessionSafety, nil } mongo-tools-r3.2.11/common/db/write_concern_test.go000066400000000000000000000143321276133741700223570ustar00rootroot00000000000000package db import ( . "github.com/smartystreets/goconvey/convey" "testing" ) func TestBuildWriteConcern(t *testing.T) { Convey("Given a write concern string value, and a boolean indicating if the "+ "write concern is to be used on a replica set, on calling BuildWriteConcern...", t, func() { Convey("no error should be returned if the write concern is valid", func() { writeConcern, err := BuildWriteConcern(`{w:34}`, ReplSet) So(err, ShouldBeNil) So(writeConcern.W, ShouldEqual, 34) writeConcern, err = BuildWriteConcern(`{w:"majority"}`, ReplSet) So(err, ShouldBeNil) So(writeConcern.WMode, ShouldEqual, "majority") writeConcern, err = BuildWriteConcern(`majority`, ReplSet) So(err, ShouldBeNil) So(writeConcern.WMode, ShouldEqual, "majority") writeConcern, err = BuildWriteConcern(`tagset`, ReplSet) So(err, ShouldBeNil) So(writeConcern.WMode, ShouldEqual, "tagset") }) Convey("on replica sets, only a write concern of 1 or 0 should be returned", func() { writeConcern, err := BuildWriteConcern(`{w:34}`, Standalone) So(err, ShouldBeNil) So(writeConcern.W, ShouldEqual, 1) writeConcern, err = BuildWriteConcern(`{w:"majority"}`, Standalone) So(err, ShouldBeNil) So(writeConcern.W, ShouldEqual, 1) writeConcern, err = BuildWriteConcern(`tagset`, Standalone) So(err, ShouldBeNil) So(writeConcern.W, ShouldEqual, 1) }) Convey("with a w value of 0, without j set, a nil write concern should be returned", func() { writeConcern, err := BuildWriteConcern(`{w:0}`, Standalone) So(err, ShouldBeNil) So(writeConcern, ShouldBeNil) }) Convey("with a negative w value, an error should be returned", func() { _, err := BuildWriteConcern(`{w:-1}`, ReplSet) So(err, ShouldNotBeNil) _, err = BuildWriteConcern(`{w:-2}`, ReplSet) So(err, ShouldNotBeNil) }) Convey("with a w value of 0, with j set, a non-nil write concern should be returned", func() { writeConcern, err := BuildWriteConcern(`{w:0, j:true}`, Standalone) So(err, ShouldBeNil) So(writeConcern.J, ShouldBeTrue) }) }) } func TestConstructWCObject(t *testing.T) { Convey("Given a write concern string value, on calling constructWCObject...", t, func() { Convey("non-JSON string values should be assigned to the 'WMode' "+ "field in their entirety", func() { writeConcernString := "majority" writeConcern, err := constructWCObject(writeConcernString) So(err, ShouldBeNil) So(writeConcern.WMode, ShouldEqual, writeConcernString) }) Convey("non-JSON int values should be assigned to the 'w' field "+ "in their entirety", func() { writeConcernString := `{w: 4}` writeConcern, err := constructWCObject(writeConcernString) So(err, ShouldBeNil) So(writeConcern.W, ShouldEqual, 4) }) Convey("JSON strings with valid j, wtimeout, fsync and w, should be "+ "assigned accordingly", func() { writeConcernString := `{w: 3, j: true, fsync: false, wtimeout: 43}` expectedW := 3 expectedWTimeout := 43 writeConcern, err := constructWCObject(writeConcernString) So(err, ShouldBeNil) So(writeConcern.W, ShouldEqual, expectedW) So(writeConcern.J, ShouldBeTrue) So(writeConcern.FSync, ShouldBeFalse) So(writeConcern.WTimeout, ShouldEqual, expectedWTimeout) }) Convey("JSON strings with an argument for j that is not false should set j true", func() { writeConcernString := `{w: 3, j: "rue"}` writeConcern, err := constructWCObject(writeConcernString) So(err, ShouldBeNil) So(writeConcern.W, ShouldEqual, 3) So(writeConcern.J, ShouldBeTrue) }) Convey("JSON strings with an argument for fsync that is not false should set fsync true", func() { writeConcernString := `{w: 3, fsync: "rue"}` writeConcern, err := constructWCObject(writeConcernString) So(err, ShouldBeNil) So(writeConcern.W, ShouldEqual, 3) So(writeConcern.FSync, ShouldBeTrue) }) Convey("JSON strings with an invalid wtimeout argument should error out", func() { writeConcernString := `{w: 3, wtimeout: "rue"}` _, err := constructWCObject(writeConcernString) So(err, ShouldNotBeNil) writeConcernString = `{w: 3, wtimeout: "43"}` _, err = constructWCObject(writeConcernString) So(err, ShouldNotBeNil) }) Convey("JSON strings with any non-false j argument should not error out", func() { writeConcernString := `{w: 3, j: "t"}` writeConcern, err := constructWCObject(writeConcernString) So(err, ShouldBeNil) So(writeConcern.J, ShouldBeTrue) writeConcernString = `{w: 3, j: "f"}` writeConcern, err = constructWCObject(writeConcernString) So(err, ShouldBeNil) So(writeConcern.J, ShouldBeTrue) writeConcernString = `{w: 3, j: false}` writeConcern, err = constructWCObject(writeConcernString) So(err, ShouldBeNil) So(writeConcern.J, ShouldBeFalse) writeConcernString = `{w: 3, j: 0}` writeConcern, err = constructWCObject(writeConcernString) So(err, ShouldBeNil) So(writeConcern.J, ShouldBeFalse) }) Convey("JSON strings with a shorthand fsync argument should not error out", func() { writeConcernString := `{w: 3, fsync: "t"}` writeConcern, err := constructWCObject(writeConcernString) So(err, ShouldBeNil) So(writeConcern.FSync, ShouldBeTrue) writeConcernString = `{w: "3", fsync: "f"}` writeConcern, err = constructWCObject(writeConcernString) So(err, ShouldBeNil) So(writeConcern.FSync, ShouldBeTrue) writeConcernString = `{w: "3", fsync: false}` writeConcern, err = constructWCObject(writeConcernString) So(err, ShouldBeNil) So(writeConcern.FSync, ShouldBeFalse) writeConcernString = `{w: "3", fsync: 0}` writeConcern, err = constructWCObject(writeConcernString) So(err, ShouldBeNil) So(writeConcern.FSync, ShouldBeFalse) }) Convey("Unacknowledge write concern strings should return a nil object "+ "if journaling is not required", func() { writeConcernString := `{w: 0}` writeConcern, err := constructWCObject(writeConcernString) So(err, ShouldBeNil) So(writeConcern, ShouldBeNil) writeConcernString = `{w: 0}` writeConcern, err = constructWCObject(writeConcernString) So(err, ShouldBeNil) So(writeConcern, ShouldBeNil) writeConcernString = `0` writeConcern, err = constructWCObject(writeConcernString) So(err, ShouldBeNil) So(writeConcern, ShouldBeNil) }) }) } mongo-tools-r3.2.11/common/intents/000077500000000000000000000000001276133741700172245ustar00rootroot00000000000000mongo-tools-r3.2.11/common/intents/intent.go000066400000000000000000000236171276133741700210650ustar00rootroot00000000000000// Package intents provides utilities for performing dump/restore operations. package intents import ( "github.com/mongodb/mongo-tools/common/log" "gopkg.in/mgo.v2/bson" "io" "sync" ) type file interface { io.ReadWriteCloser Open() error Pos() int64 } // FileNeedsIOBuffer is an interface that denotes that a struct needs // an IO buffer that is managed by an outside control. This interface // is used to both hand off a buffer to a struct and signal that it should // release its buffer. Added to reduce memory usage as outlined in TOOLS-1088. type FileNeedsIOBuffer interface { TakeIOBuffer([]byte) ReleaseIOBuffer() } // mongorestore first scans the directory to generate a list // of all files to restore and what they map to. TODO comments type Intent struct { // Namespace info DB string C string // File locations as absolute paths BSONFile file BSONSize int64 MetadataFile file // Indicates where the intent will be read from or written to Location string MetadataLocation string // Collection options Options *bson.D // File/collection size, for some prioritizer implementations. // Units don't matter as long as they are consistent for a given use case. Size int64 } func (it *Intent) Namespace() string { return it.DB + "." + it.C } func (it *Intent) IsOplog() bool { return it.DB == "" && it.C == "oplog" } func (it *Intent) IsUsers() bool { if it.C == "$admin.system.users" { return true } if it.DB == "admin" && it.C == "system.users" { return true } return false } func (it *Intent) IsRoles() bool { if it.C == "$admin.system.roles" { return true } if it.DB == "admin" && it.C == "system.roles" { return true } return false } func (it *Intent) IsAuthVersion() bool { if it.C == "$admin.system.version" { return true } if it.DB == "admin" && it.C == "system.version" { return true } return false } func (it *Intent) IsSystemIndexes() bool { return it.C == "system.indexes" && it.BSONFile != nil } func (intent *Intent) IsSpecialCollection() bool { return intent.IsSystemIndexes() || intent.IsUsers() || intent.IsRoles() || intent.IsAuthVersion() } type Manager struct { // intents are for all of the regular user created collections intents map[string]*Intent // special intents are for all of the collections that are created by mongod // and require special handling specialIntents map[string]*Intent // legacy mongorestore works in the order that paths are discovered, // so we need an ordered data structure to preserve this behavior. intentsByDiscoveryOrder []*Intent // we need different scheduling order depending on the target // mongod/mongos and whether or not we are multi threading; // the IntentPrioritizer interface encapsulates this. prioritizer IntentPrioritizer priotitizerLock *sync.Mutex // special cases that should be saved but not be part of the queue. // used to deal with oplog and user/roles restoration, which are // handled outside of the basic logic of the tool oplogIntent *Intent usersIntent *Intent rolesIntent *Intent versionIntent *Intent indexIntents map[string]*Intent } func NewIntentManager() *Manager { return &Manager{ intents: map[string]*Intent{}, specialIntents: map[string]*Intent{}, intentsByDiscoveryOrder: []*Intent{}, priotitizerLock: &sync.Mutex{}, indexIntents: map[string]*Intent{}, } } // HasConfigDBIntent returns a bool indicating if any of the intents refer to the "config" database. // This can be used to check for possible unwanted conflicts before restoring to a sharded system. func (mgr *Manager) HasConfigDBIntent() bool { for _, intent := range mgr.intentsByDiscoveryOrder { if intent.DB == "config" { return true } } return false } // Put inserts an intent into the manager. Intents for the same collection // are merged together, so that BSON and metadata files for the same collection // are returned in the same intent. func (manager *Manager) Put(intent *Intent) { if intent == nil { panic("cannot insert nil *Intent into IntentManager") } // bucket special-case collections if intent.IsOplog() { manager.oplogIntent = intent manager.specialIntents[intent.Namespace()] = intent return } if intent.IsSystemIndexes() { manager.indexIntents[intent.DB] = intent manager.specialIntents[intent.Namespace()] = intent return } if intent.IsUsers() { if intent.BSONFile != nil { manager.usersIntent = intent manager.specialIntents[intent.Namespace()] = intent } return } if intent.IsRoles() { if intent.BSONFile != nil { manager.rolesIntent = intent manager.specialIntents[intent.Namespace()] = intent } return } if intent.IsAuthVersion() { if intent.BSONFile != nil { manager.versionIntent = intent manager.specialIntents[intent.Namespace()] = intent } return } // BSON and metadata files for the same collection are merged // into the same intent. This is done to allow for simple // pairing of BSON + metadata without keeping track of the // state of the filepath walker if existing := manager.intents[intent.Namespace()]; existing != nil { // merge new intent into old intent if existing.BSONFile == nil { existing.BSONFile = intent.BSONFile } if existing.Size == 0 { existing.Size = intent.Size } if existing.Location == "" { existing.Location = intent.Location } if existing.MetadataFile == nil { existing.MetadataFile = intent.MetadataFile } if existing.MetadataLocation == "" { existing.MetadataLocation = intent.MetadataLocation } return } // if key doesn't already exist, add it to the manager manager.intents[intent.Namespace()] = intent manager.intentsByDiscoveryOrder = append(manager.intentsByDiscoveryOrder, intent) } // Intents returns a slice containing all of the intents in the manager. // Intents is not thread safe func (manager *Manager) Intents() []*Intent { allIntents := []*Intent{} for _, intent := range manager.intents { allIntents = append(allIntents, intent) } for _, intent := range manager.indexIntents { allIntents = append(allIntents, intent) } if manager.oplogIntent != nil { allIntents = append(allIntents, manager.oplogIntent) } if manager.usersIntent != nil { allIntents = append(allIntents, manager.usersIntent) } if manager.rolesIntent != nil { allIntents = append(allIntents, manager.rolesIntent) } if manager.versionIntent != nil { allIntents = append(allIntents, manager.versionIntent) } return allIntents } func (manager *Manager) IntentForNamespace(ns string) *Intent { intent := manager.intents[ns] if intent != nil { return intent } intent = manager.specialIntents[ns] return intent } // Pop returns the next available intent from the manager. If the manager is // empty, it returns nil. Pop is thread safe. func (manager *Manager) Pop() *Intent { manager.priotitizerLock.Lock() defer manager.priotitizerLock.Unlock() intent := manager.prioritizer.Get() return intent } // Peek returns a copy of a stored intent from the manager without removing // the intent. This method is useful for edge cases that need to look ahead // at what collections are in the manager before they are scheduled. // // NOTE: There are no guarantees that peek will return a usable // intent after Finalize() is called. func (manager *Manager) Peek() *Intent { if len(manager.intentsByDiscoveryOrder) == 0 { return nil } intentCopy := *manager.intentsByDiscoveryOrder[0] return &intentCopy } // Finish tells the prioritizer that mongorestore is done restoring // the given collection intent. func (manager *Manager) Finish(intent *Intent) { manager.priotitizerLock.Lock() defer manager.priotitizerLock.Unlock() manager.prioritizer.Finish(intent) } // Oplog returns the intent representing the oplog, which isn't // stored with the other intents, because it is dumped and restored in // a very different way from other collections. func (manager *Manager) Oplog() *Intent { return manager.oplogIntent } // SystemIndexes returns the system.indexes bson for a database func (manager *Manager) SystemIndexes(dbName string) *Intent { return manager.indexIntents[dbName] } // SystemIndexes returns the databases for which there are system.indexes func (manager *Manager) SystemIndexDBs() []string { databases := []string{} for dbname := range manager.indexIntents { databases = append(databases, dbname) } return databases } // Users returns the intent of the users collection to restore, a special case func (manager *Manager) Users() *Intent { return manager.usersIntent } // Roles returns the intent of the user roles collection to restore, a special case func (manager *Manager) Roles() *Intent { return manager.rolesIntent } // AuthVersion returns the intent of the version collection to restore, a special case func (manager *Manager) AuthVersion() *Intent { return manager.versionIntent } // Finalize processes the intents for prioritization. Currently only two // kinds of prioritizers are supported. No more "Put" operations may be done // after finalize is called. func (manager *Manager) Finalize(pType PriorityType) { switch pType { case Legacy: log.Log(log.DebugHigh, "finalizing intent manager with legacy prioritizer") manager.prioritizer = NewLegacyPrioritizer(manager.intentsByDiscoveryOrder) case LongestTaskFirst: log.Log(log.DebugHigh, "finalizing intent manager with longest task first prioritizer") manager.prioritizer = NewLongestTaskFirstPrioritizer(manager.intentsByDiscoveryOrder) case MultiDatabaseLTF: log.Log(log.DebugHigh, "finalizing intent manager with multi-database longest task first prioritizer") manager.prioritizer = NewMultiDatabaseLTFPrioritizer(manager.intentsByDiscoveryOrder) default: panic("cannot initialize IntentPrioritizer with unknown type") } // release these for the garbage collector and to ensure code correctness manager.intents = nil manager.intentsByDiscoveryOrder = nil } func (manager *Manager) UsePrioritizer(prioritizer IntentPrioritizer) { manager.prioritizer = prioritizer } mongo-tools-r3.2.11/common/intents/intent_prioritizer.go000066400000000000000000000157131276133741700235250ustar00rootroot00000000000000package intents import ( "container/heap" "sort" ) type PriorityType int const ( Legacy PriorityType = iota LongestTaskFirst MultiDatabaseLTF ) // IntentPrioritizer encapsulates the logic of scheduling intents // for restoration. It can know about which intents are in the // process of being restored through the "Finish" hook. // // Oplog entries and auth entries are not handled by the prioritizer, // as these are special cases handled by the regular mongorestore code. type IntentPrioritizer interface { Get() *Intent Finish(*Intent) } //===== Legacy ===== // legacyPrioritizer processes the intents in the order they were read off the // file system, keeping with legacy mongorestore behavior. type legacyPrioritizer struct { queue []*Intent } func NewLegacyPrioritizer(intentList []*Intent) *legacyPrioritizer { return &legacyPrioritizer{queue: intentList} } func (legacy *legacyPrioritizer) Get() *Intent { var intent *Intent if len(legacy.queue) == 0 { return nil } intent, legacy.queue = legacy.queue[0], legacy.queue[1:] return intent } func (legacy *legacyPrioritizer) Finish(*Intent) { // no-op return } //===== Longest Task First ===== // longestTaskFirstPrioritizer returns intents in the order of largest -> smallest, // which is better at minimizing total runtime in parallel environments than // other simple orderings. type longestTaskFirstPrioritizer struct { queue []*Intent } // NewLongestTaskFirstPrioritizer returns an initialized LTP prioritizer func NewLongestTaskFirstPrioritizer(intents []*Intent) *longestTaskFirstPrioritizer { sort.Sort(BySize(intents)) return &longestTaskFirstPrioritizer{ queue: intents, } } func (ltf *longestTaskFirstPrioritizer) Get() *Intent { var intent *Intent if len(ltf.queue) == 0 { return nil } intent, ltf.queue = ltf.queue[0], ltf.queue[1:] return intent } func (ltf *longestTaskFirstPrioritizer) Finish(*Intent) { // no-op return } // For sorting intents from largest to smallest size type BySize []*Intent func (s BySize) Len() int { return len(s) } func (s BySize) Swap(i, j int) { s[i], s[j] = s[j], s[i] } func (s BySize) Less(i, j int) bool { return s[i].Size > s[j].Size } //===== Multi Database Longest Task First ===== // multiDatabaseLTF is designed to properly schedule intents with two constraints: // 1. it is optimized to run in a multi-processor environment // 2. it is optimized for parallelism against 2.6's db-level write lock // These goals result in a design that attempts to have as many different // database's intents being restored as possible and attempts to restore the // largest collections first. // // If we can have a minimum number of collections in flight for a given db, // we avoid lock contention in an optimal way on 2.6 systems. That is, // it is better to have two restore jobs where // job1 = "test.mycollection" // job2 = "mydb2.othercollection" // so that these collections do not compete for the db-level write lock. // // We also schedule the largest jobs first, in a greedy fashion, in order // to minimize total restoration time. Each database's intents are sorted // by decreasing file size at initialization, so that the largest jobs are // run first. Admittedly, .bson file size is not a direct predictor of restore // time, but there is certainly a strong correlation. Note that this attribute // is secondary to the multi-db scheduling laid out above, since multi-db will // get us bigger wins in terms of parallelism. type multiDatabaseLTFPrioritizer struct { dbHeap heap.Interface counterMap map[string]*dbCounter } // NewMultiDatabaseLTFPrioritizer takes in a list of intents and returns an // initialized prioritizer. func NewMultiDatabaseLTFPrioritizer(intents []*Intent) *multiDatabaseLTFPrioritizer { prioritizer := &multiDatabaseLTFPrioritizer{ counterMap: map[string]*dbCounter{}, dbHeap: &DBHeap{}, } heap.Init(prioritizer.dbHeap) // first, create all database counters for _, intent := range intents { counter, exists := prioritizer.counterMap[intent.DB] if !exists { // initialize a new counter if one doesn't exist for DB counter = &dbCounter{} prioritizer.counterMap[intent.DB] = counter } counter.collections = append(counter.collections, intent) } // then ensure that all the dbCounters have sorted intents for _, counter := range prioritizer.counterMap { counter.SortCollectionsBySize() heap.Push(prioritizer.dbHeap, counter) } return prioritizer } // Get returns the next prioritized intent and updates the count of active // restores for the returned intent's DB. Get is not thread safe, and depends // on the implementation of the intent manager to lock around it. func (mdb *multiDatabaseLTFPrioritizer) Get() *Intent { if mdb.dbHeap.Len() == 0 { // we're out of things to return return nil } optimalDB := heap.Pop(mdb.dbHeap).(*dbCounter) optimalDB.active++ nextIntent := optimalDB.PopIntent() // only release the db counter if it's out of collections if len(optimalDB.collections) > 0 { heap.Push(mdb.dbHeap, optimalDB) } return nextIntent } // Finish decreases the number of active restore jobs for the given intent's // database, and reshuffles the heap accordingly. Finish is not thread safe, // and depends on the implementation of the intent manager to lock around it. func (mdb *multiDatabaseLTFPrioritizer) Finish(intent *Intent) { counter := mdb.counterMap[intent.DB] counter.active-- // only fix up the heap if the counter is still in the heap if len(counter.collections) > 0 { // This is an O(n) operation on the heap. We could make all heap // operations O(log(n)) if we set up dbCounters to track their own // position in the heap, but in practice this overhead is likely negligible. heap.Init(mdb.dbHeap) } } type dbCounter struct { active int collections []*Intent } func (dbc *dbCounter) SortCollectionsBySize() { sort.Sort(BySize(dbc.collections)) } // PopIntent returns the largest intent remaining for the database func (dbc *dbCounter) PopIntent() *Intent { var intent *Intent if len(dbc.collections) > 0 { intent, dbc.collections = dbc.collections[0], dbc.collections[1:] } return intent } // Returns the largest collection of the databases with the least active restores. // Implements the container/heap interface. None of its methods are meant to be // called directly. type DBHeap []*dbCounter func (dbh DBHeap) Len() int { return len(dbh) } func (dbh DBHeap) Swap(i, j int) { dbh[i], dbh[j] = dbh[j], dbh[i] } func (dbh DBHeap) Less(i, j int) bool { if dbh[i].active == dbh[j].active { // prioritize the largest bson file if dbs have the same number // of restorations in progress return dbh[i].collections[0].Size > dbh[j].collections[0].Size } return dbh[i].active < dbh[j].active } func (dbh *DBHeap) Push(x interface{}) { *dbh = append(*dbh, x.(*dbCounter)) } func (dbh *DBHeap) Pop() interface{} { // for container/heap package: removes the top entry and resizes the heap array old := *dbh n := len(old) toPop := old[n-1] *dbh = old[0 : n-1] return toPop } mongo-tools-r3.2.11/common/intents/intent_prioritizer_test.go000066400000000000000000000121541276133741700245600ustar00rootroot00000000000000package intents import ( "container/heap" "github.com/mongodb/mongo-tools/common/testutil" . "github.com/smartystreets/goconvey/convey" "testing" ) func TestLegacyPrioritizer(t *testing.T) { testutil.VerifyTestType(t, testutil.UnitTestType) Convey("With a legacyPrioritizer initialized with an ordered intent list", t, func() { testList := []*Intent{ &Intent{DB: "1"}, &Intent{DB: "2"}, &Intent{DB: "3"}, } legacy := NewLegacyPrioritizer(testList) So(legacy, ShouldNotBeNil) Convey("the priority should be defined by 'first-in-first-out'", func() { it0 := legacy.Get() it1 := legacy.Get() it2 := legacy.Get() it3 := legacy.Get() So(it3, ShouldBeNil) So(it0.DB, ShouldBeLessThan, it1.DB) So(it1.DB, ShouldBeLessThan, it2.DB) }) }) } func TestBasicDBHeapBehavior(t *testing.T) { var dbheap heap.Interface testutil.VerifyTestType(t, testutil.UnitTestType) Convey("With an empty dbHeap", t, func() { dbheap = &DBHeap{} heap.Init(dbheap) Convey("when inserting unordered dbCounters with different active counts", func() { heap.Push(dbheap, &dbCounter{75, nil}) heap.Push(dbheap, &dbCounter{121, nil}) heap.Push(dbheap, &dbCounter{76, nil}) heap.Push(dbheap, &dbCounter{51, nil}) heap.Push(dbheap, &dbCounter{82, nil}) heap.Push(dbheap, &dbCounter{117, nil}) heap.Push(dbheap, &dbCounter{49, nil}) heap.Push(dbheap, &dbCounter{101, nil}) heap.Push(dbheap, &dbCounter{122, nil}) heap.Push(dbheap, &dbCounter{33, nil}) heap.Push(dbheap, &dbCounter{0, nil}) Convey("they should pop in active order, least to greatest", func() { prev := -1 for dbheap.Len() > 0 { popped := heap.Pop(dbheap).(*dbCounter) So(popped.active, ShouldBeGreaterThan, prev) prev = popped.active } }) }) Convey("when inserting unordered dbCounters with different bson sizes", func() { heap.Push(dbheap, &dbCounter{0, []*Intent{&Intent{Size: 70}}}) heap.Push(dbheap, &dbCounter{0, []*Intent{&Intent{Size: 1024}}}) heap.Push(dbheap, &dbCounter{0, []*Intent{&Intent{Size: 97}}}) heap.Push(dbheap, &dbCounter{0, []*Intent{&Intent{Size: 3}}}) heap.Push(dbheap, &dbCounter{0, []*Intent{&Intent{Size: 1024 * 1024}}}) Convey("they should pop in bson size order, greatest to least", func() { prev := int64(1024*1024 + 1) // Maximum for dbheap.Len() > 0 { popped := heap.Pop(dbheap).(*dbCounter) So(popped.collections[0].Size, ShouldBeLessThan, prev) prev = popped.collections[0].Size } }) }) }) } func TestDBCounterCollectionSorting(t *testing.T) { testutil.VerifyTestType(t, testutil.UnitTestType) Convey("With a dbCounter and an unordered collection of intents", t, func() { dbc := &dbCounter{ collections: []*Intent{ &Intent{Size: 100}, &Intent{Size: 1000}, &Intent{Size: 1}, &Intent{Size: 10}, }, } Convey("popping the sorted intents should return in decreasing BSONSize", func() { dbc.SortCollectionsBySize() So(dbc.PopIntent().Size, ShouldEqual, 1000) So(dbc.PopIntent().Size, ShouldEqual, 100) So(dbc.PopIntent().Size, ShouldEqual, 10) So(dbc.PopIntent().Size, ShouldEqual, 1) So(dbc.PopIntent(), ShouldBeNil) So(dbc.PopIntent(), ShouldBeNil) }) }) } func TestSimulatedMultiDBJob(t *testing.T) { var prioritizer IntentPrioritizer testutil.VerifyTestType(t, testutil.UnitTestType) Convey("With a prioritizer initialized with a set of intents", t, func() { intents := []*Intent{ &Intent{C: "small", DB: "db2", Size: 32}, &Intent{C: "medium", DB: "db2", Size: 128}, &Intent{C: "giant", DB: "db1", Size: 1024}, &Intent{C: "tiny", DB: "db1", Size: 2}, } prioritizer = NewMultiDatabaseLTFPrioritizer(intents) So(prioritizer, ShouldNotBeNil) Convey("and a running simulation of two jobs threads:", func() { Convey("the first two intents should be of different dbs", func() { i0 := prioritizer.Get() So(i0, ShouldNotBeNil) i1 := prioritizer.Get() So(i1, ShouldNotBeNil) Convey("the first intent should be the largest bson file", func() { So(i0.C, ShouldEqual, "giant") So(i0.DB, ShouldEqual, "db1") }) Convey("the second intent should be the largest bson file of db2", func() { So(i1.C, ShouldEqual, "medium") So(i1.DB, ShouldEqual, "db2") }) Convey("with the second job finishing the smaller intents", func() { prioritizer.Finish(i1) i2 := prioritizer.Get() So(i2, ShouldNotBeNil) prioritizer.Finish(i2) i3 := prioritizer.Get() So(i3, ShouldNotBeNil) Convey("the next job should be from db2", func() { So(i2.C, ShouldEqual, "small") So(i2.DB, ShouldEqual, "db2") }) Convey("the final job should be from db1", func() { So(i3.C, ShouldEqual, "tiny") So(i3.DB, ShouldEqual, "db1") Convey("which means that there should be two active db1 jobs", func() { counter := prioritizer.(*multiDatabaseLTFPrioritizer).counterMap["db1"] So(counter.active, ShouldEqual, 2) }) }) Convey("the heap should now be empty", func() { So(prioritizer.(*multiDatabaseLTFPrioritizer).dbHeap.Len(), ShouldEqual, 0) }) }) }) }) }) } mongo-tools-r3.2.11/common/intents/intent_test.go000066400000000000000000000053771276133741700221270ustar00rootroot00000000000000package intents import ( "github.com/mongodb/mongo-tools/common/testutil" . "github.com/smartystreets/goconvey/convey" "testing" ) func TestIntentManager(t *testing.T) { var manager *Manager testutil.VerifyTestType(t, testutil.UnitTestType) Convey("With an empty IntentManager", t, func() { manager = NewIntentManager() So(manager, ShouldNotBeNil) Convey("putting a series of added bson intents", func() { manager.Put(&Intent{DB: "1", C: "1", Location: "/b1/"}) manager.Put(&Intent{DB: "1", C: "2", Location: "/b2/"}) manager.Put(&Intent{DB: "1", C: "3", Location: "/b3/"}) manager.Put(&Intent{DB: "2", C: "1", Location: "/b4/"}) So(len(manager.intentsByDiscoveryOrder), ShouldEqual, 4) So(len(manager.intents), ShouldEqual, 4) Convey("and then some matching metadata intents", func() { manager.Put(&Intent{DB: "2", C: "1", MetadataLocation: "/4m/"}) manager.Put(&Intent{DB: "1", C: "3", MetadataLocation: "/3m/"}) manager.Put(&Intent{DB: "1", C: "1", MetadataLocation: "/1m/"}) manager.Put(&Intent{DB: "1", C: "2", MetadataLocation: "/2m/"}) Convey("the size of the queue should be unchanged", func() { So(len(manager.intentsByDiscoveryOrder), ShouldEqual, 4) So(len(manager.intents), ShouldEqual, 4) }) Convey("popping them from the IntentManager", func() { manager.Finalize(Legacy) it0 := manager.Pop() it1 := manager.Pop() it2 := manager.Pop() it3 := manager.Pop() it4 := manager.Pop() So(it4, ShouldBeNil) Convey("should return them in insert order", func() { So(*it0, ShouldResemble, Intent{DB: "1", C: "1", Location: "/b1/", MetadataLocation: "/1m/"}) So(*it1, ShouldResemble, Intent{DB: "1", C: "2", Location: "/b2/", MetadataLocation: "/2m/"}) So(*it2, ShouldResemble, Intent{DB: "1", C: "3", Location: "/b3/", MetadataLocation: "/3m/"}) So(*it3, ShouldResemble, Intent{DB: "2", C: "1", Location: "/b4/", MetadataLocation: "/4m/"}) }) }) }) Convey("but adding non-matching intents", func() { manager.Put(&Intent{DB: "7", C: "49", MetadataLocation: "/5/"}) manager.Put(&Intent{DB: "27", C: "B", MetadataLocation: "/6/"}) Convey("should increase the size, because they are not merged in", func() { So(len(manager.intentsByDiscoveryOrder), ShouldEqual, 6) So(len(manager.intents), ShouldEqual, 6) }) }) Convey("using the Peek() method", func() { peeked := manager.Peek() So(peeked, ShouldNotBeNil) So(peeked, ShouldResemble, manager.intentsByDiscoveryOrder[0]) Convey("modifying the returned copy should not modify the original", func() { peeked.DB = "SHINY NEW VALUE" So(peeked, ShouldNotResemble, manager.intentsByDiscoveryOrder[0]) }) }) }) }) } mongo-tools-r3.2.11/common/json/000077500000000000000000000000001276133741700165115ustar00rootroot00000000000000mongo-tools-r3.2.11/common/json/bench_test.go000066400000000000000000000074701276133741700211660ustar00rootroot00000000000000// Copyright 2011 The Go Authors. All rights reserved. // Use of this source code is governed by a BSD-style // license that can be found in the LICENSE file. // Large data benchmark. // The JSON data is a summary of agl's changes in the // go, webkit, and chromium open source projects. // We benchmark converting between the JSON form // and in-memory data structures. package json import ( "bytes" "compress/gzip" "io/ioutil" "os" "testing" ) type codeResponse struct { Tree *codeNode `json:"tree"` Username string `json:"username"` } type codeNode struct { Name string `json:"name"` Kids []*codeNode `json:"kids"` CLWeight float64 `json:"cl_weight"` Touches int `json:"touches"` MinT int64 `json:"min_t"` MaxT int64 `json:"max_t"` MeanT int64 `json:"mean_t"` } var codeJSON []byte var codeStruct codeResponse func codeInit() { f, err := os.Open("testdata/code.json.gz") if err != nil { panic(err) } defer f.Close() gz, err := gzip.NewReader(f) if err != nil { panic(err) } data, err := ioutil.ReadAll(gz) if err != nil { panic(err) } codeJSON = data if err := Unmarshal(codeJSON, &codeStruct); err != nil { panic("unmarshal code.json: " + err.Error()) } if data, err = Marshal(&codeStruct); err != nil { panic("marshal code.json: " + err.Error()) } if !bytes.Equal(data, codeJSON) { println("different lengths", len(data), len(codeJSON)) for i := 0; i < len(data) && i < len(codeJSON); i++ { if data[i] != codeJSON[i] { println("re-marshal: changed at byte", i) println("orig: ", string(codeJSON[i-10:i+10])) println("new: ", string(data[i-10:i+10])) break } } panic("re-marshal code.json: different result") } } func BenchmarkCodeEncoder(b *testing.B) { if codeJSON == nil { b.StopTimer() codeInit() b.StartTimer() } enc := NewEncoder(ioutil.Discard) for i := 0; i < b.N; i++ { if err := enc.Encode(&codeStruct); err != nil { b.Fatal("Encode:", err) } } b.SetBytes(int64(len(codeJSON))) } func BenchmarkCodeMarshal(b *testing.B) { if codeJSON == nil { b.StopTimer() codeInit() b.StartTimer() } for i := 0; i < b.N; i++ { if _, err := Marshal(&codeStruct); err != nil { b.Fatal("Marshal:", err) } } b.SetBytes(int64(len(codeJSON))) } func BenchmarkCodeDecoder(b *testing.B) { if codeJSON == nil { b.StopTimer() codeInit() b.StartTimer() } var buf bytes.Buffer dec := NewDecoder(&buf) var r codeResponse for i := 0; i < b.N; i++ { buf.Write(codeJSON) // hide EOF buf.WriteByte('\n') buf.WriteByte('\n') buf.WriteByte('\n') if err := dec.Decode(&r); err != nil { b.Fatal("Decode:", err) } } b.SetBytes(int64(len(codeJSON))) } func BenchmarkCodeUnmarshal(b *testing.B) { if codeJSON == nil { b.StopTimer() codeInit() b.StartTimer() } for i := 0; i < b.N; i++ { var r codeResponse if err := Unmarshal(codeJSON, &r); err != nil { b.Fatal("Unmmarshal:", err) } } b.SetBytes(int64(len(codeJSON))) } func BenchmarkCodeUnmarshalReuse(b *testing.B) { if codeJSON == nil { b.StopTimer() codeInit() b.StartTimer() } var r codeResponse for i := 0; i < b.N; i++ { if err := Unmarshal(codeJSON, &r); err != nil { b.Fatal("Unmmarshal:", err) } } } func BenchmarkUnmarshalString(b *testing.B) { data := []byte(`"hello, world"`) var s string for i := 0; i < b.N; i++ { if err := Unmarshal(data, &s); err != nil { b.Fatal("Unmarshal:", err) } } } func BenchmarkUnmarshalFloat64(b *testing.B) { var f float64 data := []byte(`3.14`) for i := 0; i < b.N; i++ { if err := Unmarshal(data, &f); err != nil { b.Fatal("Unmarshal:", err) } } } func BenchmarkUnmarshalInt64(b *testing.B) { var x int64 data := []byte(`3`) for i := 0; i < b.N; i++ { if err := Unmarshal(data, &x); err != nil { b.Fatal("Unmarshal:", err) } } } mongo-tools-r3.2.11/common/json/bindata.go000066400000000000000000000033701276133741700204450ustar00rootroot00000000000000package json import ( "fmt" "reflect" ) // Transition functions for recognizing BinData. // Adapted from encoding/json/scanner.go. // stateB is the state after reading `B`. func stateB(s *scanner, c int) int { if c == 'i' { s.step = generateState("BinData", []byte("nData"), stateConstructor) return scanContinue } return s.error(c, "in literal BinData (expecting 'i')") } // Decodes a BinData literal stored in the underlying byte data into v. func (d *decodeState) storeBinData(v reflect.Value) { op := d.scanWhile(scanSkipSpace) if op != scanBeginCtor { d.error(fmt.Errorf("expected beginning of constructor")) } args, err := d.ctor("BinData", []reflect.Type{byteType, stringType}) if err != nil { d.error(err) } switch kind := v.Kind(); kind { case reflect.Interface: arg0 := byte(args[0].Uint()) arg1 := args[1].String() v.Set(reflect.ValueOf(BinData{arg0, arg1})) default: d.error(fmt.Errorf("cannot store %v value into %v type", binDataType, kind)) } } // Returns a BinData literal from the underlying byte data. func (d *decodeState) getBinData() interface{} { op := d.scanWhile(scanSkipSpace) if op != scanBeginCtor { d.error(fmt.Errorf("expected beginning of constructor")) } // Prevent d.convertNumber() from parsing the argument as a float64. useNumber := d.useNumber d.useNumber = true args := d.ctorInterface() if err := ctorNumArgsMismatch("BinData", 2, len(args)); err != nil { d.error(err) } arg0, err := args[0].(Number).Uint8() if err != nil { d.error(fmt.Errorf("expected byte for first argument of BinData constructor")) } arg1, ok := args[1].(string) if !ok { d.error(fmt.Errorf("expected string for second argument of BinData constructor")) } d.useNumber = useNumber return BinData{arg0, arg1} } mongo-tools-r3.2.11/common/json/bindata_test.go000066400000000000000000000043251276133741700215050ustar00rootroot00000000000000package json import ( "fmt" . "github.com/smartystreets/goconvey/convey" "testing" ) func TestBinDataValue(t *testing.T) { Convey("When unmarshalling JSON with BinData values", t, func() { Convey("works for a single key", func() { var jsonMap map[string]interface{} key := "key" value := `BinData(1, "xyz")` data := fmt.Sprintf(`{"%v":%v}`, key, value) err := Unmarshal([]byte(data), &jsonMap) So(err, ShouldBeNil) jsonValue, ok := jsonMap[key].(BinData) So(ok, ShouldBeTrue) So(jsonValue, ShouldResemble, BinData{1, "xyz"}) }) Convey("works for multiple keys", func() { var jsonMap map[string]interface{} key1, key2, key3 := "key1", "key2", "key3" value1, value2, value3 := `BinData(1, "abc")`, `BinData(2, "def")`, `BinData(3, "ghi")` data := fmt.Sprintf(`{"%v":%v,"%v":%v,"%v":%v}`, key1, value1, key2, value2, key3, value3) err := Unmarshal([]byte(data), &jsonMap) So(err, ShouldBeNil) jsonValue1, ok := jsonMap[key1].(BinData) So(ok, ShouldBeTrue) So(jsonValue1, ShouldResemble, BinData{1, "abc"}) jsonValue2, ok := jsonMap[key2].(BinData) So(ok, ShouldBeTrue) So(jsonValue2, ShouldResemble, BinData{2, "def"}) jsonValue3, ok := jsonMap[key3].(BinData) So(ok, ShouldBeTrue) So(jsonValue3, ShouldResemble, BinData{3, "ghi"}) }) Convey("works in an array", func() { var jsonMap map[string]interface{} key := "key" value := `BinData(42, "10")` data := fmt.Sprintf(`{"%v":[%v,%v,%v]}`, key, value, value, value) err := Unmarshal([]byte(data), &jsonMap) So(err, ShouldBeNil) jsonArray, ok := jsonMap[key].([]interface{}) So(ok, ShouldBeTrue) for _, _jsonValue := range jsonArray { jsonValue, ok := _jsonValue.(BinData) So(ok, ShouldBeTrue) So(jsonValue, ShouldResemble, BinData{42, "10"}) } }) Convey("can specify type argument using hexadecimal", func() { var jsonMap map[string]interface{} key := "key" value := `BinData(0x5f, "xyz")` data := fmt.Sprintf(`{"%v":%v}`, key, value) err := Unmarshal([]byte(data), &jsonMap) So(err, ShouldBeNil) jsonValue, ok := jsonMap[key].(BinData) So(ok, ShouldBeTrue) So(jsonValue, ShouldResemble, BinData{0x5f, "xyz"}) }) }) } mongo-tools-r3.2.11/common/json/constructor.go000066400000000000000000000051031276133741700214240ustar00rootroot00000000000000package json import ( "fmt" "reflect" ) const CtorNumArgsErrorf = "expected %v argument%v to %v constructor, but %v received" // Transition functions for recognizing object constructors. // Adapted from encoding/json/scanner.go. // stateConstructor is the state after reading a constructor name. func stateConstructor(s *scanner, c int) int { if c <= ' ' && isSpace(rune(c)) { return scanSkipSpace } if c == '(' { s.step = stateBeginCtorOrEmpty s.pushParseState(parseCtorArg) return scanBeginCtor } return s.error(c, "expected '('") } // stateBeginCtorOrEmpty is the state after reading `(`. func stateBeginCtorOrEmpty(s *scanner, c int) int { if c <= ' ' && isSpace(rune(c)) { return scanSkipSpace } if c == ')' { return stateEndValue(s, c) } return stateBeginValue(s, c) } // ctor consumes a constructor from d.data[d.off-1:], given a type specification t. // the first byte of the constructor ('(') has been read already. func (d *decodeState) ctor(name string, t []reflect.Type) ([]reflect.Value, error) { result := make([]reflect.Value, 0, len(t)) i := 0 for { // Look ahead for ) - can only happen on first iteration. op := d.scanWhile(scanSkipSpace) if op == scanEndCtor { break } // Back up so d.value can have the byte we just read. d.off-- d.scan.undo(op) if i < len(t) { v := reflect.New(t[i]).Elem() // Get argument of constructor d.value(v) result = append(result, v) i++ } // Next token must be , or ). op = d.scanWhile(scanSkipSpace) if op == scanEndCtor { break } if op != scanCtorArg { d.error(errPhase) } } return result, ctorNumArgsMismatch(name, len(t), i) } // ctorInterface is like ctor but returns []interface{}. func (d *decodeState) ctorInterface() []interface{} { var v = make([]interface{}, 0) for { // Look ahead for ) - can only happen on first iteration. op := d.scanWhile(scanSkipSpace) if op == scanEndCtor { break } // Back up so d.value can have the byte we just read. d.off-- d.scan.undo(op) v = append(v, d.valueInterface(false)) // Next token must be , or ). op = d.scanWhile(scanSkipSpace) if op == scanEndCtor { break } if op != scanCtorArg { d.error(errPhase) } } return v } // Returns a descriptive error message if the number of arguments given // to the constructor do not match what is expected. func ctorNumArgsMismatch(name string, expected, actual int) error { if expected == actual { return nil } quantifier := "" if expected > 1 { quantifier = "s" } return fmt.Errorf(CtorNumArgsErrorf, expected, quantifier, name, actual) } mongo-tools-r3.2.11/common/json/consts.go000066400000000000000000000001161276133741700203470ustar00rootroot00000000000000package json const ( ArrayStart = '[' ArraySep = ',' ArrayEnd = ']' ) mongo-tools-r3.2.11/common/json/csv_format.go000066400000000000000000000033121276133741700212020ustar00rootroot00000000000000package json import ( "encoding/base64" "fmt" "time" ) const CSV_DATE_FORMAT = "2006-01-02T15:04:05.000Z" func (b BinData) String() string { data, err := base64.StdEncoding.DecodeString(b.Base64) if err != nil { return "" // XXX: panic? } if b.Type == 0x02 { data = data[4:] // skip the first 4 bytes } return fmt.Sprintf("%X", data) // use uppercase hexadecimal } func (js JavaScript) String() string { return js.Code } func (d Date) String() string { if d.isFormatable() { n := int64(d) t := time.Unix(n/1e3, n%1e3*1e6) return t.UTC().Format(JSON_DATE_FORMAT) } // date.MarshalJSON always returns a nil err. data, _ := d.MarshalJSON() return string(data) } func (d DBRef) String() string { return fmt.Sprintf(`{ "$ref": "%v", "$id": %v, "$db": "%v" }`, d.Collection, d.Id, d.Database) } func (d DBPointer) String() string { return fmt.Sprintf(`{ "$ref": "%v", "$id": %v }`, d.Namespace, d.Id) } func (f Float) String() string { return fmt.Sprintf("%v", float64(f)) } func (_ MinKey) String() string { return "$MinKey" } func (_ MaxKey) String() string { return "$MaxKey" } func (n NumberInt) String() string { return fmt.Sprintf("%v", int32(n)) } func (n NumberLong) String() string { return fmt.Sprintf("%v", int64(n)) } // Assumes that o represents a valid ObjectId // (composed of 24 hexadecimal characters). func (o ObjectId) String() string { return fmt.Sprintf("ObjectId(%v)", string(o)) } func (r RegExp) String() string { return fmt.Sprintf("/%v/%v", r.Pattern, r.Options) } func (t Timestamp) String() string { return fmt.Sprintf(`{ "$timestamp": { "t": %v, "i": %v } }`, t.Seconds, t.Increment) } func (_ Undefined) String() string { return `{ "$undefined": true }` } mongo-tools-r3.2.11/common/json/date.go000066400000000000000000000037211276133741700177600ustar00rootroot00000000000000package json import ( "fmt" "github.com/mongodb/mongo-tools/common/util" "reflect" ) // Transition functions for recognizing Date. // Adapted from encoding/json/scanner.go. // stateDa is the state after reading `Da`. func stateDa(s *scanner, c int) int { if c == 't' { s.step = stateDat return scanContinue } return s.error(c, "in literal Date (expecting 't')") } // stateDat is the state after reading `Dat`. func stateDat(s *scanner, c int) int { if c == 'e' { s.step = stateConstructor return scanContinue } return s.error(c, "in literal Date (expecting 'e')") } // Decodes a Date literal stored in the underlying byte data into v. func (d *decodeState) storeDate(v reflect.Value) { op := d.scanWhile(scanSkipSpace) if op != scanBeginCtor { d.error(fmt.Errorf("expected beginning of constructor")) } args, err := d.ctor("Date", []reflect.Type{dateType}) if err != nil { d.error(err) } switch kind := v.Kind(); kind { case reflect.Interface: v.Set(args[0]) default: d.error(fmt.Errorf("cannot store %v value into %v type", dateType, kind)) } } // Returns a Date literal from the underlying byte data. func (d *decodeState) getDate() interface{} { op := d.scanWhile(scanSkipSpace) if op != scanBeginCtor { d.error(fmt.Errorf("expected beginning of constructor")) } // Prevent d.convertNumber() from parsing the argument as a float64. useNumber := d.useNumber d.useNumber = true args := d.ctorInterface() if err := ctorNumArgsMismatch("Date", 1, len(args)); err != nil { d.error(err) } arg0num, isNumber := args[0].(Number) if !isNumber { // validate the date format of the string _, err := util.FormatDate(args[0].(string)) if err != nil { d.error(fmt.Errorf("unexpected ISODate format")) } d.useNumber = useNumber return ISODate(args[0].(string)) } arg0, err := arg0num.Int64() if err != nil { d.error(fmt.Errorf("expected int64 for first argument of Date constructor")) } d.useNumber = useNumber return Date(arg0) } mongo-tools-r3.2.11/common/json/date_test.go000066400000000000000000000044731276133741700210240ustar00rootroot00000000000000package json import ( "fmt" . "github.com/smartystreets/goconvey/convey" "testing" ) func TestDateValue(t *testing.T) { Convey("When unmarshalling JSON with Date values", t, func() { Convey("works for a single key", func() { var jsonMap map[string]interface{} key := "key" value := "Date(123)" data := fmt.Sprintf(`{"%v":%v}`, key, value) err := Unmarshal([]byte(data), &jsonMap) So(err, ShouldBeNil) jsonValue, ok := jsonMap[key].(Date) So(ok, ShouldBeTrue) So(jsonValue, ShouldEqual, Date(123)) }) Convey("works for multiple keys", func() { var jsonMap map[string]interface{} key1, key2, key3 := "key1", "key2", "key3" value1, value2, value3 := "Date(123)", "Date(456)", "Date(789)" data := fmt.Sprintf(`{"%v":%v,"%v":%v,"%v":%v}`, key1, value1, key2, value2, key3, value3) err := Unmarshal([]byte(data), &jsonMap) So(err, ShouldBeNil) jsonValue1, ok := jsonMap[key1].(Date) So(ok, ShouldBeTrue) So(jsonValue1, ShouldEqual, Date(123)) jsonValue2, ok := jsonMap[key2].(Date) So(ok, ShouldBeTrue) So(jsonValue2, ShouldEqual, Date(456)) jsonValue3, ok := jsonMap[key3].(Date) So(ok, ShouldBeTrue) So(jsonValue3, ShouldEqual, Date(789)) }) Convey("works in an array", func() { var jsonMap map[string]interface{} key := "key" value := "Date(42)" data := fmt.Sprintf(`{"%v":[%v,%v,%v]}`, key, value, value, value) err := Unmarshal([]byte(data), &jsonMap) So(err, ShouldBeNil) jsonArray, ok := jsonMap[key].([]interface{}) So(ok, ShouldBeTrue) for _, _jsonValue := range jsonArray { jsonValue, ok := _jsonValue.(Date) So(ok, ShouldBeTrue) So(jsonValue, ShouldEqual, Date(42)) } }) Convey("cannot use string as argument", func() { var jsonMap map[string]interface{} key := "key" value := `Date("123")` data := fmt.Sprintf(`{"%v":%v}`, key, value) err := Unmarshal([]byte(data), &jsonMap) So(err, ShouldNotBeNil) }) Convey("can specify argument in hexadecimal", func() { var jsonMap map[string]interface{} key := "key" value := "Date(0x5f)" data := fmt.Sprintf(`{"%v":%v}`, key, value) err := Unmarshal([]byte(data), &jsonMap) So(err, ShouldBeNil) jsonValue, ok := jsonMap[key].(Date) So(ok, ShouldBeTrue) So(jsonValue, ShouldEqual, Date(0x5f)) }) }) } mongo-tools-r3.2.11/common/json/dbpointer.go000066400000000000000000000037631276133741700210370ustar00rootroot00000000000000package json import ( "fmt" "gopkg.in/mgo.v2/bson" "reflect" ) // Transition functions for recognizing DBPointer. // Adapted from encoding/json/scanner.go. // stateDB is the state after reading `DB`. func stateDBP(s *scanner, c int) int { if c == 'o' { s.step = generateState("DBPointer", []byte("inter"), stateConstructor) return scanContinue } return s.error(c, "in literal DBPointer (expecting 'o')") } // Decodes a DBRef literal stored in the underlying byte data into v. func (d *decodeState) storeDBPointer(v reflect.Value) { op := d.scanWhile(scanSkipSpace) if op != scanBeginCtor { d.error(fmt.Errorf("expected beginning of constructor")) } args := d.ctorInterface() if len(args) != 2 { d.error(fmt.Errorf("expected 2 arguments to DBPointer constructor, but %v received", len(args))) } switch kind := v.Kind(); kind { case reflect.Interface: arg0, ok := args[0].(string) if !ok { d.error(fmt.Errorf("expected first argument to DBPointer to be of type string")) } arg1, ok := args[1].(ObjectId) if !ok { d.error(fmt.Errorf("expected second argument to DBPointer to be of type ObjectId, but ended up being %t", args[1])) } id := bson.ObjectIdHex(string(arg1)) v.Set(reflect.ValueOf(DBPointer{arg0, id})) default: d.error(fmt.Errorf("cannot store %v value into %v type", dbPointerType, kind)) } } // Returns a DBRef literal from the underlying byte data. func (d *decodeState) getDBPointer() interface{} { op := d.scanWhile(scanSkipSpace) if op != scanBeginCtor { d.error(fmt.Errorf("expected beginning of constructor")) } args := d.ctorInterface() if err := ctorNumArgsMismatch("DBPointer", 2, len(args)); err != nil { d.error(err) } arg0, ok := args[0].(string) if !ok { d.error(fmt.Errorf("expected string for first argument of DBPointer constructor")) } arg1, ok := args[1].(ObjectId) if !ok { d.error(fmt.Errorf("expected ObjectId for second argument of DBPointer constructor")) } id := bson.ObjectIdHex(string(arg1)) return DBPointer{arg0, id} } mongo-tools-r3.2.11/common/json/dbpointer_test.go000066400000000000000000000045721276133741700220750ustar00rootroot00000000000000package json import ( "fmt" . "github.com/smartystreets/goconvey/convey" "gopkg.in/mgo.v2/bson" "testing" ) func TestDBPointerValue(t *testing.T) { Convey("Unmarshalling JSON with DBPointer values", t, func() { key := "key" value := `DBPointer("ref", ObjectId("552ffe9f5739878e73d116a9"))` Convey("works for a single key", func() { var jsonMap map[string]interface{} data := fmt.Sprintf(`{"%v":%v}`, key, value) err := Unmarshal([]byte(data), &jsonMap) So(err, ShouldBeNil) jsonValue, ok := jsonMap[key].(DBPointer) So(ok, ShouldBeTrue) So(jsonValue, ShouldResemble, DBPointer{"ref", bson.ObjectIdHex("552ffe9f5739878e73d116a9")}) }) Convey("works for multiple keys", func() { var jsonMap map[string]interface{} key1, key2, key3 := "key1", "key2", "key3" value2 := `DBPointer("ref2", ObjectId("552ffed95739878e73d116aa"))` value3 := `DBPointer("ref3", ObjectId("552fff215739878e73d116ab"))` data := fmt.Sprintf(`{"%v":%v,"%v":%v,"%v":%v}`, key1, value, key2, value2, key3, value3) err := Unmarshal([]byte(data), &jsonMap) So(err, ShouldBeNil) jsonValue1, ok := jsonMap[key1].(DBPointer) So(ok, ShouldBeTrue) So(jsonValue1, ShouldResemble, DBPointer{"ref", bson.ObjectIdHex("552ffe9f5739878e73d116a9")}) jsonValue2, ok := jsonMap[key2].(DBPointer) So(ok, ShouldBeTrue) So(jsonValue2, ShouldResemble, DBPointer{"ref2", bson.ObjectIdHex("552ffed95739878e73d116aa")}) jsonValue3, ok := jsonMap[key3].(DBPointer) So(ok, ShouldBeTrue) So(jsonValue3, ShouldResemble, DBPointer{"ref3", bson.ObjectIdHex("552fff215739878e73d116ab")}) }) Convey("works in an array", func() { var jsonMap map[string]interface{} data := fmt.Sprintf(`{"%v":[%v,%v,%v]}`, key, value, value, value) err := Unmarshal([]byte(data), &jsonMap) So(err, ShouldBeNil) jsonArray, ok := jsonMap[key].([]interface{}) So(ok, ShouldBeTrue) for _, _jsonValue := range jsonArray { jsonValue, ok := _jsonValue.(DBPointer) So(ok, ShouldBeTrue) So(jsonValue, ShouldResemble, DBPointer{"ref", bson.ObjectIdHex("552ffe9f5739878e73d116a9")}) } }) Convey("will not accept an $id type that is not an ObjectId", func() { value := `DBPointer("ref", 4)` var jsonMap map[string]interface{} data := fmt.Sprintf(`{"%v":%v}`, key, value) err := Unmarshal([]byte(data), &jsonMap) So(err, ShouldNotBeNil) }) }) } mongo-tools-r3.2.11/common/json/dbref.go000066400000000000000000000035161276133741700201270ustar00rootroot00000000000000package json import ( "fmt" "reflect" ) // Transition functions for recognizing DBRef and Dbref. // Adapted from encoding/json/scanner.go. // stateDB is the state after reading `DB`. func stateDBR(s *scanner, c int) int { if c == 'e' { s.step = generateState("DBRef", []byte("f"), stateConstructor) return scanContinue } return s.error(c, "in literal DBRef (expecting 'e')") } // stateDb is the state after reading `Db`. func stateDb(s *scanner, c int) int { if c == 'r' { s.step = generateState("Dbref", []byte("ef"), stateConstructor) return scanContinue } return s.error(c, "in literal Dbref (expecting 'r')") } // Decodes a DBRef literal stored in the underlying byte data into v. func (d *decodeState) storeDBRef(v reflect.Value) { op := d.scanWhile(scanSkipSpace) if op != scanBeginCtor { d.error(fmt.Errorf("expected beginning of constructor")) } args := d.ctorInterface() if len(args) != 2 { d.error(fmt.Errorf("expected 2 arguments to DBRef constructor, but %v received", len(args))) } switch kind := v.Kind(); kind { case reflect.Interface: arg0, ok := args[0].(string) if !ok { d.error(fmt.Errorf("expected first argument to DBRef to be of type string")) } arg1 := args[1] v.Set(reflect.ValueOf(DBRef{arg0, arg1, ""})) default: d.error(fmt.Errorf("cannot store %v value into %v type", dbRefType, kind)) } } // Returns a DBRef literal from the underlying byte data. func (d *decodeState) getDBRef() interface{} { op := d.scanWhile(scanSkipSpace) if op != scanBeginCtor { d.error(fmt.Errorf("expected beginning of constructor")) } args := d.ctorInterface() if err := ctorNumArgsMismatch("DBRef", 2, len(args)); err != nil { d.error(err) } arg0, ok := args[0].(string) if !ok { d.error(fmt.Errorf("expected string for first argument of DBRef constructor")) } return DBRef{arg0, args[1], ""} } mongo-tools-r3.2.11/common/json/dbref_test.go000066400000000000000000000214061276133741700211640ustar00rootroot00000000000000package json import ( "fmt" . "github.com/smartystreets/goconvey/convey" "math" "testing" ) func TestDBRefValue(t *testing.T) { Convey("When unmarshalling JSON with DBRef values", t, func() { Convey("works for a single key", func() { var jsonMap map[string]interface{} key := "key" value := `DBRef("ref", "123")` data := fmt.Sprintf(`{"%v":%v}`, key, value) err := Unmarshal([]byte(data), &jsonMap) So(err, ShouldBeNil) jsonValue, ok := jsonMap[key].(DBRef) So(ok, ShouldBeTrue) So(jsonValue, ShouldResemble, DBRef{"ref", "123", ""}) }) Convey("works for multiple keys", func() { var jsonMap map[string]interface{} key1, key2, key3 := "key1", "key2", "key3" value1, value2, value3 := `DBRef("ref1", "123")`, `DBRef("ref2", "456")`, `DBRef("ref3", "789")` data := fmt.Sprintf(`{"%v":%v,"%v":%v,"%v":%v}`, key1, value1, key2, value2, key3, value3) err := Unmarshal([]byte(data), &jsonMap) So(err, ShouldBeNil) jsonValue1, ok := jsonMap[key1].(DBRef) So(ok, ShouldBeTrue) So(jsonValue1, ShouldResemble, DBRef{"ref1", "123", ""}) jsonValue2, ok := jsonMap[key2].(DBRef) So(ok, ShouldBeTrue) So(jsonValue2, ShouldResemble, DBRef{"ref2", "456", ""}) jsonValue3, ok := jsonMap[key3].(DBRef) So(ok, ShouldBeTrue) So(jsonValue3, ShouldResemble, DBRef{"ref3", "789", ""}) }) Convey("works in an array", func() { var jsonMap map[string]interface{} key := "key" value := `DBRef("ref", "42")` data := fmt.Sprintf(`{"%v":[%v,%v,%v]}`, key, value, value, value) err := Unmarshal([]byte(data), &jsonMap) So(err, ShouldBeNil) jsonArray, ok := jsonMap[key].([]interface{}) So(ok, ShouldBeTrue) for _, _jsonValue := range jsonArray { jsonValue, ok := _jsonValue.(DBRef) So(ok, ShouldBeTrue) So(jsonValue, ShouldResemble, DBRef{"ref", "42", ""}) } }) Convey("can use alternative capitalization ('Dbref')", func() { var jsonMap map[string]interface{} key := "key" value := `Dbref("ref", "123")` data := fmt.Sprintf(`{"%v":%v}`, key, value) err := Unmarshal([]byte(data), &jsonMap) So(err, ShouldBeNil) jsonValue, ok := jsonMap[key].(DBRef) So(ok, ShouldBeTrue) So(jsonValue, ShouldResemble, DBRef{"ref", "123", ""}) }) Convey("can have any extended JSON value for id parameter", func() { Convey("a null literal", func() { var jsonMap map[string]interface{} key := "key" value := `DBRef("ref", null)` data := fmt.Sprintf(`{"%v":%v}`, key, value) err := Unmarshal([]byte(data), &jsonMap) So(err, ShouldBeNil) jsonValue, ok := jsonMap[key].(DBRef) So(ok, ShouldBeTrue) So(jsonValue, ShouldResemble, DBRef{"ref", nil, ""}) }) Convey("a true literal", func() { var jsonMap map[string]interface{} key := "key" value := `DBRef("ref", true)` data := fmt.Sprintf(`{"%v":%v}`, key, value) err := Unmarshal([]byte(data), &jsonMap) So(err, ShouldBeNil) jsonValue, ok := jsonMap[key].(DBRef) So(ok, ShouldBeTrue) So(jsonValue, ShouldResemble, DBRef{"ref", true, ""}) }) Convey("a false literal", func() { var jsonMap map[string]interface{} key := "key" value := `DBRef("ref", false)` data := fmt.Sprintf(`{"%v":%v}`, key, value) err := Unmarshal([]byte(data), &jsonMap) So(err, ShouldBeNil) jsonValue, ok := jsonMap[key].(DBRef) So(ok, ShouldBeTrue) So(jsonValue, ShouldResemble, DBRef{"ref", false, ""}) }) Convey("an undefined literal", func() { var jsonMap map[string]interface{} key := "key" value := `DBRef("ref", undefined)` data := fmt.Sprintf(`{"%v":%v}`, key, value) err := Unmarshal([]byte(data), &jsonMap) So(err, ShouldBeNil) jsonValue, ok := jsonMap[key].(DBRef) So(ok, ShouldBeTrue) So(jsonValue, ShouldResemble, DBRef{"ref", Undefined{}, ""}) }) Convey("a NaN literal", func() { var jsonMap map[string]interface{} key := "key" value := `DBRef("ref", NaN)` data := fmt.Sprintf(`{"%v":%v}`, key, value) err := Unmarshal([]byte(data), &jsonMap) So(err, ShouldBeNil) jsonValue, ok := jsonMap[key].(DBRef) So(ok, ShouldBeTrue) So(jsonValue.Collection, ShouldEqual, "ref") id, ok := jsonValue.Id.(float64) So(ok, ShouldBeTrue) So(math.IsNaN(id), ShouldBeTrue) }) Convey("an Infinity literal", func() { var jsonMap map[string]interface{} key := "key" value := `DBRef("ref", Infinity)` data := fmt.Sprintf(`{"%v":%v}`, key, value) err := Unmarshal([]byte(data), &jsonMap) So(err, ShouldBeNil) jsonValue, ok := jsonMap[key].(DBRef) So(ok, ShouldBeTrue) So(jsonValue.Collection, ShouldEqual, "ref") id, ok := jsonValue.Id.(float64) So(ok, ShouldBeTrue) So(math.IsInf(id, 1), ShouldBeTrue) }) Convey("a MinKey literal", func() { var jsonMap map[string]interface{} key := "key" value := `DBRef("ref", MinKey)` data := fmt.Sprintf(`{"%v":%v}`, key, value) err := Unmarshal([]byte(data), &jsonMap) So(err, ShouldBeNil) jsonValue, ok := jsonMap[key].(DBRef) So(ok, ShouldBeTrue) So(jsonValue, ShouldResemble, DBRef{"ref", MinKey{}, ""}) }) Convey("a MaxKey literal", func() { var jsonMap map[string]interface{} key := "key" value := `DBRef("ref", MaxKey)` data := fmt.Sprintf(`{"%v":%v}`, key, value) err := Unmarshal([]byte(data), &jsonMap) So(err, ShouldBeNil) jsonValue, ok := jsonMap[key].(DBRef) So(ok, ShouldBeTrue) So(jsonValue, ShouldResemble, DBRef{"ref", MaxKey{}, ""}) }) Convey("an ObjectId object", func() { var jsonMap map[string]interface{} key := "key" value := `DBRef("ref", ObjectId("123"))` data := fmt.Sprintf(`{"%v":%v}`, key, value) err := Unmarshal([]byte(data), &jsonMap) So(err, ShouldBeNil) jsonValue, ok := jsonMap[key].(DBRef) So(ok, ShouldBeTrue) So(jsonValue, ShouldResemble, DBRef{"ref", ObjectId("123"), ""}) }) Convey("a NumberInt object", func() { var jsonMap map[string]interface{} key := "key" value := `DBRef("ref", NumberInt(123))` data := fmt.Sprintf(`{"%v":%v}`, key, value) err := Unmarshal([]byte(data), &jsonMap) So(err, ShouldBeNil) jsonValue, ok := jsonMap[key].(DBRef) So(ok, ShouldBeTrue) So(jsonValue, ShouldResemble, DBRef{"ref", NumberInt(123), ""}) }) Convey("a NumberLong object", func() { var jsonMap map[string]interface{} key := "key" value := `DBRef("ref", NumberLong(123))` data := fmt.Sprintf(`{"%v":%v}`, key, value) err := Unmarshal([]byte(data), &jsonMap) So(err, ShouldBeNil) jsonValue, ok := jsonMap[key].(DBRef) So(ok, ShouldBeTrue) So(jsonValue, ShouldResemble, DBRef{"ref", NumberLong(123), ""}) }) Convey("a RegExp object", func() { var jsonMap map[string]interface{} key := "key" value := `DBRef("ref", RegExp("xyz", "i"))` data := fmt.Sprintf(`{"%v":%v}`, key, value) err := Unmarshal([]byte(data), &jsonMap) So(err, ShouldBeNil) jsonValue, ok := jsonMap[key].(DBRef) So(ok, ShouldBeTrue) So(jsonValue, ShouldResemble, DBRef{"ref", RegExp{"xyz", "i"}, ""}) }) Convey("a regular expression literal", func() { var jsonMap map[string]interface{} key := "key" value := `DBRef("ref", /xyz/i)` data := fmt.Sprintf(`{"%v":%v}`, key, value) err := Unmarshal([]byte(data), &jsonMap) So(err, ShouldBeNil) jsonValue, ok := jsonMap[key].(DBRef) So(ok, ShouldBeTrue) So(jsonValue, ShouldResemble, DBRef{"ref", RegExp{"xyz", "i"}, ""}) }) Convey("a Timestamp object", func() { var jsonMap map[string]interface{} key := "key" value := `DBRef("ref", Timestamp(123, 321))` data := fmt.Sprintf(`{"%v":%v}`, key, value) err := Unmarshal([]byte(data), &jsonMap) So(err, ShouldBeNil) jsonValue, ok := jsonMap[key].(DBRef) So(ok, ShouldBeTrue) So(jsonValue, ShouldResemble, DBRef{"ref", Timestamp{123, 321}, ""}) }) Convey("a string literal", func() { var jsonMap map[string]interface{} key := "key" value := `DBRef("ref", "xyz")` data := fmt.Sprintf(`{"%v":%v}`, key, value) err := Unmarshal([]byte(data), &jsonMap) So(err, ShouldBeNil) jsonValue, ok := jsonMap[key].(DBRef) So(ok, ShouldBeTrue) So(jsonValue, ShouldResemble, DBRef{"ref", "xyz", ""}) }) Convey("a numeric literal", func() { var jsonMap map[string]interface{} key := "key" value := `DBRef("ref", 123)` data := fmt.Sprintf(`{"%v":%v}`, key, value) err := Unmarshal([]byte(data), &jsonMap) So(err, ShouldBeNil) jsonValue, ok := jsonMap[key].(DBRef) So(ok, ShouldBeTrue) So(jsonValue.Collection, ShouldEqual, "ref") id, ok := jsonValue.Id.(int32) So(ok, ShouldBeTrue) So(id, ShouldAlmostEqual, 123) }) }) }) } mongo-tools-r3.2.11/common/json/decode.go000066400000000000000000000751731276133741700203000ustar00rootroot00000000000000// Copyright 2010 The Go Authors. All rights reserved. // Use of this source code is governed by a BSD-style // license that can be found in the LICENSE file. // Represents JSON data structure using native Go types: booleans, floats, // strings, arrays, and maps. package json import ( "bytes" "encoding" "encoding/base64" "errors" "fmt" "gopkg.in/mgo.v2/bson" "math" "reflect" "runtime" "strconv" "unicode" "unicode/utf16" "unicode/utf8" ) // Unmarshal parses the JSON-encoded data and stores the result // in the value pointed to by v. // // Unmarshal uses the inverse of the encodings that // Marshal uses, allocating maps, slices, and pointers as necessary, // with the following additional rules: // // To unmarshal JSON into a pointer, Unmarshal first handles the case of // the JSON being the JSON literal null. In that case, Unmarshal sets // the pointer to nil. Otherwise, Unmarshal unmarshals the JSON into // the value pointed at by the pointer. If the pointer is nil, Unmarshal // allocates a new value for it to point to. // // To unmarshal JSON into a struct, Unmarshal matches incoming object // keys to the keys used by Marshal (either the struct field name or its tag), // preferring an exact match but also accepting a case-insensitive match. // // To unmarshal JSON into an interface value, // Unmarshal stores one of these in the interface value: // // bool, for JSON booleans // float64, for JSON numbers // string, for JSON strings // []interface{}, for JSON arrays // map[string]interface{}, for JSON objects // nil for JSON null // // If a JSON value is not appropriate for a given target type, // or if a JSON number overflows the target type, Unmarshal // skips that field and completes the unmarshalling as best it can. // If no more serious errors are encountered, Unmarshal returns // an UnmarshalTypeError describing the earliest such error. // // The JSON null value unmarshals into an interface, map, pointer, or slice // by setting that Go value to nil. Because null is often used in JSON to mean // ``not present,'' unmarshaling a JSON null into any other Go type has no effect // on the value and produces no error. // // When unmarshaling quoted strings, invalid UTF-8 or // invalid UTF-16 surrogate pairs are not treated as an error. // Instead, they are replaced by the Unicode replacement // character U+FFFD. // func Unmarshal(data []byte, v interface{}) error { // Check for well-formedness. // Avoids filling out half a data structure // before discovering a JSON syntax error. var d decodeState err := checkValid(data, &d.scan) if err != nil { return err } d.init(data) return d.unmarshal(v) } func UnmarshalMap(data []byte) (map[string]interface{}, error) { // Check for well-formedness. // Avoids filling out half a data structure // before discovering a JSON syntax error. var d decodeState err := checkValid(data, &d.scan) if err != nil { return nil, err } d.init(data) return d.unmarshalMap() } func UnmarshalBsonD(data []byte) (bson.D, error) { // Check for well-formedness. // Avoids filling out half a data structure // before discovering a JSON syntax error. var d decodeState err := checkValid(data, &d.scan) if err != nil { return nil, err } d.init(data) return d.unmarshalBsonD() } // Unmarshaler is the interface implemented by objects // that can unmarshal a JSON description of themselves. // The input can be assumed to be a valid encoding of // a JSON value. UnmarshalJSON must copy the JSON data // if it wishes to retain the data after returning. type Unmarshaler interface { UnmarshalJSON([]byte) error } // An UnmarshalTypeError describes a JSON value that was // not appropriate for a value of a specific Go type. type UnmarshalTypeError struct { Value string // description of JSON value - "bool", "array", "number -5" Type reflect.Type // type of Go value it could not be assigned to } func (e *UnmarshalTypeError) Error() string { return "json: cannot unmarshal " + e.Value + " into Go value of type " + e.Type.String() } // An UnmarshalFieldError describes a JSON object key that // led to an unexported (and therefore unwritable) struct field. // (No longer used; kept for compatibility.) type UnmarshalFieldError struct { Key string Type reflect.Type Field reflect.StructField } func (e *UnmarshalFieldError) Error() string { return "json: cannot unmarshal object key " + strconv.Quote(e.Key) + " into unexported field " + e.Field.Name + " of type " + e.Type.String() } // An InvalidUnmarshalError describes an invalid argument passed to Unmarshal. // (The argument to Unmarshal must be a non-nil pointer.) type InvalidUnmarshalError struct { Type reflect.Type } func (e *InvalidUnmarshalError) Error() string { if e.Type == nil { return "json: Unmarshal(nil)" } if e.Type.Kind() != reflect.Ptr { return "json: Unmarshal(non-pointer " + e.Type.String() + ")" } return "json: Unmarshal(nil " + e.Type.String() + ")" } func (d *decodeState) unmarshalMap() (out map[string]interface{}, err error) { defer func() { if r := recover(); r != nil { if _, ok := r.(runtime.Error); ok { panic(r) } err = r.(error) } }() d.scan.reset() // We decode rv not rv.Elem because the Unmarshaler interface // test must be applied at the top level of the value. out = d.document() return out, d.savedError } func (d *decodeState) unmarshalBsonD() (out bson.D, err error) { defer func() { if r := recover(); r != nil { if _, ok := r.(runtime.Error); ok { panic(r) } err = r.(error) } }() d.scan.reset() // We decode rv not rv.Elem because the Unmarshaler interface // test must be applied at the top level of the value. out = d.bsonDocument() return out, d.savedError } func (d *decodeState) unmarshal(v interface{}) (err error) { defer func() { if r := recover(); r != nil { if _, ok := r.(runtime.Error); ok { panic(r) } err = r.(error) } }() rv := reflect.ValueOf(v) if rv.Kind() != reflect.Ptr || rv.IsNil() { return &InvalidUnmarshalError{reflect.TypeOf(v)} } d.scan.reset() // We decode rv not rv.Elem because the Unmarshaler interface // test must be applied at the top level of the value. d.value(rv) return d.savedError } // A Number represents a JSON number literal. type Number string // String returns the literal text of the number. func (n Number) String() string { return string(n) } // Float64 returns the number as a float64. func (n Number) Float64() (float64, error) { return strconv.ParseFloat(string(n), 64) } // Int32 returns the number as an int32. func (n Number) Int32() (int32, error) { x, err := n.Int64() return int32(x), err } // Int64 returns the number as an int64. func (n Number) Int64() (int64, error) { base := 10 if isHexPrefix(string(n)) { base = 0 // strconv.ParseInt will infer base 16 } return strconv.ParseInt(string(n), base, 64) } // Uint8 returns the number as an uint8. func (n Number) Uint8() (uint8, error) { x, err := n.Uint64() return uint8(x), err } // Uint32 returns the number as an uint32. func (n Number) Uint32() (uint32, error) { x, err := n.Uint64() return uint32(x), err } // Uint64 returns the number as an uint64. func (n Number) Uint64() (uint64, error) { base := 10 if isHexPrefix(string(n)) { base = 0 // strconv.ParseUint will infer base 16 } return strconv.ParseUint(string(n), base, 64) } // decodeState represents the state while decoding a JSON value. type decodeState struct { data []byte off int // read offset in data scan scanner nextscan scanner // for calls to nextValue savedError error tempstr string // scratch space to avoid some allocations useNumber bool } // errPhase is used for errors that should not happen unless // there is a bug in the JSON decoder or something is editing // the data slice while the decoder executes. var errPhase = errors.New("JSON decoder out of sync - data changing underfoot?") func (d *decodeState) init(data []byte) *decodeState { d.data = data d.off = 0 d.savedError = nil return d } // error aborts the decoding by panicking with err. func (d *decodeState) error(err error) { panic(err) } // saveError saves the first err it is called with, // for reporting at the end of the unmarshal. func (d *decodeState) saveError(err error) { if d.savedError == nil { d.savedError = err } } // next cuts off and returns the next full JSON value in d.data[d.off:]. // The next value is known to be an object or array, not a literal. func (d *decodeState) next() []byte { c := d.data[d.off] item, rest, err := nextValue(d.data[d.off:], &d.nextscan) if err != nil { d.error(err) } d.off = len(d.data) - len(rest) // Our scanner has seen the opening brace/bracket // and thinks we're still in the middle of the object. // invent a closing brace/bracket to get it out. if c == '{' { d.scan.step(&d.scan, '}') } else { d.scan.step(&d.scan, ']') } return item } // scanWhile processes bytes in d.data[d.off:] until it // receives a scan code not equal to op. // It updates d.off and returns the new scan code. func (d *decodeState) scanWhile(op int) int { var newOp int for { if d.off >= len(d.data) { newOp = d.scan.eof() d.off = len(d.data) + 1 // mark processed EOF with len+1 } else { c := int(d.data[d.off]) d.off++ newOp = d.scan.step(&d.scan, c) } if newOp != op { break } } return newOp } func (d *decodeState) document() map[string]interface{} { switch op := d.scanWhile(scanSkipSpace); op { default: d.error(errPhase) return nil case scanBeginObject: return d.objectInterface() } } func (d *decodeState) bsonDocument() bson.D { switch op := d.scanWhile(scanSkipSpace); op { default: d.error(errPhase) return nil case scanBeginObject: return d.bsonDInterface() } } // value decodes a JSON value from d.data[d.off:] into the value. // it updates d.off to point past the decoded value. func (d *decodeState) value(v reflect.Value) { if !v.IsValid() { _, rest, err := nextValue(d.data[d.off:], &d.nextscan) if err != nil { d.error(err) } d.off = len(d.data) - len(rest) // d.scan thinks we're still at the beginning of the item. // Feed in an empty string - the shortest, simplest value - // so that it knows we got to the end of the value. if d.scan.redo { // rewind. d.scan.redo = false d.scan.step = stateBeginValue } d.scan.step(&d.scan, '"') d.scan.step(&d.scan, '"') n := len(d.scan.parseState) if n > 0 && d.scan.parseState[n-1] == parseObjectKey { // d.scan thinks we just read an object key; finish the object d.scan.step(&d.scan, ':') d.scan.step(&d.scan, '"') d.scan.step(&d.scan, '"') d.scan.step(&d.scan, '}') } return } switch op := d.scanWhile(scanSkipSpace); op { default: d.error(errPhase) case scanBeginArray: d.array(v) case scanBeginObject: d.object(v) case scanBeginLiteral: d.literal(v) } } // indirect walks down v allocating pointers as needed, // until it gets to a non-pointer. // if it encounters an Unmarshaler, indirect stops and returns that. // if decodingNull is true, indirect stops at the last pointer so it can be set to nil. func (d *decodeState) indirect(v reflect.Value, decodingNull bool) (Unmarshaler, encoding.TextUnmarshaler, reflect.Value) { // If v is a named type and is addressable, // start with its address, so that if the type has pointer methods, // we find them. if v.Kind() != reflect.Ptr && v.Type().Name() != "" && v.CanAddr() { v = v.Addr() } for { // Load value from interface, but only if the result will be // usefully addressable. if v.Kind() == reflect.Interface && !v.IsNil() { e := v.Elem() if e.Kind() == reflect.Ptr && !e.IsNil() && (!decodingNull || e.Elem().Kind() == reflect.Ptr) { v = e continue } } if v.Kind() != reflect.Ptr { break } if v.Elem().Kind() != reflect.Ptr && decodingNull && v.CanSet() { break } if v.IsNil() { v.Set(reflect.New(v.Type().Elem())) } if v.Type().NumMethod() > 0 { if u, ok := v.Interface().(Unmarshaler); ok { return u, nil, reflect.Value{} } if u, ok := v.Interface().(encoding.TextUnmarshaler); ok { return nil, u, reflect.Value{} } } v = v.Elem() } return nil, nil, v } // array consumes an array from d.data[d.off-1:], decoding into the value v. // the first byte of the array ('[') has been read already. func (d *decodeState) array(v reflect.Value) { // Check for unmarshaler. u, ut, pv := d.indirect(v, false) if u != nil { d.off-- err := u.UnmarshalJSON(d.next()) if err != nil { d.error(err) } return } if ut != nil { d.saveError(&UnmarshalTypeError{"array", v.Type()}) d.off-- d.next() return } v = pv // Check type of target. switch v.Kind() { case reflect.Interface: if v.NumMethod() == 0 { // Decoding into nil interface? Switch to non-reflect code. v.Set(reflect.ValueOf(d.arrayInterface(false))) return } // Otherwise it's invalid. fallthrough default: d.saveError(&UnmarshalTypeError{"array", v.Type()}) d.off-- d.next() return case reflect.Array: case reflect.Slice: break } i := 0 for { // Look ahead for ] - can only happen on first iteration. op := d.scanWhile(scanSkipSpace) if op == scanEndArray { break } // Back up so d.value can have the byte we just read. d.off-- d.scan.undo(op) // Get element of array, growing if necessary. if v.Kind() == reflect.Slice { // Grow slice if necessary if i >= v.Cap() { newcap := v.Cap() + v.Cap()/2 if newcap < 4 { newcap = 4 } newv := reflect.MakeSlice(v.Type(), v.Len(), newcap) reflect.Copy(newv, v) v.Set(newv) } if i >= v.Len() { v.SetLen(i + 1) } } if i < v.Len() { // Decode into element. d.value(v.Index(i)) } else { // Ran out of fixed array: skip. d.value(reflect.Value{}) } i++ // Next token must be , or ]. op = d.scanWhile(scanSkipSpace) if op == scanEndArray { break } if op != scanArrayValue { d.error(errPhase) } } if i < v.Len() { if v.Kind() == reflect.Array { // Array. Zero the rest. z := reflect.Zero(v.Type().Elem()) for ; i < v.Len(); i++ { v.Index(i).Set(z) } } else { v.SetLen(i) } } if i == 0 && v.Kind() == reflect.Slice { v.Set(reflect.MakeSlice(v.Type(), 0, 0)) } } // object consumes an object from d.data[d.off-1:], decoding into the value v. // the first byte of the object ('{') has been read already. func (d *decodeState) object(v reflect.Value) { // Check for unmarshaler. u, ut, pv := d.indirect(v, false) if u != nil { d.off-- err := u.UnmarshalJSON(d.next()) if err != nil { d.error(err) } return } if ut != nil { d.saveError(&UnmarshalTypeError{"object", v.Type()}) d.off-- d.next() // skip over { } in input return } v = pv // Decoding into nil interface? Switch to non-reflect code. if v.Kind() == reflect.Interface && v.NumMethod() == 0 { v.Set(reflect.ValueOf(d.objectInterface())) return } // Check type of target: struct or map[string]T switch v.Kind() { case reflect.Map: // map must have string kind t := v.Type() if t.Key().Kind() != reflect.String { d.saveError(&UnmarshalTypeError{"object", v.Type()}) break } if v.IsNil() { v.Set(reflect.MakeMap(t)) } case reflect.Struct: // do nothing case reflect.Slice: // this is only a valid case if the output type is a bson.D t := v.Type() if t == orderedBSONType { v.Set(reflect.ValueOf(d.bsonDInterface())) return } fallthrough // can't unmarshal into a regular slice, goto "default" error case default: d.saveError(&UnmarshalTypeError{"object", v.Type()}) d.off-- d.next() // skip over { } in input return } var mapElem reflect.Value for { // Read opening " of string key or closing }. op := d.scanWhile(scanSkipSpace) if op == scanEndObject { // closing } - can only happen on first iteration. break } if op != scanBeginLiteral { d.error(errPhase) } // Read key. start := d.off - 1 op = d.scanWhile(scanContinue) item := d.data[start : d.off-1] key, ok := maybeUnquoteBytes(item) if !ok { d.error(errPhase) } // Figure out field corresponding to key. var subv reflect.Value destring := false // whether the value is wrapped in a string to be decoded first if v.Kind() == reflect.Map { elemType := v.Type().Elem() if !mapElem.IsValid() { mapElem = reflect.New(elemType).Elem() } else { mapElem.Set(reflect.Zero(elemType)) } subv = mapElem } else { var f *field fields := cachedTypeFields(v.Type()) for i := range fields { ff := &fields[i] if bytes.Equal(ff.nameBytes, key) { f = ff break } if f == nil && ff.equalFold(ff.nameBytes, key) { f = ff } } if f != nil { subv = v destring = f.quoted for _, i := range f.index { if subv.Kind() == reflect.Ptr { if subv.IsNil() { subv.Set(reflect.New(subv.Type().Elem())) } subv = subv.Elem() } subv = subv.Field(i) } } } // Read : before value. if op == scanSkipSpace { op = d.scanWhile(scanSkipSpace) } if op != scanObjectKey { d.error(errPhase) } // Read value. if destring { d.value(reflect.ValueOf(&d.tempstr)) d.literalStore([]byte(d.tempstr), subv, true) d.tempstr = "" // Zero scratch space for successive values. } else { d.value(subv) } // Write value back to map; // if using struct, subv points into struct already. if v.Kind() == reflect.Map { kv := reflect.ValueOf(key).Convert(v.Type().Key()) v.SetMapIndex(kv, subv) } // Next token must be , or }. op = d.scanWhile(scanSkipSpace) if op == scanEndObject { break } if op != scanObjectValue { d.error(errPhase) } } } // literal consumes a literal from d.data[d.off-1:], decoding into the value v. // The first byte of the literal has been read already // (that's how the caller knows it's a literal). func (d *decodeState) literal(v reflect.Value) { // All bytes inside literal return scanContinue op code. start := d.off - 1 op := d.scanWhile(scanContinue) // Scan read one byte too far; back up. d.off-- d.scan.undo(op) d.literalStore(d.data[start:d.off], v, false) } // convertNumber converts the number literal s to an int32, int64, a float64, // or a Number depending on the setting of d.useNumber and whether the // string is specified in hexadecimal. It does this by parsing the string to see if it // can an integer, if not it is treated as a float. If the integer is within the bounds of an int32 it // is returned as an int32. func (d *decodeState) convertNumber(s string) (interface{}, error) { if d.useNumber { return Number(s), nil } parsedInteger, err := strconv.ParseInt(s, 0, 64) if err != nil { parsedFloat, err := strconv.ParseFloat(s, 64) if err != nil { return nil, &UnmarshalTypeError{"number " + s, reflect.TypeOf(0.0)} } return parsedFloat, nil } if parsedInteger <= math.MaxInt32 && parsedInteger >= math.MinInt32 { return int32(parsedInteger), nil } return int64(parsedInteger), nil } var numberType = reflect.TypeOf(Number("")) // literalStore decodes a literal stored in item into v. // // fromQuoted indicates whether this literal came from unwrapping a // string from the ",string" struct tag option. this is used only to // produce more helpful error messages. func (d *decodeState) literalStore(item []byte, v reflect.Value, fromQuoted bool) { // Check for unmarshaler. if len(item) == 0 { // Empty string given d.saveError(fmt.Errorf("json: invalid use of ,string struct tag, trying to unmarshal %q into %v", item, v.Type())) return } wantptr := isNull(item) // null u, ut, pv := d.indirect(v, wantptr) if u != nil { err := u.UnmarshalJSON(item) if err != nil { d.error(err) } return } if ut != nil { if item[0] != '"' { if fromQuoted { d.saveError(fmt.Errorf("json: invalid use of ,string struct tag, trying to unmarshal %q into %v", item, v.Type())) } else { d.saveError(&UnmarshalTypeError{"string", v.Type()}) } } s, ok := unquoteBytes(item) if !ok { if fromQuoted { d.error(fmt.Errorf("json: invalid use of ,string struct tag, trying to unmarshal %q into %v", item, v.Type())) } else { d.error(errPhase) } } err := ut.UnmarshalText(s) if err != nil { d.error(err) } return } v = pv switch c := item[0]; { case isNull(item): // null switch v.Kind() { case reflect.Interface, reflect.Ptr, reflect.Map, reflect.Slice: v.Set(reflect.Zero(v.Type())) // otherwise, ignore null for primitives/string } case c == 't', c == 'f': // true, false value := c == 't' switch v.Kind() { default: if fromQuoted { d.saveError(fmt.Errorf("json: invalid use of ,string struct tag, trying to unmarshal %q into %v", item, v.Type())) } else { d.saveError(&UnmarshalTypeError{"bool", v.Type()}) } case reflect.Bool: v.SetBool(value) case reflect.Interface: if v.NumMethod() == 0 { v.Set(reflect.ValueOf(value)) } else { d.saveError(&UnmarshalTypeError{"bool", v.Type()}) } } case c == '"', c == '\'': // string s, ok := unquoteBytes(item) if !ok { if fromQuoted { d.error(fmt.Errorf("json: invalid use of ,string struct tag, trying to unmarshal %q into %v", item, v.Type())) } else { d.error(errPhase) } } switch v.Kind() { default: d.saveError(&UnmarshalTypeError{"string", v.Type()}) case reflect.Slice: if v.Type() != byteSliceType { d.saveError(&UnmarshalTypeError{"string", v.Type()}) break } b := make([]byte, base64.StdEncoding.DecodedLen(len(s))) n, err := base64.StdEncoding.Decode(b, s) if err != nil { d.saveError(err) break } v.Set(reflect.ValueOf(b[0:n])) case reflect.String: v.SetString(string(s)) case reflect.Interface: if v.NumMethod() == 0 { v.Set(reflect.ValueOf(string(s))) } else { d.saveError(&UnmarshalTypeError{"string", v.Type()}) } } case isNumber(item): s := string(item) switch v.Kind() { default: if v.Kind() == reflect.String && v.Type() == numberType { v.SetString(s) break } if fromQuoted { d.error(fmt.Errorf("json: invalid use of ,string struct tag, trying to unmarshal %q into %v", item, v.Type())) } else { d.error(&UnmarshalTypeError{"number", v.Type()}) } case reflect.Interface: n, err := d.convertNumber(s) if err != nil { d.saveError(err) break } if v.NumMethod() != 0 { d.saveError(&UnmarshalTypeError{"number", v.Type()}) break } v.Set(reflect.ValueOf(n)) case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64: base := 10 if isHexPrefix(s) { base = 0 // strconv.ParseInt will infer base 16 } n, err := strconv.ParseInt(s, base, 64) if err != nil || v.OverflowInt(n) { d.saveError(&UnmarshalTypeError{"number " + s, v.Type()}) break } v.SetInt(n) case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Uintptr: base := 10 if isHexPrefix(s) { base = 0 // strconv.ParseUint will infer base 16 } n, err := strconv.ParseUint(s, base, 64) if err != nil || v.OverflowUint(n) { d.saveError(&UnmarshalTypeError{"number " + s, v.Type()}) break } v.SetUint(n) case reflect.Float32, reflect.Float64: n, err := strconv.ParseFloat(s, v.Type().Bits()) if err != nil || v.OverflowFloat(n) { d.saveError(&UnmarshalTypeError{"number " + s, v.Type()}) break } v.SetFloat(n) } default: if ok := d.storeExtendedLiteral(item, v, fromQuoted); !ok { if fromQuoted { d.error(fmt.Errorf("json: invalid use of ,string struct tag, trying to unmarshal %q into %v", item, v.Type())) } else { d.error(errPhase) } } } } // The xxxInterface routines build up a value to be stored // in an empty interface. They are not strictly necessary, // but they avoid the weight of reflection in this common case. // valueInterface is like value but returns interface{}. It takes a boolean // parameter denoting whether or not the value is being unmarshalled within // a bson.D, so that bson.Ds can be the default object type when // they are inside other bson.D documents. func (d *decodeState) valueInterface(insideBSOND bool) interface{} { switch d.scanWhile(scanSkipSpace) { default: d.error(errPhase) panic("unreachable") case scanBeginArray: return d.arrayInterface(insideBSOND) case scanBeginObject: if insideBSOND { return d.bsonDInterface() } return d.objectInterface() case scanBeginLiteral: return d.literalInterface() } } // arrayInterface is like array but returns []interface{}. It takes a boolean // parameter denoting whether or not the value is being unmarshalled within // a bson.D, so that bson.Ds can be the default object type when // they are inside other bson.D documents. func (d *decodeState) arrayInterface(insideBSOND bool) []interface{} { var v = make([]interface{}, 0) for { // Look ahead for ] - can only happen on first iteration. op := d.scanWhile(scanSkipSpace) if op == scanEndArray { break } // Back up so d.value can have the byte we just read. d.off-- d.scan.undo(op) v = append(v, d.valueInterface(insideBSOND)) // Next token must be , or ]. op = d.scanWhile(scanSkipSpace) if op == scanEndArray { break } if op != scanArrayValue { d.error(errPhase) } } return v } // bsonDInterface is like object but returns bson.D{}. func (d *decodeState) bsonDInterface() bson.D { m := bson.D{} for { // Read opening " of string key or closing }. op := d.scanWhile(scanSkipSpace) if op == scanEndObject { // closing } - can only happen on first iteration. break } if op != scanBeginLiteral { d.error(errPhase) } // Read string key. start := d.off - 1 op = d.scanWhile(scanContinue) item := d.data[start : d.off-1] key, ok := maybeUnquote(item) if !ok { d.error(errPhase) } // Read : before value. if op == scanSkipSpace { op = d.scanWhile(scanSkipSpace) } if op != scanObjectKey { d.error(errPhase) } // Read value. m = append(m, bson.DocElem{key, d.valueInterface(true)}) // Next token must be , or }. op = d.scanWhile(scanSkipSpace) if op == scanEndObject { break } if op != scanObjectValue { d.error(errPhase) } } return m } // objectInterface is like object but returns map[string]interface{}. func (d *decodeState) objectInterface() map[string]interface{} { m := make(map[string]interface{}) for { // Read opening " of string key or closing }. op := d.scanWhile(scanSkipSpace) if op == scanEndObject { // closing } - can only happen on first iteration. break } if op != scanBeginLiteral { d.error(errPhase) } // Read string key. start := d.off - 1 op = d.scanWhile(scanContinue) item := d.data[start : d.off-1] key, ok := maybeUnquote(item) if !ok { d.error(errPhase) } // Read : before value. if op == scanSkipSpace { op = d.scanWhile(scanSkipSpace) } if op != scanObjectKey { d.error(errPhase) } // Read value. m[key] = d.valueInterface(false) // Next token must be , or }. op = d.scanWhile(scanSkipSpace) if op == scanEndObject { break } if op != scanObjectValue { d.error(errPhase) } } return m } // literalInterface is like literal but returns an interface value. func (d *decodeState) literalInterface() interface{} { // All bytes inside literal return scanContinue op code. start := d.off - 1 op := d.scanWhile(scanContinue) // Scan read one byte too far; back up. d.off-- d.scan.undo(op) item := d.data[start:d.off] switch c := item[0]; { case isNull(item): // null return nil case c == 't', c == 'f': // true, false return c == 't' case c == '"', c == '\'': // string s, ok := unquote(item) if !ok { d.error(errPhase) } return s case isNumber(item): // number n, err := d.convertNumber(string(item)) if err != nil { d.saveError(err) } return n default: if value, ok := d.getExtendedLiteral(item); ok { return value } d.error(errPhase) panic("unreachable") } } // getu4 decodes \uXXXX from the beginning of s, returning the hex value, // or it returns -1. func getu4(s []byte) rune { if len(s) < 6 || s[0] != '\\' || s[1] != 'u' { return -1 } r, err := strconv.ParseUint(string(s[2:6]), 16, 64) if err != nil { return -1 } return rune(r) } // unquote converts a quoted JSON string literal s into an actual string t. // The rules are different than for Go, so cannot use strconv.Unquote. func unquote(s []byte) (t string, ok bool) { s, ok = unquoteBytes(s) t = string(s) return } // unquote converts a quoted JSON string literal s into an actual string t. // The rules are different than for Go, so cannot use strconv.Unquote. func maybeUnquote(s []byte) (t string, ok bool) { s, ok = maybeUnquoteBytes(s) t = string(s) return } func unquoteBytes(s []byte) (t []byte, ok bool) { if len(s) < 2 || s[0] != '"' && s[0] != '\'' || s[len(s)-1] != '"' && s[len(s)-1] != '\'' { return } singleQuoted := s[0] == '\'' s = s[1 : len(s)-1] // Check for unusual characters. If there are none, // then no unquoting is needed, so return a slice of the // original bytes. r := 0 for r < len(s) { c := s[r] if c == '\\' || c == '"' || c == '\'' || c < ' ' { break } if c < utf8.RuneSelf { r++ continue } rr, size := utf8.DecodeRune(s[r:]) if rr == utf8.RuneError && size == 1 { break } r += size } if r == len(s) { return s, true } b := make([]byte, len(s)+2*utf8.UTFMax) w := copy(b, s[0:r]) for r < len(s) { // Out of room? Can only happen if s is full of // malformed UTF-8 and we're replacing each // byte with RuneError. if w >= len(b)-2*utf8.UTFMax { nb := make([]byte, (len(b)+utf8.UTFMax)*2) copy(nb, b[0:w]) b = nb } switch c := s[r]; { case c == '\\': r++ if r >= len(s) { return } switch s[r] { default: return case '"', '\\', '/', '\'': b[w] = s[r] r++ w++ case 'b': b[w] = '\b' r++ w++ case 'f': b[w] = '\f' r++ w++ case 'n': b[w] = '\n' r++ w++ case 'r': b[w] = '\r' r++ w++ case 't': b[w] = '\t' r++ w++ case 'u': r-- rr := getu4(s[r:]) if rr < 0 { return } r += 6 if utf16.IsSurrogate(rr) { rr1 := getu4(s[r:]) if dec := utf16.DecodeRune(rr, rr1); dec != unicode.ReplacementChar { // A valid pair; consume. r += 6 w += utf8.EncodeRune(b[w:], dec) break } // Invalid surrogate; fall back to replacement rune. rr = unicode.ReplacementChar } w += utf8.EncodeRune(b[w:], rr) } // Double quote, control characters are invalid. case !singleQuoted && c == '"', c < ' ': return // Single quote characters are invalid. case singleQuoted && c == '\'': return // ASCII case c < utf8.RuneSelf: b[w] = c r++ w++ // Coerce to well-formed UTF-8. default: rr, size := utf8.DecodeRune(s[r:]) r += size w += utf8.EncodeRune(b[w:], rr) } } return b[0:w], true } mongo-tools-r3.2.11/common/json/decode_d_test.go000066400000000000000000000073531276133741700216350ustar00rootroot00000000000000package json import ( "fmt" . "github.com/smartystreets/goconvey/convey" "gopkg.in/mgo.v2/bson" "testing" ) func TestDecodeBsonD(t *testing.T) { Convey("When unmarshalling JSON into a bson.D", t, func() { Convey("a document should be stored with keys in the same order", func() { data := `{"a":1, "b":2, "c":3, "d":4, "e":5, "f":6}` out := bson.D{} err := Unmarshal([]byte(data), &out) So(err, ShouldBeNil) So(len(out), ShouldEqual, 6) So(out[0].Name, ShouldEqual, "a") So(out[1].Name, ShouldEqual, "b") So(out[2].Name, ShouldEqual, "c") So(out[3].Name, ShouldEqual, "d") So(out[4].Name, ShouldEqual, "e") So(out[5].Name, ShouldEqual, "f") }) Convey("a nested bson.D should be parsed", func() { data := `{"a": 17, "b":{"foo":"bar", "baz":"boo"}, c:"wow" }` out := struct { A int `json:"a"` B bson.D `json:"b"` C string `json:"c"` }{} err := Unmarshal([]byte(data), &out) So(err, ShouldBeNil) So(out.A, ShouldEqual, 17) So(out.C, ShouldEqual, "wow") So(len(out.B), ShouldEqual, 2) So(out.B[0].Name, ShouldEqual, "foo") So(out.B[0].Value, ShouldEqual, "bar") So(out.B[1].Name, ShouldEqual, "baz") So(out.B[1].Value, ShouldEqual, "boo") }) Convey("objects nested within DocElems should still be parsed", func() { data := `{"a":["x", "y","z"], "b":{"foo":"bar", "baz":"boo"}}` out := bson.D{} err := Unmarshal([]byte(data), &out) So(err, ShouldBeNil) So(len(out), ShouldEqual, 2) So(out[0].Name, ShouldEqual, "a") So(out[1].Name, ShouldEqual, "b") So(out[0].Value, ShouldResemble, []interface{}{"x", "y", "z"}) So(out[1].Value, ShouldResemble, bson.D{{"foo", "bar"}, {"baz", "boo"}}) }) Convey("only subdocuments inside a bson.D should be parsed into a bson.D", func() { data := `{subA: {a:{b:{c:9}}}, subB:{a:{b:{c:9}}}}` out := struct { A interface{} `json:"subA"` B bson.D `json:"subB"` }{} err := Unmarshal([]byte(data), &out) So(err, ShouldBeNil) aMap := out.A.(map[string]interface{}) So(len(aMap), ShouldEqual, 1) aMapSub := aMap["a"].(map[string]interface{}) So(len(aMapSub), ShouldEqual, 1) aMapSubSub := aMapSub["b"].(map[string]interface{}) So(aMapSubSub["c"], ShouldEqual, 9) So(len(out.B), ShouldEqual, 1) // using string comparison for simplicity So(fmt.Sprintf("%v", out.B), ShouldEqual, fmt.Sprintf("%v", bson.D{{"a", bson.D{{"b", bson.D{{"c", 9}}}}}})) }) Convey("subdocuments inside arrays inside bson.D should be parsed into a bson.D", func() { data := `{"a":[1,2,{b:"inner"}]}` out := bson.D{} err := Unmarshal([]byte(data), &out) So(err, ShouldBeNil) So(len(out), ShouldEqual, 1) So(out[0].Value, ShouldHaveSameTypeAs, []interface{}{}) innerArray := out[0].Value.([]interface{}) So(len(innerArray), ShouldEqual, 3) So(innerArray[0], ShouldEqual, 1) So(innerArray[1], ShouldEqual, 2) So(innerArray[2], ShouldHaveSameTypeAs, bson.D{}) innerD := innerArray[2].(bson.D) So(len(innerD), ShouldEqual, 1) So(innerD[0].Name, ShouldEqual, "b") So(innerD[0].Value, ShouldEqual, "inner") }) Convey("null should be a valid value", func() { data := `{"a":true, "b":null, "c": 5}` out := bson.D{} err := Unmarshal([]byte(data), &out) So(err, ShouldBeNil) So(len(out), ShouldEqual, 3) So(out[0].Name, ShouldEqual, "a") So(out[0].Value, ShouldEqual, true) So(out[1].Name, ShouldEqual, "b") So(out[1].Value, ShouldBeNil) So(out[2].Name, ShouldEqual, "c") So(out[2].Value, ShouldEqual, 5) }) }) Convey("Unmarshalling to a non-bson.D slice types should fail", t, func() { data := `{"a":["x", "y","z"], "b":{"foo":"bar", "baz":"boo"}}` out := []interface{}{} err := Unmarshal([]byte(data), &out) So(err, ShouldNotBeNil) }) } mongo-tools-r3.2.11/common/json/decode_test.go000066400000000000000000000746161276133741700213400ustar00rootroot00000000000000// Copyright 2010 The Go Authors. All rights reserved. // Use of this source code is governed by a BSD-style // license that can be found in the LICENSE file. package json import ( "bytes" "encoding" "fmt" "image" "reflect" "strings" "testing" "time" ) type T struct { X string Y int Z int `json:"-"` } type U struct { Alphabet string `json:"alpha"` } type V struct { F1 interface{} F2 int32 F3 Number } // ifaceNumAsFloat64/ifaceNumAsNumber are used to test unmarshaling with and // without UseNumber var ifaceNumAsFloat64 = map[string]interface{}{ "k1": float64(1), "k2": "s", "k3": []interface{}{float64(1), float64(2.0), float64(3e-3)}, "k4": map[string]interface{}{"kk1": "s", "kk2": float64(2)}, } // ifaceNumAsMixedTypes is used to test unmarshalling with extended JSON var ifaceNumAsMixedTypes = map[string]interface{}{ "k1": int32(1), "k2": "s", "k3": []interface{}{int32(1), int32(2), float64(3e-3)}, "k4": map[string]interface{}{"kk1": "s", "kk2": int32(2)}, } var ifaceNumAsNumber = map[string]interface{}{ "k1": Number("1"), "k2": "s", "k3": []interface{}{Number("1"), Number("2.0"), Number("3e-3")}, "k4": map[string]interface{}{"kk1": "s", "kk2": Number("2")}, } type tx struct { x int } // A type that can unmarshal itself. type unmarshaler struct { T bool } func (u *unmarshaler) UnmarshalJSON(b []byte) error { *u = unmarshaler{true} // All we need to see that UnmarshalJSON is called. return nil } type ustruct struct { M unmarshaler } type unmarshalerText struct { T bool } // needed for re-marshaling tests func (u *unmarshalerText) MarshalText() ([]byte, error) { return []byte(""), nil } func (u *unmarshalerText) UnmarshalText(b []byte) error { *u = unmarshalerText{true} // All we need to see that UnmarshalText is called. return nil } var _ encoding.TextUnmarshaler = (*unmarshalerText)(nil) type ustructText struct { M unmarshalerText } var ( um0, um1 unmarshaler // target2 of unmarshaling ump = &um1 umtrue = unmarshaler{true} umslice = []unmarshaler{{true}} umslicep = new([]unmarshaler) umstruct = ustruct{unmarshaler{true}} um0T, um1T unmarshalerText // target2 of unmarshaling umpT = &um1T umtrueT = unmarshalerText{true} umsliceT = []unmarshalerText{{true}} umslicepT = new([]unmarshalerText) umstructT = ustructText{unmarshalerText{true}} ) // Test data structures for anonymous fields. type Point struct { Z int } type Top struct { Level0 int Embed0 *Embed0a *Embed0b `json:"e,omitempty"` // treated as named Embed0c `json:"-"` // ignored Loop Embed0p // has Point with X, Y, used Embed0q // has Point with Z, used } type Embed0 struct { Level1a int // overridden by Embed0a's Level1a with json tag Level1b int // used because Embed0a's Level1b is renamed Level1c int // used because Embed0a's Level1c is ignored Level1d int // annihilated by Embed0a's Level1d Level1e int `json:"x"` // annihilated by Embed0a.Level1e } type Embed0a struct { Level1a int `json:"Level1a,omitempty"` Level1b int `json:"LEVEL1B,omitempty"` Level1c int `json:"-"` Level1d int // annihilated by Embed0's Level1d Level1f int `json:"x"` // annihilated by Embed0's Level1e } type Embed0b Embed0 type Embed0c Embed0 type Embed0p struct { image.Point } type Embed0q struct { Point } type Loop struct { Loop1 int `json:",omitempty"` Loop2 int `json:",omitempty"` *Loop } // From reflect test: // The X in S6 and S7 annihilate, but they also block the X in S8.S9. type S5 struct { S6 S7 S8 } type S6 struct { X int } type S7 S6 type S8 struct { S9 } type S9 struct { X int Y int } // From reflect test: // The X in S11.S6 and S12.S6 annihilate, but they also block the X in S13.S8.S9. type S10 struct { S11 S12 S13 } type S11 struct { S6 } type S12 struct { S6 } type S13 struct { S8 } type unmarshalTest struct { in string ptr interface{} out interface{} err error useNumber bool } type Ambig struct { // Given "hello", the first match should win. First int `json:"HELLO"` Second int `json:"Hello"` } type XYZ struct { X interface{} Y interface{} Z interface{} } var unmarshalTests = []unmarshalTest{ // basic types {in: `true`, ptr: new(bool), out: true}, {in: `1`, ptr: new(int), out: 1}, {in: `1.2`, ptr: new(float64), out: 1.2}, {in: `-5`, ptr: new(int16), out: int16(-5)}, {in: `2`, ptr: new(Number), out: Number("2"), useNumber: true}, {in: `2`, ptr: new(Number), out: Number("2")}, {in: `2`, ptr: new(interface{}), out: int32(2)}, {in: `2`, ptr: new(interface{}), out: Number("2"), useNumber: true}, {in: `"a\u1234"`, ptr: new(string), out: "a\u1234"}, {in: `"http:\/\/"`, ptr: new(string), out: "http://"}, {in: `"g-clef: \uD834\uDD1E"`, ptr: new(string), out: "g-clef: \U0001D11E"}, {in: `"invalid: \uD834x\uDD1E"`, ptr: new(string), out: "invalid: \uFFFDx\uFFFD"}, {in: "null", ptr: new(interface{}), out: nil}, {in: `{"X": [1,2,3], "Y": 4}`, ptr: new(T), out: T{Y: 4}, err: &UnmarshalTypeError{"array", reflect.TypeOf("")}}, {in: `{"x": 1}`, ptr: new(tx), out: tx{}}, {in: `{"F1":1,"F2":2,"F3":3}`, ptr: new(V), out: V{F1: int32(1), F2: int32(2), F3: Number("3")}}, {in: `{"F1":1,"F2":2,"F3":3}`, ptr: new(V), out: V{F1: Number("1"), F2: int32(2), F3: Number("3")}, useNumber: true}, {in: `{"k1":1,"k2":"s","k3":[1,2,3e-3],"k4":{"kk1":"s","kk2":2}}`, ptr: new(interface{}), out: ifaceNumAsMixedTypes}, {in: `{"k1":1,"k2":"s","k3":[1,2.0,3e-3],"k4":{"kk1":"s","kk2":2}}`, ptr: new(interface{}), out: ifaceNumAsNumber, useNumber: true}, // raw values with whitespace {in: "\n true ", ptr: new(bool), out: true}, {in: "\t 1 ", ptr: new(int), out: 1}, {in: "\r 1.2 ", ptr: new(float64), out: 1.2}, {in: "\t -5 \n", ptr: new(int16), out: int16(-5)}, {in: "\t \"a\\u1234\" \n", ptr: new(string), out: "a\u1234"}, // Z has a "-" tag. {in: `{"Y": 1, "Z": 2}`, ptr: new(T), out: T{Y: 1}}, {in: `{"alpha": "abc", "alphabet": "xyz"}`, ptr: new(U), out: U{Alphabet: "abc"}}, {in: `{"alpha": "abc"}`, ptr: new(U), out: U{Alphabet: "abc"}}, {in: `{"alphabet": "xyz"}`, ptr: new(U), out: U{}}, // syntax errors {in: `{"X": "foo", "Y"}`, err: &SyntaxError{"invalid character '}' after object key", 17}}, {in: `[1, 2, 3+]`, err: &SyntaxError{"invalid character '+' after array element", 9}}, {in: `{"X":12x}`, err: &SyntaxError{"invalid character 'x' after object key:value pair", 8}, useNumber: true}, // raw value errors {in: "\x01 42", err: &SyntaxError{"invalid character '\\x01' looking for beginning of value", 1}}, {in: " 42 \x01", err: &SyntaxError{"invalid character '\\x01' after top-level value", 5}}, {in: "\x01 true", err: &SyntaxError{"invalid character '\\x01' looking for beginning of value", 1}}, {in: " false \x01", err: &SyntaxError{"invalid character '\\x01' after top-level value", 8}}, {in: "\x01 1.2", err: &SyntaxError{"invalid character '\\x01' looking for beginning of value", 1}}, {in: " 3.4 \x01", err: &SyntaxError{"invalid character '\\x01' after top-level value", 6}}, {in: "\x01 \"string\"", err: &SyntaxError{"invalid character '\\x01' looking for beginning of value", 1}}, {in: " \"string\" \x01", err: &SyntaxError{"invalid character '\\x01' after top-level value", 11}}, // array tests {in: `[1, 2, 3]`, ptr: new([3]int), out: [3]int{1, 2, 3}}, {in: `[1, 2, 3]`, ptr: new([1]int), out: [1]int{1}}, {in: `[1, 2, 3]`, ptr: new([5]int), out: [5]int{1, 2, 3, 0, 0}}, // empty array to interface test {in: `[]`, ptr: new([]interface{}), out: []interface{}{}}, {in: `null`, ptr: new([]interface{}), out: []interface{}(nil)}, {in: `{"T":[]}`, ptr: new(map[string]interface{}), out: map[string]interface{}{"T": []interface{}{}}}, {in: `{"T":null}`, ptr: new(map[string]interface{}), out: map[string]interface{}{"T": interface{}(nil)}}, // composite tests {in: allValueIndent, ptr: new(All), out: allValue}, {in: allValueCompact, ptr: new(All), out: allValue}, {in: allValueIndent, ptr: new(*All), out: &allValue}, {in: allValueCompact, ptr: new(*All), out: &allValue}, {in: pallValueIndent, ptr: new(All), out: pallValue}, {in: pallValueCompact, ptr: new(All), out: pallValue}, {in: pallValueIndent, ptr: new(*All), out: &pallValue}, {in: pallValueCompact, ptr: new(*All), out: &pallValue}, // unmarshal interface test {in: `{"T":false}`, ptr: &um0, out: umtrue}, // use "false" so test will fail if custom unmarshaler is not called {in: `{"T":false}`, ptr: &ump, out: &umtrue}, {in: `[{"T":false}]`, ptr: &umslice, out: umslice}, {in: `[{"T":false}]`, ptr: &umslicep, out: &umslice}, {in: `{"M":{"T":false}}`, ptr: &umstruct, out: umstruct}, // UnmarshalText interface test {in: `"X"`, ptr: &um0T, out: umtrueT}, // use "false" so test will fail if custom unmarshaler is not called {in: `"X"`, ptr: &umpT, out: &umtrueT}, {in: `["X"]`, ptr: &umsliceT, out: umsliceT}, {in: `["X"]`, ptr: &umslicepT, out: &umsliceT}, {in: `{"M":"X"}`, ptr: &umstructT, out: umstructT}, { in: `{ "Level0": 1, "Level1b": 2, "Level1c": 3, "x": 4, "Level1a": 5, "LEVEL1B": 6, "e": { "Level1a": 8, "Level1b": 9, "Level1c": 10, "Level1d": 11, "x": 12 }, "Loop1": 13, "Loop2": 14, "X": 15, "Y": 16, "Z": 17 }`, ptr: new(Top), out: Top{ Level0: 1, Embed0: Embed0{ Level1b: 2, Level1c: 3, }, Embed0a: &Embed0a{ Level1a: 5, Level1b: 6, }, Embed0b: &Embed0b{ Level1a: 8, Level1b: 9, Level1c: 10, Level1d: 11, Level1e: 12, }, Loop: Loop{ Loop1: 13, Loop2: 14, }, Embed0p: Embed0p{ Point: image.Point{X: 15, Y: 16}, }, Embed0q: Embed0q{ Point: Point{Z: 17}, }, }, }, { in: `{"hello": 1}`, ptr: new(Ambig), out: Ambig{First: 1}, }, { in: `{"X": 1,"Y":2}`, ptr: new(S5), out: S5{S8: S8{S9: S9{Y: 2}}}, }, { in: `{"X": 1,"Y":2}`, ptr: new(S10), out: S10{S13: S13{S8: S8{S9: S9{Y: 2}}}}, }, // invalid UTF-8 is coerced to valid UTF-8. { in: "\"hello\xffworld\"", ptr: new(string), out: "hello\ufffdworld", }, { in: "\"hello\xc2\xc2world\"", ptr: new(string), out: "hello\ufffd\ufffdworld", }, { in: "\"hello\xc2\xffworld\"", ptr: new(string), out: "hello\ufffd\ufffdworld", }, { in: "\"hello\\ud800world\"", ptr: new(string), out: "hello\ufffdworld", }, { in: "\"hello\\ud800\\ud800world\"", ptr: new(string), out: "hello\ufffd\ufffdworld", }, { in: "\"hello\\ud800\\ud800world\"", ptr: new(string), out: "hello\ufffd\ufffdworld", }, { in: "\"hello\xed\xa0\x80\xed\xb0\x80world\"", ptr: new(string), out: "hello\ufffd\ufffd\ufffd\ufffd\ufffd\ufffdworld", }, } func TestMarshal(t *testing.T) { b, err := Marshal(allValue) if err != nil { t.Fatalf("Marshal allValue: %v", err) } if string(b) != allValueCompact { t.Errorf("Marshal allValueCompact") diff(t, b, []byte(allValueCompact)) return } b, err = Marshal(pallValue) if err != nil { t.Fatalf("Marshal pallValue: %v", err) } if string(b) != pallValueCompact { t.Errorf("Marshal pallValueCompact") diff(t, b, []byte(pallValueCompact)) return } } var badUTF8 = []struct { in, out string }{ {"hello\xffworld", `"hello\ufffdworld"`}, {"", `""`}, {"\xff", `"\ufffd"`}, {"\xff\xff", `"\ufffd\ufffd"`}, {"a\xffb", `"a\ufffdb"`}, {"\xe6\x97\xa5\xe6\x9c\xac\xff\xaa\x9e", `"日本\ufffd\ufffd\ufffd"`}, } func TestMarshalBadUTF8(t *testing.T) { for _, tt := range badUTF8 { b, err := Marshal(tt.in) if string(b) != tt.out || err != nil { t.Errorf("Marshal(%q) = %#q, %v, want %#q, nil", tt.in, b, err, tt.out) } } } func TestMarshalNumberZeroVal(t *testing.T) { var n Number out, err := Marshal(n) if err != nil { t.Fatal(err) } outStr := string(out) if outStr != "0" { t.Fatalf("Invalid zero val for Number: %q", outStr) } } func TestMarshalEmbeds(t *testing.T) { top := &Top{ Level0: 1, Embed0: Embed0{ Level1b: 2, Level1c: 3, }, Embed0a: &Embed0a{ Level1a: 5, Level1b: 6, }, Embed0b: &Embed0b{ Level1a: 8, Level1b: 9, Level1c: 10, Level1d: 11, Level1e: 12, }, Loop: Loop{ Loop1: 13, Loop2: 14, }, Embed0p: Embed0p{ Point: image.Point{X: 15, Y: 16}, }, Embed0q: Embed0q{ Point: Point{Z: 17}, }, } b, err := Marshal(top) if err != nil { t.Fatal(err) } want := "{\"Level0\":1,\"Level1b\":2,\"Level1c\":3,\"Level1a\":5,\"LEVEL1B\":6,\"e\":{\"Level1a\":8,\"Level1b\":9,\"Level1c\":10,\"Level1d\":11,\"x\":12},\"Loop1\":13,\"Loop2\":14,\"X\":15,\"Y\":16,\"Z\":17}" if string(b) != want { t.Errorf("Wrong marshal result.\n got: %q\nwant: %q", b, want) } } func TestUnmarshal(t *testing.T) { for i, tt := range unmarshalTests { var scan scanner in := []byte(tt.in) if err := checkValid(in, &scan); err != nil { if !reflect.DeepEqual(err, tt.err) { t.Errorf("#%d: checkValid: %#v", i, err) continue } } if tt.ptr == nil { continue } // v = new(right-type) v := reflect.New(reflect.TypeOf(tt.ptr).Elem()) dec := NewDecoder(bytes.NewReader(in)) if tt.useNumber { dec.UseNumber() } if err := dec.Decode(v.Interface()); !reflect.DeepEqual(err, tt.err) { t.Errorf("#%d: %v want %v", i, err, tt.err) continue } if !reflect.DeepEqual(v.Elem().Interface(), tt.out) { t.Errorf("#%d: mismatch\nhave: %#+v\nwant: %#+v", i, v.Elem().Interface(), tt.out) data, _ := Marshal(v.Elem().Interface()) println(string(data)) data, _ = Marshal(tt.out) println(string(data)) continue } // Check round trip. if tt.err == nil { enc, err := Marshal(v.Interface()) if err != nil { t.Errorf("#%d: error re-marshaling: %v", i, err) continue } vv := reflect.New(reflect.TypeOf(tt.ptr).Elem()) dec = NewDecoder(bytes.NewReader(enc)) if tt.useNumber { dec.UseNumber() } if err := dec.Decode(vv.Interface()); err != nil { t.Errorf("#%d: error re-unmarshaling %#q: %v", i, enc, err) continue } if !reflect.DeepEqual(v.Elem().Interface(), vv.Elem().Interface()) { t.Errorf("#%d: mismatch\nhave: %#+v\nwant: %#+v", i, v.Elem().Interface(), vv.Elem().Interface()) t.Errorf(" In: %q", strings.Map(noSpace, string(in))) t.Errorf("Marshal: %q", strings.Map(noSpace, string(enc))) continue } } } } func TestUnmarshalMarshal(t *testing.T) { initBig() var v interface{} if err := Unmarshal(jsonBig, &v); err != nil { t.Fatalf("Unmarshal: %v", err) } b, err := Marshal(v) if err != nil { t.Fatalf("Marshal: %v", err) } if !bytes.Equal(jsonBig, b) { t.Errorf("Marshal jsonBig") diff(t, b, jsonBig) return } } var numberTests = []struct { in string i int64 intErr string f float64 floatErr string }{ {in: "-1.23e1", intErr: "strconv.ParseInt: parsing \"-1.23e1\": invalid syntax", f: -1.23e1}, {in: "-12", i: -12, f: -12.0}, {in: "1e1000", intErr: "strconv.ParseInt: parsing \"1e1000\": invalid syntax", floatErr: "strconv.ParseFloat: parsing \"1e1000\": value out of range"}, } // Independent of Decode, basic coverage of the accessors in Number func TestNumberAccessors(t *testing.T) { for _, tt := range numberTests { n := Number(tt.in) if s := n.String(); s != tt.in { t.Errorf("Number(%q).String() is %q", tt.in, s) } if i, err := n.Int64(); err == nil && tt.intErr == "" && i != tt.i { t.Errorf("Number(%q).Int64() is %d", tt.in, i) } else if (err == nil && tt.intErr != "") || (err != nil && err.Error() != tt.intErr) { t.Errorf("Number(%q).Int64() wanted error %q but got: %v", tt.in, tt.intErr, err) } if f, err := n.Float64(); err == nil && tt.floatErr == "" && f != tt.f { t.Errorf("Number(%q).Float64() is %g", tt.in, f) } else if (err == nil && tt.floatErr != "") || (err != nil && err.Error() != tt.floatErr) { t.Errorf("Number(%q).Float64() wanted error %q but got: %v", tt.in, tt.floatErr, err) } } } func TestLargeByteSlice(t *testing.T) { s0 := make([]byte, 2000) for i := range s0 { s0[i] = byte(i) } b, err := Marshal(s0) if err != nil { t.Fatalf("Marshal: %v", err) } var s1 []byte if err := Unmarshal(b, &s1); err != nil { t.Fatalf("Unmarshal: %v", err) } if !bytes.Equal(s0, s1) { t.Errorf("Marshal large byte slice") diff(t, s0, s1) } } type Xint struct { X int } func TestUnmarshalInterface(t *testing.T) { var xint Xint var i interface{} = &xint if err := Unmarshal([]byte(`{"X":1}`), &i); err != nil { t.Fatalf("Unmarshal: %v", err) } if xint.X != 1 { t.Fatalf("Did not write to xint") } } func TestUnmarshalPtrPtr(t *testing.T) { var xint Xint pxint := &xint if err := Unmarshal([]byte(`{"X":1}`), &pxint); err != nil { t.Fatalf("Unmarshal: %v", err) } if xint.X != 1 { t.Fatalf("Did not write to xint") } } func TestEscape(t *testing.T) { const input = `"foobar"` + " [\u2028 \u2029]" const expected = `"\"foobar\"\u003chtml\u003e [\u2028 \u2029]"` b, err := Marshal(input) if err != nil { t.Fatalf("Marshal error: %v", err) } if s := string(b); s != expected { t.Errorf("Encoding of [%s]:\n got [%s]\nwant [%s]", input, s, expected) } } // WrongString is a struct that's misusing the ,string modifier. type WrongString struct { Message string `json:"result,string"` } type wrongStringTest struct { in, err string } var wrongStringTests = []wrongStringTest{ {`{"result":"x"}`, `json: invalid use of ,string struct tag, trying to unmarshal "x" into string`}, {`{"result":"foo"}`, `json: invalid use of ,string struct tag, trying to unmarshal "foo" into string`}, {`{"result":"123"}`, `json: invalid use of ,string struct tag, trying to unmarshal "123" into string`}, } // If people misuse the ,string modifier, the error message should be // helpful, telling the user that they're doing it wrong. func TestErrorMessageFromMisusedString(t *testing.T) { for n, tt := range wrongStringTests { r := strings.NewReader(tt.in) var s WrongString err := NewDecoder(r).Decode(&s) got := fmt.Sprintf("%v", err) if got != tt.err { t.Errorf("%d. got err = %q, want %q", n, got, tt.err) } } } func noSpace(c rune) rune { if isSpace(c) { return -1 } return c } type All struct { Bool bool Int int Int8 int8 Int16 int16 Int32 int32 Int64 int64 Uint uint Uint8 uint8 Uint16 uint16 Uint32 uint32 Uint64 uint64 Uintptr uintptr Float32 float32 Float64 float64 Foo string `json:"bar"` Foo2 string `json:"bar2,dummyopt"` IntStr int64 `json:",string"` PBool *bool PInt *int PInt8 *int8 PInt16 *int16 PInt32 *int32 PInt64 *int64 PUint *uint PUint8 *uint8 PUint16 *uint16 PUint32 *uint32 PUint64 *uint64 PUintptr *uintptr PFloat32 *float32 PFloat64 *float64 String string PString *string Map map[string]Small MapP map[string]*Small PMap *map[string]Small PMapP *map[string]*Small EmptyMap map[string]Small NilMap map[string]Small Slice []Small SliceP []*Small PSlice *[]Small PSliceP *[]*Small EmptySlice []Small NilSlice []Small StringSlice []string ByteSlice []byte Small Small PSmall *Small PPSmall **Small Interface interface{} PInterface *interface{} unexported int } type Small struct { Tag string } var allValue = All{ Bool: true, Int: 2, Int8: 3, Int16: 4, Int32: 5, Int64: 6, Uint: 7, Uint8: 8, Uint16: 9, Uint32: 10, Uint64: 11, Uintptr: 12, Float32: 14.1, Float64: 15.1, Foo: "foo", Foo2: "foo2", IntStr: 42, String: "16", Map: map[string]Small{ "17": {Tag: "tag17"}, "18": {Tag: "tag18"}, }, MapP: map[string]*Small{ "19": {Tag: "tag19"}, "20": nil, }, EmptyMap: map[string]Small{}, Slice: []Small{{Tag: "tag20"}, {Tag: "tag21"}}, SliceP: []*Small{{Tag: "tag22"}, nil, {Tag: "tag23"}}, EmptySlice: []Small{}, StringSlice: []string{"str24", "str25", "str26"}, ByteSlice: []byte{27, 28, 29}, Small: Small{Tag: "tag30"}, PSmall: &Small{Tag: "tag31"}, Interface: 5.2, } var pallValue = All{ PBool: &allValue.Bool, PInt: &allValue.Int, PInt8: &allValue.Int8, PInt16: &allValue.Int16, PInt32: &allValue.Int32, PInt64: &allValue.Int64, PUint: &allValue.Uint, PUint8: &allValue.Uint8, PUint16: &allValue.Uint16, PUint32: &allValue.Uint32, PUint64: &allValue.Uint64, PUintptr: &allValue.Uintptr, PFloat32: &allValue.Float32, PFloat64: &allValue.Float64, PString: &allValue.String, PMap: &allValue.Map, PMapP: &allValue.MapP, PSlice: &allValue.Slice, PSliceP: &allValue.SliceP, PPSmall: &allValue.PSmall, PInterface: &allValue.Interface, } var allValueIndent = `{ "Bool": true, "Int": 2, "Int8": 3, "Int16": 4, "Int32": 5, "Int64": 6, "Uint": 7, "Uint8": 8, "Uint16": 9, "Uint32": 10, "Uint64": 11, "Uintptr": 12, "Float32": 14.1, "Float64": 15.1, "bar": "foo", "bar2": "foo2", "IntStr": "42", "PBool": null, "PInt": null, "PInt8": null, "PInt16": null, "PInt32": null, "PInt64": null, "PUint": null, "PUint8": null, "PUint16": null, "PUint32": null, "PUint64": null, "PUintptr": null, "PFloat32": null, "PFloat64": null, "String": "16", "PString": null, "Map": { "17": { "Tag": "tag17" }, "18": { "Tag": "tag18" } }, "MapP": { "19": { "Tag": "tag19" }, "20": null }, "PMap": null, "PMapP": null, "EmptyMap": {}, "NilMap": null, "Slice": [ { "Tag": "tag20" }, { "Tag": "tag21" } ], "SliceP": [ { "Tag": "tag22" }, null, { "Tag": "tag23" } ], "PSlice": null, "PSliceP": null, "EmptySlice": [], "NilSlice": null, "StringSlice": [ "str24", "str25", "str26" ], "ByteSlice": "Gxwd", "Small": { "Tag": "tag30" }, "PSmall": { "Tag": "tag31" }, "PPSmall": null, "Interface": 5.2, "PInterface": null }` var allValueCompact = strings.Map(noSpace, allValueIndent) var pallValueIndent = `{ "Bool": false, "Int": 0, "Int8": 0, "Int16": 0, "Int32": 0, "Int64": 0, "Uint": 0, "Uint8": 0, "Uint16": 0, "Uint32": 0, "Uint64": 0, "Uintptr": 0, "Float32": 0, "Float64": 0, "bar": "", "bar2": "", "IntStr": "0", "PBool": true, "PInt": 2, "PInt8": 3, "PInt16": 4, "PInt32": 5, "PInt64": 6, "PUint": 7, "PUint8": 8, "PUint16": 9, "PUint32": 10, "PUint64": 11, "PUintptr": 12, "PFloat32": 14.1, "PFloat64": 15.1, "String": "", "PString": "16", "Map": null, "MapP": null, "PMap": { "17": { "Tag": "tag17" }, "18": { "Tag": "tag18" } }, "PMapP": { "19": { "Tag": "tag19" }, "20": null }, "EmptyMap": null, "NilMap": null, "Slice": null, "SliceP": null, "PSlice": [ { "Tag": "tag20" }, { "Tag": "tag21" } ], "PSliceP": [ { "Tag": "tag22" }, null, { "Tag": "tag23" } ], "EmptySlice": null, "NilSlice": null, "StringSlice": null, "ByteSlice": null, "Small": { "Tag": "" }, "PSmall": null, "PPSmall": { "Tag": "tag31" }, "Interface": null, "PInterface": 5.2 }` var pallValueCompact = strings.Map(noSpace, pallValueIndent) func TestRefUnmarshal(t *testing.T) { type S struct { // Ref is defined in encode_test.go. R0 Ref R1 *Ref R2 RefText R3 *RefText } want := S{ R0: 12, R1: new(Ref), R2: 13, R3: new(RefText), } *want.R1 = 12 *want.R3 = 13 var got S if err := Unmarshal([]byte(`{"R0":"ref","R1":"ref","R2":"ref","R3":"ref"}`), &got); err != nil { t.Fatalf("Unmarshal: %v", err) } if !reflect.DeepEqual(got, want) { t.Errorf("got %+v, want %+v", got, want) } } // Test that the empty string doesn't panic decoding when ,string is specified // Issue 3450 func TestEmptyString(t *testing.T) { type T2 struct { Number1 int `json:",string"` Number2 int `json:",string"` } data := `{"Number1":"1", "Number2":""}` dec := NewDecoder(strings.NewReader(data)) var t2 T2 err := dec.Decode(&t2) if err == nil { t.Fatal("Decode: did not return error") } if t2.Number1 != 1 { t.Fatal("Decode: did not set Number1") } } // Test that the returned error is non-nil when trying to unmarshal null string into int, for successive ,string option // Issue 7046 func TestNullString(t *testing.T) { type T struct { A int `json:",string"` B int `json:",string"` } data := []byte(`{"A": "1", "B": null}`) var s T err := Unmarshal(data, &s) if err == nil { t.Fatalf("expected error; got %v", s) } } func intp(x int) *int { p := new(int) *p = x return p } func intpp(x *int) **int { pp := new(*int) *pp = x return pp } var interfaceSetTests = []struct { pre interface{} json string post interface{} }{ {"foo", `"bar"`, "bar"}, {"foo", `2`, int32(2)}, {"foo", `true`, true}, {"foo", `null`, nil}, {nil, `null`, nil}, {new(int), `null`, nil}, {(*int)(nil), `null`, nil}, {new(*int), `null`, new(*int)}, {(**int)(nil), `null`, nil}, {intp(1), `null`, nil}, {intpp(nil), `null`, intpp(nil)}, {intpp(intp(1)), `null`, intpp(nil)}, } func TestInterfaceSet(t *testing.T) { for _, tt := range interfaceSetTests { b := struct{ X interface{} }{tt.pre} blob := `{"X":` + tt.json + `}` if err := Unmarshal([]byte(blob), &b); err != nil { t.Errorf("Unmarshal %#q: %v", blob, err) continue } if !reflect.DeepEqual(b.X, tt.post) { t.Errorf("Unmarshal %#q into %#v: X=%#v, want %#v", blob, tt.pre, b.X, tt.post) } } } // JSON null values should be ignored for primitives and string values instead of resulting in an error. // Issue 2540 func TestUnmarshalNulls(t *testing.T) { jsonData := []byte(`{ "Bool" : null, "Int" : null, "Int8" : null, "Int16" : null, "Int32" : null, "Int64" : null, "Uint" : null, "Uint8" : null, "Uint16" : null, "Uint32" : null, "Uint64" : null, "Float32" : null, "Float64" : null, "String" : null}`) nulls := All{ Bool: true, Int: 2, Int8: 3, Int16: 4, Int32: 5, Int64: 6, Uint: 7, Uint8: 8, Uint16: 9, Uint32: 10, Uint64: 11, Float32: 12.1, Float64: 13.1, String: "14"} err := Unmarshal(jsonData, &nulls) if err != nil { t.Errorf("Unmarshal of null values failed: %v", err) } if !nulls.Bool || nulls.Int != 2 || nulls.Int8 != 3 || nulls.Int16 != 4 || nulls.Int32 != 5 || nulls.Int64 != 6 || nulls.Uint != 7 || nulls.Uint8 != 8 || nulls.Uint16 != 9 || nulls.Uint32 != 10 || nulls.Uint64 != 11 || nulls.Float32 != 12.1 || nulls.Float64 != 13.1 || nulls.String != "14" { t.Errorf("Unmarshal of null values affected primitives") } } func TestStringKind(t *testing.T) { type stringKind string var m1, m2 map[stringKind]int m1 = map[stringKind]int{ "foo": 42, } data, err := Marshal(m1) if err != nil { t.Errorf("Unexpected error marshalling: %v", err) } err = Unmarshal(data, &m2) if err != nil { t.Errorf("Unexpected error unmarshalling: %v", err) } if !reflect.DeepEqual(m1, m2) { t.Error("Items should be equal after encoding and then decoding") } } var decodeTypeErrorTests = []struct { dest interface{} src string }{ {new(string), `{"user": "name"}`}, // issue 4628. {new(error), `{}`}, // issue 4222 {new(error), `[]`}, {new(error), `""`}, {new(error), `123`}, {new(error), `true`}, } func TestUnmarshalTypeError(t *testing.T) { for _, item := range decodeTypeErrorTests { err := Unmarshal([]byte(item.src), item.dest) if _, ok := err.(*UnmarshalTypeError); !ok { t.Errorf("expected type error for Unmarshal(%q, type %T): got %T", item.src, item.dest, err) } } } var unmarshalSyntaxTests = []string{ "tru", "fals", "nul", "123e", `"hello`, `[1,2,3`, `{"key":1`, `{"key":1,`, } func TestUnmarshalSyntax(t *testing.T) { var x interface{} for _, src := range unmarshalSyntaxTests { err := Unmarshal([]byte(src), &x) if _, ok := err.(*SyntaxError); !ok { t.Errorf("expected syntax error for Unmarshal(%q): got %T", src, err) } } } // Test handling of unexported fields that should be ignored. // Issue 4660 type unexportedFields struct { Name string m map[string]interface{} `json:"-"` m2 map[string]interface{} `json:"abcd"` } func TestUnmarshalUnexported(t *testing.T) { input := `{"Name": "Bob", "m": {"x": 123}, "m2": {"y": 456}, "abcd": {"z": 789}}` want := &unexportedFields{Name: "Bob"} out := &unexportedFields{} err := Unmarshal([]byte(input), out) if err != nil { t.Errorf("got error %v, expected nil", err) } if !reflect.DeepEqual(out, want) { t.Errorf("got %q, want %q", out, want) } } // Time3339 is a time.Time which encodes to and from JSON // as an RFC 3339 time in UTC. type Time3339 time.Time func (t *Time3339) UnmarshalJSON(b []byte) error { if len(b) < 2 || b[0] != '"' || b[len(b)-1] != '"' { return fmt.Errorf("types: failed to unmarshal non-string value %q as an RFC 3339 time", b) } tm, err := time.Parse(time.RFC3339, string(b[1:len(b)-1])) if err != nil { return err } *t = Time3339(tm) return nil } func TestUnmarshalJSONLiteralError(t *testing.T) { var t3 Time3339 err := Unmarshal([]byte(`"0000-00-00T00:00:00Z"`), &t3) if err == nil { t.Fatalf("expected error; got time %v", time.Time(t3)) } if !strings.Contains(err.Error(), "range") { t.Errorf("got err = %v; want out of range error", err) } } // Test that extra object elements in an array do not result in a // "data changing underfoot" error. // Issue 3717 func TestSkipArrayObjects(t *testing.T) { json := `[{}]` var dest [0]interface{} err := Unmarshal([]byte(json), &dest) if err != nil { t.Errorf("got error %q, want nil", err) } } // Test semantics of pre-filled struct fields and pre-filled map fields. // Issue 4900. func TestPrefilled(t *testing.T) { ptrToMap := func(m map[string]interface{}) *map[string]interface{} { return &m } // Values here change, cannot reuse table across runs. var prefillTests = []struct { in string ptr interface{} out interface{} }{ { in: `{"X": 1, "Y": 2}`, ptr: &XYZ{X: float32(3), Y: int16(4), Z: 1.5}, out: &XYZ{X: int32(1), Y: int32(2), Z: 1.5}, }, { in: `{"X": 1, "Y": 2}`, ptr: ptrToMap(map[string]interface{}{"X": float32(3), "Y": int16(4), "Z": 1.5}), out: ptrToMap(map[string]interface{}{"X": int32(1), "Y": int32(2), "Z": 1.5}), }, } for _, tt := range prefillTests { ptrstr := fmt.Sprintf("%v", tt.ptr) err := Unmarshal([]byte(tt.in), tt.ptr) // tt.ptr edited here if err != nil { t.Errorf("Unmarshal: %v", err) } if !reflect.DeepEqual(tt.ptr, tt.out) { t.Errorf("Unmarshal(%#q, %s): have %v, want %v", tt.in, ptrstr, tt.ptr, tt.out) } } } var invalidUnmarshalTests = []struct { v interface{} want string }{ {nil, "json: Unmarshal(nil)"}, {struct{}{}, "json: Unmarshal(non-pointer struct {})"}, {(*int)(nil), "json: Unmarshal(nil *int)"}, } func TestInvalidUnmarshal(t *testing.T) { buf := []byte(`{"a":"1"}`) for _, tt := range invalidUnmarshalTests { err := Unmarshal(buf, tt.v) if err == nil { t.Errorf("Unmarshal expecting error, got nil") continue } if got := err.Error(); got != tt.want { t.Errorf("Unmarshal = %q; want %q", got, tt.want) } } } mongo-tools-r3.2.11/common/json/encode.go000066400000000000000000000741561276133741700203120ustar00rootroot00000000000000// Copyright 2010 The Go Authors. All rights reserved. // Use of this source code is governed by a BSD-style // license that can be found in the LICENSE file. // Package json implements encoding and decoding of JSON objects as defined in // RFC 4627. The mapping between JSON objects and Go values is described // in the documentation for the Marshal and Unmarshal functions. // // See "JSON and Go" for an introduction to this package: // http://golang.org/doc/articles/json_and_go.html package json import ( "bytes" "encoding" "encoding/base64" "math" "reflect" "runtime" "sort" "strconv" "strings" "sync" "unicode" "unicode/utf8" ) // Marshal returns the JSON encoding of v. // // Marshal traverses the value v recursively. // If an encountered value implements the Marshaler interface // and is not a nil pointer, Marshal calls its MarshalJSON method // to produce JSON. The nil pointer exception is not strictly necessary // but mimics a similar, necessary exception in the behavior of // UnmarshalJSON. // // Otherwise, Marshal uses the following type-dependent default encodings: // // Boolean values encode as JSON booleans. // // Floating point, integer, and Number values encode as JSON numbers. // // String values encode as JSON strings. InvalidUTF8Error will be returned // if an invalid UTF-8 sequence is encountered. // The angle brackets "<" and ">" are escaped to "\u003c" and "\u003e" // to keep some browsers from misinterpreting JSON output as HTML. // Ampersand "&" is also escaped to "\u0026" for the same reason. // // Array and slice values encode as JSON arrays, except that // []byte encodes as a base64-encoded string, and a nil slice // encodes as the null JSON object. // // Struct values encode as JSON objects. Each exported struct field // becomes a member of the object unless // - the field's tag is "-", or // - the field is empty and its tag specifies the "omitempty" option. // The empty values are false, 0, any // nil pointer or interface value, and any array, slice, map, or string of // length zero. The object's default key string is the struct field name // but can be specified in the struct field's tag value. The "json" key in // the struct field's tag value is the key name, followed by an optional comma // and options. Examples: // // // Field is ignored by this package. // Field int `json:"-"` // // // Field appears in JSON as key "myName". // Field int `json:"myName"` // // // Field appears in JSON as key "myName" and // // the field is omitted from the object if its value is empty, // // as defined above. // Field int `json:"myName,omitempty"` // // // Field appears in JSON as key "Field" (the default), but // // the field is skipped if empty. // // Note the leading comma. // Field int `json:",omitempty"` // // The "string" option signals that a field is stored as JSON inside a // JSON-encoded string. It applies only to fields of string, floating point, // or integer types. This extra level of encoding is sometimes used when // communicating with JavaScript programs: // // Int64String int64 `json:",string"` // // The key name will be used if it's a non-empty string consisting of // only Unicode letters, digits, dollar signs, percent signs, hyphens, // underscores and slashes. // // Anonymous struct fields are usually marshaled as if their inner exported fields // were fields in the outer struct, subject to the usual Go visibility rules amended // as described in the next paragraph. // An anonymous struct field with a name given in its JSON tag is treated as // having that name, rather than being anonymous. // // The Go visibility rules for struct fields are amended for JSON when // deciding which field to marshal or unmarshal. If there are // multiple fields at the same level, and that level is the least // nested (and would therefore be the nesting level selected by the // usual Go rules), the following extra rules apply: // // 1) Of those fields, if any are JSON-tagged, only tagged fields are considered, // even if there are multiple untagged fields that would otherwise conflict. // 2) If there is exactly one field (tagged or not according to the first rule), that is selected. // 3) Otherwise there are multiple fields, and all are ignored; no error occurs. // // Handling of anonymous struct fields is new in Go 1.1. // Prior to Go 1.1, anonymous struct fields were ignored. To force ignoring of // an anonymous struct field in both current and earlier versions, give the field // a JSON tag of "-". // // Map values encode as JSON objects. // The map's key type must be string; the object keys are used directly // as map keys. // // Pointer values encode as the value pointed to. // A nil pointer encodes as the null JSON object. // // Interface values encode as the value contained in the interface. // A nil interface value encodes as the null JSON object. // // Channel, complex, and function values cannot be encoded in JSON. // Attempting to encode such a value causes Marshal to return // an UnsupportedTypeError. // // JSON cannot represent cyclic data structures and Marshal does not // handle them. Passing cyclic structures to Marshal will result in // an infinite recursion. // func Marshal(v interface{}) ([]byte, error) { e := &encodeState{} err := e.marshal(v) if err != nil { return nil, err } return e.Bytes(), nil } // MarshalIndent is like Marshal but applies Indent to format the output. func MarshalIndent(v interface{}, prefix, indent string) ([]byte, error) { b, err := Marshal(v) if err != nil { return nil, err } var buf bytes.Buffer err = Indent(&buf, b, prefix, indent) if err != nil { return nil, err } return buf.Bytes(), nil } // HTMLEscape appends to dst the JSON-encoded src with <, >, &, U+2028 and U+2029 // characters inside string literals changed to \u003c, \u003e, \u0026, \u2028, \u2029 // so that the JSON will be safe to embed inside HTML

mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/client/favicon.ico000066400000000000000000000353561276133741700313450ustar00rootroot00000000000000 h6  ¨ž00 ¨%F(  × × /šš0 }} ¥/.¦ /T‚2V$e€3//}†òÈõSzì´ô}/íq:ÿS¹žÿIÿEÿSÁ™ ÿPœšÿ@F¶ÿSÁ™ ÿPššÿ@NN·¡ÿFšÿ@oóÇîœÿH!ü7Ol$0}Òbø}0.=êùûr. ¦/&.¥ }} /œ™™œ0ðÃÃñ€ D@@G‚ |€ùñÃÃð( @ × × W™ÈéøøéÈ™W $šôç¤nK::Kn¤çô›$ ‰ù¹GG¹ù‰ #ÐÛCCÚÐ#/è« ªè/#è——è# Ы  «Ð ‰Û w×þþ·xß‘¡òÿ÷¯ Û‰$ùC¸ÿÿÿÿÿ Øÿ¥»ÿÿÿÿÿÓDù$š¹_ÿÿÉC3‡ûÿÿ¥.ÿÿµ5ŸÿÿM¸š ôG¾ÿõŸÿÿ¥gÿÿG(ÿÿ†Eô Wç÷ÿ´Hÿÿ¥~ÿÿ4ÿÿœæW™¤ÿÿ‘#ÿÿ¥„ÿÿ3ÿÿ¡£™Èp0ÿÿƒÿÿ¥„ÿÿ3ÿÿ¡nÈéK7ÿÿQhrÿÿ¥„ÿÿ3ÿÿ¡Jêø:8ÿÿÇÿÿÿÿ¥„ÿÿ3ÿÿ¡:ùø:8ÿÿy››››e|ÿÿ5ÿÿš:ùéK8ÿÿ`ÿÿP1ÿÿJêÈp8ÿÿ!þÿÏ_¾ÿÿ>nÈ™¤6ÿÿÿÿÿÿÿº£™Wç+ÿÿ„:ùùvuÈà΃ æW ôGþÿIÿÿdEô ›¹Óÿàyÿÿ9¸›$ùCwÿÿ‘ ,äÿäDù$‰Ú Ôÿÿúìÿÿÿ^ Û‰ Ъ®ýÿÿÿîjªÐ #è—GQ:•è#/è« ªè0#ÐÛDDÛÐ# ‰ù¸EE¸ù‰ $šôæ£nJ::Jn£æô›$ W™ÈéøøéÈ™W ÿÀÿÿÿüð?øþðÿÿáÿÿ‡ÃÏøÃÆ`#Œ`œ@8BxB8xB8xB8`B8`B8`B8Â8À8à<x`8xøœ8ùŒñÄÿãÂÿÃáƒÿ‡ðÿÿøþüð?ÿÿÿÀÿ(0` $× × #W¶ÚëùùëÚ¶W#kÄñÿþóçßÔÔßçóþÿñÅlŒêÿýÌ…I))I…ÌýÿëŒ tóÿã%%ãÿót -ÆþèvvèþÆ-Oìþ® ®þìOjøñkiñøjjøìDCìøkOøì;<ìùO-ìñDDñì- ÆþkS€‡>MPH AyŠm+kþÆ tþ® ”îÿÿÿÿú†Zþÿå&Êþÿÿÿÿø› ¯þtóè ÎÿÿÿÿÿÿÿûU±ÿÿåÜÿÿÿÿÿÿÿÿ !éóŒÿv¦ÿÿÿþãÆÐôÿÑöÿÿå€ÿÿÿ÷ÏÜýÿÿü0vÿŒêã4öÿÿü‰1ÈÿÿÿÿåÔÿÿüd «ÿÿÿ‡âêkÿŒÿÿÿ³êÿÿÿå!ðÿÿæEÿÿÿ¾~ÿkÄý%ÒÿÿþAžÿÿÿå7ûÿÿÔ(ÿÿÿÕ "üÄ#ñÌ ûÿÿì_ÿÿÿåBÿÿÿÍ!ÿÿÿÞÉñ#Wÿ†(ÿÿÿÔ 9ÿÿÿåFÿÿÿÍÿÿÿá„ÿWþJ>ÿÿÿÇ*ÿÿÿåFÿÿÿÍÿÿÿáHþ¶ô*JÿÿÿÁÿÿÿåFÿÿÿÍÿÿÿá(ó¶ÚçQÿÿÿ¿(((>ÿÿÿåFÿÿÿÍÿÿÿáæÜëßTÿÿÿ¿)óóóôÿÿÿåFÿÿÿÍÿÿÿáßíùÔTÿÿÿ¿+ÿÿÿÿÿÿÿåFÿÿÿÍÿÿÿáÔùùÔTÿÿÿ¿%ÛÛÛÛÛÛÛÅ?þÿÿÍ#ÿÿÿÜÔùëßTÿÿÿ¿ 4ùÿÿØ,ÿÿÿÒ ßíÚçTÿÿÿ¿ìÿÿëVÿÿÿ´æܶô*Tÿÿÿ¿Ãÿÿÿ“0Íÿÿÿs(ó¶þJTÿÿÿ¿cþÿÿÿïõÿÿÿòHþWÿ…Pÿÿÿ¿ ¹ÿÿÿÿÿÿÿüs„ÿW#ñÌEÿÿÿÂTööö³ ‘èÿÿÿýÕ_Éñ#Åý%/ÿÿÿÌ`ÿÿÿ©;QL/ "üÅlÿ ýÿÿãrÿÿÿ‘~ÿlëãÏÿÿþ=¦ÿÿÿbâëŒÿv‚ÿÿÿ®âÿÿý&vÿŒóè 'ñÿÿü™#³ÿÿÿÄ!éótþ®}þÿÿÿüæäÿÿÿÿö@®þt Æþi ›ýÿÿÿÿÿÿÿÿù|iþÆ -ìñChÞÿÿÿÿÿÿÙSBñì-Oøì<3dyya/9ëùOjøìDBëøkkùñkiñùkOìþ¯!!®þìO-ÆþévvéþÆ- tóÿâ~$"~âÿót ŒêÿýÊ„H((H„ÉüÿëŒkÄñÿþóæßÔÔßæóþÿñÅl#W¶ÚëùùëÚ¶W#ÿÿÿÿÿøÿÿð€ÿÿÀþÿÿÿÿÿÿÿÿàÿüÿÿð?üÿÿø?øÿÿþðÿÿÿÿàð<áà8‡ÃÀ0Ç€0ᇀÀ @áà `qƒà àpð àxð àxð àxð àx àx àx? à|? à| `xÿà`xÿàøÿðøÿðøðxøð|ððÿñ‡ƒðÿá‡àÿáÀÀÿÿÃáÀÿÿ‡àÀÿÿðàÿÿøxÿþüÿÿø?üÿÿð?ÿÿÿàÿÿÿÿÿÿÀþÿÿð€ÿÿøÿÿÿÿÿmongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/client/index.html000066400000000000000000000416741276133741700312210ustar00rootroot00000000000000 GoConvey
PASS
Controls
NOTICE:
Coverage
Ignored
No Test Functions
No Test Files
No Go Files
Build Failures
Panics
Failures
Stories
LOG
Last test : / /
LIVE REPLAY PAUSED
mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/000077500000000000000000000000001276133741700272405ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/api/000077500000000000000000000000001276133741700300115ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/api/api.goconvey000066400000000000000000000000241276133741700323310ustar00rootroot00000000000000#ignore -timeout=1s mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/api/server.go000066400000000000000000000106441276133741700316530ustar00rootroot00000000000000package api import ( "encoding/json" "fmt" "net/http" "os" "strconv" "time" "github.com/smartystreets/goconvey/web/server/contract" "github.com/smartystreets/goconvey/web/server/messaging" ) type HTTPServer struct { watcher chan messaging.WatcherCommand executor contract.Executor latest *contract.CompleteOutput currentRoot string longpoll chan chan string paused bool } func (self *HTTPServer) ReceiveUpdate(root string, update *contract.CompleteOutput) { self.currentRoot = root self.latest = update } func (self *HTTPServer) Watch(response http.ResponseWriter, request *http.Request) { if request.Method == "POST" { self.adjustRoot(response, request) } else if request.Method == "GET" { response.Write([]byte(self.currentRoot)) } } func (self *HTTPServer) adjustRoot(response http.ResponseWriter, request *http.Request) { newRoot := self.parseQueryString("root", response, request) if newRoot == "" { return } info, err := os.Stat(newRoot) // TODO: how to unit test? if !info.IsDir() || err != nil { http.Error(response, err.Error(), http.StatusNotFound) return } self.watcher <- messaging.WatcherCommand{ Instruction: messaging.WatcherAdjustRoot, Details: newRoot, } } func (self *HTTPServer) Ignore(response http.ResponseWriter, request *http.Request) { paths := self.parseQueryString("paths", response, request) if paths != "" { self.watcher <- messaging.WatcherCommand{ Instruction: messaging.WatcherIgnore, Details: paths, } } } func (self *HTTPServer) Reinstate(response http.ResponseWriter, request *http.Request) { paths := self.parseQueryString("paths", response, request) if paths != "" { self.watcher <- messaging.WatcherCommand{ Instruction: messaging.WatcherReinstate, Details: paths, } } } func (self *HTTPServer) parseQueryString(key string, response http.ResponseWriter, request *http.Request) string { value := request.URL.Query()[key] if len(value) == 0 { http.Error(response, fmt.Sprintf("No '%s' query string parameter included!", key), http.StatusBadRequest) return "" } path := value[0] if path == "" { http.Error(response, "You must provide a non-blank path.", http.StatusBadRequest) } return path } func (self *HTTPServer) Status(response http.ResponseWriter, request *http.Request) { status := self.executor.Status() response.Write([]byte(status)) } func (self *HTTPServer) LongPollStatus(response http.ResponseWriter, request *http.Request) { if self.executor.ClearStatusFlag() { response.Write([]byte(self.executor.Status())) return } timeout, err := strconv.Atoi(request.URL.Query().Get("timeout")) if err != nil || timeout > 180000 || timeout < 0 { timeout = 60000 // default timeout is 60 seconds } myReqChan := make(chan string) select { case self.longpoll <- myReqChan: // this case means the executor's status is changing case <-time.After(time.Duration(timeout) * time.Millisecond): // this case means the executor hasn't changed status return } out := <-myReqChan if out != "" { // TODO: Why is this check necessary? Sometimes it writes empty string... response.Write([]byte(out)) } } func (self *HTTPServer) Results(response http.ResponseWriter, request *http.Request) { response.Header().Set("Content-Type", "application/json") response.Header().Set("Cache-Control", "no-cache, no-store, must-revalidate") response.Header().Set("Pragma", "no-cache") response.Header().Set("Expires", "0") if self.latest != nil { self.latest.Paused = self.paused } stuff, _ := json.Marshal(self.latest) response.Write(stuff) } func (self *HTTPServer) Execute(response http.ResponseWriter, request *http.Request) { go self.execute() } func (self *HTTPServer) execute() { self.watcher <- messaging.WatcherCommand{Instruction: messaging.WatcherExecute} } func (self *HTTPServer) TogglePause(response http.ResponseWriter, request *http.Request) { instruction := messaging.WatcherPause if self.paused { instruction = messaging.WatcherResume } self.watcher <- messaging.WatcherCommand{Instruction: instruction} self.paused = !self.paused fmt.Fprint(response, self.paused) // we could write out whatever helps keep the UI honest... } func NewHTTPServer( root string, watcher chan messaging.WatcherCommand, executor contract.Executor, status chan chan string) *HTTPServer { self := new(HTTPServer) self.currentRoot = root self.watcher = watcher self.executor = executor self.longpoll = status return self } mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/api/server_test.go000066400000000000000000000331501276133741700327070ustar00rootroot00000000000000package api import ( "encoding/json" "fmt" "net/http" "net/http/httptest" "net/url" "strings" "testing" "time" . "github.com/smartystreets/goconvey/convey" "github.com/smartystreets/goconvey/web/server/contract" "github.com/smartystreets/goconvey/web/server/messaging" ) const initialRoot = "/root/gopath/src/github.com/smartystreets/project" const nonexistentRoot = "I don't exist" const unreadableContent = "!!error!!" func TestHTTPServer(t *testing.T) { // TODO: fix the skipped tests... Convey("Subject: HttpServer responds to requests appropriately", t, func() { fixture := newServerFixture() Convey("Before any update is recieved", func() { Convey("When the update is requested", func() { update, _ := fixture.RequestLatest() Convey("No panic should occur", func() { So(func() { fixture.RequestLatest() }, ShouldNotPanic) }) Convey("The update will be empty", func() { So(update, ShouldResemble, new(contract.CompleteOutput)) }) }) }) Convey("Given an update is received", func() { fixture.ReceiveUpdate("", &contract.CompleteOutput{Revision: "asdf"}) Convey("When the update is requested", func() { update, response := fixture.RequestLatest() Convey("The server returns it", func() { So(update, ShouldResemble, &contract.CompleteOutput{Revision: "asdf"}) }) Convey("The server returns 200", func() { So(response.Code, ShouldEqual, http.StatusOK) }) Convey("The server should include important cache-related headers", func() { So(len(response.HeaderMap), ShouldEqual, 4) So(response.HeaderMap["Content-Type"][0], ShouldEqual, "application/json") So(response.HeaderMap["Cache-Control"][0], ShouldEqual, "no-cache, no-store, must-revalidate") So(response.HeaderMap["Pragma"][0], ShouldEqual, "no-cache") So(response.HeaderMap["Expires"][0], ShouldEqual, "0") }) }) }) Convey("When the root watch is queried", func() { root, status := fixture.QueryRootWatch(false) SkipConvey("The server returns it", func() { So(root, ShouldEqual, initialRoot) }) Convey("The server returns HTTP 200 - OK", func() { So(status, ShouldEqual, http.StatusOK) }) }) SkipConvey("When the root watch is adjusted", func() { Convey("But the request has no root parameter", func() { status, body := fixture.AdjustRootWatchMalformed() Convey("The server returns HTTP 400 - Bad Input", func() { So(status, ShouldEqual, http.StatusBadRequest) }) Convey("The body should contain a helpful error message", func() { So(body, ShouldEqual, "No 'root' query string parameter included!") }) Convey("The server should not change the existing root", func() { root, _ := fixture.QueryRootWatch(false) So(root, ShouldEqual, initialRoot) }) }) Convey("But the root parameter is empty", func() { status, body := fixture.AdjustRootWatch("") Convey("The server returns HTTP 400 - Bad Input", func() { So(status, ShouldEqual, http.StatusBadRequest) }) Convey("The server should provide a helpful error message", func() { So(body, ShouldEqual, "You must provide a non-blank path.") }) Convey("The server should not change the existing root", func() { root, _ := fixture.QueryRootWatch(false) So(root, ShouldEqual, initialRoot) }) }) Convey("And the new root exists", func() { status, body := fixture.AdjustRootWatch(initialRoot + "/package") Convey("The server returns HTTP 200 - OK", func() { So(status, ShouldEqual, http.StatusOK) }) Convey("The body should NOT contain any error message or content", func() { So(body, ShouldEqual, "") }) Convey("The server informs the watcher of the new root", func() { root, _ := fixture.QueryRootWatch(false) So(root, ShouldEqual, initialRoot+"/package") }) }) Convey("And the new root does NOT exist", func() { status, body := fixture.AdjustRootWatch(nonexistentRoot) Convey("The server returns HTTP 404 - Not Found", func() { So(status, ShouldEqual, http.StatusNotFound) }) Convey("The body should contain a helpful error message", func() { So(body, ShouldEqual, fmt.Sprintf("Directory does not exist: '%s'", nonexistentRoot)) }) Convey("The server should not change the existing root", func() { root, _ := fixture.QueryRootWatch(false) So(root, ShouldEqual, initialRoot) }) }) }) SkipConvey("When a packge is ignored", func() { Convey("But the request has no path parameter", func() { status, body := fixture.IgnoreMalformed() Convey("The server returns HTTP 400 - Bad Input", func() { So(status, ShouldEqual, http.StatusBadRequest) }) Convey("The body should contain a helpful error message", func() { So(body, ShouldEqual, "No 'paths' query string parameter included!") }) SkipConvey("The server should not ignore anything", func() { // So(fixture.watcher.ignored, ShouldEqual, "") }) }) Convey("But the request is blank", func() { status, body := fixture.Ignore("") Convey("The server returns HTTP 400 - Bad Input", func() { So(status, ShouldEqual, http.StatusBadRequest) }) Convey("The body should contain a helpful error message", func() { So(body, ShouldEqual, "You must provide a non-blank path.") }) }) Convey("And the request is well formed", func() { status, _ := fixture.Ignore(initialRoot) SkipConvey("The server informs the watcher", func() { // So(fixture.watcher.ignored, ShouldEqual, initialRoot) }) Convey("The server returns HTTP 200 - OK", func() { So(status, ShouldEqual, http.StatusOK) }) }) }) SkipConvey("When a package is reinstated", func() { Convey("But the request has no path parameter", func() { status, body := fixture.ReinstateMalformed() Convey("The server returns HTTP 400 - Bad Input", func() { So(status, ShouldEqual, http.StatusBadRequest) }) Convey("The body should contain a helpful error message", func() { So(body, ShouldEqual, "No 'paths' query string parameter included!") }) SkipConvey("The server should not ignore anything", func() { // So(fixture.watcher.reinstated, ShouldEqual, "") }) }) Convey("But the request is blank", func() { status, body := fixture.Reinstate("") Convey("The server returns HTTP 400 - Bad Input", func() { So(status, ShouldEqual, http.StatusBadRequest) }) Convey("The body should contain a helpful error message", func() { So(body, ShouldEqual, "You must provide a non-blank path.") }) }) Convey("And the request is well formed", func() { status, _ := fixture.Reinstate(initialRoot) SkipConvey("The server informs the watcher", func() { // So(fixture.watcher.reinstated, ShouldEqual, initialRoot) }) Convey("The server returns HTTP 200 - OK", func() { So(status, ShouldEqual, http.StatusOK) }) }) }) Convey("When the status of the executor is requested", func() { fixture.executor.status = "blah blah blah" statusCode, statusBody := fixture.RequestExecutorStatus() Convey("The server asks the executor its status and returns it", func() { So(statusBody, ShouldEqual, "blah blah blah") }) Convey("The server returns HTTP 200 - OK", func() { So(statusCode, ShouldEqual, http.StatusOK) }) }) Convey("When a manual execution of the test packages is requested", func() { status := fixture.ManualExecution() update, _ := fixture.RequestLatest() SkipConvey("The server invokes the executor using the watcher's listing and save the result", func() { So(update, ShouldResemble, &contract.CompleteOutput{Revision: initialRoot}) }) Convey("The server returns HTTP 200 - OK", func() { So(status, ShouldEqual, http.StatusOK) }) }) SkipConvey("When the pause setting is toggled via the server", func() { paused := fixture.TogglePause() SkipConvey("The pause channel buffer should have a true value", func() { // var value bool // select { // case value = <-fixture.pauseUpdate: // default: // } // So(value, ShouldBeTrue) }) Convey("The latest results should show that the server is paused", func() { fixture.ReceiveUpdate("", &contract.CompleteOutput{Revision: "asdf"}) update, _ := fixture.RequestLatest() So(update.Paused, ShouldBeTrue) }) Convey("The toggle handler should return its new status", func() { So(paused, ShouldEqual, "true") }) }) }) } /********* Server Fixture *********/ type ServerFixture struct { server *HTTPServer watcher chan messaging.WatcherCommand executor *FakeExecutor statusUpdate chan bool } func (self *ServerFixture) ReceiveUpdate(root string, update *contract.CompleteOutput) { self.server.ReceiveUpdate(root, update) } func (self *ServerFixture) RequestLatest() (*contract.CompleteOutput, *httptest.ResponseRecorder) { request, _ := http.NewRequest("GET", "http://localhost:8080/results", nil) response := httptest.NewRecorder() self.server.Results(response, request) decoder := json.NewDecoder(strings.NewReader(response.Body.String())) update := new(contract.CompleteOutput) decoder.Decode(update) return update, response } func (self *ServerFixture) QueryRootWatch(newclient bool) (string, int) { url := "http://localhost:8080/watch" if newclient { url += "?newclient=1" } request, _ := http.NewRequest("GET", url, nil) response := httptest.NewRecorder() self.server.Watch(response, request) return strings.TrimSpace(response.Body.String()), response.Code } func (self *ServerFixture) AdjustRootWatchMalformed() (status int, body string) { request, _ := http.NewRequest("POST", "http://localhost:8080/watch", nil) response := httptest.NewRecorder() self.server.Watch(response, request) status, body = response.Code, strings.TrimSpace(response.Body.String()) return } func (self *ServerFixture) AdjustRootWatch(newRoot string) (status int, body string) { escapedRoot := url.QueryEscape(newRoot) request, _ := http.NewRequest("POST", "http://localhost:8080/watch?root="+escapedRoot, nil) response := httptest.NewRecorder() self.server.Watch(response, request) status, body = response.Code, strings.TrimSpace(response.Body.String()) return } func (self *ServerFixture) IgnoreMalformed() (status int, body string) { request, _ := http.NewRequest("POST", "http://localhost:8080/ignore", nil) response := httptest.NewRecorder() self.server.Ignore(response, request) status, body = response.Code, strings.TrimSpace(response.Body.String()) return } func (self *ServerFixture) Ignore(folder string) (status int, body string) { escapedFolder := url.QueryEscape(folder) request, _ := http.NewRequest("POST", "http://localhost:8080/ignore?paths="+escapedFolder, nil) response := httptest.NewRecorder() self.server.Ignore(response, request) status, body = response.Code, strings.TrimSpace(response.Body.String()) return } func (self *ServerFixture) ReinstateMalformed() (status int, body string) { request, _ := http.NewRequest("POST", "http://localhost:8080/reinstate", nil) response := httptest.NewRecorder() self.server.Reinstate(response, request) status, body = response.Code, strings.TrimSpace(response.Body.String()) return } func (self *ServerFixture) Reinstate(folder string) (status int, body string) { escapedFolder := url.QueryEscape(folder) request, _ := http.NewRequest("POST", "http://localhost:8080/reinstate?paths="+escapedFolder, nil) response := httptest.NewRecorder() self.server.Reinstate(response, request) status, body = response.Code, strings.TrimSpace(response.Body.String()) return } func (self *ServerFixture) SetExecutorStatus(status string) { // self.executor.status = status // select { // case self.executor.statusUpdate <- make(chan string): // default: // } } func (self *ServerFixture) RequestExecutorStatus() (code int, status string) { request, _ := http.NewRequest("GET", "http://localhost:8080/status", nil) response := httptest.NewRecorder() self.server.Status(response, request) code, status = response.Code, strings.TrimSpace(response.Body.String()) return } func (self *ServerFixture) ManualExecution() int { request, _ := http.NewRequest("POST", "http://localhost:8080/execute", nil) response := httptest.NewRecorder() self.server.Execute(response, request) nap, _ := time.ParseDuration("100ms") time.Sleep(nap) return response.Code } func (self *ServerFixture) TogglePause() string { request, _ := http.NewRequest("POST", "http://localhost:8080/pause", nil) response := httptest.NewRecorder() self.server.TogglePause(response, request) return response.Body.String() } func newServerFixture() *ServerFixture { self := new(ServerFixture) self.watcher = make(chan messaging.WatcherCommand) // self.watcher.SetRootWatch(initialRoot) statusUpdate := make(chan chan string) self.executor = newFakeExecutor("", statusUpdate) self.server = NewHTTPServer("initial-working-dir", self.watcher, self.executor, statusUpdate) return self } /********* Fake Executor *********/ type FakeExecutor struct { status string executed bool statusFlag bool statusUpdate chan chan string } func (self *FakeExecutor) Status() string { return self.status } func (self *FakeExecutor) ClearStatusFlag() bool { hasNewStatus := self.statusFlag self.statusFlag = false return hasNewStatus } func (self *FakeExecutor) ExecuteTests(watched []*contract.Package) *contract.CompleteOutput { output := new(contract.CompleteOutput) output.Revision = watched[0].Path return output } func newFakeExecutor(status string, statusUpdate chan chan string) *FakeExecutor { self := new(FakeExecutor) self.status = status self.statusUpdate = statusUpdate return self } mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/contract/000077500000000000000000000000001276133741700310555ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/contract/contracts.go000066400000000000000000000015171276133741700334100ustar00rootroot00000000000000package contract import "net/http" type ( Server interface { ReceiveUpdate(root string, update *CompleteOutput) Watch(writer http.ResponseWriter, request *http.Request) Ignore(writer http.ResponseWriter, request *http.Request) Reinstate(writer http.ResponseWriter, request *http.Request) Status(writer http.ResponseWriter, request *http.Request) LongPollStatus(writer http.ResponseWriter, request *http.Request) Results(writer http.ResponseWriter, request *http.Request) Execute(writer http.ResponseWriter, request *http.Request) TogglePause(writer http.ResponseWriter, request *http.Request) } Executor interface { ExecuteTests([]*Package) *CompleteOutput Status() string ClearStatusFlag() bool } Shell interface { GoTest(directory, packageName string, tags, arguments []string) (output string, err error) } ) mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/contract/doc_test.go000066400000000000000000000000211276133741700332010ustar00rootroot00000000000000package contract mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/contract/result.go000066400000000000000000000050471276133741700327300ustar00rootroot00000000000000package contract import ( "path/filepath" "strings" "go/build" "github.com/smartystreets/goconvey/convey/reporting" "github.com/smartystreets/goconvey/web/server/messaging" ) type Package struct { Path string Name string Ignored bool Disabled bool BuildTags []string TestArguments []string Error error Output string Result *PackageResult HasImportCycle bool } func NewPackage(folder *messaging.Folder, hasImportCycle bool) *Package { self := new(Package) self.Path = folder.Path self.Name = resolvePackageName(self.Path) self.Result = NewPackageResult(self.Name) self.Ignored = folder.Ignored self.Disabled = folder.Disabled self.BuildTags = folder.BuildTags self.TestArguments = folder.TestArguments self.HasImportCycle = hasImportCycle return self } func (self *Package) Active() bool { return !self.Disabled && !self.Ignored } func (self *Package) HasUsableResult() bool { return self.Active() && (self.Error == nil || (self.Output != "")) } type CompleteOutput struct { Packages []*PackageResult Revision string Paused bool } var ( // PackageResult.Outcome values: Ignored = "ignored" Disabled = "disabled" Passed = "passed" Failed = "failed" Panicked = "panicked" BuildFailure = "build failure" NoTestFiles = "no test files" NoTestFunctions = "no test functions" NoGoFiles = "no go code" TestRunAbortedUnexpectedly = "test run aborted unexpectedly" ) type PackageResult struct { PackageName string Elapsed float64 Coverage float64 Outcome string BuildOutput string TestResults []TestResult } func NewPackageResult(packageName string) *PackageResult { self := new(PackageResult) self.PackageName = packageName self.TestResults = []TestResult{} self.Coverage = -1 return self } type TestResult struct { TestName string Elapsed float64 Passed bool Skipped bool File string Line int Message string Error string Stories []reporting.ScopeResult RawLines []string `json:",omitempty"` } func NewTestResult(testName string) *TestResult { self := new(TestResult) self.Stories = []reporting.ScopeResult{} self.RawLines = []string{} self.TestName = testName return self } func resolvePackageName(path string) string { pkg, err := build.ImportDir(path, build.FindOnly) if err == nil { return pkg.ImportPath } nameArr := strings.Split(path, endGoPath) return nameArr[len(nameArr)-1] } const ( separator = string(filepath.Separator) endGoPath = separator + "src" + separator ) mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/executor/000077500000000000000000000000001276133741700310765ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/executor/contract.go000066400000000000000000000003461276133741700332450ustar00rootroot00000000000000package executor import "github.com/smartystreets/goconvey/web/server/contract" type Parser interface { Parse([]*contract.Package) } type Tester interface { SetBatchSize(batchSize int) TestAll(folders []*contract.Package) } mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/executor/coordinator.go000066400000000000000000000033771276133741700337620ustar00rootroot00000000000000package executor import ( "errors" "fmt" "log" "strings" "sync" "github.com/smartystreets/goconvey/web/server/contract" ) type concurrentCoordinator struct { batchSize int queue chan *contract.Package folders []*contract.Package shell contract.Shell waiter sync.WaitGroup } func (self *concurrentCoordinator) ExecuteConcurrently() { self.enlistWorkers() self.scheduleTasks() self.awaitCompletion() } func (self *concurrentCoordinator) enlistWorkers() { for i := 0; i < self.batchSize; i++ { self.waiter.Add(1) go self.worker(i) } } func (self *concurrentCoordinator) worker(id int) { for folder := range self.queue { packageName := strings.Replace(folder.Name, "\\", "/", -1) if !folder.Active() { log.Printf("Skipping concurrent execution: %s\n", packageName) continue } if folder.HasImportCycle { message := fmt.Sprintf("can't load package: import cycle not allowed\npackage %s\n\timports %s", packageName, packageName) log.Println(message) folder.Output, folder.Error = message, errors.New(message) } else { log.Printf("Executing concurrent tests: %s\n", packageName) folder.Output, folder.Error = self.shell.GoTest(folder.Path, packageName, folder.BuildTags, folder.TestArguments) } } self.waiter.Done() } func (self *concurrentCoordinator) scheduleTasks() { for _, folder := range self.folders { self.queue <- folder } } func (self *concurrentCoordinator) awaitCompletion() { close(self.queue) self.waiter.Wait() } func newConcurrentCoordinator(folders []*contract.Package, batchSize int, shell contract.Shell) *concurrentCoordinator { self := new(concurrentCoordinator) self.queue = make(chan *contract.Package) self.folders = folders self.batchSize = batchSize self.shell = shell return self } mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/executor/executor.go000066400000000000000000000031401276133741700332610ustar00rootroot00000000000000package executor import ( "log" "time" "github.com/smartystreets/goconvey/web/server/contract" ) const ( Idle = "idle" Executing = "executing" ) type Executor struct { tester Tester parser Parser status string statusChan chan chan string statusFlag bool } func (self *Executor) Status() string { return self.status } func (self *Executor) ClearStatusFlag() bool { hasNewStatus := self.statusFlag self.statusFlag = false return hasNewStatus } func (self *Executor) ExecuteTests(folders []*contract.Package) *contract.CompleteOutput { defer func() { self.setStatus(Idle) }() self.execute(folders) result := self.parse(folders) return result } func (self *Executor) execute(folders []*contract.Package) { self.setStatus(Executing) self.tester.TestAll(folders) } func (self *Executor) parse(folders []*contract.Package) *contract.CompleteOutput { result := &contract.CompleteOutput{Revision: now().String()} self.parser.Parse(folders) for _, folder := range folders { result.Packages = append(result.Packages, folder.Result) } return result } func (self *Executor) setStatus(status string) { self.status = status self.statusFlag = true Loop: for { select { case c := <-self.statusChan: self.statusFlag = false c <- status default: break Loop } } log.Printf("Executor status: '%s'\n", self.status) } func NewExecutor(tester Tester, parser Parser, ch chan chan string) *Executor { return &Executor{ tester: tester, parser: parser, status: Idle, statusChan: ch, statusFlag: false, } } var now = func() time.Time { return time.Now() } executor.goconvey000066400000000000000000000000241276133741700344240ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/executor#ignore -timeout=1s executor_test.go000066400000000000000000000075401276133741700342510ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/executorpackage executor import ( "strings" "testing" "time" . "github.com/smartystreets/goconvey/convey" "github.com/smartystreets/goconvey/web/server/contract" ) func TestExecutor(t *testing.T) { t.Skip("BROKEN!") Convey("Subject: Execution of test packages and aggregation of parsed results", t, func() { fixture := newExecutorFixture() Convey("When tests packages are executed", func() { fixture.ExecuteTests() Convey("The result should include parsed results for each test package.", fixture.ResultShouldBePopulated) }) Convey("When the executor is idle", func() { Convey("The status of the executor should be 'idle'", func() { So(fixture.executor.Status(), ShouldEqual, Idle) }) }) Convey("When the status is updated", func() { fixture.executor.setStatus(Executing) Convey("The status flag should be set to true", func() { So(fixture.executor.statusFlag, ShouldBeTrue) }) }) Convey("During test execution", func() { status := fixture.CaptureStatusDuringExecutionPhase() Convey("The status of the executor should be 'executing'", func() { So(status, ShouldEqual, Executing) }) }) }) } type ExecutorFixture struct { executor *Executor tester *FakeTester parser *FakeParser folders []*contract.Package result *contract.CompleteOutput expected *contract.CompleteOutput stamp time.Time } func (self *ExecutorFixture) ExecuteTests() { self.result = self.executor.ExecuteTests(self.folders) } func (self *ExecutorFixture) CaptureStatusDuringExecutionPhase() string { nap, _ := time.ParseDuration("25ms") self.tester.addDelay(nap) return self.delayedExecution(nap) } func (self *ExecutorFixture) delayedExecution(nap time.Duration) string { go self.ExecuteTests() time.Sleep(nap) return self.executor.Status() } func (self *ExecutorFixture) ResultShouldBePopulated() { So(self.result, ShouldResemble, self.expected) } var ( prefix = "/Users/blah/gopath/src/" packageA = "github.com/smartystreets/goconvey/a" packageB = "github.com/smartystreets/goconvey/b" resultA = &contract.PackageResult{PackageName: packageA} resultB = &contract.PackageResult{PackageName: packageB} ) func newExecutorFixture() *ExecutorFixture { self := new(ExecutorFixture) self.tester = newFakeTester() self.parser = newFakeParser() self.executor = NewExecutor(self.tester, self.parser, make(chan chan string)) self.folders = []*contract.Package{ &contract.Package{Path: prefix + packageA, Name: packageA}, &contract.Package{Path: prefix + packageB, Name: packageB}, } self.stamp = time.Now() now = func() time.Time { return self.stamp } self.expected = &contract.CompleteOutput{ Packages: []*contract.PackageResult{ resultA, resultB, }, Revision: self.stamp.String(), } return self } /******** FakeTester ********/ type FakeTester struct { nap time.Duration } func (self *FakeTester) SetBatchSize(batchSize int) { panic("NOT SUPPORTED") } func (self *FakeTester) TestAll(folders []*contract.Package) { for _, p := range folders { p.Output = p.Path } time.Sleep(self.nap) } func (self *FakeTester) addDelay(nap time.Duration) { self.nap = nap } func newFakeTester() *FakeTester { self := new(FakeTester) zero, _ := time.ParseDuration("0") self.nap = zero return self } /******** FakeParser ********/ type FakeParser struct { nap time.Duration } func (self *FakeParser) Parse(packages []*contract.Package) { time.Sleep(self.nap) for _, package_ := range packages { if package_.Name == packageA && strings.HasSuffix(package_.Output, packageA) { package_.Result = resultA } if package_.Name == packageB && strings.HasSuffix(package_.Output, packageB) { package_.Result = resultB } } } func (self *FakeParser) addDelay(nap time.Duration) { self.nap = nap } func newFakeParser() *FakeParser { self := new(FakeParser) zero, _ := time.ParseDuration("0") self.nap = zero return self } mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/executor/tester.go000066400000000000000000000027511276133741700327400ustar00rootroot00000000000000package executor import ( "errors" "fmt" "log" "strings" "github.com/smartystreets/goconvey/web/server/contract" ) type ConcurrentTester struct { shell contract.Shell batchSize int } func (self *ConcurrentTester) SetBatchSize(batchSize int) { self.batchSize = batchSize log.Printf("Now configured to test %d packages concurrently.\n", self.batchSize) } func (self *ConcurrentTester) TestAll(folders []*contract.Package) { if self.batchSize == 1 { self.executeSynchronously(folders) } else { newConcurrentCoordinator(folders, self.batchSize, self.shell).ExecuteConcurrently() } return } func (self *ConcurrentTester) executeSynchronously(folders []*contract.Package) { for _, folder := range folders { packageName := strings.Replace(folder.Name, "\\", "/", -1) if !folder.Active() { log.Printf("Skipping execution: %s\n", packageName) continue } if folder.HasImportCycle { message := fmt.Sprintf("can't load package: import cycle not allowed\npackage %s\n\timports %s", packageName, packageName) log.Println(message) folder.Output, folder.Error = message, errors.New(message) } else { log.Printf("Executing tests: %s\n", packageName) folder.Output, folder.Error = self.shell.GoTest(folder.Path, packageName, folder.BuildTags, folder.TestArguments) } } } func NewConcurrentTester(shell contract.Shell) *ConcurrentTester { self := new(ConcurrentTester) self.shell = shell self.batchSize = defaultBatchSize return self } const defaultBatchSize = 10 mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/executor/tester_test.go000066400000000000000000000145111276133741700337740ustar00rootroot00000000000000package executor import ( "errors" "fmt" "io/ioutil" "log" "testing" "time" . "github.com/smartystreets/goconvey/convey" "github.com/smartystreets/goconvey/web/server/contract" ) func init() { log.SetOutput(ioutil.Discard) } func TestConcurrentTester(t *testing.T) { t.Skip("BROKEN!") Convey("Subject: Controlled execution of test packages", t, func() { fixture := NewTesterFixture() Convey("Whenever tests for each package are executed", func() { fixture.InBatchesOf(1).RunTests() Convey("The tester should execute the tests in each active package with the correct arguments", fixture.ShouldHaveRecordOfExecutionCommands) Convey("There should be a test output result for each active package", fixture.ShouldHaveOneOutputPerInput) Convey("The output should be as expected", fixture.OutputShouldBeAsExpected) }) Convey("When the tests for each package are executed synchronously", func() { fixture.InBatchesOf(1).RunTests() Convey("Each active package should be run synchronously and in the given order", fixture.TestsShouldHaveRunContiguously) }) Convey("When the tests for each package are executed synchronously with failures", func() { fixture.InBatchesOf(1).SetupFailedTestSuites().RunTests() Convey("The failed test packages should not result in any panics", func() { So(fixture.recovered, ShouldBeNil) }) }) Convey("When packages are tested concurrently", func() { fixture.InBatchesOf(concurrentBatchSize).RunTests() Convey("Active packages should be arranged and tested in batches of the appropriate size", fixture.TestsShouldHaveRunInBatchesOfTwo) }) Convey("When packages are tested concurrently with failures", func() { fixture.InBatchesOf(concurrentBatchSize).SetupFailedTestSuites().RunTests() Convey("The failed test packages should not result in any panics", func() { So(fixture.recovered, ShouldBeNil) }) }) }) } const concurrentBatchSize = 2 type TesterFixture struct { tester *ConcurrentTester shell *TimedShell results []string compilations []*ShellCommand executions []*ShellCommand packages []*contract.Package recovered error } func NewTesterFixture() *TesterFixture { self := new(TesterFixture) self.shell = NewTimedShell() self.tester = NewConcurrentTester(self.shell) self.packages = []*contract.Package{ {Path: "a"}, {Path: "b"}, {Path: "c"}, {Path: "d"}, {Path: "e", Ignored: true}, {Path: "f"}, {Path: "g", HasImportCycle: true}, } return self } func (self *TesterFixture) InBatchesOf(batchSize int) *TesterFixture { self.tester.SetBatchSize(batchSize) return self } func (self *TesterFixture) SetupAbnormalError(message string) *TesterFixture { self.shell.setTripWire(message) return self } func (self *TesterFixture) SetupFailedTestSuites() *TesterFixture { self.shell.setExitWithError() return self } func (self *TesterFixture) RunTests() { defer func() { if r := recover(); r != nil { self.recovered = r.(error) } }() self.tester.TestAll(self.packages) for _, p := range self.packages { self.results = append(self.results, p.Output) } self.executions = self.shell.Executions() } func (self *TesterFixture) ShouldHaveRecordOfExecutionCommands() { executed := []string{"a", "b", "c", "d", "f"} ignored := "e" importCycle := "g" actual := []string{} for _, pkg := range self.executions { actual = append(actual, pkg.Command) } So(actual, ShouldResemble, executed) So(actual, ShouldNotContain, ignored) So(actual, ShouldNotContain, importCycle) } func (self *TesterFixture) ShouldHaveOneOutputPerInput() { So(len(self.results), ShouldEqual, len(self.packages)) } func (self *TesterFixture) OutputShouldBeAsExpected() { for _, p := range self.packages { if p.HasImportCycle { So(p.Output, ShouldContainSubstring, "can't load package: import cycle not allowed") So(p.Error.Error(), ShouldContainSubstring, "can't load package: import cycle not allowed") } else { if p.Active() { So(p.Output, ShouldEndWith, p.Path) } else { So(p.Output, ShouldBeBlank) } So(p.Error, ShouldBeNil) } } } func (self *TesterFixture) TestsShouldHaveRunContiguously() { self.OutputShouldBeAsExpected() So(self.shell.MaxConcurrentCommands(), ShouldEqual, 1) for i := 0; i < len(self.executions)-1; i++ { current := self.executions[i] next := self.executions[i+1] So(current.Started, ShouldHappenBefore, next.Started) So(current.Ended, ShouldHappenOnOrBefore, next.Started) } } func (self *TesterFixture) TestsShouldHaveRunInBatchesOfTwo() { self.OutputShouldBeAsExpected() So(self.shell.MaxConcurrentCommands(), ShouldEqual, concurrentBatchSize) } /**** Fakes ****/ type ShellCommand struct { Command string Started time.Time Ended time.Time } type TimedShell struct { executions []*ShellCommand panicMessage string err error } func (self *TimedShell) Executions() []*ShellCommand { return self.executions } func (self *TimedShell) MaxConcurrentCommands() int { var concurrent int for x, current := range self.executions { concurrentWith_x := 1 for y, comparison := range self.executions { if y == x { continue } else if concurrentWith(current, comparison) { concurrentWith_x++ } } if concurrentWith_x > concurrent { concurrent = concurrentWith_x } } return concurrent } func concurrentWith(current, comparison *ShellCommand) bool { return ((comparison.Started == current.Started || comparison.Started.After(current.Started)) && (comparison.Started.Before(current.Ended))) } func (self *TimedShell) setTripWire(message string) { self.panicMessage = message } func (self *TimedShell) setExitWithError() { self.err = errors.New("Simulate test failure") } func (self *TimedShell) GoTest(directory, packageName string, arguments, tags []string) (output string, err error) { if self.panicMessage != "" { return "", errors.New(self.panicMessage) } output = directory err = self.err self.executions = append(self.executions, self.composeCommand(directory)) return } func (self *TimedShell) composeCommand(commandText string) *ShellCommand { start := time.Now() time.Sleep(nap) end := time.Now() return &ShellCommand{commandText, start, end} } func NewTimedShell() *TimedShell { self := new(TimedShell) self.executions = []*ShellCommand{} return self } var nap, _ = time.ParseDuration("10ms") var _ = fmt.Sprintf("fmt") mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/messaging/000077500000000000000000000000001276133741700312155ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/messaging/doc_test.go000066400000000000000000000000221276133741700333420ustar00rootroot00000000000000package messaging mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/messaging/messages.go000066400000000000000000000021271276133741700333550ustar00rootroot00000000000000package messaging /////////////////////////////////////////////////////////////////////////////// type WatcherCommand struct { Instruction WatcherInstruction Details string } type WatcherInstruction int func (this WatcherInstruction) String() string { switch this { case WatcherPause: return "Pause" case WatcherResume: return "Resume" case WatcherIgnore: return "Ignore" case WatcherReinstate: return "Reinstate" case WatcherAdjustRoot: return "AdjustRoot" case WatcherExecute: return "Execute" case WatcherStop: return "Stop" default: return "UNKNOWN INSTRUCTION" } } const ( WatcherPause WatcherInstruction = iota WatcherResume WatcherIgnore WatcherReinstate WatcherAdjustRoot WatcherExecute WatcherStop ) /////////////////////////////////////////////////////////////////////////////// type Folders map[string]*Folder type Folder struct { Path string // key Root string Ignored bool Disabled bool BuildTags []string TestArguments []string } /////////////////////////////////////////////////////////////////////////////// mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/parser/000077500000000000000000000000001276133741700305345ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/parser/packageParser.go000066400000000000000000000107501276133741700336360ustar00rootroot00000000000000package parser import ( "fmt" "regexp" "sort" "strconv" "strings" "github.com/smartystreets/goconvey/web/server/contract" ) var ( testNamePattern = regexp.MustCompile("^=== RUN:? +(.+)$") ) func ParsePackageResults(result *contract.PackageResult, rawOutput string) { newOutputParser(result, rawOutput).parse() } type outputParser struct { raw string lines []string result *contract.PackageResult tests []*contract.TestResult // place holders for loops line string test *contract.TestResult testMap map[string]*contract.TestResult } func newOutputParser(result *contract.PackageResult, rawOutput string) *outputParser { self := new(outputParser) self.raw = strings.TrimSpace(rawOutput) self.lines = strings.Split(self.raw, "\n") self.result = result self.tests = []*contract.TestResult{} self.testMap = make(map[string]*contract.TestResult) return self } func (self *outputParser) parse() { self.separateTestFunctionsAndMetadata() self.parseEachTestFunction() } func (self *outputParser) separateTestFunctionsAndMetadata() { for _, self.line = range self.lines { if self.processNonTestOutput() { break } self.processTestOutput() } } func (self *outputParser) processNonTestOutput() bool { if noGoFiles(self.line) { self.recordFinalOutcome(contract.NoGoFiles) } else if buildFailed(self.line) { self.recordFinalOutcome(contract.BuildFailure) } else if noTestFiles(self.line) { self.recordFinalOutcome(contract.NoTestFiles) } else if noTestFunctions(self.line) { self.recordFinalOutcome(contract.NoTestFunctions) } else { return false } return true } func (self *outputParser) recordFinalOutcome(outcome string) { self.result.Outcome = outcome self.result.BuildOutput = strings.Join(self.lines, "\n") } func (self *outputParser) processTestOutput() { if isNewTest(self.line) { self.registerTestFunction() } else if isTestResult(self.line) { self.recordTestMetadata() } else if isPackageReport(self.line) { self.recordPackageMetadata() } else { self.saveLineForParsingLater() } } func (self *outputParser) registerTestFunction() { testName := testNamePattern.FindStringSubmatch(self.line)[1] self.test = contract.NewTestResult(testName) self.tests = append(self.tests, self.test) self.testMap[self.test.TestName] = self.test } func (self *outputParser) recordTestMetadata() { testName := strings.Split(self.line, " ")[2] if test, ok := self.testMap[testName]; ok { self.test = test self.test.Passed = !strings.HasPrefix(self.line, "--- FAIL: ") self.test.Skipped = strings.HasPrefix(self.line, "--- SKIP: ") self.test.Elapsed = parseTestFunctionDuration(self.line) } } func (self *outputParser) recordPackageMetadata() { if packageFailed(self.line) { self.recordTestingOutcome(contract.Failed) } else if packagePassed(self.line) { self.recordTestingOutcome(contract.Passed) } else if isCoverageSummary(self.line) { self.recordCoverageSummary(self.line) } } func (self *outputParser) recordTestingOutcome(outcome string) { self.result.Outcome = outcome fields := strings.Split(self.line, "\t") self.result.PackageName = strings.TrimSpace(fields[1]) self.result.Elapsed = parseDurationInSeconds(fields[2], 3) } func (self *outputParser) recordCoverageSummary(summary string) { start := len("coverage: ") end := strings.Index(summary, "%") value := summary[start:end] parsed, err := strconv.ParseFloat(value, 64) if err != nil { self.result.Coverage = -1 } else { self.result.Coverage = parsed } } func (self *outputParser) saveLineForParsingLater() { self.line = strings.TrimLeft(self.line, "\t") if self.test == nil { fmt.Println("Potential error parsing output of", self.result.PackageName, "; couldn't handle this stray line:", self.line) return } self.test.RawLines = append(self.test.RawLines, self.line) } // TestResults is a collection of TestResults that implements sort.Interface. type TestResults []contract.TestResult func (r TestResults) Len() int { return len(r) } // Less compares TestResults on TestName func (r TestResults) Less(i, j int) bool { return r[i].TestName < r[j].TestName } func (r TestResults) Swap(i, j int) { r[i], r[j] = r[j], r[i] } func (self *outputParser) parseEachTestFunction() { for _, self.test = range self.tests { self.test = parseTestOutput(self.test) if self.test.Error != "" { self.result.Outcome = contract.Panicked } self.test.RawLines = []string{} self.result.TestResults = append(self.result.TestResults, *self.test) } sort.Sort(TestResults(self.result.TestResults)) } package_parser_test.go000066400000000000000000000723461276133741700350260ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/parserpackage parser import ( "encoding/json" "fmt" "io/ioutil" "log" "strings" "testing" "github.com/smartystreets/goconvey/convey/reporting" "github.com/smartystreets/goconvey/web/server/contract" ) func init() { log.SetOutput(ioutil.Discard) } func TestParsePackage_NoGoFiles_ReturnsPackageResult(t *testing.T) { actual := &contract.PackageResult{PackageName: expected_NoGoFiles.PackageName} ParsePackageResults(actual, input_NoGoFiles) assertEqual(t, expected_NoGoFiles, *actual) } func TestParsePackage_NoTestFiles_ReturnsPackageResult(t *testing.T) { actual := &contract.PackageResult{PackageName: expected_NoTestFiles.PackageName} ParsePackageResults(actual, input_NoTestFiles) assertEqual(t, expected_NoTestFiles, *actual) } func TestParsePacakge_NoTestFunctions_ReturnsPackageResult(t *testing.T) { actual := &contract.PackageResult{PackageName: expected_NoTestFunctions.PackageName} ParsePackageResults(actual, input_NoTestFunctions) assertEqual(t, expected_NoTestFunctions, *actual) } func TestParsePackage_BuildFailed_ReturnsPackageResult(t *testing.T) { actual := &contract.PackageResult{PackageName: expected_BuildFailed_InvalidPackageDeclaration.PackageName} ParsePackageResults(actual, input_BuildFailed_InvalidPackageDeclaration) assertEqual(t, expected_BuildFailed_InvalidPackageDeclaration, *actual) actual = &contract.PackageResult{PackageName: expected_BuildFailed_OtherErrors.PackageName} ParsePackageResults(actual, input_BuildFailed_OtherErrors) assertEqual(t, expected_BuildFailed_OtherErrors, *actual) actual = &contract.PackageResult{PackageName: expected_BuildFailed_ImportCycle.PackageName} ParsePackageResults(actual, input_BuildFailed_ImportCycle) assertEqual(t, expected_BuildFailed_ImportCycle, *actual) actual = &contract.PackageResult{PackageName: expected_BuildFailed_CantFindPackage.PackageName} ParsePackageResults(actual, input_BuildFailed_CantFindPackage) assertEqual(t, expected_BuildFailed_CantFindPackage, *actual) actual = &contract.PackageResult{PackageName: expected_BuildFailed_ConflictingImport.PackageName} ParsePackageResults(actual, input_BuildFailed_ConfictingImport) assertEqual(t, expected_BuildFailed_ConflictingImport, *actual) } func TestParsePackage_OldSchoolWithFailureOutput_ReturnsCompletePackageResult(t *testing.T) { actual := &contract.PackageResult{PackageName: expectedOldSchool_Fails.PackageName} ParsePackageResults(actual, inputOldSchool_Fails) assertEqual(t, expectedOldSchool_Fails, *actual) } func TestParsePackage_OldSchoolWithSuccessOutput_ReturnsCompletePackageResult(t *testing.T) { actual := &contract.PackageResult{PackageName: expectedOldSchool_Passes.PackageName} ParsePackageResults(actual, inputOldSchool_Passes) assertEqual(t, expectedOldSchool_Passes, *actual) } func TestParsePackage_OldSchoolWithPanicOutput_ReturnsCompletePackageResult(t *testing.T) { actual := &contract.PackageResult{PackageName: expectedOldSchool_Panics.PackageName} ParsePackageResults(actual, inputOldSchool_Panics) assertEqual(t, expectedOldSchool_Panics, *actual) } func TestParsePackage_GoConveyOutput_ReturnsCompletePackageResult(t *testing.T) { actual := &contract.PackageResult{PackageName: expectedGoConvey.PackageName} ParsePackageResults(actual, inputGoConvey) assertEqual(t, expectedGoConvey, *actual) } func TestParsePackage_ActualPackageNameDifferentThanDirectoryName_ReturnsActualPackageName(t *testing.T) { actual := &contract.PackageResult{PackageName: strings.Replace(expectedGoConvey.PackageName, "examples", "stuff", -1)} ParsePackageResults(actual, inputGoConvey) assertEqual(t, expectedGoConvey, *actual) } func TestParsePackage_GoConveyOutputMalformed_CausesPanic(t *testing.T) { defer func() { if r := recover(); r != nil { message := fmt.Sprintf("%v", r) if !strings.Contains(message, "bug report") { t.Errorf("Should have panicked with a request to file a bug report but we received this error instead: %s", message) } } else { t.Errorf("Should have panicked with a request to file a bug report but we received no error.") } }() actual := &contract.PackageResult{PackageName: expectedGoConvey.PackageName} ParsePackageResults(actual, inputGoConvey_Malformed) } func TestParsePackage_GoConveyWithRandomOutput_ReturnsPackageResult(t *testing.T) { actual := &contract.PackageResult{PackageName: expectedGoConvey_WithRandomOutput.PackageName} ParsePackageResults(actual, inputGoConvey_WithRandomOutput) assertEqual(t, expectedGoConvey_WithRandomOutput, *actual) } func TestParsePackage_OldSchoolWithSuccessAndBogusCoverage_ReturnsCompletePackageResult(t *testing.T) { actual := &contract.PackageResult{PackageName: expectedOldSchool_PassesButCoverageIsBogus.PackageName} ParsePackageResults(actual, inputOldSchool_PassesButCoverageIsBogus) assertEqual(t, expectedOldSchool_PassesButCoverageIsBogus, *actual) } func TestParsePackage_NestedTests_ReturnsPackageResult(t *testing.T) { actual := &contract.PackageResult{PackageName: expectedNestedTests.PackageName} ParsePackageResults(actual, inputNestedTests) assertEqual(t, expectedNestedTests, *actual) } func TestParsePackage_WithExampleFunctions_ReturnsPackageResult(t *testing.T) { actual := &contract.PackageResult{PackageName: expectedExampleFunctions.PackageName} ParsePackageResults(actual, inputExampleFunctions) assertEqual(t, expectedExampleFunctions, *actual) } func TestParsePackage_Golang15Output_ShouldNotPanic(t *testing.T) { actual := &contract.PackageResult{PackageName: expectedGolang15.PackageName} ParsePackageResults(actual, inputGolang15) assertEqual(t, expectedGolang15, *actual) } func assertEqual(t *testing.T, expected, actual interface{}) { a, _ := json.Marshal(expected) b, _ := json.Marshal(actual) if string(a) != string(b) { t.Errorf(failureTemplate, string(a), string(b)) } } const failureTemplate = "Comparison failed:\n Expected: %v\n Actual: %v\n" const input_NoGoFiles = `can't load package: package github.com/smartystreets/goconvey: no buildable Go source files in /Users/matt/Work/Dev/goconvey/src/github.com/smartystreets/goconvey` var expected_NoGoFiles = contract.PackageResult{ PackageName: "github.com/smartystreets/goconvey", Outcome: contract.NoGoFiles, BuildOutput: input_NoGoFiles, } const input_NoTestFiles = `? pkg.smartystreets.net/liveaddress-zipapi [no test files]` var expected_NoTestFiles = contract.PackageResult{ PackageName: "pkg.smartystreets.net/liveaddress-zipapi", Outcome: contract.NoTestFiles, BuildOutput: input_NoTestFiles, } const input_NoTestFunctions = `testing: warning: no tests to run PASS ok github.com/smartystreets/goconvey/scripts 0.011s` var expected_NoTestFunctions = contract.PackageResult{ PackageName: "github.com/smartystreets/goconvey/scripts", Outcome: contract.NoTestFunctions, BuildOutput: input_NoTestFunctions, } const input_BuildFailed_InvalidPackageDeclaration = ` can't load package: package github.com/smartystreets/goconvey/examples: bowling_game_test.go:9:1: expected 'package', found 'IDENT' asdf bowling_game_test.go:10:1: invalid package name _ ` var expected_BuildFailed_InvalidPackageDeclaration = contract.PackageResult{ PackageName: "github.com/smartystreets/goconvey/examples", Outcome: contract.BuildFailure, BuildOutput: strings.TrimSpace(input_BuildFailed_InvalidPackageDeclaration), } const input_BuildFailed_CantFindPackage = ` bowling_game.go:3:8: cannot find package "format" in any of: /usr/local/go/src/pkg/format (from $GOROOT) /Users/mike/work/dev/goconvey/src/format (from $GOPATH) FAIL github.com/smartystreets/goconvey/examples [setup failed] ` var expected_BuildFailed_CantFindPackage = contract.PackageResult{ PackageName: "github.com/smartystreets/goconvey/examples", Outcome: contract.BuildFailure, BuildOutput: strings.TrimSpace(input_BuildFailed_CantFindPackage), } const input_BuildFailed_ConfictingImport = ` mutustus.go:4:2: found packages e (e.go) and err (prepend.go) in /Users/mike/src/utensils.git/e ` var expected_BuildFailed_ConflictingImport = contract.PackageResult{ PackageName: "github.com/smartystreets/goconvey/examples", Outcome: contract.BuildFailure, BuildOutput: strings.TrimSpace(input_BuildFailed_ConfictingImport), } const input_BuildFailed_OtherErrors = ` # github.com/smartystreets/goconvey/examples ./bowling_game_test.go:22: undefined: game ./bowling_game_test.go:22: cannot assign to game ./bowling_game_test.go:25: undefined: game ./bowling_game_test.go:28: undefined: game ./bowling_game_test.go:33: undefined: game ./bowling_game_test.go:36: undefined: game ./bowling_game_test.go:41: undefined: game ./bowling_game_test.go:42: undefined: game ./bowling_game_test.go:43: undefined: game ./bowling_game_test.go:46: undefined: game ./bowling_game_test.go:46: too many errors FAIL github.com/smartystreets/goconvey/examples [build failed] ` var expected_BuildFailed_OtherErrors = contract.PackageResult{ PackageName: "github.com/smartystreets/goconvey/examples", Outcome: contract.BuildFailure, BuildOutput: strings.TrimSpace(input_BuildFailed_OtherErrors), } const input_BuildFailed_ImportCycle = ` # github.com/smartystreets/goconvey/t ./t_test.go:23: import "github.com/smartystreets/goconvey/t" while compiling that package (import cycle) FAIL github.com/smartystreets/goconvey/t [build failed] ` var expected_BuildFailed_ImportCycle = contract.PackageResult{ PackageName: "github.com/smartystreets/goconvey/t", Outcome: contract.BuildFailure, BuildOutput: strings.TrimSpace(input_BuildFailed_ImportCycle), } const inputOldSchool_Passes = ` === RUN TestOldSchool_Passes --- PASS: TestOldSchool_Passes (0.02 seconds) === RUN TestSkippingTests --- SKIP: TestSkippingTests (0.00 seconds) old_school_test.go:8: blah === RUN TestOldSchool_PassesWithMessage --- PASS: TestOldSchool_PassesWithMessage (0.05 seconds) old_school_test.go:10: I am a passing test. With a newline. PASS coverage: 100.0%% of statements in github.com/smartystreets/goconvey/convey, github.com/smartystreets/goconvey/convey/gotest, github.com/smartystreets/goconvey/convey/reporting ok github.com/smartystreets/goconvey/webserver/examples 0.018s ` var expectedOldSchool_Passes = contract.PackageResult{ PackageName: "github.com/smartystreets/goconvey/webserver/examples", Elapsed: 0.018, Coverage: 100, Outcome: contract.Passed, TestResults: []contract.TestResult{ contract.TestResult{ TestName: "TestOldSchool_Passes", Elapsed: 0.02, Passed: true, File: "", Line: 0, Message: "", Stories: []reporting.ScopeResult{}, }, contract.TestResult{ TestName: "TestOldSchool_PassesWithMessage", Elapsed: 0.05, Passed: true, File: "old_school_test.go", Line: 10, Message: "old_school_test.go:10: I am a passing test.\nWith a newline.", Stories: []reporting.ScopeResult{}, }, contract.TestResult{ TestName: "TestSkippingTests", Elapsed: 0, Passed: true, Skipped: true, File: "old_school_test.go", Line: 8, Message: "old_school_test.go:8: blah", Stories: []reporting.ScopeResult{}, }, }, } const inputOldSchool_Fails = ` === RUN TestOldSchool_Passes --- PASS: TestOldSchool_Passes (0.01 seconds) === RUN TestOldSchool_PassesWithMessage --- PASS: TestOldSchool_PassesWithMessage (0.03 seconds) old_school_test.go:10: I am a passing test. With a newline. === RUN TestOldSchool_Failure --- FAIL: TestOldSchool_Failure (0.06 seconds) === RUN TestOldSchool_FailureWithReason --- FAIL: TestOldSchool_FailureWithReason (0.11 seconds) old_school_test.go:18: I am a failing test. FAIL exit status 1 FAIL github.com/smartystreets/goconvey/webserver/examples 0.017s ` var expectedOldSchool_Fails = contract.PackageResult{ PackageName: "github.com/smartystreets/goconvey/webserver/examples", Outcome: contract.Failed, Elapsed: 0.017, TestResults: []contract.TestResult{ contract.TestResult{ TestName: "TestOldSchool_Failure", Elapsed: 0.06, Passed: false, File: "", Line: 0, Message: "", Stories: []reporting.ScopeResult{}, }, contract.TestResult{ TestName: "TestOldSchool_FailureWithReason", Elapsed: 0.11, Passed: false, File: "old_school_test.go", Line: 18, Message: "old_school_test.go:18: I am a failing test.", Stories: []reporting.ScopeResult{}, }, contract.TestResult{ TestName: "TestOldSchool_Passes", Elapsed: 0.01, Passed: true, File: "", Line: 0, Message: "", Stories: []reporting.ScopeResult{}, }, contract.TestResult{ TestName: "TestOldSchool_PassesWithMessage", Elapsed: 0.03, Passed: true, File: "old_school_test.go", Line: 10, Message: "old_school_test.go:10: I am a passing test.\nWith a newline.", Stories: []reporting.ScopeResult{}, }, }, } const inputOldSchool_Panics = ` === RUN TestOldSchool_Panics --- FAIL: TestOldSchool_Panics (0.02 seconds) panic: runtime error: index out of range [recovered] panic: runtime error: index out of range goroutine 3 [running]: testing.func·004() /usr/local/go/src/pkg/testing/testing.go:348 +0xcd github.com/smartystreets/goconvey/webserver/examples.TestOldSchool_Panics(0x210292000) /Users/mike/work/dev/goconvey/src/github.com/smartystreets/goconvey/webserver/examples/something_test.go:15 +0xec testing.tRunner(0x210292000, 0x1b09f0) /usr/local/go/src/pkg/testing/testing.go:353 +0x8a created by testing.RunTests /usr/local/go/src/pkg/testing/testing.go:433 +0x86b goroutine 1 [chan receive]: testing.RunTests(0x138f38, 0x1b09f0, 0x1, 0x1, 0x1, ...) /usr/local/go/src/pkg/testing/testing.go:434 +0x88e testing.Main(0x138f38, 0x1b09f0, 0x1, 0x1, 0x1b7f60, ...) /usr/local/go/src/pkg/testing/testing.go:365 +0x8a main.main() github.com/smartystreets/goconvey/webserver/examples/_test/_testmain.go:43 +0x9a exit status 2 FAIL github.com/smartystreets/goconvey/webserver/examples 0.014s ` var expectedOldSchool_Panics = contract.PackageResult{ PackageName: "github.com/smartystreets/goconvey/webserver/examples", Elapsed: 0.014, Outcome: contract.Panicked, TestResults: []contract.TestResult{ contract.TestResult{ TestName: "TestOldSchool_Panics", Elapsed: 0.02, Passed: false, File: "/Users/mike/work/dev/goconvey/src/github.com/smartystreets/goconvey/webserver/examples/something_test.go", Line: 15, Message: "", Error: strings.Replace(`panic: runtime error: index out of range [recovered] panic: runtime error: index out of range goroutine 3 [running]: testing.func·004() /usr/local/go/src/pkg/testing/testing.go:348 +0xcd github.com/smartystreets/goconvey/webserver/examples.TestOldSchool_Panics(0x210292000) /Users/mike/work/dev/goconvey/src/github.com/smartystreets/goconvey/webserver/examples/something_test.go:15 +0xec testing.tRunner(0x210292000, 0x1b09f0) /usr/local/go/src/pkg/testing/testing.go:353 +0x8a created by testing.RunTests /usr/local/go/src/pkg/testing/testing.go:433 +0x86b goroutine 1 [chan receive]: testing.RunTests(0x138f38, 0x1b09f0, 0x1, 0x1, 0x1, ...) /usr/local/go/src/pkg/testing/testing.go:434 +0x88e testing.Main(0x138f38, 0x1b09f0, 0x1, 0x1, 0x1b7f60, ...) /usr/local/go/src/pkg/testing/testing.go:365 +0x8a main.main() github.com/smartystreets/goconvey/webserver/examples/_test/_testmain.go:43 +0x9a`, "\u0009", "\t", -1), Stories: []reporting.ScopeResult{}, }, }, } const inputGoConvey_Malformed = ` === RUN TestPassingStory >->->OPEN-JSON->->-> { "Title": "A passing story", "File": "/Users/mike/work/dev/goconvey/src/github.com/smartystreets/goconvey/webserver/examples/old_school_test.go", "Line": 11, "Depth": 0, "Assertions": [ { "File": "/Users/mike/work/dev/goconvey/src/github.com/smartystreets/goconvey/webserver/examples/old_school_test.go", "Line": 10, "Failure": "", ;aiwheopinen39 n3902n92m "Error": null, "Skipped": false, "StackTrace": "goroutine 3 [running]:\ngithub.com/smartystreets/goconvey/webserver/examples.func·001()\n\u0009/Users/mike/work/dev/goconvey/src/github.com/smartystreets/goconvey/webserver/examples/old_school_test.go:10 +0xe3\ngithub.com/smartystreets/goconvey/webserver/examples.TestPassingStory(0x210314000)\n\u0009/Users/mike/work/dev/goconvey/src/github.com/smartystreets/goconvey/webserver/examples/old_school_test.go:11 +0xec\ntesting.tRunner(0x210314000, 0x21ab10)\n\u0009/usr/local/go/src/pkg/testing/testing.go:353 +0x8a\ncreated by testing.RunTests\n\u0009/usr/local/go/src/pkg/testing/testing.go:433 +0x86b\n" } ] }, <-<-<-CLOSE-JSON<-<-< --- PASS: TestPassingStory (0.01 seconds) PASS ok github.com/smartystreets/goconvey/webserver/examples 0.019s ` const inputGoConvey = ` === RUN TestPassingStory >->->OPEN-JSON->->-> { "Title": "A passing story", "File": "/Users/mike/work/dev/goconvey/src/github.com/smartystreets/goconvey/webserver/examples/old_school_test.go", "Line": 11, "Depth": 0, "Assertions": [ { "File": "/Users/mike/work/dev/goconvey/src/github.com/smartystreets/goconvey/webserver/examples/old_school_test.go", "Line": 10, "Failure": "", "Error": null, "Skipped": false, "StackTrace": "goroutine 3 [running]:\ngithub.com/smartystreets/goconvey/webserver/examples.func·001()\n\u0009/Users/mike/work/dev/goconvey/src/github.com/smartystreets/goconvey/webserver/examples/old_school_test.go:10 +0xe3\ngithub.com/smartystreets/goconvey/webserver/examples.TestPassingStory(0x210314000)\n\u0009/Users/mike/work/dev/goconvey/src/github.com/smartystreets/goconvey/webserver/examples/old_school_test.go:11 +0xec\ntesting.tRunner(0x210314000, 0x21ab10)\n\u0009/usr/local/go/src/pkg/testing/testing.go:353 +0x8a\ncreated by testing.RunTests\n\u0009/usr/local/go/src/pkg/testing/testing.go:433 +0x86b\n" } ] }, <-<-<-CLOSE-JSON<-<-< --- PASS: TestPassingStory (0.01 seconds) PASS coverage: 75.5%% of statements ok github.com/smartystreets/goconvey/webserver/examples 0.019s ` var expectedGoConvey = contract.PackageResult{ PackageName: "github.com/smartystreets/goconvey/webserver/examples", Elapsed: 0.019, Outcome: contract.Passed, Coverage: 75.5, TestResults: []contract.TestResult{ contract.TestResult{ TestName: "TestPassingStory", Elapsed: 0.01, Passed: true, File: "", Line: 0, Message: "", Stories: []reporting.ScopeResult{ reporting.ScopeResult{ Title: "A passing story", File: "/Users/mike/work/dev/goconvey/src/github.com/smartystreets/goconvey/webserver/examples/old_school_test.go", Line: 11, Depth: 0, Assertions: []*reporting.AssertionResult{ &reporting.AssertionResult{ File: "/Users/mike/work/dev/goconvey/src/github.com/smartystreets/goconvey/webserver/examples/old_school_test.go", Line: 10, Failure: "", Error: nil, Skipped: false, StackTrace: "goroutine 3 [running]:\ngithub.com/smartystreets/goconvey/webserver/examples.func·001()\n\u0009/Users/mike/work/dev/goconvey/src/github.com/smartystreets/goconvey/webserver/examples/old_school_test.go:10 +0xe3\ngithub.com/smartystreets/goconvey/webserver/examples.TestPassingStory(0x210314000)\n\u0009/Users/mike/work/dev/goconvey/src/github.com/smartystreets/goconvey/webserver/examples/old_school_test.go:11 +0xec\ntesting.tRunner(0x210314000, 0x21ab10)\n\u0009/usr/local/go/src/pkg/testing/testing.go:353 +0x8a\ncreated by testing.RunTests\n\u0009/usr/local/go/src/pkg/testing/testing.go:433 +0x86b\n", }, }, }, }, }, }, } const inputGoConvey_WithRandomOutput = ` === RUN TestPassingStory *** Hello, World! (1) *** *** Hello, World! (2) *** *** Hello, World! (3) ***>->->OPEN-JSON->->-> { "Title": "A passing story", "File": "/Users/mike/work/dev/goconvey/src/github.com/smartystreets/goconvey/web/server/testing/go_test.go", "Line": 16, "Depth": 0, "Assertions": [ { "File": "/Users/mike/work/dev/goconvey/src/github.com/smartystreets/goconvey/web/server/testing/go_test.go", "Line": 14, "Failure": "", "Error": null, "Skipped": false, "StackTrace": "goroutine 3 [running]:\ngithub.com/smartystreets/goconvey/web/server/testing.func·001()\n\u0009/Users/mike/work/dev/goconvey/src/github.com/smartystreets/goconvey/web/server/testing/go_test.go:14 +0x186\ngithub.com/smartystreets/goconvey/web/server/testing.TestPassingStory(0x210315000)\n\u0009/Users/mike/work/dev/goconvey/src/github.com/smartystreets/goconvey/web/server/testing/go_test.go:16 +0x1b9\ntesting.tRunner(0x210315000, 0x21bb10)\n\u0009/usr/local/go/src/pkg/testing/testing.go:353 +0x8a\ncreated by testing.RunTests\n\u0009/usr/local/go/src/pkg/testing/testing.go:433 +0x86b\n" } ] }, <-<-<-CLOSE-JSON<-<-< *** Hello, World! (4)*** *** Hello, World! (5) *** >->->OPEN-JSON->->-> { "Title": "A passing story", "File": "/Users/mike/work/dev/goconvey/src/github.com/smartystreets/goconvey/web/server/testing/go_test.go", "Line": 22, "Depth": 0, "Assertions": [ { "File": "/Users/mike/work/dev/goconvey/src/github.com/smartystreets/goconvey/web/server/testing/go_test.go", "Line": 20, "Failure": "", "Error": null, "Skipped": false, "StackTrace": "goroutine 3 [running]:\ngithub.com/smartystreets/goconvey/web/server/testing.func·002()\n\u0009/Users/mike/work/dev/goconvey/src/github.com/smartystreets/goconvey/web/server/testing/go_test.go:20 +0x186\ngithub.com/smartystreets/goconvey/web/server/testing.TestPassingStory(0x210315000)\n\u0009/Users/mike/work/dev/goconvey/src/github.com/smartystreets/goconvey/web/server/testing/go_test.go:22 +0x294\ntesting.tRunner(0x210315000, 0x21bb10)\n\u0009/usr/local/go/src/pkg/testing/testing.go:353 +0x8a\ncreated by testing.RunTests\n\u0009/usr/local/go/src/pkg/testing/testing.go:433 +0x86b\n" } ] }, <-<-<-CLOSE-JSON<-<-< *** Hello, World! (6) *** --- PASS: TestPassingStory (0.03 seconds) PASS coverage: 45.0%% of statements ok github.com/smartystreets/goconvey/web/server/testing 0.024s ` var expectedGoConvey_WithRandomOutput = contract.PackageResult{ PackageName: "github.com/smartystreets/goconvey/web/server/testing", Elapsed: 0.024, Outcome: contract.Passed, Coverage: 45.0, TestResults: []contract.TestResult{ contract.TestResult{ TestName: "TestPassingStory", Elapsed: 0.03, Passed: true, File: "", Line: 0, Message: "*** Hello, World! (1) ***\n*** Hello, World! (2) ***\n*** Hello, World! (3) ***\n*** Hello, World! (4)***\n*** Hello, World! (5) ***\n*** Hello, World! (6) ***", Stories: []reporting.ScopeResult{ reporting.ScopeResult{ Title: "A passing story", File: "/Users/mike/work/dev/goconvey/src/github.com/smartystreets/goconvey/web/server/testing/go_test.go", Line: 16, Depth: 0, Assertions: []*reporting.AssertionResult{ &reporting.AssertionResult{ File: "/Users/mike/work/dev/goconvey/src/github.com/smartystreets/goconvey/web/server/testing/go_test.go", Line: 14, Failure: "", Error: nil, Skipped: false, StackTrace: "goroutine 3 [running]:\ngithub.com/smartystreets/goconvey/web/server/testing.func·001()\n\u0009/Users/mike/work/dev/goconvey/src/github.com/smartystreets/goconvey/web/server/testing/go_test.go:14 +0x186\ngithub.com/smartystreets/goconvey/web/server/testing.TestPassingStory(0x210315000)\n\u0009/Users/mike/work/dev/goconvey/src/github.com/smartystreets/goconvey/web/server/testing/go_test.go:16 +0x1b9\ntesting.tRunner(0x210315000, 0x21bb10)\n\u0009/usr/local/go/src/pkg/testing/testing.go:353 +0x8a\ncreated by testing.RunTests\n\u0009/usr/local/go/src/pkg/testing/testing.go:433 +0x86b\n", }, }, }, reporting.ScopeResult{ Title: "A passing story", File: "/Users/mike/work/dev/goconvey/src/github.com/smartystreets/goconvey/web/server/testing/go_test.go", Line: 22, Depth: 0, Assertions: []*reporting.AssertionResult{ &reporting.AssertionResult{ File: "/Users/mike/work/dev/goconvey/src/github.com/smartystreets/goconvey/web/server/testing/go_test.go", Line: 20, Failure: "", Error: nil, Skipped: false, StackTrace: "goroutine 3 [running]:\ngithub.com/smartystreets/goconvey/web/server/testing.func·002()\n\u0009/Users/mike/work/dev/goconvey/src/github.com/smartystreets/goconvey/web/server/testing/go_test.go:20 +0x186\ngithub.com/smartystreets/goconvey/web/server/testing.TestPassingStory(0x210315000)\n\u0009/Users/mike/work/dev/goconvey/src/github.com/smartystreets/goconvey/web/server/testing/go_test.go:22 +0x294\ntesting.tRunner(0x210315000, 0x21bb10)\n\u0009/usr/local/go/src/pkg/testing/testing.go:353 +0x8a\ncreated by testing.RunTests\n\u0009/usr/local/go/src/pkg/testing/testing.go:433 +0x86b\n", }, }, }, }, }, }, } const inputOldSchool_PassesButCoverageIsBogus = ` === RUN TestOldSchool_Passes --- PASS: TestOldSchool_Passes (0.02 seconds) === RUN TestOldSchool_PassesWithMessage --- PASS: TestOldSchool_PassesWithMessage (0.05 seconds) old_school_test.go:10: I am a passing test. With a newline. PASS coverage: bogus%% of statements ok github.com/smartystreets/goconvey/webserver/examples 0.018s ` var expectedOldSchool_PassesButCoverageIsBogus = contract.PackageResult{ PackageName: "github.com/smartystreets/goconvey/webserver/examples", Elapsed: 0.018, Coverage: -1, Outcome: contract.Passed, TestResults: []contract.TestResult{ contract.TestResult{ TestName: "TestOldSchool_Passes", Elapsed: 0.02, Passed: true, File: "", Line: 0, Message: "", Stories: []reporting.ScopeResult{}, }, contract.TestResult{ TestName: "TestOldSchool_PassesWithMessage", Elapsed: 0.05, Passed: true, File: "old_school_test.go", Line: 10, Message: "old_school_test.go:10: I am a passing test.\nWith a newline.", Stories: []reporting.ScopeResult{}, }, }, } const inputNestedTests = ` === RUN TestNestedTests === RUN TestNestedTests_Passes --- PASS: TestNestedTests_Passes (0.02 seconds) === RUN TestNestedTests_Failure --- FAIL: TestNestedTests_Failure (0.06 seconds) === RUN TestNestedTests_FailureWithReason --- FAIL: TestNestedTests_FailureWithReason (0.11 seconds) nested_test.go:18: I am a failing test. === RUN TestNestedTests_Skipping --- SKIP: TestNestedTests_Skipping (0.00 seconds) nested_test.go:8: blah === RUN TestNestedTests_PassesWithMessage --- PASS: TestNestedTests_PassesWithMessage (0.05 seconds) nested_test.go:10: I am a passing test. With a newline. --- FAIL: TestNestedTests (0.25 seconds) FAIL exit status 1 FAIL github.com/smartystreets/goconvey/webserver/examples 0.018s ` var expectedNestedTests = contract.PackageResult{ PackageName: "github.com/smartystreets/goconvey/webserver/examples", Elapsed: 0.018, Outcome: contract.Failed, TestResults: []contract.TestResult{ contract.TestResult{ TestName: "TestNestedTests", Elapsed: 0.25, Passed: false, File: "", Line: 0, Message: "", Stories: []reporting.ScopeResult{}, }, contract.TestResult{ TestName: "TestNestedTests_Failure", Elapsed: 0.06, Passed: false, File: "", Line: 0, Message: "", Stories: []reporting.ScopeResult{}, }, contract.TestResult{ TestName: "TestNestedTests_FailureWithReason", Elapsed: 0.11, Passed: false, File: "nested_test.go", Line: 18, Message: "nested_test.go:18: I am a failing test.", Stories: []reporting.ScopeResult{}, }, contract.TestResult{ TestName: "TestNestedTests_Passes", Elapsed: 0.02, Passed: true, File: "", Line: 0, Message: "", Stories: []reporting.ScopeResult{}, }, contract.TestResult{ TestName: "TestNestedTests_PassesWithMessage", Elapsed: 0.05, Passed: true, File: "nested_test.go", Line: 10, Message: "nested_test.go:10: I am a passing test.\nWith a newline.", Stories: []reporting.ScopeResult{}, }, contract.TestResult{ TestName: "TestNestedTests_Skipping", Elapsed: 0.00, Passed: true, Skipped: true, File: "nested_test.go", Line: 8, Message: "nested_test.go:8: blah", Stories: []reporting.ScopeResult{}, }, }, } const inputExampleFunctions = ` === RUN Example_Failure --- FAIL: Example_Failure (0.11 seconds) got: actuall output want: real output === RUN Example_Pass --- PASS: Example_Pass (0.06 seconds) FAIL exit status 1 FAIL github.com/smartystreets/goconvey/webserver/examples 0.18s ` var expectedExampleFunctions = contract.PackageResult{ PackageName: "github.com/smartystreets/goconvey/webserver/examples", Elapsed: 0.18, Outcome: contract.Failed, TestResults: []contract.TestResult{ contract.TestResult{ TestName: "Example_Failure", Elapsed: 0.11, Passed: false, File: "", Line: 0, Message: "got:\nactuall output\nwant:\nreal output", Stories: []reporting.ScopeResult{}, }, contract.TestResult{ TestName: "Example_Pass", Elapsed: 0.06, Passed: true, File: "", Line: 0, Message: "", Stories: []reporting.ScopeResult{}, }, }, } const inputGolang15 = ` === RUN Golang15 --- PASS: Golang15 (0.00s) PASS ok github.com/smartystreets/goconvey/webserver/examples 0.008s ` var expectedGolang15 = contract.PackageResult{ PackageName: "github.com/smartystreets/goconvey/webserver/examples", Elapsed: 0.008, Outcome: contract.Passed, TestResults: []contract.TestResult{ contract.TestResult{ TestName: "Golang15", Elapsed: 0.00, Passed: true, File: "", Line: 0, Message: "", Stories: []reporting.ScopeResult{}, }, }, } mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/parser/parser.go000066400000000000000000000013141276133741700323560ustar00rootroot00000000000000package parser import ( "log" "github.com/smartystreets/goconvey/web/server/contract" ) type Parser struct { parser func(*contract.PackageResult, string) } func (self *Parser) Parse(packages []*contract.Package) { for _, p := range packages { if p.Active() && p.HasUsableResult() { self.parser(p.Result, p.Output) } else if p.Ignored { p.Result.Outcome = contract.Ignored } else if p.Disabled { p.Result.Outcome = contract.Disabled } else { p.Result.Outcome = contract.TestRunAbortedUnexpectedly } log.Printf("[%s]: %s\n", p.Result.Outcome, p.Name) } } func NewParser(helper func(*contract.PackageResult, string)) *Parser { self := new(Parser) self.parser = helper return self } mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/parser/parser.goconvey000066400000000000000000000000241276133741700335770ustar00rootroot00000000000000#ignore -timeout=1s mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/parser/parser_test.go000066400000000000000000000030011276133741700334100ustar00rootroot00000000000000package parser import ( "errors" "testing" . "github.com/smartystreets/goconvey/convey" "github.com/smartystreets/goconvey/web/server/contract" ) func TestParser(t *testing.T) { Convey("Subject: Parser parses test output for active packages", t, func() { packages := []*contract.Package{ &contract.Package{Ignored: false, Output: "Active", Result: contract.NewPackageResult("asdf")}, &contract.Package{Ignored: true, Output: "Inactive", Result: contract.NewPackageResult("qwer")}, } parser := NewParser(fakeParserImplementation) Convey("When given a collection of packages", func() { parser.Parse(packages) Convey("The parser uses its internal parsing mechanism to parse the output of only the active packages", func() { So(packages[0].Result.Outcome, ShouldEqual, packages[0].Output) }) Convey("The parser should mark inactive packages as ignored", func() { So(packages[1].Result.Outcome, ShouldEqual, contract.Ignored) }) }) Convey("When a package could not be tested (maybe it was deleted between scanning and execution?)", func() { packages[0].Output = "" packages[0].Error = errors.New("Directory does not exist") parser.Parse(packages) Convey("The package result should not be parsed and the outcome should actually resemble the problem", func() { So(packages[0].Result.Outcome, ShouldEqual, contract.TestRunAbortedUnexpectedly) }) }) }) } func fakeParserImplementation(result *contract.PackageResult, rawOutput string) { result.Outcome = rawOutput } mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/parser/rules.go000066400000000000000000000026571276133741700322270ustar00rootroot00000000000000package parser import "strings" func noGoFiles(line string) bool { return strings.HasPrefix(line, "can't load package: ") && strings.Contains(line, ": no buildable Go source files in ") } func buildFailed(line string) bool { return strings.HasPrefix(line, "# ") || strings.Contains(line, "cannot find package") || (strings.HasPrefix(line, "can't load package: ") && !strings.Contains(line, ": no Go source files in ")) || (strings.Contains(line, ": found packages ") && strings.Contains(line, ".go) and ") && strings.Contains(line, ".go) in ")) } func noTestFunctions(line string) bool { return line == "testing: warning: no tests to run" } func noTestFiles(line string) bool { return strings.HasPrefix(line, "?") && strings.Contains(line, "[no test files]") } func isNewTest(line string) bool { return strings.HasPrefix(line, "=== ") } func isTestResult(line string) bool { return strings.HasPrefix(line, "--- ") } func isPackageReport(line string) bool { return (strings.HasPrefix(line, "FAIL") || strings.HasPrefix(line, "exit status") || strings.HasPrefix(line, "PASS") || isCoverageSummary(line) || packagePassed(line)) } func packageFailed(line string) bool { return strings.HasPrefix(line, "FAIL\t") } func packagePassed(line string) bool { return strings.HasPrefix(line, "ok \t") } func isCoverageSummary(line string) bool { return strings.HasPrefix(line, "coverage: ") && strings.Contains(line, "% of statements") } mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/parser/testParser.go000066400000000000000000000106511276133741700332220ustar00rootroot00000000000000package parser import ( "encoding/json" "fmt" "strconv" "strings" "github.com/smartystreets/goconvey/convey/reporting" "github.com/smartystreets/goconvey/web/server/contract" ) type testParser struct { test *contract.TestResult line string index int inJson bool jsonLines []string otherLines []string } func parseTestOutput(test *contract.TestResult) *contract.TestResult { parser := newTestParser(test) parser.parseTestFunctionOutput() return test } func newTestParser(test *contract.TestResult) *testParser { self := new(testParser) self.test = test return self } func (self *testParser) parseTestFunctionOutput() { if len(self.test.RawLines) > 0 { self.processLines() self.deserializeJson() self.composeCapturedOutput() } } func (self *testParser) processLines() { for self.index, self.line = range self.test.RawLines { if !self.processLine() { break } } } func (self *testParser) processLine() bool { if strings.HasSuffix(self.line, reporting.OpenJson) { self.inJson = true self.accountForOutputWithoutNewline() } else if self.line == reporting.CloseJson { self.inJson = false } else if self.inJson { self.jsonLines = append(self.jsonLines, self.line) } else if isPanic(self.line) { self.parsePanicOutput() return false } else if isGoTestLogOutput(self.line) { self.parseLogLocation() } else { self.otherLines = append(self.otherLines, self.line) } return true } // If fmt.Print(f) produces output with no \n and that output // is that last output before the framework spits out json // (which starts with ''>>>>>'') then without this code // all of the json is counted as output, not as json to be // parsed and displayed by the web UI. func (self *testParser) accountForOutputWithoutNewline() { prefix := strings.Split(self.line, reporting.OpenJson)[0] if prefix != "" { self.otherLines = append(self.otherLines, prefix) } } func (self *testParser) deserializeJson() { formatted := createArrayForJsonItems(self.jsonLines) var scopes []reporting.ScopeResult err := json.Unmarshal(formatted, &scopes) if err != nil { panic(fmt.Sprintf(bugReportRequest, err, formatted)) } self.test.Stories = scopes } func (self *testParser) parsePanicOutput() { for index, line := range self.test.RawLines[self.index:] { self.parsePanicLocation(index, line) self.preserveStackTraceIndentation(index, line) } self.test.Error = strings.Join(self.test.RawLines, "\n") } func (self *testParser) parsePanicLocation(index int, line string) { if !panicLineHasMetadata(line) { return } metaLine := self.test.RawLines[index+4] fields := strings.Split(metaLine, " ") fileAndLine := strings.Split(fields[0], ":") self.test.File = fileAndLine[0] if len(fileAndLine) >= 2 { self.test.Line, _ = strconv.Atoi(fileAndLine[1]) } } func (self *testParser) preserveStackTraceIndentation(index int, line string) { if panicLineShouldBeIndented(index, line) { self.test.RawLines[index] = "\t" + line } } func (self *testParser) parseLogLocation() { self.otherLines = append(self.otherLines, self.line) lineFields := self.line fields := strings.Split(lineFields, ":") self.test.File = strings.TrimSpace(fields[0]) self.test.Line, _ = strconv.Atoi(fields[1]) } func (self *testParser) composeCapturedOutput() { self.test.Message = strings.Join(self.otherLines, "\n") } func createArrayForJsonItems(lines []string) []byte { jsonArrayItems := strings.Join(lines, "") jsonArrayItems = removeTrailingComma(jsonArrayItems) return []byte(fmt.Sprintf("[%s]\n", jsonArrayItems)) } func removeTrailingComma(rawJson string) string { if trailingComma(rawJson) { return rawJson[:len(rawJson)-1] } return rawJson } func trailingComma(value string) bool { return strings.HasSuffix(value, ",") } func isGoTestLogOutput(line string) bool { return strings.Count(line, ":") == 2 } func isPanic(line string) bool { return strings.HasPrefix(line, "panic: ") } func panicLineHasMetadata(line string) bool { return strings.HasPrefix(line, "goroutine") && strings.Contains(line, "[running]") } func panicLineShouldBeIndented(index int, line string) bool { return strings.Contains(line, "+") || (index > 0 && strings.Contains(line, "panic: ")) } const bugReportRequest = ` Uh-oh! Looks like something went wrong. Please copy the following text and file a bug report at: https://github.com/smartystreets/goconvey/issues?state=open ======= BEGIN BUG REPORT ======= ERROR: %v OUTPUT: %s ======= END BUG REPORT ======= ` mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/parser/util.go000066400000000000000000000022001276133741700320320ustar00rootroot00000000000000package parser import ( "math" "strings" "time" ) // parseTestFunctionDuration parses the duration in seconds as a float64 // from a line of go test output that looks something like this: // --- PASS: TestOldSchool_PassesWithMessage (0.03 seconds) func parseTestFunctionDuration(line string) float64 { line = strings.Replace(line, "(", "", 1) fields := strings.Split(line, " ") return parseDurationInSeconds(fields[3]+"s", 2) } func parseDurationInSeconds(raw string, precision int) float64 { elapsed, _ := time.ParseDuration(raw) return round(elapsed.Seconds(), precision) } // round returns the rounded version of x with precision. // // Special cases are: // round(±0) = ±0 // round(±Inf) = ±Inf // round(NaN) = NaN // // Why, oh why doesn't the math package come with a round function? // Inspiration: http://play.golang.org/p/ZmFfr07oHp func round(x float64, precision int) float64 { var rounder float64 pow := math.Pow(10, float64(precision)) intermediate := x * pow if intermediate < 0.0 { intermediate -= 0.5 } else { intermediate += 0.5 } rounder = float64(int64(intermediate)) return rounder / float64(pow) } mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/system/000077500000000000000000000000001276133741700305645ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/system/shell.go000066400000000000000000000127131276133741700322260ustar00rootroot00000000000000package system import ( "log" "os/exec" "path/filepath" "regexp" "strings" ) /////////////////////////////////////////////////////////////////////////////// // Integration: /////////////////////////////////////////////////////////////// /////////////////////////////////////////////////////////////////////////////// type Shell struct { coverage bool gobin string reportsPath string defaultTimeout string } func NewShell(gobin, reportsPath string, coverage bool, defaultTimeout string) *Shell { return &Shell{ coverage: coverage, gobin: gobin, reportsPath: reportsPath, defaultTimeout: defaultTimeout, } } func (self *Shell) GoTest(directory, packageName string, tags, arguments []string) (output string, err error) { reportFilename := strings.Replace(packageName, "/", "-", -1) reportPath := filepath.Join(self.reportsPath, reportFilename) reportData := reportPath + ".txt" reportHTML := reportPath + ".html" tagsArg := "-tags=" + strings.Join(tags, ",") goconvey := findGoConvey(directory, self.gobin, packageName, tagsArg).Execute() compilation := compile(directory, self.gobin, tagsArg).Execute() withCoverage := runWithCoverage(compilation, goconvey, self.coverage, reportData, directory, self.gobin, self.defaultTimeout, tagsArg, arguments).Execute() final := runWithoutCoverage(compilation, withCoverage, goconvey, directory, self.gobin, self.defaultTimeout, tagsArg, arguments).Execute() go generateReports(final, self.coverage, directory, self.gobin, reportData, reportHTML).Execute() return final.Output, final.Error } /////////////////////////////////////////////////////////////////////////////// // Functional Core://////////////////////////////////////////////////////////// /////////////////////////////////////////////////////////////////////////////// func findGoConvey(directory, gobin, packageName, tagsArg string) Command { return NewCommand(directory, gobin, "list", "-f", "'{{.TestImports}}'", tagsArg, packageName) } func compile(directory, gobin, tagsArg string) Command { return NewCommand(directory, gobin, "test", "-i", tagsArg) } func runWithCoverage(compile, goconvey Command, coverage bool, reportPath, directory, gobin, defaultTimeout, tagsArg string, customArguments []string) Command { if compile.Error != nil || goconvey.Error != nil { return compile } if !coverage { return compile } arguments := []string{"test", "-v", "-coverprofile=" + reportPath, tagsArg} customArgsText := strings.Join(customArguments, "\t") if !strings.Contains(customArgsText, "-covermode=") { arguments = append(arguments, "-covermode=set") } if !strings.Contains(customArgsText, "-timeout=") { arguments = append(arguments, "-timeout="+defaultTimeout) } if strings.Contains(goconvey.Output, goconveyDSLImport) { arguments = append(arguments, "-json") } arguments = append(arguments, customArguments...) return NewCommand(directory, gobin, arguments...) } func runWithoutCoverage(compile, withCoverage, goconvey Command, directory, gobin, defaultTimeout, tagsArg string, customArguments []string) Command { if compile.Error != nil { return compile } if goconvey.Error != nil { log.Println(gopathProblem, goconvey.Output, goconvey.Error) return goconvey } if coverageStatementRE.MatchString(withCoverage.Output) { return withCoverage } log.Printf("Coverage output: %v", withCoverage.Output) log.Print("Run without coverage") arguments := []string{"test", "-v", tagsArg} customArgsText := strings.Join(customArguments, "\t") if !strings.Contains(customArgsText, "-timeout=") { arguments = append(arguments, "-timeout="+defaultTimeout) } if strings.Contains(goconvey.Output, goconveyDSLImport) { arguments = append(arguments, "-json") } arguments = append(arguments, customArguments...) return NewCommand(directory, gobin, arguments...) } func generateReports(previous Command, coverage bool, directory, gobin, reportData, reportHTML string) Command { if previous.Error != nil { return previous } if !coverage { return previous } return NewCommand(directory, gobin, "tool", "cover", "-html="+reportData, "-o", reportHTML) } /////////////////////////////////////////////////////////////////////////////// // Imperative Shell: ////////////////////////////////////////////////////////// /////////////////////////////////////////////////////////////////////////////// type Command struct { directory string executable string arguments []string Output string Error error } func NewCommand(directory, executable string, arguments ...string) Command { return Command{ directory: directory, executable: executable, arguments: arguments, } } func (this Command) Execute() Command { if len(this.executable) == 0 { return this } if len(this.Output) > 0 || this.Error != nil { return this } command := exec.Command(this.executable, this.arguments...) command.Dir = this.directory var rawOutput []byte rawOutput, this.Error = command.CombinedOutput() this.Output = string(rawOutput) return this } /////////////////////////////////////////////////////////////////////////////// const goconveyDSLImport = "github.com/smartystreets/goconvey/convey " // note the trailing space: we don't want to target packages nested in the /convey package. const gopathProblem = "Please run goconvey from within $GOPATH/src (also, symlinks might be problematic). Output and Error: " var coverageStatementRE = regexp.MustCompile(`(?m)^coverage: \d+\.\d% of statements(.*)$|^panic: test timed out after `) shell_integration_test.go000066400000000000000000000016151276133741700356100ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/systempackage system import ( "log" "path/filepath" "runtime" "strings" "testing" ) func TestShellIntegration(t *testing.T) { if testing.Short() { t.Skip("Skipping potentially long-running integration test...") return } log.SetFlags(log.LstdFlags | log.Lshortfile | log.Lmicroseconds) _, filename, _, _ := runtime.Caller(0) directory := filepath.Join(filepath.Dir(filename), "..", "watch", "integration_testing", "sub") packageName := "github.com/smartystreets/goconvey/web/server/watch/integration_testing/sub" shell := NewShell("go", "", true, "5s") output, err := shell.GoTest(directory, packageName, []string{}, []string{"-short"}) if !strings.Contains(output, "PASS\n") || !strings.Contains(output, "ok") { t.Errorf("Expected output that resembed tests passing but got this instead: [%s]", output) } if err != nil { t.Error("Test run resulted in the following error:", err) } } mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/system/shell_test.go000066400000000000000000000213261276133741700332650ustar00rootroot00000000000000package system import ( "errors" "testing" . "github.com/smartystreets/goconvey/convey" ) func TestShellCommandComposition(t *testing.T) { var ( buildFailed = Command{Error: errors.New("BUILD FAILURE!")} buildSucceeded = Command{Output: "ok"} goConvey = Command{Output: "[fmt github.com/smartystreets/goconvey/convey net/http net/http/httptest path runtime strconv strings testing time]"} noGoConvey = Command{Output: "[fmt net/http net/http/httptest path runtime strconv strings testing time]"} errorGoConvey = Command{Output: "This is a wacky error", Error: errors.New("This happens when running goconvey outside your $GOPATH (symlinked code).")} noCoveragePassed = Command{Output: "PASS\nok github.com/smartystreets/goconvey/examples 0.012s"} coveragePassed = Command{Output: "PASS\ncoverage: 100.0% of statements\nok github.com/smartystreets/goconvey/examples 0.012s"} coverageFailed = Command{ Error: errors.New("Tests bombed!"), Output: "--- FAIL: TestIntegerManipulation (0.00 seconds)\nFAIL\ncoverage: 100.0% of statements\nexit status 1\nFAIL github.com/smartystreets/goconvey/examples 0.013s", } coverageFailedTimeout = Command{ Error: errors.New("Tests bombed!"), Output: "=== RUN SomeTest\n--- PASS: SomeTest (0.00 seconds)\n=== RUN TimeoutTest\npanic: test timed out after 5s\n\ngoroutine 27 [running]:\n", } ) const ( yesCoverage = true noCoverage = false ) Convey("When attempting to run tests with coverage flags", t, func() { Convey("And buildSucceeded failed", func() { result := runWithCoverage(buildFailed, goConvey, noCoverage, "", "", "", "", "-tags=", nil) Convey("Then no action should be taken", func() { So(result, ShouldResemble, buildFailed) }) }) Convey("And coverage is not wanted", func() { result := runWithCoverage(buildSucceeded, goConvey, noCoverage, "", "", "", "", "-tags=", nil) Convey("Then no action should be taken", func() { So(result, ShouldResemble, buildSucceeded) }) }) Convey("And the package being tested usees the GoConvey DSL (`convey` package)", func() { result := runWithCoverage(buildSucceeded, goConvey, yesCoverage, "reportsPath", "/directory", "go", "5s", "-tags=bob", []string{"-arg1", "-arg2"}) Convey("The returned command should be well formed (and include the -json flag)", func() { So(result, ShouldResemble, Command{ directory: "/directory", executable: "go", arguments: []string{"test", "-v", "-coverprofile=reportsPath", "-tags=bob", "-covermode=set", "-timeout=5s", "-json", "-arg1", "-arg2"}, }) }) }) Convey("And the package being tested does NOT use the GoConvey DSL", func() { result := runWithCoverage(buildSucceeded, noGoConvey, yesCoverage, "reportsPath", "/directory", "go", "1s", "-tags=bob", []string{"-arg1", "-arg2"}) Convey("The returned command should be well formed (and NOT include the -json flag)", func() { So(result, ShouldResemble, Command{ directory: "/directory", executable: "go", arguments: []string{"test", "-v", "-coverprofile=reportsPath", "-tags=bob", "-covermode=set", "-timeout=1s", "-arg1", "-arg2"}, }) }) }) Convey("And the package being tested has been symlinked outside the $GOAPTH", func() { result := runWithCoverage(buildSucceeded, errorGoConvey, yesCoverage, "reportsPath", "/directory", "go", "1s", "-tags=", nil) Convey("The returned command should be the compilation command", func() { So(result, ShouldResemble, buildSucceeded) }) }) Convey("And the package being tested specifies an alternate covermode", func() { result := runWithCoverage(buildSucceeded, noGoConvey, yesCoverage, "reportsPath", "/directory", "go", "1s", "-tags=", []string{"-covermode=atomic"}) Convey("The returned command should allow the alternate value", func() { So(result, ShouldResemble, Command{ directory: "/directory", executable: "go", arguments: []string{"test", "-v", "-coverprofile=reportsPath", "-tags=", "-timeout=1s", "-covermode=atomic"}, }) }) }) Convey("And the package being tested specifies an alternate timeout", func() { result := runWithCoverage(buildSucceeded, noGoConvey, yesCoverage, "reportsPath", "/directory", "go", "1s", "-tags=", []string{"-timeout=5s"}) Convey("The returned command should allow the alternate value", func() { So(result, ShouldResemble, Command{ directory: "/directory", executable: "go", arguments: []string{"test", "-v", "-coverprofile=reportsPath", "-tags=", "-covermode=set", "-timeout=5s"}, }) }) }) }) Convey("When attempting to run tests without the coverage flags", t, func() { Convey("And tests already succeeded with coverage", func() { result := runWithoutCoverage(buildSucceeded, coveragePassed, goConvey, "/directory", "go", "1s", "-tags=", []string{"-arg1", "-arg2"}) Convey("Then no action should be taken", func() { So(result, ShouldResemble, coveragePassed) }) }) Convey("And tests already failed (legitimately) with coverage", func() { result := runWithoutCoverage(buildSucceeded, coverageFailed, goConvey, "/directory", "go", "1s", "-tags=", []string{"-arg1", "-arg2"}) Convey("Then no action should be taken", func() { So(result, ShouldResemble, coverageFailed) }) }) Convey("And tests already failed (timeout) with coverage", func() { result := runWithoutCoverage(buildSucceeded, coverageFailedTimeout, goConvey, "/directory", "go", "1s", "-tags=", []string{"-arg1", "-arg2"}) Convey("Then no action should be taken", func() { So(result, ShouldResemble, coverageFailedTimeout) }) }) Convey("And the build failed earlier", func() { result := runWithoutCoverage(buildFailed, Command{}, goConvey, "/directory", "go", "1s", "-tags=", []string{"-arg1", "-arg2"}) Convey("Then no action should be taken", func() { So(result, ShouldResemble, buildFailed) }) }) Convey("And the goconvey dsl command failed (probably because of symlinks)", func() { result := runWithoutCoverage(buildSucceeded, Command{}, errorGoConvey, "", "", "", "-tags=", nil) Convey("Then no action should be taken", func() { So(result, ShouldResemble, errorGoConvey) }) }) Convey("And the package being tested uses the GoConvey DSL (`convey` package)", func() { result := runWithoutCoverage(buildSucceeded, buildSucceeded, goConvey, "/directory", "go", "1s", "-tags=", []string{"-arg1", "-arg2"}) Convey("Then the returned command should be well formed (and include the -json flag)", func() { So(result, ShouldResemble, Command{ directory: "/directory", executable: "go", arguments: []string{"test", "-v", "-tags=", "-timeout=1s", "-json", "-arg1", "-arg2"}, }) }) }) Convey("And the package being tested does NOT use the GoConvey DSL", func() { result := runWithoutCoverage(buildSucceeded, noCoveragePassed, noGoConvey, "/directory", "go", "1s", "-tags=", []string{"-arg1", "-arg2"}) Convey("Then the returned command should be well formed (and NOT include the -json flag)", func() { So(result, ShouldResemble, Command{ directory: "/directory", executable: "go", arguments: []string{"test", "-v", "-tags=", "-timeout=1s", "-arg1", "-arg2"}, }) }) }) Convey("And the package being tested specifies an alternate timeout", func() { result := runWithoutCoverage(buildSucceeded, buildSucceeded, noGoConvey, "/directory", "go", "1s", "-tags=", []string{"-timeout=5s"}) Convey("The returned command should allow the alternate value", func() { So(result, ShouldResemble, Command{ directory: "/directory", executable: "go", arguments: []string{"test", "-v", "-tags=", "-timeout=5s"}, }) }) }) }) Convey("When generating coverage reports", t, func() { Convey("And the previous command failed for any reason (compilation or failed tests)", func() { result := generateReports(buildFailed, yesCoverage, "/directory", "go", "reportData", "reportHTML") Convey("Then no action should be taken", func() { So(result, ShouldResemble, buildFailed) }) }) Convey("And coverage reports are unwanted", func() { result := generateReports(noCoveragePassed, noCoverage, "/directory", "go", "reportData", "reportHTML") Convey("Then no action should beg taken", func() { So(result, ShouldResemble, noCoveragePassed) }) }) Convey("And tests passed and coverage reports are wanted", func() { result := generateReports(coveragePassed, yesCoverage, "/directory", "go", "reportData", "reportHTML") Convey("Then the resulting command should be well-formed", func() { So(result, ShouldResemble, Command{ directory: "/directory", executable: "go", arguments: []string{"tool", "cover", "-html=reportData", "-o", "reportHTML"}, }) }) }) }) } mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/system/system.goconvey000066400000000000000000000000321276133741700336560ustar00rootroot00000000000000#ignore -timeout=1s -shortmongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/watch/000077500000000000000000000000001276133741700303465ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/watch/functional_core.go000066400000000000000000000105631276133741700340540ustar00rootroot00000000000000package watch import ( "os" "path/filepath" "strings" "github.com/smartystreets/goconvey/web/server/messaging" ) /////////////////////////////////////////////////////////////////////////////// func Categorize(items chan *FileSystemItem, root string, watchSuffixes []string) (folders, profiles, goFiles []*FileSystemItem) { for item := range items { if item.IsFolder && !isHidden(item.Name) && !foundInHiddenDirectory(item, root) { folders = append(folders, item) } else if strings.HasSuffix(item.Name, ".goconvey") && len(item.Name) > len(".goconvey") { profiles = append(profiles, item) } else { for _, suffix := range watchSuffixes { if strings.HasSuffix(item.Name, suffix) && !isHidden(item.Name) && !foundInHiddenDirectory(item, root) { goFiles = append(goFiles, item) } } } } return folders, profiles, goFiles } func foundInHiddenDirectory(item *FileSystemItem, root string) bool { path := item.Path if len(path) > len(root) { path = path[len(root):] } for _, folder := range strings.Split(filepath.Dir(path), slash) { if isHidden(folder) { return true } } return false } func isHidden(path string) bool { return strings.HasPrefix(path, ".") || strings.HasPrefix(path, "_") || strings.HasPrefix(path, "flymake_") } /////////////////////////////////////////////////////////////////////////////// func ParseProfile(profile string) (isDisabled bool, tags, arguments []string) { lines := strings.Split(profile, "\n") for _, line := range lines { line = strings.TrimSpace(line) if len(arguments) == 0 && strings.ToLower(line) == "ignore" { return true, nil, nil } else if strings.HasPrefix(line, "-tags=") { tags = append(tags, strings.Split(strings.SplitN(line, "=", 2)[1], ",")...) continue } else if len(line) == 0 { continue } else if strings.HasPrefix(line, "#") { continue } else if strings.HasPrefix(line, "//") { continue } else if line == "-cover" || strings.HasPrefix(line, "-coverprofile") { continue } else if line == "-v" { continue // Verbose mode is always enabled so there is no need to record it here. } arguments = append(arguments, line) } return false, tags, arguments } /////////////////////////////////////////////////////////////////////////////// func CreateFolders(items []*FileSystemItem) messaging.Folders { folders := map[string]*messaging.Folder{} for _, item := range items { folders[item.Path] = &messaging.Folder{Path: item.Path, Root: item.Root} } return folders } /////////////////////////////////////////////////////////////////////////////// func LimitDepth(folders messaging.Folders, depth int) { if depth < 0 { return } for path, folder := range folders { if strings.Count(path[len(folder.Root):], slash) > depth { delete(folders, path) } } } /////////////////////////////////////////////////////////////////////////////// func AttachProfiles(folders messaging.Folders, items []*FileSystemItem) { for _, profile := range items { if folder, exists := folders[filepath.Dir(profile.Path)]; exists { folder.Disabled, folder.BuildTags, folder.TestArguments = profile.ProfileDisabled, profile.ProfileTags, profile.ProfileArguments } } } /////////////////////////////////////////////////////////////////////////////// func MarkIgnored(folders messaging.Folders, ignored map[string]struct{}) { if len(ignored) == 0 { return } for _, folder := range folders { for ignored := range ignored { if !folder.Ignored && strings.HasSuffix(folder.Path, ignored) { folder.Ignored = true } } } } /////////////////////////////////////////////////////////////////////////////// func ActiveFolders(folders messaging.Folders) messaging.Folders { var active messaging.Folders = map[string]*messaging.Folder{} for path, folder := range folders { if folder.Ignored || folder.Disabled { continue } active[path] = folder } return active } /////////////////////////////////////////////////////////////////////////////// func Sum(folders messaging.Folders, items []*FileSystemItem) int64 { var sum int64 for _, item := range items { if _, exists := folders[filepath.Dir(item.Path)]; exists { sum += item.Size + item.Modified } } return sum } /////////////////////////////////////////////////////////////////////////////// const slash = string(os.PathSeparator) /////////////////////////////////////////////////////////////////////////////// functional_core_test.go000066400000000000000000000252061276133741700350340ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/watchpackage watch import ( "fmt" "testing" . "github.com/smartystreets/goconvey/convey" "github.com/smartystreets/goconvey/web/server/messaging" ) func TestCategorize(t *testing.T) { fileSystem := []*FileSystemItem{ { Root: "/.hello", Path: "/.hello", Name: "hello", IsFolder: true, }, { Root: "/.hello", Path: "/.hello/1/hello/world.txt", Name: "world.txt", IsFolder: false, }, { Root: "/.hello", Path: "/.hello/1/2/3/4/5/hello/world.go", Name: "world.go", IsFolder: false, }, { Root: "/.hello", Path: "/.hello/world.go", Name: "world.go", IsFolder: false, }, { Root: "/.hello", Path: "/.hello/hello/world.tmpl", Name: "world.tmpl", IsFolder: false, }, { Root: "/.hello", Path: "/.hello/hello/.world.go", Name: ".world.go", IsFolder: false, }, { Root: "/.hello", Path: "/.hello/hello/_world.go", Name: ".world.go", IsFolder: false, }, { Root: "/.hello", Path: "/.hello/hello/flymake_world.go", Name: "flymake_world.go", IsFolder: false, }, { Root: "/.hello", Path: "/.hello/.hello", Name: ".hello", IsFolder: true, }, { Root: "/.hello", Path: "/.hello/.hello/hello", Name: "hello", IsFolder: true, }, { Root: "/.hello", Path: "/.hello/.hello/world.go", Name: "world.go", IsFolder: false, }, { Root: "/.hello", Path: "/.hello/hello/hi.goconvey", Name: "hi.goconvey", IsFolder: false, }, { Root: "/.hello", Path: "/.hello/hello2/.goconvey", Name: ".goconvey", IsFolder: false, }, { Root: "/.hello", Path: "/.hello/_hello", Name: "_hello", IsFolder: true, }, } Convey("A stream of file system items should be categorized correctly", t, func() { items := make(chan *FileSystemItem) go func() { for _, item := range fileSystem { items <- item } close(items) }() folders, profiles, goFiles := Categorize(items, "/.hello", []string{".go"}) So(folders, ShouldResemble, fileSystem[:1]) So(profiles, ShouldResemble, fileSystem[11:12]) So(goFiles, ShouldResemble, fileSystem[2:4]) }) Convey("A stream of file system items should be categorized correctly", t, func() { items := make(chan *FileSystemItem) go func() { for _, item := range fileSystem { items <- item } close(items) }() folders, profiles, goFiles := Categorize(items, "/.hello", []string{".go", ".tmpl"}) So(folders, ShouldResemble, fileSystem[:1]) So(profiles, ShouldResemble, fileSystem[11:12]) So(goFiles, ShouldResemble, fileSystem[2:5]) }) } func TestParseProfile(t *testing.T) { var parseProfileTestCases = []struct { SKIP bool description string input string resultIgnored bool resultTestTags []string resultTestArgs []string }{ { SKIP: false, description: "Blank profile", input: "", resultIgnored: false, }, { SKIP: false, description: "All lines are blank or whitespace", input: "\n \n \t\t\t \n \n \n", resultIgnored: false, }, { SKIP: false, description: "Ignored package, no args included", input: "IGNORE\n-timeout=4s", resultIgnored: true, }, { SKIP: false, description: "Ignore directive is commented, all args are included", input: "#IGNORE\n-timeout=4s\n-parallel=5", resultIgnored: false, resultTestArgs: []string{"-timeout=4s", "-parallel=5"}, }, { SKIP: false, description: "No ignore directive, all args are included", input: "-run=TestBlah\n-timeout=42s", resultIgnored: false, resultTestArgs: []string{"-run=TestBlah", "-timeout=42s"}, }, { SKIP: false, description: "Some args are commented, therefore ignored", input: "-run=TestBlah\n #-timeout=42s", resultIgnored: false, resultTestArgs: []string{"-run=TestBlah"}, }, { SKIP: false, description: "All args are commented, therefore all are ignored", input: "#-run=TestBlah\n//-timeout=42", resultIgnored: false, }, { SKIP: false, description: "We ignore certain flags like -v and -cover and -coverprofile because they are specified by the shell", input: "-v\n-cover\n-coverprofile=blah.out", resultIgnored: false, }, { SKIP: false, description: "We allow certain coverage flags like -coverpkg and -covermode", input: "-coverpkg=blah\n-covermode=atomic", resultIgnored: false, resultTestArgs: []string{"-coverpkg=blah", "-covermode=atomic"}, }, { SKIP: false, description: "We parse out -tags particularly", input: "-coverpkg=blah\n-covermode=atomic\n-tags=foo,bar", resultIgnored: false, resultTestTags: []string{"foo", "bar"}, resultTestArgs: []string{"-coverpkg=blah", "-covermode=atomic"}, }, } for i, test := range parseProfileTestCases { if test.SKIP { SkipConvey(fmt.Sprintf("Profile Parsing, Test Case #%d: %s (SKIPPED)", i, test.description), t, nil) } else { Convey(fmt.Sprintf("Profile Parsing, Test Case #%d: %s", i, test.description), t, func() { ignored, testTags, testArgs := ParseProfile(test.input) So(ignored, ShouldEqual, test.resultIgnored) So(testTags, ShouldResemble, test.resultTestTags) So(testArgs, ShouldResemble, test.resultTestArgs) }) } } } func TestCreateFolders(t *testing.T) { Convey("File system items that represent folders should be converted to folder structs correctly", t, func() { expected := map[string]*messaging.Folder{ "/root/1": {Path: "/root/1", Root: "/root"}, "/root/1/2": {Path: "/root/1/2", Root: "/root"}, "/root/1/2/3": {Path: "/root/1/2/3", Root: "/root"}, } inputs := []*FileSystemItem{ {Path: "/root/1", Root: "/root", IsFolder: true}, {Path: "/root/1/2", Root: "/root", IsFolder: true}, {Path: "/root/1/2/3", Root: "/root", IsFolder: true}, } actual := CreateFolders(inputs) for key, actualValue := range actual { So(actualValue, ShouldResemble, expected[key]) } }) } func TestLimitDepth(t *testing.T) { Convey("Subject: Limiting folders based on relative depth from a common root", t, func() { folders := map[string]*messaging.Folder{ "/root/1": { Path: "/root/1", Root: "/root", }, "/root/1/2": { Path: "/root/1/2", Root: "/root", }, "/root/1/2/3": { Path: "/root/1/2/3", Root: "/root", }, } Convey("When there is no depth limit", func() { LimitDepth(folders, -1) Convey("No folders should be excluded", func() { So(len(folders), ShouldEqual, 3) }) }) Convey("When there is a limit", func() { LimitDepth(folders, 2) Convey("The deepest folder (in this case) should be excluded", func() { So(len(folders), ShouldEqual, 2) _, exists := folders["/root/1/2/3"] So(exists, ShouldBeFalse) }) }) }) } func TestAttachProfiles(t *testing.T) { Convey("Subject: Attaching profile information to a folder", t, func() { folders := map[string]*messaging.Folder{ "/root/1": { Path: "/root/1", Root: "/root", }, "/root/1/2": { Path: "/root/1/2", Root: "/root", }, "/root/1/2/3": { Path: "/root/1/2/3", Root: "/root", }, } profiles := []*FileSystemItem{ { Path: "/root/too-shallow.goconvey", ProfileDisabled: true, ProfileArguments: []string{"1", "2"}, }, { Path: "/root/1/2/hi.goconvey", ProfileDisabled: true, ProfileArguments: []string{"1", "2"}, }, { Path: "/root/1/2/3/4/does-not-exist", ProfileDisabled: true, ProfileArguments: []string{"1", "2", "3", "4"}, }, } Convey("Profiles that match folders should be merged with those folders", func() { AttachProfiles(folders, profiles) Convey("No profiles matched the first folder, so no assignments should occur", func() { So(folders["/root/1"].Disabled, ShouldBeFalse) So(folders["/root/1"].TestArguments, ShouldBeEmpty) }) Convey("The second folder should match the first profile", func() { So(folders["/root/1/2"].Disabled, ShouldBeTrue) So(folders["/root/1/2"].TestArguments, ShouldResemble, []string{"1", "2"}) }) Convey("No profiles match the third folder so no assignments should occur", func() { So(folders["/root/1/2/3"].Disabled, ShouldBeFalse) So(folders["/root/1/2/3"].TestArguments, ShouldBeEmpty) }) }) }) } func TestMarkIgnored(t *testing.T) { Convey("Subject: folders that have been ignored should be marked as such", t, func() { folders := map[string]*messaging.Folder{ "/root/1": { Path: "/root/1", Root: "/root", }, "/root/1/2": { Path: "/root/1/2", Root: "/root", }, "/root/1/2/3": { Path: "/root/1/2/3", Root: "/root", }, } Convey("When there are no ignored folders", func() { ignored := map[string]struct{}{} MarkIgnored(folders, ignored) Convey("No folders should be marked as ignored", func() { So(folders["/root/1"].Ignored, ShouldBeFalse) So(folders["/root/1/2"].Ignored, ShouldBeFalse) So(folders["/root/1/2/3"].Ignored, ShouldBeFalse) }) }) Convey("When there are ignored folders", func() { ignored := map[string]struct{}{"1/2": {}} MarkIgnored(folders, ignored) Convey("The ignored folders should be marked as ignored", func() { So(folders["/root/1"].Ignored, ShouldBeFalse) So(folders["/root/1/2"].Ignored, ShouldBeTrue) So(folders["/root/1/2/3"].Ignored, ShouldBeFalse) }) }) }) } func TestActiveFolders(t *testing.T) { Convey("Subject: Folders that are not ignored or disabled are active", t, func() { folders := map[string]*messaging.Folder{ "/root/1": { Path: "/root/1", Root: "/root", Ignored: true, }, "/root/1/2": { Path: "/root/1/2", Root: "/root", }, "/root/1/2/3": { Path: "/root/1/2/3", Root: "/root", Disabled: true, }, } active := ActiveFolders(folders) So(len(active), ShouldEqual, 1) So(active["/root/1/2"], ShouldResemble, folders["/root/1/2"]) }) } func TestSum(t *testing.T) { Convey("Subject: file system items within specified directores should be counted and summed", t, func() { folders := map[string]*messaging.Folder{ "/root/1": {Path: "/root/1", Root: "/root", Ignored: true}, } items := []*FileSystemItem{ {Size: 1, Modified: 3, Path: "/root/1/hi.go"}, {Size: 7, Modified: 13, Path: "/root/1/bye.go"}, {Size: 33, Modified: 45, Path: "/root/1/2/salutations.go"}, // not counted } So(Sum(folders, items), ShouldEqual, 1+3+7+13) }) } imperative_shell.go000066400000000000000000000034511276133741700341550ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/watchpackage watch import ( "io" "io/ioutil" "os" "path/filepath" ) /////////////////////////////////////////////////////////////////////////////// type FileSystemItem struct { Root string Path string Name string Size int64 Modified int64 IsFolder bool ProfileDisabled bool ProfileTags []string ProfileArguments []string } /////////////////////////////////////////////////////////////////////////////// func YieldFileSystemItems(root string, excludedDirs []string) chan *FileSystemItem { items := make(chan *FileSystemItem) go func() { filepath.Walk(root, func(path string, info os.FileInfo, err error) error { if err != nil { return filepath.SkipDir } basePath := filepath.Base(path) for _, item := range excludedDirs { if item == basePath && info.IsDir() && item != "" && basePath != "" { return filepath.SkipDir } } items <- &FileSystemItem{ Root: root, Path: path, Name: info.Name(), Size: info.Size(), Modified: info.ModTime().Unix(), IsFolder: info.IsDir(), } return nil }) close(items) }() return items } /////////////////////////////////////////////////////////////////////////////// // ReadContents reads files wholesale. This function is only called on files // that end in '.goconvey'. These files should be very small, probably not // ever more than a few hundred bytes. The ignored errors are ok because in // the event of an IO error all that need be returned is an empty string. func ReadContents(path string) string { file, err := os.Open(path) if err != nil { return "" } defer file.Close() reader := io.LimitReader(file, 1024*4) content, _ := ioutil.ReadAll(reader) return string(content) } /////////////////////////////////////////////////////////////////////////////// mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/watch/integration.go000066400000000000000000000106431276133741700332240ustar00rootroot00000000000000package watch import ( "log" "os" "strings" "sync" "time" "github.com/smartystreets/goconvey/web/server/messaging" ) type Watcher struct { nap time.Duration rootFolder string folderDepth int ignoredFolders map[string]struct{} fileSystemState int64 paused bool stopped bool watchSuffixes []string excludedDirs []string input chan messaging.WatcherCommand output chan messaging.Folders lock sync.RWMutex } func NewWatcher(rootFolder string, folderDepth int, nap time.Duration, input chan messaging.WatcherCommand, output chan messaging.Folders, watchSuffixes string, excludedDirs []string) *Watcher { return &Watcher{ nap: nap, rootFolder: rootFolder, folderDepth: folderDepth, input: input, output: output, watchSuffixes: strings.Split(watchSuffixes, ","), excludedDirs: excludedDirs, ignoredFolders: make(map[string]struct{}), } } func (this *Watcher) Listen() { for { if this.stopped { return } select { case command := <-this.input: this.respond(command) default: if !this.paused { this.scan() } time.Sleep(nap) } } } func (this *Watcher) respond(command messaging.WatcherCommand) { switch command.Instruction { case messaging.WatcherAdjustRoot: log.Println("Adjusting root...") this.rootFolder = command.Details this.execute() case messaging.WatcherIgnore: log.Println("Ignoring specified folders") this.ignore(command.Details) // Prevent a filesystem change due to the number of active folders changing _, checksum := this.gather() this.set(checksum) case messaging.WatcherReinstate: log.Println("Reinstating specified folders") this.reinstate(command.Details) // Prevent a filesystem change due to the number of active folders changing _, checksum := this.gather() this.set(checksum) case messaging.WatcherPause: log.Println("Pausing watcher...") this.paused = true case messaging.WatcherResume: log.Println("Resuming watcher...") this.paused = false case messaging.WatcherExecute: log.Println("Gathering folders for immediate execution...") this.execute() case messaging.WatcherStop: log.Println("Stopping the watcher...") close(this.output) this.stopped = true default: log.Println("Unrecognized command from server:", command.Instruction) } } func (this *Watcher) execute() { folders, _ := this.gather() this.sendToExecutor(folders) } func (this *Watcher) scan() { folders, checksum := this.gather() if checksum == this.fileSystemState { return } log.Println("File system state modified, publishing current folders...", this.fileSystemState, checksum) defer this.set(checksum) this.sendToExecutor(folders) } func (this *Watcher) gather() (folders messaging.Folders, checksum int64) { items := YieldFileSystemItems(this.rootFolder, this.excludedDirs) folderItems, profileItems, goFileItems := Categorize(items, this.rootFolder, this.watchSuffixes) for _, item := range profileItems { // TODO: don't even bother if the item's size is over a few hundred bytes... contents := ReadContents(item.Path) item.ProfileDisabled, item.ProfileTags, item.ProfileArguments = ParseProfile(contents) } folders = CreateFolders(folderItems) LimitDepth(folders, this.folderDepth) AttachProfiles(folders, profileItems) this.protectedRead(func() { MarkIgnored(folders, this.ignoredFolders) }) active := ActiveFolders(folders) checksum = int64(len(active)) checksum += Sum(active, profileItems) checksum += Sum(active, goFileItems) return folders, checksum } func (this *Watcher) set(state int64) { this.fileSystemState = state } func (this *Watcher) sendToExecutor(folders messaging.Folders) { this.output <- folders } func (this *Watcher) ignore(paths string) { this.protectedWrite(func() { for _, folder := range strings.Split(paths, string(os.PathListSeparator)) { this.ignoredFolders[folder] = struct{}{} log.Println("Currently ignored folders:", this.ignoredFolders) } }) } func (this *Watcher) reinstate(paths string) { this.protectedWrite(func() { for _, folder := range strings.Split(paths, string(os.PathListSeparator)) { delete(this.ignoredFolders, folder) } }) } func (this *Watcher) protectedWrite(protected func()) { this.lock.Lock() defer this.lock.Unlock() protected() } func (this *Watcher) protectedRead(protected func()) { this.lock.RLock() defer this.lock.RUnlock() protected() } const nap = time.Millisecond * 250 integration_test.go000066400000000000000000000116701276133741700342050ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/watchpackage watch import ( "bytes" "io/ioutil" "log" "os" "os/exec" "path/filepath" "runtime" "testing" "time" . "github.com/smartystreets/goconvey/convey" "github.com/smartystreets/goconvey/web/server/messaging" ) func TestWatcher(t *testing.T) { if testing.Short() { t.Skip("Skipping potentially long-running integration test...") return } log.SetFlags(log.LstdFlags | log.Lshortfile | log.Lmicroseconds) output := new(bytes.Buffer) log.SetOutput(output) defer func() { t.Log(output.String()) }() _, filename, _, _ := runtime.Caller(0) originalRoot := filepath.Join(filepath.Dir(filename), "integration_testing") temporary, err := ioutil.TempDir("/tmp", "goconvey") if err != nil { t.Fatal(err) } root := filepath.Join(temporary, "integration_testing") sub := filepath.Join(root, "sub") err = CopyDir(originalRoot, root) if err != nil { t.Fatal(err) } defer func() { err = os.RemoveAll(temporary) if err != nil { t.Fatal(err) } }() var ( // commands pause = messaging.WatcherCommand{Instruction: messaging.WatcherPause} resume = messaging.WatcherCommand{Instruction: messaging.WatcherResume} ignore = messaging.WatcherCommand{Instruction: messaging.WatcherIgnore, Details: sub} reinstate = messaging.WatcherCommand{Instruction: messaging.WatcherReinstate, Details: sub} adjustToSub = messaging.WatcherCommand{Instruction: messaging.WatcherAdjustRoot, Details: sub} adjustToRoot = messaging.WatcherCommand{Instruction: messaging.WatcherAdjustRoot, Details: root} execute = messaging.WatcherCommand{Instruction: messaging.WatcherExecute} bogus = messaging.WatcherCommand{Instruction: 42} stop = messaging.WatcherCommand{Instruction: messaging.WatcherStop} ) Convey("Subject: Watcher operations", t, func() { input := make(chan messaging.WatcherCommand) output := make(chan messaging.Folders) excludedDirs := []string{} watcher := NewWatcher(root, -1, time.Millisecond, input, output, ".go", excludedDirs) go watcher.Listen() Convey("Initial scan results", func() { go func() { input <- stop }() results := []messaging.Folders{} for result := range output { results = append(results, result) } So(len(results), ShouldEqual, 1) }) Convey("Manual execution produces additional results", func() { go func() { input <- execute input <- stop }() results := []messaging.Folders{} for result := range output { results = append(results, result) } So(len(results), ShouldEqual, 2) So(len(results[0]), ShouldEqual, 2) So(len(results[1]), ShouldEqual, 2) }) Convey("Ignore and Reinstate commands are not reflected in the scan results", func() { go func() { input <- ignore input <- reinstate input <- stop }() results := []messaging.Folders{} for result := range output { results = append(results, result) } So(len(results), ShouldEqual, 1) So(results[0][sub].Ignored, ShouldBeFalse) // initial }) Convey("Adjusting the root changes the number of folders in the scanned results", func() { go func() { input <- adjustToSub input <- adjustToRoot input <- stop }() results := []messaging.Folders{} for result := range output { results = append(results, result) } So(len(results), ShouldEqual, 3) So(len(results[0]), ShouldEqual, 2) // initial So(len(results[1]), ShouldEqual, 1) // root moved to sub So(len(results[2]), ShouldEqual, 2) // root moved back to original location }) Convey("A bogus command does not cause any additional scanning beyond the initial scan", func() { go func() { input <- bogus input <- stop }() results := []messaging.Folders{} for result := range output { results = append(results, result) } So(len(results), ShouldEqual, 1) // initial scan }) Convey("Scanning occurs as a result of a file system update", func() { go func() { time.Sleep(time.Second) exec.Command("touch", filepath.Join(root, "main.go")).Run() time.Sleep(time.Second) input <- stop }() results := []messaging.Folders{} for result := range output { results = append(results, result) } So(len(results), ShouldEqual, 2) }) Convey("Scanning does not occur as a result of resuming after a pause", func() { go func() { input <- pause input <- resume input <- stop }() results := []messaging.Folders{} for result := range output { results = append(results, result) } So(len(results), ShouldEqual, 1) }) Convey("Scanning should not occur when the watcher is paused", func() { go func() { input <- pause for x := 0; x < 2; x++ { time.Sleep(time.Millisecond * 250) exec.Command("touch", filepath.Join(root, "main.go")).Run() time.Sleep(time.Millisecond * 250) } input <- resume input <- stop }() results := []messaging.Folders{} for result := range output { results = append(results, result) } So(len(results), ShouldEqual, 2) }) }) } 000077500000000000000000000000001276133741700343475ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/watch/integration_testingdoc_test.go000066400000000000000000000000151276133741700364760ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/watch/integration_testingpackage main main.go000066400000000000000000000003131276133741700356170ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/watch/integration_testing// This file's only purpose is to provide a realistic // environment from which to run integration tests // against the Watcher. package main import "fmt" func main() { fmt.Println("Hello, World!") } 000077500000000000000000000000001276133741700351405ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/watch/integration_testing/sub.gitignore000066400000000000000000000002221276133741700371240ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/watch/integration_testing/subgithub.com-smartystreets-goconvey-web-server-integration_testing-sub.html github.com-smartystreets-goconvey-web-server-integration_testing-sub.txtstuff.go000066400000000000000000000002151276133741700366140ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/watch/integration_testing/sub// This file's only purpose is to provide a realistic // environment from which to run integration tests // against the Watcher. package sub stuff_test.go000066400000000000000000000003771276133741700376640ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/watch/integration_testing/sub// This file's only purpose is to provide a realistic // environment from which to run integration tests // against the Watcher. package sub import ( "fmt" "testing" ) func TestStuff(t *testing.T) { if testing.Short() { return } fmt.Println() } sub.goconvey000066400000000000000000000002371276133741700375060ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/watch/integration_testing/subIGNORE -short -run=TestStuff // This file's only purpose is to provide a realistic // environment from which to run integration tests // against the Watcher. mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/watch/util_test.go000066400000000000000000000036551276133741700327220ustar00rootroot00000000000000// Credits: https://gist.github.com/jaybill/2876519 package watch import "os" import "io" import "io/ioutil" import "log" // Copies original source to destination destination. func CopyFile(source string, destination string) (err error) { originalFile, err := os.Open(source) if err != nil { return err } defer originalFile.Close() destinationFile, err := os.Create(destination) if err != nil { return err } defer destinationFile.Close() _, err = io.Copy(destinationFile, originalFile) if err == nil { info, err := os.Stat(source) if err != nil { err = os.Chmod(destination, info.Mode()) } } return } // Recursively copies a directory tree, attempting to preserve permissions. // Source directory must exist, destination directory must *not* exist. func CopyDir(source string, destination string) (err error) { // get properties of source dir sourceFile, err := os.Stat(source) if err != nil { return err } if !sourceFile.IsDir() { return &CustomError{Message: "Source is not a directory"} } // ensure destination dir does not already exist _, err = os.Open(destination) if !os.IsNotExist(err) { return &CustomError{Message: "Destination already exists"} } // create destination dir err = os.MkdirAll(destination, sourceFile.Mode()) if err != nil { return err } entries, err := ioutil.ReadDir(source) for _, entry := range entries { sourcePath := source + "/" + entry.Name() destinationPath := destination + "/" + entry.Name() if entry.IsDir() { err = CopyDir(sourcePath, destinationPath) if err != nil { log.Println(err) } } else { // perform copy err = CopyFile(sourcePath, destinationPath) if err != nil { log.Println(err) } } } return } // A struct for returning custom error messages type CustomError struct { Message string } // Returns the error message defined in Message as a string func (this *CustomError) Error() string { return this.Message } mongo-tools-r3.2.11/vendor/src/github.com/smartystreets/goconvey/web/server/watch/watch.goconvey000066400000000000000000000000321276133741700332220ustar00rootroot00000000000000#ignore -timeout=1s -shortmongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/000077500000000000000000000000001276133741700232375ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/openssl/000077500000000000000000000000001276133741700247225ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/openssl/LICENSE000066400000000000000000000240411276133741700257300ustar00rootroot00000000000000Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: You must give any other recipients of the Work or Derivative Works a copy of this License; and You must cause any modified files to carry prominent notices stating that You changed the files; and You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/openssl/README.md000066400000000000000000000017521276133741700262060ustar00rootroot00000000000000# OpenSSL bindings for Go Please see http://godoc.org/github.com/spacemonkeygo/openssl for more info ### License Copyright (C) 2014 Space Monkey, Inc. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. ### Using on Windows 1. Install [mingw-w64](http://mingw-w64.sourceforge.net/) 2. Install [pkg-config-lite](http://sourceforge.net/projects/pkgconfiglite) 3. Build (or install precompiled) openssl for mingw32-w64 4. Set __PKG\_CONFIG\_PATH__ to the directory containing openssl.pc (i.e. c:\mingw64\mingw64\lib\pkgconfig) mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/openssl/bio.go000066400000000000000000000165531276133741700260340ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // +build cgo package openssl /* #include #include extern int cbioNew(BIO *b); static int cbioFree(BIO *b) { return 1; } extern int writeBioWrite(BIO *b, char *buf, int size); extern long writeBioCtrl(BIO *b, int cmd, long arg1, void *arg2); static int writeBioPuts(BIO *b, const char *str) { return writeBioWrite(b, (char*)str, (int)strlen(str)); } extern int readBioRead(BIO *b, char *buf, int size); extern long readBioCtrl(BIO *b, int cmd, long arg1, void *arg2); static BIO_METHOD writeBioMethod = { BIO_TYPE_SOURCE_SINK, "Go Write BIO", (int (*)(BIO *, const char *, int))writeBioWrite, NULL, writeBioPuts, NULL, writeBioCtrl, cbioNew, cbioFree, NULL}; static BIO_METHOD* BIO_s_writeBio() { return &writeBioMethod; } static BIO_METHOD readBioMethod = { BIO_TYPE_SOURCE_SINK, "Go Read BIO", NULL, readBioRead, NULL, NULL, readBioCtrl, cbioNew, cbioFree, NULL}; static BIO_METHOD* BIO_s_readBio() { return &readBioMethod; } */ import "C" import ( "errors" "io" "reflect" "sync" "unsafe" ) const ( SSLRecordSize = 16 * 1024 ) func nonCopyGoBytes(ptr uintptr, length int) []byte { var slice []byte header := (*reflect.SliceHeader)(unsafe.Pointer(&slice)) header.Cap = length header.Len = length header.Data = ptr return slice } func nonCopyCString(data *C.char, size C.int) []byte { return nonCopyGoBytes(uintptr(unsafe.Pointer(data)), int(size)) } //export cbioNew func cbioNew(b *C.BIO) C.int { b.shutdown = 1 b.init = 1 b.num = -1 b.ptr = nil b.flags = 0 return 1 } type writeBio struct { data_mtx sync.Mutex op_mtx sync.Mutex buf []byte release_buffers bool } func loadWritePtr(b *C.BIO) *writeBio { return (*writeBio)(unsafe.Pointer(b.ptr)) } func bioClearRetryFlags(b *C.BIO) { // from BIO_clear_retry_flags and BIO_clear_flags b.flags &= ^(C.BIO_FLAGS_RWS | C.BIO_FLAGS_SHOULD_RETRY) } func bioSetRetryRead(b *C.BIO) { // from BIO_set_retry_read and BIO_set_flags b.flags |= (C.BIO_FLAGS_READ | C.BIO_FLAGS_SHOULD_RETRY) } //export writeBioWrite func writeBioWrite(b *C.BIO, data *C.char, size C.int) (rc C.int) { defer func() { if err := recover(); err != nil { logger.Critf("openssl: writeBioWrite panic'd: %v", err) rc = -1 } }() ptr := loadWritePtr(b) if ptr == nil || data == nil || size < 0 { return -1 } ptr.data_mtx.Lock() defer ptr.data_mtx.Unlock() bioClearRetryFlags(b) ptr.buf = append(ptr.buf, nonCopyCString(data, size)...) return size } //export writeBioCtrl func writeBioCtrl(b *C.BIO, cmd C.int, arg1 C.long, arg2 unsafe.Pointer) ( rc C.long) { defer func() { if err := recover(); err != nil { logger.Critf("openssl: writeBioCtrl panic'd: %v", err) rc = -1 } }() switch cmd { case C.BIO_CTRL_WPENDING: return writeBioPending(b) case C.BIO_CTRL_DUP, C.BIO_CTRL_FLUSH: return 1 default: return 0 } } func writeBioPending(b *C.BIO) C.long { ptr := loadWritePtr(b) if ptr == nil { return 0 } ptr.data_mtx.Lock() defer ptr.data_mtx.Unlock() return C.long(len(ptr.buf)) } func (b *writeBio) WriteTo(w io.Writer) (rv int64, err error) { b.op_mtx.Lock() defer b.op_mtx.Unlock() // write whatever data we currently have b.data_mtx.Lock() data := b.buf b.data_mtx.Unlock() if len(data) == 0 { return 0, nil } n, err := w.Write(data) // subtract however much data we wrote from the buffer b.data_mtx.Lock() b.buf = b.buf[:copy(b.buf, b.buf[n:])] if b.release_buffers && len(b.buf) == 0 { b.buf = nil } b.data_mtx.Unlock() return int64(n), err } func (self *writeBio) Disconnect(b *C.BIO) { if loadWritePtr(b) == self { b.ptr = nil } } func (b *writeBio) MakeCBIO() *C.BIO { rv := C.BIO_new(C.BIO_s_writeBio()) rv.ptr = unsafe.Pointer(b) return rv } type readBio struct { data_mtx sync.Mutex op_mtx sync.Mutex buf []byte eof bool release_buffers bool } func loadReadPtr(b *C.BIO) *readBio { return (*readBio)(unsafe.Pointer(b.ptr)) } //export readBioRead func readBioRead(b *C.BIO, data *C.char, size C.int) (rc C.int) { defer func() { if err := recover(); err != nil { logger.Critf("openssl: readBioRead panic'd: %v", err) rc = -1 } }() ptr := loadReadPtr(b) if ptr == nil || size < 0 { return -1 } ptr.data_mtx.Lock() defer ptr.data_mtx.Unlock() bioClearRetryFlags(b) if len(ptr.buf) == 0 { if ptr.eof { return 0 } bioSetRetryRead(b) return -1 } if size == 0 || data == nil { return C.int(len(ptr.buf)) } n := copy(nonCopyCString(data, size), ptr.buf) ptr.buf = ptr.buf[:copy(ptr.buf, ptr.buf[n:])] if ptr.release_buffers && len(ptr.buf) == 0 { ptr.buf = nil } return C.int(n) } //export readBioCtrl func readBioCtrl(b *C.BIO, cmd C.int, arg1 C.long, arg2 unsafe.Pointer) ( rc C.long) { defer func() { if err := recover(); err != nil { logger.Critf("openssl: readBioCtrl panic'd: %v", err) rc = -1 } }() switch cmd { case C.BIO_CTRL_PENDING: return readBioPending(b) case C.BIO_CTRL_DUP, C.BIO_CTRL_FLUSH: return 1 default: return 0 } } func readBioPending(b *C.BIO) C.long { ptr := loadReadPtr(b) if ptr == nil { return 0 } ptr.data_mtx.Lock() defer ptr.data_mtx.Unlock() return C.long(len(ptr.buf)) } func (b *readBio) ReadFromOnce(r io.Reader) (n int, err error) { b.op_mtx.Lock() defer b.op_mtx.Unlock() // make sure we have a destination that fits at least one SSL record b.data_mtx.Lock() if cap(b.buf) < len(b.buf)+SSLRecordSize { new_buf := make([]byte, len(b.buf), len(b.buf)+SSLRecordSize) copy(new_buf, b.buf) b.buf = new_buf } dst := b.buf[len(b.buf):cap(b.buf)] dst_slice := b.buf b.data_mtx.Unlock() n, err = r.Read(dst) b.data_mtx.Lock() defer b.data_mtx.Unlock() if n > 0 { if len(dst_slice) != len(b.buf) { // someone shrunk the buffer, so we read in too far ahead and we // need to slide backwards copy(b.buf[len(b.buf):len(b.buf)+n], dst) } b.buf = b.buf[:len(b.buf)+n] } return n, err } func (b *readBio) MakeCBIO() *C.BIO { rv := C.BIO_new(C.BIO_s_readBio()) rv.ptr = unsafe.Pointer(b) return rv } func (self *readBio) Disconnect(b *C.BIO) { if loadReadPtr(b) == self { b.ptr = nil } } func (b *readBio) MarkEOF() { b.data_mtx.Lock() defer b.data_mtx.Unlock() b.eof = true } type anyBio C.BIO func asAnyBio(b *C.BIO) *anyBio { return (*anyBio)(b) } func (b *anyBio) Read(buf []byte) (n int, err error) { if len(buf) == 0 { return 0, nil } n = int(C.BIO_read((*C.BIO)(b), unsafe.Pointer(&buf[0]), C.int(len(buf)))) if n <= 0 { return 0, io.EOF } return n, nil } func (b *anyBio) Write(buf []byte) (written int, err error) { if len(buf) == 0 { return 0, nil } n := int(C.BIO_write((*C.BIO)(b), unsafe.Pointer(&buf[0]), C.int(len(buf)))) if n != len(buf) { return n, errors.New("BIO write failed") } return n, nil } mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/openssl/build.go000066400000000000000000000014541276133741700263540ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // +build cgo package openssl // #cgo linux pkg-config: openssl // #cgo windows CFLAGS: -DWIN32_LEAN_AND_MEAN // #cgo darwin CFLAGS: -Wno-deprecated-declarations // #cgo darwin LDFLAGS: -lssl -lcrypto import "C" mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/openssl/cert.go000066400000000000000000000252621276133741700262150ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // +build cgo package openssl // #include // #include // #include // // void OPENSSL_free_not_a_macro(void *ref) { OPENSSL_free(ref); } // import "C" import ( "errors" "io/ioutil" "math/big" "runtime" "time" "unsafe" ) type EVP_MD int const ( EVP_NULL EVP_MD = iota EVP_MD5 EVP_MD = iota EVP_SHA EVP_MD = iota EVP_SHA1 EVP_MD = iota EVP_DSS EVP_MD = iota EVP_DSS1 EVP_MD = iota EVP_MDC2 EVP_MD = iota EVP_RIPEMD160 EVP_MD = iota EVP_SHA224 EVP_MD = iota EVP_SHA256 EVP_MD = iota EVP_SHA384 EVP_MD = iota EVP_SHA512 EVP_MD = iota ) type Certificate struct { x *C.X509 Issuer *Certificate ref interface{} pubKey PublicKey } type CertificateInfo struct { Serial *big.Int Issued time.Duration Expires time.Duration Country string Organization string CommonName string } type Name struct { name *C.X509_NAME } // Allocate and return a new Name object. func NewName() (*Name, error) { n := C.X509_NAME_new() if n == nil { return nil, errors.New("could not create x509 name") } name := &Name{name: n} runtime.SetFinalizer(name, func(n *Name) { C.X509_NAME_free(n.name) }) return name, nil } // AddTextEntry appends a text entry to an X509 NAME. func (n *Name) AddTextEntry(field, value string) error { cfield := C.CString(field) defer C.free(unsafe.Pointer(cfield)) cvalue := (*C.uchar)(unsafe.Pointer(C.CString(value))) defer C.free(unsafe.Pointer(cvalue)) ret := C.X509_NAME_add_entry_by_txt( n.name, cfield, C.MBSTRING_ASC, cvalue, -1, -1, 0) if ret != 1 { return errors.New("failed to add x509 name text entry") } return nil } // AddTextEntries allows adding multiple entries to a name in one call. func (n *Name) AddTextEntries(entries map[string]string) error { for f, v := range entries { if err := n.AddTextEntry(f, v); err != nil { return err } } return nil } // GetEntry returns a name entry based on NID. If no entry, then ("", false) is // returned. func (n *Name) GetEntry(nid NID) (entry string, ok bool) { entrylen := C.X509_NAME_get_text_by_NID(n.name, C.int(nid), nil, 0) if entrylen == -1 { return "", false } buf := (*C.char)(C.malloc(C.size_t(entrylen + 1))) defer C.free(unsafe.Pointer(buf)) C.X509_NAME_get_text_by_NID(n.name, C.int(nid), buf, entrylen+1) return C.GoStringN(buf, entrylen), true } // NewCertificate generates a basic certificate based // on the provided CertificateInfo struct func NewCertificate(info *CertificateInfo, key PublicKey) (*Certificate, error) { c := &Certificate{x: C.X509_new()} runtime.SetFinalizer(c, func(c *Certificate) { C.X509_free(c.x) }) name, err := c.GetSubjectName() if err != nil { return nil, err } err = name.AddTextEntries(map[string]string{ "C": info.Country, "O": info.Organization, "CN": info.CommonName, }) if err != nil { return nil, err } // self-issue for now if err := c.SetIssuerName(name); err != nil { return nil, err } if err := c.SetSerial(info.Serial); err != nil { return nil, err } if err := c.SetIssueDate(info.Issued); err != nil { return nil, err } if err := c.SetExpireDate(info.Expires); err != nil { return nil, err } if err := c.SetPubKey(key); err != nil { return nil, err } return c, nil } func (c *Certificate) GetSubjectName() (*Name, error) { n := C.X509_get_subject_name(c.x) if n == nil { return nil, errors.New("failed to get subject name") } return &Name{name: n}, nil } func (c *Certificate) GetIssuerName() (*Name, error) { n := C.X509_get_issuer_name(c.x) if n == nil { return nil, errors.New("failed to get issuer name") } return &Name{name: n}, nil } func (c *Certificate) SetSubjectName(name *Name) error { if C.X509_set_subject_name(c.x, name.name) != 1 { return errors.New("failed to set subject name") } return nil } // SetIssuer updates the stored Issuer cert // and the internal x509 Issuer Name of a certificate. // The stored Issuer reference is used when adding extensions. func (c *Certificate) SetIssuer(issuer *Certificate) error { name, err := issuer.GetSubjectName() if err != nil { return err } if err = c.SetIssuerName(name); err != nil { return err } c.Issuer = issuer return nil } // SetIssuerName populates the issuer name of a certificate. // Use SetIssuer instead, if possible. func (c *Certificate) SetIssuerName(name *Name) error { if C.X509_set_issuer_name(c.x, name.name) != 1 { return errors.New("failed to set subject name") } return nil } // SetSerial sets the serial of a certificate. func (c *Certificate) SetSerial(serial *big.Int) error { sno := C.ASN1_INTEGER_new() defer C.ASN1_INTEGER_free(sno) bn := C.BN_new() defer C.BN_free(bn) serialBytes := serial.Bytes() if bn = C.BN_bin2bn((*C.uchar)(unsafe.Pointer(&serialBytes[0])), C.int(len(serialBytes)), bn); bn == nil { return errors.New("failed to set serial") } if sno = C.BN_to_ASN1_INTEGER(bn, sno); sno == nil { return errors.New("failed to set serial") } if C.X509_set_serialNumber(c.x, sno) != 1 { return errors.New("failed to set serial") } return nil } // SetIssueDate sets the certificate issue date relative to the current time. func (c *Certificate) SetIssueDate(when time.Duration) error { offset := C.long(when / time.Second) result := C.X509_gmtime_adj(c.x.cert_info.validity.notBefore, offset) if result == nil { return errors.New("failed to set issue date") } return nil } // SetExpireDate sets the certificate issue date relative to the current time. func (c *Certificate) SetExpireDate(when time.Duration) error { offset := C.long(when / time.Second) result := C.X509_gmtime_adj(c.x.cert_info.validity.notAfter, offset) if result == nil { return errors.New("failed to set expire date") } return nil } // SetPubKey assigns a new public key to a certificate. func (c *Certificate) SetPubKey(pubKey PublicKey) error { c.pubKey = pubKey if C.X509_set_pubkey(c.x, pubKey.evpPKey()) != 1 { return errors.New("failed to set public key") } return nil } // Sign a certificate using a private key and a digest name. // Accepted digest names are 'sha256', 'sha384', and 'sha512'. func (c *Certificate) Sign(privKey PrivateKey, digest EVP_MD) error { switch digest { case EVP_SHA256: case EVP_SHA384: case EVP_SHA512: default: return errors.New("Unsupported digest" + "You're probably looking for 'EVP_SHA256' or 'EVP_SHA512'.") } return c.insecureSign(privKey, digest) } func (c *Certificate) insecureSign(privKey PrivateKey, digest EVP_MD) error { var md *C.EVP_MD switch digest { // please don't use these digest functions case EVP_NULL: md = C.EVP_md_null() case EVP_MD5: md = C.EVP_md5() case EVP_SHA: md = C.EVP_sha() case EVP_SHA1: md = C.EVP_sha1() case EVP_DSS: md = C.EVP_dss() case EVP_DSS1: md = C.EVP_dss1() case EVP_RIPEMD160: md = C.EVP_ripemd160() case EVP_SHA224: md = C.EVP_sha224() // you actually want one of these case EVP_SHA256: md = C.EVP_sha256() case EVP_SHA384: md = C.EVP_sha384() case EVP_SHA512: md = C.EVP_sha512() } if C.X509_sign(c.x, privKey.evpPKey(), md) <= 0 { return errors.New("failed to sign certificate") } return nil } // Add an extension to a certificate. // Extension constants are NID_* as found in openssl. func (c *Certificate) AddExtension(nid NID, value string) error { issuer := c if c.Issuer != nil { issuer = c.Issuer } var ctx C.X509V3_CTX C.X509V3_set_ctx(&ctx, c.x, issuer.x, nil, nil, 0) ex := C.X509V3_EXT_conf_nid(nil, &ctx, C.int(nid), C.CString(value)) if ex == nil { return errors.New("failed to create x509v3 extension") } defer C.X509_EXTENSION_free(ex) if C.X509_add_ext(c.x, ex, -1) <= 0 { return errors.New("failed to add x509v3 extension") } return nil } // Wraps AddExtension using a map of NID to text extension. // Will return without finishing if it encounters an error. func (c *Certificate) AddExtensions(extensions map[NID]string) error { for nid, value := range extensions { if err := c.AddExtension(nid, value); err != nil { return err } } return nil } // LoadCertificateFromPEM loads an X509 certificate from a PEM-encoded block. func LoadCertificateFromPEM(pem_block []byte) (*Certificate, error) { if len(pem_block) == 0 { return nil, errors.New("empty pem block") } runtime.LockOSThread() defer runtime.UnlockOSThread() bio := C.BIO_new_mem_buf(unsafe.Pointer(&pem_block[0]), C.int(len(pem_block))) cert := C.PEM_read_bio_X509(bio, nil, nil, nil) C.BIO_free(bio) if cert == nil { return nil, errorFromErrorQueue() } x := &Certificate{x: cert} runtime.SetFinalizer(x, func(x *Certificate) { C.X509_free(x.x) }) return x, nil } // MarshalPEM converts the X509 certificate to PEM-encoded format func (c *Certificate) MarshalPEM() (pem_block []byte, err error) { bio := C.BIO_new(C.BIO_s_mem()) if bio == nil { return nil, errors.New("failed to allocate memory BIO") } defer C.BIO_free(bio) if int(C.PEM_write_bio_X509(bio, c.x)) != 1 { return nil, errors.New("failed dumping certificate") } return ioutil.ReadAll(asAnyBio(bio)) } // PublicKey returns the public key embedded in the X509 certificate. func (c *Certificate) PublicKey() (PublicKey, error) { pkey := C.X509_get_pubkey(c.x) if pkey == nil { return nil, errors.New("no public key found") } key := &pKey{key: pkey} runtime.SetFinalizer(key, func(key *pKey) { C.EVP_PKEY_free(key.key) }) return key, nil } // GetSerialNumberHex returns the certificate's serial number in hex format func (c *Certificate) GetSerialNumberHex() (serial string) { asn1_i := C.X509_get_serialNumber(c.x) bignum := C.ASN1_INTEGER_to_BN(asn1_i, nil) hex := C.BN_bn2hex(bignum) serial = C.GoString(hex) C.BN_free(bignum) C.OPENSSL_free_not_a_macro(unsafe.Pointer(hex)) return } func (c *Certificate) X509NamePrintEx() (out []byte, err error) { bio := C.BIO_new(C.BIO_s_mem()) if bio == nil { return nil, errors.New("failed to allocate memory BIO") } defer C.BIO_free(bio) name := C.X509_get_subject_name(c.x) // TODO, pass in flags instead of using this hardcoded one if int(C.X509_NAME_print_ex(bio, name, 0, C.XN_FLAG_RFC2253)) < 0 { return nil, errors.New("failed formatting subject") } return ioutil.ReadAll(asAnyBio(bio)) } mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/openssl/cert_test.go000066400000000000000000000063261276133741700272540ustar00rootroot00000000000000// Copyright (C) 2014 Ryan Hileman // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. package openssl import ( "math/big" "testing" "time" ) func TestCertGenerate(t *testing.T) { key, err := GenerateRSAKey(2048) if err != nil { t.Fatal(err) } info := &CertificateInfo{ Serial: big.NewInt(int64(1)), Issued: 0, Expires: 24 * time.Hour, Country: "US", Organization: "Test", CommonName: "localhost", } cert, err := NewCertificate(info, key) if err != nil { t.Fatal(err) } if err := cert.Sign(key, EVP_SHA256); err != nil { t.Fatal(err) } } func TestCAGenerate(t *testing.T) { cakey, err := GenerateRSAKey(2048) if err != nil { t.Fatal(err) } info := &CertificateInfo{ Serial: big.NewInt(int64(1)), Issued: 0, Expires: 24 * time.Hour, Country: "US", Organization: "Test CA", CommonName: "CA", } ca, err := NewCertificate(info, cakey) if err != nil { t.Fatal(err) } if err := ca.AddExtensions(map[NID]string{ NID_basic_constraints: "critical,CA:TRUE", NID_key_usage: "critical,keyCertSign,cRLSign", NID_subject_key_identifier: "hash", NID_netscape_cert_type: "sslCA", }); err != nil { t.Fatal(err) } if err := ca.Sign(cakey, EVP_SHA256); err != nil { t.Fatal(err) } key, err := GenerateRSAKey(2048) if err != nil { t.Fatal(err) } info = &CertificateInfo{ Serial: big.NewInt(int64(1)), Issued: 0, Expires: 24 * time.Hour, Country: "US", Organization: "Test", CommonName: "localhost", } cert, err := NewCertificate(info, key) if err != nil { t.Fatal(err) } if err := cert.AddExtensions(map[NID]string{ NID_basic_constraints: "critical,CA:FALSE", NID_key_usage: "keyEncipherment", NID_ext_key_usage: "serverAuth", }); err != nil { t.Fatal(err) } if err := cert.SetIssuer(ca); err != nil { t.Fatal(err) } if err := cert.Sign(cakey, EVP_SHA256); err != nil { t.Fatal(err) } } func TestCertGetNameEntry(t *testing.T) { key, err := GenerateRSAKey(2048) if err != nil { t.Fatal(err) } info := &CertificateInfo{ Serial: big.NewInt(int64(1)), Issued: 0, Expires: 24 * time.Hour, Country: "US", Organization: "Test", CommonName: "localhost", } cert, err := NewCertificate(info, key) if err != nil { t.Fatal(err) } name, err := cert.GetSubjectName() if err != nil { t.Fatal(err) } entry, ok := name.GetEntry(NID_commonName) if !ok { t.Fatal("no common name") } if entry != "localhost" { t.Fatalf("expected localhost; got %q", entry) } entry, ok = name.GetEntry(NID_localityName) if ok { t.Fatal("did not expect a locality name") } if entry != "" { t.Fatalf("entry should be empty; got %q", entry) } } mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/openssl/ciphers.go000066400000000000000000000217661276133741700267220ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // +build cgo package openssl // #include // // int EVP_CIPHER_block_size_not_a_macro(EVP_CIPHER *c) { // return EVP_CIPHER_block_size(c); // } // // int EVP_CIPHER_key_length_not_a_macro(EVP_CIPHER *c) { // return EVP_CIPHER_key_length(c); // } // // int EVP_CIPHER_iv_length_not_a_macro(EVP_CIPHER *c) { // return EVP_CIPHER_iv_length(c); // } // // int EVP_CIPHER_nid_not_a_macro(EVP_CIPHER *c) { // return EVP_CIPHER_nid(c); // } // // int EVP_CIPHER_CTX_block_size_not_a_macro(EVP_CIPHER_CTX *ctx) { // return EVP_CIPHER_CTX_block_size(ctx); // } // // int EVP_CIPHER_CTX_key_length_not_a_macro(EVP_CIPHER_CTX *ctx) { // return EVP_CIPHER_CTX_key_length(ctx); // } // // int EVP_CIPHER_CTX_iv_length_not_a_macro(EVP_CIPHER_CTX *ctx) { // return EVP_CIPHER_CTX_iv_length(ctx); // } // // const EVP_CIPHER *EVP_CIPHER_CTX_cipher_not_a_macro(EVP_CIPHER_CTX *ctx) { // return EVP_CIPHER_CTX_cipher(ctx); // } import "C" import ( "errors" "fmt" "runtime" "unsafe" ) const ( GCM_TAG_MAXLEN = 16 ) type CipherCtx interface { Cipher() *Cipher BlockSize() int KeySize() int IVSize() int } type Cipher struct { ptr *C.EVP_CIPHER } func (c *Cipher) Nid() NID { return NID(C.EVP_CIPHER_nid_not_a_macro(c.ptr)) } func (c *Cipher) ShortName() (string, error) { return Nid2ShortName(c.Nid()) } func (c *Cipher) BlockSize() int { return int(C.EVP_CIPHER_block_size_not_a_macro(c.ptr)) } func (c *Cipher) KeySize() int { return int(C.EVP_CIPHER_key_length_not_a_macro(c.ptr)) } func (c *Cipher) IVSize() int { return int(C.EVP_CIPHER_iv_length_not_a_macro(c.ptr)) } func Nid2ShortName(nid NID) (string, error) { sn := C.OBJ_nid2sn(C.int(nid)) if sn == nil { return "", fmt.Errorf("NID %d not found", nid) } return C.GoString(sn), nil } func GetCipherByName(name string) (*Cipher, error) { cname := C.CString(name) defer C.free(unsafe.Pointer(cname)) p := C.EVP_get_cipherbyname(cname) if p == nil { return nil, fmt.Errorf("Cipher %v not found", name) } // we can consider ciphers to use static mem; don't need to free return &Cipher{ptr: p}, nil } func GetCipherByNid(nid NID) (*Cipher, error) { sn, err := Nid2ShortName(nid) if err != nil { return nil, err } return GetCipherByName(sn) } type cipherCtx struct { ctx *C.EVP_CIPHER_CTX } func newCipherCtx() (*cipherCtx, error) { cctx := C.EVP_CIPHER_CTX_new() if cctx == nil { return nil, errors.New("failed to allocate cipher context") } ctx := &cipherCtx{cctx} runtime.SetFinalizer(ctx, func(ctx *cipherCtx) { C.EVP_CIPHER_CTX_free(ctx.ctx) }) return ctx, nil } func (ctx *cipherCtx) applyKeyAndIV(key, iv []byte) error { var kptr, iptr *C.uchar if key != nil { if len(key) != ctx.KeySize() { return fmt.Errorf("bad key size (%d bytes instead of %d)", len(key), ctx.KeySize()) } kptr = (*C.uchar)(&key[0]) } if iv != nil { if len(iv) != ctx.IVSize() { return fmt.Errorf("bad IV size (%d bytes instead of %d)", len(iv), ctx.IVSize()) } iptr = (*C.uchar)(&iv[0]) } if kptr != nil || iptr != nil { var res C.int if ctx.ctx.encrypt != 0 { res = C.EVP_EncryptInit_ex(ctx.ctx, nil, nil, kptr, iptr) } else { res = C.EVP_DecryptInit_ex(ctx.ctx, nil, nil, kptr, iptr) } if 1 != res { return errors.New("failed to apply key/IV") } } return nil } func (ctx *cipherCtx) Cipher() *Cipher { return &Cipher{ptr: C.EVP_CIPHER_CTX_cipher_not_a_macro(ctx.ctx)} } func (ctx *cipherCtx) BlockSize() int { return int(C.EVP_CIPHER_CTX_block_size_not_a_macro(ctx.ctx)) } func (ctx *cipherCtx) KeySize() int { return int(C.EVP_CIPHER_CTX_key_length_not_a_macro(ctx.ctx)) } func (ctx *cipherCtx) IVSize() int { return int(C.EVP_CIPHER_CTX_iv_length_not_a_macro(ctx.ctx)) } func (ctx *cipherCtx) setCtrl(code, arg int) error { res := C.EVP_CIPHER_CTX_ctrl(ctx.ctx, C.int(code), C.int(arg), nil) if res != 1 { return fmt.Errorf("failed to set code %d to %d [result %d]", code, arg, res) } return nil } func (ctx *cipherCtx) setCtrlBytes(code, arg int, value []byte) error { res := C.EVP_CIPHER_CTX_ctrl(ctx.ctx, C.int(code), C.int(arg), unsafe.Pointer(&value[0])) if res != 1 { return fmt.Errorf("failed to set code %d with arg %d to %x [result %d]", code, arg, value, res) } return nil } func (ctx *cipherCtx) getCtrlInt(code, arg int) (int, error) { var returnVal C.int res := C.EVP_CIPHER_CTX_ctrl(ctx.ctx, C.int(code), C.int(arg), unsafe.Pointer(&returnVal)) if res != 1 { return 0, fmt.Errorf("failed to get code %d with arg %d [result %d]", code, arg, res) } return int(returnVal), nil } func (ctx *cipherCtx) getCtrlBytes(code, arg, expectsize int) ([]byte, error) { returnVal := make([]byte, expectsize) res := C.EVP_CIPHER_CTX_ctrl(ctx.ctx, C.int(code), C.int(arg), unsafe.Pointer(&returnVal[0])) if res != 1 { return nil, fmt.Errorf("failed to get code %d with arg %d [result %d]", code, arg, res) } return returnVal, nil } type EncryptionCipherCtx interface { CipherCtx // pass in plaintext, get back ciphertext. can be called // multiple times as needed EncryptUpdate(input []byte) ([]byte, error) // call after all plaintext has been passed in; may return // additional ciphertext if needed to finish off a block // or extra padding information EncryptFinal() ([]byte, error) } type DecryptionCipherCtx interface { CipherCtx // pass in ciphertext, get back plaintext. can be called // multiple times as needed DecryptUpdate(input []byte) ([]byte, error) // call after all ciphertext has been passed in; may return // additional plaintext if needed to finish off a block DecryptFinal() ([]byte, error) } type encryptionCipherCtx struct { *cipherCtx } type decryptionCipherCtx struct { *cipherCtx } func newEncryptionCipherCtx(c *Cipher, e *Engine, key, iv []byte) ( *encryptionCipherCtx, error) { if c == nil { return nil, errors.New("null cipher not allowed") } ctx, err := newCipherCtx() if err != nil { return nil, err } var eptr *C.ENGINE if e != nil { eptr = e.e } if 1 != C.EVP_EncryptInit_ex(ctx.ctx, c.ptr, eptr, nil, nil) { return nil, errors.New("failed to initialize cipher context") } err = ctx.applyKeyAndIV(key, iv) if err != nil { return nil, err } return &encryptionCipherCtx{cipherCtx: ctx}, nil } func newDecryptionCipherCtx(c *Cipher, e *Engine, key, iv []byte) ( *decryptionCipherCtx, error) { if c == nil { return nil, errors.New("null cipher not allowed") } ctx, err := newCipherCtx() if err != nil { return nil, err } var eptr *C.ENGINE if e != nil { eptr = e.e } if 1 != C.EVP_DecryptInit_ex(ctx.ctx, c.ptr, eptr, nil, nil) { return nil, errors.New("failed to initialize cipher context") } err = ctx.applyKeyAndIV(key, iv) if err != nil { return nil, err } return &decryptionCipherCtx{cipherCtx: ctx}, nil } func NewEncryptionCipherCtx(c *Cipher, e *Engine, key, iv []byte) ( EncryptionCipherCtx, error) { return newEncryptionCipherCtx(c, e, key, iv) } func NewDecryptionCipherCtx(c *Cipher, e *Engine, key, iv []byte) ( DecryptionCipherCtx, error) { return newDecryptionCipherCtx(c, e, key, iv) } func (ctx *encryptionCipherCtx) EncryptUpdate(input []byte) ([]byte, error) { outbuf := make([]byte, len(input)+ctx.BlockSize()) outlen := C.int(len(outbuf)) res := C.EVP_EncryptUpdate(ctx.ctx, (*C.uchar)(&outbuf[0]), &outlen, (*C.uchar)(&input[0]), C.int(len(input))) if res != 1 { return nil, fmt.Errorf("failed to encrypt [result %d]", res) } return outbuf[:outlen], nil } func (ctx *decryptionCipherCtx) DecryptUpdate(input []byte) ([]byte, error) { outbuf := make([]byte, len(input)+ctx.BlockSize()) outlen := C.int(len(outbuf)) res := C.EVP_DecryptUpdate(ctx.ctx, (*C.uchar)(&outbuf[0]), &outlen, (*C.uchar)(&input[0]), C.int(len(input))) if res != 1 { return nil, fmt.Errorf("failed to decrypt [result %d]", res) } return outbuf[:outlen], nil } func (ctx *encryptionCipherCtx) EncryptFinal() ([]byte, error) { outbuf := make([]byte, ctx.BlockSize()) var outlen C.int if 1 != C.EVP_EncryptFinal_ex(ctx.ctx, (*C.uchar)(&outbuf[0]), &outlen) { return nil, errors.New("encryption failed") } return outbuf[:outlen], nil } func (ctx *decryptionCipherCtx) DecryptFinal() ([]byte, error) { outbuf := make([]byte, ctx.BlockSize()) var outlen C.int if 1 != C.EVP_DecryptFinal_ex(ctx.ctx, (*C.uchar)(&outbuf[0]), &outlen) { // this may mean the tag failed to verify- all previous plaintext // returned must be considered faked and invalid return nil, errors.New("decryption failed") } return outbuf[:outlen], nil } mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/openssl/ciphers_test.go000066400000000000000000000217631276133741700277560ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // +build !darwin package openssl import ( "bytes" "fmt" "strings" "testing" ) func expectError(t *testing.T, err error, msg string) { if err == nil { t.Fatalf("Expected error containing %#v, but got none", msg) } if !strings.Contains(err.Error(), msg) { t.Fatalf("Expected error containing %#v, but got %s", msg, err) } } func TestBadInputs(t *testing.T) { _, err := NewGCMEncryptionCipherCtx(256, nil, []byte("abcdefghijklmnopqrstuvwxyz"), nil) expectError(t, err, "bad key size") _, err = NewGCMEncryptionCipherCtx(128, nil, []byte("abcdefghijklmnopqrstuvwxyz"), nil) expectError(t, err, "bad key size") _, err = NewGCMEncryptionCipherCtx(200, nil, []byte("abcdefghijklmnopqrstuvwxy"), nil) expectError(t, err, "unknown block size") c, err := GetCipherByName("AES-128-CBC") if err != nil { t.Fatal("Could not look up AES-128-CBC") } _, err = NewEncryptionCipherCtx(c, nil, []byte("abcdefghijklmnop"), []byte("abc")) expectError(t, err, "bad IV size") } func doEncryption(key, iv, aad, plaintext []byte, blocksize, bufsize int) ( ciphertext, tag []byte, err error) { ectx, err := NewGCMEncryptionCipherCtx(blocksize, nil, key, iv) if err != nil { return nil, nil, fmt.Errorf("Failed making GCM encryption ctx: %s", err) } err = ectx.ExtraData(aad) if err != nil { return nil, nil, fmt.Errorf("Failed to add authenticated data: %s", err) } plainb := bytes.NewBuffer(plaintext) cipherb := new(bytes.Buffer) for plainb.Len() > 0 { moar, err := ectx.EncryptUpdate(plainb.Next(bufsize)) if err != nil { return nil, nil, fmt.Errorf("Failed to perform an encryption: %s", err) } cipherb.Write(moar) } moar, err := ectx.EncryptFinal() if err != nil { return nil, nil, fmt.Errorf("Failed to finalize encryption: %s", err) } cipherb.Write(moar) tag, err = ectx.GetTag() if err != nil { return nil, nil, fmt.Errorf("Failed to get GCM tag: %s", err) } return cipherb.Bytes(), tag, nil } func doDecryption(key, iv, aad, ciphertext, tag []byte, blocksize, bufsize int) (plaintext []byte, err error) { dctx, err := NewGCMDecryptionCipherCtx(blocksize, nil, key, iv) if err != nil { return nil, fmt.Errorf("Failed making GCM decryption ctx: %s", err) } aadbuf := bytes.NewBuffer(aad) for aadbuf.Len() > 0 { err = dctx.ExtraData(aadbuf.Next(bufsize)) if err != nil { return nil, fmt.Errorf("Failed to add authenticated data: %s", err) } } plainb := new(bytes.Buffer) cipherb := bytes.NewBuffer(ciphertext) for cipherb.Len() > 0 { moar, err := dctx.DecryptUpdate(cipherb.Next(bufsize)) if err != nil { return nil, fmt.Errorf("Failed to perform a decryption: %s", err) } plainb.Write(moar) } err = dctx.SetTag(tag) if err != nil { return nil, fmt.Errorf("Failed to set expected GCM tag: %s", err) } moar, err := dctx.DecryptFinal() if err != nil { return nil, fmt.Errorf("Failed to finalize decryption: %s", err) } plainb.Write(moar) return plainb.Bytes(), nil } func checkEqual(t *testing.T, output []byte, original string) { output_s := string(output) if output_s != original { t.Fatalf("output != original! %#v != %#v", output_s, original) } } func TestGCM(t *testing.T) { aad := []byte("foo bar baz") key := []byte("nobody can guess this i'm sure..") // len=32 iv := []byte("just a bunch of bytes") plaintext := "Long long ago, in a land far away..." blocksizes_to_test := []int{256, 192, 128} // best for this to have no common factors with blocksize, so that the // buffering layer inside the CIPHER_CTX gets exercised bufsize := 33 if len(plaintext)%8 == 0 { plaintext += "!" // make sure padding is exercised } for _, bsize := range blocksizes_to_test { subkey := key[:bsize/8] ciphertext, tag, err := doEncryption(subkey, iv, aad, []byte(plaintext), bsize, bufsize) if err != nil { t.Fatalf("Encryption with b=%d: %s", bsize, err) } plaintext_out, err := doDecryption(subkey, iv, aad, ciphertext, tag, bsize, bufsize) if err != nil { t.Fatalf("Decryption with b=%d: %s", bsize, err) } checkEqual(t, plaintext_out, plaintext) } } func TestGCMWithNoAAD(t *testing.T) { key := []byte("0000111122223333") iv := []byte("9999") plaintext := "ABORT ABORT ABORT DANGAR" ciphertext, tag, err := doEncryption(key, iv, nil, []byte(plaintext), 128, 32) if err != nil { t.Fatal("Encryption failure:", err) } plaintext_out, err := doDecryption(key, iv, nil, ciphertext, tag, 128, 129) if err != nil { t.Fatal("Decryption failure:", err) } checkEqual(t, plaintext_out, plaintext) } func TestBadTag(t *testing.T) { key := []byte("abcdefghijklmnop") iv := []byte("v7239qjfv3qr793fuaj") plaintext := "The red rooster has flown the coop I REPEAT" + "the red rooster has flown the coop!!1!" ciphertext, tag, err := doEncryption(key, iv, nil, []byte(plaintext), 128, 32) if err != nil { t.Fatal("Encryption failure:", err) } // flip the last bit tag[len(tag)-1] ^= 1 plaintext_out, err := doDecryption(key, iv, nil, ciphertext, tag, 128, 129) if err == nil { t.Fatal("Expected error for bad tag, but got none") } // flip it back, try again just to make sure tag[len(tag)-1] ^= 1 plaintext_out, err = doDecryption(key, iv, nil, ciphertext, tag, 128, 129) if err != nil { t.Fatal("Decryption failure:", err) } checkEqual(t, plaintext_out, plaintext) } func TestBadCiphertext(t *testing.T) { key := []byte("hard boiled eggs & bacon") iv := []byte("x") // it's not a very /good/ IV, is it aad := []byte("mu") plaintext := "Roger roger bingo charlie, we have a niner fourteen tango" ciphertext, tag, err := doEncryption(key, iv, aad, []byte(plaintext), 192, 1) if err != nil { t.Fatal("Encryption failure:", err) } // flip the last bit ciphertext[len(ciphertext)-1] ^= 1 plaintext_out, err := doDecryption(key, iv, aad, ciphertext, tag, 192, 192) if err == nil { t.Fatal("Expected error for bad ciphertext, but got none") } // flip it back, try again just to make sure ciphertext[len(ciphertext)-1] ^= 1 plaintext_out, err = doDecryption(key, iv, aad, ciphertext, tag, 192, 192) if err != nil { t.Fatal("Decryption failure:", err) } checkEqual(t, plaintext_out, plaintext) } func TestBadAAD(t *testing.T) { key := []byte("Ive got a lovely buncha coconuts") iv := []byte("aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab") aad := []byte("Hi i am a plain") plaintext := "Whatever." ciphertext, tag, err := doEncryption(key, iv, aad, []byte(plaintext), 256, 256) if err != nil { t.Fatal("Encryption failure:", err) } // flip the last bit aad[len(aad)-1] ^= 1 plaintext_out, err := doDecryption(key, iv, aad, ciphertext, tag, 256, 256) if err == nil { t.Fatal("Expected error for bad AAD, but got none") } // flip it back, try again just to make sure aad[len(aad)-1] ^= 1 plaintext_out, err = doDecryption(key, iv, aad, ciphertext, tag, 256, 256) if err != nil { t.Fatal("Decryption failure:", err) } checkEqual(t, plaintext_out, plaintext) } func TestNonAuthenticatedEncryption(t *testing.T) { key := []byte("never gonna give you up, never g") iv := []byte("onna let you dow") plaintext1 := "n, never gonna run around" plaintext2 := " and desert you" cipher, err := GetCipherByName("aes-256-cbc") if err != nil { t.Fatal("Could not get cipher: ", err) } eCtx, err := NewEncryptionCipherCtx(cipher, nil, key, iv) if err != nil { t.Fatal("Could not create encryption context: ", err) } cipherbytes, err := eCtx.EncryptUpdate([]byte(plaintext1)) if err != nil { t.Fatal("EncryptUpdate(plaintext1) failure: ", err) } ciphertext := string(cipherbytes) cipherbytes, err = eCtx.EncryptUpdate([]byte(plaintext2)) if err != nil { t.Fatal("EncryptUpdate(plaintext2) failure: ", err) } ciphertext += string(cipherbytes) cipherbytes, err = eCtx.EncryptFinal() if err != nil { t.Fatal("EncryptFinal() failure: ", err) } ciphertext += string(cipherbytes) dCtx, err := NewDecryptionCipherCtx(cipher, nil, key, iv) if err != nil { t.Fatal("Could not create decryption context: ", err) } plainbytes, err := dCtx.DecryptUpdate([]byte(ciphertext[:15])) if err != nil { t.Fatal("DecryptUpdate(ciphertext part 1) failure: ", err) } plainOutput := string(plainbytes) plainbytes, err = dCtx.DecryptUpdate([]byte(ciphertext[15:])) if err != nil { t.Fatal("DecryptUpdate(ciphertext part 2) failure: ", err) } plainOutput += string(plainbytes) plainbytes, err = dCtx.DecryptFinal() if err != nil { t.Fatal("DecryptFinal() failure: ", err) } plainOutput += string(plainbytes) checkEqual(t, []byte(plainOutput), plaintext1+plaintext2) } mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/openssl/conn.go000066400000000000000000000437321276133741700262170ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // +build cgo package openssl /* #include #include #include #include int sk_X509_num_not_a_macro(STACK_OF(X509) *sk) { return sk_X509_num(sk); } X509 *sk_X509_value_not_a_macro(STACK_OF(X509)* sk, int i) { return sk_X509_value(sk, i); } const char * SSL_get_cipher_name_not_a_macro(const SSL *ssl) { return SSL_get_cipher_name(ssl); } static int SSL_session_reused_not_a_macro(SSL *ssl) { return SSL_session_reused(ssl); } */ import "C" import ( "errors" "fmt" "io" "net" "runtime" "sync" "time" "unsafe" "github.com/spacemonkeygo/openssl/utils" ) var ( zeroReturn = errors.New("zero return") wantRead = errors.New("want read") wantWrite = errors.New("want write") tryAgain = errors.New("try again") ) type Conn struct { conn net.Conn ssl *C.SSL ctx *Ctx // for gc into_ssl *readBio from_ssl *writeBio is_shutdown bool mtx sync.Mutex want_read_future *utils.Future } type VerifyResult int const ( Ok VerifyResult = C.X509_V_OK UnableToGetIssuerCert VerifyResult = C.X509_V_ERR_UNABLE_TO_GET_ISSUER_CERT UnableToGetCrl VerifyResult = C.X509_V_ERR_UNABLE_TO_GET_CRL UnableToDecryptCertSignature VerifyResult = C.X509_V_ERR_UNABLE_TO_DECRYPT_CERT_SIGNATURE UnableToDecryptCrlSignature VerifyResult = C.X509_V_ERR_UNABLE_TO_DECRYPT_CRL_SIGNATURE UnableToDecodeIssuerPublicKey VerifyResult = C.X509_V_ERR_UNABLE_TO_DECODE_ISSUER_PUBLIC_KEY CertSignatureFailure VerifyResult = C.X509_V_ERR_CERT_SIGNATURE_FAILURE CrlSignatureFailure VerifyResult = C.X509_V_ERR_CRL_SIGNATURE_FAILURE CertNotYetValid VerifyResult = C.X509_V_ERR_CERT_NOT_YET_VALID CertHasExpired VerifyResult = C.X509_V_ERR_CERT_HAS_EXPIRED CrlNotYetValid VerifyResult = C.X509_V_ERR_CRL_NOT_YET_VALID CrlHasExpired VerifyResult = C.X509_V_ERR_CRL_HAS_EXPIRED ErrorInCertNotBeforeField VerifyResult = C.X509_V_ERR_ERROR_IN_CERT_NOT_BEFORE_FIELD ErrorInCertNotAfterField VerifyResult = C.X509_V_ERR_ERROR_IN_CERT_NOT_AFTER_FIELD ErrorInCrlLastUpdateField VerifyResult = C.X509_V_ERR_ERROR_IN_CRL_LAST_UPDATE_FIELD ErrorInCrlNextUpdateField VerifyResult = C.X509_V_ERR_ERROR_IN_CRL_NEXT_UPDATE_FIELD OutOfMem VerifyResult = C.X509_V_ERR_OUT_OF_MEM DepthZeroSelfSignedCert VerifyResult = C.X509_V_ERR_DEPTH_ZERO_SELF_SIGNED_CERT SelfSignedCertInChain VerifyResult = C.X509_V_ERR_SELF_SIGNED_CERT_IN_CHAIN UnableToGetIssuerCertLocally VerifyResult = C.X509_V_ERR_UNABLE_TO_GET_ISSUER_CERT_LOCALLY UnableToVerifyLeafSignature VerifyResult = C.X509_V_ERR_UNABLE_TO_VERIFY_LEAF_SIGNATURE CertChainTooLong VerifyResult = C.X509_V_ERR_CERT_CHAIN_TOO_LONG CertRevoked VerifyResult = C.X509_V_ERR_CERT_REVOKED InvalidCa VerifyResult = C.X509_V_ERR_INVALID_CA PathLengthExceeded VerifyResult = C.X509_V_ERR_PATH_LENGTH_EXCEEDED InvalidPurpose VerifyResult = C.X509_V_ERR_INVALID_PURPOSE CertUntrusted VerifyResult = C.X509_V_ERR_CERT_UNTRUSTED CertRejected VerifyResult = C.X509_V_ERR_CERT_REJECTED SubjectIssuerMismatch VerifyResult = C.X509_V_ERR_SUBJECT_ISSUER_MISMATCH AkidSkidMismatch VerifyResult = C.X509_V_ERR_AKID_SKID_MISMATCH AkidIssuerSerialMismatch VerifyResult = C.X509_V_ERR_AKID_ISSUER_SERIAL_MISMATCH KeyusageNoCertsign VerifyResult = C.X509_V_ERR_KEYUSAGE_NO_CERTSIGN UnableToGetCrlIssuer VerifyResult = C.X509_V_ERR_UNABLE_TO_GET_CRL_ISSUER UnhandledCriticalExtension VerifyResult = C.X509_V_ERR_UNHANDLED_CRITICAL_EXTENSION KeyusageNoCrlSign VerifyResult = C.X509_V_ERR_KEYUSAGE_NO_CRL_SIGN UnhandledCriticalCrlExtension VerifyResult = C.X509_V_ERR_UNHANDLED_CRITICAL_CRL_EXTENSION InvalidNonCa VerifyResult = C.X509_V_ERR_INVALID_NON_CA ProxyPathLengthExceeded VerifyResult = C.X509_V_ERR_PROXY_PATH_LENGTH_EXCEEDED KeyusageNoDigitalSignature VerifyResult = C.X509_V_ERR_KEYUSAGE_NO_DIGITAL_SIGNATURE ProxyCertificatesNotAllowed VerifyResult = C.X509_V_ERR_PROXY_CERTIFICATES_NOT_ALLOWED InvalidExtension VerifyResult = C.X509_V_ERR_INVALID_EXTENSION InvalidPolicyExtension VerifyResult = C.X509_V_ERR_INVALID_POLICY_EXTENSION NoExplicitPolicy VerifyResult = C.X509_V_ERR_NO_EXPLICIT_POLICY UnnestedResource VerifyResult = C.X509_V_ERR_UNNESTED_RESOURCE ApplicationVerification VerifyResult = C.X509_V_ERR_APPLICATION_VERIFICATION ) func newSSL(ctx *C.SSL_CTX) (*C.SSL, error) { runtime.LockOSThread() defer runtime.UnlockOSThread() ssl := C.SSL_new(ctx) if ssl == nil { return nil, errorFromErrorQueue() } return ssl, nil } func newConn(conn net.Conn, ctx *Ctx) (*Conn, error) { ssl, err := newSSL(ctx.ctx) if err != nil { return nil, err } into_ssl := &readBio{} from_ssl := &writeBio{} if ctx.GetMode()&ReleaseBuffers > 0 { into_ssl.release_buffers = true from_ssl.release_buffers = true } into_ssl_cbio := into_ssl.MakeCBIO() from_ssl_cbio := from_ssl.MakeCBIO() if into_ssl_cbio == nil || from_ssl_cbio == nil { // these frees are null safe C.BIO_free(into_ssl_cbio) C.BIO_free(from_ssl_cbio) C.SSL_free(ssl) return nil, errors.New("failed to allocate memory BIO") } // the ssl object takes ownership of these objects now C.SSL_set_bio(ssl, into_ssl_cbio, from_ssl_cbio) c := &Conn{ conn: conn, ssl: ssl, ctx: ctx, into_ssl: into_ssl, from_ssl: from_ssl} runtime.SetFinalizer(c, func(c *Conn) { c.into_ssl.Disconnect(into_ssl_cbio) c.from_ssl.Disconnect(from_ssl_cbio) C.SSL_free(c.ssl) }) return c, nil } // Client wraps an existing stream connection and puts it in the connect state // for any subsequent handshakes. // // IMPORTANT NOTE: if you use this method instead of Dial to construct an SSL // connection, you are responsible for verifying the peer's hostname. // Otherwise, you are vulnerable to MITM attacks. // // Client also does not set up SNI for you like Dial does. // // Client connections probably won't work for you unless you set a verify // location or add some certs to the certificate store of the client context // you're using. This library is not nice enough to use the system certificate // store by default for you yet. func Client(conn net.Conn, ctx *Ctx) (*Conn, error) { c, err := newConn(conn, ctx) if err != nil { return nil, err } C.SSL_set_connect_state(c.ssl) return c, nil } // Server wraps an existing stream connection and puts it in the accept state // for any subsequent handshakes. func Server(conn net.Conn, ctx *Ctx) (*Conn, error) { c, err := newConn(conn, ctx) if err != nil { return nil, err } C.SSL_set_accept_state(c.ssl) return c, nil } func (c *Conn) CurrentCipher() (string, error) { p := C.SSL_get_cipher_name_not_a_macro(c.ssl) if p == nil { return "", errors.New("Session not established") } return C.GoString(p), nil } func (c *Conn) fillInputBuffer() error { for { n, err := c.into_ssl.ReadFromOnce(c.conn) if n == 0 && err == nil { continue } if err == io.EOF { c.into_ssl.MarkEOF() return c.Close() } return err } } func (c *Conn) flushOutputBuffer() error { _, err := c.from_ssl.WriteTo(c.conn) return err } func (c *Conn) getErrorHandler(rv C.int, errno error) func() error { errcode := C.SSL_get_error(c.ssl, rv) switch errcode { case C.SSL_ERROR_ZERO_RETURN: return func() error { c.Close() return io.ErrUnexpectedEOF } case C.SSL_ERROR_WANT_READ: go c.flushOutputBuffer() if c.want_read_future != nil { want_read_future := c.want_read_future return func() error { _, err := want_read_future.Get() return err } } c.want_read_future = utils.NewFuture() want_read_future := c.want_read_future return func() (err error) { defer func() { c.mtx.Lock() c.want_read_future = nil c.mtx.Unlock() want_read_future.Set(nil, err) }() err = c.fillInputBuffer() if err != nil { return err } return tryAgain } case C.SSL_ERROR_WANT_WRITE: return func() error { err := c.flushOutputBuffer() if err != nil { return err } return tryAgain } case C.SSL_ERROR_SYSCALL: var err error if C.ERR_peek_error() == 0 { switch rv { case 0: err = errors.New("protocol-violating EOF") case -1: err = errno default: err = errorFromErrorQueue() } } else { err = errorFromErrorQueue() } return func() error { return err } default: err := errorFromErrorQueue() return func() error { return err } } } func (c *Conn) handleError(errcb func() error) error { if errcb != nil { return errcb() } return nil } func (c *Conn) handshake() func() error { c.mtx.Lock() defer c.mtx.Unlock() if c.is_shutdown { return func() error { return io.ErrUnexpectedEOF } } runtime.LockOSThread() defer runtime.UnlockOSThread() rv, errno := C.SSL_do_handshake(c.ssl) if rv > 0 { return nil } return c.getErrorHandler(rv, errno) } // Handshake performs an SSL handshake. If a handshake is not manually // triggered, it will run before the first I/O on the encrypted stream. func (c *Conn) Handshake() error { err := tryAgain for err == tryAgain { err = c.handleError(c.handshake()) } go c.flushOutputBuffer() return err } // PeerCertificate returns the Certificate of the peer with which you're // communicating. Only valid after a handshake. func (c *Conn) PeerCertificate() (*Certificate, error) { c.mtx.Lock() defer c.mtx.Unlock() if c.is_shutdown { return nil, errors.New("connection closed") } x := C.SSL_get_peer_certificate(c.ssl) if x == nil { return nil, errors.New("no peer certificate found") } cert := &Certificate{x: x} runtime.SetFinalizer(cert, func(cert *Certificate) { C.X509_free(cert.x) }) return cert, nil } // PeerCertificateChain returns the certificate chain of the peer. If called on // the client side, the stack also contains the peer's certificate; if called // on the server side, the peer's certificate must be obtained separately using // PeerCertificate. func (c *Conn) PeerCertificateChain() (rv []*Certificate, err error) { c.mtx.Lock() defer c.mtx.Unlock() if c.is_shutdown { return nil, errors.New("connection closed") } sk := C.SSL_get_peer_cert_chain(c.ssl) if sk == nil { return nil, errors.New("no peer certificates found") } sk_num := int(C.sk_X509_num_not_a_macro(sk)) rv = make([]*Certificate, 0, sk_num) for i := 0; i < sk_num; i++ { x := C.sk_X509_value_not_a_macro(sk, C.int(i)) // ref holds on to the underlying connection memory so we don't need to // worry about incrementing refcounts manually or freeing the X509 rv = append(rv, &Certificate{x: x, ref: c}) } return rv, nil } // GetVerifyResult gets result of peer certificate verification // SSL_get_verify_result() returns the result of the verification of the X509 // certificate presented by the peer, if any. See // https://www.openssl.org/docs/ssl/SSL_get_verify_result.html func (c *Conn) GetVerifyResults() error { result := C.SSL_get_verify_result(c.ssl) if int(result) != 0 { return errors.New(C.GoString( C.X509_verify_cert_error_string(result))) } return nil } type ConnectionState struct { Certificate *Certificate CertificateError error CertificateChain []*Certificate CertificateChainError error SessionReused bool } func (c *Conn) ConnectionState() (rv ConnectionState) { rv.Certificate, rv.CertificateError = c.PeerCertificate() rv.CertificateChain, rv.CertificateChainError = c.PeerCertificateChain() rv.SessionReused = c.SessionReused() return } func (c *Conn) shutdown() func() error { c.mtx.Lock() defer c.mtx.Unlock() runtime.LockOSThread() defer runtime.UnlockOSThread() rv, errno := C.SSL_shutdown(c.ssl) if rv > 0 { return nil } if rv == 0 { // The OpenSSL docs say that in this case, the shutdown is not // finished, and we should call SSL_shutdown() a second time, if a // bidirectional shutdown is going to be performed. Further, the // output of SSL_get_error may be misleading, as an erroneous // SSL_ERROR_SYSCALL may be flagged even though no error occurred. // So, TODO: revisit bidrectional shutdown, possibly trying again. // Note: some broken clients won't engage in bidirectional shutdown // without tickling them to close by sending a TCP_FIN packet, or // shutting down the write-side of the connection. return nil } else { return c.getErrorHandler(rv, errno) } } func (c *Conn) shutdownLoop() error { err := tryAgain shutdown_tries := 0 for err == tryAgain { shutdown_tries = shutdown_tries + 1 err = c.handleError(c.shutdown()) if err == nil { return c.flushOutputBuffer() } if err == tryAgain && shutdown_tries >= 2 { return errors.New("shutdown requested a third time?") } } if err == io.ErrUnexpectedEOF { err = nil } return err } // Close shuts down the SSL connection and closes the underlying wrapped // connection. func (c *Conn) Close() error { c.mtx.Lock() if c.is_shutdown { c.mtx.Unlock() return nil } c.is_shutdown = true c.mtx.Unlock() var errs utils.ErrorGroup errs.Add(c.shutdownLoop()) errs.Add(c.conn.Close()) return errs.Finalize() } func (c *Conn) read(b []byte) (int, func() error) { if len(b) == 0 { return 0, nil } c.mtx.Lock() defer c.mtx.Unlock() if c.is_shutdown { return 0, func() error { return io.EOF } } runtime.LockOSThread() defer runtime.UnlockOSThread() rv, errno := C.SSL_read(c.ssl, unsafe.Pointer(&b[0]), C.int(len(b))) if rv > 0 { return int(rv), nil } return 0, c.getErrorHandler(rv, errno) } // Read reads up to len(b) bytes into b. It returns the number of bytes read // and an error if applicable. io.EOF is returned when the caller can expect // to see no more data. func (c *Conn) Read(b []byte) (n int, err error) { if len(b) == 0 { return 0, nil } err = tryAgain for err == tryAgain { n, errcb := c.read(b) err = c.handleError(errcb) if err == nil { go c.flushOutputBuffer() return n, nil } if err == io.ErrUnexpectedEOF { err = io.EOF } } return 0, err } func (c *Conn) write(b []byte) (int, func() error) { if len(b) == 0 { return 0, nil } c.mtx.Lock() defer c.mtx.Unlock() if c.is_shutdown { err := errors.New("connection closed") return 0, func() error { return err } } runtime.LockOSThread() defer runtime.UnlockOSThread() rv, errno := C.SSL_write(c.ssl, unsafe.Pointer(&b[0]), C.int(len(b))) if rv > 0 { return int(rv), nil } return 0, c.getErrorHandler(rv, errno) } // Write will encrypt the contents of b and write it to the underlying stream. // Performance will be vastly improved if the size of b is a multiple of // SSLRecordSize. func (c *Conn) Write(b []byte) (written int, err error) { if len(b) == 0 { return 0, nil } err = tryAgain for err == tryAgain { n, errcb := c.write(b) err = c.handleError(errcb) if err == nil { return n, c.flushOutputBuffer() } } return 0, err } // VerifyHostname pulls the PeerCertificate and calls VerifyHostname on the // certificate. func (c *Conn) VerifyHostname(host string) error { cert, err := c.PeerCertificate() if err != nil { return err } return cert.VerifyHostname(host) } // LocalAddr returns the underlying connection's local address func (c *Conn) LocalAddr() net.Addr { return c.conn.LocalAddr() } // RemoteAddr returns the underlying connection's remote address func (c *Conn) RemoteAddr() net.Addr { return c.conn.RemoteAddr() } // SetDeadline calls SetDeadline on the underlying connection. func (c *Conn) SetDeadline(t time.Time) error { return c.conn.SetDeadline(t) } // SetReadDeadline calls SetReadDeadline on the underlying connection. func (c *Conn) SetReadDeadline(t time.Time) error { return c.conn.SetReadDeadline(t) } // SetWriteDeadline calls SetWriteDeadline on the underlying connection. func (c *Conn) SetWriteDeadline(t time.Time) error { return c.conn.SetWriteDeadline(t) } func (c *Conn) UnderlyingConn() net.Conn { return c.conn } func (c *Conn) VerifyResult() VerifyResult { return VerifyResult(C.SSL_get_verify_result(c.ssl)) } func (c *Conn) SessionReused() bool { return C.SSL_session_reused_not_a_macro(c.ssl) == 1 } func (c *Conn) GetSession() ([]byte, error) { runtime.LockOSThread() defer runtime.UnlockOSThread() // get1 increases the refcount of the session, so we have to free it. session := (*C.SSL_SESSION)(C.SSL_get1_session(c.ssl)) if session == nil { return nil, errors.New("failed to get session") } defer C.SSL_SESSION_free(session) // get the size of the encoding slen := C.i2d_SSL_SESSION(session, nil) buf := (*C.uchar)(C.malloc(C.size_t(slen))) defer C.free(unsafe.Pointer(buf)) // this modifies the value of buf (seriously), so we have to pass in a temp // var so that we can actually read the bytes from buf. tmp := buf slen2 := C.i2d_SSL_SESSION(session, &tmp) if slen != slen2 { return nil, errors.New("session had different lengths") } return C.GoBytes(unsafe.Pointer(buf), slen), nil } func (c *Conn) setSession(session []byte) error { runtime.LockOSThread() defer runtime.UnlockOSThread() ptr := (*C.uchar)(&session[0]) s := C.d2i_SSL_SESSION(nil, &ptr, C.long(len(session))) if s == nil { return fmt.Errorf("unable to load session: %s", errorFromErrorQueue()) } defer C.SSL_SESSION_free(s) ret := C.SSL_set_session(c.ssl, s) if ret != 1 { return fmt.Errorf("unable to set session: %s", errorFromErrorQueue()) } return nil } mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/openssl/ctx.go000066400000000000000000000601311276133741700260500ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // +build cgo package openssl /* #include #include #include #include #include static long SSL_CTX_set_options_not_a_macro(SSL_CTX* ctx, long options) { return SSL_CTX_set_options(ctx, options); } static long SSL_CTX_clear_options_not_a_macro(SSL_CTX* ctx, long options) { return SSL_CTX_clear_options(ctx, options); } static long SSL_CTX_get_options_not_a_macro(SSL_CTX* ctx) { return SSL_CTX_get_options(ctx); } static long SSL_CTX_set_mode_not_a_macro(SSL_CTX* ctx, long modes) { return SSL_CTX_set_mode(ctx, modes); } static long SSL_CTX_get_mode_not_a_macro(SSL_CTX* ctx) { return SSL_CTX_get_mode(ctx); } static long SSL_CTX_set_session_cache_mode_not_a_macro(SSL_CTX* ctx, long modes) { return SSL_CTX_set_session_cache_mode(ctx, modes); } static long SSL_CTX_sess_set_cache_size_not_a_macro(SSL_CTX* ctx, long t) { return SSL_CTX_sess_set_cache_size(ctx, t); } static long SSL_CTX_sess_get_cache_size_not_a_macro(SSL_CTX* ctx) { return SSL_CTX_sess_get_cache_size(ctx); } static long SSL_CTX_set_timeout_not_a_macro(SSL_CTX* ctx, long t) { return SSL_CTX_set_timeout(ctx, t); } static long SSL_CTX_get_timeout_not_a_macro(SSL_CTX* ctx) { return SSL_CTX_get_timeout(ctx); } static int CRYPTO_add_not_a_macro(int *pointer,int amount,int type) { return CRYPTO_add(pointer, amount, type); } static long SSL_CTX_add_extra_chain_cert_not_a_macro(SSL_CTX* ctx, X509 *cert) { return SSL_CTX_add_extra_chain_cert(ctx, cert); } #ifndef SSL_MODE_RELEASE_BUFFERS #define SSL_MODE_RELEASE_BUFFERS 0 #endif #ifndef SSL_OP_NO_COMPRESSION #define SSL_OP_NO_COMPRESSION 0 #endif static const SSL_METHOD *OUR_TLSv1_1_method() { #if OPENSSL_VERSION_NUMBER > 0x1000100fL && defined(TLS1_1_VERSION) && !defined(OPENSSL_SYSNAME_MACOSX) return TLSv1_1_method(); #else return NULL; #endif } static const SSL_METHOD *OUR_TLSv1_2_method() { #if OPENSSL_VERSION_NUMBER > 0x1000100fL && defined(TLS1_2_VERSION) && !defined(OPENSSL_SYSNAME_MACOSX) return TLSv1_2_method(); #else return NULL; #endif } #if defined SSL_CTRL_SET_TLSEXT_HOSTNAME extern int sni_cb(SSL *ssl_conn, int *ad, void *arg); #endif extern int verify_cb(int ok, X509_STORE_CTX* store); typedef STACK_OF(X509_NAME) *STACK_OF_X509_NAME_not_a_macro; static void sk_X509_NAME_pop_free_not_a_macro(STACK_OF_X509_NAME_not_a_macro st) { sk_X509_NAME_pop_free(st, X509_NAME_free); } extern int password_cb(char *buf, int size, int rwflag, void *password); */ import "C" import ( "errors" "fmt" "io/ioutil" "os" "runtime" "time" "unsafe" "github.com/spacemonkeygo/spacelog" ) var ( ssl_ctx_idx = C.SSL_CTX_get_ex_new_index(0, nil, nil, nil, nil) logger = spacelog.GetLogger() ) type Ctx struct { ctx *C.SSL_CTX cert *Certificate chain []*Certificate key PrivateKey verify_cb VerifyCallback sni_cb TLSExtServernameCallback } //export get_ssl_ctx_idx func get_ssl_ctx_idx() C.int { return ssl_ctx_idx } func newCtx(method *C.SSL_METHOD) (*Ctx, error) { runtime.LockOSThread() defer runtime.UnlockOSThread() ctx := C.SSL_CTX_new(method) if ctx == nil { return nil, errorFromErrorQueue() } c := &Ctx{ctx: ctx} C.SSL_CTX_set_ex_data(ctx, get_ssl_ctx_idx(), unsafe.Pointer(c)) runtime.SetFinalizer(c, func(c *Ctx) { C.SSL_CTX_free(c.ctx) }) return c, nil } type SSLVersion int const ( SSLv3 SSLVersion = 0x02 // Vulnerable to "POODLE" attack. TLSv1 SSLVersion = 0x03 TLSv1_1 SSLVersion = 0x04 TLSv1_2 SSLVersion = 0x05 // Make sure to disable SSLv2 and SSLv3 if you use this. SSLv3 is vulnerable // to the "POODLE" attack, and SSLv2 is what, just don't even. AnyVersion SSLVersion = 0x06 ) // NewCtxWithVersion creates an SSL context that is specific to the provided // SSL version. See http://www.openssl.org/docs/ssl/SSL_CTX_new.html for more. func NewCtxWithVersion(version SSLVersion) (*Ctx, error) { var method *C.SSL_METHOD switch version { case TLSv1: method = C.TLSv1_method() case TLSv1_1: method = C.OUR_TLSv1_1_method() case TLSv1_2: method = C.OUR_TLSv1_2_method() case AnyVersion: method = C.SSLv23_method() } if method == nil { return nil, errors.New("unknown ssl/tls version") } return newCtx(method) } // NewCtx creates a context that supports any TLS version 1.0 and newer. func NewCtx() (*Ctx, error) { c, err := NewCtxWithVersion(AnyVersion) if err == nil { c.SetOptions(NoSSLv2 | NoSSLv3) } return c, err } // NewCtxFromFiles calls NewCtx, loads the provided files, and configures the // context to use them. func NewCtxFromFiles(cert_file string, key_file string) (*Ctx, error) { ctx, err := NewCtx() if err != nil { return nil, err } cert_bytes, err := ioutil.ReadFile(cert_file) if err != nil { return nil, err } certs := SplitPEM(cert_bytes) if len(certs) == 0 { return nil, fmt.Errorf("No PEM certificate found in '%s'", cert_file) } first, certs := certs[0], certs[1:] cert, err := LoadCertificateFromPEM(first) if err != nil { return nil, err } err = ctx.UseCertificate(cert) if err != nil { return nil, err } for _, pem := range certs { cert, err := LoadCertificateFromPEM(pem) if err != nil { return nil, err } err = ctx.AddChainCertificate(cert) if err != nil { return nil, err } } key_bytes, err := ioutil.ReadFile(key_file) if err != nil { return nil, err } key, err := LoadPrivateKeyFromPEM(key_bytes) if err != nil { return nil, err } err = ctx.UsePrivateKey(key) if err != nil { return nil, err } return ctx, nil } // EllipticCurve repesents the ASN.1 OID of an elliptic curve. // see https://www.openssl.org/docs/apps/ecparam.html for a list of implemented curves. type EllipticCurve int const ( // P-256: X9.62/SECG curve over a 256 bit prime field Prime256v1 EllipticCurve = C.NID_X9_62_prime256v1 // P-384: NIST/SECG curve over a 384 bit prime field Secp384r1 EllipticCurve = C.NID_secp384r1 ) // UseCertificate configures the context to present the given certificate to // peers. func (c *Ctx) UseCertificate(cert *Certificate) error { runtime.LockOSThread() defer runtime.UnlockOSThread() c.cert = cert if int(C.SSL_CTX_use_certificate(c.ctx, cert.x)) != 1 { return errorFromErrorQueue() } return nil } // UseCertificateChainFromFile loads a certificate chain from file into ctx. // The certificates must be in PEM format and must be sorted starting with the // subject's certificate (actual client or server certificate), followed by // intermediate CA certificates if applicable, and ending at the highest level // (root) CA. See // https://www.openssl.org/docs/ssl/SSL_CTX_use_certificate.html func (c *Ctx) UseCertificateChainFile(cert_file string) error { runtime.LockOSThread() defer runtime.UnlockOSThread() var c_cert_file *C.char if cert_file != "" { c_cert_file = C.CString(cert_file) defer C.free(unsafe.Pointer(c_cert_file)) } if int(C.SSL_CTX_use_certificate_chain_file(c.ctx, c_cert_file)) != 1 { return errorFromErrorQueue() } return nil } // UsePrivateKeyFile adds the first private key found in file to the *Ctx, c. The // formatting type of the certificate must be specified from the known types // FiletypePEM, and FiletypeASN1 func (c *Ctx) UsePrivateKeyFile(key_file string, file_type Filetypes) error { runtime.LockOSThread() defer runtime.UnlockOSThread() var c_key_file *C.char if key_file != "" { c_key_file = C.CString(key_file) defer C.free(unsafe.Pointer(c_key_file)) } if int(C.SSL_CTX_use_PrivateKey_file(c.ctx, c_key_file, C.int(file_type))) != 1 { return errorFromErrorQueue() } return nil } func (c *Ctx) UsePrivateKeyFileWithPassword(key_file string, file_type Filetypes, password string) error { runtime.LockOSThread() defer runtime.UnlockOSThread() var c_key_file *C.char c_pwd := C.CString(password) defer C.free(unsafe.Pointer(c_pwd)) C.SSL_CTX_set_default_passwd_cb_userdata(c.ctx, unsafe.Pointer(c_pwd)) C.SSL_CTX_set_default_passwd_cb(c.ctx, (*C.pem_password_cb)(C.password_cb)) if key_file != "" { c_key_file = C.CString(key_file) defer C.free(unsafe.Pointer(c_key_file)) } if int(C.SSL_CTX_use_PrivateKey_file(c.ctx, c_key_file, C.int(file_type))) != 1 { return errorFromErrorQueue() } return nil } // CheckPrivateKey verifies that the private key agrees with the corresponding // public key in the certificate func (c *Ctx) CheckPrivateKey() error { runtime.LockOSThread() defer runtime.UnlockOSThread() if int(C.SSL_CTX_check_private_key(c.ctx)) != 1 { return errorFromErrorQueue() } return nil } type StackOfX509Name struct { stack C.STACK_OF_X509_NAME_not_a_macro // shared indicates weather we are the sole owner of this pointer, and implies // weather we should or shouldn't free the underlying data structure // when this go data structure goes out of scope shared bool } // LoadClientCAFile reads certificates from file and returns a StackOfX509Name // with the subject names found. See // https://www.openssl.org/docs/ssl/SSL_load_client_CA_file.html func LoadClientCAFile(ca_file string) (*StackOfX509Name, error) { runtime.LockOSThread() defer runtime.UnlockOSThread() var c_ca_file *C.char if ca_file != "" { c_ca_file = C.CString(ca_file) defer C.free(unsafe.Pointer(c_ca_file)) } stack := C.SSL_load_client_CA_file(c_ca_file) if stack == nil { return nil, errorFromErrorQueue() } caList := StackOfX509Name{ stack: stack, shared: false, } runtime.SetFinalizer(&caList, func(c *StackOfX509Name) { if !c.shared { C.sk_X509_NAME_pop_free_not_a_macro(c.stack) } }) return &caList, nil } // SetClientCAList sets the list of CAs sent to the client when requesting a // client certificate for Ctx. See // https://www.openssl.org/docs/ssl/SSL_CTX_set_client_CA_list.html func (c *Ctx) SetClientCAList(caList *StackOfX509Name) { C.SSL_CTX_set_client_CA_list(c.ctx, caList.stack) caList.shared = true } // AddChainCertificate adds a certificate to the chain presented in the // handshake. func (c *Ctx) AddChainCertificate(cert *Certificate) error { runtime.LockOSThread() defer runtime.UnlockOSThread() c.chain = append(c.chain, cert) if int(C.SSL_CTX_add_extra_chain_cert_not_a_macro(c.ctx, cert.x)) != 1 { return errorFromErrorQueue() } // OpenSSL takes ownership via SSL_CTX_add_extra_chain_cert runtime.SetFinalizer(cert, nil) return nil } // UsePrivateKey configures the context to use the given private key for SSL // handshakes. func (c *Ctx) UsePrivateKey(key PrivateKey) error { runtime.LockOSThread() defer runtime.UnlockOSThread() c.key = key if int(C.SSL_CTX_use_PrivateKey(c.ctx, key.evpPKey())) != 1 { return errorFromErrorQueue() } return nil } type CertificateStore struct { store *C.X509_STORE // for GC ctx *Ctx certs []*Certificate } // Allocate a new, empty CertificateStore func NewCertificateStore() (*CertificateStore, error) { s := C.X509_STORE_new() if s == nil { return nil, errors.New("failed to allocate X509_STORE") } store := &CertificateStore{store: s} runtime.SetFinalizer(store, func(s *CertificateStore) { C.X509_STORE_free(s.store) }) return store, nil } // Parse a chained PEM file, loading all certificates into the Store. func (s *CertificateStore) LoadCertificatesFromPEM(data []byte) error { pems := SplitPEM(data) for _, pem := range pems { cert, err := LoadCertificateFromPEM(pem) if err != nil { return err } err = s.AddCertificate(cert) if err != nil { return err } } return nil } // GetCertificateStore returns the context's certificate store that will be // used for peer validation. func (c *Ctx) GetCertificateStore() *CertificateStore { // we don't need to dealloc the cert store pointer here, because it points // to a ctx internal. so we do need to keep the ctx around return &CertificateStore{ store: C.SSL_CTX_get_cert_store(c.ctx), ctx: c} } // AddCertificate marks the provided Certificate as a trusted certificate in // the given CertificateStore. func (s *CertificateStore) AddCertificate(cert *Certificate) error { runtime.LockOSThread() defer runtime.UnlockOSThread() s.certs = append(s.certs, cert) if int(C.X509_STORE_add_cert(s.store, cert.x)) != 1 { return errorFromErrorQueue() } return nil } type X509VerificationFlag int func (s *CertificateStore) SetFlags(flags X509VerificationFlag) error { runtime.LockOSThread() defer runtime.UnlockOSThread() if int(C.X509_STORE_set_flags(s.store, C.ulong(flags))) != 1 { return errorFromErrorQueue() } return nil } // See https://www.openssl.org/docs/crypto/X509_VERIFY_PARAM_set_flags.html const ( CBIssuerCheck X509VerificationFlag = C.X509_V_FLAG_CB_ISSUER_CHECK UseCheckTime X509VerificationFlag = C.X509_V_FLAG_USE_CHECK_TIME CRLCheck X509VerificationFlag = C.X509_V_FLAG_CRL_CHECK CRLCheckAll X509VerificationFlag = C.X509_V_FLAG_CRL_CHECK_ALL IgnoreCritical X509VerificationFlag = C.X509_V_FLAG_IGNORE_CRITICAL X509Strict X509VerificationFlag = C.X509_V_FLAG_X509_STRICT AllowProxyCerts X509VerificationFlag = C.X509_V_FLAG_ALLOW_PROXY_CERTS PolicyCheck X509VerificationFlag = C.X509_V_FLAG_POLICY_CHECK ExplicitPolicy X509VerificationFlag = C.X509_V_FLAG_EXPLICIT_POLICY InhibitAny X509VerificationFlag = C.X509_V_FLAG_INHIBIT_ANY InhibitMap X509VerificationFlag = C.X509_V_FLAG_INHIBIT_MAP NotifyPolicy X509VerificationFlag = C.X509_V_FLAG_NOTIFY_POLICY // ExtendedCRLSupport X509VerificationFlag = C.X509_V_FLAG_EXTENDED_CRL_SUPPORT // UseDeltas X509VerificationFlag = C.X509_V_FLAG_USE_DELTAS // CheckSsSignature X509VerificationFlag = C.X509_V_FLAG_CHECK_SS_SIGNATURE // TrustedFirst X509VerificationFlag = C.X509_V_FLAG_TRUSTED_FIRST PolicyMask X509VerificationFlag = C.X509_V_FLAG_POLICY_MASK ) type CertificateStoreLookup struct { lookup *C.X509_LOOKUP store *CertificateStore } // an X509LookupMethod is required to build a a CertificateStoreLookup in a // CertificateStore. The X509LookupMethod indicates the type or functionality // of the CertificateStoreLookup type X509LookupMethod *C.X509_LOOKUP_METHOD // CertificateStoreLookups with X509LookupFile methods look for certs in a file func X509LookupFile() X509LookupMethod { return X509LookupMethod(C.X509_LOOKUP_file()) } // CertificateStoreLookups with X509LookupHashDir methods look for certs in a // directory func X509LookupHashDir() X509LookupMethod { return X509LookupMethod(C.X509_LOOKUP_hash_dir()) } // AddLookup creates a CertificateStoreLookup of type X509LookupMethod in the // CertificateStore func (s *CertificateStore) AddLookup(method X509LookupMethod) (*CertificateStoreLookup, error) { runtime.LockOSThread() defer runtime.UnlockOSThread() var lookup *C.X509_LOOKUP lookup = C.X509_STORE_add_lookup(s.store, method) if lookup != nil { return &CertificateStoreLookup{ lookup: lookup, store: s, }, nil } return nil, errorFromErrorQueue() } // LoadCRLFile adds a file to a CertificateStoreLookup in the // CertificateStore // I suspect that the CertificateStoreLookup needs to have been created with // X509LookupFile as the lookup method func (l *CertificateStoreLookup) LoadCRLFile(crl_file string) error { runtime.LockOSThread() defer runtime.UnlockOSThread() var c_crl_file *C.char if crl_file != "" { c_crl_file = C.CString(crl_file) defer C.free(unsafe.Pointer(c_crl_file)) } if int(C.X509_load_crl_file(l.lookup, c_crl_file, C.X509_FILETYPE_PEM)) != 1 { return errorFromErrorQueue() } return nil } type CertificateStoreCtx struct { ctx *C.X509_STORE_CTX ssl_ctx *Ctx } func (self *CertificateStoreCtx) VerifyResult() VerifyResult { return VerifyResult(C.X509_STORE_CTX_get_error(self.ctx)) } func (self *CertificateStoreCtx) Err() error { code := C.X509_STORE_CTX_get_error(self.ctx) if code == C.X509_V_OK { return nil } return fmt.Errorf("openssl: %s", C.GoString(C.X509_verify_cert_error_string(C.long(code)))) } func (self *CertificateStoreCtx) Depth() int { return int(C.X509_STORE_CTX_get_error_depth(self.ctx)) } // the certicate returned is only valid for the lifetime of the underlying // X509_STORE_CTX func (self *CertificateStoreCtx) GetCurrentCert() *Certificate { x509 := C.X509_STORE_CTX_get_current_cert(self.ctx) if x509 == nil { return nil } // add a ref C.CRYPTO_add_not_a_macro(&x509.references, 1, C.CRYPTO_LOCK_X509) cert := &Certificate{ x: x509, } runtime.SetFinalizer(cert, func(cert *Certificate) { C.X509_free(cert.x) }) return cert } // LoadVerifyLocations tells the context to trust all certificate authorities // provided in either the ca_file or the ca_path. // See http://www.openssl.org/docs/ssl/SSL_CTX_load_verify_locations.html for // more. func (c *Ctx) LoadVerifyLocations(ca_file string, ca_path string) error { runtime.LockOSThread() defer runtime.UnlockOSThread() var c_ca_file, c_ca_path *C.char if ca_file != "" { c_ca_file = C.CString(ca_file) defer C.free(unsafe.Pointer(c_ca_file)) } if ca_path != "" { c_ca_path = C.CString(ca_path) defer C.free(unsafe.Pointer(c_ca_path)) } if C.SSL_CTX_load_verify_locations(c.ctx, c_ca_file, c_ca_path) != 1 { return errorFromErrorQueue() } return nil } type Options uint const ( // NoCompression is only valid if you are using OpenSSL 1.0.1 or newer NoCompression Options = C.SSL_OP_NO_COMPRESSION NoSSLv2 Options = C.SSL_OP_NO_SSLv2 NoSSLv3 Options = C.SSL_OP_NO_SSLv3 NoTLSv1 Options = C.SSL_OP_NO_TLSv1 CipherServerPreference Options = C.SSL_OP_CIPHER_SERVER_PREFERENCE NoSessionResumptionOrRenegotiation Options = C.SSL_OP_NO_SESSION_RESUMPTION_ON_RENEGOTIATION OpAll Options = C.SSL_OP_ALL ) // SetOptions sets context options. See // http://www.openssl.org/docs/ssl/SSL_CTX_set_options.html func (c *Ctx) SetOptions(options Options) Options { return Options(C.SSL_CTX_set_options_not_a_macro( c.ctx, C.long(options))) } func (c *Ctx) ClearOptions(options Options) Options { return Options(C.SSL_CTX_clear_options_not_a_macro( c.ctx, C.long(options))) } // GetOptions returns context options. See // https://www.openssl.org/docs/ssl/SSL_CTX_set_options.html func (c *Ctx) GetOptions() Options { return Options(C.SSL_CTX_get_options_not_a_macro(c.ctx)) } type Modes int const ( // ReleaseBuffers is only valid if you are using OpenSSL 1.0.1 or newer ReleaseBuffers Modes = C.SSL_MODE_RELEASE_BUFFERS AutoRetry Modes = C.SSL_MODE_AUTO_RETRY ) // SetMode sets context modes. See // http://www.openssl.org/docs/ssl/SSL_CTX_set_mode.html func (c *Ctx) SetMode(modes Modes) Modes { return Modes(C.SSL_CTX_set_mode_not_a_macro(c.ctx, C.long(modes))) } // GetMode returns context modes. See // http://www.openssl.org/docs/ssl/SSL_CTX_set_mode.html func (c *Ctx) GetMode() Modes { return Modes(C.SSL_CTX_get_mode_not_a_macro(c.ctx)) } type VerifyOptions int const ( VerifyNone VerifyOptions = C.SSL_VERIFY_NONE VerifyPeer VerifyOptions = C.SSL_VERIFY_PEER VerifyFailIfNoPeerCert VerifyOptions = C.SSL_VERIFY_FAIL_IF_NO_PEER_CERT VerifyClientOnce VerifyOptions = C.SSL_VERIFY_CLIENT_ONCE ) type Filetypes int const ( FiletypePEM Filetypes = C.SSL_FILETYPE_PEM FiletypeASN1 Filetypes = C.SSL_FILETYPE_ASN1 ) type VerifyCallback func(ok bool, store *CertificateStoreCtx) bool //export verify_cb_thunk func verify_cb_thunk(p unsafe.Pointer, ok C.int, ctx *C.X509_STORE_CTX) C.int { defer func() { if err := recover(); err != nil { logger.Critf("openssl: verify callback panic'd: %v", err) os.Exit(1) } }() verify_cb := (*Ctx)(p).verify_cb // set up defaults just in case verify_cb is nil if verify_cb != nil { store := &CertificateStoreCtx{ctx: ctx} if verify_cb(ok == 1, store) { ok = 1 } else { ok = 0 } } return ok } // SetVerify controls peer verification settings. See // http://www.openssl.org/docs/ssl/SSL_CTX_set_verify.html func (c *Ctx) SetVerify(options VerifyOptions, verify_cb VerifyCallback) { c.verify_cb = verify_cb if verify_cb != nil { C.SSL_CTX_set_verify(c.ctx, C.int(options), (*[0]byte)(C.verify_cb)) } else { C.SSL_CTX_set_verify(c.ctx, C.int(options), nil) } } func (c *Ctx) SetVerifyMode(options VerifyOptions) { c.SetVerify(options, c.verify_cb) } func (c *Ctx) SetVerifyCallback(verify_cb VerifyCallback) { c.SetVerify(c.VerifyMode(), verify_cb) } func (c *Ctx) GetVerifyCallback() VerifyCallback { return c.verify_cb } func (c *Ctx) VerifyMode() VerifyOptions { return VerifyOptions(C.SSL_CTX_get_verify_mode(c.ctx)) } // SetVerifyDepth controls how many certificates deep the certificate // verification logic is willing to follow a certificate chain. See // https://www.openssl.org/docs/ssl/SSL_CTX_set_verify.html func (c *Ctx) SetVerifyDepth(depth int) { C.SSL_CTX_set_verify_depth(c.ctx, C.int(depth)) } // GetVerifyDepth controls how many certificates deep the certificate // verification logic is willing to follow a certificate chain. See // https://www.openssl.org/docs/ssl/SSL_CTX_set_verify.html func (c *Ctx) GetVerifyDepth() int { return int(C.SSL_CTX_get_verify_depth(c.ctx)) } type TLSExtServernameCallback func(ssl *SSL) SSLTLSExtErr func (c *Ctx) SetSessionId(session_id []byte) error { runtime.LockOSThread() defer runtime.UnlockOSThread() var ptr *C.uchar if len(session_id) > 0 { ptr = (*C.uchar)(unsafe.Pointer(&session_id[0])) } if int(C.SSL_CTX_set_session_id_context(c.ctx, ptr, C.uint(len(session_id)))) == 0 { return errorFromErrorQueue() } return nil } // SetCipherList sets the list of available ciphers. The format of the list is // described at http://www.openssl.org/docs/apps/ciphers.html, but see // http://www.openssl.org/docs/ssl/SSL_CTX_set_cipher_list.html for more. func (c *Ctx) SetCipherList(list string) error { runtime.LockOSThread() defer runtime.UnlockOSThread() clist := C.CString(list) defer C.free(unsafe.Pointer(clist)) if int(C.SSL_CTX_set_cipher_list(c.ctx, clist)) == 0 { return errorFromErrorQueue() } return nil } type SessionCacheModes int const ( SessionCacheOff SessionCacheModes = C.SSL_SESS_CACHE_OFF SessionCacheClient SessionCacheModes = C.SSL_SESS_CACHE_CLIENT SessionCacheServer SessionCacheModes = C.SSL_SESS_CACHE_SERVER SessionCacheBoth SessionCacheModes = C.SSL_SESS_CACHE_BOTH NoAutoClear SessionCacheModes = C.SSL_SESS_CACHE_NO_AUTO_CLEAR NoInternalLookup SessionCacheModes = C.SSL_SESS_CACHE_NO_INTERNAL_LOOKUP NoInternalStore SessionCacheModes = C.SSL_SESS_CACHE_NO_INTERNAL_STORE NoInternal SessionCacheModes = C.SSL_SESS_CACHE_NO_INTERNAL ) // SetSessionCacheMode enables or disables session caching. See // http://www.openssl.org/docs/ssl/SSL_CTX_set_session_cache_mode.html func (c *Ctx) SetSessionCacheMode(modes SessionCacheModes) SessionCacheModes { return SessionCacheModes( C.SSL_CTX_set_session_cache_mode_not_a_macro(c.ctx, C.long(modes))) } // Set session cache timeout. Returns previously set value. // See https://www.openssl.org/docs/ssl/SSL_CTX_set_timeout.html func (c *Ctx) SetTimeout(t time.Duration) time.Duration { prev := C.SSL_CTX_set_timeout_not_a_macro(c.ctx, C.long(t/time.Second)) return time.Duration(prev) * time.Second } // Get session cache timeout. // See https://www.openssl.org/docs/ssl/SSL_CTX_set_timeout.html func (c *Ctx) GetTimeout() time.Duration { return time.Duration(C.SSL_CTX_get_timeout_not_a_macro(c.ctx)) * time.Second } // Set session cache size. Returns previously set value. // https://www.openssl.org/docs/ssl/SSL_CTX_sess_set_cache_size.html func (c *Ctx) SessSetCacheSize(t int) int { return int(C.SSL_CTX_sess_set_cache_size_not_a_macro(c.ctx, C.long(t))) } // Get session cache size. // https://www.openssl.org/docs/ssl/SSL_CTX_sess_set_cache_size.html func (c *Ctx) SessGetCacheSize() int { return int(C.SSL_CTX_sess_get_cache_size_not_a_macro(c.ctx)) } mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/openssl/ctx_test.go000066400000000000000000000026661276133741700271200ustar00rootroot00000000000000// Copyright (C) 2014 Ryan Hileman // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. package openssl import ( "testing" "time" ) func TestCtxTimeoutOption(t *testing.T) { ctx, _ := NewCtx() oldTimeout1 := ctx.GetTimeout() newTimeout1 := oldTimeout1 + (time.Duration(99) * time.Second) oldTimeout2 := ctx.SetTimeout(newTimeout1) newTimeout2 := ctx.GetTimeout() if oldTimeout1 != oldTimeout2 { t.Error("SetTimeout() returns something undocumented") } if newTimeout1 != newTimeout2 { t.Error("SetTimeout() does not save anything to ctx") } } func TestCtxSessCacheSizeOption(t *testing.T) { ctx, _ := NewCtx() oldSize1 := ctx.SessGetCacheSize() newSize1 := oldSize1 + 42 oldSize2 := ctx.SessSetCacheSize(newSize1) newSize2 := ctx.SessGetCacheSize() if oldSize1 != oldSize2 { t.Error("SessSetCacheSize() returns something undocumented") } if newSize1 != newSize2 { t.Error("SessSetCacheSize() does not save anything to ctx") } } mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/openssl/dhparam.go000066400000000000000000000027101276133741700266650ustar00rootroot00000000000000// +build cgo package openssl /* #include #include #include #include #include static long SSL_CTX_set_tmp_dh_not_a_macro(SSL_CTX* ctx, DH *dh) { return SSL_CTX_set_tmp_dh(ctx, dh); } static long PEM_read_DHparams_not_a_macro(SSL_CTX* ctx, DH *dh) { return SSL_CTX_set_tmp_dh(ctx, dh); } */ import "C" import ( "errors" "runtime" "unsafe" ) type DH struct { dh *C.struct_dh_st } // LoadDHParametersFromPEM loads the Diffie-Hellman parameters from // a PEM-encoded block. func LoadDHParametersFromPEM(pem_block []byte) (*DH, error) { if len(pem_block) == 0 { return nil, errors.New("empty pem block") } bio := C.BIO_new_mem_buf(unsafe.Pointer(&pem_block[0]), C.int(len(pem_block))) if bio == nil { return nil, errors.New("failed creating bio") } defer C.BIO_free(bio) params := C.PEM_read_bio_DHparams(bio, nil, nil, nil) if params == nil { return nil, errors.New("failed reading dh parameters") } dhparams := &DH{dh: params} runtime.SetFinalizer(dhparams, func(dhparams *DH) { C.DH_free(dhparams.dh) }) return dhparams, nil } // SetDHParameters sets the DH group (DH parameters) used to // negotiate an emphemeral DH key during handshaking. func (c *Ctx) SetDHParameters(dh *DH) error { runtime.LockOSThread() defer runtime.UnlockOSThread() if int(C.SSL_CTX_set_tmp_dh_not_a_macro(c.ctx, dh.dh)) != 1 { return errorFromErrorQueue() } return nil } mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/openssl/digest.go000066400000000000000000000026671276133741700265430ustar00rootroot00000000000000// Copyright (C) 2015 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // +build cgo package openssl // #include import "C" import ( "fmt" "unsafe" ) // Digest represents and openssl message digest. type Digest struct { ptr *C.EVP_MD } // GetDigestByName returns the Digest with the name or nil and an error if the // digest was not found. func GetDigestByName(name string) (*Digest, error) { cname := C.CString(name) defer C.free(unsafe.Pointer(cname)) p := C.EVP_get_digestbyname(cname) if p == nil { return nil, fmt.Errorf("Digest %v not found", name) } // we can consider digests to use static mem; don't need to free return &Digest{ptr: p}, nil } // GetDigestByName returns the Digest with the NID or nil and an error if the // digest was not found. func GetDigestByNid(nid NID) (*Digest, error) { sn, err := Nid2ShortName(nid) if err != nil { return nil, err } return GetDigestByName(sn) } mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/openssl/engine.go000066400000000000000000000022641276133741700265220ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // +build cgo package openssl /* #include "openssl/engine.h" */ import "C" import ( "fmt" "runtime" "unsafe" ) type Engine struct { e *C.ENGINE } func EngineById(name string) (*Engine, error) { cname := C.CString(name) defer C.free(unsafe.Pointer(cname)) e := &Engine{ e: C.ENGINE_by_id(cname), } if e.e == nil { return nil, fmt.Errorf("engine %s missing", name) } if C.ENGINE_init(e.e) == 0 { C.ENGINE_free(e.e) return nil, fmt.Errorf("engine %s not initialized", name) } runtime.SetFinalizer(e, func(e *Engine) { C.ENGINE_finish(e.e) C.ENGINE_free(e.e) }) return e, nil } mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/openssl/fips.go000066400000000000000000000004251276133741700262130ustar00rootroot00000000000000// +build cgo // +build -darwin package openssl /* #include */ import "C" func FIPSModeSet(mode bool) error { var r C.int if mode { r = C.FIPS_mode_set(1) } else { r = C.FIPS_mode_set(0) } if r != 1 { return errorFromErrorQueue() } return nil } mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/openssl/hostname.c000066400000000000000000000250221276133741700267050ustar00rootroot00000000000000/* Go-OpenSSL notice: This file is required for all OpenSSL versions prior to 1.1.0. This simply provides the new 1.1.0 X509_check_* methods for hostname validation if they don't already exist. */ #include #ifndef X509_CHECK_FLAG_ALWAYS_CHECK_SUBJECT /* portions from x509v3.h and v3_utl.c */ /* Written by Dr Stephen N Henson (steve@openssl.org) for the OpenSSL * project. */ /* ==================================================================== * Copyright (c) 1999-2003 The OpenSSL Project. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in * the documentation and/or other materials provided with the * distribution. * * 3. All advertising materials mentioning features or use of this * software must display the following acknowledgment: * "This product includes software developed by the OpenSSL Project * for use in the OpenSSL Toolkit. (http://www.OpenSSL.org/)" * * 4. The names "OpenSSL Toolkit" and "OpenSSL Project" must not be used to * endorse or promote products derived from this software without * prior written permission. For written permission, please contact * licensing@OpenSSL.org. * * 5. Products derived from this software may not be called "OpenSSL" * nor may "OpenSSL" appear in their names without prior written * permission of the OpenSSL Project. * * 6. Redistributions of any form whatsoever must retain the following * acknowledgment: * "This product includes software developed by the OpenSSL Project * for use in the OpenSSL Toolkit (http://www.OpenSSL.org/)" * * THIS SOFTWARE IS PROVIDED BY THE OpenSSL PROJECT ``AS IS'' AND ANY * EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE OpenSSL PROJECT OR * ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED * OF THE POSSIBILITY OF SUCH DAMAGE. * ==================================================================== * * This product includes cryptographic software written by Eric Young * (eay@cryptsoft.com). This product includes software written by Tim * Hudson (tjh@cryptsoft.com). * */ /* X509 v3 extension utilities */ #include #include #include #include #define X509_CHECK_FLAG_ALWAYS_CHECK_SUBJECT 0x1 #define X509_CHECK_FLAG_NO_WILDCARDS 0x2 typedef int (*equal_fn)(const unsigned char *pattern, size_t pattern_len, const unsigned char *subject, size_t subject_len); /* Compare while ASCII ignoring case. */ static int equal_nocase(const unsigned char *pattern, size_t pattern_len, const unsigned char *subject, size_t subject_len) { if (pattern_len != subject_len) return 0; while (pattern_len) { unsigned char l = *pattern; unsigned char r = *subject; /* The pattern must not contain NUL characters. */ if (l == 0) return 0; if (l != r) { if ('A' <= l && l <= 'Z') l = (l - 'A') + 'a'; if ('A' <= r && r <= 'Z') r = (r - 'A') + 'a'; if (l != r) return 0; } ++pattern; ++subject; --pattern_len; } return 1; } /* Compare using memcmp. */ static int equal_case(const unsigned char *pattern, size_t pattern_len, const unsigned char *subject, size_t subject_len) { /* The pattern must not contain NUL characters. */ if (memchr(pattern, '\0', pattern_len) != NULL) return 0; if (pattern_len != subject_len) return 0; return !memcmp(pattern, subject, pattern_len); } /* RFC 5280, section 7.5, requires that only the domain is compared in a case-insensitive manner. */ static int equal_email(const unsigned char *a, size_t a_len, const unsigned char *b, size_t b_len) { size_t i = a_len; if (a_len != b_len) return 0; /* We search backwards for the '@' character, so that we do not have to deal with quoted local-parts. The domain part is compared in a case-insensitive manner. */ while (i > 0) { --i; if (a[i] == '@' || b[i] == '@') { if (!equal_nocase(a + i, a_len - i, b + i, a_len - i)) return 0; break; } } if (i == 0) i = a_len; return equal_case(a, i, b, i); } /* Compare the prefix and suffix with the subject, and check that the characters in-between are valid. */ static int wildcard_match(const unsigned char *prefix, size_t prefix_len, const unsigned char *suffix, size_t suffix_len, const unsigned char *subject, size_t subject_len) { const unsigned char *wildcard_start; const unsigned char *wildcard_end; const unsigned char *p; if (subject_len < prefix_len + suffix_len) return 0; if (!equal_nocase(prefix, prefix_len, subject, prefix_len)) return 0; wildcard_start = subject + prefix_len; wildcard_end = subject + (subject_len - suffix_len); if (!equal_nocase(wildcard_end, suffix_len, suffix, suffix_len)) return 0; /* The wildcard must match at least one character. */ if (wildcard_start == wildcard_end) return 0; /* Check that the part matched by the wildcard contains only permitted characters and only matches a single label. */ for (p = wildcard_start; p != wildcard_end; ++p) if (!(('0' <= *p && *p <= '9') || ('A' <= *p && *p <= 'Z') || ('a' <= *p && *p <= 'z') || *p == '-')) return 0; return 1; } /* Checks if the memory region consistens of [0-9A-Za-z.-]. */ static int valid_domain_characters(const unsigned char *p, size_t len) { while (len) { if (!(('0' <= *p && *p <= '9') || ('A' <= *p && *p <= 'Z') || ('a' <= *p && *p <= 'z') || *p == '-' || *p == '.')) return 0; ++p; --len; } return 1; } /* Find the '*' in a wildcard pattern. If no such character is found or the pattern is otherwise invalid, returns NULL. */ static const unsigned char *wildcard_find_star(const unsigned char *pattern, size_t pattern_len) { const unsigned char *star = memchr(pattern, '*', pattern_len); size_t dot_count = 0; const unsigned char *suffix_start; size_t suffix_length; if (star == NULL) return NULL; suffix_start = star + 1; suffix_length = (pattern + pattern_len) - (star + 1); if (!(valid_domain_characters(pattern, star - pattern) && valid_domain_characters(suffix_start, suffix_length))) return NULL; /* Check that the suffix matches at least two labels. */ while (suffix_length) { if (*suffix_start == '.') ++dot_count; ++suffix_start; --suffix_length; } if (dot_count < 2) return NULL; return star; } /* Compare using wildcards. */ static int equal_wildcard(const unsigned char *pattern, size_t pattern_len, const unsigned char *subject, size_t subject_len) { const unsigned char *star = wildcard_find_star(pattern, pattern_len); if (star == NULL) return equal_nocase(pattern, pattern_len, subject, subject_len); return wildcard_match(pattern, star - pattern, star + 1, (pattern + pattern_len) - star - 1, subject, subject_len); } /* Compare an ASN1_STRING to a supplied string. If they match * return 1. If cmp_type > 0 only compare if string matches the * type, otherwise convert it to UTF8. */ static int do_check_string(ASN1_STRING *a, int cmp_type, equal_fn equal, const unsigned char *b, size_t blen) { if (!a->data || !a->length) return 0; if (cmp_type > 0) { if (cmp_type != a->type) return 0; if (cmp_type == V_ASN1_IA5STRING) return equal(a->data, a->length, b, blen); if (a->length == (int)blen && !memcmp(a->data, b, blen)) return 1; else return 0; } else { int astrlen, rv; unsigned char *astr; astrlen = ASN1_STRING_to_UTF8(&astr, a); if (astrlen < 0) return -1; rv = equal(astr, astrlen, b, blen); OPENSSL_free(astr); return rv; } } static int do_x509_check(X509 *x, const unsigned char *chk, size_t chklen, unsigned int flags, int check_type) { STACK_OF(GENERAL_NAME) *gens = NULL; X509_NAME *name = NULL; int i; int cnid; int alt_type; equal_fn equal; if (check_type == GEN_EMAIL) { cnid = NID_pkcs9_emailAddress; alt_type = V_ASN1_IA5STRING; equal = equal_email; } else if (check_type == GEN_DNS) { cnid = NID_commonName; alt_type = V_ASN1_IA5STRING; if (flags & X509_CHECK_FLAG_NO_WILDCARDS) equal = equal_nocase; else equal = equal_wildcard; } else { cnid = 0; alt_type = V_ASN1_OCTET_STRING; equal = equal_case; } if (chklen == 0) chklen = strlen((const char *)chk); gens = X509_get_ext_d2i(x, NID_subject_alt_name, NULL, NULL); if (gens) { int rv = 0; for (i = 0; i < sk_GENERAL_NAME_num(gens); i++) { GENERAL_NAME *gen; ASN1_STRING *cstr; gen = sk_GENERAL_NAME_value(gens, i); if(gen->type != check_type) continue; if (check_type == GEN_EMAIL) cstr = gen->d.rfc822Name; else if (check_type == GEN_DNS) cstr = gen->d.dNSName; else cstr = gen->d.iPAddress; if (do_check_string(cstr, alt_type, equal, chk, chklen)) { rv = 1; break; } } GENERAL_NAMES_free(gens); if (rv) return 1; if (!(flags & X509_CHECK_FLAG_ALWAYS_CHECK_SUBJECT) || !cnid) return 0; } i = -1; name = X509_get_subject_name(x); while((i = X509_NAME_get_index_by_NID(name, cnid, i)) >= 0) { X509_NAME_ENTRY *ne; ASN1_STRING *str; ne = X509_NAME_get_entry(name, i); str = X509_NAME_ENTRY_get_data(ne); if (do_check_string(str, -1, equal, chk, chklen)) return 1; } return 0; } int _X509_check_host(X509 *x, const unsigned char *chk, size_t chklen, unsigned int flags) { return do_x509_check(x, chk, chklen, flags, GEN_DNS); } int _X509_check_email(X509 *x, const unsigned char *chk, size_t chklen, unsigned int flags) { return do_x509_check(x, chk, chklen, flags, GEN_EMAIL); } int _X509_check_ip(X509 *x, const unsigned char *chk, size_t chklen, unsigned int flags) { return do_x509_check(x, chk, chklen, flags, GEN_IPADD); } #endif mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/openssl/hostname.go000066400000000000000000000076011276133741700270730ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // +build cgo package openssl /* #include #include #include #ifndef X509_CHECK_FLAG_ALWAYS_CHECK_SUBJECT #define X509_CHECK_FLAG_ALWAYS_CHECK_SUBJECT 0x1 #define X509_CHECK_FLAG_NO_WILDCARDS 0x2 extern int _X509_check_host(X509 *x, const unsigned char *chk, size_t chklen, unsigned int flags); extern int _X509_check_email(X509 *x, const unsigned char *chk, size_t chklen, unsigned int flags); extern int _X509_check_ip(X509 *x, const unsigned char *chk, size_t chklen, unsigned int flags); #endif */ import "C" import ( "errors" "net" "unsafe" ) var ( ValidationError = errors.New("Host validation error") ) type CheckFlags int const ( AlwaysCheckSubject CheckFlags = C.X509_CHECK_FLAG_ALWAYS_CHECK_SUBJECT NoWildcards CheckFlags = C.X509_CHECK_FLAG_NO_WILDCARDS ) // CheckHost checks that the X509 certificate is signed for the provided // host name. See http://www.openssl.org/docs/crypto/X509_check_host.html for // more. Note that CheckHost does not check the IP field. See VerifyHostname. // Specifically returns ValidationError if the Certificate didn't match but // there was no internal error. func (c *Certificate) CheckHost(host string, flags CheckFlags) error { chost := unsafe.Pointer(C.CString(host)) defer C.free(chost) rv := C._X509_check_host(c.x, (*C.uchar)(chost), C.size_t(len(host)), C.uint(flags)) if rv > 0 { return nil } if rv == 0 { return ValidationError } return errors.New("hostname validation had an internal failure") } // CheckEmail checks that the X509 certificate is signed for the provided // email address. See http://www.openssl.org/docs/crypto/X509_check_host.html // for more. // Specifically returns ValidationError if the Certificate didn't match but // there was no internal error. func (c *Certificate) CheckEmail(email string, flags CheckFlags) error { cemail := unsafe.Pointer(C.CString(email)) defer C.free(cemail) rv := C._X509_check_email(c.x, (*C.uchar)(cemail), C.size_t(len(email)), C.uint(flags)) if rv > 0 { return nil } if rv == 0 { return ValidationError } return errors.New("email validation had an internal failure") } // CheckIP checks that the X509 certificate is signed for the provided // IP address. See http://www.openssl.org/docs/crypto/X509_check_host.html // for more. // Specifically returns ValidationError if the Certificate didn't match but // there was no internal error. func (c *Certificate) CheckIP(ip net.IP, flags CheckFlags) error { cip := unsafe.Pointer(&ip[0]) rv := C._X509_check_ip(c.x, (*C.uchar)(cip), C.size_t(len(ip)), C.uint(flags)) if rv > 0 { return nil } if rv == 0 { return ValidationError } return errors.New("ip validation had an internal failure") } // VerifyHostname is a combination of CheckHost and CheckIP. If the provided // hostname looks like an IP address, it will be checked as an IP address, // otherwise it will be checked as a hostname. // Specifically returns ValidationError if the Certificate didn't match but // there was no internal error. func (c *Certificate) VerifyHostname(host string) error { var ip net.IP if len(host) >= 3 && host[0] == '[' && host[len(host)-1] == ']' { ip = net.ParseIP(host[1 : len(host)-1]) } else { ip = net.ParseIP(host) } if ip != nil { return c.CheckIP(ip, 0) } return c.CheckHost(host, 0) } mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/openssl/http.go000066400000000000000000000041761276133741700262400ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. package openssl import ( "net/http" ) // ListenAndServeTLS will take an http.Handler and serve it using OpenSSL over // the given tcp address, configured to use the provided cert and key files. func ListenAndServeTLS(addr string, cert_file string, key_file string, handler http.Handler) error { return ServerListenAndServeTLS( &http.Server{Addr: addr, Handler: handler}, cert_file, key_file) } // ServerListenAndServeTLS will take an http.Server and serve it using OpenSSL // configured to use the provided cert and key files. func ServerListenAndServeTLS(srv *http.Server, cert_file, key_file string) error { addr := srv.Addr if addr == "" { addr = ":https" } ctx, err := NewCtxFromFiles(cert_file, key_file) if err != nil { return err } l, err := Listen("tcp", addr, ctx) if err != nil { return err } return srv.Serve(l) } // TODO: http client integration // holy crap, getting this integrated nicely with the Go stdlib HTTP client // stack so that it does proxying, connection pooling, and most importantly // hostname verification is really hard. So much stuff is hardcoded to just use // the built-in TLS lib. I think to get this to work either some crazy // hacktackery beyond me, an almost straight up fork of the HTTP client, or // serious stdlib internal refactoring is necessary. // even more so, good luck getting openssl to use the operating system default // root certificates if the user doesn't provide any. sadlol // NOTE: if you're going to try and write your own round tripper, at least use // openssl.Dial, or equivalent logic mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/openssl/init.go000066400000000000000000000114511276133741700262160ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // +build cgo /* Package openssl is a light wrapper around OpenSSL for Go. It strives to provide a near-drop-in replacement for the Go standard library tls package, while allowing for: Performance OpenSSL is battle-tested and optimized C. While Go's built-in library shows great promise, it is still young and in some places, inefficient. This simple OpenSSL wrapper can often do at least 2x with the same cipher and protocol. On my lappytop, I get the following benchmarking speeds: BenchmarkSHA1Large_openssl 1000 2611282 ns/op 401.56 MB/s BenchmarkSHA1Large_stdlib 500 3963983 ns/op 264.53 MB/s BenchmarkSHA1Small_openssl 1000000 3476 ns/op 0.29 MB/s BenchmarkSHA1Small_stdlib 5000000 550 ns/op 1.82 MB/s BenchmarkSHA256Large_openssl 200 8085314 ns/op 129.69 MB/s BenchmarkSHA256Large_stdlib 100 18948189 ns/op 55.34 MB/s BenchmarkSHA256Small_openssl 1000000 4262 ns/op 0.23 MB/s BenchmarkSHA256Small_stdlib 1000000 1444 ns/op 0.69 MB/s BenchmarkOpenSSLThroughput 100000 21634 ns/op 47.33 MB/s BenchmarkStdlibThroughput 50000 58974 ns/op 17.36 MB/s Interoperability Many systems support OpenSSL with a variety of plugins and modules for things, such as hardware acceleration in embedded devices. Greater flexibility and configuration OpenSSL allows for far greater configuration of corner cases and backwards compatibility (such as support of SSLv2). You shouldn't be using SSLv2 if you can help but, but sometimes you can't help it. Security Yeah yeah, Heartbleed. But according to the author of the standard library's TLS implementation, Go's TLS library is vulnerable to timing attacks. And whether or not OpenSSL received the appropriate amount of scrutiny pre-Heartbleed, it sure is receiving it now. Usage Starting an HTTP server that uses OpenSSL is very easy. It's as simple as: log.Fatal(openssl.ListenAndServeTLS( ":8443", "my_server.crt", "my_server.key", myHandler)) Getting a net.Listener that uses OpenSSL is also easy: ctx, err := openssl.NewCtxFromFiles("my_server.crt", "my_server.key") if err != nil { log.Fatal(err) } l, err := openssl.Listen("tcp", ":7777", ctx) Making a client connection is straightforward too: ctx, err := NewCtx() if err != nil { log.Fatal(err) } err = ctx.LoadVerifyLocations("/etc/ssl/certs/ca-certificates.crt", "") if err != nil { log.Fatal(err) } conn, err := openssl.Dial("tcp", "localhost:7777", ctx, 0) Help wanted: To get this library to work with net/http's client, we had to fork net/http. It would be nice if an alternate http client library supported the generality needed to use OpenSSL instead of crypto/tls. */ package openssl /* #include #include #include #include #include extern int Goopenssl_init_locks(); extern void Goopenssl_thread_locking_callback(int, int, const char*, int); static int Goopenssl_init_threadsafety() { // Set up OPENSSL thread safety callbacks. We only set the locking // callback because the default id callback implementation is good // enough for us. int rc = Goopenssl_init_locks(); if (rc == 0) { CRYPTO_set_locking_callback(Goopenssl_thread_locking_callback); } return rc; } static void OpenSSL_add_all_algorithms_not_a_macro() { OpenSSL_add_all_algorithms(); } */ import "C" import ( "errors" "fmt" "strings" "sync" ) var ( sslMutexes []sync.Mutex ) func init() { C.OPENSSL_config(nil) C.ENGINE_load_builtin_engines() C.SSL_load_error_strings() C.SSL_library_init() C.OpenSSL_add_all_algorithms_not_a_macro() rc := C.Goopenssl_init_threadsafety() if rc != 0 { panic(fmt.Errorf("Goopenssl_init_locks failed with %d", rc)) } } // errorFromErrorQueue needs to run in the same OS thread as the operation // that caused the possible error func errorFromErrorQueue() error { var errs []string for { err := C.ERR_get_error() if err == 0 { break } errs = append(errs, fmt.Sprintf("%s:%s:%s", C.GoString(C.ERR_lib_error_string(err)), C.GoString(C.ERR_func_error_string(err)), C.GoString(C.ERR_reason_error_string(err)))) } return errors.New(fmt.Sprintf("SSL errors: %s", strings.Join(errs, "\n"))) } mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/openssl/init_posix.go000066400000000000000000000030201276133741700274310ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // +build linux darwin cgo // +build !windows package openssl /* #include #include #include pthread_mutex_t* goopenssl_locks; int Goopenssl_init_locks() { int rc = 0; int nlock; int i; int locks_needed = CRYPTO_num_locks(); goopenssl_locks = (pthread_mutex_t*)malloc( sizeof(pthread_mutex_t) * locks_needed); if (!goopenssl_locks) { return ENOMEM; } for (nlock = 0; nlock < locks_needed; ++nlock) { rc = pthread_mutex_init(&goopenssl_locks[nlock], NULL); if (rc != 0) { break; } } if (rc != 0) { for (i = nlock - 1; i >= 0; --i) { pthread_mutex_destroy(&goopenssl_locks[i]); } free(goopenssl_locks); goopenssl_locks = NULL; } return rc; } void Goopenssl_thread_locking_callback(int mode, int n, const char *file, int line) { if (mode & CRYPTO_LOCK) { pthread_mutex_lock(&goopenssl_locks[n]); } else { pthread_mutex_unlock(&goopenssl_locks[n]); } } */ import "C" mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/openssl/init_windows.go000066400000000000000000000027471276133741700300000ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // +build windows cgo package openssl /* #cgo windows LDFLAGS: -lssleay32 -llibeay32 -L c:/openssl/bin #cgo windows CFLAGS: -I"c:/openssl/include" #ifndef WIN32_LEAN_AND_MEAN #define WIN32_LEAN_AND_MEAN #endif #include #include #include CRITICAL_SECTION* goopenssl_locks; int Goopenssl_init_locks() { int rc = 0; int nlock; int i; int locks_needed = CRYPTO_num_locks(); goopenssl_locks = (CRITICAL_SECTION*)malloc( sizeof(*goopenssl_locks) * locks_needed); if (!goopenssl_locks) { return ENOMEM; } for (nlock = 0; nlock < locks_needed; ++nlock) { InitializeCriticalSection(&goopenssl_locks[nlock]); } return 0; } void Goopenssl_thread_locking_callback(int mode, int n, const char *file, int line) { if (mode & CRYPTO_LOCK) { EnterCriticalSection(&goopenssl_locks[n]); } else { LeaveCriticalSection(&goopenssl_locks[n]); } } */ import "C" mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/openssl/key.go000066400000000000000000000241531276133741700260460ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // +build cgo package openssl // #include // #include // #include // // int EVP_SignInit_not_a_macro(EVP_MD_CTX *ctx, const EVP_MD *type) { // return EVP_SignInit(ctx, type); // } // // int EVP_SignUpdate_not_a_macro(EVP_MD_CTX *ctx, const void *d, // unsigned int cnt) { // return EVP_SignUpdate(ctx, d, cnt); // } // // int EVP_VerifyInit_not_a_macro(EVP_MD_CTX *ctx, const EVP_MD *type) { // return EVP_VerifyInit(ctx, type); // } // // int EVP_VerifyUpdate_not_a_macro(EVP_MD_CTX *ctx, const void *d, // unsigned int cnt) { // return EVP_VerifyUpdate(ctx, d, cnt); // } // // int EVP_PKEY_assign_charp(EVP_PKEY *pkey, int type, char *key) { // return EVP_PKEY_assign(pkey, type, key); // } import "C" import ( "errors" "io/ioutil" "runtime" "unsafe" ) type Method *C.EVP_MD var ( SHA256_Method Method = C.EVP_sha256() ) type PublicKey interface { // Verifies the data signature using PKCS1.15 VerifyPKCS1v15(method Method, data, sig []byte) error // MarshalPKIXPublicKeyPEM converts the public key to PEM-encoded PKIX // format MarshalPKIXPublicKeyPEM() (pem_block []byte, err error) // MarshalPKIXPublicKeyDER converts the public key to DER-encoded PKIX // format MarshalPKIXPublicKeyDER() (der_block []byte, err error) evpPKey() *C.EVP_PKEY } type PrivateKey interface { PublicKey // Signs the data using PKCS1.15 SignPKCS1v15(Method, []byte) ([]byte, error) // MarshalPKCS1PrivateKeyPEM converts the private key to PEM-encoded PKCS1 // format MarshalPKCS1PrivateKeyPEM() (pem_block []byte, err error) // MarshalPKCS1PrivateKeyDER converts the private key to DER-encoded PKCS1 // format MarshalPKCS1PrivateKeyDER() (der_block []byte, err error) } type pKey struct { key *C.EVP_PKEY } func (key *pKey) evpPKey() *C.EVP_PKEY { return key.key } func (key *pKey) SignPKCS1v15(method Method, data []byte) ([]byte, error) { var ctx C.EVP_MD_CTX C.EVP_MD_CTX_init(&ctx) defer C.EVP_MD_CTX_cleanup(&ctx) if 1 != C.EVP_SignInit_not_a_macro(&ctx, method) { return nil, errors.New("signpkcs1v15: failed to init signature") } if len(data) > 0 { if 1 != C.EVP_SignUpdate_not_a_macro( &ctx, unsafe.Pointer(&data[0]), C.uint(len(data))) { return nil, errors.New("signpkcs1v15: failed to update signature") } } sig := make([]byte, C.EVP_PKEY_size(key.key)) var sigblen C.uint if 1 != C.EVP_SignFinal(&ctx, ((*C.uchar)(unsafe.Pointer(&sig[0]))), &sigblen, key.key) { return nil, errors.New("signpkcs1v15: failed to finalize signature") } return sig[:sigblen], nil } func (key *pKey) VerifyPKCS1v15(method Method, data, sig []byte) error { var ctx C.EVP_MD_CTX C.EVP_MD_CTX_init(&ctx) defer C.EVP_MD_CTX_cleanup(&ctx) if 1 != C.EVP_VerifyInit_not_a_macro(&ctx, method) { return errors.New("verifypkcs1v15: failed to init verify") } if len(data) > 0 { if 1 != C.EVP_VerifyUpdate_not_a_macro( &ctx, unsafe.Pointer(&data[0]), C.uint(len(data))) { return errors.New("verifypkcs1v15: failed to update verify") } } if 1 != C.EVP_VerifyFinal(&ctx, ((*C.uchar)(unsafe.Pointer(&sig[0]))), C.uint(len(sig)), key.key) { return errors.New("verifypkcs1v15: failed to finalize verify") } return nil } func (key *pKey) MarshalPKCS1PrivateKeyPEM() (pem_block []byte, err error) { bio := C.BIO_new(C.BIO_s_mem()) if bio == nil { return nil, errors.New("failed to allocate memory BIO") } defer C.BIO_free(bio) rsa := (*C.RSA)(C.EVP_PKEY_get1_RSA(key.key)) if rsa == nil { return nil, errors.New("failed getting rsa key") } defer C.RSA_free(rsa) if int(C.PEM_write_bio_RSAPrivateKey(bio, rsa, nil, nil, C.int(0), nil, nil)) != 1 { return nil, errors.New("failed dumping private key") } return ioutil.ReadAll(asAnyBio(bio)) } func (key *pKey) MarshalPKCS1PrivateKeyDER() (der_block []byte, err error) { bio := C.BIO_new(C.BIO_s_mem()) if bio == nil { return nil, errors.New("failed to allocate memory BIO") } defer C.BIO_free(bio) rsa := (*C.RSA)(C.EVP_PKEY_get1_RSA(key.key)) if rsa == nil { return nil, errors.New("failed getting rsa key") } defer C.RSA_free(rsa) if int(C.i2d_RSAPrivateKey_bio(bio, rsa)) != 1 { return nil, errors.New("failed dumping private key der") } return ioutil.ReadAll(asAnyBio(bio)) } func (key *pKey) MarshalPKIXPublicKeyPEM() (pem_block []byte, err error) { bio := C.BIO_new(C.BIO_s_mem()) if bio == nil { return nil, errors.New("failed to allocate memory BIO") } defer C.BIO_free(bio) rsa := (*C.RSA)(C.EVP_PKEY_get1_RSA(key.key)) if rsa == nil { return nil, errors.New("failed getting rsa key") } defer C.RSA_free(rsa) if int(C.PEM_write_bio_RSA_PUBKEY(bio, rsa)) != 1 { return nil, errors.New("failed dumping public key pem") } return ioutil.ReadAll(asAnyBio(bio)) } func (key *pKey) MarshalPKIXPublicKeyDER() (der_block []byte, err error) { bio := C.BIO_new(C.BIO_s_mem()) if bio == nil { return nil, errors.New("failed to allocate memory BIO") } defer C.BIO_free(bio) rsa := (*C.RSA)(C.EVP_PKEY_get1_RSA(key.key)) if rsa == nil { return nil, errors.New("failed getting rsa key") } defer C.RSA_free(rsa) if int(C.i2d_RSA_PUBKEY_bio(bio, rsa)) != 1 { return nil, errors.New("failed dumping public key der") } return ioutil.ReadAll(asAnyBio(bio)) } // LoadPrivateKeyFromPEM loads a private key from a PEM-encoded block. func LoadPrivateKeyFromPEM(pem_block []byte) (PrivateKey, error) { if len(pem_block) == 0 { return nil, errors.New("empty pem block") } bio := C.BIO_new_mem_buf(unsafe.Pointer(&pem_block[0]), C.int(len(pem_block))) if bio == nil { return nil, errors.New("failed creating bio") } defer C.BIO_free(bio) rsakey := C.PEM_read_bio_RSAPrivateKey(bio, nil, nil, nil) if rsakey == nil { return nil, errors.New("failed reading rsa key") } defer C.RSA_free(rsakey) // convert to PKEY key := C.EVP_PKEY_new() if key == nil { return nil, errors.New("failed converting to evp_pkey") } if C.EVP_PKEY_set1_RSA(key, (*C.struct_rsa_st)(rsakey)) != 1 { C.EVP_PKEY_free(key) return nil, errors.New("failed converting to evp_pkey") } p := &pKey{key: key} runtime.SetFinalizer(p, func(p *pKey) { C.EVP_PKEY_free(p.key) }) return p, nil } // LoadPrivateKeyFromPEM loads a private key from a PEM-encoded block. func LoadPrivateKeyFromPEMWidthPassword(pem_block []byte, password string) ( PrivateKey, error) { if len(pem_block) == 0 { return nil, errors.New("empty pem block") } bio := C.BIO_new_mem_buf(unsafe.Pointer(&pem_block[0]), C.int(len(pem_block))) if bio == nil { return nil, errors.New("failed creating bio") } defer C.BIO_free(bio) cs := C.CString(password) defer C.free(unsafe.Pointer(cs)) rsakey := C.PEM_read_bio_RSAPrivateKey(bio, nil, nil, unsafe.Pointer(cs)) if rsakey == nil { return nil, errors.New("failed reading rsa key") } defer C.RSA_free(rsakey) // convert to PKEY key := C.EVP_PKEY_new() if key == nil { return nil, errors.New("failed converting to evp_pkey") } if C.EVP_PKEY_set1_RSA(key, (*C.struct_rsa_st)(rsakey)) != 1 { C.EVP_PKEY_free(key) return nil, errors.New("failed converting to evp_pkey") } p := &pKey{key: key} runtime.SetFinalizer(p, func(p *pKey) { C.EVP_PKEY_free(p.key) }) return p, nil } // LoadPublicKeyFromPEM loads a public key from a PEM-encoded block. func LoadPublicKeyFromPEM(pem_block []byte) (PublicKey, error) { if len(pem_block) == 0 { return nil, errors.New("empty pem block") } bio := C.BIO_new_mem_buf(unsafe.Pointer(&pem_block[0]), C.int(len(pem_block))) if bio == nil { return nil, errors.New("failed creating bio") } defer C.BIO_free(bio) rsakey := C.PEM_read_bio_RSA_PUBKEY(bio, nil, nil, nil) if rsakey == nil { return nil, errors.New("failed reading rsa key") } defer C.RSA_free(rsakey) // convert to PKEY key := C.EVP_PKEY_new() if key == nil { return nil, errors.New("failed converting to evp_pkey") } if C.EVP_PKEY_set1_RSA(key, (*C.struct_rsa_st)(rsakey)) != 1 { C.EVP_PKEY_free(key) return nil, errors.New("failed converting to evp_pkey") } p := &pKey{key: key} runtime.SetFinalizer(p, func(p *pKey) { C.EVP_PKEY_free(p.key) }) return p, nil } // LoadPublicKeyFromDER loads a public key from a DER-encoded block. func LoadPublicKeyFromDER(der_block []byte) (PublicKey, error) { if len(der_block) == 0 { return nil, errors.New("empty der block") } bio := C.BIO_new_mem_buf(unsafe.Pointer(&der_block[0]), C.int(len(der_block))) if bio == nil { return nil, errors.New("failed creating bio") } defer C.BIO_free(bio) rsakey := C.d2i_RSA_PUBKEY_bio(bio, nil) if rsakey == nil { return nil, errors.New("failed reading rsa key") } defer C.RSA_free(rsakey) // convert to PKEY key := C.EVP_PKEY_new() if key == nil { return nil, errors.New("failed converting to evp_pkey") } if C.EVP_PKEY_set1_RSA(key, (*C.struct_rsa_st)(rsakey)) != 1 { C.EVP_PKEY_free(key) return nil, errors.New("failed converting to evp_pkey") } p := &pKey{key: key} runtime.SetFinalizer(p, func(p *pKey) { C.EVP_PKEY_free(p.key) }) return p, nil } // GenerateRSAKey generates a new RSA private key with an exponent of 3. func GenerateRSAKey(bits int) (PrivateKey, error) { exponent := 3 rsa := C.RSA_generate_key(C.int(bits), C.ulong(exponent), nil, nil) if rsa == nil { return nil, errors.New("failed to generate RSA key") } key := C.EVP_PKEY_new() if key == nil { return nil, errors.New("failed to allocate EVP_PKEY") } if C.EVP_PKEY_assign_charp(key, C.EVP_PKEY_RSA, (*C.char)(unsafe.Pointer(rsa))) != 1 { C.EVP_PKEY_free(key) return nil, errors.New("failed to assign RSA key") } p := &pKey{key: key} runtime.SetFinalizer(p, func(p *pKey) { C.EVP_PKEY_free(p.key) }) return p, nil } mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/openssl/key_test.go000066400000000000000000000072151276133741700271050ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. package openssl import ( "bytes" "crypto/rsa" "crypto/tls" "crypto/x509" "encoding/hex" pem_pkg "encoding/pem" "io/ioutil" "testing" ) func TestMarshal(t *testing.T) { key, err := LoadPrivateKeyFromPEM(keyBytes) if err != nil { t.Fatal(err) } cert, err := LoadCertificateFromPEM(certBytes) if err != nil { t.Fatal(err) } pem, err := cert.MarshalPEM() if err != nil { t.Fatal(err) } if !bytes.Equal(pem, certBytes) { ioutil.WriteFile("generated", pem, 0644) ioutil.WriteFile("hardcoded", certBytes, 0644) t.Fatal("invalid cert pem bytes") } pem, err = key.MarshalPKCS1PrivateKeyPEM() if err != nil { t.Fatal(err) } if !bytes.Equal(pem, keyBytes) { ioutil.WriteFile("generated", pem, 0644) ioutil.WriteFile("hardcoded", keyBytes, 0644) t.Fatal("invalid private key pem bytes") } tls_cert, err := tls.X509KeyPair(certBytes, keyBytes) if err != nil { t.Fatal(err) } tls_key, ok := tls_cert.PrivateKey.(*rsa.PrivateKey) if !ok { t.Fatal("FASDFASDF") } _ = tls_key der, err := key.MarshalPKCS1PrivateKeyDER() if err != nil { t.Fatal(err) } tls_der := x509.MarshalPKCS1PrivateKey(tls_key) if !bytes.Equal(der, tls_der) { t.Fatal("invalid private key der bytes: %s\n v.s. %s\n", hex.Dump(der), hex.Dump(tls_der)) } der, err = key.MarshalPKIXPublicKeyDER() if err != nil { t.Fatal(err) } tls_der, err = x509.MarshalPKIXPublicKey(&tls_key.PublicKey) if err != nil { t.Fatal(err) } if !bytes.Equal(der, tls_der) { ioutil.WriteFile("generated", []byte(hex.Dump(der)), 0644) ioutil.WriteFile("hardcoded", []byte(hex.Dump(tls_der)), 0644) t.Fatal("invalid public key der bytes") } pem, err = key.MarshalPKIXPublicKeyPEM() if err != nil { t.Fatal(err) } tls_pem := pem_pkg.EncodeToMemory(&pem_pkg.Block{ Type: "PUBLIC KEY", Bytes: tls_der}) if !bytes.Equal(pem, tls_pem) { ioutil.WriteFile("generated", pem, 0644) ioutil.WriteFile("hardcoded", tls_pem, 0644) t.Fatal("invalid public key pem bytes") } loaded_pubkey_from_pem, err := LoadPublicKeyFromPEM(pem) if err != nil { t.Fatal(err) } loaded_pubkey_from_der, err := LoadPublicKeyFromDER(der) if err != nil { t.Fatal(err) } new_der_from_pem, err := loaded_pubkey_from_pem.MarshalPKIXPublicKeyDER() if err != nil { t.Fatal(err) } new_der_from_der, err := loaded_pubkey_from_der.MarshalPKIXPublicKeyDER() if err != nil { t.Fatal(err) } if !bytes.Equal(new_der_from_der, tls_der) { ioutil.WriteFile("generated", []byte(hex.Dump(new_der_from_der)), 0644) ioutil.WriteFile("hardcoded", []byte(hex.Dump(tls_der)), 0644) t.Fatal("invalid public key der bytes") } if !bytes.Equal(new_der_from_pem, tls_der) { ioutil.WriteFile("generated", []byte(hex.Dump(new_der_from_pem)), 0644) ioutil.WriteFile("hardcoded", []byte(hex.Dump(tls_der)), 0644) t.Fatal("invalid public key der bytes") } } func TestGenerate(t *testing.T) { key, err := GenerateRSAKey(2048) if err != nil { t.Fatal(err) } _, err = key.MarshalPKIXPublicKeyPEM() if err != nil { t.Fatal(err) } _, err = key.MarshalPKCS1PrivateKeyPEM() if err != nil { t.Fatal(err) } } mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/openssl/net.go000066400000000000000000000074311276133741700260440ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. package openssl import ( "errors" "net" ) type listener struct { net.Listener ctx *Ctx } func (l *listener) Accept() (c net.Conn, err error) { c, err = l.Listener.Accept() if err != nil { return nil, err } ssl_c, err := Server(c, l.ctx) if err != nil { c.Close() return nil, err } return ssl_c, nil } // NewListener wraps an existing net.Listener such that all accepted // connections are wrapped as OpenSSL server connections using the provided // context ctx. func NewListener(inner net.Listener, ctx *Ctx) net.Listener { return &listener{ Listener: inner, ctx: ctx} } // Listen is a wrapper around net.Listen that wraps incoming connections with // an OpenSSL server connection using the provided context ctx. func Listen(network, laddr string, ctx *Ctx) (net.Listener, error) { if ctx == nil { return nil, errors.New("no ssl context provided") } l, err := net.Listen(network, laddr) if err != nil { return nil, err } return NewListener(l, ctx), nil } type DialFlags int const ( InsecureSkipHostVerification DialFlags = 1 << iota DisableSNI ) // Dial will connect to network/address and then wrap the corresponding // underlying connection with an OpenSSL client connection using context ctx. // If flags includes InsecureSkipHostVerification, the server certificate's // hostname will not be checked to match the hostname in addr. Otherwise, flags // should be 0. // // Dial probably won't work for you unless you set a verify location or add // some certs to the certificate store of the client context you're using. // This library is not nice enough to use the system certificate store by // default for you yet. func Dial(network, addr string, ctx *Ctx, flags DialFlags) (*Conn, error) { return DialSession(network, addr, ctx, flags, nil) } // DialSession will connect to network/address and then wrap the corresponding // underlying connection with an OpenSSL client connection using context ctx. // If flags includes InsecureSkipHostVerification, the server certificate's // hostname will not be checked to match the hostname in addr. Otherwise, flags // should be 0. // // Dial probably won't work for you unless you set a verify location or add // some certs to the certificate store of the client context you're using. // This library is not nice enough to use the system certificate store by // default for you yet. // // If session is not nil it will be used to resume the tls state. The session // can be retrieved from the GetSession method on the Conn. func DialSession(network, addr string, ctx *Ctx, flags DialFlags, session []byte) (*Conn, error) { host, _, err := net.SplitHostPort(addr) if err != nil { return nil, err } if ctx == nil { var err error ctx, err = NewCtx() if err != nil { return nil, err } // TODO: use operating system default certificate chain? } c, err := net.Dial(network, addr) if err != nil { return nil, err } conn, err := Client(c, ctx) if err != nil { c.Close() return nil, err } // XXX removed SNI err = conn.Handshake() if err != nil { conn.Close() return nil, err } if flags&InsecureSkipHostVerification == 0 { err = conn.VerifyHostname(host) if err != nil { conn.Close() return nil, err } } return conn, nil } mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/openssl/nid.go000066400000000000000000000224031276133741700260240ustar00rootroot00000000000000// Copyright (C) 2014 Ryan Hileman // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. package openssl type NID int const ( NID_rsadsi NID = 1 NID_pkcs NID = 2 NID_md2 NID = 3 NID_md5 NID = 4 NID_rc4 NID = 5 NID_rsaEncryption NID = 6 NID_md2WithRSAEncryption NID = 7 NID_md5WithRSAEncryption NID = 8 NID_pbeWithMD2AndDES_CBC NID = 9 NID_pbeWithMD5AndDES_CBC NID = 10 NID_X500 NID = 11 NID_X509 NID = 12 NID_commonName NID = 13 NID_countryName NID = 14 NID_localityName NID = 15 NID_stateOrProvinceName NID = 16 NID_organizationName NID = 17 NID_organizationalUnitName NID = 18 NID_rsa NID = 19 NID_pkcs7 NID = 20 NID_pkcs7_data NID = 21 NID_pkcs7_signed NID = 22 NID_pkcs7_enveloped NID = 23 NID_pkcs7_signedAndEnveloped NID = 24 NID_pkcs7_digest NID = 25 NID_pkcs7_encrypted NID = 26 NID_pkcs3 NID = 27 NID_dhKeyAgreement NID = 28 NID_des_ecb NID = 29 NID_des_cfb64 NID = 30 NID_des_cbc NID = 31 NID_des_ede NID = 32 NID_des_ede3 NID = 33 NID_idea_cbc NID = 34 NID_idea_cfb64 NID = 35 NID_idea_ecb NID = 36 NID_rc2_cbc NID = 37 NID_rc2_ecb NID = 38 NID_rc2_cfb64 NID = 39 NID_rc2_ofb64 NID = 40 NID_sha NID = 41 NID_shaWithRSAEncryption NID = 42 NID_des_ede_cbc NID = 43 NID_des_ede3_cbc NID = 44 NID_des_ofb64 NID = 45 NID_idea_ofb64 NID = 46 NID_pkcs9 NID = 47 NID_pkcs9_emailAddress NID = 48 NID_pkcs9_unstructuredName NID = 49 NID_pkcs9_contentType NID = 50 NID_pkcs9_messageDigest NID = 51 NID_pkcs9_signingTime NID = 52 NID_pkcs9_countersignature NID = 53 NID_pkcs9_challengePassword NID = 54 NID_pkcs9_unstructuredAddress NID = 55 NID_pkcs9_extCertAttributes NID = 56 NID_netscape NID = 57 NID_netscape_cert_extension NID = 58 NID_netscape_data_type NID = 59 NID_des_ede_cfb64 NID = 60 NID_des_ede3_cfb64 NID = 61 NID_des_ede_ofb64 NID = 62 NID_des_ede3_ofb64 NID = 63 NID_sha1 NID = 64 NID_sha1WithRSAEncryption NID = 65 NID_dsaWithSHA NID = 66 NID_dsa_2 NID = 67 NID_pbeWithSHA1AndRC2_CBC NID = 68 NID_id_pbkdf2 NID = 69 NID_dsaWithSHA1_2 NID = 70 NID_netscape_cert_type NID = 71 NID_netscape_base_url NID = 72 NID_netscape_revocation_url NID = 73 NID_netscape_ca_revocation_url NID = 74 NID_netscape_renewal_url NID = 75 NID_netscape_ca_policy_url NID = 76 NID_netscape_ssl_server_name NID = 77 NID_netscape_comment NID = 78 NID_netscape_cert_sequence NID = 79 NID_desx_cbc NID = 80 NID_id_ce NID = 81 NID_subject_key_identifier NID = 82 NID_key_usage NID = 83 NID_private_key_usage_period NID = 84 NID_subject_alt_name NID = 85 NID_issuer_alt_name NID = 86 NID_basic_constraints NID = 87 NID_crl_number NID = 88 NID_certificate_policies NID = 89 NID_authority_key_identifier NID = 90 NID_bf_cbc NID = 91 NID_bf_ecb NID = 92 NID_bf_cfb64 NID = 93 NID_bf_ofb64 NID = 94 NID_mdc2 NID = 95 NID_mdc2WithRSA NID = 96 NID_rc4_40 NID = 97 NID_rc2_40_cbc NID = 98 NID_givenName NID = 99 NID_surname NID = 100 NID_initials NID = 101 NID_uniqueIdentifier NID = 102 NID_crl_distribution_points NID = 103 NID_md5WithRSA NID = 104 NID_serialNumber NID = 105 NID_title NID = 106 NID_description NID = 107 NID_cast5_cbc NID = 108 NID_cast5_ecb NID = 109 NID_cast5_cfb64 NID = 110 NID_cast5_ofb64 NID = 111 NID_pbeWithMD5AndCast5_CBC NID = 112 NID_dsaWithSHA1 NID = 113 NID_md5_sha1 NID = 114 NID_sha1WithRSA NID = 115 NID_dsa NID = 116 NID_ripemd160 NID = 117 NID_ripemd160WithRSA NID = 119 NID_rc5_cbc NID = 120 NID_rc5_ecb NID = 121 NID_rc5_cfb64 NID = 122 NID_rc5_ofb64 NID = 123 NID_rle_compression NID = 124 NID_zlib_compression NID = 125 NID_ext_key_usage NID = 126 NID_id_pkix NID = 127 NID_id_kp NID = 128 NID_server_auth NID = 129 NID_client_auth NID = 130 NID_code_sign NID = 131 NID_email_protect NID = 132 NID_time_stamp NID = 133 NID_ms_code_ind NID = 134 NID_ms_code_com NID = 135 NID_ms_ctl_sign NID = 136 NID_ms_sgc NID = 137 NID_ms_efs NID = 138 NID_ns_sgc NID = 139 NID_delta_crl NID = 140 NID_crl_reason NID = 141 NID_invalidity_date NID = 142 NID_sxnet NID = 143 NID_pbe_WithSHA1And128BitRC4 NID = 144 NID_pbe_WithSHA1And40BitRC4 NID = 145 NID_pbe_WithSHA1And3_Key_TripleDES_CBC NID = 146 NID_pbe_WithSHA1And2_Key_TripleDES_CBC NID = 147 NID_pbe_WithSHA1And128BitRC2_CBC NID = 148 NID_pbe_WithSHA1And40BitRC2_CBC NID = 149 NID_keyBag NID = 150 NID_pkcs8ShroudedKeyBag NID = 151 NID_certBag NID = 152 NID_crlBag NID = 153 NID_secretBag NID = 154 NID_safeContentsBag NID = 155 NID_friendlyName NID = 156 NID_localKeyID NID = 157 NID_x509Certificate NID = 158 NID_sdsiCertificate NID = 159 NID_x509Crl NID = 160 NID_pbes2 NID = 161 NID_pbmac1 NID = 162 NID_hmacWithSHA1 NID = 163 NID_id_qt_cps NID = 164 NID_id_qt_unotice NID = 165 NID_rc2_64_cbc NID = 166 NID_SMIMECapabilities NID = 167 NID_pbeWithMD2AndRC2_CBC NID = 168 NID_pbeWithMD5AndRC2_CBC NID = 169 NID_pbeWithSHA1AndDES_CBC NID = 170 NID_ms_ext_req NID = 171 NID_ext_req NID = 172 NID_name NID = 173 NID_dnQualifier NID = 174 NID_id_pe NID = 175 NID_id_ad NID = 176 NID_info_access NID = 177 NID_ad_OCSP NID = 178 NID_ad_ca_issuers NID = 179 NID_OCSP_sign NID = 180 ) mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/openssl/oracle_stubs.go000066400000000000000000000104621276133741700277410ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // +build !cgo package openssl import ( "errors" "net" "time" ) const ( SSLRecordSize = 16 * 1024 ) type Conn struct{} func Client(conn net.Conn, ctx *Ctx) (*Conn, error) func Server(conn net.Conn, ctx *Ctx) (*Conn, error) func (c *Conn) Handshake() error func (c *Conn) PeerCertificate() (*Certificate, error) func (c *Conn) Close() error func (c *Conn) Read(b []byte) (n int, err error) func (c *Conn) Write(b []byte) (written int, err error) func (c *Conn) VerifyHostname(host string) error func (c *Conn) LocalAddr() net.Addr func (c *Conn) RemoteAddr() net.Addr func (c *Conn) SetDeadline(t time.Time) error func (c *Conn) SetReadDeadline(t time.Time) error func (c *Conn) SetWriteDeadline(t time.Time) error type Ctx struct{} type SSLVersion int const ( SSLv3 SSLVersion = 0x02 TLSv1 SSLVersion = 0x03 TLSv1_1 SSLVersion = 0x04 TLSv1_2 SSLVersion = 0x05 AnyVersion SSLVersion = 0x06 ) func NewCtxWithVersion(version SSLVersion) (*Ctx, error) func NewCtx() (*Ctx, error) func NewCtxFromFiles(cert_file string, key_file string) (*Ctx, error) func (c *Ctx) UseCertificate(cert *Certificate) error func (c *Ctx) UsePrivateKey(key PrivateKey) error type CertificateStore struct{} func (c *Ctx) GetCertificateStore() *CertificateStore func (s *CertificateStore) AddCertificate(cert *Certificate) error func (c *Ctx) LoadVerifyLocations(ca_file string, ca_path string) error type Options int const ( NoCompression Options = 0 NoSSLv2 Options = 0 NoSSLv3 Options = 0 NoTLSv1 Options = 0 CipherServerPreference Options = 0 NoSessionResumptionOrRenegotiation Options = 0 NoTicket Options = 0 ) func (c *Ctx) SetOptions(options Options) Options type Modes int const ( ReleaseBuffers Modes = 0 ) func (c *Ctx) SetMode(modes Modes) Modes type VerifyOptions int const ( VerifyNone VerifyOptions = 0 VerifyPeer VerifyOptions = 0 VerifyFailIfNoPeerCert VerifyOptions = 0 VerifyClientOnce VerifyOptions = 0 ) func (c *Ctx) SetVerify(options VerifyOptions) func (c *Ctx) SetVerifyDepth(depth int) func (c *Ctx) SetSessionId(session_id []byte) error func (c *Ctx) SetCipherList(list string) error type SessionCacheModes int const ( SessionCacheOff SessionCacheModes = 0 SessionCacheClient SessionCacheModes = 0 SessionCacheServer SessionCacheModes = 0 SessionCacheBoth SessionCacheModes = 0 NoAutoClear SessionCacheModes = 0 NoInternalLookup SessionCacheModes = 0 NoInternalStore SessionCacheModes = 0 NoInternal SessionCacheModes = 0 ) func (c *Ctx) SetSessionCacheMode(modes SessionCacheModes) SessionCacheModes var ( ValidationError = errors.New("Host validation error") ) type CheckFlags int const ( AlwaysCheckSubject CheckFlags = 0 NoWildcards CheckFlags = 0 ) func (c *Certificate) CheckHost(host string, flags CheckFlags) error func (c *Certificate) CheckEmail(email string, flags CheckFlags) error func (c *Certificate) CheckIP(ip net.IP, flags CheckFlags) error func (c *Certificate) VerifyHostname(host string) error type PublicKey interface { MarshalPKIXPublicKeyPEM() (pem_block []byte, err error) MarshalPKIXPublicKeyDER() (der_block []byte, err error) evpPKey() struct{} } type PrivateKey interface { PublicKey MarshalPKCS1PrivateKeyPEM() (pem_block []byte, err error) MarshalPKCS1PrivateKeyDER() (der_block []byte, err error) } func LoadPrivateKeyFromPEM(pem_block []byte) (PrivateKey, error) type Certificate struct{} func LoadCertificateFromPEM(pem_block []byte) (*Certificate, error) func (c *Certificate) MarshalPEM() (pem_block []byte, err error) func (c *Certificate) PublicKey() (PublicKey, error) mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/openssl/password.c000066400000000000000000000003671276133741700267360ustar00rootroot00000000000000#include #include "_cgo_export.h" int password_cb(char *buf,int buf_len, int rwflag,void *userdata) { char* pw = (char *)userdata; int l = strlen(pw); if (l + 1 > buf_len) return 0; strcpy(buf,pw); return l; } mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/openssl/pem.go000066400000000000000000000016451276133741700260400ustar00rootroot00000000000000// Copyright (C) 2014 Ryan Hileman // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. package openssl import ( "regexp" ) var pemSplit *regexp.Regexp = regexp.MustCompile(`(?sm)` + `(^-----[\s-]*?BEGIN.*?-----$` + `.*?` + `^-----[\s-]*?END.*?-----$)`) func SplitPEM(data []byte) [][]byte { var results [][]byte for _, block := range pemSplit.FindAll(data, -1) { results = append(results, block) } return results } mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/openssl/sha1.go000066400000000000000000000043371276133741700261140ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // +build cgo package openssl /* #include #include #include #include #include "openssl/evp.h" */ import "C" import ( "errors" "runtime" "unsafe" ) type SHA1Hash struct { ctx C.EVP_MD_CTX engine *Engine } func NewSHA1Hash() (*SHA1Hash, error) { return NewSHA1HashWithEngine(nil) } func NewSHA1HashWithEngine(e *Engine) (*SHA1Hash, error) { hash := &SHA1Hash{engine: e} C.EVP_MD_CTX_init(&hash.ctx) runtime.SetFinalizer(hash, func(hash *SHA1Hash) { hash.Close() }) if err := hash.Reset(); err != nil { return nil, err } return hash, nil } func (s *SHA1Hash) Close() { C.EVP_MD_CTX_cleanup(&s.ctx) } func engineRef(e *Engine) *C.ENGINE { if e == nil { return nil } return e.e } func (s *SHA1Hash) Reset() error { if 1 != C.EVP_DigestInit_ex(&s.ctx, C.EVP_sha1(), engineRef(s.engine)) { return errors.New("openssl: sha1: cannot init digest ctx") } return nil } func (s *SHA1Hash) Write(p []byte) (n int, err error) { if len(p) == 0 { return 0, nil } if 1 != C.EVP_DigestUpdate(&s.ctx, unsafe.Pointer(&p[0]), C.size_t(len(p))) { return 0, errors.New("openssl: sha1: cannot update digest") } return len(p), nil } func (s *SHA1Hash) Sum() (result [20]byte, err error) { if 1 != C.EVP_DigestFinal_ex(&s.ctx, (*C.uchar)(unsafe.Pointer(&result[0])), nil) { return result, errors.New("openssl: sha1: cannot finalize ctx") } return result, s.Reset() } func SHA1(data []byte) (result [20]byte, err error) { hash, err := NewSHA1Hash() if err != nil { return result, err } defer hash.Close() if _, err := hash.Write(data); err != nil { return result, err } return hash.Sum() } mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/openssl/sha1_test.go000066400000000000000000000045161276133741700271520ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // +build cgo package openssl import ( "crypto/rand" "crypto/sha1" "io" "testing" ) func TestSHA1(t *testing.T) { for i := 0; i < 100; i++ { buf := make([]byte, 10*1024-i) if _, err := io.ReadFull(rand.Reader, buf); err != nil { t.Fatal(err) } expected := sha1.Sum(buf) got, err := SHA1(buf) if err != nil { t.Fatal(err) } if expected != got { t.Fatal("exp:%x got:%x", expected, got) } } } func TestSHA1Writer(t *testing.T) { ohash, err := NewSHA1Hash() if err != nil { t.Fatal(err) } hash := sha1.New() for i := 0; i < 100; i++ { if err := ohash.Reset(); err != nil { t.Fatal(err) } hash.Reset() buf := make([]byte, 10*1024-i) if _, err := io.ReadFull(rand.Reader, buf); err != nil { t.Fatal(err) } if _, err := ohash.Write(buf); err != nil { t.Fatal(err) } if _, err := hash.Write(buf); err != nil { t.Fatal(err) } var got, exp [20]byte hash.Sum(exp[:0]) got, err := ohash.Sum() if err != nil { t.Fatal(err) } if got != exp { t.Fatal("exp:%x got:%x", exp, got) } } } type shafunc func([]byte) func benchmarkSHA1(b *testing.B, length int64, fn shafunc) { buf := make([]byte, length) if _, err := io.ReadFull(rand.Reader, buf); err != nil { b.Fatal(err) } b.SetBytes(length) b.ResetTimer() for i := 0; i < b.N; i++ { fn(buf) } } func BenchmarkSHA1Large_openssl(b *testing.B) { benchmarkSHA1(b, 1024*1024, func(buf []byte) { SHA1(buf) }) } func BenchmarkSHA1Large_stdlib(b *testing.B) { benchmarkSHA1(b, 1024*1024, func(buf []byte) { sha1.Sum(buf) }) } func BenchmarkSHA1Small_openssl(b *testing.B) { benchmarkSHA1(b, 1, func(buf []byte) { SHA1(buf) }) } func BenchmarkSHA1Small_stdlib(b *testing.B) { benchmarkSHA1(b, 1, func(buf []byte) { sha1.Sum(buf) }) } mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/openssl/sha256.go000066400000000000000000000042571276133741700262710ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // +build cgo package openssl /* #include #include #include #include #include "openssl/evp.h" */ import "C" import ( "errors" "runtime" "unsafe" ) type SHA256Hash struct { ctx C.EVP_MD_CTX engine *Engine } func NewSHA256Hash() (*SHA256Hash, error) { return NewSHA256HashWithEngine(nil) } func NewSHA256HashWithEngine(e *Engine) (*SHA256Hash, error) { hash := &SHA256Hash{engine: e} C.EVP_MD_CTX_init(&hash.ctx) runtime.SetFinalizer(hash, func(hash *SHA256Hash) { hash.Close() }) if err := hash.Reset(); err != nil { return nil, err } return hash, nil } func (s *SHA256Hash) Close() { C.EVP_MD_CTX_cleanup(&s.ctx) } func (s *SHA256Hash) Reset() error { if 1 != C.EVP_DigestInit_ex(&s.ctx, C.EVP_sha256(), engineRef(s.engine)) { return errors.New("openssl: sha256: cannot init digest ctx") } return nil } func (s *SHA256Hash) Write(p []byte) (n int, err error) { if len(p) == 0 { return 0, nil } if 1 != C.EVP_DigestUpdate(&s.ctx, unsafe.Pointer(&p[0]), C.size_t(len(p))) { return 0, errors.New("openssl: sha256: cannot update digest") } return len(p), nil } func (s *SHA256Hash) Sum() (result [32]byte, err error) { if 1 != C.EVP_DigestFinal_ex(&s.ctx, (*C.uchar)(unsafe.Pointer(&result[0])), nil) { return result, errors.New("openssl: sha256: cannot finalize ctx") } return result, s.Reset() } func SHA256(data []byte) (result [32]byte, err error) { hash, err := NewSHA256Hash() if err != nil { return result, err } defer hash.Close() if _, err := hash.Write(data); err != nil { return result, err } return hash.Sum() } mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/openssl/sha256_test.go000066400000000000000000000045441276133741700273270ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // +build cgo package openssl import ( "crypto/rand" "crypto/sha256" "io" "testing" ) func TestSHA256(t *testing.T) { for i := 0; i < 100; i++ { buf := make([]byte, 10*1024-i) if _, err := io.ReadFull(rand.Reader, buf); err != nil { t.Fatal(err) } expected := sha256.Sum256(buf) got, err := SHA256(buf) if err != nil { t.Fatal(err) } if expected != got { t.Fatal("exp:%x got:%x", expected, got) } } } func TestSHA256Writer(t *testing.T) { ohash, err := NewSHA256Hash() if err != nil { t.Fatal(err) } hash := sha256.New() for i := 0; i < 100; i++ { if err := ohash.Reset(); err != nil { t.Fatal(err) } hash.Reset() buf := make([]byte, 10*1024-i) if _, err := io.ReadFull(rand.Reader, buf); err != nil { t.Fatal(err) } if _, err := ohash.Write(buf); err != nil { t.Fatal(err) } if _, err := hash.Write(buf); err != nil { t.Fatal(err) } var got, exp [32]byte hash.Sum(exp[:0]) got, err := ohash.Sum() if err != nil { t.Fatal(err) } if got != exp { t.Fatal("exp:%x got:%x", exp, got) } } } func benchmarkSHA256(b *testing.B, length int64, fn shafunc) { buf := make([]byte, length) if _, err := io.ReadFull(rand.Reader, buf); err != nil { b.Fatal(err) } b.SetBytes(length) b.ResetTimer() for i := 0; i < b.N; i++ { fn(buf) } } func BenchmarkSHA256Large_openssl(b *testing.B) { benchmarkSHA256(b, 1024*1024, func(buf []byte) { SHA256(buf) }) } func BenchmarkSHA256Large_stdlib(b *testing.B) { benchmarkSHA256(b, 1024*1024, func(buf []byte) { sha256.Sum256(buf) }) } func BenchmarkSHA256Small_openssl(b *testing.B) { benchmarkSHA256(b, 1, func(buf []byte) { SHA256(buf) }) } func BenchmarkSHA256Small_stdlib(b *testing.B) { benchmarkSHA256(b, 1, func(buf []byte) { sha256.Sum256(buf) }) } mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/openssl/sni.c000066400000000000000000000015401276133741700256570ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. #include #include "_cgo_export.h" #include int sni_cb(SSL *con, int *ad, void *arg) { SSL_CTX* ssl_ctx = ssl_ctx = SSL_get_SSL_CTX(con); void* p = SSL_CTX_get_ex_data(ssl_ctx, get_ssl_ctx_idx()); return sni_cb_thunk(p, con, ad, arg); } mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/openssl/sni_test.go000066400000000000000000000015611276133741700271040ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. package openssl import "fmt" // We can implemant SNI rfc6066 (http://tools.ietf.org/html/rfc6066) on the server side using foolowing callback. // You should implement context storage (tlsCtxStorage) by your self. func ExampleSetTLSExtServernameCallback() { fmt.Println("Hello") } mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/openssl/ssl.go000066400000000000000000000112461276133741700260560ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // +build cgo package openssl /* #include #include #include #include static long SSL_set_options_not_a_macro(SSL* ssl, long options) { return SSL_set_options(ssl, options); } static long SSL_get_options_not_a_macro(SSL* ssl) { return SSL_get_options(ssl); } static long SSL_clear_options_not_a_macro(SSL* ssl, long options) { return SSL_clear_options(ssl, options); } extern int verify_ssl_cb(int ok, X509_STORE_CTX* store); */ import "C" import ( "os" "unsafe" ) type SSLTLSExtErr int var ( ssl_idx = C.SSL_get_ex_new_index(0, nil, nil, nil, nil) ) //export get_ssl_idx func get_ssl_idx() C.int { return ssl_idx } type SSL struct { ssl *C.SSL verify_cb VerifyCallback } //export verify_ssl_cb_thunk func verify_ssl_cb_thunk(p unsafe.Pointer, ok C.int, ctx *C.X509_STORE_CTX) C.int { defer func() { if err := recover(); err != nil { logger.Critf("openssl: verify callback panic'd: %v", err) os.Exit(1) } }() verify_cb := (*SSL)(p).verify_cb // set up defaults just in case verify_cb is nil if verify_cb != nil { store := &CertificateStoreCtx{ctx: ctx} if verify_cb(ok == 1, store) { ok = 1 } else { ok = 0 } } return ok } // GetOptions returns SSL options. See // https://www.openssl.org/docs/ssl/SSL_CTX_set_options.html func (s *SSL) GetOptions() Options { return Options(C.SSL_get_options_not_a_macro(s.ssl)) } // SetOptions sets SSL options. See // https://www.openssl.org/docs/ssl/SSL_CTX_set_options.html func (s *SSL) SetOptions(options Options) Options { return Options(C.SSL_set_options_not_a_macro(s.ssl, C.long(options))) } // ClearOptions clear SSL options. See // https://www.openssl.org/docs/ssl/SSL_CTX_set_options.html func (s *SSL) ClearOptions(options Options) Options { return Options(C.SSL_clear_options_not_a_macro(s.ssl, C.long(options))) } // SetVerify controls peer verification settings. See // http://www.openssl.org/docs/ssl/SSL_CTX_set_verify.html func (s *SSL) SetVerify(options VerifyOptions, verify_cb VerifyCallback) { s.verify_cb = verify_cb if verify_cb != nil { C.SSL_set_verify(s.ssl, C.int(options), (*[0]byte)(C.verify_ssl_cb)) } else { C.SSL_set_verify(s.ssl, C.int(options), nil) } } // SetVerifyMode controls peer verification setting. See // http://www.openssl.org/docs/ssl/SSL_CTX_set_verify.html func (s *SSL) SetVerifyMode(options VerifyOptions) { s.SetVerify(options, s.verify_cb) } // SetVerifyCallback controls peer verification setting. See // http://www.openssl.org/docs/ssl/SSL_CTX_set_verify.html func (s *SSL) SetVerifyCallback(verify_cb VerifyCallback) { s.SetVerify(s.VerifyMode(), s.verify_cb) } // GetVerifyCallback returns callback function. See // http://www.openssl.org/docs/ssl/SSL_CTX_set_verify.html func (s *SSL) GetVerifyCallback() VerifyCallback { return s.verify_cb } // VerifyMode returns peer verification setting. See // http://www.openssl.org/docs/ssl/SSL_CTX_set_verify.html func (s *SSL) VerifyMode() VerifyOptions { return VerifyOptions(C.SSL_get_verify_mode(s.ssl)) } // SetVerifyDepth controls how many certificates deep the certificate // verification logic is willing to follow a certificate chain. See // https://www.openssl.org/docs/ssl/SSL_CTX_set_verify.html func (s *SSL) SetVerifyDepth(depth int) { C.SSL_set_verify_depth(s.ssl, C.int(depth)) } // GetVerifyDepth controls how many certificates deep the certificate // verification logic is willing to follow a certificate chain. See // https://www.openssl.org/docs/ssl/SSL_CTX_set_verify.html func (s *SSL) GetVerifyDepth() int { return int(C.SSL_get_verify_depth(s.ssl)) } //export sni_cb_thunk func sni_cb_thunk(p unsafe.Pointer, con *C.SSL, ad unsafe.Pointer, arg unsafe.Pointer) C.int { defer func() { if err := recover(); err != nil { logger.Critf("openssl: verify callback sni panic'd: %v", err) os.Exit(1) } }() sni_cb := (*Ctx)(p).sni_cb s := &SSL{ssl: con} // This attaches a pointer to our SSL struct into the SNI callback. C.SSL_set_ex_data(s.ssl, get_ssl_idx(), unsafe.Pointer(s)) // Note: this is ctx.sni_cb, not C.sni_cb return C.int(sni_cb(s)) } mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/openssl/ssl_test.go000066400000000000000000000372441276133741700271230ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. package openssl import ( "bytes" "crypto/rand" "crypto/tls" "io" "io/ioutil" "net" "sync" "testing" "time" "github.com/spacemonkeygo/openssl/utils" ) var ( certBytes = []byte(`-----BEGIN CERTIFICATE----- MIIDxDCCAqygAwIBAgIVAMcK/0VWQr2O3MNfJCydqR7oVELcMA0GCSqGSIb3DQEB BQUAMIGQMUkwRwYDVQQDE0A1NjdjZGRmYzRjOWZiNTYwZTk1M2ZlZjA1N2M0NGFm MDdiYjc4MDIzODIxYTA5NThiY2RmMGMwNzJhOTdiMThhMQswCQYDVQQGEwJVUzEN MAsGA1UECBMEVXRhaDEQMA4GA1UEBxMHTWlkdmFsZTEVMBMGA1UEChMMU3BhY2Ug TW9ua2V5MB4XDTEzMTIxNzE4MzgyMloXDTIzMTIxNTE4MzgyMlowgZAxSTBHBgNV BAMTQDM4NTg3ODRkMjU1NTdiNTM1MWZmNjRmMmQzMTQ1ZjkwYTJlMTIzMDM4Y2Yz Mjc1Yzg1OTM1MjcxYWIzMmNiMDkxCzAJBgNVBAYTAlVTMQ0wCwYDVQQIEwRVdGFo MRAwDgYDVQQHEwdNaWR2YWxlMRUwEwYDVQQKEwxTcGFjZSBNb25rZXkwggEiMA0G CSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDdf3icNvFsrlrnNLi8SocscqlSbFq+ pEvmhcSoqgDLqebnqu8Ld73HJJ74MGXEgRX8xZT5FinOML31CR6t9E/j3dqV6p+G fdlFLe3IqtC0/bPVnCDBirBygBI4uCrMq+1VhAxPWclrDo7l9QRYbsExH9lfn+Ry vxeNMZiOASasvVZNncY8E9usBGRdH17EfDL/TPwXqWOLyxSN5o54GTztjjy9w9CG QP7jcCueKYyQJQCtEmnwc6P/q6/EPv5R6drBkX6loAPtmCUAkHqxkWOJrRq/v7Pw zRYhfY+ZpVHGc7WEkDnLzRiUypr1C9oxvLKS10etZEIwEdKyOkSg2fdPAgMBAAGj EzARMA8GA1UdEwEB/wQFMAMCAQAwDQYJKoZIhvcNAQEFBQADggEBAEcz0RTTJ99l HTK/zTyfV5VZEhtwqu6bwre/hD7lhI+1ji0DZYGIgCbJLKuZhj+cHn2h5nPhN7zE M9tc4pn0TgeVS0SVFSe6TGnIFipNogvP17E+vXpDZcW/xn9kPKeVCZc1hlDt1W4Z 5q+ub3aUwuMwYs7bcArtDrumCmciJ3LFyNhebPi4mntb5ooeLFLaujEmVYyrQnpo tWKC9sMlJmLm4yAso64Sv9KLS2T9ivJBNn0ZtougozBCCTqrqgZVjha+B2yjHe9f sRkg/uxcJf7wC5Y0BLlp1+aPwdmZD87T3a1uQ1Ij93jmHG+2T9U20MklHAePOl0q yTqdSPnSH1c= -----END CERTIFICATE----- `) keyBytes = []byte(`-----BEGIN RSA PRIVATE KEY----- MIIEpQIBAAKCAQEA3X94nDbxbK5a5zS4vEqHLHKpUmxavqRL5oXEqKoAy6nm56rv C3e9xySe+DBlxIEV/MWU+RYpzjC99QkerfRP493aleqfhn3ZRS3tyKrQtP2z1Zwg wYqwcoASOLgqzKvtVYQMT1nJaw6O5fUEWG7BMR/ZX5/kcr8XjTGYjgEmrL1WTZ3G PBPbrARkXR9exHwy/0z8F6lji8sUjeaOeBk87Y48vcPQhkD+43ArnimMkCUArRJp 8HOj/6uvxD7+UenawZF+paAD7ZglAJB6sZFjia0av7+z8M0WIX2PmaVRxnO1hJA5 y80YlMqa9QvaMbyyktdHrWRCMBHSsjpEoNn3TwIDAQABAoIBAQCwgp6YzmgCFce3 LBpzYmjqEM3CMzr1ZXRe1gbr6d4Mbu7leyBX4SpJAnP0kIzo1X2yG7ol7XWPLOST 2pqqQWFQ00EX6wsJYEy+hmVRXl5HfU3MUkkAMwd9l3Xt4UWqKPBPD5XHvmN2fvl9 Y4388vXdseXGAGNK1eFs0TMjJuOtDxDyrmJcnxpJ7y/77y/Hb5rUa9DCvj8tkKHg HmeIwQE0HhIFofj+qCYbqeVyjbPAaYZMrISXb2HmcyULKEOGRbMH24IzInKA0NxV kdP9qmV8Y2bJ609Fft/y8Vpj31iEdq/OFXyobdVvnXMnaVyAetoaWy7AOTIQ2Cnw wGbJ/F8BAoGBAN/pCnLQrWREeVMuFjf+MgYgCtRRaQ8EOVvjYcXXi0PhtOMFTAb7 djqhlgmBOFsmeXcb8YRZsF+pNtu1xk5RJOquyKfK8j1rUdAJfoxGHiaUFI2/1i9E zuXX/Ao0xNRkWMxMKuwYBmmt1fMuVo+1M8UEwFMdHRtgxe+/+eOV1J2PAoGBAP09 7GLOYSYAI1OO3BN/bEVNau6tAxP5YShGmX2Qxy0+ooxHZ1V3D8yo6C0hSg+H+fPT mjMgGcvaW6K+QyCdHDjgbk2hfdZ+Beq92JApPrH9gMV7MPhwHzgwjzDDio9OFxYY 3vjBQ2yX+9jvz9lkvq2NM3fqFqbsG6Et+5mCc6pBAoGBAI62bxVtEgbladrtdfXs S6ABzkUzOl362EBL9iZuUnJKqstDtgiBQALwuLuIJA5cwHB9W/t6WuMt7CwveJy0 NW5rRrNDtBAXlgad9o2bp135ZfxO+EoadjCi8B7lMUsaRkq4hWcDjRrQVJxxvXRN DxkVBSw0Uzf+/0nnN3OqLODbAoGACCY+/isAC1YDzQOS53m5RT2pjEa7C6CB1Ob4 t4a6MiWK25LMq35qXr6swg8JMBjDHWqY0r5ctievvTv8Mwd7SgVG526j+wwRKq2z U2hQYS/0Peap+8S37Hn7kakpQ1VS/t4MBttJTSxS6XdGLAvG6xTZLCm3UuXUOcqe ByGgkUECgYEAmop45kRi974g4MPvyLplcE4syb19ifrHj76gPRBi94Cp8jZosY1T ucCCa4lOGgPtXJ0Qf1c8yq5vh4yqkQjrgUTkr+CFDGR6y4CxmNDQxEMYIajaIiSY qmgvgyRayemfO2zR0CPgC6wSoGBth+xW6g+WA8y0z76ZSaWpFi8lVM4= -----END RSA PRIVATE KEY----- `) ) func NetPipe(t testing.TB) (net.Conn, net.Conn) { l, err := net.Listen("tcp", "localhost:0") if err != nil { t.Fatal(err) } defer l.Close() client_future := utils.NewFuture() go func() { client_future.Set(net.Dial(l.Addr().Network(), l.Addr().String())) }() var errs utils.ErrorGroup server_conn, err := l.Accept() errs.Add(err) client_conn, err := client_future.Get() errs.Add(err) err = errs.Finalize() if err != nil { if server_conn != nil { server_conn.Close() } if client_conn != nil { client_conn.(net.Conn).Close() } t.Fatal(err) } return server_conn, client_conn.(net.Conn) } type HandshakingConn interface { net.Conn Handshake() error } func SimpleConnTest(t testing.TB, constructor func( t testing.TB, conn1, conn2 net.Conn) (sslconn1, sslconn2 HandshakingConn)) { server_conn, client_conn := NetPipe(t) defer server_conn.Close() defer client_conn.Close() data := "first test string\n" server, client := constructor(t, server_conn, client_conn) defer close_both(server, client) var wg sync.WaitGroup wg.Add(2) go func() { defer wg.Done() err := client.Handshake() if err != nil { t.Fatal(err) } _, err = io.Copy(client, bytes.NewReader([]byte(data))) if err != nil { t.Fatal(err) } err = client.Close() if err != nil { t.Fatal(err) } }() go func() { defer wg.Done() err := server.Handshake() if err != nil { t.Fatal(err) } buf := bytes.NewBuffer(make([]byte, 0, len(data))) _, err = io.CopyN(buf, server, int64(len(data))) if err != nil { t.Fatal(err) } if string(buf.Bytes()) != data { t.Fatal("mismatched data") } err = server.Close() if err != nil { t.Fatal(err) } }() wg.Wait() } func close_both(closer1, closer2 io.Closer) { var wg sync.WaitGroup wg.Add(2) go func() { defer wg.Done() closer1.Close() }() go func() { defer wg.Done() closer2.Close() }() wg.Wait() } func ClosingTest(t testing.TB, constructor func( t testing.TB, conn1, conn2 net.Conn) (sslconn1, sslconn2 HandshakingConn)) { run_test := func(close_tcp bool, server_writes bool) { server_conn, client_conn := NetPipe(t) defer server_conn.Close() defer client_conn.Close() server, client := constructor(t, server_conn, client_conn) defer close_both(server, client) var sslconn1, sslconn2 HandshakingConn var conn1 net.Conn if server_writes { sslconn1 = server conn1 = server_conn sslconn2 = client } else { sslconn1 = client conn1 = client_conn sslconn2 = server } var wg sync.WaitGroup wg.Add(2) go func() { defer wg.Done() _, err := sslconn1.Write([]byte("hello")) if err != nil { t.Fatal(err) } if close_tcp { err = conn1.Close() } else { err = sslconn1.Close() } if err != nil { t.Fatal(err) } }() go func() { defer wg.Done() data, err := ioutil.ReadAll(sslconn2) if err != nil { t.Fatal(err) } if !bytes.Equal(data, []byte("hello")) { t.Fatal("bytes don't match") } }() wg.Wait() } run_test(true, false) run_test(false, false) run_test(true, true) run_test(false, true) } func ThroughputBenchmark(b *testing.B, constructor func( t testing.TB, conn1, conn2 net.Conn) (sslconn1, sslconn2 HandshakingConn)) { server_conn, client_conn := NetPipe(b) defer server_conn.Close() defer client_conn.Close() server, client := constructor(b, server_conn, client_conn) defer close_both(server, client) b.SetBytes(1024) data := make([]byte, b.N*1024) _, err := io.ReadFull(rand.Reader, data[:]) if err != nil { b.Fatal(err) } b.ResetTimer() var wg sync.WaitGroup wg.Add(2) go func() { defer wg.Done() _, err = io.Copy(client, bytes.NewReader([]byte(data))) if err != nil { b.Fatal(err) } }() go func() { defer wg.Done() buf := &bytes.Buffer{} _, err = io.CopyN(buf, server, int64(len(data))) if err != nil { b.Fatal(err) } if !bytes.Equal(buf.Bytes(), data) { b.Fatal("mismatched data") } }() wg.Wait() b.StopTimer() } func StdlibConstructor(t testing.TB, server_conn, client_conn net.Conn) ( server, client HandshakingConn) { cert, err := tls.X509KeyPair(certBytes, keyBytes) if err != nil { t.Fatal(err) } config := &tls.Config{ Certificates: []tls.Certificate{cert}, InsecureSkipVerify: true, CipherSuites: []uint16{tls.TLS_RSA_WITH_AES_128_CBC_SHA}} server = tls.Server(server_conn, config) client = tls.Client(client_conn, config) return server, client } func passThruVerify(t testing.TB) func(bool, *CertificateStoreCtx) bool { x := func(ok bool, store *CertificateStoreCtx) bool { cert := store.GetCurrentCert() if cert == nil { t.Fatalf("Could not obtain cert from store\n") } sn := cert.GetSerialNumberHex() if len(sn) == 0 { t.Fatalf("Could not obtain serial number from cert") } return ok } return x } func OpenSSLConstructor(t testing.TB, server_conn, client_conn net.Conn) ( server, client HandshakingConn) { ctx, err := NewCtx() if err != nil { t.Fatal(err) } ctx.SetVerify(VerifyNone, passThruVerify(t)) key, err := LoadPrivateKeyFromPEM(keyBytes) if err != nil { t.Fatal(err) } err = ctx.UsePrivateKey(key) if err != nil { t.Fatal(err) } cert, err := LoadCertificateFromPEM(certBytes) if err != nil { t.Fatal(err) } err = ctx.UseCertificate(cert) if err != nil { t.Fatal(err) } err = ctx.SetCipherList("AES128-SHA") if err != nil { t.Fatal(err) } server, err = Server(server_conn, ctx) if err != nil { t.Fatal(err) } client, err = Client(client_conn, ctx) if err != nil { t.Fatal(err) } return server, client } func StdlibOpenSSLConstructor(t testing.TB, server_conn, client_conn net.Conn) ( server, client HandshakingConn) { server_std, _ := StdlibConstructor(t, server_conn, client_conn) _, client_ssl := OpenSSLConstructor(t, server_conn, client_conn) return server_std, client_ssl } func OpenSSLStdlibConstructor(t testing.TB, server_conn, client_conn net.Conn) ( server, client HandshakingConn) { _, client_std := StdlibConstructor(t, server_conn, client_conn) server_ssl, _ := OpenSSLConstructor(t, server_conn, client_conn) return server_ssl, client_std } func TestStdlibSimple(t *testing.T) { SimpleConnTest(t, StdlibConstructor) } func TestOpenSSLSimple(t *testing.T) { SimpleConnTest(t, OpenSSLConstructor) } func TestStdlibClosing(t *testing.T) { ClosingTest(t, StdlibConstructor) } func TestOpenSSLClosing(t *testing.T) { ClosingTest(t, OpenSSLConstructor) } func BenchmarkStdlibThroughput(b *testing.B) { ThroughputBenchmark(b, StdlibConstructor) } func BenchmarkOpenSSLThroughput(b *testing.B) { ThroughputBenchmark(b, OpenSSLConstructor) } func TestStdlibOpenSSLSimple(t *testing.T) { SimpleConnTest(t, StdlibOpenSSLConstructor) } func TestOpenSSLStdlibSimple(t *testing.T) { SimpleConnTest(t, OpenSSLStdlibConstructor) } func TestStdlibOpenSSLClosing(t *testing.T) { ClosingTest(t, StdlibOpenSSLConstructor) } func TestOpenSSLStdlibClosing(t *testing.T) { ClosingTest(t, OpenSSLStdlibConstructor) } func BenchmarkStdlibOpenSSLThroughput(b *testing.B) { ThroughputBenchmark(b, StdlibOpenSSLConstructor) } func BenchmarkOpenSSLStdlibThroughput(b *testing.B) { ThroughputBenchmark(b, OpenSSLStdlibConstructor) } func FullDuplexRenegotiationTest(t testing.TB, constructor func( t testing.TB, conn1, conn2 net.Conn) (sslconn1, sslconn2 HandshakingConn)) { server_conn, client_conn := NetPipe(t) defer server_conn.Close() defer client_conn.Close() times := 256 data_len := 4 * SSLRecordSize data1 := make([]byte, data_len) _, err := io.ReadFull(rand.Reader, data1[:]) if err != nil { t.Fatal(err) } data2 := make([]byte, data_len) _, err = io.ReadFull(rand.Reader, data1[:]) if err != nil { t.Fatal(err) } server, client := constructor(t, server_conn, client_conn) defer close_both(server, client) var wg sync.WaitGroup send_func := func(sender HandshakingConn, data []byte) { defer wg.Done() for i := 0; i < times; i++ { if i == times/2 { wg.Add(1) go func() { defer wg.Done() err := sender.Handshake() if err != nil { t.Fatal(err) } }() } _, err := sender.Write(data) if err != nil { t.Fatal(err) } } } recv_func := func(receiver net.Conn, data []byte) { defer wg.Done() buf := make([]byte, len(data)) for i := 0; i < times; i++ { n, err := io.ReadFull(receiver, buf[:]) if err != nil { t.Fatal(err) } if !bytes.Equal(buf[:n], data) { t.Fatal(err) } } } wg.Add(4) go recv_func(server, data1) go send_func(client, data1) go send_func(server, data2) go recv_func(client, data2) wg.Wait() } func TestStdlibFullDuplexRenegotiation(t *testing.T) { FullDuplexRenegotiationTest(t, StdlibConstructor) } func TestOpenSSLFullDuplexRenegotiation(t *testing.T) { FullDuplexRenegotiationTest(t, OpenSSLConstructor) } func TestOpenSSLStdlibFullDuplexRenegotiation(t *testing.T) { FullDuplexRenegotiationTest(t, OpenSSLStdlibConstructor) } func TestStdlibOpenSSLFullDuplexRenegotiation(t *testing.T) { FullDuplexRenegotiationTest(t, StdlibOpenSSLConstructor) } func LotsOfConns(t *testing.T, payload_size int64, loops, clients int, sleep time.Duration, newListener func(net.Listener) net.Listener, newClient func(net.Conn) (net.Conn, error)) { tcp_listener, err := net.Listen("tcp", "localhost:0") if err != nil { t.Fatal(err) } ssl_listener := newListener(tcp_listener) go func() { for { conn, err := ssl_listener.Accept() if err != nil { t.Fatalf("failed accept: %s", err) continue } go func() { defer func() { err = conn.Close() if err != nil { t.Fatalf("failed closing: %s", err) } }() for i := 0; i < loops; i++ { _, err := io.Copy(ioutil.Discard, io.LimitReader(conn, payload_size)) if err != nil { t.Fatalf("failed reading: %s", err) return } _, err = io.Copy(conn, io.LimitReader(rand.Reader, payload_size)) if err != nil { t.Fatalf("failed writing: %s", err) return } } time.Sleep(sleep) }() } }() var wg sync.WaitGroup for i := 0; i < clients; i++ { tcp_client, err := net.Dial(tcp_listener.Addr().Network(), tcp_listener.Addr().String()) if err != nil { t.Fatal(err) } ssl_client, err := newClient(tcp_client) if err != nil { t.Fatal(err) } wg.Add(1) go func(i int) { defer func() { err = ssl_client.Close() if err != nil { t.Fatalf("failed closing: %s", err) } wg.Done() }() for i := 0; i < loops; i++ { _, err := io.Copy(ssl_client, io.LimitReader(rand.Reader, payload_size)) if err != nil { t.Fatalf("failed writing: %s", err) return } _, err = io.Copy(ioutil.Discard, io.LimitReader(ssl_client, payload_size)) if err != nil { t.Fatalf("failed reading: %s", err) return } } time.Sleep(sleep) }(i) } wg.Wait() } func TestStdlibLotsOfConns(t *testing.T) { tls_cert, err := tls.X509KeyPair(certBytes, keyBytes) if err != nil { t.Fatal(err) } tls_config := &tls.Config{ Certificates: []tls.Certificate{tls_cert}, InsecureSkipVerify: true, CipherSuites: []uint16{tls.TLS_RSA_WITH_AES_128_CBC_SHA}} LotsOfConns(t, 1024*64, 10, 100, 0*time.Second, func(l net.Listener) net.Listener { return tls.NewListener(l, tls_config) }, func(c net.Conn) (net.Conn, error) { return tls.Client(c, tls_config), nil }) } func TestOpenSSLLotsOfConns(t *testing.T) { ctx, err := NewCtx() if err != nil { t.Fatal(err) } key, err := LoadPrivateKeyFromPEM(keyBytes) if err != nil { t.Fatal(err) } err = ctx.UsePrivateKey(key) if err != nil { t.Fatal(err) } cert, err := LoadCertificateFromPEM(certBytes) if err != nil { t.Fatal(err) } err = ctx.UseCertificate(cert) if err != nil { t.Fatal(err) } err = ctx.SetCipherList("AES128-SHA") if err != nil { t.Fatal(err) } LotsOfConns(t, 1024*64, 10, 100, 0*time.Second, func(l net.Listener) net.Listener { return NewListener(l, ctx) }, func(c net.Conn) (net.Conn, error) { return Client(c, ctx) }) } mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/openssl/utils/000077500000000000000000000000001276133741700260625ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/openssl/utils/errors.go000066400000000000000000000026411276133741700277300ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. package utils import ( "errors" "strings" ) // ErrorGroup collates errors type ErrorGroup struct { Errors []error } // Add adds an error to an existing error group func (e *ErrorGroup) Add(err error) { if err != nil { e.Errors = append(e.Errors, err) } } // Finalize returns an error corresponding to the ErrorGroup state. If there's // no errors in the group, finalize returns nil. If there's only one error, // Finalize returns that error. Otherwise, Finalize will make a new error // consisting of the messages from the constituent errors. func (e *ErrorGroup) Finalize() error { if len(e.Errors) == 0 { return nil } if len(e.Errors) == 1 { return e.Errors[0] } msgs := make([]string, 0, len(e.Errors)) for _, err := range e.Errors { msgs = append(msgs, err.Error()) } return errors.New(strings.Join(msgs, "\n")) } mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/openssl/utils/future.go000066400000000000000000000041221276133741700277220ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. package utils import ( "sync" ) // Future is a type that is essentially the inverse of a channel. With a // channel, you have multiple senders and one receiver. With a future, you can // have multiple receivers and one sender. Additionally, a future protects // against double-sends. Since this is usually used for returning function // results, we also capture and return error values as well. Use NewFuture // to initialize. type Future struct { mutex *sync.Mutex cond *sync.Cond received bool val interface{} err error } // NewFuture returns an initialized and ready Future. func NewFuture() *Future { mutex := &sync.Mutex{} return &Future{ mutex: mutex, cond: sync.NewCond(mutex), received: false, val: nil, err: nil, } } // Get blocks until the Future has a value set. func (self *Future) Get() (interface{}, error) { self.mutex.Lock() defer self.mutex.Unlock() for { if self.received { return self.val, self.err } self.cond.Wait() } } // Fired returns whether or not a value has been set. If Fired is true, Get // won't block. func (self *Future) Fired() bool { self.mutex.Lock() defer self.mutex.Unlock() return self.received } // Set provides the value to present and future Get calls. If Set has already // been called, this is a no-op. func (self *Future) Set(val interface{}, err error) { self.mutex.Lock() defer self.mutex.Unlock() if self.received { return } self.received = true self.val = val self.err = err self.cond.Broadcast() } mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/openssl/verify.c000066400000000000000000000023301276133741700263700ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. #include #include "_cgo_export.h" int verify_cb(int ok, X509_STORE_CTX* store) { SSL* ssl = (SSL *)X509_STORE_CTX_get_app_data(store); SSL_CTX* ssl_ctx = SSL_get_SSL_CTX(ssl); void* p = SSL_CTX_get_ex_data(ssl_ctx, get_ssl_ctx_idx()); // get the pointer to the go Ctx object and pass it back into the thunk return verify_cb_thunk(p, ok, store); } int verify_ssl_cb(int ok, X509_STORE_CTX* store) { SSL* ssl = (SSL *)X509_STORE_CTX_get_app_data(store); void* p = SSL_get_ex_data(ssl, get_ssl_idx()); // get the pointer to the go Ctx object and pass it back into the thunk return verify_ssl_cb_thunk(p, ok, store); } mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/openssl/version.go000066400000000000000000000013211276133741700267330ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // +build cgo package openssl // #include import "C" const Version string = C.OPENSSL_VERSION_TEXT mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/spacelog/000077500000000000000000000000001276133741700250345ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/spacelog/.travis.yml000066400000000000000000000000561276133741700271460ustar00rootroot00000000000000language: go go: - 1.2 - release - tip mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/spacelog/LICENSE000066400000000000000000000240411276133741700260420ustar00rootroot00000000000000Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: You must give any other recipients of the Work or Derivative Works a copy of this License; and You must cause any modified files to carry prominent notices stating that You changed the files; and You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/spacelog/README.md000066400000000000000000000014261276133741700263160ustar00rootroot00000000000000# spacelog [![Build Status](https://api.travis-ci.org/spacemonkeygo/spacelog.svg?branch=master)](https://travis-ci.org/spacemonkeygo/spacelog) Please see http://godoc.org/github.com/spacemonkeygo/spacelog for info ### License Copyright (C) 2014 Space Monkey, Inc. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/spacelog/capture.go000066400000000000000000000034401276133741700270270ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. package spacelog import ( "fmt" "os" "os/exec" ) // CaptureOutputToFile opens a filehandle using the given path, then calls // CaptureOutputToFd on the associated filehandle. func CaptureOutputToFile(path string) error { fh, err := os.OpenFile(path, os.O_RDWR|os.O_CREATE, 0644) if err != nil { return err } defer fh.Close() return CaptureOutputToFd(int(fh.Fd())) } // CaptureOutputToProcess starts a process and using CaptureOutputToFd, // redirects stdout and stderr to the subprocess' stdin. // CaptureOutputToProcess expects the subcommand to last the lifetime of the // process, and if the subprocess dies, will panic. func CaptureOutputToProcess(command string, args ...string) error { cmd := exec.Command(command, args...) out, err := cmd.StdinPipe() if err != nil { return err } defer out.Close() type fder interface { Fd() uintptr } out_fder, ok := out.(fder) if !ok { return fmt.Errorf("unable to get underlying pipe") } err = CaptureOutputToFd(int(out_fder.Fd())) if err != nil { return err } err = cmd.Start() if err != nil { return err } go func() { err := cmd.Wait() if err != nil { panic(fmt.Errorf("captured output process died! %s", err)) } }() return nil } mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/spacelog/capture_other.go000066400000000000000000000017531276133741700302350ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // +build !windows package spacelog import ( "syscall" ) // CaptureOutputToFd redirects the current process' stdout and stderr file // descriptors to the given file descriptor, using the dup2 syscall. func CaptureOutputToFd(fd int) error { err := syscall.Dup2(fd, syscall.Stdout) if err != nil { return err } err = syscall.Dup2(fd, syscall.Stderr) if err != nil { return err } return nil } mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/spacelog/capture_windows.go000066400000000000000000000013451276133741700306030ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. package spacelog import ( "fmt" ) func CaptureOutputToFd(fd int) error { return fmt.Errorf("CaptureOutputToFd not supported on Windows") } mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/spacelog/collection.go000066400000000000000000000157141276133741700275260ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. package spacelog import ( "regexp" "runtime" "strings" "sync" "text/template" ) var ( // If set, these prefixes will be stripped out of automatic logger names. IgnoredPrefixes []string badChars = regexp.MustCompile("[^a-zA-Z0-9_.-]") slashes = regexp.MustCompile("[/]") ) func callerName() string { pc, _, _, ok := runtime.Caller(2) if !ok { return "unknown.unknown" } f := runtime.FuncForPC(pc) if f == nil { return "unknown.unknown" } name := f.Name() for _, prefix := range IgnoredPrefixes { name = strings.TrimPrefix(name, prefix) } return badChars.ReplaceAllLiteralString( slashes.ReplaceAllLiteralString(name, "."), "_") } // LoggerCollections contain all of the loggers a program might use. Typically // a codebase will just use the default logger collection. type LoggerCollection struct { mtx sync.Mutex loggers map[string]*Logger level LogLevel handler Handler } // NewLoggerCollection creates a new logger collection. It's unlikely you will // ever practically need this method. Use the DefaultLoggerCollection instead. func NewLoggerCollection() *LoggerCollection { return &LoggerCollection{ loggers: make(map[string]*Logger), level: DefaultLevel, handler: defaultHandler} } // GetLogger returns a new Logger with a name automatically generated using // the callstack. If you want to avoid automatic name generation check out // GetLoggerNamed func (c *LoggerCollection) GetLogger() *Logger { return GetLoggerNamed(callerName()) } func (c *LoggerCollection) getLogger(name string, level LogLevel, handler Handler) *Logger { c.mtx.Lock() defer c.mtx.Unlock() logger, exists := c.loggers[name] if !exists { logger = &Logger{level: level, collection: c, name: name, handler: handler} c.loggers[name] = logger } return logger } // GetLoggerNamed returns a new Logger with the provided name. GetLogger is // more frequently used. func (c *LoggerCollection) GetLoggerNamed(name string) *Logger { c.mtx.Lock() defer c.mtx.Unlock() logger, exists := c.loggers[name] if !exists { logger = &Logger{level: c.level, collection: c, name: name, handler: c.handler} c.loggers[name] = logger } return logger } // SetLevel will set the current log level for all loggers with names that // match a provided regular expression. If the regular expression is nil, then // all loggers match. func (c *LoggerCollection) SetLevel(re *regexp.Regexp, level LogLevel) { c.mtx.Lock() defer c.mtx.Unlock() if re == nil { c.level = level } for name, logger := range c.loggers { if re == nil || re.MatchString(name) { logger.setLevel(level) } } } // SetHandler will set the current log handler for all loggers with names that // match a provided regular expression. If the regular expression is nil, then // all loggers match. func (c *LoggerCollection) SetHandler(re *regexp.Regexp, handler Handler) { c.mtx.Lock() defer c.mtx.Unlock() if re == nil { c.handler = handler } for name, logger := range c.loggers { if re == nil || re.MatchString(name) { logger.setHandler(handler) } } } // SetTextTemplate will set the current text template for all loggers with // names that match a provided regular expression. If the regular expression // is nil, then all loggers match. Note that not every handler is guaranteed // to support text templates and a text template will only apply to // text-oriented and unstructured handlers. func (c *LoggerCollection) SetTextTemplate(re *regexp.Regexp, t *template.Template) { c.mtx.Lock() defer c.mtx.Unlock() if re == nil { c.handler.SetTextTemplate(t) } for name, logger := range c.loggers { if re == nil || re.MatchString(name) { logger.getHandler().SetTextTemplate(t) } } } // SetTextOutput will set the current output interface for all loggers with // names that match a provided regular expression. If the regular expression // is nil, then all loggers match. Note that not every handler is guaranteed // to support text output and a text output interface will only apply to // text-oriented and unstructured handlers. func (c *LoggerCollection) SetTextOutput(re *regexp.Regexp, output TextOutput) { c.mtx.Lock() defer c.mtx.Unlock() if re == nil { c.handler.SetTextOutput(output) } for name, logger := range c.loggers { if re == nil || re.MatchString(name) { logger.getHandler().SetTextOutput(output) } } } var ( // It's unlikely you'll need to use this directly DefaultLoggerCollection = NewLoggerCollection() ) // GetLogger returns an automatically-named logger on the default logger // collection. func GetLogger() *Logger { return DefaultLoggerCollection.GetLoggerNamed(callerName()) } // GetLoggerNamed returns a new Logger with the provided name on the default // logger collection. GetLogger is more frequently used. func GetLoggerNamed(name string) *Logger { return DefaultLoggerCollection.GetLoggerNamed(name) } // SetLevel will set the current log level for all loggers on the default // collection with names that match a provided regular expression. If the // regular expression is nil, then all loggers match. func SetLevel(re *regexp.Regexp, level LogLevel) { DefaultLoggerCollection.SetLevel(re, level) } // SetHandler will set the current log handler for all loggers on the default // collection with names that match a provided regular expression. If the // regular expression is nil, then all loggers match. func SetHandler(re *regexp.Regexp, handler Handler) { DefaultLoggerCollection.SetHandler(re, handler) } // SetTextTemplate will set the current text template for all loggers on the // default collection with names that match a provided regular expression. If // the regular expression is nil, then all loggers match. Note that not every // handler is guaranteed to support text templates and a text template will // only apply to text-oriented and unstructured handlers. func SetTextTemplate(re *regexp.Regexp, t *template.Template) { DefaultLoggerCollection.SetTextTemplate(re, t) } // SetTextOutput will set the current output interface for all loggers on the // default collection with names that match a provided regular expression. If // the regular expression is nil, then all loggers match. Note that not every // handler is guaranteed to support text output and a text output interface // will only apply to text-oriented and unstructured handlers. func SetTextOutput(re *regexp.Regexp, output TextOutput) { DefaultLoggerCollection.SetTextOutput(re, output) } mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/spacelog/convenience.go000066400000000000000000000175511276133741700276700ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. package spacelog import ( "fmt" "io" ) // Debug logs a collection of values if the logger's level is debug or even // more permissive. func (l *Logger) Debug(v ...interface{}) { if l.getLevel() <= Debug { l.getHandler().Log(l.name, Debug, fmt.Sprint(v...), 1) } } // Debugf logs a format string with values if the logger's level is debug or // even more permissive. func (l *Logger) Debugf(format string, v ...interface{}) { if l.getLevel() <= Debug { l.getHandler().Log(l.name, Debug, fmt.Sprintf(format, v...), 1) } } // Debuge logs an error value if the error is not nil and the logger's level // is debug or even more permissive. func (l *Logger) Debuge(err error) { if l.getLevel() <= Debug && err != nil { l.getHandler().Log(l.name, Debug, err.Error(), 1) } } // DebugEnabled returns true if the logger's level is debug or even more // permissive. func (l *Logger) DebugEnabled() bool { return l.getLevel() <= Debug } // Info logs a collection of values if the logger's level is info or even // more permissive. func (l *Logger) Info(v ...interface{}) { if l.getLevel() <= Info { l.getHandler().Log(l.name, Info, fmt.Sprint(v...), 1) } } // Infof logs a format string with values if the logger's level is info or // even more permissive. func (l *Logger) Infof(format string, v ...interface{}) { if l.getLevel() <= Info { l.getHandler().Log(l.name, Info, fmt.Sprintf(format, v...), 1) } } // Infoe logs an error value if the error is not nil and the logger's level // is info or even more permissive. func (l *Logger) Infoe(err error) { if l.getLevel() <= Info && err != nil { l.getHandler().Log(l.name, Info, err.Error(), 1) } } // InfoEnabled returns true if the logger's level is info or even more // permissive. func (l *Logger) InfoEnabled() bool { return l.getLevel() <= Info } // Notice logs a collection of values if the logger's level is notice or even // more permissive. func (l *Logger) Notice(v ...interface{}) { if l.getLevel() <= Notice { l.getHandler().Log(l.name, Notice, fmt.Sprint(v...), 1) } } // Noticef logs a format string with values if the logger's level is notice or // even more permissive. func (l *Logger) Noticef(format string, v ...interface{}) { if l.getLevel() <= Notice { l.getHandler().Log(l.name, Notice, fmt.Sprintf(format, v...), 1) } } // Noticee logs an error value if the error is not nil and the logger's level // is notice or even more permissive. func (l *Logger) Noticee(err error) { if l.getLevel() <= Notice && err != nil { l.getHandler().Log(l.name, Notice, err.Error(), 1) } } // NoticeEnabled returns true if the logger's level is notice or even more // permissive. func (l *Logger) NoticeEnabled() bool { return l.getLevel() <= Notice } // Warn logs a collection of values if the logger's level is warning or even // more permissive. func (l *Logger) Warn(v ...interface{}) { if l.getLevel() <= Warning { l.getHandler().Log(l.name, Warning, fmt.Sprint(v...), 1) } } // Warnf logs a format string with values if the logger's level is warning or // even more permissive. func (l *Logger) Warnf(format string, v ...interface{}) { if l.getLevel() <= Warning { l.getHandler().Log(l.name, Warning, fmt.Sprintf(format, v...), 1) } } // Warne logs an error value if the error is not nil and the logger's level // is warning or even more permissive. func (l *Logger) Warne(err error) { if l.getLevel() <= Warning && err != nil { l.getHandler().Log(l.name, Warning, err.Error(), 1) } } // WarnEnabled returns true if the logger's level is warning or even more // permissive. func (l *Logger) WarnEnabled() bool { return l.getLevel() <= Warning } // Error logs a collection of values if the logger's level is error or even // more permissive. func (l *Logger) Error(v ...interface{}) { if l.getLevel() <= Error { l.getHandler().Log(l.name, Error, fmt.Sprint(v...), 1) } } // Errorf logs a format string with values if the logger's level is error or // even more permissive. func (l *Logger) Errorf(format string, v ...interface{}) { if l.getLevel() <= Error { l.getHandler().Log(l.name, Error, fmt.Sprintf(format, v...), 1) } } // Errore logs an error value if the error is not nil and the logger's level // is error or even more permissive. func (l *Logger) Errore(err error) { if l.getLevel() <= Error && err != nil { l.getHandler().Log(l.name, Error, err.Error(), 1) } } // ErrorEnabled returns true if the logger's level is error or even more // permissive. func (l *Logger) ErrorEnabled() bool { return l.getLevel() <= Error } // Crit logs a collection of values if the logger's level is critical or even // more permissive. func (l *Logger) Crit(v ...interface{}) { if l.getLevel() <= Critical { l.getHandler().Log(l.name, Critical, fmt.Sprint(v...), 1) } } // Critf logs a format string with values if the logger's level is critical or // even more permissive. func (l *Logger) Critf(format string, v ...interface{}) { if l.getLevel() <= Critical { l.getHandler().Log(l.name, Critical, fmt.Sprintf(format, v...), 1) } } // Crite logs an error value if the error is not nil and the logger's level // is critical or even more permissive. func (l *Logger) Crite(err error) { if l.getLevel() <= Critical && err != nil { l.getHandler().Log(l.name, Critical, err.Error(), 1) } } // CritEnabled returns true if the logger's level is critical or even more // permissive. func (l *Logger) CritEnabled() bool { return l.getLevel() <= Critical } // Log logs a collection of values if the logger's level is the provided level // or even more permissive. func (l *Logger) Log(level LogLevel, v ...interface{}) { if l.getLevel() <= level { l.getHandler().Log(l.name, level, fmt.Sprint(v...), 1) } } // Logf logs a format string with values if the logger's level is the provided // level or even more permissive. func (l *Logger) Logf(level LogLevel, format string, v ...interface{}) { if l.getLevel() <= level { l.getHandler().Log(l.name, level, fmt.Sprintf(format, v...), 1) } } // Loge logs an error value if the error is not nil and the logger's level // is the provided level or even more permissive. func (l *Logger) Loge(level LogLevel, err error) { if l.getLevel() <= level && err != nil { l.getHandler().Log(l.name, level, err.Error(), 1) } } // LevelEnabled returns true if the logger's level is the provided level or // even more permissive. func (l *Logger) LevelEnabled(level LogLevel) bool { return l.getLevel() <= level } type writer struct { l *Logger level LogLevel } func (w *writer) Write(data []byte) (int, error) { if w.l.getLevel() <= w.level { w.l.getHandler().Log(w.l.name, w.level, string(data), 1) } return len(data), nil } // Writer returns an io.Writer that writes messages at the given log level. func (l *Logger) Writer(level LogLevel) io.Writer { return &writer{l: l, level: level} } type writerNoCaller struct { l *Logger level LogLevel } func (w *writerNoCaller) Write(data []byte) (int, error) { if w.l.getLevel() <= w.level { w.l.getHandler().Log(w.l.name, w.level, string(data), -1) } return len(data), nil } // WriterWithoutCaller returns an io.Writer that writes messages at the given // log level, but does not attempt to collect the Write caller, and provides // no caller information to the log event. func (l *Logger) WriterWithoutCaller(level LogLevel) io.Writer { return &writerNoCaller{l: l, level: level} } mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/spacelog/doc.go000066400000000000000000000031541276133741700261330ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. /* Package spacelog is a collection of interface lego bricks designed to help you build a flexible logging system. spacelog is loosely inspired by the Python logging library. The basic interaction is between a Logger and a Handler. A Logger is what the programmer typically interacts with for creating log messages. A Logger will be at a given log level, and if log messages can clear that specific logger's log level filter, they will be passed off to the Handler. Loggers are instantiated from GetLogger and GetLoggerNamed. A Handler is a very generic interface for handling log events. You can provide your own Handler for doing structured JSON output or colorized output or countless other things. Provided are a simple TextHandler with a variety of log event templates and TextOutput sinks, such as io.Writer, Syslog, and so forth. Make sure to see the source of the setup subpackage for an example of easy and configurable logging setup at process start: http://godoc.org/github.com/spacemonkeygo/spacelog/setup */ package spacelog mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/spacelog/event.go000066400000000000000000000044241276133741700265100ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. package spacelog import ( "path/filepath" "strings" "time" ) // TermColors is a type that knows how to output terminal colors and formatting type TermColors struct{} // LogEvent is a type made by the default text handler for feeding to log // templates. It has as much contextual data about the log event as possible. type LogEvent struct { LoggerName string Level LogLevel Message string Filepath string Line int Timestamp time.Time TermColors } // Reset resets the color palette for terminals that support color func (TermColors) Reset() string { return "\x1b[0m" } func (TermColors) Bold() string { return "\x1b[1m" } func (TermColors) Underline() string { return "\x1b[4m" } func (TermColors) Black() string { return "\x1b[30m" } func (TermColors) Red() string { return "\x1b[31m" } func (TermColors) Green() string { return "\x1b[32m" } func (TermColors) Yellow() string { return "\x1b[33m" } func (TermColors) Blue() string { return "\x1b[34m" } func (TermColors) Magenta() string { return "\x1b[35m" } func (TermColors) Cyan() string { return "\x1b[36m" } func (TermColors) White() string { return "\x1b[37m" } func (l *LogEvent) Filename() string { if l.Filepath == "" { return "" } return filepath.Base(l.Filepath) } func (l *LogEvent) Time() string { return l.Timestamp.Format("15:04:05") } func (l *LogEvent) Date() string { return l.Timestamp.Format("2006/01/02") } // LevelJustified returns the log level in string form justified so that all // log levels take the same text width. func (l *LogEvent) LevelJustified() (rv string) { rv = l.Level.String() if len(rv) < 5 { rv += strings.Repeat(" ", 5-len(rv)) } return rv } mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/spacelog/handler.go000066400000000000000000000034321276133741700270020ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. package spacelog import ( "text/template" ) // Handler is an interface that knows how to process log events. This is the // basic interface type for building a logging system. If you want to route // structured log data somewhere, you would implement this interface. type Handler interface { // Log is called for every message. if calldepth is negative, caller // information is missing Log(logger_name string, level LogLevel, msg string, calldepth int) // These two calls are expected to be no-ops on non-text-output handlers SetTextTemplate(t *template.Template) SetTextOutput(output TextOutput) } // HandlerFunc is a type to make implementation of the Handler interface easier type HandlerFunc func(logger_name string, level LogLevel, msg string, calldepth int) // Log simply calls f(logger_name, level, msg, calldepth) func (f HandlerFunc) Log(logger_name string, level LogLevel, msg string, calldepth int) { f(logger_name, level, msg, calldepth) } // SetTextTemplate is a no-op func (HandlerFunc) SetTextTemplate(t *template.Template) {} // SetTextOutput is a no-op func (HandlerFunc) SetTextOutput(output TextOutput) {} var ( defaultHandler = NewTextHandler(StdlibTemplate, &StdlibOutput{}) ) mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/spacelog/level.go000066400000000000000000000051461276133741700265000ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. package spacelog import ( "fmt" "strconv" "strings" ) type LogLevel int32 const ( Debug LogLevel = 10 Info LogLevel = 20 Notice LogLevel = 30 Warning LogLevel = 40 Error LogLevel = 50 Critical LogLevel = 60 // syslog has Alert // syslog has Emerg DefaultLevel = Notice ) // String returns the log level name in short form func (l LogLevel) String() string { switch l.Match() { case Critical: return "CRIT" case Error: return "ERR" case Warning: return "WARN" case Notice: return "NOTE" case Info: return "INFO" case Debug: return "DEBUG" default: return "UNSET" } } // String returns the log level name in long human readable form func (l LogLevel) Name() string { switch l.Match() { case Critical: return "critical" case Error: return "error" case Warning: return "warning" case Notice: return "notice" case Info: return "info" case Debug: return "debug" default: return "unset" } } // Match returns the greatest named log level that is less than or equal to // the receiver log level. For example, if the log level is 43, Match() will // return 40 (Warning) func (l LogLevel) Match() LogLevel { if l >= Critical { return Critical } if l >= Error { return Error } if l >= Warning { return Warning } if l >= Notice { return Notice } if l >= Info { return Info } if l >= Debug { return Debug } return 0 } // LevelFromString will convert a named log level to its corresponding value // type, or error if both the name was unknown and an integer value was unable // to be parsed. func LevelFromString(str string) (LogLevel, error) { switch strings.ToLower(str) { case "crit", "critical": return Critical, nil case "err", "error": return Error, nil case "warn", "warning": return Warning, nil case "note", "notice": return Notice, nil case "info": return Info, nil case "debug": return Debug, nil } val, err := strconv.ParseInt(str, 10, 32) if err == nil { return LogLevel(val), nil } return 0, fmt.Errorf("Invalid log level: %s", str) } mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/spacelog/logger.go000066400000000000000000000034331276133741700266450ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. package spacelog import ( "sync" "sync/atomic" ) // Logger is the basic type that allows for logging. A logger has an associated // name, given to it during construction, either through a logger collection, // GetLogger, GetLoggerNamed, or another Logger's Scope method. A logger also // has an associated level and handler, typically configured through the logger // collection to which it belongs. type Logger struct { level LogLevel name string collection *LoggerCollection handler_mtx sync.RWMutex handler Handler } // Scope returns a new Logger with the same level and handler, using the // receiver Logger's name as a prefix. func (l *Logger) Scope(name string) *Logger { return l.collection.getLogger(l.name+"."+name, l.getLevel(), l.getHandler()) } func (l *Logger) setLevel(level LogLevel) { atomic.StoreInt32((*int32)(&l.level), int32(level)) } func (l *Logger) getLevel() LogLevel { return LogLevel(atomic.LoadInt32((*int32)(&l.level))) } func (l *Logger) setHandler(handler Handler) { l.handler_mtx.Lock() defer l.handler_mtx.Unlock() l.handler = handler } func (l *Logger) getHandler() Handler { l.handler_mtx.RLock() defer l.handler_mtx.RUnlock() return l.handler } mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/spacelog/output.go000066400000000000000000000120101276133741700267150ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. package spacelog import ( "bytes" "fmt" "io" "log" "os" "sync" ) type TextOutput interface { Output(LogLevel, []byte) } // WriterOutput is an io.Writer wrapper that matches the TextOutput interface type WriterOutput struct { w io.Writer } // NewWriterOutput returns a TextOutput that writes messages to an io.Writer func NewWriterOutput(w io.Writer) *WriterOutput { return &WriterOutput{w: w} } func (o *WriterOutput) Output(_ LogLevel, message []byte) { o.w.Write(append(bytes.TrimRight(message, "\r\n"), '\n')) } // StdlibOutput is a TextOutput that simply writes to the default Go stdlib // logging system. It is the default. If you configure the Go stdlib to write // to spacelog, make sure to provide a new TextOutput to your logging // collection type StdlibOutput struct{} func (*StdlibOutput) Output(_ LogLevel, message []byte) { log.Print(string(message)) } type bufferMsg struct { level LogLevel message []byte } // BufferedOutput uses a channel to synchronize writes to a wrapped TextOutput // and allows for buffering a limited amount of log events. type BufferedOutput struct { o TextOutput c chan bufferMsg running sync.Mutex close_once sync.Once } // NewBufferedOutput returns a BufferedOutput wrapping output with a buffer // size of buffer. func NewBufferedOutput(output TextOutput, buffer int) *BufferedOutput { if buffer < 0 { buffer = 0 } b := &BufferedOutput{ o: output, c: make(chan bufferMsg, buffer)} go b.process() return b } // Close shuts down the BufferedOutput's processing func (b *BufferedOutput) Close() { b.close_once.Do(func() { close(b.c) }) b.running.Lock() b.running.Unlock() } func (b *BufferedOutput) Output(level LogLevel, message []byte) { b.c <- bufferMsg{level: level, message: message} } func (b *BufferedOutput) process() { b.running.Lock() defer b.running.Unlock() for { msg, open := <-b.c if !open { break } b.o.Output(msg.level, msg.message) } } // A TextOutput object that also implements HupHandlingTextOutput may have its // OnHup() method called when an administrative signal is sent to this process. type HupHandlingTextOutput interface { TextOutput OnHup() } // FileWriterOutput is like WriterOutput with a plain file handle, but it // knows how to reopen the file (or try to reopen it) if it hasn't been able // to open the file previously, or if an appropriate signal has been received. type FileWriterOutput struct { *WriterOutput path string } // Creates a new FileWriterOutput object. This is the only case where an // error opening the file will be reported to the caller; if we try to // reopen it later and the reopen fails, we'll just keep trying until it // works. func NewFileWriterOutput(path string) (*FileWriterOutput, error) { fo := &FileWriterOutput{path: path} fh, err := fo.openFile() if err != nil { return nil, err } fo.WriterOutput = NewWriterOutput(fh) return fo, nil } // Try to open the file with the path associated with this object. func (fo *FileWriterOutput) openFile() (*os.File, error) { return os.OpenFile(fo.path, os.O_WRONLY|os.O_CREATE|os.O_APPEND, 0644) } // Try to communicate a message without using our log file. In all likelihood, // stderr is closed or redirected to /dev/null, but at least we can try // writing there. In the very worst case, if an admin attaches a ptrace to // this process, it will be more clear what the problem is. func (fo *FileWriterOutput) fallbackLog(tmpl string, args ...interface{}) { fmt.Fprintf(os.Stderr, tmpl, args...) } // Output a log line by writing it to the file. If the file has been // released, try to open it again. If that fails, cry for a little // while, then throw away the message and carry on. func (fo *FileWriterOutput) Output(ll LogLevel, message []byte) { if fo.WriterOutput == nil { fh, err := fo.openFile() if err != nil { fo.fallbackLog("Could not open %#v: %s", fo.path, err) return } fo.WriterOutput = NewWriterOutput(fh) } fo.WriterOutput.Output(ll, message) } // Throw away any references/handles to the output file. This probably // means the admin wants to rotate the file out and have this process // open a new one. Close the underlying io.Writer if that is a thing // that it knows how to do. func (fo *FileWriterOutput) OnHup() { if fo.WriterOutput != nil { wc, ok := fo.WriterOutput.w.(io.Closer) if ok { err := wc.Close() if err != nil { fo.fallbackLog("Closing %#v failed: %s", fo.path, err) } } fo.WriterOutput = nil } } mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/spacelog/setup.go000066400000000000000000000126661276133741700265360ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. package spacelog import ( "bytes" "fmt" "log" "math" "os" "os/signal" "regexp" "strings" "syscall" "text/template" ) // SetupConfig is a configuration struct meant to be used with // github.com/spacemonkeygo/flagfile/utils.Setup // but can be used independently. type SetupConfig struct { Output string `default:"stderr" usage:"log output. can be stdout, stderr, syslog, or a path"` Level string `default:"" usage:"base logger level"` Filter string `default:"" usage:"sets loggers matching this regular expression to the lowest level"` Format string `default:"" usage:"format string to use"` Stdlevel string `default:"warn" usage:"logger level for stdlib log integration"` Subproc string `default:"" usage:"process to run for stdout/stderr-captured logging. The command is first processed as a Go template that supports {{.Facility}}, {{.Level}}, and {{.Name}} fields, and then passed to sh. If set, will redirect stdout and stderr to the given process. A good default is 'setsid logger --priority {{.Facility}}.{{.Level}} --tag {{.Name}}'"` Buffer int `default:"0" usage:"the number of messages to buffer. 0 for no buffer"` // Facility defaults to syslog.LOG_USER (which is 8) Facility int `default:"8" usage:"the syslog facility to use if syslog output is configured"` HupRotate bool `default:"false" usage:"if true, sending a HUP signal will reopen log files"` } var ( stdlog = GetLoggerNamed("stdlog") funcmap = template.FuncMap{"ColorizeLevel": ColorizeLevel} ) // SetFormatMethod adds functions to the template function map, such that // command-line and Setup provided templates can call methods added to the map // via this method. The map comes prepopulated with ColorizeLevel, but can be // overridden. SetFormatMethod should be called (if at all) before one of // this package's Setup methods. func SetFormatMethod(name string, fn interface{}) { funcmap[name] = fn } // MustSetup is the same as Setup, but panics instead of returning an error func MustSetup(procname string, config SetupConfig) { err := Setup(procname, config) if err != nil { panic(err) } } type subprocInfo struct { Facility string Level string Name string } // Setup takes a given procname and sets spacelog up with the given // configuration. Setup supports: // * capturing stdout and stderr to a subprocess // * configuring the default level // * configuring log filters (enabling only some loggers) // * configuring the logging template // * configuring the output (a file, syslog, stdout, stderr) // * configuring log event buffering // * capturing all standard library logging with configurable log level // It is expected that this method will be called once at process start. func Setup(procname string, config SetupConfig) error { if config.Subproc != "" { t, err := template.New("subproc").Parse(config.Subproc) if err != nil { return err } var buf bytes.Buffer err = t.Execute(&buf, &subprocInfo{ Facility: fmt.Sprintf("%d", config.Facility), Level: fmt.Sprintf("%d", 2), // syslog.LOG_CRIT Name: procname}) if err != nil { return err } err = CaptureOutputToProcess("sh", "-c", string(buf.Bytes())) if err != nil { return err } } if config.Level != "" { level_val, err := LevelFromString(config.Level) if err != nil { return err } if level_val != DefaultLevel { SetLevel(nil, level_val) } } if config.Filter != "" { re, err := regexp.Compile(config.Filter) if err != nil { return err } SetLevel(re, LogLevel(math.MinInt32)) } var t *template.Template if config.Format != "" { var err error t, err = template.New("user").Funcs(funcmap).Parse(config.Format) if err != nil { return err } } var textout TextOutput switch strings.ToLower(config.Output) { case "syslog": w, err := NewSyslogOutput(SyslogPriority(config.Facility), procname) if err != nil { return err } if t == nil { t = SyslogTemplate } textout = w case "stdout": if t == nil { t = DefaultTemplate } textout = NewWriterOutput(os.Stdout) case "stderr", "": if t == nil { t = DefaultTemplate } textout = NewWriterOutput(os.Stderr) default: if t == nil { t = StandardTemplate } var err error textout, err = NewFileWriterOutput(config.Output) if err != nil { return err } } if config.HupRotate { if hh, ok := textout.(HupHandlingTextOutput); ok { sigchan := make(chan os.Signal) signal.Notify(sigchan, syscall.SIGHUP) go func() { for _ = range sigchan { hh.OnHup() } }() } } if config.Buffer > 0 { textout = NewBufferedOutput(textout, config.Buffer) } SetHandler(nil, NewTextHandler(t, textout)) log.SetFlags(log.Lshortfile) if config.Stdlevel == "" { config.Stdlevel = "warn" } stdlog_level_val, err := LevelFromString(config.Stdlevel) if err != nil { return err } log.SetOutput(stdlog.WriterWithoutCaller(stdlog_level_val)) return nil } mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/spacelog/setup/000077500000000000000000000000001276133741700261745ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/spacelog/setup/setup.go000066400000000000000000000051601276133741700276650ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. /* Package setup provides simple helpers for configuring spacelog from flags. This package adds the following flags: --log.output - can either be stdout, stderr, syslog, or a file path --log.level - the base logger level --log.filter - loggers that match this regular expression get set to the lowest level --log.format - a go text template for log lines --log.stdlevel - the logger level to assume the standard library logger is using --log.subproc - a process to run for stdout/stderr capturing --log.buffer - the number of message to buffer */ package setup import ( "github.com/spacemonkeygo/flagfile/utils" "github.com/spacemonkeygo/spacelog" ) var ( config spacelog.SetupConfig ) func init() { utils.Setup("log", &config) } // SetFormatMethod in this subpackage is deprecated and will be removed soon. // Please see spacelog.SetFormatMethod instead func SetFormatMethod(name string, fn interface{}) { spacelog.SetFormatMethod(name, fn) } // MustSetup calls spacelog.MustSetup with a flag-configured config struct // It's pretty useless to call this method without parsing flags first, via // flagfile.Load() func MustSetup(procname string) { spacelog.MustSetup(procname, config) } // Setup calls spacelog.Setup with a flag-configured config struct // It's pretty useless to call this method without parsing flags first, via // flagfile.Load() func Setup(procname string) error { return spacelog.Setup(procname, config) } // MustSetupWithFacility is deprecated and will be removed soon. Please // configure facility through the facility flag option. func MustSetupWithFacility(procname string, facility spacelog.SyslogPriority) { err := SetupWithFacility(procname, facility) if err != nil { panic(err) } } // SetupWithFacility is deprecated and will be removed soon. Please // configure facility through the facility flag option. func SetupWithFacility(procname string, facility spacelog.SyslogPriority) error { config_copy := config config_copy.Facility = int(facility) return spacelog.Setup(procname, config_copy) } mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/spacelog/syslog.go000066400000000000000000000031501276133741700267020ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // +build !windows package spacelog import ( "bytes" "log/syslog" ) type SyslogPriority syslog.Priority // SyslogOutput is a syslog client that matches the TextOutput interface type SyslogOutput struct { w *syslog.Writer } // NewSyslogOutput returns a TextOutput object that writes to syslog using // the given facility and tag. The log level will be determined by the log // event. func NewSyslogOutput(facility SyslogPriority, tag string) ( TextOutput, error) { w, err := syslog.New(syslog.Priority(facility), tag) if err != nil { return nil, err } return &SyslogOutput{w: w}, nil } func (o *SyslogOutput) Output(level LogLevel, message []byte) { level = level.Match() for _, msg := range bytes.Split(message, []byte{'\n'}) { switch level { case Critical: o.w.Crit(string(msg)) case Error: o.w.Err(string(msg)) case Warning: o.w.Warning(string(msg)) case Notice: o.w.Notice(string(msg)) case Info: o.w.Info(string(msg)) case Debug: fallthrough default: o.w.Debug(string(msg)) } } } mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/spacelog/syslog_windows.go000066400000000000000000000014511276133741700304560ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. package spacelog import ( "fmt" ) type SyslogPriority int func NewSyslogOutput(facility SyslogPriority, tag string) ( TextOutput, error) { return nil, fmt.Errorf("SyslogOutput not supported on Windows") } mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/spacelog/templates.go000066400000000000000000000045771276133741700273760ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. package spacelog import ( "text/template" ) // ColorizeLevel returns a TermColor byte sequence for the appropriate color // for the level. If you'd like to configure your own color choices, you can // make your own template with its own function map to your own colorize // function. func ColorizeLevel(level LogLevel) string { switch level.Match() { case Critical, Error: return TermColors{}.Red() case Warning: return TermColors{}.Magenta() case Notice: return TermColors{}.Yellow() case Info, Debug: return TermColors{}.Green() } return "" } var ( // ColorTemplate uses the default ColorizeLevel method for color choices. ColorTemplate = template.Must(template.New("color").Funcs(template.FuncMap{ "ColorizeLevel": ColorizeLevel}).Parse( `{{.Blue}}{{.Date}} {{.Time}}{{.Reset}} ` + `{{.Bold}}{{ColorizeLevel .Level}}{{.LevelJustified}}{{.Reset}} ` + `{{.Underline}}{{.LoggerName}}{{.Reset}} ` + `{{if .Filename}}{{.Filename}}:{{.Line}} {{end}}- ` + `{{ColorizeLevel .Level}}{{.Message}}{{.Reset}}`)) // StandardTemplate is like ColorTemplate with no color. StandardTemplate = template.Must(template.New("standard").Parse( `{{.Date}} {{.Time}} ` + `{{.Level}} {{.LoggerName}} ` + `{{if .Filename}}{{.Filename}}:{{.Line}} {{end}}` + `- {{.Message}}`)) // SyslogTemplate is missing the date and time as syslog adds those // things. SyslogTemplate = template.Must(template.New("syslog").Parse( `{{.Level}} {{.LoggerName}} ` + `{{if .Filename}}{{.Filename}}:{{.Line}} {{end}}` + `- {{.Message}}`)) // StdlibTemplate is missing the date and time as the stdlib logger often // adds those things. StdlibTemplate = template.Must(template.New("stdlib").Parse( `{{.Level}} {{.LoggerName}} ` + `{{if .Filename}}{{.Filename}}:{{.Line}} {{end}}` + `- {{.Message}}`)) ) mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/spacelog/templates_others.go000066400000000000000000000013601276133741700307450ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. // +build !windows package spacelog var ( // DefaultTemplate is default template for stdout/stderr for the platform DefaultTemplate = ColorTemplate ) mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/spacelog/templates_windows.go000066400000000000000000000013371276133741700311370ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. package spacelog var ( // DefaultTemplate is default template for stdout/stderr for the platform DefaultTemplate = StandardTemplate ) mongo-tools-r3.2.11/vendor/src/github.com/spacemonkeygo/spacelog/text.go000066400000000000000000000045311276133741700263520ustar00rootroot00000000000000// Copyright (C) 2014 Space Monkey, Inc. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. package spacelog import ( "bytes" "fmt" "runtime" "strings" "sync" "text/template" "time" ) // TextHandler is the default implementation of the Handler interface. A // TextHandler, on log events, makes LogEvent structures, passes them to the // configured template, and then passes that output to a configured TextOutput // interface. type TextHandler struct { mtx sync.RWMutex template *template.Template output TextOutput } // NewTextHandler creates a Handler that creates LogEvents, passes them to // the given template, and passes the result to output func NewTextHandler(t *template.Template, output TextOutput) *TextHandler { return &TextHandler{template: t, output: output} } // Log makes a LogEvent, formats it with the configured template, then passes // the output to configured output sink func (h *TextHandler) Log(logger_name string, level LogLevel, msg string, calldepth int) { h.mtx.RLock() output, template := h.output, h.template h.mtx.RUnlock() event := LogEvent{ LoggerName: logger_name, Level: level, Message: strings.TrimRight(msg, "\n\r"), Timestamp: time.Now()} if calldepth >= 0 { _, event.Filepath, event.Line, _ = runtime.Caller(calldepth + 1) } var buf bytes.Buffer err := template.Execute(&buf, &event) if err != nil { output.Output(level, []byte( fmt.Sprintf("log format template failed: %s", err))) return } output.Output(level, buf.Bytes()) } // SetTextTemplate changes the TextHandler's text formatting template func (h *TextHandler) SetTextTemplate(t *template.Template) { h.mtx.Lock() defer h.mtx.Unlock() h.template = t } // SetTextOutput changes the TextHandler's TextOutput sink func (h *TextHandler) SetTextOutput(output TextOutput) { h.mtx.Lock() defer h.mtx.Unlock() h.output = output } mongo-tools-r3.2.11/vendor/src/golang.org/000077500000000000000000000000001276133741700203715ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/golang.org/x/000077500000000000000000000000001276133741700206405ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/golang.org/x/crypto/000077500000000000000000000000001276133741700221605ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/golang.org/x/crypto/ssh/000077500000000000000000000000001276133741700227555ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/golang.org/x/crypto/ssh/terminal/000077500000000000000000000000001276133741700245705ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/golang.org/x/crypto/ssh/terminal/terminal.go000066400000000000000000000510001276133741700267260ustar00rootroot00000000000000// Copyright 2011 The Go Authors. All rights reserved. // Use of this source code is governed by a BSD-style // license that can be found in the LICENSE file. package terminal import ( "bytes" "io" "sync" "unicode/utf8" ) // EscapeCodes contains escape sequences that can be written to the terminal in // order to achieve different styles of text. type EscapeCodes struct { // Foreground colors Black, Red, Green, Yellow, Blue, Magenta, Cyan, White []byte // Reset all attributes Reset []byte } var vt100EscapeCodes = EscapeCodes{ Black: []byte{keyEscape, '[', '3', '0', 'm'}, Red: []byte{keyEscape, '[', '3', '1', 'm'}, Green: []byte{keyEscape, '[', '3', '2', 'm'}, Yellow: []byte{keyEscape, '[', '3', '3', 'm'}, Blue: []byte{keyEscape, '[', '3', '4', 'm'}, Magenta: []byte{keyEscape, '[', '3', '5', 'm'}, Cyan: []byte{keyEscape, '[', '3', '6', 'm'}, White: []byte{keyEscape, '[', '3', '7', 'm'}, Reset: []byte{keyEscape, '[', '0', 'm'}, } // Terminal contains the state for running a VT100 terminal that is capable of // reading lines of input. type Terminal struct { // AutoCompleteCallback, if non-null, is called for each keypress with // the full input line and the current position of the cursor (in // bytes, as an index into |line|). If it returns ok=false, the key // press is processed normally. Otherwise it returns a replacement line // and the new cursor position. AutoCompleteCallback func(line string, pos int, key rune) (newLine string, newPos int, ok bool) // Escape contains a pointer to the escape codes for this terminal. // It's always a valid pointer, although the escape codes themselves // may be empty if the terminal doesn't support them. Escape *EscapeCodes // lock protects the terminal and the state in this object from // concurrent processing of a key press and a Write() call. lock sync.Mutex c io.ReadWriter prompt []rune // line is the current line being entered. line []rune // pos is the logical position of the cursor in line pos int // echo is true if local echo is enabled echo bool // pasteActive is true iff there is a bracketed paste operation in // progress. pasteActive bool // cursorX contains the current X value of the cursor where the left // edge is 0. cursorY contains the row number where the first row of // the current line is 0. cursorX, cursorY int // maxLine is the greatest value of cursorY so far. maxLine int termWidth, termHeight int // outBuf contains the terminal data to be sent. outBuf []byte // remainder contains the remainder of any partial key sequences after // a read. It aliases into inBuf. remainder []byte inBuf [256]byte // history contains previously entered commands so that they can be // accessed with the up and down keys. history stRingBuffer // historyIndex stores the currently accessed history entry, where zero // means the immediately previous entry. historyIndex int // When navigating up and down the history it's possible to return to // the incomplete, initial line. That value is stored in // historyPending. historyPending string } // NewTerminal runs a VT100 terminal on the given ReadWriter. If the ReadWriter is // a local terminal, that terminal must first have been put into raw mode. // prompt is a string that is written at the start of each input line (i.e. // "> "). func NewTerminal(c io.ReadWriter, prompt string) *Terminal { return &Terminal{ Escape: &vt100EscapeCodes, c: c, prompt: []rune(prompt), termWidth: 80, termHeight: 24, echo: true, historyIndex: -1, } } const ( keyCtrlD = 4 keyCtrlU = 21 keyEnter = '\r' keyEscape = 27 keyBackspace = 127 keyUnknown = 0xd800 /* UTF-16 surrogate area */ + iota keyUp keyDown keyLeft keyRight keyAltLeft keyAltRight keyHome keyEnd keyDeleteWord keyDeleteLine keyClearScreen keyPasteStart keyPasteEnd ) var pasteStart = []byte{keyEscape, '[', '2', '0', '0', '~'} var pasteEnd = []byte{keyEscape, '[', '2', '0', '1', '~'} // bytesToKey tries to parse a key sequence from b. If successful, it returns // the key and the remainder of the input. Otherwise it returns utf8.RuneError. func bytesToKey(b []byte, pasteActive bool) (rune, []byte) { if len(b) == 0 { return utf8.RuneError, nil } if !pasteActive { switch b[0] { case 1: // ^A return keyHome, b[1:] case 5: // ^E return keyEnd, b[1:] case 8: // ^H return keyBackspace, b[1:] case 11: // ^K return keyDeleteLine, b[1:] case 12: // ^L return keyClearScreen, b[1:] case 23: // ^W return keyDeleteWord, b[1:] } } if b[0] != keyEscape { if !utf8.FullRune(b) { return utf8.RuneError, b } r, l := utf8.DecodeRune(b) return r, b[l:] } if !pasteActive && len(b) >= 3 && b[0] == keyEscape && b[1] == '[' { switch b[2] { case 'A': return keyUp, b[3:] case 'B': return keyDown, b[3:] case 'C': return keyRight, b[3:] case 'D': return keyLeft, b[3:] case 'H': return keyHome, b[3:] case 'F': return keyEnd, b[3:] } } if !pasteActive && len(b) >= 6 && b[0] == keyEscape && b[1] == '[' && b[2] == '1' && b[3] == ';' && b[4] == '3' { switch b[5] { case 'C': return keyAltRight, b[6:] case 'D': return keyAltLeft, b[6:] } } if !pasteActive && len(b) >= 6 && bytes.Equal(b[:6], pasteStart) { return keyPasteStart, b[6:] } if pasteActive && len(b) >= 6 && bytes.Equal(b[:6], pasteEnd) { return keyPasteEnd, b[6:] } // If we get here then we have a key that we don't recognise, or a // partial sequence. It's not clear how one should find the end of a // sequence without knowing them all, but it seems that [a-zA-Z~] only // appears at the end of a sequence. for i, c := range b[0:] { if c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z' || c == '~' { return keyUnknown, b[i+1:] } } return utf8.RuneError, b } // queue appends data to the end of t.outBuf func (t *Terminal) queue(data []rune) { t.outBuf = append(t.outBuf, []byte(string(data))...) } var eraseUnderCursor = []rune{' ', keyEscape, '[', 'D'} var space = []rune{' '} func isPrintable(key rune) bool { isInSurrogateArea := key >= 0xd800 && key <= 0xdbff return key >= 32 && !isInSurrogateArea } // moveCursorToPos appends data to t.outBuf which will move the cursor to the // given, logical position in the text. func (t *Terminal) moveCursorToPos(pos int) { if !t.echo { return } x := visualLength(t.prompt) + pos y := x / t.termWidth x = x % t.termWidth up := 0 if y < t.cursorY { up = t.cursorY - y } down := 0 if y > t.cursorY { down = y - t.cursorY } left := 0 if x < t.cursorX { left = t.cursorX - x } right := 0 if x > t.cursorX { right = x - t.cursorX } t.cursorX = x t.cursorY = y t.move(up, down, left, right) } func (t *Terminal) move(up, down, left, right int) { movement := make([]rune, 3*(up+down+left+right)) m := movement for i := 0; i < up; i++ { m[0] = keyEscape m[1] = '[' m[2] = 'A' m = m[3:] } for i := 0; i < down; i++ { m[0] = keyEscape m[1] = '[' m[2] = 'B' m = m[3:] } for i := 0; i < left; i++ { m[0] = keyEscape m[1] = '[' m[2] = 'D' m = m[3:] } for i := 0; i < right; i++ { m[0] = keyEscape m[1] = '[' m[2] = 'C' m = m[3:] } t.queue(movement) } func (t *Terminal) clearLineToRight() { op := []rune{keyEscape, '[', 'K'} t.queue(op) } const maxLineLength = 4096 func (t *Terminal) setLine(newLine []rune, newPos int) { if t.echo { t.moveCursorToPos(0) t.writeLine(newLine) for i := len(newLine); i < len(t.line); i++ { t.writeLine(space) } t.moveCursorToPos(newPos) } t.line = newLine t.pos = newPos } func (t *Terminal) advanceCursor(places int) { t.cursorX += places t.cursorY += t.cursorX / t.termWidth if t.cursorY > t.maxLine { t.maxLine = t.cursorY } t.cursorX = t.cursorX % t.termWidth if places > 0 && t.cursorX == 0 { // Normally terminals will advance the current position // when writing a character. But that doesn't happen // for the last character in a line. However, when // writing a character (except a new line) that causes // a line wrap, the position will be advanced two // places. // // So, if we are stopping at the end of a line, we // need to write a newline so that our cursor can be // advanced to the next line. t.outBuf = append(t.outBuf, '\n') } } func (t *Terminal) eraseNPreviousChars(n int) { if n == 0 { return } if t.pos < n { n = t.pos } t.pos -= n t.moveCursorToPos(t.pos) copy(t.line[t.pos:], t.line[n+t.pos:]) t.line = t.line[:len(t.line)-n] if t.echo { t.writeLine(t.line[t.pos:]) for i := 0; i < n; i++ { t.queue(space) } t.advanceCursor(n) t.moveCursorToPos(t.pos) } } // countToLeftWord returns then number of characters from the cursor to the // start of the previous word. func (t *Terminal) countToLeftWord() int { if t.pos == 0 { return 0 } pos := t.pos - 1 for pos > 0 { if t.line[pos] != ' ' { break } pos-- } for pos > 0 { if t.line[pos] == ' ' { pos++ break } pos-- } return t.pos - pos } // countToRightWord returns then number of characters from the cursor to the // start of the next word. func (t *Terminal) countToRightWord() int { pos := t.pos for pos < len(t.line) { if t.line[pos] == ' ' { break } pos++ } for pos < len(t.line) { if t.line[pos] != ' ' { break } pos++ } return pos - t.pos } // visualLength returns the number of visible glyphs in s. func visualLength(runes []rune) int { inEscapeSeq := false length := 0 for _, r := range runes { switch { case inEscapeSeq: if (r >= 'a' && r <= 'z') || (r >= 'A' && r <= 'Z') { inEscapeSeq = false } case r == '\x1b': inEscapeSeq = true default: length++ } } return length } // handleKey processes the given key and, optionally, returns a line of text // that the user has entered. func (t *Terminal) handleKey(key rune) (line string, ok bool) { if t.pasteActive && key != keyEnter { t.addKeyToLine(key) return } switch key { case keyBackspace: if t.pos == 0 { return } t.eraseNPreviousChars(1) case keyAltLeft: // move left by a word. t.pos -= t.countToLeftWord() t.moveCursorToPos(t.pos) case keyAltRight: // move right by a word. t.pos += t.countToRightWord() t.moveCursorToPos(t.pos) case keyLeft: if t.pos == 0 { return } t.pos-- t.moveCursorToPos(t.pos) case keyRight: if t.pos == len(t.line) { return } t.pos++ t.moveCursorToPos(t.pos) case keyHome: if t.pos == 0 { return } t.pos = 0 t.moveCursorToPos(t.pos) case keyEnd: if t.pos == len(t.line) { return } t.pos = len(t.line) t.moveCursorToPos(t.pos) case keyUp: entry, ok := t.history.NthPreviousEntry(t.historyIndex + 1) if !ok { return "", false } if t.historyIndex == -1 { t.historyPending = string(t.line) } t.historyIndex++ runes := []rune(entry) t.setLine(runes, len(runes)) case keyDown: switch t.historyIndex { case -1: return case 0: runes := []rune(t.historyPending) t.setLine(runes, len(runes)) t.historyIndex-- default: entry, ok := t.history.NthPreviousEntry(t.historyIndex - 1) if ok { t.historyIndex-- runes := []rune(entry) t.setLine(runes, len(runes)) } } case keyEnter: t.moveCursorToPos(len(t.line)) t.queue([]rune("\r\n")) line = string(t.line) ok = true t.line = t.line[:0] t.pos = 0 t.cursorX = 0 t.cursorY = 0 t.maxLine = 0 case keyDeleteWord: // Delete zero or more spaces and then one or more characters. t.eraseNPreviousChars(t.countToLeftWord()) case keyDeleteLine: // Delete everything from the current cursor position to the // end of line. for i := t.pos; i < len(t.line); i++ { t.queue(space) t.advanceCursor(1) } t.line = t.line[:t.pos] t.moveCursorToPos(t.pos) case keyCtrlD: // Erase the character under the current position. // The EOF case when the line is empty is handled in // readLine(). if t.pos < len(t.line) { t.pos++ t.eraseNPreviousChars(1) } case keyCtrlU: t.eraseNPreviousChars(t.pos) case keyClearScreen: // Erases the screen and moves the cursor to the home position. t.queue([]rune("\x1b[2J\x1b[H")) t.queue(t.prompt) t.cursorX, t.cursorY = 0, 0 t.advanceCursor(visualLength(t.prompt)) t.setLine(t.line, t.pos) default: if t.AutoCompleteCallback != nil { prefix := string(t.line[:t.pos]) suffix := string(t.line[t.pos:]) t.lock.Unlock() newLine, newPos, completeOk := t.AutoCompleteCallback(prefix+suffix, len(prefix), key) t.lock.Lock() if completeOk { t.setLine([]rune(newLine), utf8.RuneCount([]byte(newLine)[:newPos])) return } } if !isPrintable(key) { return } if len(t.line) == maxLineLength { return } t.addKeyToLine(key) } return } // addKeyToLine inserts the given key at the current position in the current // line. func (t *Terminal) addKeyToLine(key rune) { if len(t.line) == cap(t.line) { newLine := make([]rune, len(t.line), 2*(1+len(t.line))) copy(newLine, t.line) t.line = newLine } t.line = t.line[:len(t.line)+1] copy(t.line[t.pos+1:], t.line[t.pos:]) t.line[t.pos] = key if t.echo { t.writeLine(t.line[t.pos:]) } t.pos++ t.moveCursorToPos(t.pos) } func (t *Terminal) writeLine(line []rune) { for len(line) != 0 { remainingOnLine := t.termWidth - t.cursorX todo := len(line) if todo > remainingOnLine { todo = remainingOnLine } t.queue(line[:todo]) t.advanceCursor(visualLength(line[:todo])) line = line[todo:] } } func (t *Terminal) Write(buf []byte) (n int, err error) { t.lock.Lock() defer t.lock.Unlock() if t.cursorX == 0 && t.cursorY == 0 { // This is the easy case: there's nothing on the screen that we // have to move out of the way. return t.c.Write(buf) } // We have a prompt and possibly user input on the screen. We // have to clear it first. t.move(0 /* up */, 0 /* down */, t.cursorX /* left */, 0 /* right */) t.cursorX = 0 t.clearLineToRight() for t.cursorY > 0 { t.move(1 /* up */, 0, 0, 0) t.cursorY-- t.clearLineToRight() } if _, err = t.c.Write(t.outBuf); err != nil { return } t.outBuf = t.outBuf[:0] if n, err = t.c.Write(buf); err != nil { return } t.writeLine(t.prompt) if t.echo { t.writeLine(t.line) } t.moveCursorToPos(t.pos) if _, err = t.c.Write(t.outBuf); err != nil { return } t.outBuf = t.outBuf[:0] return } // ReadPassword temporarily changes the prompt and reads a password, without // echo, from the terminal. func (t *Terminal) ReadPassword(prompt string) (line string, err error) { t.lock.Lock() defer t.lock.Unlock() oldPrompt := t.prompt t.prompt = []rune(prompt) t.echo = false line, err = t.readLine() t.prompt = oldPrompt t.echo = true return } // ReadLine returns a line of input from the terminal. func (t *Terminal) ReadLine() (line string, err error) { t.lock.Lock() defer t.lock.Unlock() return t.readLine() } func (t *Terminal) readLine() (line string, err error) { // t.lock must be held at this point if t.cursorX == 0 && t.cursorY == 0 { t.writeLine(t.prompt) t.c.Write(t.outBuf) t.outBuf = t.outBuf[:0] } lineIsPasted := t.pasteActive for { rest := t.remainder lineOk := false for !lineOk { var key rune key, rest = bytesToKey(rest, t.pasteActive) if key == utf8.RuneError { break } if !t.pasteActive { if key == keyCtrlD { if len(t.line) == 0 { return "", io.EOF } } if key == keyPasteStart { t.pasteActive = true if len(t.line) == 0 { lineIsPasted = true } continue } } else if key == keyPasteEnd { t.pasteActive = false continue } if !t.pasteActive { lineIsPasted = false } line, lineOk = t.handleKey(key) } if len(rest) > 0 { n := copy(t.inBuf[:], rest) t.remainder = t.inBuf[:n] } else { t.remainder = nil } t.c.Write(t.outBuf) t.outBuf = t.outBuf[:0] if lineOk { if t.echo { t.historyIndex = -1 t.history.Add(line) } if lineIsPasted { err = ErrPasteIndicator } return } // t.remainder is a slice at the beginning of t.inBuf // containing a partial key sequence readBuf := t.inBuf[len(t.remainder):] var n int t.lock.Unlock() n, err = t.c.Read(readBuf) t.lock.Lock() if err != nil { return } t.remainder = t.inBuf[:n+len(t.remainder)] } panic("unreachable") // for Go 1.0. } // SetPrompt sets the prompt to be used when reading subsequent lines. func (t *Terminal) SetPrompt(prompt string) { t.lock.Lock() defer t.lock.Unlock() t.prompt = []rune(prompt) } func (t *Terminal) clearAndRepaintLinePlusNPrevious(numPrevLines int) { // Move cursor to column zero at the start of the line. t.move(t.cursorY, 0, t.cursorX, 0) t.cursorX, t.cursorY = 0, 0 t.clearLineToRight() for t.cursorY < numPrevLines { // Move down a line t.move(0, 1, 0, 0) t.cursorY++ t.clearLineToRight() } // Move back to beginning. t.move(t.cursorY, 0, 0, 0) t.cursorX, t.cursorY = 0, 0 t.queue(t.prompt) t.advanceCursor(visualLength(t.prompt)) t.writeLine(t.line) t.moveCursorToPos(t.pos) } func (t *Terminal) SetSize(width, height int) error { t.lock.Lock() defer t.lock.Unlock() if width == 0 { width = 1 } oldWidth := t.termWidth t.termWidth, t.termHeight = width, height switch { case width == oldWidth: // If the width didn't change then nothing else needs to be // done. return nil case len(t.line) == 0 && t.cursorX == 0 && t.cursorY == 0: // If there is nothing on current line and no prompt printed, // just do nothing return nil case width < oldWidth: // Some terminals (e.g. xterm) will truncate lines that were // too long when shinking. Others, (e.g. gnome-terminal) will // attempt to wrap them. For the former, repainting t.maxLine // works great, but that behaviour goes badly wrong in the case // of the latter because they have doubled every full line. // We assume that we are working on a terminal that wraps lines // and adjust the cursor position based on every previous line // wrapping and turning into two. This causes the prompt on // xterms to move upwards, which isn't great, but it avoids a // huge mess with gnome-terminal. if t.cursorX >= t.termWidth { t.cursorX = t.termWidth - 1 } t.cursorY *= 2 t.clearAndRepaintLinePlusNPrevious(t.maxLine * 2) case width > oldWidth: // If the terminal expands then our position calculations will // be wrong in the future because we think the cursor is // |t.pos| chars into the string, but there will be a gap at // the end of any wrapped line. // // But the position will actually be correct until we move, so // we can move back to the beginning and repaint everything. t.clearAndRepaintLinePlusNPrevious(t.maxLine) } _, err := t.c.Write(t.outBuf) t.outBuf = t.outBuf[:0] return err } type pasteIndicatorError struct{} func (pasteIndicatorError) Error() string { return "terminal: ErrPasteIndicator not correctly handled" } // ErrPasteIndicator may be returned from ReadLine as the error, in addition // to valid line data. It indicates that bracketed paste mode is enabled and // that the returned line consists only of pasted data. Programs may wish to // interpret pasted data more literally than typed data. var ErrPasteIndicator = pasteIndicatorError{} // SetBracketedPasteMode requests that the terminal bracket paste operations // with markers. Not all terminals support this but, if it is supported, then // enabling this mode will stop any autocomplete callback from running due to // pastes. Additionally, any lines that are completely pasted will be returned // from ReadLine with the error set to ErrPasteIndicator. func (t *Terminal) SetBracketedPasteMode(on bool) { if on { io.WriteString(t.c, "\x1b[?2004h") } else { io.WriteString(t.c, "\x1b[?2004l") } } // stRingBuffer is a ring buffer of strings. type stRingBuffer struct { // entries contains max elements. entries []string max int // head contains the index of the element most recently added to the ring. head int // size contains the number of elements in the ring. size int } func (s *stRingBuffer) Add(a string) { if s.entries == nil { const defaultNumEntries = 100 s.entries = make([]string, defaultNumEntries) s.max = defaultNumEntries } s.head = (s.head + 1) % s.max s.entries[s.head] = a if s.size < s.max { s.size++ } } // NthPreviousEntry returns the value passed to the nth previous call to Add. // If n is zero then the immediately prior value is returned, if one, then the // next most recent, and so on. If such an element doesn't exist then ok is // false. func (s *stRingBuffer) NthPreviousEntry(n int) (value string, ok bool) { if n >= s.size { return "", false } index := s.head - n if index < 0 { index += s.max } return s.entries[index], true } mongo-tools-r3.2.11/vendor/src/golang.org/x/crypto/ssh/terminal/terminal_test.go000066400000000000000000000123261276133741700277750ustar00rootroot00000000000000// Copyright 2011 The Go Authors. All rights reserved. // Use of this source code is governed by a BSD-style // license that can be found in the LICENSE file. package terminal import ( "io" "testing" ) type MockTerminal struct { toSend []byte bytesPerRead int received []byte } func (c *MockTerminal) Read(data []byte) (n int, err error) { n = len(data) if n == 0 { return } if n > len(c.toSend) { n = len(c.toSend) } if n == 0 { return 0, io.EOF } if c.bytesPerRead > 0 && n > c.bytesPerRead { n = c.bytesPerRead } copy(data, c.toSend[:n]) c.toSend = c.toSend[n:] return } func (c *MockTerminal) Write(data []byte) (n int, err error) { c.received = append(c.received, data...) return len(data), nil } func TestClose(t *testing.T) { c := &MockTerminal{} ss := NewTerminal(c, "> ") line, err := ss.ReadLine() if line != "" { t.Errorf("Expected empty line but got: %s", line) } if err != io.EOF { t.Errorf("Error should have been EOF but got: %s", err) } } var keyPressTests = []struct { in string line string err error throwAwayLines int }{ { err: io.EOF, }, { in: "\r", line: "", }, { in: "foo\r", line: "foo", }, { in: "a\x1b[Cb\r", // right line: "ab", }, { in: "a\x1b[Db\r", // left line: "ba", }, { in: "a\177b\r", // backspace line: "b", }, { in: "\x1b[A\r", // up }, { in: "\x1b[B\r", // down }, { in: "line\x1b[A\x1b[B\r", // up then down line: "line", }, { in: "line1\rline2\x1b[A\r", // recall previous line. line: "line1", throwAwayLines: 1, }, { // recall two previous lines and append. in: "line1\rline2\rline3\x1b[A\x1b[Axxx\r", line: "line1xxx", throwAwayLines: 2, }, { // Ctrl-A to move to beginning of line followed by ^K to kill // line. in: "a b \001\013\r", line: "", }, { // Ctrl-A to move to beginning of line, Ctrl-E to move to end, // finally ^K to kill nothing. in: "a b \001\005\013\r", line: "a b ", }, { in: "\027\r", line: "", }, { in: "a\027\r", line: "", }, { in: "a \027\r", line: "", }, { in: "a b\027\r", line: "a ", }, { in: "a b \027\r", line: "a ", }, { in: "one two thr\x1b[D\027\r", line: "one two r", }, { in: "\013\r", line: "", }, { in: "a\013\r", line: "a", }, { in: "ab\x1b[D\013\r", line: "a", }, { in: "Ξεσκεπάζω\r", line: "Ξεσκεπάζω", }, { in: "£\r\x1b[A\177\r", // non-ASCII char, enter, up, backspace. line: "", throwAwayLines: 1, }, { in: "£\r££\x1b[A\x1b[B\177\r", // non-ASCII char, enter, 2x non-ASCII, up, down, backspace, enter. line: "£", throwAwayLines: 1, }, { // Ctrl-D at the end of the line should be ignored. in: "a\004\r", line: "a", }, { // a, b, left, Ctrl-D should erase the b. in: "ab\x1b[D\004\r", line: "a", }, { // a, b, c, d, left, left, ^U should erase to the beginning of // the line. in: "abcd\x1b[D\x1b[D\025\r", line: "cd", }, { // Bracketed paste mode: control sequences should be returned // verbatim in paste mode. in: "abc\x1b[200~de\177f\x1b[201~\177\r", line: "abcde\177", }, { // Enter in bracketed paste mode should still work. in: "abc\x1b[200~d\refg\x1b[201~h\r", line: "efgh", throwAwayLines: 1, }, { // Lines consisting entirely of pasted data should be indicated as such. in: "\x1b[200~a\r", line: "a", err: ErrPasteIndicator, }, } func TestKeyPresses(t *testing.T) { for i, test := range keyPressTests { for j := 1; j < len(test.in); j++ { c := &MockTerminal{ toSend: []byte(test.in), bytesPerRead: j, } ss := NewTerminal(c, "> ") for k := 0; k < test.throwAwayLines; k++ { _, err := ss.ReadLine() if err != nil { t.Errorf("Throwaway line %d from test %d resulted in error: %s", k, i, err) } } line, err := ss.ReadLine() if line != test.line { t.Errorf("Line resulting from test %d (%d bytes per read) was '%s', expected '%s'", i, j, line, test.line) break } if err != test.err { t.Errorf("Error resulting from test %d (%d bytes per read) was '%v', expected '%v'", i, j, err, test.err) break } } } } func TestPasswordNotSaved(t *testing.T) { c := &MockTerminal{ toSend: []byte("password\r\x1b[A\r"), bytesPerRead: 1, } ss := NewTerminal(c, "> ") pw, _ := ss.ReadPassword("> ") if pw != "password" { t.Fatalf("failed to read password, got %s", pw) } line, _ := ss.ReadLine() if len(line) > 0 { t.Fatalf("password was saved in history") } } var setSizeTests = []struct { width, height int }{ {40, 13}, {80, 24}, {132, 43}, } func TestTerminalSetSize(t *testing.T) { for _, setSize := range setSizeTests { c := &MockTerminal{ toSend: []byte("password\r\x1b[A\r"), bytesPerRead: 1, } ss := NewTerminal(c, "> ") ss.SetSize(setSize.width, setSize.height) pw, _ := ss.ReadPassword("Password: ") if pw != "password" { t.Fatalf("failed to read password, got %s", pw) } if string(c.received) != "Password: \r\n" { t.Errorf("failed to set the temporary prompt expected %q, got %q", "Password: ", c.received) } } } mongo-tools-r3.2.11/vendor/src/golang.org/x/crypto/ssh/terminal/util.go000066400000000000000000000076711276133741700261070ustar00rootroot00000000000000// Copyright 2011 The Go Authors. All rights reserved. // Use of this source code is governed by a BSD-style // license that can be found in the LICENSE file. // +build darwin dragonfly freebsd linux,!appengine netbsd openbsd // Package terminal provides support functions for dealing with terminals, as // commonly found on UNIX systems. // // Putting a terminal into raw mode is the most common requirement: // // oldState, err := terminal.MakeRaw(0) // if err != nil { // panic(err) // } // defer terminal.Restore(0, oldState) package terminal // import "golang.org/x/crypto/ssh/terminal" import ( "io" "syscall" "unsafe" ) // State contains the state of a terminal. type State struct { termios syscall.Termios } // IsTerminal returns true if the given file descriptor is a terminal. func IsTerminal(fd int) bool { var termios syscall.Termios _, _, err := syscall.Syscall6(syscall.SYS_IOCTL, uintptr(fd), ioctlReadTermios, uintptr(unsafe.Pointer(&termios)), 0, 0, 0) return err == 0 } // MakeRaw put the terminal connected to the given file descriptor into raw // mode and returns the previous state of the terminal so that it can be // restored. func MakeRaw(fd int) (*State, error) { var oldState State if _, _, err := syscall.Syscall6(syscall.SYS_IOCTL, uintptr(fd), ioctlReadTermios, uintptr(unsafe.Pointer(&oldState.termios)), 0, 0, 0); err != 0 { return nil, err } newState := oldState.termios newState.Iflag &^= syscall.ISTRIP | syscall.INLCR | syscall.ICRNL | syscall.IGNCR | syscall.IXON | syscall.IXOFF newState.Lflag &^= syscall.ECHO | syscall.ICANON | syscall.ISIG if _, _, err := syscall.Syscall6(syscall.SYS_IOCTL, uintptr(fd), ioctlWriteTermios, uintptr(unsafe.Pointer(&newState)), 0, 0, 0); err != 0 { return nil, err } return &oldState, nil } // GetState returns the current state of a terminal which may be useful to // restore the terminal after a signal. func GetState(fd int) (*State, error) { var oldState State if _, _, err := syscall.Syscall6(syscall.SYS_IOCTL, uintptr(fd), ioctlReadTermios, uintptr(unsafe.Pointer(&oldState.termios)), 0, 0, 0); err != 0 { return nil, err } return &oldState, nil } // Restore restores the terminal connected to the given file descriptor to a // previous state. func Restore(fd int, state *State) error { _, _, err := syscall.Syscall6(syscall.SYS_IOCTL, uintptr(fd), ioctlWriteTermios, uintptr(unsafe.Pointer(&state.termios)), 0, 0, 0) return err } // GetSize returns the dimensions of the given terminal. func GetSize(fd int) (width, height int, err error) { var dimensions [4]uint16 if _, _, err := syscall.Syscall6(syscall.SYS_IOCTL, uintptr(fd), uintptr(syscall.TIOCGWINSZ), uintptr(unsafe.Pointer(&dimensions)), 0, 0, 0); err != 0 { return -1, -1, err } return int(dimensions[1]), int(dimensions[0]), nil } // ReadPassword reads a line of input from a terminal without local echo. This // is commonly used for inputting passwords and other sensitive data. The slice // returned does not include the \n. func ReadPassword(fd int) ([]byte, error) { var oldState syscall.Termios if _, _, err := syscall.Syscall6(syscall.SYS_IOCTL, uintptr(fd), ioctlReadTermios, uintptr(unsafe.Pointer(&oldState)), 0, 0, 0); err != 0 { return nil, err } newState := oldState newState.Lflag &^= syscall.ECHO newState.Lflag |= syscall.ICANON | syscall.ISIG newState.Iflag |= syscall.ICRNL if _, _, err := syscall.Syscall6(syscall.SYS_IOCTL, uintptr(fd), ioctlWriteTermios, uintptr(unsafe.Pointer(&newState)), 0, 0, 0); err != 0 { return nil, err } defer func() { syscall.Syscall6(syscall.SYS_IOCTL, uintptr(fd), ioctlWriteTermios, uintptr(unsafe.Pointer(&oldState)), 0, 0, 0) }() var buf [16]byte var ret []byte for { n, err := syscall.Read(fd, buf[:]) if err != nil { return nil, err } if n == 0 { if len(ret) == 0 { return nil, io.EOF } break } if buf[n-1] == '\n' { n-- } ret = append(ret, buf[:n]...) if n < len(buf) { break } } return ret, nil } mongo-tools-r3.2.11/vendor/src/golang.org/x/crypto/ssh/terminal/util_bsd.go000066400000000000000000000005141276133741700267240ustar00rootroot00000000000000// Copyright 2013 The Go Authors. All rights reserved. // Use of this source code is governed by a BSD-style // license that can be found in the LICENSE file. // +build darwin dragonfly freebsd netbsd openbsd package terminal import "syscall" const ioctlReadTermios = syscall.TIOCGETA const ioctlWriteTermios = syscall.TIOCSETA mongo-tools-r3.2.11/vendor/src/golang.org/x/crypto/ssh/terminal/util_linux.go000066400000000000000000000007111276133741700273120ustar00rootroot00000000000000// Copyright 2013 The Go Authors. All rights reserved. // Use of this source code is governed by a BSD-style // license that can be found in the LICENSE file. package terminal // These constants are declared here, rather than importing // them from the syscall package as some syscall packages, even // on linux, for example gccgo, do not declare them. const ioctlReadTermios = 0x5401 // syscall.TCGETS const ioctlWriteTermios = 0x5402 // syscall.TCSETS mongo-tools-r3.2.11/vendor/src/golang.org/x/crypto/ssh/terminal/util_windows.go000066400000000000000000000105071276133741700276510ustar00rootroot00000000000000// Copyright 2011 The Go Authors. All rights reserved. // Use of this source code is governed by a BSD-style // license that can be found in the LICENSE file. // +build windows // Package terminal provides support functions for dealing with terminals, as // commonly found on UNIX systems. // // Putting a terminal into raw mode is the most common requirement: // // oldState, err := terminal.MakeRaw(0) // if err != nil { // panic(err) // } // defer terminal.Restore(0, oldState) package terminal import ( "io" "syscall" "unsafe" ) const ( enableLineInput = 2 enableEchoInput = 4 enableProcessedInput = 1 enableWindowInput = 8 enableMouseInput = 16 enableInsertMode = 32 enableQuickEditMode = 64 enableExtendedFlags = 128 enableAutoPosition = 256 enableProcessedOutput = 1 enableWrapAtEolOutput = 2 ) var kernel32 = syscall.NewLazyDLL("kernel32.dll") var ( procGetConsoleMode = kernel32.NewProc("GetConsoleMode") procSetConsoleMode = kernel32.NewProc("SetConsoleMode") procGetConsoleScreenBufferInfo = kernel32.NewProc("GetConsoleScreenBufferInfo") ) type ( short int16 word uint16 coord struct { x short y short } smallRect struct { left short top short right short bottom short } consoleScreenBufferInfo struct { size coord cursorPosition coord attributes word window smallRect maximumWindowSize coord } ) type State struct { mode uint32 } // IsTerminal returns true if the given file descriptor is a terminal. func IsTerminal(fd int) bool { var st uint32 r, _, e := syscall.Syscall(procGetConsoleMode.Addr(), 2, uintptr(fd), uintptr(unsafe.Pointer(&st)), 0) return r != 0 && e == 0 } // MakeRaw put the terminal connected to the given file descriptor into raw // mode and returns the previous state of the terminal so that it can be // restored. func MakeRaw(fd int) (*State, error) { var st uint32 _, _, e := syscall.Syscall(procGetConsoleMode.Addr(), 2, uintptr(fd), uintptr(unsafe.Pointer(&st)), 0) if e != 0 { return nil, error(e) } st &^= (enableEchoInput | enableProcessedInput | enableLineInput | enableProcessedOutput) _, _, e = syscall.Syscall(procSetConsoleMode.Addr(), 2, uintptr(fd), uintptr(st), 0) if e != 0 { return nil, error(e) } return &State{st}, nil } // GetState returns the current state of a terminal which may be useful to // restore the terminal after a signal. func GetState(fd int) (*State, error) { var st uint32 _, _, e := syscall.Syscall(procGetConsoleMode.Addr(), 2, uintptr(fd), uintptr(unsafe.Pointer(&st)), 0) if e != 0 { return nil, error(e) } return &State{st}, nil } // Restore restores the terminal connected to the given file descriptor to a // previous state. func Restore(fd int, state *State) error { _, _, err := syscall.Syscall(procSetConsoleMode.Addr(), 2, uintptr(fd), uintptr(state.mode), 0) return err } // GetSize returns the dimensions of the given terminal. func GetSize(fd int) (width, height int, err error) { var info consoleScreenBufferInfo _, _, e := syscall.Syscall(procGetConsoleScreenBufferInfo.Addr(), 2, uintptr(fd), uintptr(unsafe.Pointer(&info)), 0) if e != 0 { return 0, 0, error(e) } return int(info.size.x), int(info.size.y), nil } // ReadPassword reads a line of input from a terminal without local echo. This // is commonly used for inputting passwords and other sensitive data. The slice // returned does not include the \n. func ReadPassword(fd int) ([]byte, error) { var st uint32 _, _, e := syscall.Syscall(procGetConsoleMode.Addr(), 2, uintptr(fd), uintptr(unsafe.Pointer(&st)), 0) if e != 0 { return nil, error(e) } old := st st &^= (enableEchoInput) st |= (enableProcessedInput | enableLineInput | enableProcessedOutput) _, _, e = syscall.Syscall(procSetConsoleMode.Addr(), 2, uintptr(fd), uintptr(st), 0) if e != 0 { return nil, error(e) } defer func() { syscall.Syscall(procSetConsoleMode.Addr(), 2, uintptr(fd), uintptr(old), 0) }() var buf [16]byte var ret []byte for { n, err := syscall.Read(syscall.Handle(fd), buf[:]) if err != nil { return nil, err } if n == 0 { if len(ret) == 0 { return nil, io.EOF } break } if buf[n-1] == '\n' { n-- } if n > 0 && buf[n-1] == '\r' { n-- } ret = append(ret, buf[:n]...) if n < len(buf) { break } } return ret, nil } mongo-tools-r3.2.11/vendor/src/gopkg.in/000077500000000000000000000000001276133741700200505ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/000077500000000000000000000000001276133741700211605ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/.travis.yml000066400000000000000000000020041276133741700232650ustar00rootroot00000000000000language: go go_import_path: gopkg.in/mgo.v2 addons: apt: packages: env: global: - BUCKET=https://niemeyer.s3.amazonaws.com matrix: - GO=1.4.1 MONGODB=x86_64-2.2.7 - GO=1.4.1 MONGODB=x86_64-2.4.14 - GO=1.4.1 MONGODB=x86_64-2.6.11 - GO=1.4.1 MONGODB=x86_64-3.0.9 - GO=1.4.1 MONGODB=x86_64-3.2.3-nojournal - GO=1.5.3 MONGODB=x86_64-3.0.9 - GO=1.6 MONGODB=x86_64-3.0.9 install: - eval "$(gimme $GO)" - wget $BUCKET/mongodb-linux-$MONGODB.tgz - tar xzvf mongodb-linux-$MONGODB.tgz - export PATH=$PWD/mongodb-linux-$MONGODB/bin:$PATH - wget $BUCKET/daemontools.tar.gz - tar xzvf daemontools.tar.gz - export PATH=$PWD/daemontools:$PATH - go get gopkg.in/check.v1 - go get gopkg.in/yaml.v2 - go get gopkg.in/tomb.v2 before_script: - export NOIPV6=1 - make startdb script: - (cd bson && go test -check.v) - go test -check.v -fast - (cd txn && go test -check.v) # vim:sw=4:ts=4:et mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/LICENSE000066400000000000000000000025241276133741700221700ustar00rootroot00000000000000mgo - MongoDB driver for Go Copyright (c) 2010-2013 - Gustavo Niemeyer All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/Makefile000066400000000000000000000001031276133741700226120ustar00rootroot00000000000000startdb: @harness/setup.sh start stopdb: @harness/setup.sh stop mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/README.md000066400000000000000000000002101276133741700224300ustar00rootroot00000000000000The MongoDB driver for Go ------------------------- Please go to [http://labix.org/mgo](http://labix.org/mgo) for all project details. mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/auth.go000066400000000000000000000302151276133741700224510ustar00rootroot00000000000000// mgo - MongoDB driver for Go // // Copyright (c) 2010-2012 - Gustavo Niemeyer // // All rights reserved. // // Redistribution and use in source and binary forms, with or without // modification, are permitted provided that the following conditions are met: // // 1. Redistributions of source code must retain the above copyright notice, this // list of conditions and the following disclaimer. // 2. Redistributions in binary form must reproduce the above copyright notice, // this list of conditions and the following disclaimer in the documentation // and/or other materials provided with the distribution. // // THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND // ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED // WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE // DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR // ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES // (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; // LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND // ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT // (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS // SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. package mgo import ( "crypto/md5" "crypto/sha1" "encoding/hex" "errors" "fmt" "sync" "gopkg.in/mgo.v2/bson" "gopkg.in/mgo.v2/internal/scram" ) type authCmd struct { Authenticate int Nonce string User string Key string } type startSaslCmd struct { StartSASL int `bson:"startSasl"` } type authResult struct { ErrMsg string Ok bool } type getNonceCmd struct { GetNonce int } type getNonceResult struct { Nonce string Err string "$err" Code int } type logoutCmd struct { Logout int } type saslCmd struct { Start int `bson:"saslStart,omitempty"` Continue int `bson:"saslContinue,omitempty"` ConversationId int `bson:"conversationId,omitempty"` Mechanism string `bson:"mechanism,omitempty"` Payload []byte } type saslResult struct { Ok bool `bson:"ok"` NotOk bool `bson:"code"` // Server <= 2.3.2 returns ok=1 & code>0 on errors (WTF?) Done bool ConversationId int `bson:"conversationId"` Payload []byte ErrMsg string } type saslStepper interface { Step(serverData []byte) (clientData []byte, done bool, err error) Close() } func (socket *mongoSocket) getNonce() (nonce string, err error) { socket.Lock() for socket.cachedNonce == "" && socket.dead == nil { debugf("Socket %p to %s: waiting for nonce", socket, socket.addr) socket.gotNonce.Wait() } if socket.cachedNonce == "mongos" { socket.Unlock() return "", errors.New("Can't authenticate with mongos; see http://j.mp/mongos-auth") } debugf("Socket %p to %s: got nonce", socket, socket.addr) nonce, err = socket.cachedNonce, socket.dead socket.cachedNonce = "" socket.Unlock() if err != nil { nonce = "" } return } func (socket *mongoSocket) resetNonce() { debugf("Socket %p to %s: requesting a new nonce", socket, socket.addr) op := &queryOp{} op.query = &getNonceCmd{GetNonce: 1} op.collection = "admin.$cmd" op.limit = -1 op.replyFunc = func(err error, reply *replyOp, docNum int, docData []byte) { if err != nil { socket.kill(errors.New("getNonce: "+err.Error()), true) return } result := &getNonceResult{} err = bson.Unmarshal(docData, &result) if err != nil { socket.kill(errors.New("Failed to unmarshal nonce: "+err.Error()), true) return } debugf("Socket %p to %s: nonce unmarshalled: %#v", socket, socket.addr, result) if result.Code == 13390 { // mongos doesn't yet support auth (see http://j.mp/mongos-auth) result.Nonce = "mongos" } else if result.Nonce == "" { var msg string if result.Err != "" { msg = fmt.Sprintf("Got an empty nonce: %s (%d)", result.Err, result.Code) } else { msg = "Got an empty nonce" } socket.kill(errors.New(msg), true) return } socket.Lock() if socket.cachedNonce != "" { socket.Unlock() panic("resetNonce: nonce already cached") } socket.cachedNonce = result.Nonce socket.gotNonce.Signal() socket.Unlock() } err := socket.Query(op) if err != nil { socket.kill(errors.New("resetNonce: "+err.Error()), true) } } func (socket *mongoSocket) Login(cred Credential) error { socket.Lock() if cred.Mechanism == "" && socket.serverInfo.MaxWireVersion >= 3 { cred.Mechanism = "SCRAM-SHA-1" } for _, sockCred := range socket.creds { if sockCred == cred { debugf("Socket %p to %s: login: db=%q user=%q (already logged in)", socket, socket.addr, cred.Source, cred.Username) socket.Unlock() return nil } } if socket.dropLogout(cred) { debugf("Socket %p to %s: login: db=%q user=%q (cached)", socket, socket.addr, cred.Source, cred.Username) socket.creds = append(socket.creds, cred) socket.Unlock() return nil } socket.Unlock() debugf("Socket %p to %s: login: db=%q user=%q", socket, socket.addr, cred.Source, cred.Username) var err error switch cred.Mechanism { case "", "MONGODB-CR", "MONGO-CR": // Name changed to MONGODB-CR in SERVER-8501. err = socket.loginClassic(cred) case "PLAIN": err = socket.loginPlain(cred) case "MONGODB-X509": err = socket.loginX509(cred) default: // Try SASL for everything else, if it is available. err = socket.loginSASL(cred) } if err != nil { debugf("Socket %p to %s: login error: %s", socket, socket.addr, err) } else { debugf("Socket %p to %s: login successful", socket, socket.addr) } return err } func (socket *mongoSocket) loginClassic(cred Credential) error { // Note that this only works properly because this function is // synchronous, which means the nonce won't get reset while we're // using it and any other login requests will block waiting for a // new nonce provided in the defer call below. nonce, err := socket.getNonce() if err != nil { return err } defer socket.resetNonce() psum := md5.New() psum.Write([]byte(cred.Username + ":mongo:" + cred.Password)) ksum := md5.New() ksum.Write([]byte(nonce + cred.Username)) ksum.Write([]byte(hex.EncodeToString(psum.Sum(nil)))) key := hex.EncodeToString(ksum.Sum(nil)) cmd := authCmd{Authenticate: 1, User: cred.Username, Nonce: nonce, Key: key} res := authResult{} return socket.loginRun(cred.Source, &cmd, &res, func() error { if !res.Ok { return errors.New(res.ErrMsg) } socket.Lock() socket.dropAuth(cred.Source) socket.creds = append(socket.creds, cred) socket.Unlock() return nil }) } type authX509Cmd struct { Authenticate int User string Mechanism string } func (socket *mongoSocket) loginX509(cred Credential) error { cmd := authX509Cmd{Authenticate: 1, User: cred.Username, Mechanism: "MONGODB-X509"} res := authResult{} return socket.loginRun(cred.Source, &cmd, &res, func() error { if !res.Ok { return errors.New(res.ErrMsg) } socket.Lock() socket.dropAuth(cred.Source) socket.creds = append(socket.creds, cred) socket.Unlock() return nil }) } func (socket *mongoSocket) loginPlain(cred Credential) error { cmd := saslCmd{Start: 1, Mechanism: "PLAIN", Payload: []byte("\x00" + cred.Username + "\x00" + cred.Password)} res := authResult{} return socket.loginRun(cred.Source, &cmd, &res, func() error { if !res.Ok { return errors.New(res.ErrMsg) } socket.Lock() socket.dropAuth(cred.Source) socket.creds = append(socket.creds, cred) socket.Unlock() return nil }) } func (socket *mongoSocket) loginSASL(cred Credential) error { var sasl saslStepper var err error if cred.Mechanism == "SCRAM-SHA-1" { // SCRAM is handled without external libraries. sasl = saslNewScram(cred) } else if len(cred.ServiceHost) > 0 { sasl, err = saslNew(cred, cred.ServiceHost) } else { sasl, err = saslNew(cred, socket.Server().Addr) } if err != nil { return err } defer sasl.Close() // The goal of this logic is to carry a locked socket until the // local SASL step confirms the auth is valid; the socket needs to be // locked so that concurrent action doesn't leave the socket in an // auth state that doesn't reflect the operations that took place. // As a simple case, imagine inverting login=>logout to logout=>login. // // The logic below works because the lock func isn't called concurrently. locked := false lock := func(b bool) { if locked != b { locked = b if b { socket.Lock() } else { socket.Unlock() } } } lock(true) defer lock(false) start := 1 cmd := saslCmd{} res := saslResult{} for { payload, done, err := sasl.Step(res.Payload) if err != nil { return err } if done && res.Done { socket.dropAuth(cred.Source) socket.creds = append(socket.creds, cred) break } lock(false) cmd = saslCmd{ Start: start, Continue: 1 - start, ConversationId: res.ConversationId, Mechanism: cred.Mechanism, Payload: payload, } start = 0 err = socket.loginRun(cred.Source, &cmd, &res, func() error { // See the comment on lock for why this is necessary. lock(true) if !res.Ok || res.NotOk { return fmt.Errorf("server returned error on SASL authentication step: %s", res.ErrMsg) } return nil }) if err != nil { return err } if done && res.Done { socket.dropAuth(cred.Source) socket.creds = append(socket.creds, cred) break } } return nil } func saslNewScram(cred Credential) *saslScram { credsum := md5.New() credsum.Write([]byte(cred.Username + ":mongo:" + cred.Password)) client := scram.NewClient(sha1.New, cred.Username, hex.EncodeToString(credsum.Sum(nil))) return &saslScram{cred: cred, client: client} } type saslScram struct { cred Credential client *scram.Client } func (s *saslScram) Close() {} func (s *saslScram) Step(serverData []byte) (clientData []byte, done bool, err error) { more := s.client.Step(serverData) return s.client.Out(), !more, s.client.Err() } func (socket *mongoSocket) loginRun(db string, query, result interface{}, f func() error) error { var mutex sync.Mutex var replyErr error mutex.Lock() op := queryOp{} op.query = query op.collection = db + ".$cmd" op.limit = -1 op.replyFunc = func(err error, reply *replyOp, docNum int, docData []byte) { defer mutex.Unlock() if err != nil { replyErr = err return } err = bson.Unmarshal(docData, result) if err != nil { replyErr = err } else { // Must handle this within the read loop for the socket, so // that concurrent login requests are properly ordered. replyErr = f() } } err := socket.Query(&op) if err != nil { return err } mutex.Lock() // Wait. return replyErr } func (socket *mongoSocket) Logout(db string) { socket.Lock() cred, found := socket.dropAuth(db) if found { debugf("Socket %p to %s: logout: db=%q (flagged)", socket, socket.addr, db) socket.logout = append(socket.logout, cred) } socket.Unlock() } func (socket *mongoSocket) LogoutAll() { socket.Lock() if l := len(socket.creds); l > 0 { debugf("Socket %p to %s: logout all (flagged %d)", socket, socket.addr, l) socket.logout = append(socket.logout, socket.creds...) socket.creds = socket.creds[0:0] } socket.Unlock() } func (socket *mongoSocket) flushLogout() (ops []interface{}) { socket.Lock() if l := len(socket.logout); l > 0 { debugf("Socket %p to %s: logout all (flushing %d)", socket, socket.addr, l) for i := 0; i != l; i++ { op := queryOp{} op.query = &logoutCmd{1} op.collection = socket.logout[i].Source + ".$cmd" op.limit = -1 ops = append(ops, &op) } socket.logout = socket.logout[0:0] } socket.Unlock() return } func (socket *mongoSocket) dropAuth(db string) (cred Credential, found bool) { for i, sockCred := range socket.creds { if sockCred.Source == db { copy(socket.creds[i:], socket.creds[i+1:]) socket.creds = socket.creds[:len(socket.creds)-1] return sockCred, true } } return cred, false } func (socket *mongoSocket) dropLogout(cred Credential) (found bool) { for i, sockCred := range socket.logout { if sockCred == cred { copy(socket.logout[i:], socket.logout[i+1:]) socket.logout = socket.logout[:len(socket.logout)-1] return true } } return false } mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/auth_test.go000066400000000000000000000761051276133741700235200ustar00rootroot00000000000000// mgo - MongoDB driver for Go // // Copyright (c) 2010-2012 - Gustavo Niemeyer // // All rights reserved. // // Redistribution and use in source and binary forms, with or without // modification, are permitted provided that the following conditions are met: // // 1. Redistributions of source code must retain the above copyright notice, this // list of conditions and the following disclaimer. // 2. Redistributions in binary form must reproduce the above copyright notice, // this list of conditions and the following disclaimer in the documentation // and/or other materials provided with the distribution. // // THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND // ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED // WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE // DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR // ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES // (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; // LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND // ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT // (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS // SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. package mgo_test import ( "crypto/tls" "flag" "fmt" "io/ioutil" "net" "net/url" "os" "runtime" "sync" "time" . "gopkg.in/check.v1" "gopkg.in/mgo.v2" ) func (s *S) TestAuthLoginDatabase(c *C) { // Test both with a normal database and with an authenticated shard. for _, addr := range []string{"localhost:40002", "localhost:40203"} { session, err := mgo.Dial(addr) c.Assert(err, IsNil) defer session.Close() coll := session.DB("mydb").C("mycoll") err = coll.Insert(M{"n": 1}) c.Assert(err, ErrorMatches, "unauthorized|need to login|not authorized .*") admindb := session.DB("admin") err = admindb.Login("root", "wrong") c.Assert(err, ErrorMatches, "auth fail(s|ed)|.*Authentication failed.") err = admindb.Login("root", "rapadura") c.Assert(err, IsNil) err = coll.Insert(M{"n": 1}) c.Assert(err, IsNil) } } func (s *S) TestAuthLoginSession(c *C) { // Test both with a normal database and with an authenticated shard. for _, addr := range []string{"localhost:40002", "localhost:40203"} { session, err := mgo.Dial(addr) c.Assert(err, IsNil) defer session.Close() coll := session.DB("mydb").C("mycoll") err = coll.Insert(M{"n": 1}) c.Assert(err, ErrorMatches, "unauthorized|need to login|not authorized .*") cred := mgo.Credential{ Username: "root", Password: "wrong", } err = session.Login(&cred) c.Assert(err, ErrorMatches, "auth fail(s|ed)|.*Authentication failed.") cred.Password = "rapadura" err = session.Login(&cred) c.Assert(err, IsNil) err = coll.Insert(M{"n": 1}) c.Assert(err, IsNil) } } func (s *S) TestAuthLoginLogout(c *C) { // Test both with a normal database and with an authenticated shard. for _, addr := range []string{"localhost:40002", "localhost:40203"} { session, err := mgo.Dial(addr) c.Assert(err, IsNil) defer session.Close() admindb := session.DB("admin") err = admindb.Login("root", "rapadura") c.Assert(err, IsNil) admindb.Logout() coll := session.DB("mydb").C("mycoll") err = coll.Insert(M{"n": 1}) c.Assert(err, ErrorMatches, "unauthorized|need to login|not authorized .*") // Must have dropped auth from the session too. session = session.Copy() defer session.Close() coll = session.DB("mydb").C("mycoll") err = coll.Insert(M{"n": 1}) c.Assert(err, ErrorMatches, "unauthorized|need to login|not authorized .*") } } func (s *S) TestAuthLoginLogoutAll(c *C) { session, err := mgo.Dial("localhost:40002") c.Assert(err, IsNil) defer session.Close() admindb := session.DB("admin") err = admindb.Login("root", "rapadura") c.Assert(err, IsNil) session.LogoutAll() coll := session.DB("mydb").C("mycoll") err = coll.Insert(M{"n": 1}) c.Assert(err, ErrorMatches, "unauthorized|need to login|not authorized .*") // Must have dropped auth from the session too. session = session.Copy() defer session.Close() coll = session.DB("mydb").C("mycoll") err = coll.Insert(M{"n": 1}) c.Assert(err, ErrorMatches, "unauthorized|need to login|not authorized .*") } func (s *S) TestAuthUpsertUserErrors(c *C) { session, err := mgo.Dial("localhost:40002") c.Assert(err, IsNil) defer session.Close() admindb := session.DB("admin") err = admindb.Login("root", "rapadura") c.Assert(err, IsNil) mydb := session.DB("mydb") err = mydb.UpsertUser(&mgo.User{}) c.Assert(err, ErrorMatches, "user has no Username") err = mydb.UpsertUser(&mgo.User{Username: "user", Password: "pass", UserSource: "source"}) c.Assert(err, ErrorMatches, "user has both Password/PasswordHash and UserSource set") err = mydb.UpsertUser(&mgo.User{Username: "user", Password: "pass", OtherDBRoles: map[string][]mgo.Role{"db": nil}}) c.Assert(err, ErrorMatches, "user with OtherDBRoles is only supported in the admin or \\$external databases") } func (s *S) TestAuthUpsertUser(c *C) { if !s.versionAtLeast(2, 4) { c.Skip("UpsertUser only works on 2.4+") } session, err := mgo.Dial("localhost:40002") c.Assert(err, IsNil) defer session.Close() admindb := session.DB("admin") err = admindb.Login("root", "rapadura") c.Assert(err, IsNil) mydb := session.DB("mydb") ruser := &mgo.User{ Username: "myruser", Password: "mypass", Roles: []mgo.Role{mgo.RoleRead}, } rwuser := &mgo.User{ Username: "myrwuser", Password: "mypass", Roles: []mgo.Role{mgo.RoleReadWrite}, } err = mydb.UpsertUser(ruser) c.Assert(err, IsNil) err = mydb.UpsertUser(rwuser) c.Assert(err, IsNil) err = mydb.Login("myruser", "mypass") c.Assert(err, IsNil) admindb.Logout() coll := session.DB("mydb").C("mycoll") err = coll.Insert(M{"n": 1}) c.Assert(err, ErrorMatches, "unauthorized|not authorized .*") err = mydb.Login("myrwuser", "mypass") c.Assert(err, IsNil) err = coll.Insert(M{"n": 1}) c.Assert(err, IsNil) myotherdb := session.DB("myotherdb") err = admindb.Login("root", "rapadura") c.Assert(err, IsNil) // Test UserSource. rwuserother := &mgo.User{ Username: "myrwuser", UserSource: "mydb", Roles: []mgo.Role{mgo.RoleRead}, } err = myotherdb.UpsertUser(rwuserother) if s.versionAtLeast(2, 6) { c.Assert(err, ErrorMatches, `MongoDB 2.6\+ does not support the UserSource setting`) return } c.Assert(err, IsNil) admindb.Logout() // Test indirection via UserSource: we can't write to it, because // the roles for myrwuser are different there. othercoll := myotherdb.C("myothercoll") err = othercoll.Insert(M{"n": 1}) c.Assert(err, ErrorMatches, "unauthorized|not authorized .*") // Reading works, though. err = othercoll.Find(nil).One(nil) c.Assert(err, Equals, mgo.ErrNotFound) // Can't login directly into the database using UserSource, though. err = myotherdb.Login("myrwuser", "mypass") c.Assert(err, ErrorMatches, "auth fail(s|ed)|.*Authentication failed.") } func (s *S) TestAuthUpsertUserOtherDBRoles(c *C) { if !s.versionAtLeast(2, 4) { c.Skip("UpsertUser only works on 2.4+") } session, err := mgo.Dial("localhost:40002") c.Assert(err, IsNil) defer session.Close() admindb := session.DB("admin") err = admindb.Login("root", "rapadura") c.Assert(err, IsNil) ruser := &mgo.User{ Username: "myruser", Password: "mypass", OtherDBRoles: map[string][]mgo.Role{"mydb": []mgo.Role{mgo.RoleRead}}, } err = admindb.UpsertUser(ruser) c.Assert(err, IsNil) defer admindb.RemoveUser("myruser") admindb.Logout() err = admindb.Login("myruser", "mypass") coll := session.DB("mydb").C("mycoll") err = coll.Insert(M{"n": 1}) c.Assert(err, ErrorMatches, "unauthorized|not authorized .*") err = coll.Find(nil).One(nil) c.Assert(err, Equals, mgo.ErrNotFound) } func (s *S) TestAuthUpsertUserUpdates(c *C) { if !s.versionAtLeast(2, 4) { c.Skip("UpsertUser only works on 2.4+") } session, err := mgo.Dial("localhost:40002") c.Assert(err, IsNil) defer session.Close() admindb := session.DB("admin") err = admindb.Login("root", "rapadura") c.Assert(err, IsNil) mydb := session.DB("mydb") // Insert a user that can read. user := &mgo.User{ Username: "myruser", Password: "mypass", Roles: []mgo.Role{mgo.RoleRead}, } err = mydb.UpsertUser(user) c.Assert(err, IsNil) // Now update the user password. user = &mgo.User{ Username: "myruser", Password: "mynewpass", } err = mydb.UpsertUser(user) c.Assert(err, IsNil) // Login with the new user. usession, err := mgo.Dial("myruser:mynewpass@localhost:40002/mydb") c.Assert(err, IsNil) defer usession.Close() // Can read, but not write. err = usession.DB("mydb").C("mycoll").Find(nil).One(nil) c.Assert(err, Equals, mgo.ErrNotFound) err = usession.DB("mydb").C("mycoll").Insert(M{"ok": 1}) c.Assert(err, ErrorMatches, "unauthorized|not authorized .*") // Update the user role. user = &mgo.User{ Username: "myruser", Roles: []mgo.Role{mgo.RoleReadWrite}, } err = mydb.UpsertUser(user) c.Assert(err, IsNil) // Dial again to ensure the password hasn't changed. usession, err = mgo.Dial("myruser:mynewpass@localhost:40002/mydb") c.Assert(err, IsNil) defer usession.Close() // Now it can write. err = usession.DB("mydb").C("mycoll").Insert(M{"ok": 1}) c.Assert(err, IsNil) } func (s *S) TestAuthAddUser(c *C) { session, err := mgo.Dial("localhost:40002") c.Assert(err, IsNil) defer session.Close() admindb := session.DB("admin") err = admindb.Login("root", "rapadura") c.Assert(err, IsNil) mydb := session.DB("mydb") err = mydb.AddUser("myruser", "mypass", true) c.Assert(err, IsNil) err = mydb.AddUser("mywuser", "mypass", false) c.Assert(err, IsNil) err = mydb.Login("myruser", "mypass") c.Assert(err, IsNil) admindb.Logout() coll := session.DB("mydb").C("mycoll") err = coll.Insert(M{"n": 1}) c.Assert(err, ErrorMatches, "unauthorized|not authorized .*") err = mydb.Login("mywuser", "mypass") c.Assert(err, IsNil) err = coll.Insert(M{"n": 1}) c.Assert(err, IsNil) } func (s *S) TestAuthAddUserReplaces(c *C) { session, err := mgo.Dial("localhost:40002") c.Assert(err, IsNil) defer session.Close() admindb := session.DB("admin") err = admindb.Login("root", "rapadura") c.Assert(err, IsNil) mydb := session.DB("mydb") err = mydb.AddUser("myuser", "myoldpass", false) c.Assert(err, IsNil) err = mydb.AddUser("myuser", "mynewpass", true) c.Assert(err, IsNil) admindb.Logout() err = mydb.Login("myuser", "myoldpass") c.Assert(err, ErrorMatches, "auth fail(s|ed)|.*Authentication failed.") err = mydb.Login("myuser", "mynewpass") c.Assert(err, IsNil) // ReadOnly flag was changed too. err = mydb.C("mycoll").Insert(M{"n": 1}) c.Assert(err, ErrorMatches, "unauthorized|not authorized .*") } func (s *S) TestAuthRemoveUser(c *C) { session, err := mgo.Dial("localhost:40002") c.Assert(err, IsNil) defer session.Close() admindb := session.DB("admin") err = admindb.Login("root", "rapadura") c.Assert(err, IsNil) mydb := session.DB("mydb") err = mydb.AddUser("myuser", "mypass", true) c.Assert(err, IsNil) err = mydb.RemoveUser("myuser") c.Assert(err, IsNil) err = mydb.RemoveUser("myuser") c.Assert(err, Equals, mgo.ErrNotFound) err = mydb.Login("myuser", "mypass") c.Assert(err, ErrorMatches, "auth fail(s|ed)|.*Authentication failed.") } func (s *S) TestAuthLoginTwiceDoesNothing(c *C) { session, err := mgo.Dial("localhost:40002") c.Assert(err, IsNil) defer session.Close() admindb := session.DB("admin") err = admindb.Login("root", "rapadura") c.Assert(err, IsNil) oldStats := mgo.GetStats() err = admindb.Login("root", "rapadura") c.Assert(err, IsNil) newStats := mgo.GetStats() c.Assert(newStats.SentOps, Equals, oldStats.SentOps) } func (s *S) TestAuthLoginLogoutLoginDoesNothing(c *C) { session, err := mgo.Dial("localhost:40002") c.Assert(err, IsNil) defer session.Close() admindb := session.DB("admin") err = admindb.Login("root", "rapadura") c.Assert(err, IsNil) oldStats := mgo.GetStats() admindb.Logout() err = admindb.Login("root", "rapadura") c.Assert(err, IsNil) newStats := mgo.GetStats() c.Assert(newStats.SentOps, Equals, oldStats.SentOps) } func (s *S) TestAuthLoginSwitchUser(c *C) { session, err := mgo.Dial("localhost:40002") c.Assert(err, IsNil) defer session.Close() admindb := session.DB("admin") err = admindb.Login("root", "rapadura") c.Assert(err, IsNil) coll := session.DB("mydb").C("mycoll") err = coll.Insert(M{"n": 1}) c.Assert(err, IsNil) err = admindb.Login("reader", "rapadura") c.Assert(err, IsNil) // Can't write. err = coll.Insert(M{"n": 1}) c.Assert(err, ErrorMatches, "unauthorized|not authorized .*") // But can read. result := struct{ N int }{} err = coll.Find(nil).One(&result) c.Assert(err, IsNil) c.Assert(result.N, Equals, 1) } func (s *S) TestAuthLoginChangePassword(c *C) { session, err := mgo.Dial("localhost:40002") c.Assert(err, IsNil) defer session.Close() admindb := session.DB("admin") err = admindb.Login("root", "rapadura") c.Assert(err, IsNil) mydb := session.DB("mydb") err = mydb.AddUser("myuser", "myoldpass", false) c.Assert(err, IsNil) err = mydb.Login("myuser", "myoldpass") c.Assert(err, IsNil) err = mydb.AddUser("myuser", "mynewpass", true) c.Assert(err, IsNil) err = mydb.Login("myuser", "mynewpass") c.Assert(err, IsNil) admindb.Logout() // The second login must be in effect, which means read-only. err = mydb.C("mycoll").Insert(M{"n": 1}) c.Assert(err, ErrorMatches, "unauthorized|not authorized .*") } func (s *S) TestAuthLoginCachingWithSessionRefresh(c *C) { session, err := mgo.Dial("localhost:40002") c.Assert(err, IsNil) defer session.Close() admindb := session.DB("admin") err = admindb.Login("root", "rapadura") c.Assert(err, IsNil) session.Refresh() coll := session.DB("mydb").C("mycoll") err = coll.Insert(M{"n": 1}) c.Assert(err, IsNil) } func (s *S) TestAuthLoginCachingWithSessionCopy(c *C) { session, err := mgo.Dial("localhost:40002") c.Assert(err, IsNil) defer session.Close() admindb := session.DB("admin") err = admindb.Login("root", "rapadura") c.Assert(err, IsNil) session = session.Copy() defer session.Close() coll := session.DB("mydb").C("mycoll") err = coll.Insert(M{"n": 1}) c.Assert(err, IsNil) } func (s *S) TestAuthLoginCachingWithSessionClone(c *C) { session, err := mgo.Dial("localhost:40002") c.Assert(err, IsNil) defer session.Close() admindb := session.DB("admin") err = admindb.Login("root", "rapadura") c.Assert(err, IsNil) session = session.Clone() defer session.Close() coll := session.DB("mydb").C("mycoll") err = coll.Insert(M{"n": 1}) c.Assert(err, IsNil) } func (s *S) TestAuthLoginCachingWithNewSession(c *C) { session, err := mgo.Dial("localhost:40002") c.Assert(err, IsNil) defer session.Close() admindb := session.DB("admin") err = admindb.Login("root", "rapadura") c.Assert(err, IsNil) session = session.New() defer session.Close() coll := session.DB("mydb").C("mycoll") err = coll.Insert(M{"n": 1}) c.Assert(err, ErrorMatches, "unauthorized|need to login|not authorized .*") } func (s *S) TestAuthLoginCachingAcrossPool(c *C) { // Logins are cached even when the conenction goes back // into the pool. session, err := mgo.Dial("localhost:40002") c.Assert(err, IsNil) defer session.Close() admindb := session.DB("admin") err = admindb.Login("root", "rapadura") c.Assert(err, IsNil) // Add another user to test the logout case at the same time. mydb := session.DB("mydb") err = mydb.AddUser("myuser", "mypass", false) c.Assert(err, IsNil) err = mydb.Login("myuser", "mypass") c.Assert(err, IsNil) // Logout root explicitly, to test both cases. admindb.Logout() // Give socket back to pool. session.Refresh() // Brand new session, should use socket from the pool. other := session.New() defer other.Close() oldStats := mgo.GetStats() err = other.DB("admin").Login("root", "rapadura") c.Assert(err, IsNil) err = other.DB("mydb").Login("myuser", "mypass") c.Assert(err, IsNil) // Both logins were cached, so no ops. newStats := mgo.GetStats() c.Assert(newStats.SentOps, Equals, oldStats.SentOps) // And they actually worked. err = other.DB("mydb").C("mycoll").Insert(M{"n": 1}) c.Assert(err, IsNil) other.DB("admin").Logout() err = other.DB("mydb").C("mycoll").Insert(M{"n": 1}) c.Assert(err, IsNil) } func (s *S) TestAuthLoginCachingAcrossPoolWithLogout(c *C) { // Now verify that logouts are properly flushed if they // are not revalidated after leaving the pool. session, err := mgo.Dial("localhost:40002") c.Assert(err, IsNil) defer session.Close() admindb := session.DB("admin") err = admindb.Login("root", "rapadura") c.Assert(err, IsNil) // Add another user to test the logout case at the same time. mydb := session.DB("mydb") err = mydb.AddUser("myuser", "mypass", true) c.Assert(err, IsNil) err = mydb.Login("myuser", "mypass") c.Assert(err, IsNil) // Just some data to query later. err = session.DB("mydb").C("mycoll").Insert(M{"n": 1}) c.Assert(err, IsNil) // Give socket back to pool. session.Refresh() // Brand new session, should use socket from the pool. other := session.New() defer other.Close() oldStats := mgo.GetStats() err = other.DB("mydb").Login("myuser", "mypass") c.Assert(err, IsNil) // Login was cached, so no ops. newStats := mgo.GetStats() c.Assert(newStats.SentOps, Equals, oldStats.SentOps) // Can't write, since root has been implicitly logged out // when the collection went into the pool, and not revalidated. err = other.DB("mydb").C("mycoll").Insert(M{"n": 1}) c.Assert(err, ErrorMatches, "unauthorized|not authorized .*") // But can read due to the revalidated myuser login. result := struct{ N int }{} err = other.DB("mydb").C("mycoll").Find(nil).One(&result) c.Assert(err, IsNil) c.Assert(result.N, Equals, 1) } func (s *S) TestAuthEventual(c *C) { // Eventual sessions don't keep sockets around, so they are // an interesting test case. session, err := mgo.Dial("localhost:40002") c.Assert(err, IsNil) defer session.Close() admindb := session.DB("admin") err = admindb.Login("root", "rapadura") c.Assert(err, IsNil) err = session.DB("mydb").C("mycoll").Insert(M{"n": 1}) c.Assert(err, IsNil) var wg sync.WaitGroup wg.Add(20) for i := 0; i != 10; i++ { go func() { defer wg.Done() var result struct{ N int } err := session.DB("mydb").C("mycoll").Find(nil).One(&result) c.Assert(err, IsNil) c.Assert(result.N, Equals, 1) }() } for i := 0; i != 10; i++ { go func() { defer wg.Done() err := session.DB("mydb").C("mycoll").Insert(M{"n": 1}) c.Assert(err, IsNil) }() } wg.Wait() } func (s *S) TestAuthURL(c *C) { session, err := mgo.Dial("mongodb://root:rapadura@localhost:40002/") c.Assert(err, IsNil) defer session.Close() err = session.DB("mydb").C("mycoll").Insert(M{"n": 1}) c.Assert(err, IsNil) } func (s *S) TestAuthURLWrongCredentials(c *C) { session, err := mgo.Dial("mongodb://root:wrong@localhost:40002/") if session != nil { session.Close() } c.Assert(err, ErrorMatches, "auth fail(s|ed)|.*Authentication failed.") c.Assert(session, IsNil) } func (s *S) TestAuthURLWithNewSession(c *C) { // When authentication is in the URL, the new session will // actually carry it on as well, even if logged out explicitly. session, err := mgo.Dial("mongodb://root:rapadura@localhost:40002/") c.Assert(err, IsNil) defer session.Close() session.DB("admin").Logout() // Do it twice to ensure it passes the needed data on. session = session.New() defer session.Close() session = session.New() defer session.Close() err = session.DB("mydb").C("mycoll").Insert(M{"n": 1}) c.Assert(err, IsNil) } func (s *S) TestAuthURLWithDatabase(c *C) { session, err := mgo.Dial("mongodb://root:rapadura@localhost:40002") c.Assert(err, IsNil) defer session.Close() mydb := session.DB("mydb") err = mydb.AddUser("myruser", "mypass", true) c.Assert(err, IsNil) // Test once with database, and once with source. for i := 0; i < 2; i++ { var url string if i == 0 { url = "mongodb://myruser:mypass@localhost:40002/mydb" } else { url = "mongodb://myruser:mypass@localhost:40002/admin?authSource=mydb" } usession, err := mgo.Dial(url) c.Assert(err, IsNil) defer usession.Close() ucoll := usession.DB("mydb").C("mycoll") err = ucoll.FindId(0).One(nil) c.Assert(err, Equals, mgo.ErrNotFound) err = ucoll.Insert(M{"n": 1}) c.Assert(err, ErrorMatches, "unauthorized|not authorized .*") } } func (s *S) TestDefaultDatabase(c *C) { tests := []struct{ url, db string }{ {"mongodb://root:rapadura@localhost:40002", "test"}, {"mongodb://root:rapadura@localhost:40002/admin", "admin"}, {"mongodb://localhost:40001", "test"}, {"mongodb://localhost:40001/", "test"}, {"mongodb://localhost:40001/mydb", "mydb"}, } for _, test := range tests { session, err := mgo.Dial(test.url) c.Assert(err, IsNil) defer session.Close() c.Logf("test: %#v", test) c.Assert(session.DB("").Name, Equals, test.db) scopy := session.Copy() c.Check(scopy.DB("").Name, Equals, test.db) scopy.Close() } } func (s *S) TestAuthDirect(c *C) { // Direct connections must work to the master and slaves. for _, port := range []string{"40031", "40032", "40033"} { url := fmt.Sprintf("mongodb://root:rapadura@localhost:%s/?connect=direct", port) session, err := mgo.Dial(url) c.Assert(err, IsNil) defer session.Close() session.SetMode(mgo.Monotonic, true) var result struct{} err = session.DB("mydb").C("mycoll").Find(nil).One(&result) c.Assert(err, Equals, mgo.ErrNotFound) } } func (s *S) TestAuthDirectWithLogin(c *C) { // Direct connections must work to the master and slaves. for _, port := range []string{"40031", "40032", "40033"} { url := fmt.Sprintf("mongodb://localhost:%s/?connect=direct", port) session, err := mgo.Dial(url) c.Assert(err, IsNil) defer session.Close() session.SetMode(mgo.Monotonic, true) session.SetSyncTimeout(3 * time.Second) err = session.DB("admin").Login("root", "rapadura") c.Assert(err, IsNil) var result struct{} err = session.DB("mydb").C("mycoll").Find(nil).One(&result) c.Assert(err, Equals, mgo.ErrNotFound) } } func (s *S) TestAuthScramSha1Cred(c *C) { if !s.versionAtLeast(2, 7, 7) { c.Skip("SCRAM-SHA-1 tests depend on 2.7.7") } cred := &mgo.Credential{ Username: "root", Password: "rapadura", Mechanism: "SCRAM-SHA-1", Source: "admin", } host := "localhost:40002" c.Logf("Connecting to %s...", host) session, err := mgo.Dial(host) c.Assert(err, IsNil) defer session.Close() mycoll := session.DB("admin").C("mycoll") c.Logf("Connected! Testing the need for authentication...") err = mycoll.Find(nil).One(nil) c.Assert(err, ErrorMatches, "unauthorized|not authorized .*") c.Logf("Authenticating...") err = session.Login(cred) c.Assert(err, IsNil) c.Logf("Authenticated!") c.Logf("Connected! Testing the need for authentication...") err = mycoll.Find(nil).One(nil) c.Assert(err, Equals, mgo.ErrNotFound) } func (s *S) TestAuthScramSha1URL(c *C) { if !s.versionAtLeast(2, 7, 7) { c.Skip("SCRAM-SHA-1 tests depend on 2.7.7") } host := "localhost:40002" c.Logf("Connecting to %s...", host) session, err := mgo.Dial(fmt.Sprintf("root:rapadura@%s?authMechanism=SCRAM-SHA-1", host)) c.Assert(err, IsNil) defer session.Close() mycoll := session.DB("admin").C("mycoll") c.Logf("Connected! Testing the need for authentication...") err = mycoll.Find(nil).One(nil) c.Assert(err, Equals, mgo.ErrNotFound) } func (s *S) TestAuthX509Cred(c *C) { session, err := mgo.Dial("localhost:40001") c.Assert(err, IsNil) defer session.Close() binfo, err := session.BuildInfo() c.Assert(err, IsNil) if binfo.OpenSSLVersion == "" { c.Skip("server does not support SSL") } clientCertPEM, err := ioutil.ReadFile("harness/certs/client.pem") c.Assert(err, IsNil) clientCert, err := tls.X509KeyPair(clientCertPEM, clientCertPEM) c.Assert(err, IsNil) tlsConfig := &tls.Config{ // Isolating tests to client certs, don't care about server validation. InsecureSkipVerify: true, Certificates: []tls.Certificate{clientCert}, } var host = "localhost:40003" c.Logf("Connecting to %s...", host) session, err = mgo.DialWithInfo(&mgo.DialInfo{ Addrs: []string{host}, DialServer: func(addr *mgo.ServerAddr) (net.Conn, error) { return tls.Dial("tcp", addr.String(), tlsConfig) }, }) c.Assert(err, IsNil) defer session.Close() err = session.Login(&mgo.Credential{Username: "root", Password: "rapadura"}) c.Assert(err, IsNil) // This needs to be kept in sync with client.pem x509Subject := "CN=localhost,OU=Client,O=MGO,L=MGO,ST=MGO,C=GO" externalDB := session.DB("$external") var x509User mgo.User = mgo.User{ Username: x509Subject, OtherDBRoles: map[string][]mgo.Role{"admin": []mgo.Role{mgo.RoleRoot}}, } err = externalDB.UpsertUser(&x509User) c.Assert(err, IsNil) session.LogoutAll() c.Logf("Connected! Ensuring authentication is required...") names, err := session.DatabaseNames() c.Assert(err, ErrorMatches, "not authorized .*") cred := &mgo.Credential{ Username: x509Subject, Mechanism: "MONGODB-X509", Source: "$external", } c.Logf("Authenticating...") err = session.Login(cred) c.Assert(err, IsNil) c.Logf("Authenticated!") names, err = session.DatabaseNames() c.Assert(err, IsNil) c.Assert(len(names) > 0, Equals, true) } var ( plainFlag = flag.String("plain", "", "Host to test PLAIN authentication against (depends on custom environment)") plainUser = "einstein" plainPass = "password" ) func (s *S) TestAuthPlainCred(c *C) { if *plainFlag == "" { c.Skip("no -plain") } cred := &mgo.Credential{ Username: plainUser, Password: plainPass, Source: "$external", Mechanism: "PLAIN", } c.Logf("Connecting to %s...", *plainFlag) session, err := mgo.Dial(*plainFlag) c.Assert(err, IsNil) defer session.Close() records := session.DB("records").C("records") c.Logf("Connected! Testing the need for authentication...") err = records.Find(nil).One(nil) c.Assert(err, ErrorMatches, "unauthorized|not authorized .*") c.Logf("Authenticating...") err = session.Login(cred) c.Assert(err, IsNil) c.Logf("Authenticated!") c.Logf("Connected! Testing the need for authentication...") err = records.Find(nil).One(nil) c.Assert(err, Equals, mgo.ErrNotFound) } func (s *S) TestAuthPlainURL(c *C) { if *plainFlag == "" { c.Skip("no -plain") } c.Logf("Connecting to %s...", *plainFlag) session, err := mgo.Dial(fmt.Sprintf("%s:%s@%s?authMechanism=PLAIN", url.QueryEscape(plainUser), url.QueryEscape(plainPass), *plainFlag)) c.Assert(err, IsNil) defer session.Close() c.Logf("Connected! Testing the need for authentication...") err = session.DB("records").C("records").Find(nil).One(nil) c.Assert(err, Equals, mgo.ErrNotFound) } var ( kerberosFlag = flag.Bool("kerberos", false, "Test Kerberos authentication (depends on custom environment)") kerberosHost = "ldaptest.10gen.cc" kerberosUser = "drivers@LDAPTEST.10GEN.CC" winKerberosPasswordEnv = "MGO_KERBEROS_PASSWORD" ) // Kerberos has its own suite because it talks to a remote server // that is prepared to authenticate against a kerberos deployment. type KerberosSuite struct{} var _ = Suite(&KerberosSuite{}) func (kerberosSuite *KerberosSuite) SetUpSuite(c *C) { mgo.SetDebug(true) mgo.SetStats(true) } func (kerberosSuite *KerberosSuite) TearDownSuite(c *C) { mgo.SetDebug(false) mgo.SetStats(false) } func (kerberosSuite *KerberosSuite) SetUpTest(c *C) { mgo.SetLogger((*cLogger)(c)) mgo.ResetStats() } func (kerberosSuite *KerberosSuite) TearDownTest(c *C) { mgo.SetLogger(nil) } func (kerberosSuite *KerberosSuite) TestAuthKerberosCred(c *C) { if !*kerberosFlag { c.Skip("no -kerberos") } cred := &mgo.Credential{ Username: kerberosUser, Mechanism: "GSSAPI", } windowsAppendPasswordToCredential(cred) c.Logf("Connecting to %s...", kerberosHost) session, err := mgo.Dial(kerberosHost) c.Assert(err, IsNil) defer session.Close() c.Logf("Connected! Testing the need for authentication...") n, err := session.DB("kerberos").C("test").Find(M{}).Count() c.Assert(err, ErrorMatches, ".*authorized.*") c.Logf("Authenticating...") err = session.Login(cred) c.Assert(err, IsNil) c.Logf("Authenticated!") n, err = session.DB("kerberos").C("test").Find(M{}).Count() c.Assert(err, IsNil) c.Assert(n, Equals, 1) } func (kerberosSuite *KerberosSuite) TestAuthKerberosURL(c *C) { if !*kerberosFlag { c.Skip("no -kerberos") } c.Logf("Connecting to %s...", kerberosHost) connectUri := url.QueryEscape(kerberosUser) + "@" + kerberosHost + "?authMechanism=GSSAPI" if runtime.GOOS == "windows" { connectUri = url.QueryEscape(kerberosUser) + ":" + url.QueryEscape(getWindowsKerberosPassword()) + "@" + kerberosHost + "?authMechanism=GSSAPI" } session, err := mgo.Dial(connectUri) c.Assert(err, IsNil) defer session.Close() n, err := session.DB("kerberos").C("test").Find(M{}).Count() c.Assert(err, IsNil) c.Assert(n, Equals, 1) } func (kerberosSuite *KerberosSuite) TestAuthKerberosServiceName(c *C) { if !*kerberosFlag { c.Skip("no -kerberos") } wrongServiceName := "wrong" rightServiceName := "mongodb" cred := &mgo.Credential{ Username: kerberosUser, Mechanism: "GSSAPI", Service: wrongServiceName, } windowsAppendPasswordToCredential(cred) c.Logf("Connecting to %s...", kerberosHost) session, err := mgo.Dial(kerberosHost) c.Assert(err, IsNil) defer session.Close() c.Logf("Authenticating with incorrect service name...") err = session.Login(cred) c.Assert(err, ErrorMatches, ".*@LDAPTEST.10GEN.CC not found.*") cred.Service = rightServiceName c.Logf("Authenticating with correct service name...") err = session.Login(cred) c.Assert(err, IsNil) c.Logf("Authenticated!") n, err := session.DB("kerberos").C("test").Find(M{}).Count() c.Assert(err, IsNil) c.Assert(n, Equals, 1) } func (kerberosSuite *KerberosSuite) TestAuthKerberosServiceHost(c *C) { if !*kerberosFlag { c.Skip("no -kerberos") } wrongServiceHost := "eggs.bacon.tk" rightServiceHost := kerberosHost cred := &mgo.Credential{ Username: kerberosUser, Mechanism: "GSSAPI", ServiceHost: wrongServiceHost, } windowsAppendPasswordToCredential(cred) c.Logf("Connecting to %s...", kerberosHost) session, err := mgo.Dial(kerberosHost) c.Assert(err, IsNil) defer session.Close() c.Logf("Authenticating with incorrect service host...") err = session.Login(cred) c.Assert(err, ErrorMatches, ".*@LDAPTEST.10GEN.CC not found.*") cred.ServiceHost = rightServiceHost c.Logf("Authenticating with correct service host...") err = session.Login(cred) c.Assert(err, IsNil) c.Logf("Authenticated!") n, err := session.DB("kerberos").C("test").Find(M{}).Count() c.Assert(err, IsNil) c.Assert(n, Equals, 1) } // No kinit on SSPI-style Kerberos, so we need to provide a password. In order // to avoid inlining password, require it to be set as an environment variable, // for instance: `SET MGO_KERBEROS_PASSWORD=this_isnt_the_password` func getWindowsKerberosPassword() string { pw := os.Getenv(winKerberosPasswordEnv) if pw == "" { panic(fmt.Sprintf("Need to set %v environment variable to run Kerberos tests on Windows", winKerberosPasswordEnv)) } return pw } func windowsAppendPasswordToCredential(cred *mgo.Credential) { if runtime.GOOS == "windows" { cred.Password = getWindowsKerberosPassword() } } mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/bson/000077500000000000000000000000001276133741700221215ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/bson/LICENSE000066400000000000000000000025141276133741700231300ustar00rootroot00000000000000BSON library for Go Copyright (c) 2010-2012 - Gustavo Niemeyer All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/bson/bson.go000066400000000000000000000551571276133741700234260ustar00rootroot00000000000000// BSON library for Go // // Copyright (c) 2010-2012 - Gustavo Niemeyer // // All rights reserved. // // Redistribution and use in source and binary forms, with or without // modification, are permitted provided that the following conditions are met: // // 1. Redistributions of source code must retain the above copyright notice, this // list of conditions and the following disclaimer. // 2. Redistributions in binary form must reproduce the above copyright notice, // this list of conditions and the following disclaimer in the documentation // and/or other materials provided with the distribution. // // THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND // ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED // WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE // DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR // ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES // (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; // LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND // ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT // (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS // SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. // Package bson is an implementation of the BSON specification for Go: // // http://bsonspec.org // // It was created as part of the mgo MongoDB driver for Go, but is standalone // and may be used on its own without the driver. package bson import ( "bytes" "crypto/md5" "crypto/rand" "encoding/binary" "encoding/hex" "encoding/json" "errors" "fmt" "io" "os" "reflect" "runtime" "strings" "sync" "sync/atomic" "time" ) // -------------------------------------------------------------------------- // The public API. // A value implementing the bson.Getter interface will have its GetBSON // method called when the given value has to be marshalled, and the result // of this method will be marshaled in place of the actual object. // // If GetBSON returns return a non-nil error, the marshalling procedure // will stop and error out with the provided value. type Getter interface { GetBSON() (interface{}, error) } // A value implementing the bson.Setter interface will receive the BSON // value via the SetBSON method during unmarshaling, and the object // itself will not be changed as usual. // // If setting the value works, the method should return nil or alternatively // bson.SetZero to set the respective field to its zero value (nil for // pointer types). If SetBSON returns a value of type bson.TypeError, the // BSON value will be omitted from a map or slice being decoded and the // unmarshalling will continue. If it returns any other non-nil error, the // unmarshalling procedure will stop and error out with the provided value. // // This interface is generally useful in pointer receivers, since the method // will want to change the receiver. A type field that implements the Setter // interface doesn't have to be a pointer, though. // // Unlike the usual behavior, unmarshalling onto a value that implements a // Setter interface will NOT reset the value to its zero state. This allows // the value to decide by itself how to be unmarshalled. // // For example: // // type MyString string // // func (s *MyString) SetBSON(raw bson.Raw) error { // return raw.Unmarshal(s) // } // type Setter interface { SetBSON(raw Raw) error } // SetZero may be returned from a SetBSON method to have the value set to // its respective zero value. When used in pointer values, this will set the // field to nil rather than to the pre-allocated value. var SetZero = errors.New("set to zero") // M is a convenient alias for a map[string]interface{} map, useful for // dealing with BSON in a native way. For instance: // // bson.M{"a": 1, "b": true} // // There's no special handling for this type in addition to what's done anyway // for an equivalent map type. Elements in the map will be dumped in an // undefined ordered. See also the bson.D type for an ordered alternative. type M map[string]interface{} // D represents a BSON document containing ordered elements. For example: // // bson.D{{"a", 1}, {"b", true}} // // In some situations, such as when creating indexes for MongoDB, the order in // which the elements are defined is important. If the order is not important, // using a map is generally more comfortable. See bson.M and bson.RawD. type D []DocElem // DocElem is an element of the bson.D document representation. type DocElem struct { Name string Value interface{} } // Map returns a map out of the ordered element name/value pairs in d. func (d D) Map() (m M) { m = make(M, len(d)) for _, item := range d { m[item.Name] = item.Value } return m } // The Raw type represents raw unprocessed BSON documents and elements. // Kind is the kind of element as defined per the BSON specification, and // Data is the raw unprocessed data for the respective element. // Using this type it is possible to unmarshal or marshal values partially. // // Relevant documentation: // // http://bsonspec.org/#/specification // type Raw struct { Kind byte Data []byte } // RawD represents a BSON document containing raw unprocessed elements. // This low-level representation may be useful when lazily processing // documents of uncertain content, or when manipulating the raw content // documents in general. type RawD []RawDocElem // See the RawD type. type RawDocElem struct { Name string Value Raw } // ObjectId is a unique ID identifying a BSON value. It must be exactly 12 bytes // long. MongoDB objects by default have such a property set in their "_id" // property. // // http://www.mongodb.org/display/DOCS/Object+IDs type ObjectId string // ObjectIdHex returns an ObjectId from the provided hex representation. // Calling this function with an invalid hex representation will // cause a runtime panic. See the IsObjectIdHex function. func ObjectIdHex(s string) ObjectId { d, err := hex.DecodeString(s) if err != nil || len(d) != 12 { panic(fmt.Sprintf("invalid input to ObjectIdHex: %q", s)) } return ObjectId(d) } // IsObjectIdHex returns whether s is a valid hex representation of // an ObjectId. See the ObjectIdHex function. func IsObjectIdHex(s string) bool { if len(s) != 24 { return false } _, err := hex.DecodeString(s) return err == nil } // objectIdCounter is atomically incremented when generating a new ObjectId // using NewObjectId() function. It's used as a counter part of an id. var objectIdCounter uint32 = readRandomUint32() // readRandomUint32 returns a random objectIdCounter. func readRandomUint32() uint32 { var b [4]byte _, err := io.ReadFull(rand.Reader, b[:]) if err != nil { panic(fmt.Errorf("cannot read random object id: %v", err)) } return uint32((uint32(b[0]) << 0) | (uint32(b[1]) << 8) | (uint32(b[2]) << 16) | (uint32(b[3]) << 24)) } // machineId stores machine id generated once and used in subsequent calls // to NewObjectId function. var machineId = readMachineId() var processId = os.Getpid() // readMachineId generates and returns a machine id. // If this function fails to get the hostname it will cause a runtime error. func readMachineId() []byte { var sum [3]byte id := sum[:] hostname, err1 := os.Hostname() if err1 != nil { _, err2 := io.ReadFull(rand.Reader, id) if err2 != nil { panic(fmt.Errorf("cannot get hostname: %v; %v", err1, err2)) } return id } hw := md5.New() hw.Write([]byte(hostname)) copy(id, hw.Sum(nil)) return id } // NewObjectId returns a new unique ObjectId. func NewObjectId() ObjectId { var b [12]byte // Timestamp, 4 bytes, big endian binary.BigEndian.PutUint32(b[:], uint32(time.Now().Unix())) // Machine, first 3 bytes of md5(hostname) b[4] = machineId[0] b[5] = machineId[1] b[6] = machineId[2] // Pid, 2 bytes, specs don't specify endianness, but we use big endian. b[7] = byte(processId >> 8) b[8] = byte(processId) // Increment, 3 bytes, big endian i := atomic.AddUint32(&objectIdCounter, 1) b[9] = byte(i >> 16) b[10] = byte(i >> 8) b[11] = byte(i) return ObjectId(b[:]) } // NewObjectIdWithTime returns a dummy ObjectId with the timestamp part filled // with the provided number of seconds from epoch UTC, and all other parts // filled with zeroes. It's not safe to insert a document with an id generated // by this method, it is useful only for queries to find documents with ids // generated before or after the specified timestamp. func NewObjectIdWithTime(t time.Time) ObjectId { var b [12]byte binary.BigEndian.PutUint32(b[:4], uint32(t.Unix())) return ObjectId(string(b[:])) } // String returns a hex string representation of the id. // Example: ObjectIdHex("4d88e15b60f486e428412dc9"). func (id ObjectId) String() string { return fmt.Sprintf(`ObjectIdHex("%x")`, string(id)) } // Hex returns a hex representation of the ObjectId. func (id ObjectId) Hex() string { return hex.EncodeToString([]byte(id)) } // MarshalJSON turns a bson.ObjectId into a json.Marshaller. func (id ObjectId) MarshalJSON() ([]byte, error) { return []byte(fmt.Sprintf(`"%x"`, string(id))), nil } var nullBytes = []byte("null") // UnmarshalJSON turns *bson.ObjectId into a json.Unmarshaller. func (id *ObjectId) UnmarshalJSON(data []byte) error { if len(data) > 0 && (data[0] == '{' || data[0] == 'O') { var v struct { Id json.RawMessage `json:"$oid"` Func struct { Id json.RawMessage } `json:"$oidFunc"` } err := jdec(data, &v) if err == nil { if len(v.Id) > 0 { data = []byte(v.Id) } else { data = []byte(v.Func.Id) } } } if len(data) == 2 && data[0] == '"' && data[1] == '"' || bytes.Equal(data, nullBytes) { *id = "" return nil } if len(data) != 26 || data[0] != '"' || data[25] != '"' { return errors.New(fmt.Sprintf("invalid ObjectId in JSON: %s", string(data))) } var buf [12]byte _, err := hex.Decode(buf[:], data[1:25]) if err != nil { return errors.New(fmt.Sprintf("invalid ObjectId in JSON: %s (%s)", string(data), err)) } *id = ObjectId(string(buf[:])) return nil } // MarshalText turns bson.ObjectId into an encoding.TextMarshaler. func (id ObjectId) MarshalText() ([]byte, error) { return []byte(fmt.Sprintf("%x", string(id))), nil } // UnmarshalText turns *bson.ObjectId into an encoding.TextUnmarshaler. func (id *ObjectId) UnmarshalText(data []byte) error { if len(data) == 1 && data[0] == ' ' || len(data) == 0 { *id = "" return nil } if len(data) != 24 { return fmt.Errorf("invalid ObjectId: %s", data) } var buf [12]byte _, err := hex.Decode(buf[:], data[:]) if err != nil { return fmt.Errorf("invalid ObjectId: %s (%s)", data, err) } *id = ObjectId(string(buf[:])) return nil } // Valid returns true if id is valid. A valid id must contain exactly 12 bytes. func (id ObjectId) Valid() bool { return len(id) == 12 } // byteSlice returns byte slice of id from start to end. // Calling this function with an invalid id will cause a runtime panic. func (id ObjectId) byteSlice(start, end int) []byte { if len(id) != 12 { panic(fmt.Sprintf("invalid ObjectId: %q", string(id))) } return []byte(string(id)[start:end]) } // Time returns the timestamp part of the id. // It's a runtime error to call this method with an invalid id. func (id ObjectId) Time() time.Time { // First 4 bytes of ObjectId is 32-bit big-endian seconds from epoch. secs := int64(binary.BigEndian.Uint32(id.byteSlice(0, 4))) return time.Unix(secs, 0) } // Machine returns the 3-byte machine id part of the id. // It's a runtime error to call this method with an invalid id. func (id ObjectId) Machine() []byte { return id.byteSlice(4, 7) } // Pid returns the process id part of the id. // It's a runtime error to call this method with an invalid id. func (id ObjectId) Pid() uint16 { return binary.BigEndian.Uint16(id.byteSlice(7, 9)) } // Counter returns the incrementing value part of the id. // It's a runtime error to call this method with an invalid id. func (id ObjectId) Counter() int32 { b := id.byteSlice(9, 12) // Counter is stored as big-endian 3-byte value return int32(uint32(b[0])<<16 | uint32(b[1])<<8 | uint32(b[2])) } // The Symbol type is similar to a string and is used in languages with a // distinct symbol type. type Symbol string // Now returns the current time with millisecond precision. MongoDB stores // timestamps with the same precision, so a Time returned from this method // will not change after a roundtrip to the database. That's the only reason // why this function exists. Using the time.Now function also works fine // otherwise. func Now() time.Time { return time.Unix(0, time.Now().UnixNano()/1e6*1e6) } // MongoTimestamp is a special internal type used by MongoDB that for some // strange reason has its own datatype defined in BSON. type MongoTimestamp int64 type orderKey int64 // MaxKey is a special value that compares higher than all other possible BSON // values in a MongoDB database. var MaxKey = orderKey(1<<63 - 1) // MinKey is a special value that compares lower than all other possible BSON // values in a MongoDB database. var MinKey = orderKey(-1 << 63) type undefined struct{} // Undefined represents the undefined BSON value. var Undefined undefined // Binary is a representation for non-standard binary values. Any kind should // work, but the following are known as of this writing: // // 0x00 - Generic. This is decoded as []byte(data), not Binary{0x00, data}. // 0x01 - Function (!?) // 0x02 - Obsolete generic. // 0x03 - UUID // 0x05 - MD5 // 0x80 - User defined. // type Binary struct { Kind byte Data []byte } // RegEx represents a regular expression. The Options field may contain // individual characters defining the way in which the pattern should be // applied, and must be sorted. Valid options as of this writing are 'i' for // case insensitive matching, 'm' for multi-line matching, 'x' for verbose // mode, 'l' to make \w, \W, and similar be locale-dependent, 's' for dot-all // mode (a '.' matches everything), and 'u' to make \w, \W, and similar match // unicode. The value of the Options parameter is not verified before being // marshaled into the BSON format. type RegEx struct { Pattern string Options string } // JavaScript is a type that holds JavaScript code. If Scope is non-nil, it // will be marshaled as a mapping from identifiers to values that may be // used when evaluating the provided Code. type JavaScript struct { Code string Scope interface{} } // DBPointer refers to a document id in a namespace. // // This type is deprecated in the BSON specification and should not be used // except for backwards compatibility with ancient applications. type DBPointer struct { Namespace string Id ObjectId } const initialBufferSize = 64 func handleErr(err *error) { if r := recover(); r != nil { if _, ok := r.(runtime.Error); ok { panic(r) } else if _, ok := r.(externalPanic); ok { panic(r) } else if s, ok := r.(string); ok { *err = errors.New(s) } else if e, ok := r.(error); ok { *err = e } else { panic(r) } } } // Marshal serializes the in value, which may be a map or a struct value. // In the case of struct values, only exported fields will be serialized, // and the order of serialized fields will match that of the struct itself. // The lowercased field name is used as the key for each exported field, // but this behavior may be changed using the respective field tag. // The tag may also contain flags to tweak the marshalling behavior for // the field. The tag formats accepted are: // // "[][,[,]]" // // `(...) bson:"[][,[,]]" (...)` // // The following flags are currently supported: // // omitempty Only include the field if it's not set to the zero // value for the type or to empty slices or maps. // // minsize Marshal an int64 value as an int32, if that's feasible // while preserving the numeric value. // // inline Inline the field, which must be a struct or a map, // causing all of its fields or keys to be processed as if // they were part of the outer struct. For maps, keys must // not conflict with the bson keys of other struct fields. // // Some examples: // // type T struct { // A bool // B int "myb" // C string "myc,omitempty" // D string `bson:",omitempty" json:"jsonkey"` // E int64 ",minsize" // F int64 "myf,omitempty,minsize" // } // func Marshal(in interface{}) (out []byte, err error) { defer handleErr(&err) e := &encoder{make([]byte, 0, initialBufferSize)} e.addDoc(reflect.ValueOf(in)) return e.out, nil } // Unmarshal deserializes data from in into the out value. The out value // must be a map, a pointer to a struct, or a pointer to a bson.D value. // In the case of struct values, only exported fields will be deserialized. // The lowercased field name is used as the key for each exported field, // but this behavior may be changed using the respective field tag. // The tag may also contain flags to tweak the marshalling behavior for // the field. The tag formats accepted are: // // "[][,[,]]" // // `(...) bson:"[][,[,]]" (...)` // // The following flags are currently supported during unmarshal (see the // Marshal method for other flags): // // inline Inline the field, which must be a struct or a map. // Inlined structs are handled as if its fields were part // of the outer struct. An inlined map causes keys that do // not match any other struct field to be inserted in the // map rather than being discarded as usual. // // The target field or element types of out may not necessarily match // the BSON values of the provided data. The following conversions are // made automatically: // // - Numeric types are converted if at least the integer part of the // value would be preserved correctly // - Bools are converted to numeric types as 1 or 0 // - Numeric types are converted to bools as true if not 0 or false otherwise // - Binary and string BSON data is converted to a string, array or byte slice // // If the value would not fit the type and cannot be converted, it's // silently skipped. // // Pointer values are initialized when necessary. func Unmarshal(in []byte, out interface{}) (err error) { if raw, ok := out.(*Raw); ok { raw.Kind = 3 raw.Data = in return nil } defer handleErr(&err) v := reflect.ValueOf(out) switch v.Kind() { case reflect.Ptr: fallthrough case reflect.Map: d := newDecoder(in) d.readDocTo(v) case reflect.Struct: return errors.New("Unmarshal can't deal with struct values. Use a pointer.") default: return errors.New("Unmarshal needs a map or a pointer to a struct.") } return nil } // Unmarshal deserializes raw into the out value. If the out value type // is not compatible with raw, a *bson.TypeError is returned. // // See the Unmarshal function documentation for more details on the // unmarshalling process. func (raw Raw) Unmarshal(out interface{}) (err error) { defer handleErr(&err) v := reflect.ValueOf(out) switch v.Kind() { case reflect.Ptr: v = v.Elem() fallthrough case reflect.Map: d := newDecoder(raw.Data) good := d.readElemTo(v, raw.Kind) if !good { return &TypeError{v.Type(), raw.Kind} } case reflect.Struct: return errors.New("Raw Unmarshal can't deal with struct values. Use a pointer.") default: return errors.New("Raw Unmarshal needs a map or a valid pointer.") } return nil } type TypeError struct { Type reflect.Type Kind byte } func (e *TypeError) Error() string { return fmt.Sprintf("BSON kind 0x%02x isn't compatible with type %s", e.Kind, e.Type.String()) } // -------------------------------------------------------------------------- // Maintain a mapping of keys to structure field indexes type structInfo struct { FieldsMap map[string]fieldInfo FieldsList []fieldInfo InlineMap int Zero reflect.Value } type fieldInfo struct { Key string Num int OmitEmpty bool MinSize bool Inline []int } var structMap = make(map[reflect.Type]*structInfo) var structMapMutex sync.RWMutex type externalPanic string func (e externalPanic) String() string { return string(e) } func getStructInfo(st reflect.Type) (*structInfo, error) { structMapMutex.RLock() sinfo, found := structMap[st] structMapMutex.RUnlock() if found { return sinfo, nil } n := st.NumField() fieldsMap := make(map[string]fieldInfo) fieldsList := make([]fieldInfo, 0, n) inlineMap := -1 for i := 0; i != n; i++ { field := st.Field(i) if field.PkgPath != "" && !field.Anonymous { continue // Private field } info := fieldInfo{Num: i} tag := field.Tag.Get("bson") if tag == "" && strings.Index(string(field.Tag), ":") < 0 { tag = string(field.Tag) } if tag == "-" { continue } inline := false fields := strings.Split(tag, ",") if len(fields) > 1 { for _, flag := range fields[1:] { switch flag { case "omitempty": info.OmitEmpty = true case "minsize": info.MinSize = true case "inline": inline = true default: msg := fmt.Sprintf("Unsupported flag %q in tag %q of type %s", flag, tag, st) panic(externalPanic(msg)) } } tag = fields[0] } if inline { switch field.Type.Kind() { case reflect.Map: if inlineMap >= 0 { return nil, errors.New("Multiple ,inline maps in struct " + st.String()) } if field.Type.Key() != reflect.TypeOf("") { return nil, errors.New("Option ,inline needs a map with string keys in struct " + st.String()) } inlineMap = info.Num case reflect.Struct: sinfo, err := getStructInfo(field.Type) if err != nil { return nil, err } for _, finfo := range sinfo.FieldsList { if _, found := fieldsMap[finfo.Key]; found { msg := "Duplicated key '" + finfo.Key + "' in struct " + st.String() return nil, errors.New(msg) } if finfo.Inline == nil { finfo.Inline = []int{i, finfo.Num} } else { finfo.Inline = append([]int{i}, finfo.Inline...) } fieldsMap[finfo.Key] = finfo fieldsList = append(fieldsList, finfo) } default: panic("Option ,inline needs a struct value or map field") } continue } if tag != "" { info.Key = tag } else { info.Key = strings.ToLower(field.Name) } if _, found = fieldsMap[info.Key]; found { msg := "Duplicated key '" + info.Key + "' in struct " + st.String() return nil, errors.New(msg) } fieldsList = append(fieldsList, info) fieldsMap[info.Key] = info } sinfo = &structInfo{ fieldsMap, fieldsList, inlineMap, reflect.New(st).Elem(), } structMapMutex.Lock() structMap[st] = sinfo structMapMutex.Unlock() return sinfo, nil } mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/bson/bson_test.go000066400000000000000000001504261276133741700244600ustar00rootroot00000000000000// BSON library for Go // // Copyright (c) 2010-2012 - Gustavo Niemeyer // // All rights reserved. // // Redistribution and use in source and binary forms, with or without // modification, are permitted provided that the following conditions are met: // // 1. Redistributions of source code must retain the above copyright notice, this // list of conditions and the following disclaimer. // 2. Redistributions in binary form must reproduce the above copyright notice, // this list of conditions and the following disclaimer in the documentation // and/or other materials provided with the distribution. // // THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND // ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED // WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE // DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR // ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES // (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; // LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND // ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT // (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS // SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. // gobson - BSON library for Go. package bson_test import ( "encoding/binary" "encoding/hex" "encoding/json" "encoding/xml" "errors" "net/url" "reflect" "strings" "testing" "time" . "gopkg.in/check.v1" "gopkg.in/mgo.v2/bson" "gopkg.in/yaml.v2" ) func TestAll(t *testing.T) { TestingT(t) } type S struct{} var _ = Suite(&S{}) // Wrap up the document elements contained in data, prepending the int32 // length of the data, and appending the '\x00' value closing the document. func wrapInDoc(data string) string { result := make([]byte, len(data)+5) binary.LittleEndian.PutUint32(result, uint32(len(result))) copy(result[4:], []byte(data)) return string(result) } func makeZeroDoc(value interface{}) (zero interface{}) { v := reflect.ValueOf(value) t := v.Type() switch t.Kind() { case reflect.Map: mv := reflect.MakeMap(t) zero = mv.Interface() case reflect.Ptr: pv := reflect.New(v.Type().Elem()) zero = pv.Interface() case reflect.Slice, reflect.Int, reflect.Int64, reflect.Struct: zero = reflect.New(t).Interface() default: panic("unsupported doc type: " + t.Name()) } return zero } func testUnmarshal(c *C, data string, obj interface{}) { zero := makeZeroDoc(obj) err := bson.Unmarshal([]byte(data), zero) c.Assert(err, IsNil) c.Assert(zero, DeepEquals, obj) } type testItemType struct { obj interface{} data string } // -------------------------------------------------------------------------- // Samples from bsonspec.org: var sampleItems = []testItemType{ {bson.M{"hello": "world"}, "\x16\x00\x00\x00\x02hello\x00\x06\x00\x00\x00world\x00\x00"}, {bson.M{"BSON": []interface{}{"awesome", float64(5.05), 1986}}, "1\x00\x00\x00\x04BSON\x00&\x00\x00\x00\x020\x00\x08\x00\x00\x00" + "awesome\x00\x011\x00333333\x14@\x102\x00\xc2\x07\x00\x00\x00\x00"}, } func (s *S) TestMarshalSampleItems(c *C) { for i, item := range sampleItems { data, err := bson.Marshal(item.obj) c.Assert(err, IsNil) c.Assert(string(data), Equals, item.data, Commentf("Failed on item %d", i)) } } func (s *S) TestUnmarshalSampleItems(c *C) { for i, item := range sampleItems { value := bson.M{} err := bson.Unmarshal([]byte(item.data), value) c.Assert(err, IsNil) c.Assert(value, DeepEquals, item.obj, Commentf("Failed on item %d", i)) } } // -------------------------------------------------------------------------- // Every type, ordered by the type flag. These are not wrapped with the // length and last \x00 from the document. wrapInDoc() computes them. // Note that all of them should be supported as two-way conversions. var allItems = []testItemType{ {bson.M{}, ""}, {bson.M{"_": float64(5.05)}, "\x01_\x00333333\x14@"}, {bson.M{"_": "yo"}, "\x02_\x00\x03\x00\x00\x00yo\x00"}, {bson.M{"_": bson.M{"a": true}}, "\x03_\x00\x09\x00\x00\x00\x08a\x00\x01\x00"}, {bson.M{"_": []interface{}{true, false}}, "\x04_\x00\r\x00\x00\x00\x080\x00\x01\x081\x00\x00\x00"}, {bson.M{"_": []byte("yo")}, "\x05_\x00\x02\x00\x00\x00\x00yo"}, {bson.M{"_": bson.Binary{0x80, []byte("udef")}}, "\x05_\x00\x04\x00\x00\x00\x80udef"}, {bson.M{"_": bson.Undefined}, // Obsolete, but still seen in the wild. "\x06_\x00"}, {bson.M{"_": bson.ObjectId("0123456789ab")}, "\x07_\x000123456789ab"}, {bson.M{"_": bson.DBPointer{"testnamespace", bson.ObjectId("0123456789ab")}}, "\x0C_\x00\x0e\x00\x00\x00testnamespace\x000123456789ab"}, {bson.M{"_": false}, "\x08_\x00\x00"}, {bson.M{"_": true}, "\x08_\x00\x01"}, {bson.M{"_": time.Unix(0, 258e6)}, // Note the NS <=> MS conversion. "\x09_\x00\x02\x01\x00\x00\x00\x00\x00\x00"}, {bson.M{"_": nil}, "\x0A_\x00"}, {bson.M{"_": bson.RegEx{"ab", "cd"}}, "\x0B_\x00ab\x00cd\x00"}, {bson.M{"_": bson.JavaScript{"code", nil}}, "\x0D_\x00\x05\x00\x00\x00code\x00"}, {bson.M{"_": bson.Symbol("sym")}, "\x0E_\x00\x04\x00\x00\x00sym\x00"}, {bson.M{"_": bson.JavaScript{"code", bson.M{"": nil}}}, "\x0F_\x00\x14\x00\x00\x00\x05\x00\x00\x00code\x00" + "\x07\x00\x00\x00\x0A\x00\x00"}, {bson.M{"_": 258}, "\x10_\x00\x02\x01\x00\x00"}, {bson.M{"_": bson.MongoTimestamp(258)}, "\x11_\x00\x02\x01\x00\x00\x00\x00\x00\x00"}, {bson.M{"_": int64(258)}, "\x12_\x00\x02\x01\x00\x00\x00\x00\x00\x00"}, {bson.M{"_": int64(258 << 32)}, "\x12_\x00\x00\x00\x00\x00\x02\x01\x00\x00"}, {bson.M{"_": bson.MaxKey}, "\x7F_\x00"}, {bson.M{"_": bson.MinKey}, "\xFF_\x00"}, } func (s *S) TestMarshalAllItems(c *C) { for i, item := range allItems { data, err := bson.Marshal(item.obj) c.Assert(err, IsNil) c.Assert(string(data), Equals, wrapInDoc(item.data), Commentf("Failed on item %d: %#v", i, item)) } } func (s *S) TestUnmarshalAllItems(c *C) { for i, item := range allItems { value := bson.M{} err := bson.Unmarshal([]byte(wrapInDoc(item.data)), value) c.Assert(err, IsNil) c.Assert(value, DeepEquals, item.obj, Commentf("Failed on item %d: %#v", i, item)) } } func (s *S) TestUnmarshalRawAllItems(c *C) { for i, item := range allItems { if len(item.data) == 0 { continue } value := item.obj.(bson.M)["_"] if value == nil { continue } pv := reflect.New(reflect.ValueOf(value).Type()) raw := bson.Raw{item.data[0], []byte(item.data[3:])} c.Logf("Unmarshal raw: %#v, %#v", raw, pv.Interface()) err := raw.Unmarshal(pv.Interface()) c.Assert(err, IsNil) c.Assert(pv.Elem().Interface(), DeepEquals, value, Commentf("Failed on item %d: %#v", i, item)) } } func (s *S) TestUnmarshalRawIncompatible(c *C) { raw := bson.Raw{0x08, []byte{0x01}} // true err := raw.Unmarshal(&struct{}{}) c.Assert(err, ErrorMatches, "BSON kind 0x08 isn't compatible with type struct \\{\\}") } func (s *S) TestUnmarshalZeroesStruct(c *C) { data, err := bson.Marshal(bson.M{"b": 2}) c.Assert(err, IsNil) type T struct{ A, B int } v := T{A: 1} err = bson.Unmarshal(data, &v) c.Assert(err, IsNil) c.Assert(v.A, Equals, 0) c.Assert(v.B, Equals, 2) } func (s *S) TestUnmarshalZeroesMap(c *C) { data, err := bson.Marshal(bson.M{"b": 2}) c.Assert(err, IsNil) m := bson.M{"a": 1} err = bson.Unmarshal(data, &m) c.Assert(err, IsNil) c.Assert(m, DeepEquals, bson.M{"b": 2}) } func (s *S) TestUnmarshalNonNilInterface(c *C) { data, err := bson.Marshal(bson.M{"b": 2}) c.Assert(err, IsNil) m := bson.M{"a": 1} var i interface{} i = m err = bson.Unmarshal(data, &i) c.Assert(err, IsNil) c.Assert(i, DeepEquals, bson.M{"b": 2}) c.Assert(m, DeepEquals, bson.M{"a": 1}) } // -------------------------------------------------------------------------- // Some one way marshaling operations which would unmarshal differently. var oneWayMarshalItems = []testItemType{ // These are being passed as pointers, and will unmarshal as values. {bson.M{"": &bson.Binary{0x02, []byte("old")}}, "\x05\x00\x07\x00\x00\x00\x02\x03\x00\x00\x00old"}, {bson.M{"": &bson.Binary{0x80, []byte("udef")}}, "\x05\x00\x04\x00\x00\x00\x80udef"}, {bson.M{"": &bson.RegEx{"ab", "cd"}}, "\x0B\x00ab\x00cd\x00"}, {bson.M{"": &bson.JavaScript{"code", nil}}, "\x0D\x00\x05\x00\x00\x00code\x00"}, {bson.M{"": &bson.JavaScript{"code", bson.M{"": nil}}}, "\x0F\x00\x14\x00\x00\x00\x05\x00\x00\x00code\x00" + "\x07\x00\x00\x00\x0A\x00\x00"}, // There's no float32 type in BSON. Will encode as a float64. {bson.M{"": float32(5.05)}, "\x01\x00\x00\x00\x00@33\x14@"}, // The array will be unmarshaled as a slice instead. {bson.M{"": [2]bool{true, false}}, "\x04\x00\r\x00\x00\x00\x080\x00\x01\x081\x00\x00\x00"}, // The typed slice will be unmarshaled as []interface{}. {bson.M{"": []bool{true, false}}, "\x04\x00\r\x00\x00\x00\x080\x00\x01\x081\x00\x00\x00"}, // Will unmarshal as a []byte. {bson.M{"": bson.Binary{0x00, []byte("yo")}}, "\x05\x00\x02\x00\x00\x00\x00yo"}, {bson.M{"": bson.Binary{0x02, []byte("old")}}, "\x05\x00\x07\x00\x00\x00\x02\x03\x00\x00\x00old"}, // No way to preserve the type information here. We might encode as a zero // value, but this would mean that pointer values in structs wouldn't be // able to correctly distinguish between unset and set to the zero value. {bson.M{"": (*byte)(nil)}, "\x0A\x00"}, // No int types smaller than int32 in BSON. Could encode this as a char, // but it would still be ambiguous, take more, and be awkward in Go when // loaded without typing information. {bson.M{"": byte(8)}, "\x10\x00\x08\x00\x00\x00"}, // There are no unsigned types in BSON. Will unmarshal as int32 or int64. {bson.M{"": uint32(258)}, "\x10\x00\x02\x01\x00\x00"}, {bson.M{"": uint64(258)}, "\x12\x00\x02\x01\x00\x00\x00\x00\x00\x00"}, {bson.M{"": uint64(258 << 32)}, "\x12\x00\x00\x00\x00\x00\x02\x01\x00\x00"}, // This will unmarshal as int. {bson.M{"": int32(258)}, "\x10\x00\x02\x01\x00\x00"}, // That's a special case. The unsigned value is too large for an int32, // so an int64 is used instead. {bson.M{"": uint32(1<<32 - 1)}, "\x12\x00\xFF\xFF\xFF\xFF\x00\x00\x00\x00"}, {bson.M{"": uint(1<<32 - 1)}, "\x12\x00\xFF\xFF\xFF\xFF\x00\x00\x00\x00"}, } func (s *S) TestOneWayMarshalItems(c *C) { for i, item := range oneWayMarshalItems { data, err := bson.Marshal(item.obj) c.Assert(err, IsNil) c.Assert(string(data), Equals, wrapInDoc(item.data), Commentf("Failed on item %d", i)) } } // -------------------------------------------------------------------------- // Two-way tests for user-defined structures using the samples // from bsonspec.org. type specSample1 struct { Hello string } type specSample2 struct { BSON []interface{} "BSON" } var structSampleItems = []testItemType{ {&specSample1{"world"}, "\x16\x00\x00\x00\x02hello\x00\x06\x00\x00\x00world\x00\x00"}, {&specSample2{[]interface{}{"awesome", float64(5.05), 1986}}, "1\x00\x00\x00\x04BSON\x00&\x00\x00\x00\x020\x00\x08\x00\x00\x00" + "awesome\x00\x011\x00333333\x14@\x102\x00\xc2\x07\x00\x00\x00\x00"}, } func (s *S) TestMarshalStructSampleItems(c *C) { for i, item := range structSampleItems { data, err := bson.Marshal(item.obj) c.Assert(err, IsNil) c.Assert(string(data), Equals, item.data, Commentf("Failed on item %d", i)) } } func (s *S) TestUnmarshalStructSampleItems(c *C) { for _, item := range structSampleItems { testUnmarshal(c, item.data, item.obj) } } func (s *S) Test64bitInt(c *C) { var i int64 = (1 << 31) if int(i) > 0 { data, err := bson.Marshal(bson.M{"i": int(i)}) c.Assert(err, IsNil) c.Assert(string(data), Equals, wrapInDoc("\x12i\x00\x00\x00\x00\x80\x00\x00\x00\x00")) var result struct{ I int } err = bson.Unmarshal(data, &result) c.Assert(err, IsNil) c.Assert(int64(result.I), Equals, i) } } // -------------------------------------------------------------------------- // Generic two-way struct marshaling tests. var bytevar = byte(8) var byteptr = &bytevar var structItems = []testItemType{ {&struct{ Ptr *byte }{nil}, "\x0Aptr\x00"}, {&struct{ Ptr *byte }{&bytevar}, "\x10ptr\x00\x08\x00\x00\x00"}, {&struct{ Ptr **byte }{&byteptr}, "\x10ptr\x00\x08\x00\x00\x00"}, {&struct{ Byte byte }{8}, "\x10byte\x00\x08\x00\x00\x00"}, {&struct{ Byte byte }{0}, "\x10byte\x00\x00\x00\x00\x00"}, {&struct { V byte "Tag" }{8}, "\x10Tag\x00\x08\x00\x00\x00"}, {&struct { V *struct { Byte byte } }{&struct{ Byte byte }{8}}, "\x03v\x00" + "\x0f\x00\x00\x00\x10byte\x00\b\x00\x00\x00\x00"}, {&struct{ priv byte }{}, ""}, // The order of the dumped fields should be the same in the struct. {&struct{ A, C, B, D, F, E *byte }{}, "\x0Aa\x00\x0Ac\x00\x0Ab\x00\x0Ad\x00\x0Af\x00\x0Ae\x00"}, {&struct{ V bson.Raw }{bson.Raw{0x03, []byte("\x0f\x00\x00\x00\x10byte\x00\b\x00\x00\x00\x00")}}, "\x03v\x00" + "\x0f\x00\x00\x00\x10byte\x00\b\x00\x00\x00\x00"}, {&struct{ V bson.Raw }{bson.Raw{0x10, []byte("\x00\x00\x00\x00")}}, "\x10v\x00" + "\x00\x00\x00\x00"}, // Byte arrays. {&struct{ V [2]byte }{[2]byte{'y', 'o'}}, "\x05v\x00\x02\x00\x00\x00\x00yo"}, } func (s *S) TestMarshalStructItems(c *C) { for i, item := range structItems { data, err := bson.Marshal(item.obj) c.Assert(err, IsNil) c.Assert(string(data), Equals, wrapInDoc(item.data), Commentf("Failed on item %d", i)) } } func (s *S) TestUnmarshalStructItems(c *C) { for _, item := range structItems { testUnmarshal(c, wrapInDoc(item.data), item.obj) } } func (s *S) TestUnmarshalRawStructItems(c *C) { for i, item := range structItems { raw := bson.Raw{0x03, []byte(wrapInDoc(item.data))} zero := makeZeroDoc(item.obj) err := raw.Unmarshal(zero) c.Assert(err, IsNil) c.Assert(zero, DeepEquals, item.obj, Commentf("Failed on item %d: %#v", i, item)) } } func (s *S) TestUnmarshalRawNil(c *C) { // Regression test: shouldn't try to nil out the pointer itself, // as it's not settable. raw := bson.Raw{0x0A, []byte{}} err := raw.Unmarshal(&struct{}{}) c.Assert(err, IsNil) } // -------------------------------------------------------------------------- // One-way marshaling tests. type dOnIface struct { D interface{} } type ignoreField struct { Before string Ignore string `bson:"-"` After string } var marshalItems = []testItemType{ // Ordered document dump. Will unmarshal as a dictionary by default. {bson.D{{"a", nil}, {"c", nil}, {"b", nil}, {"d", nil}, {"f", nil}, {"e", true}}, "\x0Aa\x00\x0Ac\x00\x0Ab\x00\x0Ad\x00\x0Af\x00\x08e\x00\x01"}, {MyD{{"a", nil}, {"c", nil}, {"b", nil}, {"d", nil}, {"f", nil}, {"e", true}}, "\x0Aa\x00\x0Ac\x00\x0Ab\x00\x0Ad\x00\x0Af\x00\x08e\x00\x01"}, {&dOnIface{bson.D{{"a", nil}, {"c", nil}, {"b", nil}, {"d", true}}}, "\x03d\x00" + wrapInDoc("\x0Aa\x00\x0Ac\x00\x0Ab\x00\x08d\x00\x01")}, {bson.RawD{{"a", bson.Raw{0x0A, nil}}, {"c", bson.Raw{0x0A, nil}}, {"b", bson.Raw{0x08, []byte{0x01}}}}, "\x0Aa\x00" + "\x0Ac\x00" + "\x08b\x00\x01"}, {MyRawD{{"a", bson.Raw{0x0A, nil}}, {"c", bson.Raw{0x0A, nil}}, {"b", bson.Raw{0x08, []byte{0x01}}}}, "\x0Aa\x00" + "\x0Ac\x00" + "\x08b\x00\x01"}, {&dOnIface{bson.RawD{{"a", bson.Raw{0x0A, nil}}, {"c", bson.Raw{0x0A, nil}}, {"b", bson.Raw{0x08, []byte{0x01}}}}}, "\x03d\x00" + wrapInDoc("\x0Aa\x00"+"\x0Ac\x00"+"\x08b\x00\x01")}, {&ignoreField{"before", "ignore", "after"}, "\x02before\x00\a\x00\x00\x00before\x00\x02after\x00\x06\x00\x00\x00after\x00"}, // Marshalling a Raw document does nothing. {bson.Raw{0x03, []byte(wrapInDoc("anything"))}, "anything"}, {bson.Raw{Data: []byte(wrapInDoc("anything"))}, "anything"}, } func (s *S) TestMarshalOneWayItems(c *C) { for _, item := range marshalItems { data, err := bson.Marshal(item.obj) c.Assert(err, IsNil) c.Assert(string(data), Equals, wrapInDoc(item.data)) } } // -------------------------------------------------------------------------- // One-way unmarshaling tests. var unmarshalItems = []testItemType{ // Field is private. Should not attempt to unmarshal it. {&struct{ priv byte }{}, "\x10priv\x00\x08\x00\x00\x00"}, // Wrong casing. Field names are lowercased. {&struct{ Byte byte }{}, "\x10Byte\x00\x08\x00\x00\x00"}, // Ignore non-existing field. {&struct{ Byte byte }{9}, "\x10boot\x00\x08\x00\x00\x00" + "\x10byte\x00\x09\x00\x00\x00"}, // Do not unmarshal on ignored field. {&ignoreField{"before", "", "after"}, "\x02before\x00\a\x00\x00\x00before\x00" + "\x02-\x00\a\x00\x00\x00ignore\x00" + "\x02after\x00\x06\x00\x00\x00after\x00"}, // Ignore unsuitable types silently. {map[string]string{"str": "s"}, "\x02str\x00\x02\x00\x00\x00s\x00" + "\x10int\x00\x01\x00\x00\x00"}, {map[string][]int{"array": []int{5, 9}}, "\x04array\x00" + wrapInDoc("\x100\x00\x05\x00\x00\x00"+"\x021\x00\x02\x00\x00\x00s\x00"+"\x102\x00\x09\x00\x00\x00")}, // Wrong type. Shouldn't init pointer. {&struct{ Str *byte }{}, "\x02str\x00\x02\x00\x00\x00s\x00"}, {&struct{ Str *struct{ Str string } }{}, "\x02str\x00\x02\x00\x00\x00s\x00"}, // Ordered document. {&struct{ bson.D }{bson.D{{"a", nil}, {"c", nil}, {"b", nil}, {"d", true}}}, "\x03d\x00" + wrapInDoc("\x0Aa\x00\x0Ac\x00\x0Ab\x00\x08d\x00\x01")}, // Raw document. {&bson.Raw{0x03, []byte(wrapInDoc("\x10byte\x00\x08\x00\x00\x00"))}, "\x10byte\x00\x08\x00\x00\x00"}, // RawD document. {&struct{ bson.RawD }{bson.RawD{{"a", bson.Raw{0x0A, []byte{}}}, {"c", bson.Raw{0x0A, []byte{}}}, {"b", bson.Raw{0x08, []byte{0x01}}}}}, "\x03rawd\x00" + wrapInDoc("\x0Aa\x00\x0Ac\x00\x08b\x00\x01")}, // Decode old binary. {bson.M{"_": []byte("old")}, "\x05_\x00\x07\x00\x00\x00\x02\x03\x00\x00\x00old"}, // Decode old binary without length. According to the spec, this shouldn't happen. {bson.M{"_": []byte("old")}, "\x05_\x00\x03\x00\x00\x00\x02old"}, // Decode a doc within a doc in to a slice within a doc; shouldn't error {&struct{ Foo []string }{}, "\x03\x66\x6f\x6f\x00\x05\x00\x00\x00\x00"}, } func (s *S) TestUnmarshalOneWayItems(c *C) { for _, item := range unmarshalItems { testUnmarshal(c, wrapInDoc(item.data), item.obj) } } func (s *S) TestUnmarshalNilInStruct(c *C) { // Nil is the default value, so we need to ensure it's indeed being set. b := byte(1) v := &struct{ Ptr *byte }{&b} err := bson.Unmarshal([]byte(wrapInDoc("\x0Aptr\x00")), v) c.Assert(err, IsNil) c.Assert(v, DeepEquals, &struct{ Ptr *byte }{nil}) } // -------------------------------------------------------------------------- // Marshalling error cases. type structWithDupKeys struct { Name byte Other byte "name" // Tag should precede. } var marshalErrorItems = []testItemType{ {bson.M{"": uint64(1 << 63)}, "BSON has no uint64 type, and value is too large to fit correctly in an int64"}, {bson.M{"": bson.ObjectId("tooshort")}, "ObjectIDs must be exactly 12 bytes long \\(got 8\\)"}, {int64(123), "Can't marshal int64 as a BSON document"}, {bson.M{"": 1i}, "Can't marshal complex128 in a BSON document"}, {&structWithDupKeys{}, "Duplicated key 'name' in struct bson_test.structWithDupKeys"}, {bson.Raw{0xA, []byte{}}, "Attempted to marshal Raw kind 10 as a document"}, {bson.Raw{0x3, []byte{}}, "Attempted to marshal empty Raw document"}, {bson.M{"w": bson.Raw{0x3, []byte{}}}, "Attempted to marshal empty Raw document"}, {&inlineCantPtr{&struct{ A, B int }{1, 2}}, "Option ,inline needs a struct value or map field"}, {&inlineDupName{1, struct{ A, B int }{2, 3}}, "Duplicated key 'a' in struct bson_test.inlineDupName"}, {&inlineDupMap{}, "Multiple ,inline maps in struct bson_test.inlineDupMap"}, {&inlineBadKeyMap{}, "Option ,inline needs a map with string keys in struct bson_test.inlineBadKeyMap"}, {&inlineMap{A: 1, M: map[string]interface{}{"a": 1}}, `Can't have key "a" in inlined map; conflicts with struct field`}, } func (s *S) TestMarshalErrorItems(c *C) { for _, item := range marshalErrorItems { data, err := bson.Marshal(item.obj) c.Assert(err, ErrorMatches, item.data) c.Assert(data, IsNil) } } // -------------------------------------------------------------------------- // Unmarshalling error cases. type unmarshalErrorType struct { obj interface{} data string error string } var unmarshalErrorItems = []unmarshalErrorType{ // Tag name conflicts with existing parameter. {&structWithDupKeys{}, "\x10name\x00\x08\x00\x00\x00", "Duplicated key 'name' in struct bson_test.structWithDupKeys"}, // Non-string map key. {map[int]interface{}{}, "\x10name\x00\x08\x00\x00\x00", "BSON map must have string keys. Got: map\\[int\\]interface \\{\\}"}, {nil, "\xEEname\x00", "Unknown element kind \\(0xEE\\)"}, {struct{ Name bool }{}, "\x10name\x00\x08\x00\x00\x00", "Unmarshal can't deal with struct values. Use a pointer."}, {123, "\x10name\x00\x08\x00\x00\x00", "Unmarshal needs a map or a pointer to a struct."}, {nil, "\x08\x62\x00\x02", "encoded boolean must be 1 or 0, found 2"}, } func (s *S) TestUnmarshalErrorItems(c *C) { for _, item := range unmarshalErrorItems { data := []byte(wrapInDoc(item.data)) var value interface{} switch reflect.ValueOf(item.obj).Kind() { case reflect.Map, reflect.Ptr: value = makeZeroDoc(item.obj) case reflect.Invalid: value = bson.M{} default: value = item.obj } err := bson.Unmarshal(data, value) c.Assert(err, ErrorMatches, item.error) } } type unmarshalRawErrorType struct { obj interface{} raw bson.Raw error string } var unmarshalRawErrorItems = []unmarshalRawErrorType{ // Tag name conflicts with existing parameter. {&structWithDupKeys{}, bson.Raw{0x03, []byte("\x10byte\x00\x08\x00\x00\x00")}, "Duplicated key 'name' in struct bson_test.structWithDupKeys"}, {&struct{}{}, bson.Raw{0xEE, []byte{}}, "Unknown element kind \\(0xEE\\)"}, {struct{ Name bool }{}, bson.Raw{0x10, []byte("\x08\x00\x00\x00")}, "Raw Unmarshal can't deal with struct values. Use a pointer."}, {123, bson.Raw{0x10, []byte("\x08\x00\x00\x00")}, "Raw Unmarshal needs a map or a valid pointer."}, } func (s *S) TestUnmarshalRawErrorItems(c *C) { for i, item := range unmarshalRawErrorItems { err := item.raw.Unmarshal(item.obj) c.Assert(err, ErrorMatches, item.error, Commentf("Failed on item %d: %#v\n", i, item)) } } var corruptedData = []string{ "\x04\x00\x00\x00\x00", // Document shorter than minimum "\x06\x00\x00\x00\x00", // Not enough data "\x05\x00\x00", // Broken length "\x05\x00\x00\x00\xff", // Corrupted termination "\x0A\x00\x00\x00\x0Aooop\x00", // Unfinished C string // Array end past end of string (s[2]=0x07 is correct) wrapInDoc("\x04\x00\x09\x00\x00\x00\x0A\x00\x00"), // Array end within string, but past acceptable. wrapInDoc("\x04\x00\x08\x00\x00\x00\x0A\x00\x00"), // Document end within string, but past acceptable. wrapInDoc("\x03\x00\x08\x00\x00\x00\x0A\x00\x00"), // String with corrupted end. wrapInDoc("\x02\x00\x03\x00\x00\x00yo\xFF"), // String with negative length (issue #116). "\x0c\x00\x00\x00\x02x\x00\xff\xff\xff\xff\x00", // String with zero length (must include trailing '\x00') "\x0c\x00\x00\x00\x02x\x00\x00\x00\x00\x00\x00", // Binary with negative length. "\r\x00\x00\x00\x05x\x00\xff\xff\xff\xff\x00\x00", } func (s *S) TestUnmarshalMapDocumentTooShort(c *C) { for _, data := range corruptedData { err := bson.Unmarshal([]byte(data), bson.M{}) c.Assert(err, ErrorMatches, "Document is corrupted") err = bson.Unmarshal([]byte(data), &struct{}{}) c.Assert(err, ErrorMatches, "Document is corrupted") } } // -------------------------------------------------------------------------- // Setter test cases. var setterResult = map[string]error{} type setterType struct { received interface{} } func (o *setterType) SetBSON(raw bson.Raw) error { err := raw.Unmarshal(&o.received) if err != nil { panic("The panic:" + err.Error()) } if s, ok := o.received.(string); ok { if result, ok := setterResult[s]; ok { return result } } return nil } type ptrSetterDoc struct { Field *setterType "_" } type valSetterDoc struct { Field setterType "_" } func (s *S) TestUnmarshalAllItemsWithPtrSetter(c *C) { for _, item := range allItems { for i := 0; i != 2; i++ { var field *setterType if i == 0 { obj := &ptrSetterDoc{} err := bson.Unmarshal([]byte(wrapInDoc(item.data)), obj) c.Assert(err, IsNil) field = obj.Field } else { obj := &valSetterDoc{} err := bson.Unmarshal([]byte(wrapInDoc(item.data)), obj) c.Assert(err, IsNil) field = &obj.Field } if item.data == "" { // Nothing to unmarshal. Should be untouched. if i == 0 { c.Assert(field, IsNil) } else { c.Assert(field.received, IsNil) } } else { expected := item.obj.(bson.M)["_"] c.Assert(field, NotNil, Commentf("Pointer not initialized (%#v)", expected)) c.Assert(field.received, DeepEquals, expected) } } } } func (s *S) TestUnmarshalWholeDocumentWithSetter(c *C) { obj := &setterType{} err := bson.Unmarshal([]byte(sampleItems[0].data), obj) c.Assert(err, IsNil) c.Assert(obj.received, DeepEquals, bson.M{"hello": "world"}) } func (s *S) TestUnmarshalSetterOmits(c *C) { setterResult["2"] = &bson.TypeError{} setterResult["4"] = &bson.TypeError{} defer func() { delete(setterResult, "2") delete(setterResult, "4") }() m := map[string]*setterType{} data := wrapInDoc("\x02abc\x00\x02\x00\x00\x001\x00" + "\x02def\x00\x02\x00\x00\x002\x00" + "\x02ghi\x00\x02\x00\x00\x003\x00" + "\x02jkl\x00\x02\x00\x00\x004\x00") err := bson.Unmarshal([]byte(data), m) c.Assert(err, IsNil) c.Assert(m["abc"], NotNil) c.Assert(m["def"], IsNil) c.Assert(m["ghi"], NotNil) c.Assert(m["jkl"], IsNil) c.Assert(m["abc"].received, Equals, "1") c.Assert(m["ghi"].received, Equals, "3") } func (s *S) TestUnmarshalSetterErrors(c *C) { boom := errors.New("BOOM") setterResult["2"] = boom defer delete(setterResult, "2") m := map[string]*setterType{} data := wrapInDoc("\x02abc\x00\x02\x00\x00\x001\x00" + "\x02def\x00\x02\x00\x00\x002\x00" + "\x02ghi\x00\x02\x00\x00\x003\x00") err := bson.Unmarshal([]byte(data), m) c.Assert(err, Equals, boom) c.Assert(m["abc"], NotNil) c.Assert(m["def"], IsNil) c.Assert(m["ghi"], IsNil) c.Assert(m["abc"].received, Equals, "1") } func (s *S) TestDMap(c *C) { d := bson.D{{"a", 1}, {"b", 2}} c.Assert(d.Map(), DeepEquals, bson.M{"a": 1, "b": 2}) } func (s *S) TestUnmarshalSetterSetZero(c *C) { setterResult["foo"] = bson.SetZero defer delete(setterResult, "field") data, err := bson.Marshal(bson.M{"field": "foo"}) c.Assert(err, IsNil) m := map[string]*setterType{} err = bson.Unmarshal([]byte(data), m) c.Assert(err, IsNil) value, ok := m["field"] c.Assert(ok, Equals, true) c.Assert(value, IsNil) } // -------------------------------------------------------------------------- // Getter test cases. type typeWithGetter struct { result interface{} err error } func (t *typeWithGetter) GetBSON() (interface{}, error) { if t == nil { return "", nil } return t.result, t.err } type docWithGetterField struct { Field *typeWithGetter "_" } func (s *S) TestMarshalAllItemsWithGetter(c *C) { for i, item := range allItems { if item.data == "" { continue } obj := &docWithGetterField{} obj.Field = &typeWithGetter{result: item.obj.(bson.M)["_"]} data, err := bson.Marshal(obj) c.Assert(err, IsNil) c.Assert(string(data), Equals, wrapInDoc(item.data), Commentf("Failed on item #%d", i)) } } func (s *S) TestMarshalWholeDocumentWithGetter(c *C) { obj := &typeWithGetter{result: sampleItems[0].obj} data, err := bson.Marshal(obj) c.Assert(err, IsNil) c.Assert(string(data), Equals, sampleItems[0].data) } func (s *S) TestGetterErrors(c *C) { e := errors.New("oops") obj1 := &docWithGetterField{} obj1.Field = &typeWithGetter{sampleItems[0].obj, e} data, err := bson.Marshal(obj1) c.Assert(err, ErrorMatches, "oops") c.Assert(data, IsNil) obj2 := &typeWithGetter{sampleItems[0].obj, e} data, err = bson.Marshal(obj2) c.Assert(err, ErrorMatches, "oops") c.Assert(data, IsNil) } type intGetter int64 func (t intGetter) GetBSON() (interface{}, error) { return int64(t), nil } type typeWithIntGetter struct { V intGetter ",minsize" } func (s *S) TestMarshalShortWithGetter(c *C) { obj := typeWithIntGetter{42} data, err := bson.Marshal(obj) c.Assert(err, IsNil) m := bson.M{} err = bson.Unmarshal(data, m) c.Assert(err, IsNil) c.Assert(m["v"], Equals, 42) } func (s *S) TestMarshalWithGetterNil(c *C) { obj := docWithGetterField{} data, err := bson.Marshal(obj) c.Assert(err, IsNil) m := bson.M{} err = bson.Unmarshal(data, m) c.Assert(err, IsNil) c.Assert(m, DeepEquals, bson.M{"_": ""}) } // -------------------------------------------------------------------------- // Cross-type conversion tests. type crossTypeItem struct { obj1 interface{} obj2 interface{} } type condStr struct { V string ",omitempty" } type condStrNS struct { V string `a:"A" bson:",omitempty" b:"B"` } type condBool struct { V bool ",omitempty" } type condInt struct { V int ",omitempty" } type condUInt struct { V uint ",omitempty" } type condFloat struct { V float64 ",omitempty" } type condIface struct { V interface{} ",omitempty" } type condPtr struct { V *bool ",omitempty" } type condSlice struct { V []string ",omitempty" } type condMap struct { V map[string]int ",omitempty" } type namedCondStr struct { V string "myv,omitempty" } type condTime struct { V time.Time ",omitempty" } type condStruct struct { V struct{ A []int } ",omitempty" } type condRaw struct { V bson.Raw ",omitempty" } type shortInt struct { V int64 ",minsize" } type shortUint struct { V uint64 ",minsize" } type shortIface struct { V interface{} ",minsize" } type shortPtr struct { V *int64 ",minsize" } type shortNonEmptyInt struct { V int64 ",minsize,omitempty" } type inlineInt struct { V struct{ A, B int } ",inline" } type inlineCantPtr struct { V *struct{ A, B int } ",inline" } type inlineDupName struct { A int V struct{ A, B int } ",inline" } type inlineMap struct { A int M map[string]interface{} ",inline" } type inlineMapInt struct { A int M map[string]int ",inline" } type inlineMapMyM struct { A int M MyM ",inline" } type inlineDupMap struct { M1 map[string]interface{} ",inline" M2 map[string]interface{} ",inline" } type inlineBadKeyMap struct { M map[int]int ",inline" } type inlineUnexported struct { M map[string]interface{} ",inline" unexported ",inline" } type unexported struct { A int } type getterSetterD bson.D func (s getterSetterD) GetBSON() (interface{}, error) { if len(s) == 0 { return bson.D{}, nil } return bson.D(s[:len(s)-1]), nil } func (s *getterSetterD) SetBSON(raw bson.Raw) error { var doc bson.D err := raw.Unmarshal(&doc) doc = append(doc, bson.DocElem{"suffix", true}) *s = getterSetterD(doc) return err } type getterSetterInt int func (i getterSetterInt) GetBSON() (interface{}, error) { return bson.D{{"a", int(i)}}, nil } func (i *getterSetterInt) SetBSON(raw bson.Raw) error { var doc struct{ A int } err := raw.Unmarshal(&doc) *i = getterSetterInt(doc.A) return err } type ifaceType interface { Hello() } type ifaceSlice []ifaceType func (s *ifaceSlice) SetBSON(raw bson.Raw) error { var ns []int if err := raw.Unmarshal(&ns); err != nil { return err } *s = make(ifaceSlice, ns[0]) return nil } func (s ifaceSlice) GetBSON() (interface{}, error) { return []int{len(s)}, nil } type ( MyString string MyBytes []byte MyBool bool MyD []bson.DocElem MyRawD []bson.RawDocElem MyM map[string]interface{} ) var ( truevar = true falsevar = false int64var = int64(42) int64ptr = &int64var intvar = int(42) intptr = &intvar gsintvar = getterSetterInt(42) ) func parseURL(s string) *url.URL { u, err := url.Parse(s) if err != nil { panic(err) } return u } // That's a pretty fun test. It will dump the first item, generate a zero // value equivalent to the second one, load the dumped data onto it, and then // verify that the resulting value is deep-equal to the untouched second value. // Then, it will do the same in the *opposite* direction! var twoWayCrossItems = []crossTypeItem{ // int<=>int {&struct{ I int }{42}, &struct{ I int8 }{42}}, {&struct{ I int }{42}, &struct{ I int32 }{42}}, {&struct{ I int }{42}, &struct{ I int64 }{42}}, {&struct{ I int8 }{42}, &struct{ I int32 }{42}}, {&struct{ I int8 }{42}, &struct{ I int64 }{42}}, {&struct{ I int32 }{42}, &struct{ I int64 }{42}}, // uint<=>uint {&struct{ I uint }{42}, &struct{ I uint8 }{42}}, {&struct{ I uint }{42}, &struct{ I uint32 }{42}}, {&struct{ I uint }{42}, &struct{ I uint64 }{42}}, {&struct{ I uint8 }{42}, &struct{ I uint32 }{42}}, {&struct{ I uint8 }{42}, &struct{ I uint64 }{42}}, {&struct{ I uint32 }{42}, &struct{ I uint64 }{42}}, // float32<=>float64 {&struct{ I float32 }{42}, &struct{ I float64 }{42}}, // int<=>uint {&struct{ I uint }{42}, &struct{ I int }{42}}, {&struct{ I uint }{42}, &struct{ I int8 }{42}}, {&struct{ I uint }{42}, &struct{ I int32 }{42}}, {&struct{ I uint }{42}, &struct{ I int64 }{42}}, {&struct{ I uint8 }{42}, &struct{ I int }{42}}, {&struct{ I uint8 }{42}, &struct{ I int8 }{42}}, {&struct{ I uint8 }{42}, &struct{ I int32 }{42}}, {&struct{ I uint8 }{42}, &struct{ I int64 }{42}}, {&struct{ I uint32 }{42}, &struct{ I int }{42}}, {&struct{ I uint32 }{42}, &struct{ I int8 }{42}}, {&struct{ I uint32 }{42}, &struct{ I int32 }{42}}, {&struct{ I uint32 }{42}, &struct{ I int64 }{42}}, {&struct{ I uint64 }{42}, &struct{ I int }{42}}, {&struct{ I uint64 }{42}, &struct{ I int8 }{42}}, {&struct{ I uint64 }{42}, &struct{ I int32 }{42}}, {&struct{ I uint64 }{42}, &struct{ I int64 }{42}}, // int <=> float {&struct{ I int }{42}, &struct{ I float64 }{42}}, // int <=> bool {&struct{ I int }{1}, &struct{ I bool }{true}}, {&struct{ I int }{0}, &struct{ I bool }{false}}, // uint <=> float64 {&struct{ I uint }{42}, &struct{ I float64 }{42}}, // uint <=> bool {&struct{ I uint }{1}, &struct{ I bool }{true}}, {&struct{ I uint }{0}, &struct{ I bool }{false}}, // float64 <=> bool {&struct{ I float64 }{1}, &struct{ I bool }{true}}, {&struct{ I float64 }{0}, &struct{ I bool }{false}}, // string <=> string and string <=> []byte {&struct{ S []byte }{[]byte("abc")}, &struct{ S string }{"abc"}}, {&struct{ S []byte }{[]byte("def")}, &struct{ S bson.Symbol }{"def"}}, {&struct{ S string }{"ghi"}, &struct{ S bson.Symbol }{"ghi"}}, // map <=> struct {&struct { A struct { B, C int } }{struct{ B, C int }{1, 2}}, map[string]map[string]int{"a": map[string]int{"b": 1, "c": 2}}}, {&struct{ A bson.Symbol }{"abc"}, map[string]string{"a": "abc"}}, {&struct{ A bson.Symbol }{"abc"}, map[string][]byte{"a": []byte("abc")}}, {&struct{ A []byte }{[]byte("abc")}, map[string]string{"a": "abc"}}, {&struct{ A uint }{42}, map[string]int{"a": 42}}, {&struct{ A uint }{42}, map[string]float64{"a": 42}}, {&struct{ A uint }{1}, map[string]bool{"a": true}}, {&struct{ A int }{42}, map[string]uint{"a": 42}}, {&struct{ A int }{42}, map[string]float64{"a": 42}}, {&struct{ A int }{1}, map[string]bool{"a": true}}, {&struct{ A float64 }{42}, map[string]float32{"a": 42}}, {&struct{ A float64 }{42}, map[string]int{"a": 42}}, {&struct{ A float64 }{42}, map[string]uint{"a": 42}}, {&struct{ A float64 }{1}, map[string]bool{"a": true}}, {&struct{ A bool }{true}, map[string]int{"a": 1}}, {&struct{ A bool }{true}, map[string]uint{"a": 1}}, {&struct{ A bool }{true}, map[string]float64{"a": 1}}, {&struct{ A **byte }{&byteptr}, map[string]byte{"a": 8}}, // url.URL <=> string {&struct{ URL *url.URL }{parseURL("h://e.c/p")}, map[string]string{"url": "h://e.c/p"}}, {&struct{ URL url.URL }{*parseURL("h://e.c/p")}, map[string]string{"url": "h://e.c/p"}}, // Slices {&struct{ S []int }{[]int{1, 2, 3}}, map[string][]int{"s": []int{1, 2, 3}}}, {&struct{ S *[]int }{&[]int{1, 2, 3}}, map[string][]int{"s": []int{1, 2, 3}}}, // Conditionals {&condBool{true}, map[string]bool{"v": true}}, {&condBool{}, map[string]bool{}}, {&condInt{1}, map[string]int{"v": 1}}, {&condInt{}, map[string]int{}}, {&condUInt{1}, map[string]uint{"v": 1}}, {&condUInt{}, map[string]uint{}}, {&condFloat{}, map[string]int{}}, {&condStr{"yo"}, map[string]string{"v": "yo"}}, {&condStr{}, map[string]string{}}, {&condStrNS{"yo"}, map[string]string{"v": "yo"}}, {&condStrNS{}, map[string]string{}}, {&condSlice{[]string{"yo"}}, map[string][]string{"v": []string{"yo"}}}, {&condSlice{}, map[string][]string{}}, {&condMap{map[string]int{"k": 1}}, bson.M{"v": bson.M{"k": 1}}}, {&condMap{}, map[string][]string{}}, {&condIface{"yo"}, map[string]string{"v": "yo"}}, {&condIface{""}, map[string]string{"v": ""}}, {&condIface{}, map[string]string{}}, {&condPtr{&truevar}, map[string]bool{"v": true}}, {&condPtr{&falsevar}, map[string]bool{"v": false}}, {&condPtr{}, map[string]string{}}, {&condTime{time.Unix(123456789, 123e6)}, map[string]time.Time{"v": time.Unix(123456789, 123e6)}}, {&condTime{}, map[string]string{}}, {&condStruct{struct{ A []int }{[]int{1}}}, bson.M{"v": bson.M{"a": []interface{}{1}}}}, {&condStruct{struct{ A []int }{}}, bson.M{}}, {&condRaw{bson.Raw{Kind: 0x0A, Data: []byte{}}}, bson.M{"v": nil}}, {&condRaw{bson.Raw{Kind: 0x00}}, bson.M{}}, {&namedCondStr{"yo"}, map[string]string{"myv": "yo"}}, {&namedCondStr{}, map[string]string{}}, {&shortInt{1}, map[string]interface{}{"v": 1}}, {&shortInt{1 << 30}, map[string]interface{}{"v": 1 << 30}}, {&shortInt{1 << 31}, map[string]interface{}{"v": int64(1 << 31)}}, {&shortUint{1 << 30}, map[string]interface{}{"v": 1 << 30}}, {&shortUint{1 << 31}, map[string]interface{}{"v": int64(1 << 31)}}, {&shortIface{int64(1) << 31}, map[string]interface{}{"v": int64(1 << 31)}}, {&shortPtr{int64ptr}, map[string]interface{}{"v": intvar}}, {&shortNonEmptyInt{1}, map[string]interface{}{"v": 1}}, {&shortNonEmptyInt{1 << 31}, map[string]interface{}{"v": int64(1 << 31)}}, {&shortNonEmptyInt{}, map[string]interface{}{}}, {&inlineInt{struct{ A, B int }{1, 2}}, map[string]interface{}{"a": 1, "b": 2}}, {&inlineMap{A: 1, M: map[string]interface{}{"b": 2}}, map[string]interface{}{"a": 1, "b": 2}}, {&inlineMap{A: 1, M: nil}, map[string]interface{}{"a": 1}}, {&inlineMapInt{A: 1, M: map[string]int{"b": 2}}, map[string]int{"a": 1, "b": 2}}, {&inlineMapInt{A: 1, M: nil}, map[string]int{"a": 1}}, {&inlineMapMyM{A: 1, M: MyM{"b": MyM{"c": 3}}}, map[string]interface{}{"a": 1, "b": map[string]interface{}{"c": 3}}}, {&inlineUnexported{M: map[string]interface{}{"b": 1}, unexported: unexported{A: 2}}, map[string]interface{}{"b": 1, "a": 2}}, // []byte <=> Binary {&struct{ B []byte }{[]byte("abc")}, map[string]bson.Binary{"b": bson.Binary{Data: []byte("abc")}}}, // []byte <=> MyBytes {&struct{ B MyBytes }{[]byte("abc")}, map[string]string{"b": "abc"}}, {&struct{ B MyBytes }{[]byte{}}, map[string]string{"b": ""}}, {&struct{ B MyBytes }{}, map[string]bool{}}, {&struct{ B []byte }{[]byte("abc")}, map[string]MyBytes{"b": []byte("abc")}}, // bool <=> MyBool {&struct{ B MyBool }{true}, map[string]bool{"b": true}}, {&struct{ B MyBool }{}, map[string]bool{"b": false}}, {&struct{ B MyBool }{}, map[string]string{}}, {&struct{ B bool }{}, map[string]MyBool{"b": false}}, // arrays {&struct{ V [2]int }{[...]int{1, 2}}, map[string][2]int{"v": [2]int{1, 2}}}, {&struct{ V [2]byte }{[...]byte{1, 2}}, map[string][2]byte{"v": [2]byte{1, 2}}}, // zero time {&struct{ V time.Time }{}, map[string]interface{}{"v": time.Time{}}}, // zero time + 1 second + 1 millisecond; overflows int64 as nanoseconds {&struct{ V time.Time }{time.Unix(-62135596799, 1e6).Local()}, map[string]interface{}{"v": time.Unix(-62135596799, 1e6).Local()}}, // bson.D <=> []DocElem {&bson.D{{"a", bson.D{{"b", 1}, {"c", 2}}}}, &bson.D{{"a", bson.D{{"b", 1}, {"c", 2}}}}}, {&bson.D{{"a", bson.D{{"b", 1}, {"c", 2}}}}, &MyD{{"a", MyD{{"b", 1}, {"c", 2}}}}}, {&struct{ V MyD }{MyD{{"a", 1}}}, &bson.D{{"v", bson.D{{"a", 1}}}}}, // bson.RawD <=> []RawDocElem {&bson.RawD{{"a", bson.Raw{0x08, []byte{0x01}}}}, &bson.RawD{{"a", bson.Raw{0x08, []byte{0x01}}}}}, {&bson.RawD{{"a", bson.Raw{0x08, []byte{0x01}}}}, &MyRawD{{"a", bson.Raw{0x08, []byte{0x01}}}}}, // bson.M <=> map {bson.M{"a": bson.M{"b": 1, "c": 2}}, MyM{"a": MyM{"b": 1, "c": 2}}}, {bson.M{"a": bson.M{"b": 1, "c": 2}}, map[string]interface{}{"a": map[string]interface{}{"b": 1, "c": 2}}}, // bson.M <=> map[MyString] {bson.M{"a": bson.M{"b": 1, "c": 2}}, map[MyString]interface{}{"a": map[MyString]interface{}{"b": 1, "c": 2}}}, // json.Number <=> int64, float64 {&struct{ N json.Number }{"5"}, map[string]interface{}{"n": int64(5)}}, {&struct{ N json.Number }{"5.05"}, map[string]interface{}{"n": 5.05}}, {&struct{ N json.Number }{"9223372036854776000"}, map[string]interface{}{"n": float64(1 << 63)}}, // bson.D <=> non-struct getter/setter {&bson.D{{"a", 1}}, &getterSetterD{{"a", 1}, {"suffix", true}}}, {&bson.D{{"a", 42}}, &gsintvar}, // Interface slice setter. {&struct{ V ifaceSlice }{ifaceSlice{nil, nil, nil}}, bson.M{"v": []interface{}{3}}}, } // Same thing, but only one way (obj1 => obj2). var oneWayCrossItems = []crossTypeItem{ // map <=> struct {map[string]interface{}{"a": 1, "b": "2", "c": 3}, map[string]int{"a": 1, "c": 3}}, // inline map elides badly typed values {map[string]interface{}{"a": 1, "b": "2", "c": 3}, &inlineMapInt{A: 1, M: map[string]int{"c": 3}}}, // Can't decode int into struct. {bson.M{"a": bson.M{"b": 2}}, &struct{ A bool }{}}, // Would get decoded into a int32 too in the opposite direction. {&shortIface{int64(1) << 30}, map[string]interface{}{"v": 1 << 30}}, // Ensure omitempty on struct with private fields works properly. {&struct { V struct{ v time.Time } ",omitempty" }{}, map[string]interface{}{}}, // Attempt to marshal slice into RawD (issue #120). {bson.M{"x": []int{1, 2, 3}}, &struct{ X bson.RawD }{}}, } func testCrossPair(c *C, dump interface{}, load interface{}) { c.Logf("Dump: %#v", dump) c.Logf("Load: %#v", load) zero := makeZeroDoc(load) data, err := bson.Marshal(dump) c.Assert(err, IsNil) c.Logf("Dumped: %#v", string(data)) err = bson.Unmarshal(data, zero) c.Assert(err, IsNil) c.Logf("Loaded: %#v", zero) c.Assert(zero, DeepEquals, load) } func (s *S) TestTwoWayCrossPairs(c *C) { for _, item := range twoWayCrossItems { testCrossPair(c, item.obj1, item.obj2) testCrossPair(c, item.obj2, item.obj1) } } func (s *S) TestOneWayCrossPairs(c *C) { for _, item := range oneWayCrossItems { testCrossPair(c, item.obj1, item.obj2) } } // -------------------------------------------------------------------------- // ObjectId hex representation test. func (s *S) TestObjectIdHex(c *C) { id := bson.ObjectIdHex("4d88e15b60f486e428412dc9") c.Assert(id.String(), Equals, `ObjectIdHex("4d88e15b60f486e428412dc9")`) c.Assert(id.Hex(), Equals, "4d88e15b60f486e428412dc9") } func (s *S) TestIsObjectIdHex(c *C) { test := []struct { id string valid bool }{ {"4d88e15b60f486e428412dc9", true}, {"4d88e15b60f486e428412dc", false}, {"4d88e15b60f486e428412dc9e", false}, {"4d88e15b60f486e428412dcx", false}, } for _, t := range test { c.Assert(bson.IsObjectIdHex(t.id), Equals, t.valid) } } // -------------------------------------------------------------------------- // ObjectId parts extraction tests. type objectIdParts struct { id bson.ObjectId timestamp int64 machine []byte pid uint16 counter int32 } var objectIds = []objectIdParts{ objectIdParts{ bson.ObjectIdHex("4d88e15b60f486e428412dc9"), 1300816219, []byte{0x60, 0xf4, 0x86}, 0xe428, 4271561, }, objectIdParts{ bson.ObjectIdHex("000000000000000000000000"), 0, []byte{0x00, 0x00, 0x00}, 0x0000, 0, }, objectIdParts{ bson.ObjectIdHex("00000000aabbccddee000001"), 0, []byte{0xaa, 0xbb, 0xcc}, 0xddee, 1, }, } func (s *S) TestObjectIdPartsExtraction(c *C) { for i, v := range objectIds { t := time.Unix(v.timestamp, 0) c.Assert(v.id.Time(), Equals, t, Commentf("#%d Wrong timestamp value", i)) c.Assert(v.id.Machine(), DeepEquals, v.machine, Commentf("#%d Wrong machine id value", i)) c.Assert(v.id.Pid(), Equals, v.pid, Commentf("#%d Wrong pid value", i)) c.Assert(v.id.Counter(), Equals, v.counter, Commentf("#%d Wrong counter value", i)) } } func (s *S) TestNow(c *C) { before := time.Now() time.Sleep(1e6) now := bson.Now() time.Sleep(1e6) after := time.Now() c.Assert(now.After(before) && now.Before(after), Equals, true, Commentf("now=%s, before=%s, after=%s", now, before, after)) } // -------------------------------------------------------------------------- // ObjectId generation tests. func (s *S) TestNewObjectId(c *C) { // Generate 10 ids ids := make([]bson.ObjectId, 10) for i := 0; i < 10; i++ { ids[i] = bson.NewObjectId() } for i := 1; i < 10; i++ { prevId := ids[i-1] id := ids[i] // Test for uniqueness among all other 9 generated ids for j, tid := range ids { if j != i { c.Assert(id, Not(Equals), tid, Commentf("Generated ObjectId is not unique")) } } // Check that timestamp was incremented and is within 30 seconds of the previous one secs := id.Time().Sub(prevId.Time()).Seconds() c.Assert((secs >= 0 && secs <= 30), Equals, true, Commentf("Wrong timestamp in generated ObjectId")) // Check that machine ids are the same c.Assert(id.Machine(), DeepEquals, prevId.Machine()) // Check that pids are the same c.Assert(id.Pid(), Equals, prevId.Pid()) // Test for proper increment delta := int(id.Counter() - prevId.Counter()) c.Assert(delta, Equals, 1, Commentf("Wrong increment in generated ObjectId")) } } func (s *S) TestNewObjectIdWithTime(c *C) { t := time.Unix(12345678, 0) id := bson.NewObjectIdWithTime(t) c.Assert(id.Time(), Equals, t) c.Assert(id.Machine(), DeepEquals, []byte{0x00, 0x00, 0x00}) c.Assert(int(id.Pid()), Equals, 0) c.Assert(int(id.Counter()), Equals, 0) } // -------------------------------------------------------------------------- // ObjectId JSON marshalling. type jsonType struct { Id bson.ObjectId } var jsonIdTests = []struct { value jsonType json string marshal bool unmarshal bool error string }{{ value: jsonType{Id: bson.ObjectIdHex("4d88e15b60f486e428412dc9")}, json: `{"Id":"4d88e15b60f486e428412dc9"}`, marshal: true, unmarshal: true, }, { value: jsonType{}, json: `{"Id":""}`, marshal: true, unmarshal: true, }, { value: jsonType{}, json: `{"Id":null}`, marshal: false, unmarshal: true, }, { json: `{"Id":"4d88e15b60f486e428412dc9A"}`, error: `invalid ObjectId in JSON: "4d88e15b60f486e428412dc9A"`, marshal: false, unmarshal: true, }, { json: `{"Id":"4d88e15b60f486e428412dcZ"}`, error: `invalid ObjectId in JSON: "4d88e15b60f486e428412dcZ" .*`, marshal: false, unmarshal: true, }} func (s *S) TestObjectIdJSONMarshaling(c *C) { for _, test := range jsonIdTests { if test.marshal { data, err := json.Marshal(&test.value) if test.error == "" { c.Assert(err, IsNil) c.Assert(string(data), Equals, test.json) } else { c.Assert(err, ErrorMatches, test.error) } } if test.unmarshal { var value jsonType err := json.Unmarshal([]byte(test.json), &value) if test.error == "" { c.Assert(err, IsNil) c.Assert(value, DeepEquals, test.value) } else { c.Assert(err, ErrorMatches, test.error) } } } } // -------------------------------------------------------------------------- // Spec tests type specTest struct { Description string Documents []struct { Decoded map[string]interface{} Encoded string DecodeOnly bool `yaml:"decodeOnly"` Error interface{} } } func (s *S) TestSpecTests(c *C) { for _, data := range specTests { var test specTest err := yaml.Unmarshal([]byte(data), &test) c.Assert(err, IsNil) c.Logf("Running spec test set %q", test.Description) for _, doc := range test.Documents { if doc.Error != nil { continue } c.Logf("Ensuring %q decodes as %v", doc.Encoded, doc.Decoded) var decoded map[string]interface{} encoded, err := hex.DecodeString(doc.Encoded) c.Assert(err, IsNil) err = bson.Unmarshal(encoded, &decoded) c.Assert(err, IsNil) c.Assert(decoded, DeepEquals, doc.Decoded) } for _, doc := range test.Documents { if doc.DecodeOnly || doc.Error != nil { continue } c.Logf("Ensuring %v encodes as %q", doc.Decoded, doc.Encoded) encoded, err := bson.Marshal(doc.Decoded) c.Assert(err, IsNil) c.Assert(strings.ToUpper(hex.EncodeToString(encoded)), Equals, doc.Encoded) } for _, doc := range test.Documents { if doc.Error == nil { continue } c.Logf("Ensuring %q errors when decoded: %s", doc.Encoded, doc.Error) var decoded map[string]interface{} encoded, err := hex.DecodeString(doc.Encoded) c.Assert(err, IsNil) err = bson.Unmarshal(encoded, &decoded) c.Assert(err, NotNil) c.Logf("Failed with: %v", err) } } } // -------------------------------------------------------------------------- // ObjectId Text encoding.TextUnmarshaler. var textIdTests = []struct { value bson.ObjectId text string marshal bool unmarshal bool error string }{{ value: bson.ObjectIdHex("4d88e15b60f486e428412dc9"), text: "4d88e15b60f486e428412dc9", marshal: true, unmarshal: true, }, { text: "", marshal: true, unmarshal: true, }, { text: "4d88e15b60f486e428412dc9A", marshal: false, unmarshal: true, error: `invalid ObjectId: 4d88e15b60f486e428412dc9A`, }, { text: "4d88e15b60f486e428412dcZ", marshal: false, unmarshal: true, error: `invalid ObjectId: 4d88e15b60f486e428412dcZ .*`, }} func (s *S) TestObjectIdTextMarshaling(c *C) { for _, test := range textIdTests { if test.marshal { data, err := test.value.MarshalText() if test.error == "" { c.Assert(err, IsNil) c.Assert(string(data), Equals, test.text) } else { c.Assert(err, ErrorMatches, test.error) } } if test.unmarshal { err := test.value.UnmarshalText([]byte(test.text)) if test.error == "" { c.Assert(err, IsNil) if test.value != "" { value := bson.ObjectIdHex(test.text) c.Assert(value, DeepEquals, test.value) } } else { c.Assert(err, ErrorMatches, test.error) } } } } // -------------------------------------------------------------------------- // ObjectId XML marshalling. type xmlType struct { Id bson.ObjectId } var xmlIdTests = []struct { value xmlType xml string marshal bool unmarshal bool error string }{{ value: xmlType{Id: bson.ObjectIdHex("4d88e15b60f486e428412dc9")}, xml: "4d88e15b60f486e428412dc9", marshal: true, unmarshal: true, }, { value: xmlType{}, xml: "", marshal: true, unmarshal: true, }, { xml: "4d88e15b60f486e428412dc9A", marshal: false, unmarshal: true, error: `invalid ObjectId: 4d88e15b60f486e428412dc9A`, }, { xml: "4d88e15b60f486e428412dcZ", marshal: false, unmarshal: true, error: `invalid ObjectId: 4d88e15b60f486e428412dcZ .*`, }} func (s *S) TestObjectIdXMLMarshaling(c *C) { for _, test := range xmlIdTests { if test.marshal { data, err := xml.Marshal(&test.value) if test.error == "" { c.Assert(err, IsNil) c.Assert(string(data), Equals, test.xml) } else { c.Assert(err, ErrorMatches, test.error) } } if test.unmarshal { var value xmlType err := xml.Unmarshal([]byte(test.xml), &value) if test.error == "" { c.Assert(err, IsNil) c.Assert(value, DeepEquals, test.value) } else { c.Assert(err, ErrorMatches, test.error) } } } } // -------------------------------------------------------------------------- // Some simple benchmarks. type BenchT struct { A, B, C, D, E, F string } type BenchRawT struct { A string B int C bson.M D []float64 } func (s *S) BenchmarkUnmarhsalStruct(c *C) { v := BenchT{A: "A", D: "D", E: "E"} data, err := bson.Marshal(&v) if err != nil { panic(err) } c.ResetTimer() for i := 0; i < c.N; i++ { err = bson.Unmarshal(data, &v) } if err != nil { panic(err) } } func (s *S) BenchmarkUnmarhsalMap(c *C) { m := bson.M{"a": "a", "d": "d", "e": "e"} data, err := bson.Marshal(&m) if err != nil { panic(err) } c.ResetTimer() for i := 0; i < c.N; i++ { err = bson.Unmarshal(data, &m) } if err != nil { panic(err) } } func (s *S) BenchmarkUnmarshalRaw(c *C) { var err error m := BenchRawT{ A: "test_string", B: 123, C: bson.M{ "subdoc_int": 12312, "subdoc_doc": bson.M{"1": 1}, }, D: []float64{0.0, 1.3333, -99.9997, 3.1415}, } data, err := bson.Marshal(&m) if err != nil { panic(err) } raw := bson.Raw{} c.ResetTimer() for i := 0; i < c.N; i++ { err = bson.Unmarshal(data, &raw) } if err != nil { panic(err) } } func (s *S) BenchmarkNewObjectId(c *C) { for i := 0; i < c.N; i++ { bson.NewObjectId() } } mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/bson/decimal.go000066400000000000000000000154761276133741700240630ustar00rootroot00000000000000// BSON library for Go // // Copyright (c) 2010-2012 - Gustavo Niemeyer // // All rights reserved. // // Redistribution and use in source and binary forms, with or without // modification, are permitted provided that the following conditions are met: // // 1. Redistributions of source code must retain the above copyright notice, this // list of conditions and the following disclaimer. // 2. Redistributions in binary form must reproduce the above copyright notice, // this list of conditions and the following disclaimer in the documentation // and/or other materials provided with the distribution. // // THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND // ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED // WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE // DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR // ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES // (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; // LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND // ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT // (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS // SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. package bson import ( "fmt" "strconv" "strings" ) // Decimal128 holds decimal128 BSON values. type Decimal128 struct { h, l uint64 } func (d Decimal128) String() string { var pos int // positive sign var e int // exponent var h, l uint64 // significand high/low if d.h>>63&1 == 0 { pos = 1 } switch d.h >> 58 & (1<<5 - 1) { case 0x1F: return "NaN" case 0x1E: return "-Inf"[pos:] } l = d.l if d.h>>61&3 == 3 { // Bits: 1*sign 2*ignored 14*exponent 111*significand. // Implicit 0b100 prefix in significand. e = int(d.h>>47&(1<<14-1)) - 6176 //h = 4<<47 | d.h&(1<<47-1) // Spec says all of these values are out of range. h, l = 0, 0 } else { // Bits: 1*sign 14*exponent 113*significand e = int(d.h>>49&(1<<14-1)) - 6176 h = d.h & (1<<49 - 1) } // Would be handled by the logic below, but that's trivial and common. if h == 0 && l == 0 && e == 0 { return "-0"[pos:] } var repr [48]byte // Loop 5 times over 9 digits plus dot, negative sign, and leading zero. var last = len(repr) var i = len(repr) var dot = len(repr) + e var rem uint32 Loop: for d9 := 0; d9 < 5; d9++ { h, l, rem = divmod(h, l, 1e9) for d1 := 0; d1 < 9; d1++ { // Handle "-0.0", "0.00123400", "-1.00E-6", "1.050E+3", etc. if i < len(repr) && (dot == i || l == 0 && h == 0 && rem > 0 && rem < 10 && (dot < i-6 || e > 0)) { e += len(repr) - i i-- repr[i] = '.' last = i - 1 dot = len(repr) // Unmark. } c := '0' + byte(rem%10) rem /= 10 i-- repr[i] = c // Handle "0E+3", "1E+3", etc. if l == 0 && h == 0 && rem == 0 && i == len(repr)-1 && (dot < i-5 || e > 0) { last = i break Loop } if c != '0' { last = i } // Break early. Works without it, but why. if dot > i && l == 0 && h == 0 && rem == 0 { break Loop } } } repr[last-1] = '-' last-- if e > 0 { return string(repr[last+pos:]) + "E+" + strconv.Itoa(e) } if e < 0 { return string(repr[last+pos:]) + "E" + strconv.Itoa(e) } return string(repr[last+pos:]) } func divmod(h, l uint64, div uint32) (qh, ql uint64, rem uint32) { div64 := uint64(div) a := h >> 32 aq := a / div64 ar := a % div64 b := ar<<32 + h&(1<<32-1) bq := b / div64 br := b % div64 c := br<<32 + l>>32 cq := c / div64 cr := c % div64 d := cr<<32 + l&(1<<32-1) dq := d / div64 dr := d % div64 return (aq<<32 | bq), (cq<<32 | dq), uint32(dr) } var dNaN = Decimal128{0x1F << 58, 0} var dPosInf = Decimal128{0x1E << 58, 0} var dNegInf = Decimal128{0x3E << 58, 0} func dErr(s string) (Decimal128, error) { return dNaN, fmt.Errorf("cannot parse %q as a decimal128", s) } func ParseDecimal128(s string) (Decimal128, error) { orig := s if s == "" { return dErr(orig) } neg := s[0] == '-' if neg || s[0] == '+' { s = s[1:] } if (len(s) == 3 || len(s) == 8) && (s[0] == 'N' || s[0] == 'n' || s[0] == 'I' || s[0] == 'i') { if s == "NaN" || s == "nan" || strings.EqualFold(s, "nan") { return dNaN, nil } if s == "Inf" || s == "inf" || strings.EqualFold(s, "inf") || strings.EqualFold(s, "infinity") { if neg { return dNegInf, nil } return dPosInf, nil } return dErr(orig) } var h, l uint64 var e int var add, ovr uint32 var mul uint32 = 1 var dot = -1 var digits = 0 var i = 0 for i < len(s) { c := s[i] if mul == 1e9 { h, l, ovr = muladd(h, l, mul, add) mul, add = 1, 0 if ovr > 0 || h&((1<<15-1)<<49) > 0 { return dErr(orig) } } if c >= '0' && c <= '9' { i++ if c > '0' || digits > 0 { digits++ } if digits > 34 { if c == '0' { // Exact rounding. e++ continue } return dErr(orig) } mul *= 10 add *= 10 add += uint32(c - '0') continue } if c == '.' { i++ if dot >= 0 || i == 1 && len(s) == 1 { return dErr(orig) } if i == len(s) { break } if s[i] < '0' || s[i] > '9' || e > 0 { return dErr(orig) } dot = i continue } break } if i == 0 { return dErr(orig) } if mul > 1 { h, l, ovr = muladd(h, l, mul, add) if ovr > 0 || h&((1<<15-1)<<49) > 0 { return dErr(orig) } } if dot >= 0 { e += dot - i } if i+1 < len(s) && (s[i] == 'E' || s[i] == 'e') { i++ eneg := s[i] == '-' if eneg || s[i] == '+' { i++ if i == len(s) { return dErr(orig) } } n := 0 for i < len(s) && n < 1e4 { c := s[i] i++ if c < '0' || c > '9' { return dErr(orig) } n *= 10 n += int(c - '0') } if eneg { n = -n } e += n for e < -6176 { // Subnormal. var div uint32 = 1 for div < 1e9 && e < -6176 { div *= 10 e++ } var rem uint32 h, l, rem = divmod(h, l, div) if rem > 0 { return dErr(orig) } } for e > 6111 { // Clamped. var mul uint32 = 1 for mul < 1e9 && e > 6111 { mul *= 10 e-- } h, l, ovr = muladd(h, l, mul, 0) if ovr > 0 || h&((1<<15-1)<<49) > 0 { return dErr(orig) } } if e < -6176 || e > 6111 { return dErr(orig) } } if i < len(s) { return dErr(orig) } h |= uint64(e+6176) & uint64(1<<14-1) << 49 if neg { h |= 1 << 63 } return Decimal128{h, l}, nil } func muladd(h, l uint64, mul uint32, add uint32) (resh, resl uint64, overflow uint32) { mul64 := uint64(mul) a := mul64 * (l & (1<<32 - 1)) b := a>>32 + mul64*(l>>32) c := b>>32 + mul64*(h&(1<<32-1)) d := c>>32 + mul64*(h>>32) a = a&(1<<32-1) + uint64(add) b = b&(1<<32-1) + a>>32 c = c&(1<<32-1) + b>>32 d = d&(1<<32-1) + c>>32 return (d<<32 | c&(1<<32-1)), (b<<32 | a&(1<<32-1)), uint32(d >> 32) } mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/bson/decimal_test.go000066400000000000000000005556061276133741700251260ustar00rootroot00000000000000// BSON library for Go // // Copyright (c) 2010-2012 - Gustavo Niemeyer // // All rights reserved. // // Redistribution and use in source and binary forms, with or without // modification, are permitted provided that the following conditions are met: // // 1. Redistributions of source code must retain the above copyright notice, this // list of conditions and the following disclaimer. // 2. Redistributions in binary form must reproduce the above copyright notice, // this list of conditions and the following disclaimer in the documentation // and/or other materials provided with the distribution. // // THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND // ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED // WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE // DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR // ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES // (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; // LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND // ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT // (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS // SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. package bson_test import ( "encoding/hex" "encoding/json" "fmt" "regexp" "strings" "gopkg.in/mgo.v2/bson" . "gopkg.in/check.v1" ) // -------------------------------------------------------------------------- // Decimal tests type decimalTests struct { Valid []struct { Description string `json:"description"` BSON string `json:"bson"` CanonicalBSON string `json:"canonical_bson"` ExtJSON string `json:"extjson"` CanonicalExtJSON string `json:"canonical_extjson"` Lossy bool `json:"lossy"` } `json:"valid"` ParseErrors []struct { Description string `json:"description"` String string `json:"string"` } `json:"parseErrors"` } func extJSONRepr(s string) string { var value struct { D struct { Repr string `json:"$numberDecimal"` } `json:"d"` } err := json.Unmarshal([]byte(s), &value) if err != nil { panic(err) } return value.D.Repr } func (s *S) TestDecimalTests(c *C) { // These also conform to the spec and are used by Go elsewhere. // (e.g. math/big won't parse "Infinity"). goStr := func(s string) string { switch s { case "Infinity": return "Inf" case "-Infinity": return "-Inf" } return s } for _, testEntry := range decimalTestsJSON { testFile := testEntry.file var tests decimalTests err := json.Unmarshal([]byte(testEntry.json), &tests) c.Assert(err, IsNil) for _, test := range tests.Valid { c.Logf("Running %s test: %s", testFile, test.Description) test.BSON = strings.ToLower(test.BSON) // Unmarshal value from BSON data. bsonData, err := hex.DecodeString(test.BSON) var bsonValue struct{ D interface{} } err = bson.Unmarshal(bsonData, &bsonValue) c.Assert(err, IsNil) dec128, ok := bsonValue.D.(bson.Decimal128) c.Assert(ok, Equals, true) // Extract ExtJSON representations (canonical and not). extjRepr := extJSONRepr(test.ExtJSON) cextjRepr := extjRepr if test.CanonicalExtJSON != "" { cextjRepr = extJSONRepr(test.CanonicalExtJSON) } wantRepr := goStr(cextjRepr) // Generate canonical representation. c.Assert(dec128.String(), Equals, wantRepr) // Parse original canonical representation. parsed, err := bson.ParseDecimal128(cextjRepr) c.Assert(err, IsNil) c.Assert(parsed.String(), Equals, wantRepr) // Parse non-canonical representation. parsed, err = bson.ParseDecimal128(extjRepr) c.Assert(err, IsNil) c.Assert(parsed.String(), Equals, wantRepr) // Parse Go canonical representation (Inf vs. Infinity). parsed, err = bson.ParseDecimal128(wantRepr) c.Assert(err, IsNil) c.Assert(parsed.String(), Equals, wantRepr) // Marshal original value back into BSON data. data, err := bson.Marshal(bsonValue) c.Assert(err, IsNil) c.Assert(hex.EncodeToString(data), Equals, test.BSON) if test.Lossy { continue } // Marshal the parsed canonical representation. var parsedValue struct{ D interface{} } parsedValue.D = parsed data, err = bson.Marshal(parsedValue) c.Assert(err, IsNil) c.Assert(hex.EncodeToString(data), Equals, test.BSON) } for _, test := range tests.ParseErrors { c.Logf("Running %s parse error test: %s (string %q)", testFile, test.Description, test.String) _, err := bson.ParseDecimal128(test.String) quoted := regexp.QuoteMeta(fmt.Sprintf("%q", test.String)) c.Assert(err, ErrorMatches, `cannot parse `+quoted+` as a decimal128`) } } } const decBenchNum = "9.999999999999999999999999999999999E+6144" func (s *S) BenchmarkDecimal128String(c *C) { d, err := bson.ParseDecimal128(decBenchNum) c.Assert(err, IsNil) c.Assert(d.String(), Equals, decBenchNum) c.ResetTimer() for i := 0; i < c.N; i++ { d.String() } } func (s *S) BenchmarkDecimal128Parse(c *C) { var err error c.ResetTimer() for i := 0; i < c.N; i++ { _, err = bson.ParseDecimal128(decBenchNum) } if err != nil { panic(err) } } var decimalTestsJSON = []struct{ file, json string }{ {"decimal128-1.json", ` { "description": "Decimal128", "bson_type": "0x13", "test_key": "d", "valid": [ { "description": "Special - Canonical NaN", "bson": "180000001364000000000000000000000000000000007C00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"NaN\"}}" }, { "description": "Special - Negative NaN", "bson": "18000000136400000000000000000000000000000000FC00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"NaN\"}}", "lossy": true }, { "description": "Special - Negative NaN", "bson": "18000000136400000000000000000000000000000000FC00", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"NaN\"}}", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-NaN\"}}", "lossy": true }, { "description": "Special - Canonical SNaN", "bson": "180000001364000000000000000000000000000000007E00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"NaN\"}}", "lossy": true }, { "description": "Special - Negative SNaN", "bson": "18000000136400000000000000000000000000000000FE00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"NaN\"}}", "lossy": true }, { "description": "Special - NaN with a payload", "bson": "180000001364001200000000000000000000000000007E00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"NaN\"}}", "lossy": true }, { "description": "Special - Canonical Positive Infinity", "bson": "180000001364000000000000000000000000000000007800", "extjson": "{\"d\" : {\"$numberDecimal\" : \"Infinity\"}}" }, { "description": "Special - Canonical Negative Infinity", "bson": "18000000136400000000000000000000000000000000F800", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-Infinity\"}}" }, { "description": "Special - Invalid representation treated as 0", "bson": "180000001364000000000000000000000000000000106C00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0\"}}", "lossy": true }, { "description": "Special - Invalid representation treated as -0", "bson": "18000000136400DCBA9876543210DEADBEEF00000010EC00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0\"}}", "lossy": true }, { "description": "Special - Invalid representation treated as 0E3", "bson": "18000000136400FFFFFFFFFFFFFFFFFFFFFFFFFFFF116C00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0E+3\"}}", "lossy": true }, { "description": "Regular - Adjusted Exponent Limit", "bson": "18000000136400F2AF967ED05C82DE3297FF6FDE3CF22F00", "extjson": "{\"d\": { \"$numberDecimal\": \"0.000001234567890123456789012345678901234\" }}" }, { "description": "Regular - Smallest", "bson": "18000000136400D204000000000000000000000000343000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.001234\"}}" }, { "description": "Regular - Smallest with Trailing Zeros", "bson": "1800000013640040EF5A07000000000000000000002A3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.00123400000\"}}" }, { "description": "Regular - 0.1", "bson": "1800000013640001000000000000000000000000003E3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.1\"}}" }, { "description": "Regular - 0.1234567890123456789012345678901234", "bson": "18000000136400F2AF967ED05C82DE3297FF6FDE3CFC2F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.1234567890123456789012345678901234\"}}" }, { "description": "Regular - 0", "bson": "180000001364000000000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0\"}}" }, { "description": "Regular - -0", "bson": "18000000136400000000000000000000000000000040B000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0\"}}" }, { "description": "Regular - -0.0", "bson": "1800000013640000000000000000000000000000003EB000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0.0\"}}" }, { "description": "Regular - 2", "bson": "180000001364000200000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"2\"}}" }, { "description": "Regular - 2.000", "bson": "18000000136400D0070000000000000000000000003A3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"2.000\"}}" }, { "description": "Regular - Largest", "bson": "18000000136400F2AF967ED05C82DE3297FF6FDE3C403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1234567890123456789012345678901234\"}}" }, { "description": "Scientific - Tiniest", "bson": "18000000136400FFFFFFFF638E8D37C087ADBE09ED010000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"9.999999999999999999999999999999999E-6143\"}}" }, { "description": "Scientific - Tiny", "bson": "180000001364000100000000000000000000000000000000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E-6176\"}}" }, { "description": "Scientific - Negative Tiny", "bson": "180000001364000100000000000000000000000000008000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-1E-6176\"}}" }, { "description": "Scientific - Adjusted Exponent Limit", "bson": "18000000136400F2AF967ED05C82DE3297FF6FDE3CF02F00", "extjson": "{\"d\": { \"$numberDecimal\": \"1.234567890123456789012345678901234E-7\" }}" }, { "description": "Scientific - Fractional", "bson": "1800000013640064000000000000000000000000002CB000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-1.00E-8\"}}" }, { "description": "Scientific - 0 with Exponent", "bson": "180000001364000000000000000000000000000000205F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0E+6000\"}}" }, { "description": "Scientific - 0 with Negative Exponent", "bson": "1800000013640000000000000000000000000000007A2B00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0E-611\"}}" }, { "description": "Scientific - No Decimal with Signed Exponent", "bson": "180000001364000100000000000000000000000000463000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+3\"}}" }, { "description": "Scientific - Trailing Zero", "bson": "180000001364001A04000000000000000000000000423000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.050E+4\"}}" }, { "description": "Scientific - With Decimal", "bson": "180000001364006900000000000000000000000000423000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.05E+3\"}}" }, { "description": "Scientific - Full", "bson": "18000000136400FFFFFFFFFFFFFFFFFFFFFFFFFFFF403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"5192296858534827628530496329220095\"}}" }, { "description": "Scientific - Large", "bson": "18000000136400000000000A5BC138938D44C64D31FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.000000000000000000000000000000000E+6144\"}}" }, { "description": "Scientific - Largest", "bson": "18000000136400FFFFFFFF638E8D37C087ADBE09EDFF5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"9.999999999999999999999999999999999E+6144\"}}" }, { "description": "Non-Canonical Parsing - Exponent Normalization", "bson": "1800000013640064000000000000000000000000002CB000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-100E-10\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"-1.00E-8\"}}" }, { "description": "Non-Canonical Parsing - Unsigned Positive Exponent", "bson": "180000001364000100000000000000000000000000463000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E3\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+3\"}}" }, { "description": "Non-Canonical Parsing - Lowercase Exponent Identifier", "bson": "180000001364000100000000000000000000000000463000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1e+3\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+3\"}}" }, { "description": "Non-Canonical Parsing - Long Significand with Exponent", "bson": "1800000013640079D9E0F9763ADA429D0200000000583000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"12345689012345789012345E+12\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.2345689012345789012345E+34\"}}" }, { "description": "Non-Canonical Parsing - Positive Sign", "bson": "18000000136400F2AF967ED05C82DE3297FF6FDE3C403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"+1234567890123456789012345678901234\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1234567890123456789012345678901234\"}}" }, { "description": "Non-Canonical Parsing - Long Decimal String", "bson": "180000001364000100000000000000000000000000722800", "extjson": "{\"d\" : {\"$numberDecimal\" : \".000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1E-999\"}}" }, { "description": "Non-Canonical Parsing - nan", "bson": "180000001364000000000000000000000000000000007C00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"nan\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"NaN\"}}" }, { "description": "Non-Canonical Parsing - nAn", "bson": "180000001364000000000000000000000000000000007C00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"nAn\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"NaN\"}}" }, { "description": "Non-Canonical Parsing - +infinity", "bson": "180000001364000000000000000000000000000000007800", "extjson": "{\"d\" : {\"$numberDecimal\" : \"+infinity\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"Infinity\"}}" }, { "description": "Non-Canonical Parsing - infinity", "bson": "180000001364000000000000000000000000000000007800", "extjson": "{\"d\" : {\"$numberDecimal\" : \"infinity\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"Infinity\"}}" }, { "description": "Non-Canonical Parsing - infiniTY", "bson": "180000001364000000000000000000000000000000007800", "extjson": "{\"d\" : {\"$numberDecimal\" : \"infiniTY\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"Infinity\"}}" }, { "description": "Non-Canonical Parsing - inf", "bson": "180000001364000000000000000000000000000000007800", "extjson": "{\"d\" : {\"$numberDecimal\" : \"inf\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"Infinity\"}}" }, { "description": "Non-Canonical Parsing - inF", "bson": "180000001364000000000000000000000000000000007800", "extjson": "{\"d\" : {\"$numberDecimal\" : \"inF\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"Infinity\"}}" }, { "description": "Non-Canonical Parsing - -infinity", "bson": "18000000136400000000000000000000000000000000F800", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-infinity\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"-Infinity\"}}" }, { "description": "Non-Canonical Parsing - -infiniTy", "bson": "18000000136400000000000000000000000000000000F800", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-infiniTy\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"-Infinity\"}}" }, { "description": "Non-Canonical Parsing - -Inf", "bson": "18000000136400000000000000000000000000000000F800", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-Infinity\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"-Infinity\"}}" }, { "description": "Non-Canonical Parsing - -inf", "bson": "18000000136400000000000000000000000000000000F800", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-inf\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"-Infinity\"}}" }, { "description": "Non-Canonical Parsing - -inF", "bson": "18000000136400000000000000000000000000000000F800", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-inF\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"-Infinity\"}}" }, { "description": "Rounded Subnormal number", "bson": "180000001364000100000000000000000000000000000000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"10E-6177\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1E-6176\"}}" }, { "description": "Clamped", "bson": "180000001364000a00000000000000000000000000fe5f00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E6112\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0E+6112\"}}" }, { "description": "Exact rounding", "bson": "18000000136400000000000a5bc138938d44c64d31cc3700", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.000000000000000000000000000000000E+999\"}}" } ] } `}, {"decimal128-2.json", ` { "description": "Decimal128", "bson_type": "0x13", "test_key": "d", "valid": [ { "description": "[decq021] Normality", "bson": "18000000136400F2AF967ED05C82DE3297FF6FDE3C40B000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-1234567890123456789012345678901234\"}}" }, { "description": "[decq823] values around [u]int32 edges (zeros done earlier)", "bson": "18000000136400010000800000000000000000000040B000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-2147483649\"}}" }, { "description": "[decq822] values around [u]int32 edges (zeros done earlier)", "bson": "18000000136400000000800000000000000000000040B000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-2147483648\"}}" }, { "description": "[decq821] values around [u]int32 edges (zeros done earlier)", "bson": "18000000136400FFFFFF7F0000000000000000000040B000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-2147483647\"}}" }, { "description": "[decq820] values around [u]int32 edges (zeros done earlier)", "bson": "18000000136400FEFFFF7F0000000000000000000040B000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-2147483646\"}}" }, { "description": "[decq152] fold-downs (more below)", "bson": "18000000136400393000000000000000000000000040B000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-12345\"}}" }, { "description": "[decq154] fold-downs (more below)", "bson": "18000000136400D20400000000000000000000000040B000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-1234\"}}" }, { "description": "[decq006] derivative canonical plain strings", "bson": "18000000136400EE0200000000000000000000000040B000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-750\"}}" }, { "description": "[decq164] fold-downs (more below)", "bson": "1800000013640039300000000000000000000000003CB000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-123.45\"}}" }, { "description": "[decq156] fold-downs (more below)", "bson": "180000001364007B0000000000000000000000000040B000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-123\"}}" }, { "description": "[decq008] derivative canonical plain strings", "bson": "18000000136400EE020000000000000000000000003EB000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-75.0\"}}" }, { "description": "[decq158] fold-downs (more below)", "bson": "180000001364000C0000000000000000000000000040B000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-12\"}}" }, { "description": "[decq122] Nmax and similar", "bson": "18000000136400FFFFFFFF638E8D37C087ADBE09EDFFDF00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-9.999999999999999999999999999999999E+6144\"}}" }, { "description": "[decq002] (mostly derived from the Strawman 4 document and examples)", "bson": "18000000136400EE020000000000000000000000003CB000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-7.50\"}}" }, { "description": "[decq004] derivative canonical plain strings", "bson": "18000000136400EE0200000000000000000000000042B000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-7.50E+3\"}}" }, { "description": "[decq018] derivative canonical plain strings", "bson": "18000000136400EE020000000000000000000000002EB000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-7.50E-7\"}}" }, { "description": "[decq125] Nmax and similar", "bson": "18000000136400F2AF967ED05C82DE3297FF6FDE3CFEDF00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-1.234567890123456789012345678901234E+6144\"}}" }, { "description": "[decq131] fold-downs (more below)", "bson": "18000000136400000000807F1BCF85B27059C8A43CFEDF00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-1.230000000000000000000000000000000E+6144\"}}" }, { "description": "[decq162] fold-downs (more below)", "bson": "180000001364007B000000000000000000000000003CB000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-1.23\"}}" }, { "description": "[decq176] Nmin and below", "bson": "18000000136400010000000A5BC138938D44C64D31008000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-1.000000000000000000000000000000001E-6143\"}}" }, { "description": "[decq174] Nmin and below", "bson": "18000000136400000000000A5BC138938D44C64D31008000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-1.000000000000000000000000000000000E-6143\"}}" }, { "description": "[decq133] fold-downs (more below)", "bson": "18000000136400000000000A5BC138938D44C64D31FEDF00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-1.000000000000000000000000000000000E+6144\"}}" }, { "description": "[decq160] fold-downs (more below)", "bson": "18000000136400010000000000000000000000000040B000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-1\"}}" }, { "description": "[decq172] Nmin and below", "bson": "180000001364000100000000000000000000000000428000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-1E-6143\"}}" }, { "description": "[decq010] derivative canonical plain strings", "bson": "18000000136400EE020000000000000000000000003AB000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0.750\"}}" }, { "description": "[decq012] derivative canonical plain strings", "bson": "18000000136400EE0200000000000000000000000038B000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0.0750\"}}" }, { "description": "[decq014] derivative canonical plain strings", "bson": "18000000136400EE0200000000000000000000000034B000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0.000750\"}}" }, { "description": "[decq016] derivative canonical plain strings", "bson": "18000000136400EE0200000000000000000000000030B000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0.00000750\"}}" }, { "description": "[decq404] zeros", "bson": "180000001364000000000000000000000000000000000000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0E-6176\"}}" }, { "description": "[decq424] negative zeros", "bson": "180000001364000000000000000000000000000000008000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0E-6176\"}}" }, { "description": "[decq407] zeros", "bson": "1800000013640000000000000000000000000000003C3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.00\"}}" }, { "description": "[decq427] negative zeros", "bson": "1800000013640000000000000000000000000000003CB000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0.00\"}}" }, { "description": "[decq409] zeros", "bson": "180000001364000000000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0\"}}" }, { "description": "[decq428] negative zeros", "bson": "18000000136400000000000000000000000000000040B000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0\"}}" }, { "description": "[decq700] Selected DPD codes", "bson": "180000001364000000000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0\"}}" }, { "description": "[decq406] zeros", "bson": "1800000013640000000000000000000000000000003C3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.00\"}}" }, { "description": "[decq426] negative zeros", "bson": "1800000013640000000000000000000000000000003CB000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0.00\"}}" }, { "description": "[decq410] zeros", "bson": "180000001364000000000000000000000000000000463000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0E+3\"}}" }, { "description": "[decq431] negative zeros", "bson": "18000000136400000000000000000000000000000046B000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0E+3\"}}" }, { "description": "[decq419] clamped zeros...", "bson": "180000001364000000000000000000000000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0E+6111\"}}" }, { "description": "[decq432] negative zeros", "bson": "180000001364000000000000000000000000000000FEDF00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0E+6111\"}}" }, { "description": "[decq405] zeros", "bson": "180000001364000000000000000000000000000000000000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0E-6176\"}}" }, { "description": "[decq425] negative zeros", "bson": "180000001364000000000000000000000000000000008000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0E-6176\"}}" }, { "description": "[decq508] Specials", "bson": "180000001364000000000000000000000000000000007800", "extjson": "{\"d\" : {\"$numberDecimal\" : \"Infinity\"}}" }, { "description": "[decq528] Specials", "bson": "18000000136400000000000000000000000000000000F800", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-Infinity\"}}" }, { "description": "[decq541] Specials", "bson": "180000001364000000000000000000000000000000007C00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"NaN\"}}" }, { "description": "[decq074] Nmin and below", "bson": "18000000136400000000000A5BC138938D44C64D31000000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.000000000000000000000000000000000E-6143\"}}" }, { "description": "[decq602] fold-down full sequence", "bson": "18000000136400000000000A5BC138938D44C64D31FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.000000000000000000000000000000000E+6144\"}}" }, { "description": "[decq604] fold-down full sequence", "bson": "180000001364000000000081EFAC855B416D2DEE04FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.00000000000000000000000000000000E+6143\"}}" }, { "description": "[decq606] fold-down full sequence", "bson": "1800000013640000000080264B91C02220BE377E00FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0000000000000000000000000000000E+6142\"}}" }, { "description": "[decq608] fold-down full sequence", "bson": "1800000013640000000040EAED7446D09C2C9F0C00FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.000000000000000000000000000000E+6141\"}}" }, { "description": "[decq610] fold-down full sequence", "bson": "18000000136400000000A0CA17726DAE0F1E430100FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.00000000000000000000000000000E+6140\"}}" }, { "description": "[decq612] fold-down full sequence", "bson": "18000000136400000000106102253E5ECE4F200000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0000000000000000000000000000E+6139\"}}" }, { "description": "[decq614] fold-down full sequence", "bson": "18000000136400000000E83C80D09F3C2E3B030000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.000000000000000000000000000E+6138\"}}" }, { "description": "[decq616] fold-down full sequence", "bson": "18000000136400000000E4D20CC8DCD2B752000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.00000000000000000000000000E+6137\"}}" }, { "description": "[decq618] fold-down full sequence", "bson": "180000001364000000004A48011416954508000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0000000000000000000000000E+6136\"}}" }, { "description": "[decq620] fold-down full sequence", "bson": "18000000136400000000A1EDCCCE1BC2D300000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.000000000000000000000000E+6135\"}}" }, { "description": "[decq622] fold-down full sequence", "bson": "18000000136400000080F64AE1C7022D1500000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.00000000000000000000000E+6134\"}}" }, { "description": "[decq624] fold-down full sequence", "bson": "18000000136400000040B2BAC9E0191E0200000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0000000000000000000000E+6133\"}}" }, { "description": "[decq626] fold-down full sequence", "bson": "180000001364000000A0DEC5ADC935360000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.000000000000000000000E+6132\"}}" }, { "description": "[decq628] fold-down full sequence", "bson": "18000000136400000010632D5EC76B050000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.00000000000000000000E+6131\"}}" }, { "description": "[decq630] fold-down full sequence", "bson": "180000001364000000E8890423C78A000000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0000000000000000000E+6130\"}}" }, { "description": "[decq632] fold-down full sequence", "bson": "18000000136400000064A7B3B6E00D000000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.000000000000000000E+6129\"}}" }, { "description": "[decq634] fold-down full sequence", "bson": "1800000013640000008A5D78456301000000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.00000000000000000E+6128\"}}" }, { "description": "[decq636] fold-down full sequence", "bson": "180000001364000000C16FF2862300000000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0000000000000000E+6127\"}}" }, { "description": "[decq638] fold-down full sequence", "bson": "180000001364000080C6A47E8D0300000000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.000000000000000E+6126\"}}" }, { "description": "[decq640] fold-down full sequence", "bson": "1800000013640000407A10F35A0000000000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.00000000000000E+6125\"}}" }, { "description": "[decq642] fold-down full sequence", "bson": "1800000013640000A0724E18090000000000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0000000000000E+6124\"}}" }, { "description": "[decq644] fold-down full sequence", "bson": "180000001364000010A5D4E8000000000000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.000000000000E+6123\"}}" }, { "description": "[decq646] fold-down full sequence", "bson": "1800000013640000E8764817000000000000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.00000000000E+6122\"}}" }, { "description": "[decq648] fold-down full sequence", "bson": "1800000013640000E40B5402000000000000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0000000000E+6121\"}}" }, { "description": "[decq650] fold-down full sequence", "bson": "1800000013640000CA9A3B00000000000000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.000000000E+6120\"}}" }, { "description": "[decq652] fold-down full sequence", "bson": "1800000013640000E1F50500000000000000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.00000000E+6119\"}}" }, { "description": "[decq654] fold-down full sequence", "bson": "180000001364008096980000000000000000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0000000E+6118\"}}" }, { "description": "[decq656] fold-down full sequence", "bson": "1800000013640040420F0000000000000000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.000000E+6117\"}}" }, { "description": "[decq658] fold-down full sequence", "bson": "18000000136400A086010000000000000000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.00000E+6116\"}}" }, { "description": "[decq660] fold-down full sequence", "bson": "180000001364001027000000000000000000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0000E+6115\"}}" }, { "description": "[decq662] fold-down full sequence", "bson": "18000000136400E803000000000000000000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.000E+6114\"}}" }, { "description": "[decq664] fold-down full sequence", "bson": "180000001364006400000000000000000000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.00E+6113\"}}" }, { "description": "[decq666] fold-down full sequence", "bson": "180000001364000A00000000000000000000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0E+6112\"}}" }, { "description": "[decq060] fold-downs (more below)", "bson": "180000001364000100000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1\"}}" }, { "description": "[decq670] fold-down full sequence", "bson": "180000001364000100000000000000000000000000FC5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+6110\"}}" }, { "description": "[decq668] fold-down full sequence", "bson": "180000001364000100000000000000000000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+6111\"}}" }, { "description": "[decq072] Nmin and below", "bson": "180000001364000100000000000000000000000000420000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E-6143\"}}" }, { "description": "[decq076] Nmin and below", "bson": "18000000136400010000000A5BC138938D44C64D31000000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.000000000000000000000000000000001E-6143\"}}" }, { "description": "[decq036] fold-downs (more below)", "bson": "18000000136400000000807F1BCF85B27059C8A43CFE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.230000000000000000000000000000000E+6144\"}}" }, { "description": "[decq062] fold-downs (more below)", "bson": "180000001364007B000000000000000000000000003C3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.23\"}}" }, { "description": "[decq034] Nmax and similar", "bson": "18000000136400F2AF967ED05C82DE3297FF6FDE3CFE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.234567890123456789012345678901234E+6144\"}}" }, { "description": "[decq441] exponent lengths", "bson": "180000001364000700000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"7\"}}" }, { "description": "[decq449] exponent lengths", "bson": "1800000013640007000000000000000000000000001E5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"7E+5999\"}}" }, { "description": "[decq447] exponent lengths", "bson": "1800000013640007000000000000000000000000000E3800", "extjson": "{\"d\" : {\"$numberDecimal\" : \"7E+999\"}}" }, { "description": "[decq445] exponent lengths", "bson": "180000001364000700000000000000000000000000063100", "extjson": "{\"d\" : {\"$numberDecimal\" : \"7E+99\"}}" }, { "description": "[decq443] exponent lengths", "bson": "180000001364000700000000000000000000000000523000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"7E+9\"}}" }, { "description": "[decq842] VG testcase", "bson": "180000001364000000FED83F4E7C9FE4E269E38A5BCD1700", "extjson": "{\"d\" : {\"$numberDecimal\" : \"7.049000000000010795488000000000000E-3097\"}}" }, { "description": "[decq841] VG testcase", "bson": "180000001364000000203B9DB5056F000000000000002400", "extjson": "{\"d\" : {\"$numberDecimal\" : \"8.000000000000000000E-1550\"}}" }, { "description": "[decq840] VG testcase", "bson": "180000001364003C17258419D710C42F0000000000002400", "extjson": "{\"d\" : {\"$numberDecimal\" : \"8.81125000000001349436E-1548\"}}" }, { "description": "[decq701] Selected DPD codes", "bson": "180000001364000900000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"9\"}}" }, { "description": "[decq032] Nmax and similar", "bson": "18000000136400FFFFFFFF638E8D37C087ADBE09EDFF5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"9.999999999999999999999999999999999E+6144\"}}" }, { "description": "[decq702] Selected DPD codes", "bson": "180000001364000A00000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"10\"}}" }, { "description": "[decq057] fold-downs (more below)", "bson": "180000001364000C00000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"12\"}}" }, { "description": "[decq703] Selected DPD codes", "bson": "180000001364001300000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"19\"}}" }, { "description": "[decq704] Selected DPD codes", "bson": "180000001364001400000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"20\"}}" }, { "description": "[decq705] Selected DPD codes", "bson": "180000001364001D00000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"29\"}}" }, { "description": "[decq706] Selected DPD codes", "bson": "180000001364001E00000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"30\"}}" }, { "description": "[decq707] Selected DPD codes", "bson": "180000001364002700000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"39\"}}" }, { "description": "[decq708] Selected DPD codes", "bson": "180000001364002800000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"40\"}}" }, { "description": "[decq709] Selected DPD codes", "bson": "180000001364003100000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"49\"}}" }, { "description": "[decq710] Selected DPD codes", "bson": "180000001364003200000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"50\"}}" }, { "description": "[decq711] Selected DPD codes", "bson": "180000001364003B00000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"59\"}}" }, { "description": "[decq712] Selected DPD codes", "bson": "180000001364003C00000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"60\"}}" }, { "description": "[decq713] Selected DPD codes", "bson": "180000001364004500000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"69\"}}" }, { "description": "[decq714] Selected DPD codes", "bson": "180000001364004600000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"70\"}}" }, { "description": "[decq715] Selected DPD codes", "bson": "180000001364004700000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"71\"}}" }, { "description": "[decq716] Selected DPD codes", "bson": "180000001364004800000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"72\"}}" }, { "description": "[decq717] Selected DPD codes", "bson": "180000001364004900000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"73\"}}" }, { "description": "[decq718] Selected DPD codes", "bson": "180000001364004A00000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"74\"}}" }, { "description": "[decq719] Selected DPD codes", "bson": "180000001364004B00000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"75\"}}" }, { "description": "[decq720] Selected DPD codes", "bson": "180000001364004C00000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"76\"}}" }, { "description": "[decq721] Selected DPD codes", "bson": "180000001364004D00000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"77\"}}" }, { "description": "[decq722] Selected DPD codes", "bson": "180000001364004E00000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"78\"}}" }, { "description": "[decq723] Selected DPD codes", "bson": "180000001364004F00000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"79\"}}" }, { "description": "[decq056] fold-downs (more below)", "bson": "180000001364007B00000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"123\"}}" }, { "description": "[decq064] fold-downs (more below)", "bson": "1800000013640039300000000000000000000000003C3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"123.45\"}}" }, { "description": "[decq732] Selected DPD codes", "bson": "180000001364000802000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"520\"}}" }, { "description": "[decq733] Selected DPD codes", "bson": "180000001364000902000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"521\"}}" }, { "description": "[decq740] DPD: one of each of the huffman groups", "bson": "180000001364000903000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"777\"}}" }, { "description": "[decq741] DPD: one of each of the huffman groups", "bson": "180000001364000A03000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"778\"}}" }, { "description": "[decq742] DPD: one of each of the huffman groups", "bson": "180000001364001303000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"787\"}}" }, { "description": "[decq746] DPD: one of each of the huffman groups", "bson": "180000001364001F03000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"799\"}}" }, { "description": "[decq743] DPD: one of each of the huffman groups", "bson": "180000001364006D03000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"877\"}}" }, { "description": "[decq753] DPD all-highs cases (includes the 24 redundant codes)", "bson": "180000001364007803000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"888\"}}" }, { "description": "[decq754] DPD all-highs cases (includes the 24 redundant codes)", "bson": "180000001364007903000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"889\"}}" }, { "description": "[decq760] DPD all-highs cases (includes the 24 redundant codes)", "bson": "180000001364008203000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"898\"}}" }, { "description": "[decq764] DPD all-highs cases (includes the 24 redundant codes)", "bson": "180000001364008303000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"899\"}}" }, { "description": "[decq745] DPD: one of each of the huffman groups", "bson": "18000000136400D303000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"979\"}}" }, { "description": "[decq770] DPD all-highs cases (includes the 24 redundant codes)", "bson": "18000000136400DC03000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"988\"}}" }, { "description": "[decq774] DPD all-highs cases (includes the 24 redundant codes)", "bson": "18000000136400DD03000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"989\"}}" }, { "description": "[decq730] Selected DPD codes", "bson": "18000000136400E203000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"994\"}}" }, { "description": "[decq731] Selected DPD codes", "bson": "18000000136400E303000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"995\"}}" }, { "description": "[decq744] DPD: one of each of the huffman groups", "bson": "18000000136400E503000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"997\"}}" }, { "description": "[decq780] DPD all-highs cases (includes the 24 redundant codes)", "bson": "18000000136400E603000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"998\"}}" }, { "description": "[decq787] DPD all-highs cases (includes the 24 redundant codes)", "bson": "18000000136400E703000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"999\"}}" }, { "description": "[decq053] fold-downs (more below)", "bson": "18000000136400D204000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1234\"}}" }, { "description": "[decq052] fold-downs (more below)", "bson": "180000001364003930000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"12345\"}}" }, { "description": "[decq792] Miscellaneous (testers' queries, etc.)", "bson": "180000001364003075000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"30000\"}}" }, { "description": "[decq793] Miscellaneous (testers' queries, etc.)", "bson": "1800000013640090940D0000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"890000\"}}" }, { "description": "[decq824] values around [u]int32 edges (zeros done earlier)", "bson": "18000000136400FEFFFF7F00000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"2147483646\"}}" }, { "description": "[decq825] values around [u]int32 edges (zeros done earlier)", "bson": "18000000136400FFFFFF7F00000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"2147483647\"}}" }, { "description": "[decq826] values around [u]int32 edges (zeros done earlier)", "bson": "180000001364000000008000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"2147483648\"}}" }, { "description": "[decq827] values around [u]int32 edges (zeros done earlier)", "bson": "180000001364000100008000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"2147483649\"}}" }, { "description": "[decq828] values around [u]int32 edges (zeros done earlier)", "bson": "18000000136400FEFFFFFF00000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"4294967294\"}}" }, { "description": "[decq829] values around [u]int32 edges (zeros done earlier)", "bson": "18000000136400FFFFFFFF00000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"4294967295\"}}" }, { "description": "[decq830] values around [u]int32 edges (zeros done earlier)", "bson": "180000001364000000000001000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"4294967296\"}}" }, { "description": "[decq831] values around [u]int32 edges (zeros done earlier)", "bson": "180000001364000100000001000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"4294967297\"}}" }, { "description": "[decq022] Normality", "bson": "18000000136400C7711CC7B548F377DC80A131C836403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1111111111111111111111111111111111\"}}" }, { "description": "[decq020] Normality", "bson": "18000000136400F2AF967ED05C82DE3297FF6FDE3C403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1234567890123456789012345678901234\"}}" }, { "description": "[decq550] Specials", "bson": "18000000136400FFFFFFFF638E8D37C087ADBE09ED413000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"9999999999999999999999999999999999\"}}" } ] } `}, {"decimal128-3.json", ` { "description": "Decimal128", "bson_type": "0x13", "test_key": "d", "valid": [ { "description": "[basx066] strings without E cannot generate E in result", "bson": "18000000136400185C0ACE0000000000000000000038B000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-00345678.5432\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"-345678.5432\"}}" }, { "description": "[basx065] strings without E cannot generate E in result", "bson": "18000000136400185C0ACE0000000000000000000038B000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0345678.5432\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"-345678.5432\"}}" }, { "description": "[basx064] strings without E cannot generate E in result", "bson": "18000000136400185C0ACE0000000000000000000038B000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-345678.5432\"}}" }, { "description": "[basx041] strings without E cannot generate E in result", "bson": "180000001364004C0000000000000000000000000040B000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-76\"}}" }, { "description": "[basx027] conform to rules and exponent will be in permitted range).", "bson": "180000001364000F270000000000000000000000003AB000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-9.999\"}}" }, { "description": "[basx026] conform to rules and exponent will be in permitted range).", "bson": "180000001364009F230000000000000000000000003AB000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-9.119\"}}" }, { "description": "[basx025] conform to rules and exponent will be in permitted range).", "bson": "180000001364008F030000000000000000000000003CB000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-9.11\"}}" }, { "description": "[basx024] conform to rules and exponent will be in permitted range).", "bson": "180000001364005B000000000000000000000000003EB000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-9.1\"}}" }, { "description": "[dqbsr531] negatives (Rounded)", "bson": "1800000013640099761CC7B548F377DC80A131C836FEAF00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-1.1111111111111111111111111111123450\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"-1.111111111111111111111111111112345\"}}" }, { "description": "[basx022] conform to rules and exponent will be in permitted range).", "bson": "180000001364000A000000000000000000000000003EB000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-1.0\"}}" }, { "description": "[basx021] conform to rules and exponent will be in permitted range).", "bson": "18000000136400010000000000000000000000000040B000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-1\"}}" }, { "description": "[basx601] Zeros", "bson": "1800000013640000000000000000000000000000002E3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.000000000\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0E-9\"}}" }, { "description": "[basx622] Zeros", "bson": "1800000013640000000000000000000000000000002EB000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0.000000000\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"-0E-9\"}}" }, { "description": "[basx602] Zeros", "bson": "180000001364000000000000000000000000000000303000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.00000000\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0E-8\"}}" }, { "description": "[basx621] Zeros", "bson": "18000000136400000000000000000000000000000030B000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0.00000000\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"-0E-8\"}}" }, { "description": "[basx603] Zeros", "bson": "180000001364000000000000000000000000000000323000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.0000000\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0E-7\"}}" }, { "description": "[basx620] Zeros", "bson": "18000000136400000000000000000000000000000032B000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0.0000000\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"-0E-7\"}}" }, { "description": "[basx604] Zeros", "bson": "180000001364000000000000000000000000000000343000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.000000\"}}" }, { "description": "[basx619] Zeros", "bson": "18000000136400000000000000000000000000000034B000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0.000000\"}}" }, { "description": "[basx605] Zeros", "bson": "180000001364000000000000000000000000000000363000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.00000\"}}" }, { "description": "[basx618] Zeros", "bson": "18000000136400000000000000000000000000000036B000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0.00000\"}}" }, { "description": "[basx680] Zeros", "bson": "180000001364000000000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"000000.\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0\"}}" }, { "description": "[basx606] Zeros", "bson": "180000001364000000000000000000000000000000383000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.0000\"}}" }, { "description": "[basx617] Zeros", "bson": "18000000136400000000000000000000000000000038B000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0.0000\"}}" }, { "description": "[basx681] Zeros", "bson": "180000001364000000000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"00000.\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0\"}}" }, { "description": "[basx686] Zeros", "bson": "180000001364000000000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"+00000.\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0\"}}" }, { "description": "[basx687] Zeros", "bson": "18000000136400000000000000000000000000000040B000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-00000.\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"-0\"}}" }, { "description": "[basx019] conform to rules and exponent will be in permitted range).", "bson": "1800000013640000000000000000000000000000003CB000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-00.00\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"-0.00\"}}" }, { "description": "[basx607] Zeros", "bson": "1800000013640000000000000000000000000000003A3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.000\"}}" }, { "description": "[basx616] Zeros", "bson": "1800000013640000000000000000000000000000003AB000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0.000\"}}" }, { "description": "[basx682] Zeros", "bson": "180000001364000000000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0000.\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0\"}}" }, { "description": "[basx155] Numbers with E", "bson": "1800000013640000000000000000000000000000003A3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.000e+0\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.000\"}}" }, { "description": "[basx130] Numbers with E", "bson": "180000001364000000000000000000000000000000383000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.000E-1\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.0000\"}}" }, { "description": "[basx290] some more negative zeros [systematic tests below]", "bson": "18000000136400000000000000000000000000000038B000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0.000E-1\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"-0.0000\"}}" }, { "description": "[basx131] Numbers with E", "bson": "180000001364000000000000000000000000000000363000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.000E-2\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.00000\"}}" }, { "description": "[basx291] some more negative zeros [systematic tests below]", "bson": "18000000136400000000000000000000000000000036B000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0.000E-2\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"-0.00000\"}}" }, { "description": "[basx132] Numbers with E", "bson": "180000001364000000000000000000000000000000343000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.000E-3\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.000000\"}}" }, { "description": "[basx292] some more negative zeros [systematic tests below]", "bson": "18000000136400000000000000000000000000000034B000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0.000E-3\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"-0.000000\"}}" }, { "description": "[basx133] Numbers with E", "bson": "180000001364000000000000000000000000000000323000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.000E-4\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0E-7\"}}" }, { "description": "[basx293] some more negative zeros [systematic tests below]", "bson": "18000000136400000000000000000000000000000032B000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0.000E-4\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"-0E-7\"}}" }, { "description": "[basx608] Zeros", "bson": "1800000013640000000000000000000000000000003C3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.00\"}}" }, { "description": "[basx615] Zeros", "bson": "1800000013640000000000000000000000000000003CB000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0.00\"}}" }, { "description": "[basx683] Zeros", "bson": "180000001364000000000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"000.\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0\"}}" }, { "description": "[basx630] Zeros", "bson": "1800000013640000000000000000000000000000003C3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.00E+0\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.00\"}}" }, { "description": "[basx670] Zeros", "bson": "1800000013640000000000000000000000000000003C3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.00E-0\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.00\"}}" }, { "description": "[basx631] Zeros", "bson": "1800000013640000000000000000000000000000003E3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.00E+1\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.0\"}}" }, { "description": "[basx671] Zeros", "bson": "1800000013640000000000000000000000000000003A3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.00E-1\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.000\"}}" }, { "description": "[basx134] Numbers with E", "bson": "180000001364000000000000000000000000000000383000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.00E-2\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.0000\"}}" }, { "description": "[basx294] some more negative zeros [systematic tests below]", "bson": "18000000136400000000000000000000000000000038B000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0.00E-2\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"-0.0000\"}}" }, { "description": "[basx632] Zeros", "bson": "180000001364000000000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.00E+2\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0\"}}" }, { "description": "[basx672] Zeros", "bson": "180000001364000000000000000000000000000000383000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.00E-2\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.0000\"}}" }, { "description": "[basx135] Numbers with E", "bson": "180000001364000000000000000000000000000000363000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.00E-3\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.00000\"}}" }, { "description": "[basx295] some more negative zeros [systematic tests below]", "bson": "18000000136400000000000000000000000000000036B000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0.00E-3\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"-0.00000\"}}" }, { "description": "[basx633] Zeros", "bson": "180000001364000000000000000000000000000000423000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.00E+3\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0E+1\"}}" }, { "description": "[basx673] Zeros", "bson": "180000001364000000000000000000000000000000363000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.00E-3\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.00000\"}}" }, { "description": "[basx136] Numbers with E", "bson": "180000001364000000000000000000000000000000343000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.00E-4\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.000000\"}}" }, { "description": "[basx674] Zeros", "bson": "180000001364000000000000000000000000000000343000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.00E-4\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.000000\"}}" }, { "description": "[basx634] Zeros", "bson": "180000001364000000000000000000000000000000443000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.00E+4\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0E+2\"}}" }, { "description": "[basx137] Numbers with E", "bson": "180000001364000000000000000000000000000000323000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.00E-5\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0E-7\"}}" }, { "description": "[basx635] Zeros", "bson": "180000001364000000000000000000000000000000463000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.00E+5\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0E+3\"}}" }, { "description": "[basx675] Zeros", "bson": "180000001364000000000000000000000000000000323000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.00E-5\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0E-7\"}}" }, { "description": "[basx636] Zeros", "bson": "180000001364000000000000000000000000000000483000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.00E+6\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0E+4\"}}" }, { "description": "[basx676] Zeros", "bson": "180000001364000000000000000000000000000000303000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.00E-6\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0E-8\"}}" }, { "description": "[basx637] Zeros", "bson": "1800000013640000000000000000000000000000004A3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.00E+7\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0E+5\"}}" }, { "description": "[basx677] Zeros", "bson": "1800000013640000000000000000000000000000002E3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.00E-7\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0E-9\"}}" }, { "description": "[basx638] Zeros", "bson": "1800000013640000000000000000000000000000004C3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.00E+8\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0E+6\"}}" }, { "description": "[basx678] Zeros", "bson": "1800000013640000000000000000000000000000002C3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.00E-8\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0E-10\"}}" }, { "description": "[basx149] Numbers with E", "bson": "180000001364000000000000000000000000000000523000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"000E+9\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0E+9\"}}" }, { "description": "[basx639] Zeros", "bson": "1800000013640000000000000000000000000000004E3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.00E+9\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0E+7\"}}" }, { "description": "[basx679] Zeros", "bson": "1800000013640000000000000000000000000000002A3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.00E-9\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0E-11\"}}" }, { "description": "[basx063] strings without E cannot generate E in result", "bson": "18000000136400185C0ACE00000000000000000000383000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"+00345678.5432\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"345678.5432\"}}" }, { "description": "[basx018] conform to rules and exponent will be in permitted range).", "bson": "1800000013640000000000000000000000000000003EB000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0.0\"}}" }, { "description": "[basx609] Zeros", "bson": "1800000013640000000000000000000000000000003E3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.0\"}}" }, { "description": "[basx614] Zeros", "bson": "1800000013640000000000000000000000000000003EB000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0.0\"}}" }, { "description": "[basx684] Zeros", "bson": "180000001364000000000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"00.\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0\"}}" }, { "description": "[basx640] Zeros", "bson": "1800000013640000000000000000000000000000003E3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.0E+0\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.0\"}}" }, { "description": "[basx660] Zeros", "bson": "1800000013640000000000000000000000000000003E3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.0E-0\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.0\"}}" }, { "description": "[basx641] Zeros", "bson": "180000001364000000000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.0E+1\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0\"}}" }, { "description": "[basx661] Zeros", "bson": "1800000013640000000000000000000000000000003C3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.0E-1\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.00\"}}" }, { "description": "[basx296] some more negative zeros [systematic tests below]", "bson": "1800000013640000000000000000000000000000003AB000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0.0E-2\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"-0.000\"}}" }, { "description": "[basx642] Zeros", "bson": "180000001364000000000000000000000000000000423000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.0E+2\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0E+1\"}}" }, { "description": "[basx662] Zeros", "bson": "1800000013640000000000000000000000000000003A3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.0E-2\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.000\"}}" }, { "description": "[basx297] some more negative zeros [systematic tests below]", "bson": "18000000136400000000000000000000000000000038B000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0.0E-3\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"-0.0000\"}}" }, { "description": "[basx643] Zeros", "bson": "180000001364000000000000000000000000000000443000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.0E+3\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0E+2\"}}" }, { "description": "[basx663] Zeros", "bson": "180000001364000000000000000000000000000000383000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.0E-3\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.0000\"}}" }, { "description": "[basx644] Zeros", "bson": "180000001364000000000000000000000000000000463000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.0E+4\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0E+3\"}}" }, { "description": "[basx664] Zeros", "bson": "180000001364000000000000000000000000000000363000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.0E-4\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.00000\"}}" }, { "description": "[basx645] Zeros", "bson": "180000001364000000000000000000000000000000483000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.0E+5\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0E+4\"}}" }, { "description": "[basx665] Zeros", "bson": "180000001364000000000000000000000000000000343000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.0E-5\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.000000\"}}" }, { "description": "[basx646] Zeros", "bson": "1800000013640000000000000000000000000000004A3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.0E+6\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0E+5\"}}" }, { "description": "[basx666] Zeros", "bson": "180000001364000000000000000000000000000000323000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.0E-6\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0E-7\"}}" }, { "description": "[basx647] Zeros", "bson": "1800000013640000000000000000000000000000004C3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.0E+7\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0E+6\"}}" }, { "description": "[basx667] Zeros", "bson": "180000001364000000000000000000000000000000303000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.0E-7\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0E-8\"}}" }, { "description": "[basx648] Zeros", "bson": "1800000013640000000000000000000000000000004E3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.0E+8\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0E+7\"}}" }, { "description": "[basx668] Zeros", "bson": "1800000013640000000000000000000000000000002E3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.0E-8\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0E-9\"}}" }, { "description": "[basx160] Numbers with E", "bson": "180000001364000000000000000000000000000000523000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"00E+9\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0E+9\"}}" }, { "description": "[basx161] Numbers with E", "bson": "1800000013640000000000000000000000000000002E3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"00E-9\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0E-9\"}}" }, { "description": "[basx649] Zeros", "bson": "180000001364000000000000000000000000000000503000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.0E+9\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0E+8\"}}" }, { "description": "[basx669] Zeros", "bson": "1800000013640000000000000000000000000000002C3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.0E-9\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0E-10\"}}" }, { "description": "[basx062] strings without E cannot generate E in result", "bson": "18000000136400185C0ACE00000000000000000000383000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"+0345678.5432\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"345678.5432\"}}" }, { "description": "[basx001] conform to rules and exponent will be in permitted range).", "bson": "180000001364000000000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0\"}}" }, { "description": "[basx017] conform to rules and exponent will be in permitted range).", "bson": "18000000136400000000000000000000000000000040B000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0\"}}" }, { "description": "[basx611] Zeros", "bson": "180000001364000000000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0\"}}" }, { "description": "[basx613] Zeros", "bson": "18000000136400000000000000000000000000000040B000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0.\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"-0\"}}" }, { "description": "[basx685] Zeros", "bson": "180000001364000000000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0\"}}" }, { "description": "[basx688] Zeros", "bson": "180000001364000000000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"+0.\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0\"}}" }, { "description": "[basx689] Zeros", "bson": "18000000136400000000000000000000000000000040B000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0.\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"-0\"}}" }, { "description": "[basx650] Zeros", "bson": "180000001364000000000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0E+0\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0\"}}" }, { "description": "[basx651] Zeros", "bson": "180000001364000000000000000000000000000000423000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0E+1\"}}" }, { "description": "[basx298] some more negative zeros [systematic tests below]", "bson": "1800000013640000000000000000000000000000003CB000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0E-2\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"-0.00\"}}" }, { "description": "[basx652] Zeros", "bson": "180000001364000000000000000000000000000000443000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0E+2\"}}" }, { "description": "[basx299] some more negative zeros [systematic tests below]", "bson": "1800000013640000000000000000000000000000003AB000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0E-3\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"-0.000\"}}" }, { "description": "[basx653] Zeros", "bson": "180000001364000000000000000000000000000000463000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0E+3\"}}" }, { "description": "[basx654] Zeros", "bson": "180000001364000000000000000000000000000000483000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0E+4\"}}" }, { "description": "[basx655] Zeros", "bson": "1800000013640000000000000000000000000000004A3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0E+5\"}}" }, { "description": "[basx656] Zeros", "bson": "1800000013640000000000000000000000000000004C3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0E+6\"}}" }, { "description": "[basx657] Zeros", "bson": "1800000013640000000000000000000000000000004E3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0E+7\"}}" }, { "description": "[basx658] Zeros", "bson": "180000001364000000000000000000000000000000503000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0E+8\"}}" }, { "description": "[basx138] Numbers with E", "bson": "180000001364000000000000000000000000000000523000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"+0E+9\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0E+9\"}}" }, { "description": "[basx139] Numbers with E", "bson": "18000000136400000000000000000000000000000052B000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0E+9\"}}" }, { "description": "[basx144] Numbers with E", "bson": "180000001364000000000000000000000000000000523000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0E+9\"}}" }, { "description": "[basx154] Numbers with E", "bson": "180000001364000000000000000000000000000000523000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0E9\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0E+9\"}}" }, { "description": "[basx659] Zeros", "bson": "180000001364000000000000000000000000000000523000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0E+9\"}}" }, { "description": "[basx042] strings without E cannot generate E in result", "bson": "18000000136400FC040000000000000000000000003C3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"+12.76\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"12.76\"}}" }, { "description": "[basx143] Numbers with E", "bson": "180000001364000100000000000000000000000000523000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"+1E+009\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+9\"}}" }, { "description": "[basx061] strings without E cannot generate E in result", "bson": "18000000136400185C0ACE00000000000000000000383000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"+345678.5432\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"345678.5432\"}}" }, { "description": "[basx036] conform to rules and exponent will be in permitted range).", "bson": "1800000013640015CD5B0700000000000000000000203000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.0000000123456789\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.23456789E-8\"}}" }, { "description": "[basx035] conform to rules and exponent will be in permitted range).", "bson": "1800000013640015CD5B0700000000000000000000223000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.000000123456789\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.23456789E-7\"}}" }, { "description": "[basx034] conform to rules and exponent will be in permitted range).", "bson": "1800000013640015CD5B0700000000000000000000243000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.00000123456789\"}}" }, { "description": "[basx053] strings without E cannot generate E in result", "bson": "180000001364003200000000000000000000000000323000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.0000050\"}}" }, { "description": "[basx033] conform to rules and exponent will be in permitted range).", "bson": "1800000013640015CD5B0700000000000000000000263000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.0000123456789\"}}" }, { "description": "[basx016] conform to rules and exponent will be in permitted range).", "bson": "180000001364000C000000000000000000000000003A3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.012\"}}" }, { "description": "[basx015] conform to rules and exponent will be in permitted range).", "bson": "180000001364007B000000000000000000000000003A3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.123\"}}" }, { "description": "[basx037] conform to rules and exponent will be in permitted range).", "bson": "1800000013640078DF0D8648700000000000000000223000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.123456789012344\"}}" }, { "description": "[basx038] conform to rules and exponent will be in permitted range).", "bson": "1800000013640079DF0D8648700000000000000000223000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.123456789012345\"}}" }, { "description": "[basx250] Numbers with E", "bson": "18000000136400F104000000000000000000000000383000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.1265\"}}" }, { "description": "[basx257] Numbers with E", "bson": "18000000136400F104000000000000000000000000383000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.1265E-0\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.1265\"}}" }, { "description": "[basx256] Numbers with E", "bson": "18000000136400F104000000000000000000000000363000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.1265E-1\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.01265\"}}" }, { "description": "[basx258] Numbers with E", "bson": "18000000136400F1040000000000000000000000003A3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.1265E+1\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.265\"}}" }, { "description": "[basx251] Numbers with E", "bson": "18000000136400F104000000000000000000000000103000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.1265E-20\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.265E-21\"}}" }, { "description": "[basx263] Numbers with E", "bson": "18000000136400F104000000000000000000000000603000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.1265E+20\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.265E+19\"}}" }, { "description": "[basx255] Numbers with E", "bson": "18000000136400F104000000000000000000000000343000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.1265E-2\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.001265\"}}" }, { "description": "[basx259] Numbers with E", "bson": "18000000136400F1040000000000000000000000003C3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.1265E+2\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"12.65\"}}" }, { "description": "[basx254] Numbers with E", "bson": "18000000136400F104000000000000000000000000323000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.1265E-3\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.0001265\"}}" }, { "description": "[basx260] Numbers with E", "bson": "18000000136400F1040000000000000000000000003E3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.1265E+3\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"126.5\"}}" }, { "description": "[basx253] Numbers with E", "bson": "18000000136400F104000000000000000000000000303000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.1265E-4\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.00001265\"}}" }, { "description": "[basx261] Numbers with E", "bson": "18000000136400F104000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.1265E+4\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1265\"}}" }, { "description": "[basx252] Numbers with E", "bson": "18000000136400F104000000000000000000000000283000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.1265E-8\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.265E-9\"}}" }, { "description": "[basx262] Numbers with E", "bson": "18000000136400F104000000000000000000000000483000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.1265E+8\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.265E+7\"}}" }, { "description": "[basx159] Numbers with E", "bson": "1800000013640049000000000000000000000000002E3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.73e-7\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"7.3E-8\"}}" }, { "description": "[basx004] conform to rules and exponent will be in permitted range).", "bson": "1800000013640064000000000000000000000000003C3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.00\"}}" }, { "description": "[basx003] conform to rules and exponent will be in permitted range).", "bson": "180000001364000A000000000000000000000000003E3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0\"}}" }, { "description": "[basx002] conform to rules and exponent will be in permitted range).", "bson": "180000001364000100000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1\"}}" }, { "description": "[basx148] Numbers with E", "bson": "180000001364000100000000000000000000000000523000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+009\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+9\"}}" }, { "description": "[basx153] Numbers with E", "bson": "180000001364000100000000000000000000000000523000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E009\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+9\"}}" }, { "description": "[basx141] Numbers with E", "bson": "180000001364000100000000000000000000000000523000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1e+09\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+9\"}}" }, { "description": "[basx146] Numbers with E", "bson": "180000001364000100000000000000000000000000523000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+09\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+9\"}}" }, { "description": "[basx151] Numbers with E", "bson": "180000001364000100000000000000000000000000523000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1e09\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+9\"}}" }, { "description": "[basx142] Numbers with E", "bson": "180000001364000100000000000000000000000000F43000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+90\"}}" }, { "description": "[basx147] Numbers with E", "bson": "180000001364000100000000000000000000000000F43000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1e+90\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+90\"}}" }, { "description": "[basx152] Numbers with E", "bson": "180000001364000100000000000000000000000000F43000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E90\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+90\"}}" }, { "description": "[basx140] Numbers with E", "bson": "180000001364000100000000000000000000000000523000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+9\"}}" }, { "description": "[basx150] Numbers with E", "bson": "180000001364000100000000000000000000000000523000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E9\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+9\"}}" }, { "description": "[basx014] conform to rules and exponent will be in permitted range).", "bson": "18000000136400D2040000000000000000000000003A3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.234\"}}" }, { "description": "[basx170] Numbers with E", "bson": "18000000136400F1040000000000000000000000003A3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.265\"}}" }, { "description": "[basx177] Numbers with E", "bson": "18000000136400F1040000000000000000000000003A3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.265E-0\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.265\"}}" }, { "description": "[basx176] Numbers with E", "bson": "18000000136400F104000000000000000000000000383000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.265E-1\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.1265\"}}" }, { "description": "[basx178] Numbers with E", "bson": "18000000136400F1040000000000000000000000003C3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.265E+1\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"12.65\"}}" }, { "description": "[basx171] Numbers with E", "bson": "18000000136400F104000000000000000000000000123000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.265E-20\"}}" }, { "description": "[basx183] Numbers with E", "bson": "18000000136400F104000000000000000000000000623000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.265E+20\"}}" }, { "description": "[basx175] Numbers with E", "bson": "18000000136400F104000000000000000000000000363000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.265E-2\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.01265\"}}" }, { "description": "[basx179] Numbers with E", "bson": "18000000136400F1040000000000000000000000003E3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.265E+2\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"126.5\"}}" }, { "description": "[basx174] Numbers with E", "bson": "18000000136400F104000000000000000000000000343000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.265E-3\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.001265\"}}" }, { "description": "[basx180] Numbers with E", "bson": "18000000136400F104000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.265E+3\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1265\"}}" }, { "description": "[basx173] Numbers with E", "bson": "18000000136400F104000000000000000000000000323000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.265E-4\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.0001265\"}}" }, { "description": "[basx181] Numbers with E", "bson": "18000000136400F104000000000000000000000000423000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.265E+4\"}}" }, { "description": "[basx172] Numbers with E", "bson": "18000000136400F1040000000000000000000000002A3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.265E-8\"}}" }, { "description": "[basx182] Numbers with E", "bson": "18000000136400F1040000000000000000000000004A3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.265E+8\"}}" }, { "description": "[basx157] Numbers with E", "bson": "180000001364000400000000000000000000000000523000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"4E+9\"}}" }, { "description": "[basx067] examples", "bson": "180000001364000500000000000000000000000000343000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"5E-6\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.000005\"}}" }, { "description": "[basx069] examples", "bson": "180000001364000500000000000000000000000000323000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"5E-7\"}}" }, { "description": "[basx385] Engineering notation tests", "bson": "180000001364000700000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"7E0\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"7\"}}" }, { "description": "[basx365] Engineering notation tests", "bson": "180000001364000700000000000000000000000000543000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"7E10\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"7E+10\"}}" }, { "description": "[basx405] Engineering notation tests", "bson": "1800000013640007000000000000000000000000002C3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"7E-10\"}}" }, { "description": "[basx363] Engineering notation tests", "bson": "180000001364000700000000000000000000000000563000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"7E11\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"7E+11\"}}" }, { "description": "[basx407] Engineering notation tests", "bson": "1800000013640007000000000000000000000000002A3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"7E-11\"}}" }, { "description": "[basx361] Engineering notation tests", "bson": "180000001364000700000000000000000000000000583000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"7E12\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"7E+12\"}}" }, { "description": "[basx409] Engineering notation tests", "bson": "180000001364000700000000000000000000000000283000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"7E-12\"}}" }, { "description": "[basx411] Engineering notation tests", "bson": "180000001364000700000000000000000000000000263000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"7E-13\"}}" }, { "description": "[basx383] Engineering notation tests", "bson": "180000001364000700000000000000000000000000423000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"7E1\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"7E+1\"}}" }, { "description": "[basx387] Engineering notation tests", "bson": "1800000013640007000000000000000000000000003E3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"7E-1\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.7\"}}" }, { "description": "[basx381] Engineering notation tests", "bson": "180000001364000700000000000000000000000000443000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"7E2\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"7E+2\"}}" }, { "description": "[basx389] Engineering notation tests", "bson": "1800000013640007000000000000000000000000003C3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"7E-2\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.07\"}}" }, { "description": "[basx379] Engineering notation tests", "bson": "180000001364000700000000000000000000000000463000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"7E3\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"7E+3\"}}" }, { "description": "[basx391] Engineering notation tests", "bson": "1800000013640007000000000000000000000000003A3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"7E-3\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.007\"}}" }, { "description": "[basx377] Engineering notation tests", "bson": "180000001364000700000000000000000000000000483000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"7E4\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"7E+4\"}}" }, { "description": "[basx393] Engineering notation tests", "bson": "180000001364000700000000000000000000000000383000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"7E-4\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.0007\"}}" }, { "description": "[basx375] Engineering notation tests", "bson": "1800000013640007000000000000000000000000004A3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"7E5\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"7E+5\"}}" }, { "description": "[basx395] Engineering notation tests", "bson": "180000001364000700000000000000000000000000363000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"7E-5\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.00007\"}}" }, { "description": "[basx373] Engineering notation tests", "bson": "1800000013640007000000000000000000000000004C3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"7E6\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"7E+6\"}}" }, { "description": "[basx397] Engineering notation tests", "bson": "180000001364000700000000000000000000000000343000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"7E-6\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.000007\"}}" }, { "description": "[basx371] Engineering notation tests", "bson": "1800000013640007000000000000000000000000004E3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"7E7\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"7E+7\"}}" }, { "description": "[basx399] Engineering notation tests", "bson": "180000001364000700000000000000000000000000323000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"7E-7\"}}" }, { "description": "[basx369] Engineering notation tests", "bson": "180000001364000700000000000000000000000000503000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"7E8\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"7E+8\"}}" }, { "description": "[basx401] Engineering notation tests", "bson": "180000001364000700000000000000000000000000303000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"7E-8\"}}" }, { "description": "[basx367] Engineering notation tests", "bson": "180000001364000700000000000000000000000000523000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"7E9\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"7E+9\"}}" }, { "description": "[basx403] Engineering notation tests", "bson": "1800000013640007000000000000000000000000002E3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"7E-9\"}}" }, { "description": "[basx007] conform to rules and exponent will be in permitted range).", "bson": "1800000013640064000000000000000000000000003E3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"10.0\"}}" }, { "description": "[basx005] conform to rules and exponent will be in permitted range).", "bson": "180000001364000A00000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"10\"}}" }, { "description": "[basx165] Numbers with E", "bson": "180000001364000A00000000000000000000000000523000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"10E+009\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0E+10\"}}" }, { "description": "[basx163] Numbers with E", "bson": "180000001364000A00000000000000000000000000523000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"10E+09\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0E+10\"}}" }, { "description": "[basx325] Engineering notation tests", "bson": "180000001364000A00000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"10e0\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"10\"}}" }, { "description": "[basx305] Engineering notation tests", "bson": "180000001364000A00000000000000000000000000543000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"10e10\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0E+11\"}}" }, { "description": "[basx345] Engineering notation tests", "bson": "180000001364000A000000000000000000000000002C3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"10e-10\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0E-9\"}}" }, { "description": "[basx303] Engineering notation tests", "bson": "180000001364000A00000000000000000000000000563000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"10e11\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0E+12\"}}" }, { "description": "[basx347] Engineering notation tests", "bson": "180000001364000A000000000000000000000000002A3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"10e-11\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0E-10\"}}" }, { "description": "[basx301] Engineering notation tests", "bson": "180000001364000A00000000000000000000000000583000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"10e12\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0E+13\"}}" }, { "description": "[basx349] Engineering notation tests", "bson": "180000001364000A00000000000000000000000000283000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"10e-12\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0E-11\"}}" }, { "description": "[basx351] Engineering notation tests", "bson": "180000001364000A00000000000000000000000000263000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"10e-13\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0E-12\"}}" }, { "description": "[basx323] Engineering notation tests", "bson": "180000001364000A00000000000000000000000000423000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"10e1\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0E+2\"}}" }, { "description": "[basx327] Engineering notation tests", "bson": "180000001364000A000000000000000000000000003E3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"10e-1\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0\"}}" }, { "description": "[basx321] Engineering notation tests", "bson": "180000001364000A00000000000000000000000000443000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"10e2\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0E+3\"}}" }, { "description": "[basx329] Engineering notation tests", "bson": "180000001364000A000000000000000000000000003C3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"10e-2\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.10\"}}" }, { "description": "[basx319] Engineering notation tests", "bson": "180000001364000A00000000000000000000000000463000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"10e3\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0E+4\"}}" }, { "description": "[basx331] Engineering notation tests", "bson": "180000001364000A000000000000000000000000003A3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"10e-3\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.010\"}}" }, { "description": "[basx317] Engineering notation tests", "bson": "180000001364000A00000000000000000000000000483000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"10e4\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0E+5\"}}" }, { "description": "[basx333] Engineering notation tests", "bson": "180000001364000A00000000000000000000000000383000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"10e-4\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.0010\"}}" }, { "description": "[basx315] Engineering notation tests", "bson": "180000001364000A000000000000000000000000004A3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"10e5\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0E+6\"}}" }, { "description": "[basx335] Engineering notation tests", "bson": "180000001364000A00000000000000000000000000363000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"10e-5\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.00010\"}}" }, { "description": "[basx313] Engineering notation tests", "bson": "180000001364000A000000000000000000000000004C3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"10e6\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0E+7\"}}" }, { "description": "[basx337] Engineering notation tests", "bson": "180000001364000A00000000000000000000000000343000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"10e-6\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.000010\"}}" }, { "description": "[basx311] Engineering notation tests", "bson": "180000001364000A000000000000000000000000004E3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"10e7\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0E+8\"}}" }, { "description": "[basx339] Engineering notation tests", "bson": "180000001364000A00000000000000000000000000323000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"10e-7\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.0000010\"}}" }, { "description": "[basx309] Engineering notation tests", "bson": "180000001364000A00000000000000000000000000503000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"10e8\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0E+9\"}}" }, { "description": "[basx341] Engineering notation tests", "bson": "180000001364000A00000000000000000000000000303000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"10e-8\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0E-7\"}}" }, { "description": "[basx164] Numbers with E", "bson": "180000001364000A00000000000000000000000000F43000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"10e+90\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0E+91\"}}" }, { "description": "[basx162] Numbers with E", "bson": "180000001364000A00000000000000000000000000523000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"10E+9\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0E+10\"}}" }, { "description": "[basx307] Engineering notation tests", "bson": "180000001364000A00000000000000000000000000523000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"10e9\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0E+10\"}}" }, { "description": "[basx343] Engineering notation tests", "bson": "180000001364000A000000000000000000000000002E3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"10e-9\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0E-8\"}}" }, { "description": "[basx008] conform to rules and exponent will be in permitted range).", "bson": "1800000013640065000000000000000000000000003E3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"10.1\"}}" }, { "description": "[basx009] conform to rules and exponent will be in permitted range).", "bson": "1800000013640068000000000000000000000000003E3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"10.4\"}}" }, { "description": "[basx010] conform to rules and exponent will be in permitted range).", "bson": "1800000013640069000000000000000000000000003E3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"10.5\"}}" }, { "description": "[basx011] conform to rules and exponent will be in permitted range).", "bson": "180000001364006A000000000000000000000000003E3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"10.6\"}}" }, { "description": "[basx012] conform to rules and exponent will be in permitted range).", "bson": "180000001364006D000000000000000000000000003E3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"10.9\"}}" }, { "description": "[basx013] conform to rules and exponent will be in permitted range).", "bson": "180000001364006E000000000000000000000000003E3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"11.0\"}}" }, { "description": "[basx040] strings without E cannot generate E in result", "bson": "180000001364000C00000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"12\"}}" }, { "description": "[basx190] Numbers with E", "bson": "18000000136400F1040000000000000000000000003C3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"12.65\"}}" }, { "description": "[basx197] Numbers with E", "bson": "18000000136400F1040000000000000000000000003C3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"12.65E-0\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"12.65\"}}" }, { "description": "[basx196] Numbers with E", "bson": "18000000136400F1040000000000000000000000003A3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"12.65E-1\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.265\"}}" }, { "description": "[basx198] Numbers with E", "bson": "18000000136400F1040000000000000000000000003E3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"12.65E+1\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"126.5\"}}" }, { "description": "[basx191] Numbers with E", "bson": "18000000136400F104000000000000000000000000143000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"12.65E-20\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.265E-19\"}}" }, { "description": "[basx203] Numbers with E", "bson": "18000000136400F104000000000000000000000000643000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"12.65E+20\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.265E+21\"}}" }, { "description": "[basx195] Numbers with E", "bson": "18000000136400F104000000000000000000000000383000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"12.65E-2\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.1265\"}}" }, { "description": "[basx199] Numbers with E", "bson": "18000000136400F104000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"12.65E+2\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1265\"}}" }, { "description": "[basx194] Numbers with E", "bson": "18000000136400F104000000000000000000000000363000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"12.65E-3\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.01265\"}}" }, { "description": "[basx200] Numbers with E", "bson": "18000000136400F104000000000000000000000000423000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"12.65E+3\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.265E+4\"}}" }, { "description": "[basx193] Numbers with E", "bson": "18000000136400F104000000000000000000000000343000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"12.65E-4\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.001265\"}}" }, { "description": "[basx201] Numbers with E", "bson": "18000000136400F104000000000000000000000000443000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"12.65E+4\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.265E+5\"}}" }, { "description": "[basx192] Numbers with E", "bson": "18000000136400F1040000000000000000000000002C3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"12.65E-8\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.265E-7\"}}" }, { "description": "[basx202] Numbers with E", "bson": "18000000136400F1040000000000000000000000004C3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"12.65E+8\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.265E+9\"}}" }, { "description": "[basx044] strings without E cannot generate E in result", "bson": "18000000136400FC040000000000000000000000003C3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"012.76\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"12.76\"}}" }, { "description": "[basx042] strings without E cannot generate E in result", "bson": "18000000136400FC040000000000000000000000003C3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"12.76\"}}" }, { "description": "[basx046] strings without E cannot generate E in result", "bson": "180000001364001100000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"17.\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"17\"}}" }, { "description": "[basx049] strings without E cannot generate E in result", "bson": "180000001364002C00000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0044\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"44\"}}" }, { "description": "[basx048] strings without E cannot generate E in result", "bson": "180000001364002C00000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"044\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"44\"}}" }, { "description": "[basx158] Numbers with E", "bson": "180000001364002C00000000000000000000000000523000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"44E+9\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"4.4E+10\"}}" }, { "description": "[basx068] examples", "bson": "180000001364003200000000000000000000000000323000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"50E-7\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.0000050\"}}" }, { "description": "[basx169] Numbers with E", "bson": "180000001364006400000000000000000000000000523000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"100e+009\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.00E+11\"}}" }, { "description": "[basx167] Numbers with E", "bson": "180000001364006400000000000000000000000000523000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"100e+09\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.00E+11\"}}" }, { "description": "[basx168] Numbers with E", "bson": "180000001364006400000000000000000000000000F43000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"100E+90\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.00E+92\"}}" }, { "description": "[basx166] Numbers with E", "bson": "180000001364006400000000000000000000000000523000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"100e+9\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.00E+11\"}}" }, { "description": "[basx210] Numbers with E", "bson": "18000000136400F1040000000000000000000000003E3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"126.5\"}}" }, { "description": "[basx217] Numbers with E", "bson": "18000000136400F1040000000000000000000000003E3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"126.5E-0\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"126.5\"}}" }, { "description": "[basx216] Numbers with E", "bson": "18000000136400F1040000000000000000000000003C3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"126.5E-1\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"12.65\"}}" }, { "description": "[basx218] Numbers with E", "bson": "18000000136400F104000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"126.5E+1\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1265\"}}" }, { "description": "[basx211] Numbers with E", "bson": "18000000136400F104000000000000000000000000163000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"126.5E-20\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.265E-18\"}}" }, { "description": "[basx223] Numbers with E", "bson": "18000000136400F104000000000000000000000000663000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"126.5E+20\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.265E+22\"}}" }, { "description": "[basx215] Numbers with E", "bson": "18000000136400F1040000000000000000000000003A3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"126.5E-2\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.265\"}}" }, { "description": "[basx219] Numbers with E", "bson": "18000000136400F104000000000000000000000000423000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"126.5E+2\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.265E+4\"}}" }, { "description": "[basx214] Numbers with E", "bson": "18000000136400F104000000000000000000000000383000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"126.5E-3\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.1265\"}}" }, { "description": "[basx220] Numbers with E", "bson": "18000000136400F104000000000000000000000000443000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"126.5E+3\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.265E+5\"}}" }, { "description": "[basx213] Numbers with E", "bson": "18000000136400F104000000000000000000000000363000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"126.5E-4\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.01265\"}}" }, { "description": "[basx221] Numbers with E", "bson": "18000000136400F104000000000000000000000000463000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"126.5E+4\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.265E+6\"}}" }, { "description": "[basx212] Numbers with E", "bson": "18000000136400F1040000000000000000000000002E3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"126.5E-8\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.000001265\"}}" }, { "description": "[basx222] Numbers with E", "bson": "18000000136400F1040000000000000000000000004E3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"126.5E+8\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.265E+10\"}}" }, { "description": "[basx006] conform to rules and exponent will be in permitted range).", "bson": "18000000136400E803000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1000\"}}" }, { "description": "[basx230] Numbers with E", "bson": "18000000136400F104000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1265\"}}" }, { "description": "[basx237] Numbers with E", "bson": "18000000136400F104000000000000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1265E-0\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1265\"}}" }, { "description": "[basx236] Numbers with E", "bson": "18000000136400F1040000000000000000000000003E3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1265E-1\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"126.5\"}}" }, { "description": "[basx238] Numbers with E", "bson": "18000000136400F104000000000000000000000000423000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1265E+1\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.265E+4\"}}" }, { "description": "[basx231] Numbers with E", "bson": "18000000136400F104000000000000000000000000183000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1265E-20\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.265E-17\"}}" }, { "description": "[basx243] Numbers with E", "bson": "18000000136400F104000000000000000000000000683000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1265E+20\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.265E+23\"}}" }, { "description": "[basx235] Numbers with E", "bson": "18000000136400F1040000000000000000000000003C3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1265E-2\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"12.65\"}}" }, { "description": "[basx239] Numbers with E", "bson": "18000000136400F104000000000000000000000000443000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1265E+2\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.265E+5\"}}" }, { "description": "[basx234] Numbers with E", "bson": "18000000136400F1040000000000000000000000003A3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1265E-3\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.265\"}}" }, { "description": "[basx240] Numbers with E", "bson": "18000000136400F104000000000000000000000000463000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1265E+3\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.265E+6\"}}" }, { "description": "[basx233] Numbers with E", "bson": "18000000136400F104000000000000000000000000383000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1265E-4\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.1265\"}}" }, { "description": "[basx241] Numbers with E", "bson": "18000000136400F104000000000000000000000000483000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1265E+4\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.265E+7\"}}" }, { "description": "[basx232] Numbers with E", "bson": "18000000136400F104000000000000000000000000303000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1265E-8\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.00001265\"}}" }, { "description": "[basx242] Numbers with E", "bson": "18000000136400F104000000000000000000000000503000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1265E+8\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.265E+11\"}}" }, { "description": "[basx060] strings without E cannot generate E in result", "bson": "18000000136400185C0ACE00000000000000000000383000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"345678.5432\"}}" }, { "description": "[basx059] strings without E cannot generate E in result", "bson": "18000000136400F198670C08000000000000000000363000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0345678.54321\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"345678.54321\"}}" }, { "description": "[basx058] strings without E cannot generate E in result", "bson": "180000001364006AF90B7C50000000000000000000343000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"345678.543210\"}}" }, { "description": "[basx057] strings without E cannot generate E in result", "bson": "180000001364006A19562522020000000000000000343000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"2345678.543210\"}}" }, { "description": "[basx056] strings without E cannot generate E in result", "bson": "180000001364006AB9C8733A0B0000000000000000343000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"12345678.543210\"}}" }, { "description": "[basx031] conform to rules and exponent will be in permitted range).", "bson": "1800000013640040AF0D8648700000000000000000343000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"123456789.000000\"}}" }, { "description": "[basx030] conform to rules and exponent will be in permitted range).", "bson": "1800000013640080910F8648700000000000000000343000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"123456789.123456\"}}" }, { "description": "[basx032] conform to rules and exponent will be in permitted range).", "bson": "1800000013640080910F8648700000000000000000403000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"123456789123456\"}}" } ] } `}, {"decimal128-4.json", ` { "description": "Decimal128", "bson_type": "0x13", "test_key": "d", "valid": [ { "description": "[basx023] conform to rules and exponent will be in permitted range).", "bson": "1800000013640001000000000000000000000000003EB000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0.1\"}}" }, { "description": "[basx045] strings without E cannot generate E in result", "bson": "1800000013640003000000000000000000000000003A3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"+0.003\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.003\"}}" }, { "description": "[basx610] Zeros", "bson": "1800000013640000000000000000000000000000003E3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \".0\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.0\"}}" }, { "description": "[basx612] Zeros", "bson": "1800000013640000000000000000000000000000003EB000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-.0\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"-0.0\"}}" }, { "description": "[basx043] strings without E cannot generate E in result", "bson": "18000000136400FC040000000000000000000000003C3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"+12.76\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"12.76\"}}" }, { "description": "[basx055] strings without E cannot generate E in result", "bson": "180000001364000500000000000000000000000000303000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.00000005\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"5E-8\"}}" }, { "description": "[basx054] strings without E cannot generate E in result", "bson": "180000001364000500000000000000000000000000323000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.0000005\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"5E-7\"}}" }, { "description": "[basx052] strings without E cannot generate E in result", "bson": "180000001364000500000000000000000000000000343000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.000005\"}}" }, { "description": "[basx051] strings without E cannot generate E in result", "bson": "180000001364000500000000000000000000000000363000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"00.00005\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.00005\"}}" }, { "description": "[basx050] strings without E cannot generate E in result", "bson": "180000001364000500000000000000000000000000383000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.0005\"}}" }, { "description": "[basx047] strings without E cannot generate E in result", "bson": "1800000013640005000000000000000000000000003E3000", "extjson": "{\"d\" : {\"$numberDecimal\" : \".5\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.5\"}}" }, { "description": "[dqbsr431] check rounding modes heeded (Rounded)", "bson": "1800000013640099761CC7B548F377DC80A131C836FE2F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.1111111111111111111111111111123450\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.111111111111111111111111111112345\"}}" }, { "description": "OK2", "bson": "18000000136400000000000A5BC138938D44C64D31FC2F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \".100000000000000000000000000000000000000000000000000000000000\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0.1000000000000000000000000000000000\"}}" } ], "parseErrors": [ { "description": "[basx564] Near-specials (Conversion_syntax)", "string": "Infi" }, { "description": "[basx565] Near-specials (Conversion_syntax)", "string": "Infin" }, { "description": "[basx566] Near-specials (Conversion_syntax)", "string": "Infini" }, { "description": "[basx567] Near-specials (Conversion_syntax)", "string": "Infinit" }, { "description": "[basx568] Near-specials (Conversion_syntax)", "string": "-Infinit" }, { "description": "[basx590] some baddies with dots and Es and dots and specials (Conversion_syntax)", "string": ".Infinity" }, { "description": "[basx562] Near-specials (Conversion_syntax)", "string": "NaNq" }, { "description": "[basx563] Near-specials (Conversion_syntax)", "string": "NaNs" }, { "description": "[dqbas939] overflow results at different rounding modes (Overflow & Inexact & Rounded)", "string": "-7e10000" }, { "description": "[dqbsr534] negatives (Rounded & Inexact)", "string": "-1.11111111111111111111111111111234650" }, { "description": "[dqbsr535] negatives (Rounded & Inexact)", "string": "-1.11111111111111111111111111111234551" }, { "description": "[dqbsr533] negatives (Rounded & Inexact)", "string": "-1.11111111111111111111111111111234550" }, { "description": "[dqbsr532] negatives (Rounded & Inexact)", "string": "-1.11111111111111111111111111111234549" }, { "description": "[dqbsr432] check rounding modes heeded (Rounded & Inexact)", "string": "1.11111111111111111111111111111234549" }, { "description": "[dqbsr433] check rounding modes heeded (Rounded & Inexact)", "string": "1.11111111111111111111111111111234550" }, { "description": "[dqbsr435] check rounding modes heeded (Rounded & Inexact)", "string": "1.11111111111111111111111111111234551" }, { "description": "[dqbsr434] check rounding modes heeded (Rounded & Inexact)", "string": "1.11111111111111111111111111111234650" }, { "description": "[dqbas938] overflow results at different rounding modes (Overflow & Inexact & Rounded)", "string": "7e10000" }, { "description": "Inexact rounding#1", "string": "100000000000000000000000000000000000000000000000000000000001" }, { "description": "Inexact rounding#2", "string": "1E-6177" } ] } `}, {"decimal128-5.json", ` { "description": "Decimal128", "bson_type": "0x13", "test_key": "d", "valid": [ { "description": "[decq035] fold-downs (more below) (Clamped)", "bson": "18000000136400000000807F1BCF85B27059C8A43CFE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.23E+6144\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.230000000000000000000000000000000E+6144\"}}" }, { "description": "[decq037] fold-downs (more below) (Clamped)", "bson": "18000000136400000000000A5BC138938D44C64D31FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+6144\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.000000000000000000000000000000000E+6144\"}}" }, { "description": "[decq077] Nmin and below (Subnormal)", "bson": "180000001364000000000081EFAC855B416D2DEE04000000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.100000000000000000000000000000000E-6143\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.00000000000000000000000000000000E-6144\"}}" }, { "description": "[decq078] Nmin and below (Subnormal)", "bson": "180000001364000000000081EFAC855B416D2DEE04000000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.00000000000000000000000000000000E-6144\"}}" }, { "description": "[decq079] Nmin and below (Subnormal)", "bson": "180000001364000A00000000000000000000000000000000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.000000000000000000000000000000010E-6143\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0E-6175\"}}" }, { "description": "[decq080] Nmin and below (Subnormal)", "bson": "180000001364000A00000000000000000000000000000000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0E-6175\"}}" }, { "description": "[decq081] Nmin and below (Subnormal)", "bson": "180000001364000100000000000000000000000000020000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.00000000000000000000000000000001E-6143\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1E-6175\"}}" }, { "description": "[decq082] Nmin and below (Subnormal)", "bson": "180000001364000100000000000000000000000000020000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E-6175\"}}" }, { "description": "[decq083] Nmin and below (Subnormal)", "bson": "180000001364000100000000000000000000000000000000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0.000000000000000000000000000000001E-6143\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1E-6176\"}}" }, { "description": "[decq084] Nmin and below (Subnormal)", "bson": "180000001364000100000000000000000000000000000000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E-6176\"}}" }, { "description": "[decq090] underflows cannot be tested for simple copies, check edge cases (Subnormal)", "bson": "180000001364000100000000000000000000000000000000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1e-6176\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1E-6176\"}}" }, { "description": "[decq100] underflows cannot be tested for simple copies, check edge cases (Subnormal)", "bson": "18000000136400FFFFFFFF095BC138938D44C64D31000000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"999999999999999999999999999999999e-6176\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"9.99999999999999999999999999999999E-6144\"}}" }, { "description": "[decq130] fold-downs (more below) (Clamped)", "bson": "18000000136400000000807F1BCF85B27059C8A43CFEDF00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-1.23E+6144\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"-1.230000000000000000000000000000000E+6144\"}}" }, { "description": "[decq132] fold-downs (more below) (Clamped)", "bson": "18000000136400000000000A5BC138938D44C64D31FEDF00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-1E+6144\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"-1.000000000000000000000000000000000E+6144\"}}" }, { "description": "[decq177] Nmin and below (Subnormal)", "bson": "180000001364000000000081EFAC855B416D2DEE04008000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0.100000000000000000000000000000000E-6143\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"-1.00000000000000000000000000000000E-6144\"}}" }, { "description": "[decq178] Nmin and below (Subnormal)", "bson": "180000001364000000000081EFAC855B416D2DEE04008000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-1.00000000000000000000000000000000E-6144\"}}" }, { "description": "[decq179] Nmin and below (Subnormal)", "bson": "180000001364000A00000000000000000000000000008000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0.000000000000000000000000000000010E-6143\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"-1.0E-6175\"}}" }, { "description": "[decq180] Nmin and below (Subnormal)", "bson": "180000001364000A00000000000000000000000000008000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-1.0E-6175\"}}" }, { "description": "[decq181] Nmin and below (Subnormal)", "bson": "180000001364000100000000000000000000000000028000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0.00000000000000000000000000000001E-6143\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"-1E-6175\"}}" }, { "description": "[decq182] Nmin and below (Subnormal)", "bson": "180000001364000100000000000000000000000000028000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-1E-6175\"}}" }, { "description": "[decq183] Nmin and below (Subnormal)", "bson": "180000001364000100000000000000000000000000008000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0.000000000000000000000000000000001E-6143\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"-1E-6176\"}}" }, { "description": "[decq184] Nmin and below (Subnormal)", "bson": "180000001364000100000000000000000000000000008000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-1E-6176\"}}" }, { "description": "[decq190] underflow edge cases (Subnormal)", "bson": "180000001364000100000000000000000000000000008000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-1e-6176\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"-1E-6176\"}}" }, { "description": "[decq200] underflow edge cases (Subnormal)", "bson": "18000000136400FFFFFFFF095BC138938D44C64D31008000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-999999999999999999999999999999999e-6176\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"-9.99999999999999999999999999999999E-6144\"}}" }, { "description": "[decq400] zeros (Clamped)", "bson": "180000001364000000000000000000000000000000000000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0E-8000\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0E-6176\"}}" }, { "description": "[decq401] zeros (Clamped)", "bson": "180000001364000000000000000000000000000000000000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0E-6177\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0E-6176\"}}" }, { "description": "[decq414] clamped zeros... (Clamped)", "bson": "180000001364000000000000000000000000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0E+6112\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0E+6111\"}}" }, { "description": "[decq416] clamped zeros... (Clamped)", "bson": "180000001364000000000000000000000000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0E+6144\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0E+6111\"}}" }, { "description": "[decq418] clamped zeros... (Clamped)", "bson": "180000001364000000000000000000000000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"0E+8000\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"0E+6111\"}}" }, { "description": "[decq420] negative zeros (Clamped)", "bson": "180000001364000000000000000000000000000000008000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0E-8000\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"-0E-6176\"}}" }, { "description": "[decq421] negative zeros (Clamped)", "bson": "180000001364000000000000000000000000000000008000", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0E-6177\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"-0E-6176\"}}" }, { "description": "[decq434] clamped zeros... (Clamped)", "bson": "180000001364000000000000000000000000000000FEDF00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0E+6112\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"-0E+6111\"}}" }, { "description": "[decq436] clamped zeros... (Clamped)", "bson": "180000001364000000000000000000000000000000FEDF00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0E+6144\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"-0E+6111\"}}" }, { "description": "[decq438] clamped zeros... (Clamped)", "bson": "180000001364000000000000000000000000000000FEDF00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"-0E+8000\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"-0E+6111\"}}" }, { "description": "[decq601] fold-down full sequence (Clamped)", "bson": "18000000136400000000000A5BC138938D44C64D31FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+6144\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.000000000000000000000000000000000E+6144\"}}" }, { "description": "[decq603] fold-down full sequence (Clamped)", "bson": "180000001364000000000081EFAC855B416D2DEE04FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+6143\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.00000000000000000000000000000000E+6143\"}}" }, { "description": "[decq605] fold-down full sequence (Clamped)", "bson": "1800000013640000000080264B91C02220BE377E00FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+6142\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0000000000000000000000000000000E+6142\"}}" }, { "description": "[decq607] fold-down full sequence (Clamped)", "bson": "1800000013640000000040EAED7446D09C2C9F0C00FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+6141\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.000000000000000000000000000000E+6141\"}}" }, { "description": "[decq609] fold-down full sequence (Clamped)", "bson": "18000000136400000000A0CA17726DAE0F1E430100FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+6140\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.00000000000000000000000000000E+6140\"}}" }, { "description": "[decq611] fold-down full sequence (Clamped)", "bson": "18000000136400000000106102253E5ECE4F200000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+6139\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0000000000000000000000000000E+6139\"}}" }, { "description": "[decq613] fold-down full sequence (Clamped)", "bson": "18000000136400000000E83C80D09F3C2E3B030000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+6138\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.000000000000000000000000000E+6138\"}}" }, { "description": "[decq615] fold-down full sequence (Clamped)", "bson": "18000000136400000000E4D20CC8DCD2B752000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+6137\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.00000000000000000000000000E+6137\"}}" }, { "description": "[decq617] fold-down full sequence (Clamped)", "bson": "180000001364000000004A48011416954508000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+6136\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0000000000000000000000000E+6136\"}}" }, { "description": "[decq619] fold-down full sequence (Clamped)", "bson": "18000000136400000000A1EDCCCE1BC2D300000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+6135\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.000000000000000000000000E+6135\"}}" }, { "description": "[decq621] fold-down full sequence (Clamped)", "bson": "18000000136400000080F64AE1C7022D1500000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+6134\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.00000000000000000000000E+6134\"}}" }, { "description": "[decq623] fold-down full sequence (Clamped)", "bson": "18000000136400000040B2BAC9E0191E0200000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+6133\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0000000000000000000000E+6133\"}}" }, { "description": "[decq625] fold-down full sequence (Clamped)", "bson": "180000001364000000A0DEC5ADC935360000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+6132\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.000000000000000000000E+6132\"}}" }, { "description": "[decq627] fold-down full sequence (Clamped)", "bson": "18000000136400000010632D5EC76B050000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+6131\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.00000000000000000000E+6131\"}}" }, { "description": "[decq629] fold-down full sequence (Clamped)", "bson": "180000001364000000E8890423C78A000000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+6130\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0000000000000000000E+6130\"}}" }, { "description": "[decq631] fold-down full sequence (Clamped)", "bson": "18000000136400000064A7B3B6E00D000000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+6129\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.000000000000000000E+6129\"}}" }, { "description": "[decq633] fold-down full sequence (Clamped)", "bson": "1800000013640000008A5D78456301000000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+6128\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.00000000000000000E+6128\"}}" }, { "description": "[decq635] fold-down full sequence (Clamped)", "bson": "180000001364000000C16FF2862300000000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+6127\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0000000000000000E+6127\"}}" }, { "description": "[decq637] fold-down full sequence (Clamped)", "bson": "180000001364000080C6A47E8D0300000000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+6126\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.000000000000000E+6126\"}}" }, { "description": "[decq639] fold-down full sequence (Clamped)", "bson": "1800000013640000407A10F35A0000000000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+6125\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.00000000000000E+6125\"}}" }, { "description": "[decq641] fold-down full sequence (Clamped)", "bson": "1800000013640000A0724E18090000000000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+6124\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0000000000000E+6124\"}}" }, { "description": "[decq643] fold-down full sequence (Clamped)", "bson": "180000001364000010A5D4E8000000000000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+6123\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.000000000000E+6123\"}}" }, { "description": "[decq645] fold-down full sequence (Clamped)", "bson": "1800000013640000E8764817000000000000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+6122\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.00000000000E+6122\"}}" }, { "description": "[decq647] fold-down full sequence (Clamped)", "bson": "1800000013640000E40B5402000000000000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+6121\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0000000000E+6121\"}}" }, { "description": "[decq649] fold-down full sequence (Clamped)", "bson": "1800000013640000CA9A3B00000000000000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+6120\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.000000000E+6120\"}}" }, { "description": "[decq651] fold-down full sequence (Clamped)", "bson": "1800000013640000E1F50500000000000000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+6119\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.00000000E+6119\"}}" }, { "description": "[decq653] fold-down full sequence (Clamped)", "bson": "180000001364008096980000000000000000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+6118\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0000000E+6118\"}}" }, { "description": "[decq655] fold-down full sequence (Clamped)", "bson": "1800000013640040420F0000000000000000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+6117\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.000000E+6117\"}}" }, { "description": "[decq657] fold-down full sequence (Clamped)", "bson": "18000000136400A086010000000000000000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+6116\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.00000E+6116\"}}" }, { "description": "[decq659] fold-down full sequence (Clamped)", "bson": "180000001364001027000000000000000000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+6115\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0000E+6115\"}}" }, { "description": "[decq661] fold-down full sequence (Clamped)", "bson": "18000000136400E803000000000000000000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+6114\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.000E+6114\"}}" }, { "description": "[decq663] fold-down full sequence (Clamped)", "bson": "180000001364006400000000000000000000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+6113\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.00E+6113\"}}" }, { "description": "[decq665] fold-down full sequence (Clamped)", "bson": "180000001364000A00000000000000000000000000FE5F00", "extjson": "{\"d\" : {\"$numberDecimal\" : \"1E+6112\"}}", "canonical_extjson": "{\"d\" : {\"$numberDecimal\" : \"1.0E+6112\"}}" } ] } `}, {"decimal128-6.json", ` { "description": "Decimal128", "bson_type": "0x13", "test_key": "d", "parseErrors": [ { "description": "Incomplete Exponent", "string": "1e" }, { "description": "Exponent at the beginning", "string": "E01" }, { "description": "Just a decimal place", "string": "." }, { "description": "2 decimal places", "string": "..3" }, { "description": "2 decimal places", "string": ".13.3" }, { "description": "2 decimal places", "string": "1..3" }, { "description": "2 decimal places", "string": "1.3.4" }, { "description": "2 decimal places", "string": "1.34." }, { "description": "Decimal with no digits", "string": ".e" }, { "description": "2 signs", "string": "+-32.4" }, { "description": "2 signs", "string": "-+32.4" }, { "description": "2 negative signs", "string": "--32.4" }, { "description": "2 negative signs", "string": "-32.-4" }, { "description": "End in negative sign", "string": "32.0-" }, { "description": "2 negative signs", "string": "32.4E--21" }, { "description": "2 negative signs", "string": "32.4E-2-1" }, { "description": "2 signs", "string": "32.4E+-21" }, { "description": "Empty string", "string": "" }, { "description": "leading white space positive number", "string": " 1" }, { "description": "leading white space negative number", "string": " -1" }, { "description": "trailing white space", "string": "1 " }, { "description": "Invalid", "string": "E" }, { "description": "Invalid", "string": "invalid" }, { "description": "Invalid", "string": "i" }, { "description": "Invalid", "string": "in" }, { "description": "Invalid", "string": "-in" }, { "description": "Invalid", "string": "Na" }, { "description": "Invalid", "string": "-Na" }, { "description": "Invalid", "string": "1.23abc" }, { "description": "Invalid", "string": "1.23abcE+02" }, { "description": "Invalid", "string": "1.23E+0aabs2" } ] } `}, {"decimal128-7.json", ` { "description": "Decimal128", "bson_type": "0x13", "test_key": "d", "parseErrors": [ { "description": "[basx572] Near-specials (Conversion_syntax)", "string": "-9Inf" }, { "description": "[basx516] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "-1-" }, { "description": "[basx533] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "0000.." }, { "description": "[basx534] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": ".0000." }, { "description": "[basx535] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "00..00" }, { "description": "[basx569] Near-specials (Conversion_syntax)", "string": "0Inf" }, { "description": "[basx571] Near-specials (Conversion_syntax)", "string": "-0Inf" }, { "description": "[basx575] Near-specials (Conversion_syntax)", "string": "0sNaN" }, { "description": "[basx503] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "++1" }, { "description": "[basx504] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "--1" }, { "description": "[basx505] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "-+1" }, { "description": "[basx506] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "+-1" }, { "description": "[basx510] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": " +1" }, { "description": "[basx513] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": " + 1" }, { "description": "[basx514] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": " - 1" }, { "description": "[basx501] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "." }, { "description": "[basx502] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": ".." }, { "description": "[basx519] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "" }, { "description": "[basx525] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "e100" }, { "description": "[basx549] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "e+1" }, { "description": "[basx577] some baddies with dots and Es and dots and specials (Conversion_syntax)", "string": ".e+1" }, { "description": "[basx578] some baddies with dots and Es and dots and specials (Conversion_syntax)", "string": "+.e+1" }, { "description": "[basx581] some baddies with dots and Es and dots and specials (Conversion_syntax)", "string": "E+1" }, { "description": "[basx582] some baddies with dots and Es and dots and specials (Conversion_syntax)", "string": ".E+1" }, { "description": "[basx583] some baddies with dots and Es and dots and specials (Conversion_syntax)", "string": "+.E+1" }, { "description": "[basx579] some baddies with dots and Es and dots and specials (Conversion_syntax)", "string": "-.e+" }, { "description": "[basx580] some baddies with dots and Es and dots and specials (Conversion_syntax)", "string": "-.e" }, { "description": "[basx584] some baddies with dots and Es and dots and specials (Conversion_syntax)", "string": "-.E+" }, { "description": "[basx585] some baddies with dots and Es and dots and specials (Conversion_syntax)", "string": "-.E" }, { "description": "[basx589] some baddies with dots and Es and dots and specials (Conversion_syntax)", "string": "+.Inf" }, { "description": "[basx586] some baddies with dots and Es and dots and specials (Conversion_syntax)", "string": ".NaN" }, { "description": "[basx587] some baddies with dots and Es and dots and specials (Conversion_syntax)", "string": "-.NaN" }, { "description": "[basx545] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "ONE" }, { "description": "[basx561] Near-specials (Conversion_syntax)", "string": "qNaN" }, { "description": "[basx573] Near-specials (Conversion_syntax)", "string": "-sNa" }, { "description": "[basx588] some baddies with dots and Es and dots and specials (Conversion_syntax)", "string": "+.sNaN" }, { "description": "[basx544] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "ten" }, { "description": "[basx527] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "u0b65" }, { "description": "[basx526] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "u0e5a" }, { "description": "[basx515] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "x" }, { "description": "[basx574] Near-specials (Conversion_syntax)", "string": "xNaN" }, { "description": "[basx530] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": ".123.5" }, { "description": "[basx500] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "1..2" }, { "description": "[basx542] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "1e1.0" }, { "description": "[basx553] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "1E+1.2.3" }, { "description": "[basx543] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "1e123e" }, { "description": "[basx552] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "1E+1.2" }, { "description": "[basx546] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "1e.1" }, { "description": "[basx547] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "1e1." }, { "description": "[basx554] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "1E++1" }, { "description": "[basx555] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "1E--1" }, { "description": "[basx556] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "1E+-1" }, { "description": "[basx557] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "1E-+1" }, { "description": "[basx558] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "1E'1" }, { "description": "[basx559] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "1E\"1" }, { "description": "[basx520] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "1e-" }, { "description": "[basx560] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "1E" }, { "description": "[basx548] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "1ee" }, { "description": "[basx551] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "1.2.1" }, { "description": "[basx550] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "1.23.4" }, { "description": "[basx529] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "1.34.5" }, { "description": "[basx531] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "01.35." }, { "description": "[basx532] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "01.35-" }, { "description": "[basx518] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "3+" }, { "description": "[basx521] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "7e99999a" }, { "description": "[basx570] Near-specials (Conversion_syntax)", "string": "9Inf" }, { "description": "[basx512] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "12 " }, { "description": "[basx517] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "12-" }, { "description": "[basx507] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "12e" }, { "description": "[basx508] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "12e++" }, { "description": "[basx509] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "12f4" }, { "description": "[basx536] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "111e*123" }, { "description": "[basx537] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "111e123-" }, { "description": "[basx540] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "111e1*23" }, { "description": "[basx538] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "111e+12+" }, { "description": "[basx539] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "111e1-3-" }, { "description": "[basx541] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "111E1e+3" }, { "description": "[basx528] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "123,65" }, { "description": "[basx523] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "7e12356789012x" }, { "description": "[basx522] The 'baddies' tests from DiagBigDecimal, plus some new ones (Conversion_syntax)", "string": "7e123567890x" } ] } `}, } mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/bson/decode.go000066400000000000000000000454131276133741700237020ustar00rootroot00000000000000// BSON library for Go // // Copyright (c) 2010-2012 - Gustavo Niemeyer // // All rights reserved. // // Redistribution and use in source and binary forms, with or without // modification, are permitted provided that the following conditions are met: // // 1. Redistributions of source code must retain the above copyright notice, this // list of conditions and the following disclaimer. // 2. Redistributions in binary form must reproduce the above copyright notice, // this list of conditions and the following disclaimer in the documentation // and/or other materials provided with the distribution. // // THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND // ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED // WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE // DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR // ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES // (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; // LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND // ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT // (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS // SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. // gobson - BSON library for Go. package bson import ( "fmt" "math" "net/url" "reflect" "strconv" "sync" "time" ) type decoder struct { in []byte i int docType reflect.Type } var typeM = reflect.TypeOf(M{}) func newDecoder(in []byte) *decoder { return &decoder{in, 0, typeM} } // -------------------------------------------------------------------------- // Some helper functions. func corrupted() { panic("Document is corrupted") } func settableValueOf(i interface{}) reflect.Value { v := reflect.ValueOf(i) sv := reflect.New(v.Type()).Elem() sv.Set(v) return sv } // -------------------------------------------------------------------------- // Unmarshaling of documents. const ( setterUnknown = iota setterNone setterType setterAddr ) var setterStyles map[reflect.Type]int var setterIface reflect.Type var setterMutex sync.RWMutex func init() { var iface Setter setterIface = reflect.TypeOf(&iface).Elem() setterStyles = make(map[reflect.Type]int) } func setterStyle(outt reflect.Type) int { setterMutex.RLock() style := setterStyles[outt] setterMutex.RUnlock() if style == setterUnknown { setterMutex.Lock() defer setterMutex.Unlock() if outt.Implements(setterIface) { setterStyles[outt] = setterType } else if reflect.PtrTo(outt).Implements(setterIface) { setterStyles[outt] = setterAddr } else { setterStyles[outt] = setterNone } style = setterStyles[outt] } return style } func getSetter(outt reflect.Type, out reflect.Value) Setter { style := setterStyle(outt) if style == setterNone { return nil } if style == setterAddr { if !out.CanAddr() { return nil } out = out.Addr() } else if outt.Kind() == reflect.Ptr && out.IsNil() { out.Set(reflect.New(outt.Elem())) } return out.Interface().(Setter) } func clearMap(m reflect.Value) { var none reflect.Value for _, k := range m.MapKeys() { m.SetMapIndex(k, none) } } func (d *decoder) readDocTo(out reflect.Value) { var elemType reflect.Type outt := out.Type() outk := outt.Kind() for { if outk == reflect.Ptr && out.IsNil() { out.Set(reflect.New(outt.Elem())) } if setter := getSetter(outt, out); setter != nil { var raw Raw d.readDocTo(reflect.ValueOf(&raw)) err := setter.SetBSON(raw) if _, ok := err.(*TypeError); err != nil && !ok { panic(err) } return } if outk == reflect.Ptr { out = out.Elem() outt = out.Type() outk = out.Kind() continue } break } var fieldsMap map[string]fieldInfo var inlineMap reflect.Value start := d.i origout := out if outk == reflect.Interface { if d.docType.Kind() == reflect.Map { mv := reflect.MakeMap(d.docType) out.Set(mv) out = mv } else { dv := reflect.New(d.docType).Elem() out.Set(dv) out = dv } outt = out.Type() outk = outt.Kind() } docType := d.docType keyType := typeString convertKey := false switch outk { case reflect.Map: keyType = outt.Key() if keyType.Kind() != reflect.String { panic("BSON map must have string keys. Got: " + outt.String()) } if keyType != typeString { convertKey = true } elemType = outt.Elem() if elemType == typeIface { d.docType = outt } if out.IsNil() { out.Set(reflect.MakeMap(out.Type())) } else if out.Len() > 0 { clearMap(out) } case reflect.Struct: if outt != typeRaw { sinfo, err := getStructInfo(out.Type()) if err != nil { panic(err) } fieldsMap = sinfo.FieldsMap out.Set(sinfo.Zero) if sinfo.InlineMap != -1 { inlineMap = out.Field(sinfo.InlineMap) if !inlineMap.IsNil() && inlineMap.Len() > 0 { clearMap(inlineMap) } elemType = inlineMap.Type().Elem() if elemType == typeIface { d.docType = inlineMap.Type() } } } case reflect.Slice: switch outt.Elem() { case typeDocElem: origout.Set(d.readDocElems(outt)) return case typeRawDocElem: origout.Set(d.readRawDocElems(outt)) return } fallthrough default: panic("Unsupported document type for unmarshalling: " + out.Type().String()) } end := int(d.readInt32()) end += d.i - 4 if end <= d.i || end > len(d.in) || d.in[end-1] != '\x00' { corrupted() } for d.in[d.i] != '\x00' { kind := d.readByte() name := d.readCStr() if d.i >= end { corrupted() } switch outk { case reflect.Map: e := reflect.New(elemType).Elem() if d.readElemTo(e, kind) { k := reflect.ValueOf(name) if convertKey { k = k.Convert(keyType) } out.SetMapIndex(k, e) } case reflect.Struct: if outt == typeRaw { d.dropElem(kind) } else { if info, ok := fieldsMap[name]; ok { if info.Inline == nil { d.readElemTo(out.Field(info.Num), kind) } else { d.readElemTo(out.FieldByIndex(info.Inline), kind) } } else if inlineMap.IsValid() { if inlineMap.IsNil() { inlineMap.Set(reflect.MakeMap(inlineMap.Type())) } e := reflect.New(elemType).Elem() if d.readElemTo(e, kind) { inlineMap.SetMapIndex(reflect.ValueOf(name), e) } } else { d.dropElem(kind) } } case reflect.Slice: } if d.i >= end { corrupted() } } d.i++ // '\x00' if d.i != end { corrupted() } d.docType = docType if outt == typeRaw { out.Set(reflect.ValueOf(Raw{0x03, d.in[start:d.i]})) } } func (d *decoder) readArrayDocTo(out reflect.Value) { end := int(d.readInt32()) end += d.i - 4 if end <= d.i || end > len(d.in) || d.in[end-1] != '\x00' { corrupted() } i := 0 l := out.Len() for d.in[d.i] != '\x00' { if i >= l { panic("Length mismatch on array field") } kind := d.readByte() for d.i < end && d.in[d.i] != '\x00' { d.i++ } if d.i >= end { corrupted() } d.i++ d.readElemTo(out.Index(i), kind) if d.i >= end { corrupted() } i++ } if i != l { panic("Length mismatch on array field") } d.i++ // '\x00' if d.i != end { corrupted() } } func (d *decoder) readSliceDoc(t reflect.Type) interface{} { tmp := make([]reflect.Value, 0, 8) elemType := t.Elem() if elemType == typeRawDocElem { d.dropElem(0x04) return reflect.Zero(t).Interface() } end := int(d.readInt32()) end += d.i - 4 if end <= d.i || end > len(d.in) || d.in[end-1] != '\x00' { corrupted() } for d.in[d.i] != '\x00' { kind := d.readByte() for d.i < end && d.in[d.i] != '\x00' { d.i++ } if d.i >= end { corrupted() } d.i++ e := reflect.New(elemType).Elem() if d.readElemTo(e, kind) { tmp = append(tmp, e) } if d.i >= end { corrupted() } } d.i++ // '\x00' if d.i != end { corrupted() } n := len(tmp) slice := reflect.MakeSlice(t, n, n) for i := 0; i != n; i++ { slice.Index(i).Set(tmp[i]) } return slice.Interface() } var typeSlice = reflect.TypeOf([]interface{}{}) var typeIface = typeSlice.Elem() func (d *decoder) readDocElems(typ reflect.Type) reflect.Value { docType := d.docType d.docType = typ slice := make([]DocElem, 0, 8) d.readDocWith(func(kind byte, name string) { e := DocElem{Name: name} v := reflect.ValueOf(&e.Value) if d.readElemTo(v.Elem(), kind) { slice = append(slice, e) } }) slicev := reflect.New(typ).Elem() slicev.Set(reflect.ValueOf(slice)) d.docType = docType return slicev } func (d *decoder) readRawDocElems(typ reflect.Type) reflect.Value { docType := d.docType d.docType = typ slice := make([]RawDocElem, 0, 8) d.readDocWith(func(kind byte, name string) { e := RawDocElem{Name: name} v := reflect.ValueOf(&e.Value) if d.readElemTo(v.Elem(), kind) { slice = append(slice, e) } }) slicev := reflect.New(typ).Elem() slicev.Set(reflect.ValueOf(slice)) d.docType = docType return slicev } func (d *decoder) readDocWith(f func(kind byte, name string)) { end := int(d.readInt32()) end += d.i - 4 if end <= d.i || end > len(d.in) || d.in[end-1] != '\x00' { corrupted() } for d.in[d.i] != '\x00' { kind := d.readByte() name := d.readCStr() if d.i >= end { corrupted() } f(kind, name) if d.i >= end { corrupted() } } d.i++ // '\x00' if d.i != end { corrupted() } } // -------------------------------------------------------------------------- // Unmarshaling of individual elements within a document. var blackHole = settableValueOf(struct{}{}) func (d *decoder) dropElem(kind byte) { d.readElemTo(blackHole, kind) } // Attempt to decode an element from the document and put it into out. // If the types are not compatible, the returned ok value will be // false and out will be unchanged. func (d *decoder) readElemTo(out reflect.Value, kind byte) (good bool) { start := d.i if kind == 0x03 { // Delegate unmarshaling of documents. outt := out.Type() outk := out.Kind() switch outk { case reflect.Interface, reflect.Ptr, reflect.Struct, reflect.Map: d.readDocTo(out) return true } if setterStyle(outt) != setterNone { d.readDocTo(out) return true } if outk == reflect.Slice { switch outt.Elem() { case typeDocElem: out.Set(d.readDocElems(outt)) case typeRawDocElem: out.Set(d.readRawDocElems(outt)) default: d.readDocTo(blackHole) } return true } d.readDocTo(blackHole) return true } var in interface{} switch kind { case 0x01: // Float64 in = d.readFloat64() case 0x02: // UTF-8 string in = d.readStr() case 0x03: // Document panic("Can't happen. Handled above.") case 0x04: // Array outt := out.Type() if setterStyle(outt) != setterNone { // Skip the value so its data is handed to the setter below. d.dropElem(kind) break } for outt.Kind() == reflect.Ptr { outt = outt.Elem() } switch outt.Kind() { case reflect.Array: d.readArrayDocTo(out) return true case reflect.Slice: in = d.readSliceDoc(outt) default: in = d.readSliceDoc(typeSlice) } case 0x05: // Binary b := d.readBinary() if b.Kind == 0x00 || b.Kind == 0x02 { in = b.Data } else { in = b } case 0x06: // Undefined (obsolete, but still seen in the wild) in = Undefined case 0x07: // ObjectId in = ObjectId(d.readBytes(12)) case 0x08: // Bool in = d.readBool() case 0x09: // Timestamp // MongoDB handles timestamps as milliseconds. i := d.readInt64() if i == -62135596800000 { in = time.Time{} // In UTC for convenience. } else { in = time.Unix(i/1e3, i%1e3*1e6) } case 0x0A: // Nil in = nil case 0x0B: // RegEx in = d.readRegEx() case 0x0C: in = DBPointer{Namespace: d.readStr(), Id: ObjectId(d.readBytes(12))} case 0x0D: // JavaScript without scope in = JavaScript{Code: d.readStr()} case 0x0E: // Symbol in = Symbol(d.readStr()) case 0x0F: // JavaScript with scope d.i += 4 // Skip length js := JavaScript{d.readStr(), make(M)} d.readDocTo(reflect.ValueOf(js.Scope)) in = js case 0x10: // Int32 in = int(d.readInt32()) case 0x11: // Mongo-specific timestamp in = MongoTimestamp(d.readInt64()) case 0x12: // Int64 in = d.readInt64() case 0x13: // Decimal128 in = Decimal128{ l: uint64(d.readInt64()), h: uint64(d.readInt64()), } case 0x7F: // Max key in = MaxKey case 0xFF: // Min key in = MinKey default: panic(fmt.Sprintf("Unknown element kind (0x%02X)", kind)) } outt := out.Type() if outt == typeRaw { out.Set(reflect.ValueOf(Raw{kind, d.in[start:d.i]})) return true } if setter := getSetter(outt, out); setter != nil { err := setter.SetBSON(Raw{kind, d.in[start:d.i]}) if err == SetZero { out.Set(reflect.Zero(outt)) return true } if err == nil { return true } if _, ok := err.(*TypeError); !ok { panic(err) } return false } if in == nil { out.Set(reflect.Zero(outt)) return true } outk := outt.Kind() // Dereference and initialize pointer if necessary. first := true for outk == reflect.Ptr { if !out.IsNil() { out = out.Elem() } else { elem := reflect.New(outt.Elem()) if first { // Only set if value is compatible. first = false defer func(out, elem reflect.Value) { if good { out.Set(elem) } }(out, elem) } else { out.Set(elem) } out = elem } outt = out.Type() outk = outt.Kind() } inv := reflect.ValueOf(in) if outt == inv.Type() { out.Set(inv) return true } switch outk { case reflect.Interface: out.Set(inv) return true case reflect.String: switch inv.Kind() { case reflect.String: out.SetString(inv.String()) return true case reflect.Slice: if b, ok := in.([]byte); ok { out.SetString(string(b)) return true } case reflect.Int, reflect.Int64: if outt == typeJSONNumber { out.SetString(strconv.FormatInt(inv.Int(), 10)) return true } case reflect.Float64: if outt == typeJSONNumber { out.SetString(strconv.FormatFloat(inv.Float(), 'f', -1, 64)) return true } } case reflect.Slice, reflect.Array: // Remember, array (0x04) slices are built with the correct // element type. If we are here, must be a cross BSON kind // conversion (e.g. 0x05 unmarshalling on string). if outt.Elem().Kind() != reflect.Uint8 { break } switch inv.Kind() { case reflect.String: slice := []byte(inv.String()) out.Set(reflect.ValueOf(slice)) return true case reflect.Slice: switch outt.Kind() { case reflect.Array: reflect.Copy(out, inv) case reflect.Slice: out.SetBytes(inv.Bytes()) } return true } case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64: switch inv.Kind() { case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64: out.SetInt(inv.Int()) return true case reflect.Float32, reflect.Float64: out.SetInt(int64(inv.Float())) return true case reflect.Bool: if inv.Bool() { out.SetInt(1) } else { out.SetInt(0) } return true case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Uintptr: panic("can't happen: no uint types in BSON (!?)") } case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Uintptr: switch inv.Kind() { case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64: out.SetUint(uint64(inv.Int())) return true case reflect.Float32, reflect.Float64: out.SetUint(uint64(inv.Float())) return true case reflect.Bool: if inv.Bool() { out.SetUint(1) } else { out.SetUint(0) } return true case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Uintptr: panic("Can't happen. No uint types in BSON.") } case reflect.Float32, reflect.Float64: switch inv.Kind() { case reflect.Float32, reflect.Float64: out.SetFloat(inv.Float()) return true case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64: out.SetFloat(float64(inv.Int())) return true case reflect.Bool: if inv.Bool() { out.SetFloat(1) } else { out.SetFloat(0) } return true case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Uintptr: panic("Can't happen. No uint types in BSON?") } case reflect.Bool: switch inv.Kind() { case reflect.Bool: out.SetBool(inv.Bool()) return true case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64: out.SetBool(inv.Int() != 0) return true case reflect.Float32, reflect.Float64: out.SetBool(inv.Float() != 0) return true case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Uintptr: panic("Can't happen. No uint types in BSON?") } case reflect.Struct: if outt == typeURL && inv.Kind() == reflect.String { u, err := url.Parse(inv.String()) if err != nil { panic(err) } out.Set(reflect.ValueOf(u).Elem()) return true } if outt == typeBinary { if b, ok := in.([]byte); ok { out.Set(reflect.ValueOf(Binary{Data: b})) return true } } } return false } // -------------------------------------------------------------------------- // Parsers of basic types. func (d *decoder) readRegEx() RegEx { re := RegEx{} re.Pattern = d.readCStr() re.Options = d.readCStr() return re } func (d *decoder) readBinary() Binary { l := d.readInt32() b := Binary{} b.Kind = d.readByte() b.Data = d.readBytes(l) if b.Kind == 0x02 && len(b.Data) >= 4 { // Weird obsolete format with redundant length. b.Data = b.Data[4:] } return b } func (d *decoder) readStr() string { l := d.readInt32() b := d.readBytes(l - 1) if d.readByte() != '\x00' { corrupted() } return string(b) } func (d *decoder) readCStr() string { start := d.i end := start l := len(d.in) for ; end != l; end++ { if d.in[end] == '\x00' { break } } d.i = end + 1 if d.i > l { corrupted() } return string(d.in[start:end]) } func (d *decoder) readBool() bool { b := d.readByte() if b == 0 { return false } if b == 1 { return true } panic(fmt.Sprintf("encoded boolean must be 1 or 0, found %d", b)) } func (d *decoder) readFloat64() float64 { return math.Float64frombits(uint64(d.readInt64())) } func (d *decoder) readInt32() int32 { b := d.readBytes(4) return int32((uint32(b[0]) << 0) | (uint32(b[1]) << 8) | (uint32(b[2]) << 16) | (uint32(b[3]) << 24)) } func (d *decoder) readInt64() int64 { b := d.readBytes(8) return int64((uint64(b[0]) << 0) | (uint64(b[1]) << 8) | (uint64(b[2]) << 16) | (uint64(b[3]) << 24) | (uint64(b[4]) << 32) | (uint64(b[5]) << 40) | (uint64(b[6]) << 48) | (uint64(b[7]) << 56)) } func (d *decoder) readByte() byte { i := d.i d.i++ if d.i > len(d.in) { corrupted() } return d.in[i] } func (d *decoder) readBytes(length int32) []byte { if length < 0 { corrupted() } start := d.i d.i += int(length) if d.i < start || d.i > len(d.in) { corrupted() } return d.in[start : start+int(length)] } mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/bson/encode.go000066400000000000000000000303131276133741700237050ustar00rootroot00000000000000// BSON library for Go // // Copyright (c) 2010-2012 - Gustavo Niemeyer // // All rights reserved. // // Redistribution and use in source and binary forms, with or without // modification, are permitted provided that the following conditions are met: // // 1. Redistributions of source code must retain the above copyright notice, this // list of conditions and the following disclaimer. // 2. Redistributions in binary form must reproduce the above copyright notice, // this list of conditions and the following disclaimer in the documentation // and/or other materials provided with the distribution. // // THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND // ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED // WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE // DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR // ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES // (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; // LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND // ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT // (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS // SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. // gobson - BSON library for Go. package bson import ( "encoding/json" "fmt" "math" "net/url" "reflect" "strconv" "time" ) // -------------------------------------------------------------------------- // Some internal infrastructure. var ( typeBinary = reflect.TypeOf(Binary{}) typeObjectId = reflect.TypeOf(ObjectId("")) typeDBPointer = reflect.TypeOf(DBPointer{"", ObjectId("")}) typeSymbol = reflect.TypeOf(Symbol("")) typeMongoTimestamp = reflect.TypeOf(MongoTimestamp(0)) typeOrderKey = reflect.TypeOf(MinKey) typeDocElem = reflect.TypeOf(DocElem{}) typeRawDocElem = reflect.TypeOf(RawDocElem{}) typeRaw = reflect.TypeOf(Raw{}) typeURL = reflect.TypeOf(url.URL{}) typeTime = reflect.TypeOf(time.Time{}) typeString = reflect.TypeOf("") typeJSONNumber = reflect.TypeOf(json.Number("")) ) const itoaCacheSize = 32 var itoaCache []string func init() { itoaCache = make([]string, itoaCacheSize) for i := 0; i != itoaCacheSize; i++ { itoaCache[i] = strconv.Itoa(i) } } func itoa(i int) string { if i < itoaCacheSize { return itoaCache[i] } return strconv.Itoa(i) } // -------------------------------------------------------------------------- // Marshaling of the document value itself. type encoder struct { out []byte } func (e *encoder) addDoc(v reflect.Value) { for { if vi, ok := v.Interface().(Getter); ok { getv, err := vi.GetBSON() if err != nil { panic(err) } v = reflect.ValueOf(getv) continue } if v.Kind() == reflect.Ptr { v = v.Elem() continue } break } if v.Type() == typeRaw { raw := v.Interface().(Raw) if raw.Kind != 0x03 && raw.Kind != 0x00 { panic("Attempted to marshal Raw kind " + strconv.Itoa(int(raw.Kind)) + " as a document") } if len(raw.Data) == 0 { panic("Attempted to marshal empty Raw document") } e.addBytes(raw.Data...) return } start := e.reserveInt32() switch v.Kind() { case reflect.Map: e.addMap(v) case reflect.Struct: e.addStruct(v) case reflect.Array, reflect.Slice: e.addSlice(v) default: panic("Can't marshal " + v.Type().String() + " as a BSON document") } e.addBytes(0) e.setInt32(start, int32(len(e.out)-start)) } func (e *encoder) addMap(v reflect.Value) { for _, k := range v.MapKeys() { e.addElem(k.String(), v.MapIndex(k), false) } } func (e *encoder) addStruct(v reflect.Value) { sinfo, err := getStructInfo(v.Type()) if err != nil { panic(err) } var value reflect.Value if sinfo.InlineMap >= 0 { m := v.Field(sinfo.InlineMap) if m.Len() > 0 { for _, k := range m.MapKeys() { ks := k.String() if _, found := sinfo.FieldsMap[ks]; found { panic(fmt.Sprintf("Can't have key %q in inlined map; conflicts with struct field", ks)) } e.addElem(ks, m.MapIndex(k), false) } } } for _, info := range sinfo.FieldsList { if info.Inline == nil { value = v.Field(info.Num) } else { value = v.FieldByIndex(info.Inline) } if info.OmitEmpty && isZero(value) { continue } e.addElem(info.Key, value, info.MinSize) } } func isZero(v reflect.Value) bool { switch v.Kind() { case reflect.String: return len(v.String()) == 0 case reflect.Ptr, reflect.Interface: return v.IsNil() case reflect.Slice: return v.Len() == 0 case reflect.Map: return v.Len() == 0 case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64: return v.Int() == 0 case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Uintptr: return v.Uint() == 0 case reflect.Float32, reflect.Float64: return v.Float() == 0 case reflect.Bool: return !v.Bool() case reflect.Struct: vt := v.Type() if vt == typeTime { return v.Interface().(time.Time).IsZero() } for i := 0; i < v.NumField(); i++ { if vt.Field(i).PkgPath != "" && !vt.Field(i).Anonymous { continue // Private field } if !isZero(v.Field(i)) { return false } } return true } return false } func (e *encoder) addSlice(v reflect.Value) { vi := v.Interface() if d, ok := vi.(D); ok { for _, elem := range d { e.addElem(elem.Name, reflect.ValueOf(elem.Value), false) } return } if d, ok := vi.(RawD); ok { for _, elem := range d { e.addElem(elem.Name, reflect.ValueOf(elem.Value), false) } return } l := v.Len() et := v.Type().Elem() if et == typeDocElem { for i := 0; i < l; i++ { elem := v.Index(i).Interface().(DocElem) e.addElem(elem.Name, reflect.ValueOf(elem.Value), false) } return } if et == typeRawDocElem { for i := 0; i < l; i++ { elem := v.Index(i).Interface().(RawDocElem) e.addElem(elem.Name, reflect.ValueOf(elem.Value), false) } return } for i := 0; i < l; i++ { e.addElem(itoa(i), v.Index(i), false) } } // -------------------------------------------------------------------------- // Marshaling of elements in a document. func (e *encoder) addElemName(kind byte, name string) { e.addBytes(kind) e.addBytes([]byte(name)...) e.addBytes(0) } func (e *encoder) addElem(name string, v reflect.Value, minSize bool) { if !v.IsValid() { e.addElemName(0x0A, name) return } if getter, ok := v.Interface().(Getter); ok { getv, err := getter.GetBSON() if err != nil { panic(err) } e.addElem(name, reflect.ValueOf(getv), minSize) return } switch v.Kind() { case reflect.Interface: e.addElem(name, v.Elem(), minSize) case reflect.Ptr: e.addElem(name, v.Elem(), minSize) case reflect.String: s := v.String() switch v.Type() { case typeObjectId: if len(s) != 12 { panic("ObjectIDs must be exactly 12 bytes long (got " + strconv.Itoa(len(s)) + ")") } e.addElemName(0x07, name) e.addBytes([]byte(s)...) case typeSymbol: e.addElemName(0x0E, name) e.addStr(s) case typeJSONNumber: n := v.Interface().(json.Number) if i, err := n.Int64(); err == nil { e.addElemName(0x12, name) e.addInt64(i) } else if f, err := n.Float64(); err == nil { e.addElemName(0x01, name) e.addFloat64(f) } else { panic("failed to convert json.Number to a number: " + s) } default: e.addElemName(0x02, name) e.addStr(s) } case reflect.Float32, reflect.Float64: e.addElemName(0x01, name) e.addFloat64(v.Float()) case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Uintptr: u := v.Uint() if int64(u) < 0 { panic("BSON has no uint64 type, and value is too large to fit correctly in an int64") } else if u <= math.MaxInt32 && (minSize || v.Kind() <= reflect.Uint32) { e.addElemName(0x10, name) e.addInt32(int32(u)) } else { e.addElemName(0x12, name) e.addInt64(int64(u)) } case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64: switch v.Type() { case typeMongoTimestamp: e.addElemName(0x11, name) e.addInt64(v.Int()) case typeOrderKey: if v.Int() == int64(MaxKey) { e.addElemName(0x7F, name) } else { e.addElemName(0xFF, name) } default: i := v.Int() if (minSize || v.Type().Kind() != reflect.Int64) && i >= math.MinInt32 && i <= math.MaxInt32 { // It fits into an int32, encode as such. e.addElemName(0x10, name) e.addInt32(int32(i)) } else { e.addElemName(0x12, name) e.addInt64(i) } } case reflect.Bool: e.addElemName(0x08, name) if v.Bool() { e.addBytes(1) } else { e.addBytes(0) } case reflect.Map: e.addElemName(0x03, name) e.addDoc(v) case reflect.Slice: vt := v.Type() et := vt.Elem() if et.Kind() == reflect.Uint8 { e.addElemName(0x05, name) e.addBinary(0x00, v.Bytes()) } else if et == typeDocElem || et == typeRawDocElem { e.addElemName(0x03, name) e.addDoc(v) } else { e.addElemName(0x04, name) e.addDoc(v) } case reflect.Array: et := v.Type().Elem() if et.Kind() == reflect.Uint8 { e.addElemName(0x05, name) if v.CanAddr() { e.addBinary(0x00, v.Slice(0, v.Len()).Interface().([]byte)) } else { n := v.Len() e.addInt32(int32(n)) e.addBytes(0x00) for i := 0; i < n; i++ { el := v.Index(i) e.addBytes(byte(el.Uint())) } } } else { e.addElemName(0x04, name) e.addDoc(v) } case reflect.Struct: switch s := v.Interface().(type) { case Raw: kind := s.Kind if kind == 0x00 { kind = 0x03 } if len(s.Data) == 0 && kind != 0x06 && kind != 0x0A && kind != 0xFF && kind != 0x7F { panic("Attempted to marshal empty Raw document") } e.addElemName(kind, name) e.addBytes(s.Data...) case Binary: e.addElemName(0x05, name) e.addBinary(s.Kind, s.Data) case Decimal128: e.addElemName(0x13, name) e.addInt64(int64(s.l)) e.addInt64(int64(s.h)) case DBPointer: e.addElemName(0x0C, name) e.addStr(s.Namespace) if len(s.Id) != 12 { panic("ObjectIDs must be exactly 12 bytes long (got " + strconv.Itoa(len(s.Id)) + ")") } e.addBytes([]byte(s.Id)...) case RegEx: e.addElemName(0x0B, name) e.addCStr(s.Pattern) e.addCStr(s.Options) case JavaScript: if s.Scope == nil { e.addElemName(0x0D, name) e.addStr(s.Code) } else { e.addElemName(0x0F, name) start := e.reserveInt32() e.addStr(s.Code) e.addDoc(reflect.ValueOf(s.Scope)) e.setInt32(start, int32(len(e.out)-start)) } case time.Time: // MongoDB handles timestamps as milliseconds. e.addElemName(0x09, name) e.addInt64(s.Unix()*1000 + int64(s.Nanosecond()/1e6)) case url.URL: e.addElemName(0x02, name) e.addStr(s.String()) case undefined: e.addElemName(0x06, name) default: e.addElemName(0x03, name) e.addDoc(v) } default: panic("Can't marshal " + v.Type().String() + " in a BSON document") } } // -------------------------------------------------------------------------- // Marshaling of base types. func (e *encoder) addBinary(subtype byte, v []byte) { if subtype == 0x02 { // Wonder how that brilliant idea came to life. Obsolete, luckily. e.addInt32(int32(len(v) + 4)) e.addBytes(subtype) e.addInt32(int32(len(v))) } else { e.addInt32(int32(len(v))) e.addBytes(subtype) } e.addBytes(v...) } func (e *encoder) addStr(v string) { e.addInt32(int32(len(v) + 1)) e.addCStr(v) } func (e *encoder) addCStr(v string) { e.addBytes([]byte(v)...) e.addBytes(0) } func (e *encoder) reserveInt32() (pos int) { pos = len(e.out) e.addBytes(0, 0, 0, 0) return pos } func (e *encoder) setInt32(pos int, v int32) { e.out[pos+0] = byte(v) e.out[pos+1] = byte(v >> 8) e.out[pos+2] = byte(v >> 16) e.out[pos+3] = byte(v >> 24) } func (e *encoder) addInt32(v int32) { u := uint32(v) e.addBytes(byte(u), byte(u>>8), byte(u>>16), byte(u>>24)) } func (e *encoder) addInt64(v int64) { u := uint64(v) e.addBytes(byte(u), byte(u>>8), byte(u>>16), byte(u>>24), byte(u>>32), byte(u>>40), byte(u>>48), byte(u>>56)) } func (e *encoder) addFloat64(v float64) { e.addInt64(int64(math.Float64bits(v))) } func (e *encoder) addBytes(v ...byte) { e.out = append(e.out, v...) } mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/bson/json.go000066400000000000000000000212651276133741700234270ustar00rootroot00000000000000package bson import ( "bytes" "encoding/base64" "fmt" "gopkg.in/mgo.v2/internal/json" "strconv" "time" ) // UnmarshalJSON unmarshals a JSON value that may hold non-standard // syntax as defined in BSON's extended JSON specification. func UnmarshalJSON(data []byte, value interface{}) error { d := json.NewDecoder(bytes.NewBuffer(data)) d.Extend(&jsonExt) return d.Decode(value) } // MarshalJSON marshals a JSON value that may hold non-standard // syntax as defined in BSON's extended JSON specification. func MarshalJSON(value interface{}) ([]byte, error) { var buf bytes.Buffer e := json.NewEncoder(&buf) e.Extend(&jsonExt) err := e.Encode(value) if err != nil { return nil, err } return buf.Bytes(), nil } // jdec is used internally by the JSON decoding functions // so they may unmarshal functions without getting into endless // recursion due to keyed objects. func jdec(data []byte, value interface{}) error { d := json.NewDecoder(bytes.NewBuffer(data)) d.Extend(&funcExt) return d.Decode(value) } var jsonExt json.Extension var funcExt json.Extension // TODO // - Shell regular expressions ("/regexp/opts") func init() { jsonExt.DecodeUnquotedKeys(true) jsonExt.DecodeTrailingCommas(true) funcExt.DecodeFunc("BinData", "$binaryFunc", "$type", "$binary") jsonExt.DecodeKeyed("$binary", jdecBinary) jsonExt.DecodeKeyed("$binaryFunc", jdecBinary) jsonExt.EncodeType([]byte(nil), jencBinarySlice) jsonExt.EncodeType(Binary{}, jencBinaryType) funcExt.DecodeFunc("ISODate", "$dateFunc", "S") funcExt.DecodeFunc("new Date", "$dateFunc", "S") jsonExt.DecodeKeyed("$date", jdecDate) jsonExt.DecodeKeyed("$dateFunc", jdecDate) jsonExt.EncodeType(time.Time{}, jencDate) funcExt.DecodeFunc("Timestamp", "$timestamp", "t", "i") jsonExt.DecodeKeyed("$timestamp", jdecTimestamp) jsonExt.EncodeType(MongoTimestamp(0), jencTimestamp) funcExt.DecodeConst("undefined", Undefined) jsonExt.DecodeKeyed("$regex", jdecRegEx) jsonExt.EncodeType(RegEx{}, jencRegEx) funcExt.DecodeFunc("ObjectId", "$oidFunc", "Id") jsonExt.DecodeKeyed("$oid", jdecObjectId) jsonExt.DecodeKeyed("$oidFunc", jdecObjectId) jsonExt.EncodeType(ObjectId(""), jencObjectId) funcExt.DecodeFunc("DBRef", "$dbrefFunc", "$ref", "$id") jsonExt.DecodeKeyed("$dbrefFunc", jdecDBRef) funcExt.DecodeFunc("NumberLong", "$numberLongFunc", "N") jsonExt.DecodeKeyed("$numberLong", jdecNumberLong) jsonExt.DecodeKeyed("$numberLongFunc", jdecNumberLong) jsonExt.EncodeType(int64(0), jencNumberLong) jsonExt.EncodeType(int(0), jencInt) funcExt.DecodeConst("MinKey", MinKey) funcExt.DecodeConst("MaxKey", MaxKey) jsonExt.DecodeKeyed("$minKey", jdecMinKey) jsonExt.DecodeKeyed("$maxKey", jdecMaxKey) jsonExt.EncodeType(orderKey(0), jencMinMaxKey) jsonExt.DecodeKeyed("$undefined", jdecUndefined) jsonExt.EncodeType(Undefined, jencUndefined) jsonExt.Extend(&funcExt) } func fbytes(format string, args ...interface{}) []byte { var buf bytes.Buffer fmt.Fprintf(&buf, format, args...) return buf.Bytes() } func jdecBinary(data []byte) (interface{}, error) { var v struct { Binary []byte `json:"$binary"` Type string `json:"$type"` Func struct { Binary []byte `json:"$binary"` Type int64 `json:"$type"` } `json:"$binaryFunc"` } err := jdec(data, &v) if err != nil { return nil, err } var binData []byte var binKind int64 if v.Type == "" && v.Binary == nil { binData = v.Func.Binary binKind = v.Func.Type } else if v.Type == "" { return v.Binary, nil } else { binData = v.Binary binKind, err = strconv.ParseInt(v.Type, 0, 64) if err != nil { binKind = -1 } } if binKind == 0 { return binData, nil } if binKind < 0 || binKind > 255 { return nil, fmt.Errorf("invalid type in binary object: %s", data) } return Binary{Kind: byte(binKind), Data: binData}, nil } func jencBinarySlice(v interface{}) ([]byte, error) { in := v.([]byte) out := make([]byte, base64.StdEncoding.EncodedLen(len(in))) base64.StdEncoding.Encode(out, in) return fbytes(`{"$binary":"%s","$type":"0x0"}`, out), nil } func jencBinaryType(v interface{}) ([]byte, error) { in := v.(Binary) out := make([]byte, base64.StdEncoding.EncodedLen(len(in.Data))) base64.StdEncoding.Encode(out, in.Data) return fbytes(`{"$binary":"%s","$type":"0x%x"}`, out, in.Kind), nil } const jdateFormat = "2006-01-02T15:04:05.999Z" func jdecDate(data []byte) (interface{}, error) { var v struct { S string `json:"$date"` Func struct { S string } `json:"$dateFunc"` } _ = jdec(data, &v) if v.S == "" { v.S = v.Func.S } if v.S != "" { for _, format := range []string{jdateFormat, "2006-01-02"} { t, err := time.Parse(format, v.S) if err == nil { return t, nil } } return nil, fmt.Errorf("cannot parse date: %q", v.S) } var vn struct { Date struct { N int64 `json:"$numberLong,string"` } `json:"$date"` Func struct { S int64 } `json:"$dateFunc"` } err := jdec(data, &vn) if err != nil { return nil, fmt.Errorf("cannot parse date: %q", data) } n := vn.Date.N if n == 0 { n = vn.Func.S } return time.Unix(n/1000, n%1000*1e6).UTC(), nil } func jencDate(v interface{}) ([]byte, error) { t := v.(time.Time) return fbytes(`{"$date":%q}`, t.Format(jdateFormat)), nil } func jdecTimestamp(data []byte) (interface{}, error) { var v struct { Func struct { T int32 `json:"t"` I int32 `json:"i"` } `json:"$timestamp"` } err := jdec(data, &v) if err != nil { return nil, err } return MongoTimestamp(uint64(v.Func.T)<<32 | uint64(uint32(v.Func.I))), nil } func jencTimestamp(v interface{}) ([]byte, error) { ts := uint64(v.(MongoTimestamp)) return fbytes(`{"$timestamp":{"t":%d,"i":%d}}`, ts>>32, uint32(ts)), nil } func jdecRegEx(data []byte) (interface{}, error) { var v struct { Regex string `json:"$regex"` Options string `json:"$options"` } err := jdec(data, &v) if err != nil { return nil, err } return RegEx{v.Regex, v.Options}, nil } func jencRegEx(v interface{}) ([]byte, error) { re := v.(RegEx) type regex struct { Regex string `json:"$regex"` Options string `json:"$options"` } return json.Marshal(regex{re.Pattern, re.Options}) } func jdecObjectId(data []byte) (interface{}, error) { var v struct { Id string `json:"$oid"` Func struct { Id string } `json:"$oidFunc"` } err := jdec(data, &v) if err != nil { return nil, err } if v.Id == "" { v.Id = v.Func.Id } return ObjectIdHex(v.Id), nil } func jencObjectId(v interface{}) ([]byte, error) { return fbytes(`{"$oid":"%s"}`, v.(ObjectId).Hex()), nil } func jdecDBRef(data []byte) (interface{}, error) { // TODO Support unmarshaling $ref and $id into the input value. var v struct { Obj map[string]interface{} `json:"$dbrefFunc"` } // TODO Fix this. Must not be required. v.Obj = make(map[string]interface{}) err := jdec(data, &v) if err != nil { return nil, err } return v.Obj, nil } func jdecNumberLong(data []byte) (interface{}, error) { var v struct { N int64 `json:"$numberLong,string"` Func struct { N int64 `json:",string"` } `json:"$numberLongFunc"` } var vn struct { N int64 `json:"$numberLong"` Func struct { N int64 } `json:"$numberLongFunc"` } err := jdec(data, &v) if err != nil { err = jdec(data, &vn) v.N = vn.N v.Func.N = vn.Func.N } if err != nil { return nil, err } if v.N != 0 { return v.N, nil } return v.Func.N, nil } func jencNumberLong(v interface{}) ([]byte, error) { n := v.(int64) f := `{"$numberLong":"%d"}` if n <= 1<<53 { f = `{"$numberLong":%d}` } return fbytes(f, n), nil } func jencInt(v interface{}) ([]byte, error) { n := v.(int) f := `{"$numberLong":"%d"}` if int64(n) <= 1<<53 { f = `%d` } return fbytes(f, n), nil } func jdecMinKey(data []byte) (interface{}, error) { var v struct { N int64 `json:"$minKey"` } err := jdec(data, &v) if err != nil { return nil, err } if v.N != 1 { return nil, fmt.Errorf("invalid $minKey object: %s", data) } return MinKey, nil } func jdecMaxKey(data []byte) (interface{}, error) { var v struct { N int64 `json:"$maxKey"` } err := jdec(data, &v) if err != nil { return nil, err } if v.N != 1 { return nil, fmt.Errorf("invalid $maxKey object: %s", data) } return MaxKey, nil } func jencMinMaxKey(v interface{}) ([]byte, error) { switch v.(orderKey) { case MinKey: return []byte(`{"$minKey":1}`), nil case MaxKey: return []byte(`{"$maxKey":1}`), nil } panic(fmt.Sprintf("invalid $minKey/$maxKey value: %d", v)) } func jdecUndefined(data []byte) (interface{}, error) { var v struct { B bool `json:"$undefined"` } err := jdec(data, &v) if err != nil { return nil, err } if !v.B { return nil, fmt.Errorf("invalid $undefined object: %s", data) } return Undefined, nil } func jencUndefined(v interface{}) ([]byte, error) { return []byte(`{"$undefined":true}`), nil } mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/bson/json_test.go000066400000000000000000000074301276133741700244640ustar00rootroot00000000000000package bson_test import ( "gopkg.in/mgo.v2/bson" . "gopkg.in/check.v1" "reflect" "strings" "time" ) type jsonTest struct { a interface{} // value encoded into JSON (optional) b string // JSON expected as output of , and used as input to c interface{} // Value expected from decoding , defaults to e string // error string, if decoding (b) should fail } var jsonTests = []jsonTest{ // $binary { a: []byte("foo"), b: `{"$binary":"Zm9v","$type":"0x0"}`, }, { a: bson.Binary{Kind: 2, Data: []byte("foo")}, b: `{"$binary":"Zm9v","$type":"0x2"}`, }, { b: `BinData(2,"Zm9v")`, c: bson.Binary{Kind: 2, Data: []byte("foo")}, }, // $date { a: time.Date(2016, 5, 15, 1, 2, 3, 4000000, time.UTC), b: `{"$date":"2016-05-15T01:02:03.004Z"}`, }, { b: `{"$date": {"$numberLong": "1002"}}`, c: time.Date(1970, 1, 1, 0, 0, 1, 2e6, time.UTC), }, { b: `ISODate("2016-05-15T01:02:03.004Z")`, c: time.Date(2016, 5, 15, 1, 2, 3, 4000000, time.UTC), }, { b: `new Date(1000)`, c: time.Date(1970, 1, 1, 0, 0, 1, 0, time.UTC), }, { b: `new Date("2016-05-15")`, c: time.Date(2016, 5, 15, 0, 0, 0, 0, time.UTC), }, // $timestamp { a: bson.MongoTimestamp(4294967298), b: `{"$timestamp":{"t":1,"i":2}}`, }, { b: `Timestamp(1, 2)`, c: bson.MongoTimestamp(4294967298), }, // $regex { a: bson.RegEx{"pattern", "options"}, b: `{"$regex":"pattern","$options":"options"}`, }, // $oid { a: bson.ObjectIdHex("0123456789abcdef01234567"), b: `{"$oid":"0123456789abcdef01234567"}`, }, { b: `ObjectId("0123456789abcdef01234567")`, c: bson.ObjectIdHex("0123456789abcdef01234567"), }, // $ref (no special type) { b: `DBRef("name", "id")`, c: map[string]interface{}{"$ref": "name", "$id": "id"}, }, // $numberLong { a: 123, b: `123`, }, { a: int64(9007199254740992), b: `{"$numberLong":9007199254740992}`, }, { a: int64(1<<53 + 1), b: `{"$numberLong":"9007199254740993"}`, }, { a: 1<<53 + 1, b: `{"$numberLong":"9007199254740993"}`, c: int64(9007199254740993), }, { b: `NumberLong(9007199254740992)`, c: int64(1 << 53), }, { b: `NumberLong("9007199254740993")`, c: int64(1<<53 + 1), }, // $minKey, $maxKey { a: bson.MinKey, b: `{"$minKey":1}`, }, { a: bson.MaxKey, b: `{"$maxKey":1}`, }, { b: `MinKey`, c: bson.MinKey, }, { b: `MaxKey`, c: bson.MaxKey, }, { b: `{"$minKey":0}`, e: `invalid $minKey object: {"$minKey":0}`, }, { b: `{"$maxKey":0}`, e: `invalid $maxKey object: {"$maxKey":0}`, }, { a: bson.Undefined, b: `{"$undefined":true}`, }, { b: `undefined`, c: bson.Undefined, }, { b: `{"v": undefined}`, c: struct{ V interface{} }{bson.Undefined}, }, // Unquoted keys and trailing commas { b: `{$foo: ["bar",],}`, c: map[string]interface{}{"$foo": []interface{}{"bar"}}, }, } func (s *S) TestJSON(c *C) { for i, item := range jsonTests { c.Logf("------------ (#%d)", i) c.Logf("A: %#v", item.a) c.Logf("B: %#v", item.b) if item.c == nil { item.c = item.a } else { c.Logf("C: %#v", item.c) } if item.e != "" { c.Logf("E: %s", item.e) } if item.a != nil { data, err := bson.MarshalJSON(item.a) c.Assert(err, IsNil) c.Logf("Dumped: %#v", string(data)) c.Assert(strings.TrimSuffix(string(data), "\n"), Equals, item.b) } var zero interface{} if item.c == nil { zero = &struct{}{} } else { zero = reflect.New(reflect.TypeOf(item.c)).Interface() } err := bson.UnmarshalJSON([]byte(item.b), zero) if item.e != "" { c.Assert(err, NotNil) c.Assert(err.Error(), Equals, item.e) continue } c.Assert(err, IsNil) zerov := reflect.ValueOf(zero) value := zerov.Interface() if zerov.Kind() == reflect.Ptr { value = zerov.Elem().Interface() } c.Logf("Loaded: %#v", value) c.Assert(value, DeepEquals, item.c) } } mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/bson/specdata/000077500000000000000000000000001276133741700237055ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/bson/specdata/update.sh000077500000000000000000000005701276133741700255300ustar00rootroot00000000000000#!/bin/sh set -e if [ ! -d specifications ]; then git clone -b bson git@github.com:jyemin/specifications fi TESTFILE="../specdata_test.go" cat < $TESTFILE package bson_test var specTests = []string{ END for file in specifications/source/bson/tests/*.yml; do ( echo '`' cat $file echo -n '`,' ) >> $TESTFILE done echo '}' >> $TESTFILE gofmt -w $TESTFILE mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/bson/specdata_test.go000066400000000000000000000131101276133741700252670ustar00rootroot00000000000000package bson_test var specTests = []string{ ` --- description: "Array type" documents: - decoded: a : [] encoded: 0D000000046100050000000000 - decoded: a: [10] encoded: 140000000461000C0000001030000A0000000000 - # Decode an array that uses an empty string as the key decodeOnly : true decoded: a: [10] encoded: 130000000461000B00000010000A0000000000 - # Decode an array that uses a non-numeric string as the key decodeOnly : true decoded: a: [10] encoded: 150000000461000D000000106162000A0000000000 `, ` --- description: "Boolean type" documents: - encoded: "090000000862000100" decoded: { "b" : true } - encoded: "090000000862000000" decoded: { "b" : false } `, ` --- description: "Corrupted BSON" documents: - encoded: "09000000016600" error: "truncated double" - encoded: "09000000026600" error: "truncated string" - encoded: "09000000036600" error: "truncated document" - encoded: "09000000046600" error: "truncated array" - encoded: "09000000056600" error: "truncated binary" - encoded: "09000000076600" error: "truncated objectid" - encoded: "09000000086600" error: "truncated boolean" - encoded: "09000000096600" error: "truncated date" - encoded: "090000000b6600" error: "truncated regex" - encoded: "090000000c6600" error: "truncated db pointer" - encoded: "0C0000000d6600" error: "truncated javascript" - encoded: "0C0000000e6600" error: "truncated symbol" - encoded: "0C0000000f6600" error: "truncated javascript with scope" - encoded: "0C000000106600" error: "truncated int32" - encoded: "0C000000116600" error: "truncated timestamp" - encoded: "0C000000126600" error: "truncated int64" - encoded: "0400000000" error: basic - encoded: "0500000001" error: basic - encoded: "05000000" error: basic - encoded: "0700000002610078563412" error: basic - encoded: "090000001061000500" error: basic - encoded: "00000000000000000000" error: basic - encoded: "1300000002666f6f00040000006261720000" error: "basic" - encoded: "1800000003666f6f000f0000001062617200ffffff7f0000" error: basic - encoded: "1500000003666f6f000c0000000862617200010000" error: basic - encoded: "1c00000003666f6f001200000002626172000500000062617a000000" error: basic - encoded: "1000000002610004000000616263ff00" error: string is not null-terminated - encoded: "0c0000000200000000000000" error: bad_string_length - encoded: "120000000200ffffffff666f6f6261720000" error: bad_string_length - encoded: "0c0000000e00000000000000" error: bad_string_length - encoded: "120000000e00ffffffff666f6f6261720000" error: bad_string_length - encoded: "180000000c00fa5bd841d6585d9900" error: "" - encoded: "1e0000000c00ffffffff666f6f626172005259b56afa5bd841d6585d9900" error: bad_string_length - encoded: "0c0000000d00000000000000" error: bad_string_length - encoded: "0c0000000d00ffffffff0000" error: bad_string_length - encoded: "1c0000000f001500000000000000000c000000020001000000000000" error: bad_string_length - encoded: "1c0000000f0015000000ffffffff000c000000020001000000000000" error: bad_string_length - encoded: "1c0000000f001500000001000000000c000000020000000000000000" error: bad_string_length - encoded: "1c0000000f001500000001000000000c0000000200ffffffff000000" error: bad_string_length - encoded: "0E00000008616263646566676869707172737475" error: "Run-on CString" - encoded: "0100000000" error: "An object size that's too small to even include the object size, but is correctly encoded, along with a correct EOO (and no data)" - encoded: "1a0000000e74657374000c00000068656c6c6f20776f726c6400000500000000" error: "One object, but with object size listed smaller than it is in the data" - encoded: "05000000" error: "One object, missing the EOO at the end" - encoded: "0500000001" error: "One object, sized correctly, with a spot for an EOO, but the EOO is 0x01" - encoded: "05000000ff" error: "One object, sized correctly, with a spot for an EOO, but the EOO is 0xff" - encoded: "0500000070" error: "One object, sized correctly, with a spot for an EOO, but the EOO is 0x70" - encoded: "07000000000000" error: "Invalid BSON type low range" - encoded: "07000000800000" error: "Invalid BSON type high range" - encoded: "090000000862000200" error: "Invalid boolean value of 2" - encoded: "09000000086200ff00" error: "Invalid boolean value of -1" `, ` --- description: "Int32 type" documents: - decoded: i: -2147483648 encoded: 0C0000001069000000008000 - decoded: i: 2147483647 encoded: 0C000000106900FFFFFF7F00 - decoded: i: -1 encoded: 0C000000106900FFFFFFFF00 - decoded: i: 0 encoded: 0C0000001069000000000000 - decoded: i: 1 encoded: 0C0000001069000100000000 `, ` --- description: "String type" documents: - decoded: s : "" encoded: 0D000000027300010000000000 - decoded: s: "a" encoded: 0E00000002730002000000610000 - decoded: s: "This is a string" encoded: 1D0000000273001100000054686973206973206120737472696E670000 - decoded: s: "κόσμε" encoded: 180000000273000C000000CEBAE1BDB9CF83CEBCCEB50000 `} mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/bulk.go000066400000000000000000000234151276133741700224510ustar00rootroot00000000000000package mgo import ( "bytes" "sort" "gopkg.in/mgo.v2/bson" ) // Bulk represents an operation that can be prepared with several // orthogonal changes before being delivered to the server. // // MongoDB servers older than version 2.6 do not have proper support for bulk // operations, so the driver attempts to map its API as much as possible into // the functionality that works. In particular, in those releases updates and // removals are sent individually, and inserts are sent in bulk but have // suboptimal error reporting compared to more recent versions of the server. // See the documentation of BulkErrorCase for details on that. // // Relevant documentation: // // http://blog.mongodb.org/post/84922794768/mongodbs-new-bulk-api // type Bulk struct { c *Collection opcount int actions []bulkAction ordered bool } type bulkOp int const ( bulkInsert bulkOp = iota + 1 bulkUpdate bulkUpdateAll bulkRemove ) type bulkAction struct { op bulkOp docs []interface{} idxs []int } type bulkUpdateOp []interface{} type bulkDeleteOp []interface{} // BulkResult holds the results for a bulk operation. type BulkResult struct { Matched int Modified int // Available only for MongoDB 2.6+ // Be conservative while we understand exactly how to report these // results in a useful and convenient way, and also how to emulate // them with prior servers. private bool } // BulkError holds an error returned from running a Bulk operation. // Individual errors may be obtained and inspected via the Cases method. type BulkError struct { ecases []BulkErrorCase } func (e *BulkError) Error() string { if len(e.ecases) == 0 { return "invalid BulkError instance: no errors" } if len(e.ecases) == 1 { return e.ecases[0].Err.Error() } msgs := make([]string, 0, len(e.ecases)) seen := make(map[string]bool) for _, ecase := range e.ecases { msg := ecase.Err.Error() if !seen[msg] { seen[msg] = true msgs = append(msgs, msg) } } if len(msgs) == 1 { return msgs[0] } var buf bytes.Buffer buf.WriteString("multiple errors in bulk operation:\n") for _, msg := range msgs { buf.WriteString(" - ") buf.WriteString(msg) buf.WriteByte('\n') } return buf.String() } type bulkErrorCases []BulkErrorCase func (slice bulkErrorCases) Len() int { return len(slice) } func (slice bulkErrorCases) Less(i, j int) bool { return slice[i].Index < slice[j].Index } func (slice bulkErrorCases) Swap(i, j int) { slice[i], slice[j] = slice[j], slice[i] } // BulkErrorCase holds an individual error found while attempting a single change // within a bulk operation, and the position in which it was enqueued. // // MongoDB servers older than version 2.6 do not have proper support for bulk // operations, so the driver attempts to map its API as much as possible into // the functionality that works. In particular, only the last error is reported // for bulk inserts and without any positional information, so the Index // field is set to -1 in these cases. type BulkErrorCase struct { Index int // Position of operation that failed, or -1 if unknown. Err error } // Cases returns all individual errors found while attempting the requested changes. // // See the documentation of BulkErrorCase for limitations in older MongoDB releases. func (e *BulkError) Cases() []BulkErrorCase { return e.ecases } // Bulk returns a value to prepare the execution of a bulk operation. func (c *Collection) Bulk() *Bulk { return &Bulk{c: c, ordered: true} } // Unordered puts the bulk operation in unordered mode. // // In unordered mode the indvidual operations may be sent // out of order, which means latter operations may proceed // even if prior ones have failed. func (b *Bulk) Unordered() { b.ordered = false } func (b *Bulk) action(op bulkOp, opcount int) *bulkAction { var action *bulkAction if len(b.actions) > 0 && b.actions[len(b.actions)-1].op == op { action = &b.actions[len(b.actions)-1] } else if !b.ordered { for i := range b.actions { if b.actions[i].op == op { action = &b.actions[i] break } } } if action == nil { b.actions = append(b.actions, bulkAction{op: op}) action = &b.actions[len(b.actions)-1] } for i := 0; i < opcount; i++ { action.idxs = append(action.idxs, b.opcount) b.opcount++ } return action } // Insert queues up the provided documents for insertion. func (b *Bulk) Insert(docs ...interface{}) { action := b.action(bulkInsert, len(docs)) action.docs = append(action.docs, docs...) } // Remove queues up the provided selectors for removing matching documents. // Each selector will remove only a single matching document. func (b *Bulk) Remove(selectors ...interface{}) { action := b.action(bulkRemove, len(selectors)) for _, selector := range selectors { if selector == nil { selector = bson.D{} } action.docs = append(action.docs, &deleteOp{ Collection: b.c.FullName, Selector: selector, Flags: 1, Limit: 1, }) } } // RemoveAll queues up the provided selectors for removing all matching documents. // Each selector will remove all matching documents. func (b *Bulk) RemoveAll(selectors ...interface{}) { action := b.action(bulkRemove, len(selectors)) for _, selector := range selectors { if selector == nil { selector = bson.D{} } action.docs = append(action.docs, &deleteOp{ Collection: b.c.FullName, Selector: selector, Flags: 0, Limit: 0, }) } } // Update queues up the provided pairs of updating instructions. // The first element of each pair selects which documents must be // updated, and the second element defines how to update it. // Each pair matches exactly one document for updating at most. func (b *Bulk) Update(pairs ...interface{}) { if len(pairs)%2 != 0 { panic("Bulk.Update requires an even number of parameters") } action := b.action(bulkUpdate, len(pairs)/2) for i := 0; i < len(pairs); i += 2 { selector := pairs[i] if selector == nil { selector = bson.D{} } action.docs = append(action.docs, &updateOp{ Collection: b.c.FullName, Selector: selector, Update: pairs[i+1], }) } } // UpdateAll queues up the provided pairs of updating instructions. // The first element of each pair selects which documents must be // updated, and the second element defines how to update it. // Each pair updates all documents matching the selector. func (b *Bulk) UpdateAll(pairs ...interface{}) { if len(pairs)%2 != 0 { panic("Bulk.UpdateAll requires an even number of parameters") } action := b.action(bulkUpdate, len(pairs)/2) for i := 0; i < len(pairs); i += 2 { selector := pairs[i] if selector == nil { selector = bson.D{} } action.docs = append(action.docs, &updateOp{ Collection: b.c.FullName, Selector: selector, Update: pairs[i+1], Flags: 2, Multi: true, }) } } // Upsert queues up the provided pairs of upserting instructions. // The first element of each pair selects which documents must be // updated, and the second element defines how to update it. // Each pair matches exactly one document for updating at most. func (b *Bulk) Upsert(pairs ...interface{}) { if len(pairs)%2 != 0 { panic("Bulk.Update requires an even number of parameters") } action := b.action(bulkUpdate, len(pairs)/2) for i := 0; i < len(pairs); i += 2 { selector := pairs[i] if selector == nil { selector = bson.D{} } action.docs = append(action.docs, &updateOp{ Collection: b.c.FullName, Selector: selector, Update: pairs[i+1], Flags: 1, Upsert: true, }) } } // Run runs all the operations queued up. // // If an error is reported on an unordered bulk operation, the error value may // be an aggregation of all issues observed. As an exception to that, Insert // operations running on MongoDB versions prior to 2.6 will report the last // error only due to a limitation in the wire protocol. func (b *Bulk) Run() (*BulkResult, error) { var result BulkResult var berr BulkError var failed bool for i := range b.actions { action := &b.actions[i] var ok bool switch action.op { case bulkInsert: ok = b.runInsert(action, &result, &berr) case bulkUpdate: ok = b.runUpdate(action, &result, &berr) case bulkRemove: ok = b.runRemove(action, &result, &berr) default: panic("unknown bulk operation") } if !ok { failed = true if b.ordered { break } } } if failed { sort.Sort(bulkErrorCases(berr.ecases)) return nil, &berr } return &result, nil } func (b *Bulk) runInsert(action *bulkAction, result *BulkResult, berr *BulkError) bool { op := &insertOp{b.c.FullName, action.docs, 0} if !b.ordered { op.flags = 1 // ContinueOnError } lerr, err := b.c.writeOp(op, b.ordered) return b.checkSuccess(action, berr, lerr, err) } func (b *Bulk) runUpdate(action *bulkAction, result *BulkResult, berr *BulkError) bool { lerr, err := b.c.writeOp(bulkUpdateOp(action.docs), b.ordered) if lerr != nil { result.Matched += lerr.N result.Modified += lerr.modified } return b.checkSuccess(action, berr, lerr, err) } func (b *Bulk) runRemove(action *bulkAction, result *BulkResult, berr *BulkError) bool { lerr, err := b.c.writeOp(bulkDeleteOp(action.docs), b.ordered) if lerr != nil { result.Matched += lerr.N result.Modified += lerr.modified } return b.checkSuccess(action, berr, lerr, err) } func (b *Bulk) checkSuccess(action *bulkAction, berr *BulkError, lerr *LastError, err error) bool { if lerr != nil && len(lerr.ecases) > 0 { for i := 0; i < len(lerr.ecases); i++ { // Map back from the local error index into the visible one. ecase := lerr.ecases[i] idx := ecase.Index if idx >= 0 { idx = action.idxs[idx] } berr.ecases = append(berr.ecases, BulkErrorCase{idx, ecase.Err}) } return false } else if err != nil { for i := 0; i < len(action.idxs); i++ { berr.ecases = append(berr.ecases, BulkErrorCase{action.idxs[i], err}) } return false } return true } mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/bulk_test.go000066400000000000000000000333611276133741700235110ustar00rootroot00000000000000// mgo - MongoDB driver for Go // // Copyright (c) 2010-2015 - Gustavo Niemeyer // // All rights reserved. // // Redistribution and use in source and binary forms, with or without // modification, are permitted provided that the following conditions are met: // // 1. Redistributions of source code must retain the above copyright notice, this // list of conditions and the following disclaimer. // 2. Redistributions in binary form must reproduce the above copyright notice, // this list of conditions and the following disclaimer in the documentation // and/or other materials provided with the distribution. // // THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND // ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED // WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE // DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR // ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES // (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; // LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND // ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT // (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS // SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. package mgo_test import ( . "gopkg.in/check.v1" "gopkg.in/mgo.v2" ) func (s *S) TestBulkInsert(c *C) { session, err := mgo.Dial("localhost:40001") c.Assert(err, IsNil) defer session.Close() coll := session.DB("mydb").C("mycoll") bulk := coll.Bulk() bulk.Insert(M{"n": 1}) bulk.Insert(M{"n": 2}, M{"n": 3}) r, err := bulk.Run() c.Assert(err, IsNil) c.Assert(r, FitsTypeOf, &mgo.BulkResult{}) type doc struct{ N int } var res []doc err = coll.Find(nil).Sort("n").All(&res) c.Assert(err, IsNil) c.Assert(res, DeepEquals, []doc{{1}, {2}, {3}}) } func (s *S) TestBulkInsertError(c *C) { session, err := mgo.Dial("localhost:40001") c.Assert(err, IsNil) defer session.Close() coll := session.DB("mydb").C("mycoll") bulk := coll.Bulk() bulk.Insert(M{"_id": 1}, M{"_id": 2}, M{"_id": 2}, M{"_id": 3}) _, err = bulk.Run() c.Assert(err, ErrorMatches, ".*duplicate key.*") c.Assert(mgo.IsDup(err), Equals, true) type doc struct { N int `_id` } var res []doc err = coll.Find(nil).Sort("_id").All(&res) c.Assert(err, IsNil) c.Assert(res, DeepEquals, []doc{{1}, {2}}) } func (s *S) TestBulkInsertErrorUnordered(c *C) { session, err := mgo.Dial("localhost:40001") c.Assert(err, IsNil) defer session.Close() coll := session.DB("mydb").C("mycoll") bulk := coll.Bulk() bulk.Unordered() bulk.Insert(M{"_id": 1}, M{"_id": 2}, M{"_id": 2}, M{"_id": 3}) _, err = bulk.Run() c.Assert(err, ErrorMatches, ".*duplicate key.*") type doc struct { N int `_id` } var res []doc err = coll.Find(nil).Sort("_id").All(&res) c.Assert(err, IsNil) c.Assert(res, DeepEquals, []doc{{1}, {2}, {3}}) } func (s *S) TestBulkInsertErrorUnorderedSplitBatch(c *C) { // The server has a batch limit of 1000 documents when using write commands. // This artificial limit did not exist with the old wire protocol, so to // avoid compatibility issues the implementation internally split batches // into the proper size and delivers them one by one. This test ensures that // the behavior of unordered (that is, continue on error) remains correct // when errors happen and there are batches left. session, err := mgo.Dial("localhost:40001") c.Assert(err, IsNil) defer session.Close() coll := session.DB("mydb").C("mycoll") bulk := coll.Bulk() bulk.Unordered() const total = 4096 type doc struct { Id int `_id` } docs := make([]interface{}, total) for i := 0; i < total; i++ { docs[i] = doc{i} } docs[1] = doc{0} bulk.Insert(docs...) _, err = bulk.Run() c.Assert(err, ErrorMatches, ".*duplicate key.*") n, err := coll.Count() c.Assert(err, IsNil) c.Assert(n, Equals, total-1) var res doc err = coll.FindId(1500).One(&res) c.Assert(err, IsNil) c.Assert(res.Id, Equals, 1500) } func (s *S) TestBulkErrorString(c *C) { session, err := mgo.Dial("localhost:40001") c.Assert(err, IsNil) defer session.Close() coll := session.DB("mydb").C("mycoll") // If it's just the same string multiple times, join it into a single message. bulk := coll.Bulk() bulk.Unordered() bulk.Insert(M{"_id": 1}, M{"_id": 2}, M{"_id": 2}) _, err = bulk.Run() c.Assert(err, ErrorMatches, ".*duplicate key.*") c.Assert(err, Not(ErrorMatches), ".*duplicate key.*duplicate key") c.Assert(mgo.IsDup(err), Equals, true) // With matching errors but different messages, present them all. bulk = coll.Bulk() bulk.Unordered() bulk.Insert(M{"_id": "dupone"}, M{"_id": "dupone"}, M{"_id": "duptwo"}, M{"_id": "duptwo"}) _, err = bulk.Run() if s.versionAtLeast(2, 6) { c.Assert(err, ErrorMatches, "multiple errors in bulk operation:\n( - .*duplicate.*\n){2}$") c.Assert(err, ErrorMatches, "(?s).*dupone.*") c.Assert(err, ErrorMatches, "(?s).*duptwo.*") } else { // Wire protocol query doesn't return all errors. c.Assert(err, ErrorMatches, ".*duplicate.*") } c.Assert(mgo.IsDup(err), Equals, true) // With mixed errors, present them all. bulk = coll.Bulk() bulk.Unordered() bulk.Insert(M{"_id": 1}, M{"_id": []int{2}}) _, err = bulk.Run() if s.versionAtLeast(2, 6) { c.Assert(err, ErrorMatches, "multiple errors in bulk operation:\n - .*duplicate.*\n - .*array.*\n$") } else { // Wire protocol query doesn't return all errors. c.Assert(err, ErrorMatches, ".*array.*") } c.Assert(mgo.IsDup(err), Equals, false) } func (s *S) TestBulkErrorCases_2_6(c *C) { if !s.versionAtLeast(2, 6) { c.Skip("2.4- has poor bulk reporting") } session, err := mgo.Dial("localhost:40001") c.Assert(err, IsNil) defer session.Close() coll := session.DB("mydb").C("mycoll") bulk := coll.Bulk() bulk.Unordered() // There's a limit of 1000 operations per command, so // this forces the more complex indexing logic to act. for i := 0; i < 1010; i++ { switch i { case 3, 14: bulk.Insert(M{"_id": "dupone"}) case 5, 106: bulk.Update(M{"_id": i - 1}, M{"$set": M{"_id": 4}}) case 7, 1008: bulk.Insert(M{"_id": "duptwo"}) default: bulk.Insert(M{"_id": i}) } } _, err = bulk.Run() ecases := err.(*mgo.BulkError).Cases() c.Check(ecases[0].Err, ErrorMatches, ".*duplicate.*dupone.*") c.Check(ecases[0].Index, Equals, 14) c.Check(ecases[1].Err, ErrorMatches, ".*update.*_id.*") c.Check(ecases[1].Index, Equals, 106) c.Check(ecases[2].Err, ErrorMatches, ".*duplicate.*duptwo.*") c.Check(ecases[2].Index, Equals, 1008) } func (s *S) TestBulkErrorCases_2_4(c *C) { if s.versionAtLeast(2, 6) { c.Skip("2.6+ has better reporting") } session, err := mgo.Dial("localhost:40001") c.Assert(err, IsNil) defer session.Close() coll := session.DB("mydb").C("mycoll") bulk := coll.Bulk() bulk.Unordered() // There's a limit of 1000 operations per command, so // this forces the more complex indexing logic to act. for i := 0; i < 1010; i++ { switch i { case 3, 14: bulk.Insert(M{"_id": "dupone"}) case 5: bulk.Update(M{"_id": i - 1}, M{"$set": M{"n": 4}}) case 106: bulk.Update(M{"_id": i - 1}, M{"$bogus": M{"n": 4}}) case 7, 1008: bulk.Insert(M{"_id": "duptwo"}) default: bulk.Insert(M{"_id": i}) } } _, err = bulk.Run() ecases := err.(*mgo.BulkError).Cases() c.Check(ecases[0].Err, ErrorMatches, ".*duplicate.*duptwo.*") c.Check(ecases[0].Index, Equals, -1) c.Check(ecases[1].Err, ErrorMatches, `.*\$bogus.*`) c.Check(ecases[1].Index, Equals, 106) } func (s *S) TestBulkErrorCasesOrdered(c *C) { session, err := mgo.Dial("localhost:40001") c.Assert(err, IsNil) defer session.Close() coll := session.DB("mydb").C("mycoll") bulk := coll.Bulk() // There's a limit of 1000 operations per command, so // this forces the more complex indexing logic to act. for i := 0; i < 20; i++ { switch i { case 3, 14: bulk.Insert(M{"_id": "dupone"}) case 7, 17: bulk.Insert(M{"_id": "duptwo"}) default: bulk.Insert(M{"_id": i}) } } _, err = bulk.Run() ecases := err.(*mgo.BulkError).Cases() c.Check(ecases[0].Err, ErrorMatches, ".*duplicate.*dupone.*") if s.versionAtLeast(2, 6) { c.Check(ecases[0].Index, Equals, 14) } else { c.Check(ecases[0].Index, Equals, -1) } c.Check(ecases, HasLen, 1) } func (s *S) TestBulkUpdate(c *C) { session, err := mgo.Dial("localhost:40001") c.Assert(err, IsNil) defer session.Close() coll := session.DB("mydb").C("mycoll") err = coll.Insert(M{"n": 1}, M{"n": 2}, M{"n": 3}) c.Assert(err, IsNil) bulk := coll.Bulk() bulk.Update(M{"n": 1}, M{"$set": M{"n": 1}}) bulk.Update(M{"n": 2}, M{"$set": M{"n": 20}}) bulk.Update(M{"n": 5}, M{"$set": M{"n": 50}}) // Won't match. bulk.Update(M{"n": 1}, M{"$set": M{"n": 10}}, M{"n": 3}, M{"$set": M{"n": 30}}) r, err := bulk.Run() c.Assert(err, IsNil) c.Assert(r.Matched, Equals, 4) if s.versionAtLeast(2, 6) { c.Assert(r.Modified, Equals, 3) } type doc struct{ N int } var res []doc err = coll.Find(nil).Sort("n").All(&res) c.Assert(err, IsNil) c.Assert(res, DeepEquals, []doc{{10}, {20}, {30}}) } func (s *S) TestBulkUpdateError(c *C) { session, err := mgo.Dial("localhost:40001") c.Assert(err, IsNil) defer session.Close() coll := session.DB("mydb").C("mycoll") err = coll.Insert(M{"n": 1}, M{"n": 2}, M{"n": 3}) c.Assert(err, IsNil) bulk := coll.Bulk() bulk.Update( M{"n": 1}, M{"$set": M{"n": 10}}, M{"n": 2}, M{"$set": M{"n": 20, "_id": 20}}, M{"n": 3}, M{"$set": M{"n": 30}}, ) r, err := bulk.Run() c.Assert(err, ErrorMatches, ".*_id.*") c.Assert(r, FitsTypeOf, &mgo.BulkResult{}) type doc struct{ N int } var res []doc err = coll.Find(nil).Sort("n").All(&res) c.Assert(err, IsNil) c.Assert(res, DeepEquals, []doc{{2}, {3}, {10}}) } func (s *S) TestBulkUpdateErrorUnordered(c *C) { session, err := mgo.Dial("localhost:40001") c.Assert(err, IsNil) defer session.Close() coll := session.DB("mydb").C("mycoll") err = coll.Insert(M{"n": 1}, M{"n": 2}, M{"n": 3}) c.Assert(err, IsNil) bulk := coll.Bulk() bulk.Unordered() bulk.Update( M{"n": 1}, M{"$set": M{"n": 10}}, M{"n": 2}, M{"$set": M{"n": 20, "_id": 20}}, M{"n": 3}, M{"$set": M{"n": 30}}, ) r, err := bulk.Run() c.Assert(err, ErrorMatches, ".*_id.*") c.Assert(r, FitsTypeOf, &mgo.BulkResult{}) type doc struct{ N int } var res []doc err = coll.Find(nil).Sort("n").All(&res) c.Assert(err, IsNil) c.Assert(res, DeepEquals, []doc{{2}, {10}, {30}}) } func (s *S) TestBulkUpdateAll(c *C) { session, err := mgo.Dial("localhost:40001") c.Assert(err, IsNil) defer session.Close() coll := session.DB("mydb").C("mycoll") err = coll.Insert(M{"n": 1}, M{"n": 2}, M{"n": 3}) c.Assert(err, IsNil) bulk := coll.Bulk() bulk.UpdateAll(M{"n": 1}, M{"$set": M{"n": 10}}) bulk.UpdateAll(M{"n": 2}, M{"$set": M{"n": 2}}) // Won't change. bulk.UpdateAll(M{"n": 5}, M{"$set": M{"n": 50}}) // Won't match. bulk.UpdateAll(M{}, M{"$inc": M{"n": 1}}, M{"n": 11}, M{"$set": M{"n": 5}}) r, err := bulk.Run() c.Assert(err, IsNil) c.Assert(r.Matched, Equals, 6) if s.versionAtLeast(2, 6) { c.Assert(r.Modified, Equals, 5) } type doc struct{ N int } var res []doc err = coll.Find(nil).Sort("n").All(&res) c.Assert(err, IsNil) c.Assert(res, DeepEquals, []doc{{3}, {4}, {5}}) } func (s *S) TestBulkMixedUnordered(c *C) { session, err := mgo.Dial("localhost:40001") c.Assert(err, IsNil) defer session.Close() coll := session.DB("mydb").C("mycoll") // Abuse undefined behavior to ensure the desired implementation is in place. bulk := coll.Bulk() bulk.Unordered() bulk.Insert(M{"n": 1}) bulk.Update(M{"n": 2}, M{"$inc": M{"n": 1}}) bulk.Insert(M{"n": 2}) bulk.Update(M{"n": 3}, M{"$inc": M{"n": 1}}) bulk.Update(M{"n": 1}, M{"$inc": M{"n": 1}}) bulk.Insert(M{"n": 3}) r, err := bulk.Run() c.Assert(err, IsNil) c.Assert(r.Matched, Equals, 3) if s.versionAtLeast(2, 6) { c.Assert(r.Modified, Equals, 3) } type doc struct{ N int } var res []doc err = coll.Find(nil).Sort("n").All(&res) c.Assert(err, IsNil) c.Assert(res, DeepEquals, []doc{{2}, {3}, {4}}) } func (s *S) TestBulkUpsert(c *C) { session, err := mgo.Dial("localhost:40001") c.Assert(err, IsNil) defer session.Close() coll := session.DB("mydb").C("mycoll") err = coll.Insert(M{"n": 1}, M{"n": 2}, M{"n": 3}) c.Assert(err, IsNil) bulk := coll.Bulk() bulk.Upsert(M{"n": 2}, M{"$set": M{"n": 20}}) bulk.Upsert(M{"n": 4}, M{"$set": M{"n": 40}}, M{"n": 3}, M{"$set": M{"n": 30}}) r, err := bulk.Run() c.Assert(err, IsNil) c.Assert(r, FitsTypeOf, &mgo.BulkResult{}) type doc struct{ N int } var res []doc err = coll.Find(nil).Sort("n").All(&res) c.Assert(err, IsNil) c.Assert(res, DeepEquals, []doc{{1}, {20}, {30}, {40}}) } func (s *S) TestBulkRemove(c *C) { session, err := mgo.Dial("localhost:40001") c.Assert(err, IsNil) defer session.Close() coll := session.DB("mydb").C("mycoll") err = coll.Insert(M{"n": 1}, M{"n": 2}, M{"n": 3}, M{"n": 4}, M{"n": 4}) c.Assert(err, IsNil) bulk := coll.Bulk() bulk.Remove(M{"n": 1}) bulk.Remove(M{"n": 2}, M{"n": 4}) r, err := bulk.Run() c.Assert(err, IsNil) c.Assert(r.Matched, Equals, 3) type doc struct{ N int } var res []doc err = coll.Find(nil).Sort("n").All(&res) c.Assert(err, IsNil) c.Assert(res, DeepEquals, []doc{{3}, {4}}) } func (s *S) TestBulkRemoveAll(c *C) { session, err := mgo.Dial("localhost:40001") c.Assert(err, IsNil) defer session.Close() coll := session.DB("mydb").C("mycoll") err = coll.Insert(M{"n": 1}, M{"n": 2}, M{"n": 3}, M{"n": 4}, M{"n": 4}) c.Assert(err, IsNil) bulk := coll.Bulk() bulk.RemoveAll(M{"n": 1}) bulk.RemoveAll(M{"n": 2}, M{"n": 4}) r, err := bulk.Run() c.Assert(err, IsNil) c.Assert(r.Matched, Equals, 4) type doc struct{ N int } var res []doc err = coll.Find(nil).Sort("n").All(&res) c.Assert(err, IsNil) c.Assert(res, DeepEquals, []doc{{3}}) } mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/cluster.go000066400000000000000000000455611276133741700232030ustar00rootroot00000000000000// mgo - MongoDB driver for Go // // Copyright (c) 2010-2012 - Gustavo Niemeyer // // All rights reserved. // // Redistribution and use in source and binary forms, with or without // modification, are permitted provided that the following conditions are met: // // 1. Redistributions of source code must retain the above copyright notice, this // list of conditions and the following disclaimer. // 2. Redistributions in binary form must reproduce the above copyright notice, // this list of conditions and the following disclaimer in the documentation // and/or other materials provided with the distribution. // // THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND // ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED // WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE // DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR // ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES // (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; // LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND // ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT // (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS // SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. package mgo import ( "errors" "fmt" "net" "strconv" "strings" "sync" "time" "gopkg.in/mgo.v2/bson" ) // --------------------------------------------------------------------------- // Mongo cluster encapsulation. // // A cluster enables the communication with one or more servers participating // in a mongo cluster. This works with individual servers, a replica set, // a replica pair, one or multiple mongos routers, etc. type mongoCluster struct { sync.RWMutex serverSynced sync.Cond userSeeds []string dynaSeeds []string servers mongoServers masters mongoServers references int syncing bool direct bool failFast bool syncCount uint setName string cachedIndex map[string]bool sync chan bool dial dialer } func newCluster(userSeeds []string, direct, failFast bool, dial dialer, setName string) *mongoCluster { cluster := &mongoCluster{ userSeeds: userSeeds, references: 1, direct: direct, failFast: failFast, dial: dial, setName: setName, } cluster.serverSynced.L = cluster.RWMutex.RLocker() cluster.sync = make(chan bool, 1) stats.cluster(+1) go cluster.syncServersLoop() return cluster } // Acquire increases the reference count for the cluster. func (cluster *mongoCluster) Acquire() { cluster.Lock() cluster.references++ debugf("Cluster %p acquired (refs=%d)", cluster, cluster.references) cluster.Unlock() } // Release decreases the reference count for the cluster. Once // it reaches zero, all servers will be closed. func (cluster *mongoCluster) Release() { cluster.Lock() if cluster.references == 0 { panic("cluster.Release() with references == 0") } cluster.references-- debugf("Cluster %p released (refs=%d)", cluster, cluster.references) if cluster.references == 0 { for _, server := range cluster.servers.Slice() { server.Close() } // Wake up the sync loop so it can die. cluster.syncServers() stats.cluster(-1) } cluster.Unlock() } func (cluster *mongoCluster) LiveServers() (servers []string) { cluster.RLock() for _, serv := range cluster.servers.Slice() { servers = append(servers, serv.Addr) } cluster.RUnlock() return servers } func (cluster *mongoCluster) removeServer(server *mongoServer) { cluster.Lock() cluster.masters.Remove(server) other := cluster.servers.Remove(server) cluster.Unlock() if other != nil { other.Close() log("Removed server ", server.Addr, " from cluster.") } server.Close() } type isMasterResult struct { IsMaster bool Secondary bool Primary string Hosts []string Passives []string Tags bson.D Msg string SetName string `bson:"setName"` MaxWireVersion int `bson:"maxWireVersion"` } func (cluster *mongoCluster) isMaster(socket *mongoSocket, result *isMasterResult) error { // Monotonic let's it talk to a slave and still hold the socket. session := newSession(Monotonic, cluster, 10*time.Second) session.setSocket(socket) err := session.Run("ismaster", result) session.Close() return err } type possibleTimeout interface { Timeout() bool } var syncSocketTimeout = 5 * time.Second func (cluster *mongoCluster) syncServer(server *mongoServer) (info *mongoServerInfo, hosts []string, err error) { var syncTimeout time.Duration if raceDetector { // This variable is only ever touched by tests. globalMutex.Lock() syncTimeout = syncSocketTimeout globalMutex.Unlock() } else { syncTimeout = syncSocketTimeout } addr := server.Addr log("SYNC Processing ", addr, "...") // Retry a few times to avoid knocking a server down for a hiccup. var result isMasterResult var tryerr error for retry := 0; ; retry++ { if retry == 3 || retry == 1 && cluster.failFast { return nil, nil, tryerr } if retry > 0 { // Don't abuse the server needlessly if there's something actually wrong. if err, ok := tryerr.(possibleTimeout); ok && err.Timeout() { // Give a chance for waiters to timeout as well. cluster.serverSynced.Broadcast() } time.Sleep(syncShortDelay) } // It's not clear what would be a good timeout here. Is it // better to wait longer or to retry? socket, _, err := server.AcquireSocket(0, syncTimeout) if err != nil { tryerr = err logf("SYNC Failed to get socket to %s: %v", addr, err) continue } err = cluster.isMaster(socket, &result) socket.Release() if err != nil { tryerr = err logf("SYNC Command 'ismaster' to %s failed: %v", addr, err) continue } debugf("SYNC Result of 'ismaster' from %s: %#v", addr, result) break } if cluster.setName != "" && result.SetName != cluster.setName { logf("SYNC Server %s is not a member of replica set %q", addr, cluster.setName) return nil, nil, fmt.Errorf("server %s is not a member of replica set %q", addr, cluster.setName) } if result.IsMaster { debugf("SYNC %s is a master.", addr) if !server.info.Master { // Made an incorrect assumption above, so fix stats. stats.conn(-1, false) stats.conn(+1, true) } } else if result.Secondary { debugf("SYNC %s is a slave.", addr) } else if cluster.direct { logf("SYNC %s in unknown state. Pretending it's a slave due to direct connection.", addr) } else { logf("SYNC %s is neither a master nor a slave.", addr) // Let stats track it as whatever was known before. return nil, nil, errors.New(addr + " is not a master nor slave") } info = &mongoServerInfo{ Master: result.IsMaster, Mongos: result.Msg == "isdbgrid", Tags: result.Tags, SetName: result.SetName, MaxWireVersion: result.MaxWireVersion, } hosts = make([]string, 0, 1+len(result.Hosts)+len(result.Passives)) if result.Primary != "" { // First in the list to speed up master discovery. hosts = append(hosts, result.Primary) } hosts = append(hosts, result.Hosts...) hosts = append(hosts, result.Passives...) debugf("SYNC %s knows about the following peers: %#v", addr, hosts) return info, hosts, nil } type syncKind bool const ( completeSync syncKind = true partialSync syncKind = false ) func (cluster *mongoCluster) addServer(server *mongoServer, info *mongoServerInfo, syncKind syncKind) { cluster.Lock() current := cluster.servers.Search(server.ResolvedAddr) if current == nil { if syncKind == partialSync { cluster.Unlock() server.Close() log("SYNC Discarding unknown server ", server.Addr, " due to partial sync.") return } cluster.servers.Add(server) if info.Master { cluster.masters.Add(server) log("SYNC Adding ", server.Addr, " to cluster as a master.") } else { log("SYNC Adding ", server.Addr, " to cluster as a slave.") } } else { if server != current { panic("addServer attempting to add duplicated server") } if server.Info().Master != info.Master { if info.Master { log("SYNC Server ", server.Addr, " is now a master.") cluster.masters.Add(server) } else { log("SYNC Server ", server.Addr, " is now a slave.") cluster.masters.Remove(server) } } } server.SetInfo(info) debugf("SYNC Broadcasting availability of server %s", server.Addr) cluster.serverSynced.Broadcast() cluster.Unlock() } func (cluster *mongoCluster) getKnownAddrs() []string { cluster.RLock() max := len(cluster.userSeeds) + len(cluster.dynaSeeds) + cluster.servers.Len() seen := make(map[string]bool, max) known := make([]string, 0, max) add := func(addr string) { if _, found := seen[addr]; !found { seen[addr] = true known = append(known, addr) } } for _, addr := range cluster.userSeeds { add(addr) } for _, addr := range cluster.dynaSeeds { add(addr) } for _, serv := range cluster.servers.Slice() { add(serv.Addr) } cluster.RUnlock() return known } // syncServers injects a value into the cluster.sync channel to force // an iteration of the syncServersLoop function. func (cluster *mongoCluster) syncServers() { select { case cluster.sync <- true: default: } } // How long to wait for a checkup of the cluster topology if nothing // else kicks a synchronization before that. const syncServersDelay = 30 * time.Second const syncShortDelay = 500 * time.Millisecond // syncServersLoop loops while the cluster is alive to keep its idea of // the server topology up-to-date. It must be called just once from // newCluster. The loop iterates once syncServersDelay has passed, or // if somebody injects a value into the cluster.sync channel to force a // synchronization. A loop iteration will contact all servers in // parallel, ask them about known peers and their own role within the // cluster, and then attempt to do the same with all the peers // retrieved. func (cluster *mongoCluster) syncServersLoop() { for { debugf("SYNC Cluster %p is starting a sync loop iteration.", cluster) cluster.Lock() if cluster.references == 0 { cluster.Unlock() break } cluster.references++ // Keep alive while syncing. direct := cluster.direct cluster.Unlock() cluster.syncServersIteration(direct) // We just synchronized, so consume any outstanding requests. select { case <-cluster.sync: default: } cluster.Release() // Hold off before allowing another sync. No point in // burning CPU looking for down servers. if !cluster.failFast { time.Sleep(syncShortDelay) } cluster.Lock() if cluster.references == 0 { cluster.Unlock() break } cluster.syncCount++ // Poke all waiters so they have a chance to timeout or // restart syncing if they wish to. cluster.serverSynced.Broadcast() // Check if we have to restart immediately either way. restart := !direct && cluster.masters.Empty() || cluster.servers.Empty() cluster.Unlock() if restart { log("SYNC No masters found. Will synchronize again.") time.Sleep(syncShortDelay) continue } debugf("SYNC Cluster %p waiting for next requested or scheduled sync.", cluster) // Hold off until somebody explicitly requests a synchronization // or it's time to check for a cluster topology change again. select { case <-cluster.sync: case <-time.After(syncServersDelay): } } debugf("SYNC Cluster %p is stopping its sync loop.", cluster) } func (cluster *mongoCluster) server(addr string, tcpaddr *net.TCPAddr) *mongoServer { cluster.RLock() server := cluster.servers.Search(tcpaddr.String()) cluster.RUnlock() if server != nil { return server } return newServer(addr, tcpaddr, cluster.sync, cluster.dial) } func resolveAddr(addr string) (*net.TCPAddr, error) { // Simple cases that do not need actual resolution. Works with IPv4 and v6. if host, port, err := net.SplitHostPort(addr); err == nil { if port, _ := strconv.Atoi(port); port > 0 { zone := "" if i := strings.LastIndex(host, "%"); i >= 0 { zone = host[i+1:] host = host[:i] } ip := net.ParseIP(host) if ip != nil { return &net.TCPAddr{IP: ip, Port: port, Zone: zone}, nil } } } // Attempt to resolve IPv4 and v6 concurrently. addrChan := make(chan *net.TCPAddr, 2) for _, network := range []string{"udp4", "udp6"} { network := network go func() { // The unfortunate UDP dialing hack allows having a timeout on address resolution. conn, err := net.DialTimeout(network, addr, 10*time.Second) if err != nil { addrChan <- nil } else { addrChan <- (*net.TCPAddr)(conn.RemoteAddr().(*net.UDPAddr)) conn.Close() } }() } // Wait for the result of IPv4 and v6 resolution. Use IPv4 if available. tcpaddr := <-addrChan if tcpaddr == nil || len(tcpaddr.IP) != 4 { var timeout <-chan time.Time if tcpaddr != nil { // Don't wait too long if an IPv6 address is known. timeout = time.After(50 * time.Millisecond) } select { case <-timeout: case tcpaddr2 := <-addrChan: if tcpaddr == nil || tcpaddr2 != nil { // It's an IPv4 address or the only known address. Use it. tcpaddr = tcpaddr2 } } } if tcpaddr == nil { log("SYNC Failed to resolve server address: ", addr) return nil, errors.New("failed to resolve server address: " + addr) } if tcpaddr.String() != addr { debug("SYNC Address ", addr, " resolved as ", tcpaddr.String()) } return tcpaddr, nil } type pendingAdd struct { server *mongoServer info *mongoServerInfo } func (cluster *mongoCluster) syncServersIteration(direct bool) { log("SYNC Starting full topology synchronization...") var wg sync.WaitGroup var m sync.Mutex notYetAdded := make(map[string]pendingAdd) addIfFound := make(map[string]bool) seen := make(map[string]bool) syncKind := partialSync var spawnSync func(addr string, byMaster bool) spawnSync = func(addr string, byMaster bool) { wg.Add(1) go func() { defer wg.Done() tcpaddr, err := resolveAddr(addr) if err != nil { log("SYNC Failed to start sync of ", addr, ": ", err.Error()) return } resolvedAddr := tcpaddr.String() m.Lock() if byMaster { if pending, ok := notYetAdded[resolvedAddr]; ok { delete(notYetAdded, resolvedAddr) m.Unlock() cluster.addServer(pending.server, pending.info, completeSync) return } addIfFound[resolvedAddr] = true } if seen[resolvedAddr] { m.Unlock() return } seen[resolvedAddr] = true m.Unlock() server := cluster.server(addr, tcpaddr) info, hosts, err := cluster.syncServer(server) if err != nil { cluster.removeServer(server) return } m.Lock() add := direct || info.Master || addIfFound[resolvedAddr] if add { syncKind = completeSync } else { notYetAdded[resolvedAddr] = pendingAdd{server, info} } m.Unlock() if add { cluster.addServer(server, info, completeSync) } if !direct { for _, addr := range hosts { spawnSync(addr, info.Master) } } }() } knownAddrs := cluster.getKnownAddrs() for _, addr := range knownAddrs { spawnSync(addr, false) } wg.Wait() if syncKind == completeSync { logf("SYNC Synchronization was complete (got data from primary).") for _, pending := range notYetAdded { cluster.removeServer(pending.server) } } else { logf("SYNC Synchronization was partial (cannot talk to primary).") for _, pending := range notYetAdded { cluster.addServer(pending.server, pending.info, partialSync) } } cluster.Lock() mastersLen := cluster.masters.Len() logf("SYNC Synchronization completed: %d master(s) and %d slave(s) alive.", mastersLen, cluster.servers.Len()-mastersLen) // Update dynamic seeds, but only if we have any good servers. Otherwise, // leave them alone for better chances of a successful sync in the future. if syncKind == completeSync { dynaSeeds := make([]string, cluster.servers.Len()) for i, server := range cluster.servers.Slice() { dynaSeeds[i] = server.Addr } cluster.dynaSeeds = dynaSeeds debugf("SYNC New dynamic seeds: %#v\n", dynaSeeds) } cluster.Unlock() } // AcquireSocket returns a socket to a server in the cluster. If slaveOk is // true, it will attempt to return a socket to a slave server. If it is // false, the socket will necessarily be to a master server. func (cluster *mongoCluster) AcquireSocket(mode Mode, slaveOk bool, syncTimeout time.Duration, socketTimeout time.Duration, serverTags []bson.D, poolLimit int) (s *mongoSocket, err error) { var started time.Time var syncCount uint warnedLimit := false for { cluster.RLock() for { mastersLen := cluster.masters.Len() slavesLen := cluster.servers.Len() - mastersLen debugf("Cluster has %d known masters and %d known slaves.", mastersLen, slavesLen) if mastersLen > 0 && !(slaveOk && mode == Secondary) || slavesLen > 0 && slaveOk { break } if mastersLen > 0 && mode == Secondary && cluster.masters.HasMongos() { break } if started.IsZero() { // Initialize after fast path above. started = time.Now() syncCount = cluster.syncCount } else if syncTimeout != 0 && started.Before(time.Now().Add(-syncTimeout)) || cluster.failFast && cluster.syncCount != syncCount { cluster.RUnlock() return nil, errors.New("no reachable servers") } log("Waiting for servers to synchronize...") cluster.syncServers() // Remember: this will release and reacquire the lock. cluster.serverSynced.Wait() } var server *mongoServer if slaveOk { server = cluster.servers.BestFit(mode, serverTags) } else { server = cluster.masters.BestFit(mode, nil) } cluster.RUnlock() if server == nil { // Must have failed the requested tags. Sleep to avoid spinning. time.Sleep(1e8) continue } s, abended, err := server.AcquireSocket(poolLimit, socketTimeout) if err == errPoolLimit { if !warnedLimit { warnedLimit = true log("WARNING: Per-server connection limit reached.") } time.Sleep(100 * time.Millisecond) continue } if err != nil { cluster.removeServer(server) cluster.syncServers() continue } if abended && !slaveOk { var result isMasterResult err := cluster.isMaster(s, &result) if err != nil || !result.IsMaster { logf("Cannot confirm server %s as master (%v)", server.Addr, err) s.Release() cluster.syncServers() time.Sleep(100 * time.Millisecond) continue } } return s, nil } panic("unreached") } func (cluster *mongoCluster) CacheIndex(cacheKey string, exists bool) { cluster.Lock() if cluster.cachedIndex == nil { cluster.cachedIndex = make(map[string]bool) } if exists { cluster.cachedIndex[cacheKey] = true } else { delete(cluster.cachedIndex, cacheKey) } cluster.Unlock() } func (cluster *mongoCluster) HasCachedIndex(cacheKey string) (result bool) { cluster.RLock() if cluster.cachedIndex != nil { result = cluster.cachedIndex[cacheKey] } cluster.RUnlock() return } func (cluster *mongoCluster) ResetIndexCache() { cluster.Lock() cluster.cachedIndex = make(map[string]bool) cluster.Unlock() } mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/cluster_test.go000066400000000000000000001530241276133741700242340ustar00rootroot00000000000000// mgo - MongoDB driver for Go // // Copyright (c) 2010-2012 - Gustavo Niemeyer // // All rights reserved. // // Redistribution and use in source and binary forms, with or without // modification, are permitted provided that the following conditions are met: // // 1. Redistributions of source code must retain the above copyright notice, this // list of conditions and the following disclaimer. // 2. Redistributions in binary form must reproduce the above copyright notice, // this list of conditions and the following disclaimer in the documentation // and/or other materials provided with the distribution. // // THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND // ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED // WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE // DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR // ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES // (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; // LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND // ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT // (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS // SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. package mgo_test import ( "fmt" "io" "net" "strings" "sync" "time" . "gopkg.in/check.v1" "gopkg.in/mgo.v2" "gopkg.in/mgo.v2/bson" ) func (s *S) TestNewSession(c *C) { session, err := mgo.Dial("localhost:40001") c.Assert(err, IsNil) defer session.Close() // Do a dummy operation to wait for connection. coll := session.DB("mydb").C("mycoll") err = coll.Insert(M{"_id": 1}) c.Assert(err, IsNil) // Tweak safety and query settings to ensure other has copied those. session.SetSafe(nil) session.SetBatch(-1) other := session.New() defer other.Close() session.SetSafe(&mgo.Safe{}) // Clone was copied while session was unsafe, so no errors. otherColl := other.DB("mydb").C("mycoll") err = otherColl.Insert(M{"_id": 1}) c.Assert(err, IsNil) // Original session was made safe again. err = coll.Insert(M{"_id": 1}) c.Assert(err, NotNil) // With New(), each session has its own socket now. stats := mgo.GetStats() c.Assert(stats.MasterConns, Equals, 2) c.Assert(stats.SocketsInUse, Equals, 2) // Ensure query parameters were cloned. err = otherColl.Insert(M{"_id": 2}) c.Assert(err, IsNil) // Ping the database to ensure the nonce has been received already. c.Assert(other.Ping(), IsNil) mgo.ResetStats() iter := otherColl.Find(M{}).Iter() c.Assert(err, IsNil) m := M{} ok := iter.Next(m) c.Assert(ok, Equals, true) err = iter.Close() c.Assert(err, IsNil) // If Batch(-1) is in effect, a single document must have been received. stats = mgo.GetStats() c.Assert(stats.ReceivedDocs, Equals, 1) } func (s *S) TestCloneSession(c *C) { session, err := mgo.Dial("localhost:40001") c.Assert(err, IsNil) defer session.Close() // Do a dummy operation to wait for connection. coll := session.DB("mydb").C("mycoll") err = coll.Insert(M{"_id": 1}) c.Assert(err, IsNil) // Tweak safety and query settings to ensure clone is copying those. session.SetSafe(nil) session.SetBatch(-1) clone := session.Clone() defer clone.Close() session.SetSafe(&mgo.Safe{}) // Clone was copied while session was unsafe, so no errors. cloneColl := clone.DB("mydb").C("mycoll") err = cloneColl.Insert(M{"_id": 1}) c.Assert(err, IsNil) // Original session was made safe again. err = coll.Insert(M{"_id": 1}) c.Assert(err, NotNil) // With Clone(), same socket is shared between sessions now. stats := mgo.GetStats() c.Assert(stats.SocketsInUse, Equals, 1) c.Assert(stats.SocketRefs, Equals, 2) // Refreshing one of them should let the original socket go, // while preserving the safety settings. clone.Refresh() err = cloneColl.Insert(M{"_id": 1}) c.Assert(err, IsNil) // Must have used another connection now. stats = mgo.GetStats() c.Assert(stats.SocketsInUse, Equals, 2) c.Assert(stats.SocketRefs, Equals, 2) // Ensure query parameters were cloned. err = cloneColl.Insert(M{"_id": 2}) c.Assert(err, IsNil) // Ping the database to ensure the nonce has been received already. c.Assert(clone.Ping(), IsNil) mgo.ResetStats() iter := cloneColl.Find(M{}).Iter() c.Assert(err, IsNil) m := M{} ok := iter.Next(m) c.Assert(ok, Equals, true) err = iter.Close() c.Assert(err, IsNil) // If Batch(-1) is in effect, a single document must have been received. stats = mgo.GetStats() c.Assert(stats.ReceivedDocs, Equals, 1) } func (s *S) TestModeStrong(c *C) { session, err := mgo.Dial("localhost:40012") c.Assert(err, IsNil) defer session.Close() session.SetMode(mgo.Monotonic, false) session.SetMode(mgo.Strong, false) c.Assert(session.Mode(), Equals, mgo.Strong) result := M{} cmd := session.DB("admin").C("$cmd") err = cmd.Find(M{"ismaster": 1}).One(&result) c.Assert(err, IsNil) c.Assert(result["ismaster"], Equals, true) coll := session.DB("mydb").C("mycoll") err = coll.Insert(M{"a": 1}) c.Assert(err, IsNil) // Wait since the sync also uses sockets. for len(session.LiveServers()) != 3 { c.Log("Waiting for cluster sync to finish...") time.Sleep(5e8) } stats := mgo.GetStats() c.Assert(stats.MasterConns, Equals, 1) c.Assert(stats.SlaveConns, Equals, 2) c.Assert(stats.SocketsInUse, Equals, 1) session.SetMode(mgo.Strong, true) stats = mgo.GetStats() c.Assert(stats.SocketsInUse, Equals, 0) } func (s *S) TestModeMonotonic(c *C) { // Must necessarily connect to a slave, otherwise the // master connection will be available first. session, err := mgo.Dial("localhost:40012") c.Assert(err, IsNil) defer session.Close() session.SetMode(mgo.Monotonic, false) c.Assert(session.Mode(), Equals, mgo.Monotonic) var result struct{ IsMaster bool } cmd := session.DB("admin").C("$cmd") err = cmd.Find(M{"ismaster": 1}).One(&result) c.Assert(err, IsNil) c.Assert(result.IsMaster, Equals, false) coll := session.DB("mydb").C("mycoll") err = coll.Insert(M{"a": 1}) c.Assert(err, IsNil) err = cmd.Find(M{"ismaster": 1}).One(&result) c.Assert(err, IsNil) c.Assert(result.IsMaster, Equals, true) // Wait since the sync also uses sockets. for len(session.LiveServers()) != 3 { c.Log("Waiting for cluster sync to finish...") time.Sleep(5e8) } stats := mgo.GetStats() c.Assert(stats.MasterConns, Equals, 1) c.Assert(stats.SlaveConns, Equals, 2) c.Assert(stats.SocketsInUse, Equals, 2) session.SetMode(mgo.Monotonic, true) stats = mgo.GetStats() c.Assert(stats.SocketsInUse, Equals, 0) } func (s *S) TestModeMonotonicAfterStrong(c *C) { // Test that a strong session shifting to a monotonic // one preserves the socket untouched. session, err := mgo.Dial("localhost:40012") c.Assert(err, IsNil) defer session.Close() // Insert something to force a connection to the master. coll := session.DB("mydb").C("mycoll") err = coll.Insert(M{"a": 1}) c.Assert(err, IsNil) session.SetMode(mgo.Monotonic, false) // Wait since the sync also uses sockets. for len(session.LiveServers()) != 3 { c.Log("Waiting for cluster sync to finish...") time.Sleep(5e8) } // Master socket should still be reserved. stats := mgo.GetStats() c.Assert(stats.SocketsInUse, Equals, 1) // Confirm it's the master even though it's Monotonic by now. result := M{} cmd := session.DB("admin").C("$cmd") err = cmd.Find(M{"ismaster": 1}).One(&result) c.Assert(err, IsNil) c.Assert(result["ismaster"], Equals, true) } func (s *S) TestModeStrongAfterMonotonic(c *C) { // Test that shifting from Monotonic to Strong while // using a slave socket will keep the socket reserved // until the master socket is necessary, so that no // switch over occurs unless it's actually necessary. // Must necessarily connect to a slave, otherwise the // master connection will be available first. session, err := mgo.Dial("localhost:40012") c.Assert(err, IsNil) defer session.Close() session.SetMode(mgo.Monotonic, false) // Ensure we're talking to a slave, and reserve the socket. result := M{} err = session.Run("ismaster", &result) c.Assert(err, IsNil) c.Assert(result["ismaster"], Equals, false) // Switch to a Strong session. session.SetMode(mgo.Strong, false) // Wait since the sync also uses sockets. for len(session.LiveServers()) != 3 { c.Log("Waiting for cluster sync to finish...") time.Sleep(5e8) } // Slave socket should still be reserved. stats := mgo.GetStats() c.Assert(stats.SocketsInUse, Equals, 1) // But any operation will switch it to the master. result = M{} err = session.Run("ismaster", &result) c.Assert(err, IsNil) c.Assert(result["ismaster"], Equals, true) } func (s *S) TestModeMonotonicWriteOnIteration(c *C) { // Must necessarily connect to a slave, otherwise the // master connection will be available first. session, err := mgo.Dial("localhost:40012") c.Assert(err, IsNil) defer session.Close() session.SetMode(mgo.Monotonic, false) c.Assert(session.Mode(), Equals, mgo.Monotonic) coll1 := session.DB("mydb").C("mycoll1") coll2 := session.DB("mydb").C("mycoll2") ns := []int{40, 41, 42, 43, 44, 45, 46} for _, n := range ns { err := coll1.Insert(M{"n": n}) c.Assert(err, IsNil) } // Release master so we can grab a slave again. session.Refresh() // Wait until synchronization is done. for { n, err := coll1.Count() c.Assert(err, IsNil) if n == len(ns) { break } } iter := coll1.Find(nil).Batch(2).Iter() i := 0 m := M{} for iter.Next(&m) { i++ if i > 3 { err := coll2.Insert(M{"n": 47 + i}) c.Assert(err, IsNil) } } c.Assert(i, Equals, len(ns)) } func (s *S) TestModeEventual(c *C) { // Must necessarily connect to a slave, otherwise the // master connection will be available first. session, err := mgo.Dial("localhost:40012") c.Assert(err, IsNil) defer session.Close() session.SetMode(mgo.Eventual, false) c.Assert(session.Mode(), Equals, mgo.Eventual) result := M{} err = session.Run("ismaster", &result) c.Assert(err, IsNil) c.Assert(result["ismaster"], Equals, false) coll := session.DB("mydb").C("mycoll") err = coll.Insert(M{"a": 1}) c.Assert(err, IsNil) result = M{} err = session.Run("ismaster", &result) c.Assert(err, IsNil) c.Assert(result["ismaster"], Equals, false) // Wait since the sync also uses sockets. for len(session.LiveServers()) != 3 { c.Log("Waiting for cluster sync to finish...") time.Sleep(5e8) } stats := mgo.GetStats() c.Assert(stats.MasterConns, Equals, 1) c.Assert(stats.SlaveConns, Equals, 2) c.Assert(stats.SocketsInUse, Equals, 0) } func (s *S) TestModeEventualAfterStrong(c *C) { // Test that a strong session shifting to an eventual // one preserves the socket untouched. session, err := mgo.Dial("localhost:40012") c.Assert(err, IsNil) defer session.Close() // Insert something to force a connection to the master. coll := session.DB("mydb").C("mycoll") err = coll.Insert(M{"a": 1}) c.Assert(err, IsNil) session.SetMode(mgo.Eventual, false) // Wait since the sync also uses sockets. for len(session.LiveServers()) != 3 { c.Log("Waiting for cluster sync to finish...") time.Sleep(5e8) } // Master socket should still be reserved. stats := mgo.GetStats() c.Assert(stats.SocketsInUse, Equals, 1) // Confirm it's the master even though it's Eventual by now. result := M{} cmd := session.DB("admin").C("$cmd") err = cmd.Find(M{"ismaster": 1}).One(&result) c.Assert(err, IsNil) c.Assert(result["ismaster"], Equals, true) session.SetMode(mgo.Eventual, true) stats = mgo.GetStats() c.Assert(stats.SocketsInUse, Equals, 0) } func (s *S) TestModeStrongFallover(c *C) { if *fast { c.Skip("-fast") } session, err := mgo.Dial("localhost:40021") c.Assert(err, IsNil) defer session.Close() // With strong consistency, this will open a socket to the master. result := &struct{ Host string }{} err = session.Run("serverStatus", result) c.Assert(err, IsNil) // Kill the master. host := result.Host s.Stop(host) // This must fail, since the connection was broken. err = session.Run("serverStatus", result) c.Assert(err, Equals, io.EOF) // With strong consistency, it fails again until reset. err = session.Run("serverStatus", result) c.Assert(err, Equals, io.EOF) session.Refresh() // Now we should be able to talk to the new master. // Increase the timeout since this may take quite a while. session.SetSyncTimeout(3 * time.Minute) err = session.Run("serverStatus", result) c.Assert(err, IsNil) c.Assert(result.Host, Not(Equals), host) // Insert some data to confirm it's indeed a master. err = session.DB("mydb").C("mycoll").Insert(M{"n": 42}) c.Assert(err, IsNil) } func (s *S) TestModePrimaryHiccup(c *C) { if *fast { c.Skip("-fast") } session, err := mgo.Dial("localhost:40021") c.Assert(err, IsNil) defer session.Close() // With strong consistency, this will open a socket to the master. result := &struct{ Host string }{} err = session.Run("serverStatus", result) c.Assert(err, IsNil) // Establish a few extra sessions to create spare sockets to // the master. This increases a bit the chances of getting an // incorrect cached socket. var sessions []*mgo.Session for i := 0; i < 20; i++ { sessions = append(sessions, session.Copy()) err = sessions[len(sessions)-1].Run("serverStatus", result) c.Assert(err, IsNil) } for i := range sessions { sessions[i].Close() } // Kill the master, but bring it back immediatelly. host := result.Host s.Stop(host) s.StartAll() // This must fail, since the connection was broken. err = session.Run("serverStatus", result) c.Assert(err, Equals, io.EOF) // With strong consistency, it fails again until reset. err = session.Run("serverStatus", result) c.Assert(err, Equals, io.EOF) session.Refresh() // Now we should be able to talk to the new master. // Increase the timeout since this may take quite a while. session.SetSyncTimeout(3 * time.Minute) // Insert some data to confirm it's indeed a master. err = session.DB("mydb").C("mycoll").Insert(M{"n": 42}) c.Assert(err, IsNil) } func (s *S) TestModeMonotonicFallover(c *C) { if *fast { c.Skip("-fast") } session, err := mgo.Dial("localhost:40021") c.Assert(err, IsNil) defer session.Close() session.SetMode(mgo.Monotonic, true) // Insert something to force a switch to the master. coll := session.DB("mydb").C("mycoll") err = coll.Insert(M{"a": 1}) c.Assert(err, IsNil) // Wait a bit for this to be synchronized to slaves. time.Sleep(3 * time.Second) result := &struct{ Host string }{} err = session.Run("serverStatus", result) c.Assert(err, IsNil) // Kill the master. host := result.Host s.Stop(host) // This must fail, since the connection was broken. err = session.Run("serverStatus", result) c.Assert(err, Equals, io.EOF) // With monotonic consistency, it fails again until reset. err = session.Run("serverStatus", result) c.Assert(err, Equals, io.EOF) session.Refresh() // Now we should be able to talk to the new master. err = session.Run("serverStatus", result) c.Assert(err, IsNil) c.Assert(result.Host, Not(Equals), host) } func (s *S) TestModeMonotonicWithSlaveFallover(c *C) { if *fast { c.Skip("-fast") } session, err := mgo.Dial("localhost:40021") c.Assert(err, IsNil) defer session.Close() ssresult := &struct{ Host string }{} imresult := &struct{ IsMaster bool }{} // Figure the master while still using the strong session. err = session.Run("serverStatus", ssresult) c.Assert(err, IsNil) err = session.Run("isMaster", imresult) c.Assert(err, IsNil) master := ssresult.Host c.Assert(imresult.IsMaster, Equals, true, Commentf("%s is not the master", master)) // Create new monotonic session with an explicit address to ensure // a slave is synchronized before the master, otherwise a connection // with the master may be used below for lack of other options. var addr string switch { case strings.HasSuffix(ssresult.Host, ":40021"): addr = "localhost:40022" case strings.HasSuffix(ssresult.Host, ":40022"): addr = "localhost:40021" case strings.HasSuffix(ssresult.Host, ":40023"): addr = "localhost:40021" default: c.Fatal("Unknown host: ", ssresult.Host) } session, err = mgo.Dial(addr) c.Assert(err, IsNil) defer session.Close() session.SetMode(mgo.Monotonic, true) // Check the address of the socket associated with the monotonic session. c.Log("Running serverStatus and isMaster with monotonic session") err = session.Run("serverStatus", ssresult) c.Assert(err, IsNil) err = session.Run("isMaster", imresult) c.Assert(err, IsNil) slave := ssresult.Host c.Assert(imresult.IsMaster, Equals, false, Commentf("%s is not a slave", slave)) c.Assert(master, Not(Equals), slave) // Kill the master. s.Stop(master) // Session must still be good, since we were talking to a slave. err = session.Run("serverStatus", ssresult) c.Assert(err, IsNil) c.Assert(ssresult.Host, Equals, slave, Commentf("Monotonic session moved from %s to %s", slave, ssresult.Host)) // If we try to insert something, it'll have to hold until the new // master is available to move the connection, and work correctly. coll := session.DB("mydb").C("mycoll") err = coll.Insert(M{"a": 1}) c.Assert(err, IsNil) // Must now be talking to the new master. err = session.Run("serverStatus", ssresult) c.Assert(err, IsNil) err = session.Run("isMaster", imresult) c.Assert(err, IsNil) c.Assert(imresult.IsMaster, Equals, true, Commentf("%s is not the master", master)) // ... which is not the old one, since it's still dead. c.Assert(ssresult.Host, Not(Equals), master) } func (s *S) TestModeEventualFallover(c *C) { if *fast { c.Skip("-fast") } session, err := mgo.Dial("localhost:40021") c.Assert(err, IsNil) defer session.Close() result := &struct{ Host string }{} err = session.Run("serverStatus", result) c.Assert(err, IsNil) master := result.Host session.SetMode(mgo.Eventual, true) // Should connect to the master when needed. coll := session.DB("mydb").C("mycoll") err = coll.Insert(M{"a": 1}) c.Assert(err, IsNil) // Wait a bit for this to be synchronized to slaves. time.Sleep(3 * time.Second) // Kill the master. s.Stop(master) // Should still work, with the new master now. coll = session.DB("mydb").C("mycoll") err = coll.Insert(M{"a": 1}) c.Assert(err, IsNil) err = session.Run("serverStatus", result) c.Assert(err, IsNil) c.Assert(result.Host, Not(Equals), master) } func (s *S) TestModeSecondaryJustPrimary(c *C) { if *fast { c.Skip("-fast") } session, err := mgo.Dial("localhost:40001") c.Assert(err, IsNil) defer session.Close() session.SetMode(mgo.Secondary, true) err = session.Ping() c.Assert(err, ErrorMatches, "no reachable servers") } func (s *S) TestModeSecondaryPreferredJustPrimary(c *C) { if *fast { c.Skip("-fast") } session, err := mgo.Dial("localhost:40001") c.Assert(err, IsNil) defer session.Close() session.SetMode(mgo.SecondaryPreferred, true) result := &struct{ Host string }{} err = session.Run("serverStatus", result) c.Assert(err, IsNil) } func (s *S) TestModeSecondaryPreferredFallover(c *C) { if *fast { c.Skip("-fast") } session, err := mgo.Dial("localhost:40011") c.Assert(err, IsNil) defer session.Close() // Ensure secondaries are available for being picked up. for len(session.LiveServers()) != 3 { c.Log("Waiting for cluster sync to finish...") time.Sleep(5e8) } session.SetMode(mgo.SecondaryPreferred, true) result := &struct{ Host string }{} err = session.Run("serverStatus", result) c.Assert(err, IsNil) c.Assert(supvName(result.Host), Not(Equals), "rs1a") secondary := result.Host // Should connect to the primary when needed. coll := session.DB("mydb").C("mycoll") err = coll.Insert(M{"a": 1}) c.Assert(err, IsNil) // Wait a bit for this to be synchronized to slaves. time.Sleep(3 * time.Second) // Kill the primary. s.Stop("localhost:40011") // It can still talk to the selected secondary. err = session.Run("serverStatus", result) c.Assert(err, IsNil) c.Assert(result.Host, Equals, secondary) // But cannot speak to the primary until reset. coll = session.DB("mydb").C("mycoll") err = coll.Insert(M{"a": 1}) c.Assert(err, Equals, io.EOF) session.Refresh() // Can still talk to a secondary. err = session.Run("serverStatus", result) c.Assert(err, IsNil) c.Assert(supvName(result.Host), Not(Equals), "rs1a") s.StartAll() // Should now be able to talk to the primary again. coll = session.DB("mydb").C("mycoll") err = coll.Insert(M{"a": 1}) c.Assert(err, IsNil) } func (s *S) TestModePrimaryPreferredFallover(c *C) { if *fast { c.Skip("-fast") } session, err := mgo.Dial("localhost:40011") c.Assert(err, IsNil) defer session.Close() session.SetMode(mgo.PrimaryPreferred, true) result := &struct{ Host string }{} err = session.Run("serverStatus", result) c.Assert(err, IsNil) c.Assert(supvName(result.Host), Equals, "rs1a") // Kill the primary. s.Stop("localhost:40011") // Should now fail as there was a primary socket in use already. err = session.Run("serverStatus", result) c.Assert(err, Equals, io.EOF) // Refresh so the reserved primary socket goes away. session.Refresh() // Should be able to talk to the secondary. err = session.Run("serverStatus", result) c.Assert(err, IsNil) s.StartAll() // Should wait for the new primary to become available. coll := session.DB("mydb").C("mycoll") err = coll.Insert(M{"a": 1}) c.Assert(err, IsNil) // And should use the new primary in general, as it is preferred. err = session.Run("serverStatus", result) c.Assert(err, IsNil) c.Assert(supvName(result.Host), Equals, "rs1a") } func (s *S) TestModePrimaryFallover(c *C) { if *fast { c.Skip("-fast") } session, err := mgo.Dial("localhost:40011") c.Assert(err, IsNil) defer session.Close() session.SetSyncTimeout(3 * time.Second) session.SetMode(mgo.Primary, true) result := &struct{ Host string }{} err = session.Run("serverStatus", result) c.Assert(err, IsNil) c.Assert(supvName(result.Host), Equals, "rs1a") // Kill the primary. s.Stop("localhost:40011") session.Refresh() err = session.Ping() c.Assert(err, ErrorMatches, "no reachable servers") } func (s *S) TestModeSecondary(c *C) { if *fast { c.Skip("-fast") } session, err := mgo.Dial("localhost:40011") c.Assert(err, IsNil) defer session.Close() session.SetMode(mgo.Secondary, true) result := &struct{ Host string }{} err = session.Run("serverStatus", result) c.Assert(err, IsNil) c.Assert(supvName(result.Host), Not(Equals), "rs1a") secondary := result.Host coll := session.DB("mydb").C("mycoll") err = coll.Insert(M{"a": 1}) c.Assert(err, IsNil) err = session.Run("serverStatus", result) c.Assert(err, IsNil) c.Assert(result.Host, Equals, secondary) } func (s *S) TestPreserveSocketCountOnSync(c *C) { if *fast { c.Skip("-fast") } session, err := mgo.Dial("localhost:40011") c.Assert(err, IsNil) defer session.Close() stats := mgo.GetStats() for stats.SocketsAlive != 3 { c.Logf("Waiting for all connections to be established (sockets alive currently %d)...", stats.SocketsAlive) stats = mgo.GetStats() time.Sleep(5e8) } c.Assert(stats.SocketsAlive, Equals, 3) // Kill the master (with rs1, 'a' is always the master). s.Stop("localhost:40011") // Wait for the logic to run for a bit and bring it back. startedAll := make(chan bool) go func() { time.Sleep(5e9) s.StartAll() startedAll <- true }() // Do not allow the test to return before the goroutine above is done. defer func() { <-startedAll }() // Do an action to kick the resync logic in, and also to // wait until the cluster recognizes the server is back. result := struct{ Ok bool }{} err = session.Run("getLastError", &result) c.Assert(err, IsNil) c.Assert(result.Ok, Equals, true) for i := 0; i != 20; i++ { stats = mgo.GetStats() if stats.SocketsAlive == 3 { break } c.Logf("Waiting for 3 sockets alive, have %d", stats.SocketsAlive) time.Sleep(5e8) } // Ensure the number of sockets is preserved after syncing. stats = mgo.GetStats() c.Assert(stats.SocketsAlive, Equals, 3) c.Assert(stats.SocketsInUse, Equals, 1) c.Assert(stats.SocketRefs, Equals, 1) } // Connect to the master of a deployment with a single server, // run an insert, and then ensure the insert worked and that a // single connection was established. func (s *S) TestTopologySyncWithSingleMaster(c *C) { // Use hostname here rather than IP, to make things trickier. session, err := mgo.Dial("localhost:40001") c.Assert(err, IsNil) defer session.Close() coll := session.DB("mydb").C("mycoll") err = coll.Insert(M{"a": 1, "b": 2}) c.Assert(err, IsNil) // One connection used for discovery. Master socket recycled for // insert. Socket is reserved after insert. stats := mgo.GetStats() c.Assert(stats.MasterConns, Equals, 1) c.Assert(stats.SlaveConns, Equals, 0) c.Assert(stats.SocketsInUse, Equals, 1) // Refresh session and socket must be released. session.Refresh() stats = mgo.GetStats() c.Assert(stats.SocketsInUse, Equals, 0) } func (s *S) TestTopologySyncWithSlaveSeed(c *C) { // That's supposed to be a slave. Must run discovery // and find out master to insert successfully. session, err := mgo.Dial("localhost:40012") c.Assert(err, IsNil) defer session.Close() coll := session.DB("mydb").C("mycoll") coll.Insert(M{"a": 1, "b": 2}) result := struct{ Ok bool }{} err = session.Run("getLastError", &result) c.Assert(err, IsNil) c.Assert(result.Ok, Equals, true) // One connection to each during discovery. Master // socket recycled for insert. stats := mgo.GetStats() c.Assert(stats.MasterConns, Equals, 1) c.Assert(stats.SlaveConns, Equals, 2) // Only one socket reference alive, in the master socket owned // by the above session. c.Assert(stats.SocketsInUse, Equals, 1) // Refresh it, and it must be gone. session.Refresh() stats = mgo.GetStats() c.Assert(stats.SocketsInUse, Equals, 0) } func (s *S) TestSyncTimeout(c *C) { if *fast { c.Skip("-fast") } session, err := mgo.Dial("localhost:40001") c.Assert(err, IsNil) defer session.Close() s.Stop("localhost:40001") timeout := 3 * time.Second session.SetSyncTimeout(timeout) started := time.Now() // Do something. result := struct{ Ok bool }{} err = session.Run("getLastError", &result) c.Assert(err, ErrorMatches, "no reachable servers") c.Assert(started.Before(time.Now().Add(-timeout)), Equals, true) c.Assert(started.After(time.Now().Add(-timeout*2)), Equals, true) } func (s *S) TestDialWithTimeout(c *C) { if *fast { c.Skip("-fast") } timeout := 2 * time.Second started := time.Now() // 40009 isn't used by the test servers. session, err := mgo.DialWithTimeout("localhost:40009", timeout) if session != nil { session.Close() } c.Assert(err, ErrorMatches, "no reachable servers") c.Assert(session, IsNil) c.Assert(started.Before(time.Now().Add(-timeout)), Equals, true) c.Assert(started.After(time.Now().Add(-timeout*2)), Equals, true) } func (s *S) TestSocketTimeout(c *C) { if *fast { c.Skip("-fast") } session, err := mgo.Dial("localhost:40001") c.Assert(err, IsNil) defer session.Close() s.Freeze("localhost:40001") timeout := 3 * time.Second session.SetSocketTimeout(timeout) started := time.Now() // Do something. result := struct{ Ok bool }{} err = session.Run("getLastError", &result) c.Assert(err, ErrorMatches, ".*: i/o timeout") c.Assert(started.Before(time.Now().Add(-timeout)), Equals, true) c.Assert(started.After(time.Now().Add(-timeout*2)), Equals, true) } func (s *S) TestSocketTimeoutOnDial(c *C) { if *fast { c.Skip("-fast") } timeout := 1 * time.Second defer mgo.HackSyncSocketTimeout(timeout)() s.Freeze("localhost:40001") started := time.Now() session, err := mgo.DialWithTimeout("localhost:40001", timeout) c.Assert(err, ErrorMatches, "no reachable servers") c.Assert(session, IsNil) c.Assert(started.Before(time.Now().Add(-timeout)), Equals, true) c.Assert(started.After(time.Now().Add(-20*time.Second)), Equals, true) } func (s *S) TestSocketTimeoutOnInactiveSocket(c *C) { if *fast { c.Skip("-fast") } session, err := mgo.Dial("localhost:40001") c.Assert(err, IsNil) defer session.Close() timeout := 2 * time.Second session.SetSocketTimeout(timeout) // Do something that relies on the timeout and works. c.Assert(session.Ping(), IsNil) // Freeze and wait for the timeout to go by. s.Freeze("localhost:40001") time.Sleep(timeout + 500*time.Millisecond) s.Thaw("localhost:40001") // Do something again. The timeout above should not have killed // the socket as there was nothing to be done. c.Assert(session.Ping(), IsNil) } func (s *S) TestDialWithReplicaSetName(c *C) { seedLists := [][]string{ // rs1 primary and rs2 primary []string{"localhost:40011", "localhost:40021"}, // rs1 primary and rs2 secondary []string{"localhost:40011", "localhost:40022"}, // rs1 secondary and rs2 primary []string{"localhost:40012", "localhost:40021"}, // rs1 secondary and rs2 secondary []string{"localhost:40012", "localhost:40022"}, } rs2Members := []string{":40021", ":40022", ":40023"} verifySyncedServers := func(session *mgo.Session, numServers int) { // wait for the server(s) to be synced for len(session.LiveServers()) != numServers { c.Log("Waiting for cluster sync to finish...") time.Sleep(5e8) } // ensure none of the rs2 set members are communicated with for _, addr := range session.LiveServers() { for _, rs2Member := range rs2Members { c.Assert(strings.HasSuffix(addr, rs2Member), Equals, false) } } } // only communication with rs1 members is expected for _, seedList := range seedLists { info := mgo.DialInfo{ Addrs: seedList, Timeout: 5 * time.Second, ReplicaSetName: "rs1", } session, err := mgo.DialWithInfo(&info) c.Assert(err, IsNil) verifySyncedServers(session, 3) session.Close() info.Direct = true session, err = mgo.DialWithInfo(&info) c.Assert(err, IsNil) verifySyncedServers(session, 1) session.Close() connectionUrl := fmt.Sprintf("mongodb://%v/?replicaSet=rs1", strings.Join(seedList, ",")) session, err = mgo.Dial(connectionUrl) c.Assert(err, IsNil) verifySyncedServers(session, 3) session.Close() connectionUrl += "&connect=direct" session, err = mgo.Dial(connectionUrl) c.Assert(err, IsNil) verifySyncedServers(session, 1) session.Close() } } func (s *S) TestDirect(c *C) { session, err := mgo.Dial("localhost:40012?connect=direct") c.Assert(err, IsNil) defer session.Close() // We know that server is a slave. session.SetMode(mgo.Monotonic, true) result := &struct{ Host string }{} err = session.Run("serverStatus", result) c.Assert(err, IsNil) c.Assert(strings.HasSuffix(result.Host, ":40012"), Equals, true) stats := mgo.GetStats() c.Assert(stats.SocketsAlive, Equals, 1) c.Assert(stats.SocketsInUse, Equals, 1) c.Assert(stats.SocketRefs, Equals, 1) // We've got no master, so it'll timeout. session.SetSyncTimeout(5e8 * time.Nanosecond) coll := session.DB("mydb").C("mycoll") err = coll.Insert(M{"test": 1}) c.Assert(err, ErrorMatches, "no reachable servers") // Writing to the local database is okay. coll = session.DB("local").C("mycoll") defer coll.RemoveAll(nil) id := bson.NewObjectId() err = coll.Insert(M{"_id": id}) c.Assert(err, IsNil) // Data was stored in the right server. n, err := coll.Find(M{"_id": id}).Count() c.Assert(err, IsNil) c.Assert(n, Equals, 1) // Server hasn't changed. result.Host = "" err = session.Run("serverStatus", result) c.Assert(err, IsNil) c.Assert(strings.HasSuffix(result.Host, ":40012"), Equals, true) } func (s *S) TestDirectToUnknownStateMember(c *C) { session, err := mgo.Dial("localhost:40041?connect=direct") c.Assert(err, IsNil) defer session.Close() session.SetMode(mgo.Monotonic, true) result := &struct{ Host string }{} err = session.Run("serverStatus", result) c.Assert(err, IsNil) c.Assert(strings.HasSuffix(result.Host, ":40041"), Equals, true) // We've got no master, so it'll timeout. session.SetSyncTimeout(5e8 * time.Nanosecond) coll := session.DB("mydb").C("mycoll") err = coll.Insert(M{"test": 1}) c.Assert(err, ErrorMatches, "no reachable servers") // Slave is still reachable. result.Host = "" err = session.Run("serverStatus", result) c.Assert(err, IsNil) c.Assert(strings.HasSuffix(result.Host, ":40041"), Equals, true) } func (s *S) TestFailFast(c *C) { info := mgo.DialInfo{ Addrs: []string{"localhost:99999"}, Timeout: 5 * time.Second, FailFast: true, } started := time.Now() _, err := mgo.DialWithInfo(&info) c.Assert(err, ErrorMatches, "no reachable servers") c.Assert(started.After(time.Now().Add(-time.Second)), Equals, true) } func (s *S) countQueries(c *C, server string) (n int) { defer func() { c.Logf("Queries for %q: %d", server, n) }() session, err := mgo.Dial(server + "?connect=direct") c.Assert(err, IsNil) defer session.Close() session.SetMode(mgo.Monotonic, true) var result struct { OpCounters struct { Query int } Metrics struct { Commands struct{ Find struct{ Total int } } } } err = session.Run("serverStatus", &result) c.Assert(err, IsNil) if s.versionAtLeast(3, 2) { return result.Metrics.Commands.Find.Total } return result.OpCounters.Query } func (s *S) countCommands(c *C, server, commandName string) (n int) { defer func() { c.Logf("Queries for %q: %d", server, n) }() session, err := mgo.Dial(server + "?connect=direct") c.Assert(err, IsNil) defer session.Close() session.SetMode(mgo.Monotonic, true) var result struct { Metrics struct { Commands map[string]struct{ Total int } } } err = session.Run("serverStatus", &result) c.Assert(err, IsNil) return result.Metrics.Commands[commandName].Total } func (s *S) TestMonotonicSlaveOkFlagWithMongos(c *C) { session, err := mgo.Dial("localhost:40021") c.Assert(err, IsNil) defer session.Close() ssresult := &struct{ Host string }{} imresult := &struct{ IsMaster bool }{} // Figure the master while still using the strong session. err = session.Run("serverStatus", ssresult) c.Assert(err, IsNil) err = session.Run("isMaster", imresult) c.Assert(err, IsNil) master := ssresult.Host c.Assert(imresult.IsMaster, Equals, true, Commentf("%s is not the master", master)) // Ensure mongos is aware about the current topology. s.Stop(":40201") s.StartAll() mongos, err := mgo.Dial("localhost:40202") c.Assert(err, IsNil) defer mongos.Close() // Insert some data as otherwise 3.2+ doesn't seem to run the query at all. err = mongos.DB("mydb").C("mycoll").Insert(bson.M{"n": 1}) c.Assert(err, IsNil) // Wait until all servers see the data. for _, addr := range []string{"localhost:40021", "localhost:40022", "localhost:40023"} { session, err := mgo.Dial(addr + "?connect=direct") c.Assert(err, IsNil) defer session.Close() session.SetMode(mgo.Monotonic, true) for i := 300; i >= 0; i-- { n, err := session.DB("mydb").C("mycoll").Find(nil).Count() c.Assert(err, IsNil) if n == 1 { break } if i == 0 { c.Fatalf("Inserted data never reached " + addr) } time.Sleep(100 * time.Millisecond) } } // Collect op counters for everyone. q21a := s.countQueries(c, "localhost:40021") q22a := s.countQueries(c, "localhost:40022") q23a := s.countQueries(c, "localhost:40023") // Do a SlaveOk query through MongoS mongos.SetMode(mgo.Monotonic, true) coll := mongos.DB("mydb").C("mycoll") var result struct{ N int } for i := 0; i != 5; i++ { err = coll.Find(nil).One(&result) c.Assert(err, IsNil) c.Assert(result.N, Equals, 1) } // Collect op counters for everyone again. q21b := s.countQueries(c, "localhost:40021") q22b := s.countQueries(c, "localhost:40022") q23b := s.countQueries(c, "localhost:40023") var masterDelta, slaveDelta int switch hostPort(master) { case "40021": masterDelta = q21b - q21a slaveDelta = (q22b - q22a) + (q23b - q23a) case "40022": masterDelta = q22b - q22a slaveDelta = (q21b - q21a) + (q23b - q23a) case "40023": masterDelta = q23b - q23a slaveDelta = (q21b - q21a) + (q22b - q22a) default: c.Fatal("Uh?") } c.Check(masterDelta, Equals, 0) // Just the counting itself. c.Check(slaveDelta, Equals, 5) // The counting for both, plus 5 queries above. } func (s *S) TestSecondaryModeWithMongos(c *C) { session, err := mgo.Dial("localhost:40021") c.Assert(err, IsNil) defer session.Close() ssresult := &struct{ Host string }{} imresult := &struct{ IsMaster bool }{} // Figure the master while still using the strong session. err = session.Run("serverStatus", ssresult) c.Assert(err, IsNil) err = session.Run("isMaster", imresult) c.Assert(err, IsNil) master := ssresult.Host c.Assert(imresult.IsMaster, Equals, true, Commentf("%s is not the master", master)) // Ensure mongos is aware about the current topology. s.Stop(":40201") s.StartAll() mongos, err := mgo.Dial("localhost:40202") c.Assert(err, IsNil) defer mongos.Close() mongos.SetSyncTimeout(5 * time.Second) // Insert some data as otherwise 3.2+ doesn't seem to run the query at all. err = mongos.DB("mydb").C("mycoll").Insert(bson.M{"n": 1}) c.Assert(err, IsNil) // Wait until all servers see the data. for _, addr := range []string{"localhost:40021", "localhost:40022", "localhost:40023"} { session, err := mgo.Dial(addr + "?connect=direct") c.Assert(err, IsNil) defer session.Close() session.SetMode(mgo.Monotonic, true) for i := 300; i >= 0; i-- { n, err := session.DB("mydb").C("mycoll").Find(nil).Count() c.Assert(err, IsNil) if n == 1 { break } if i == 0 { c.Fatalf("Inserted data never reached " + addr) } time.Sleep(100 * time.Millisecond) } } // Collect op counters for everyone. q21a := s.countQueries(c, "localhost:40021") q22a := s.countQueries(c, "localhost:40022") q23a := s.countQueries(c, "localhost:40023") // Do a Secondary query through MongoS mongos.SetMode(mgo.Secondary, true) coll := mongos.DB("mydb").C("mycoll") var result struct{ N int } for i := 0; i != 5; i++ { err = coll.Find(nil).One(&result) c.Assert(err, IsNil) c.Assert(result.N, Equals, 1) } // Collect op counters for everyone again. q21b := s.countQueries(c, "localhost:40021") q22b := s.countQueries(c, "localhost:40022") q23b := s.countQueries(c, "localhost:40023") var masterDelta, slaveDelta int switch hostPort(master) { case "40021": masterDelta = q21b - q21a slaveDelta = (q22b - q22a) + (q23b - q23a) case "40022": masterDelta = q22b - q22a slaveDelta = (q21b - q21a) + (q23b - q23a) case "40023": masterDelta = q23b - q23a slaveDelta = (q21b - q21a) + (q22b - q22a) default: c.Fatal("Uh?") } c.Check(masterDelta, Equals, 0) // Just the counting itself. c.Check(slaveDelta, Equals, 5) // The counting for both, plus 5 queries above. } func (s *S) TestSecondaryModeWithMongosInsert(c *C) { if *fast { c.Skip("-fast") } session, err := mgo.Dial("localhost:40202") c.Assert(err, IsNil) defer session.Close() session.SetMode(mgo.Secondary, true) session.SetSyncTimeout(4 * time.Second) coll := session.DB("mydb").C("mycoll") err = coll.Insert(M{"a": 1}) c.Assert(err, IsNil) var result struct{ A int } coll.Find(nil).One(&result) c.Assert(result.A, Equals, 1) } func (s *S) TestRemovalOfClusterMember(c *C) { if *fast { c.Skip("-fast") } master, err := mgo.Dial("localhost:40021") c.Assert(err, IsNil) defer master.Close() // Wait for cluster to fully sync up. for i := 0; i < 10; i++ { if len(master.LiveServers()) == 3 { break } time.Sleep(5e8) } if len(master.LiveServers()) != 3 { c.Fatalf("Test started with bad cluster state: %v", master.LiveServers()) } result := &struct { IsMaster bool Me string }{} slave := master.Copy() slave.SetMode(mgo.Monotonic, true) // Monotonic can hold a non-master socket persistently. err = slave.Run("isMaster", result) c.Assert(err, IsNil) c.Assert(result.IsMaster, Equals, false) slaveAddr := result.Me defer func() { config := map[string]string{ "40021": `{_id: 1, host: "127.0.0.1:40021", priority: 1, tags: {rs2: "a"}}`, "40022": `{_id: 2, host: "127.0.0.1:40022", priority: 0, tags: {rs2: "b"}}`, "40023": `{_id: 3, host: "127.0.0.1:40023", priority: 0, tags: {rs2: "c"}}`, } master.Refresh() master.Run(bson.D{{"$eval", `rs.add(` + config[hostPort(slaveAddr)] + `)`}}, nil) master.Close() slave.Close() // Ensure suite syncs up with the changes before next test. s.Stop(":40201") s.StartAll() time.Sleep(8 * time.Second) // TODO Find a better way to find out when mongos is fully aware that all // servers are up. Without that follow up tests that depend on mongos will // break due to their expectation of things being in a working state. }() c.Logf("========== Removing slave: %s ==========", slaveAddr) master.Run(bson.D{{"$eval", `rs.remove("` + slaveAddr + `")`}}, nil) master.Refresh() // Give the cluster a moment to catch up by doing a roundtrip to the master. err = master.Ping() c.Assert(err, IsNil) time.Sleep(3e9) // This must fail since the slave has been taken off the cluster. err = slave.Ping() c.Assert(err, NotNil) for i := 0; i < 15; i++ { if len(master.LiveServers()) == 2 { break } time.Sleep(time.Second) } live := master.LiveServers() if len(live) != 2 { c.Errorf("Removed server still considered live: %#s", live) } c.Log("========== Test succeeded. ==========") } func (s *S) TestPoolLimitSimple(c *C) { for test := 0; test < 2; test++ { var session *mgo.Session var err error if test == 0 { session, err = mgo.Dial("localhost:40001") c.Assert(err, IsNil) session.SetPoolLimit(1) } else { session, err = mgo.Dial("localhost:40001?maxPoolSize=1") c.Assert(err, IsNil) } defer session.Close() // Put one socket in use. c.Assert(session.Ping(), IsNil) done := make(chan time.Duration) // Now block trying to get another one due to the pool limit. go func() { copy := session.Copy() defer copy.Close() started := time.Now() c.Check(copy.Ping(), IsNil) done <- time.Now().Sub(started) }() time.Sleep(300 * time.Millisecond) // Put the one socket back in the pool, freeing it for the copy. session.Refresh() delay := <-done c.Assert(delay > 300*time.Millisecond, Equals, true, Commentf("Delay: %s", delay)) } } func (s *S) TestPoolLimitMany(c *C) { if *fast { c.Skip("-fast") } session, err := mgo.Dial("localhost:40011") c.Assert(err, IsNil) defer session.Close() stats := mgo.GetStats() for stats.SocketsAlive != 3 { c.Logf("Waiting for all connections to be established (sockets alive currently %d)...", stats.SocketsAlive) stats = mgo.GetStats() time.Sleep(5e8) } const poolLimit = 64 session.SetPoolLimit(poolLimit) // Consume the whole limit for the master. var master []*mgo.Session for i := 0; i < poolLimit; i++ { s := session.Copy() defer s.Close() c.Assert(s.Ping(), IsNil) master = append(master, s) } before := time.Now() go func() { time.Sleep(3e9) master[0].Refresh() }() // Then, a single ping must block, since it would need another // connection to the master, over the limit. Once the goroutine // above releases its socket, it should move on. session.Ping() delay := time.Now().Sub(before) c.Assert(delay > 3e9, Equals, true) c.Assert(delay < 6e9, Equals, true) } func (s *S) TestSetModeEventualIterBug(c *C) { session1, err := mgo.Dial("localhost:40011") c.Assert(err, IsNil) defer session1.Close() session1.SetMode(mgo.Eventual, false) coll1 := session1.DB("mydb").C("mycoll") const N = 100 for i := 0; i < N; i++ { err = coll1.Insert(M{"_id": i}) c.Assert(err, IsNil) } c.Logf("Waiting until secondary syncs") for { n, err := coll1.Count() c.Assert(err, IsNil) if n == N { c.Logf("Found all") break } } session2, err := mgo.Dial("localhost:40011") c.Assert(err, IsNil) defer session2.Close() session2.SetMode(mgo.Eventual, false) coll2 := session2.DB("mydb").C("mycoll") i := 0 iter := coll2.Find(nil).Batch(10).Iter() var result struct{} for iter.Next(&result) { i++ } c.Assert(iter.Close(), Equals, nil) c.Assert(i, Equals, N) } func (s *S) TestCustomDialOld(c *C) { dials := make(chan bool, 16) dial := func(addr net.Addr) (net.Conn, error) { tcpaddr, ok := addr.(*net.TCPAddr) if !ok { return nil, fmt.Errorf("unexpected address type: %T", addr) } dials <- true return net.DialTCP("tcp", nil, tcpaddr) } info := mgo.DialInfo{ Addrs: []string{"localhost:40012"}, Dial: dial, } // Use hostname here rather than IP, to make things trickier. session, err := mgo.DialWithInfo(&info) c.Assert(err, IsNil) defer session.Close() const N = 3 for i := 0; i < N; i++ { select { case <-dials: case <-time.After(5 * time.Second): c.Fatalf("expected %d dials, got %d", N, i) } } select { case <-dials: c.Fatalf("got more dials than expected") case <-time.After(100 * time.Millisecond): } } func (s *S) TestCustomDialNew(c *C) { dials := make(chan bool, 16) dial := func(addr *mgo.ServerAddr) (net.Conn, error) { dials <- true if addr.TCPAddr().Port == 40012 { c.Check(addr.String(), Equals, "localhost:40012") } return net.DialTCP("tcp", nil, addr.TCPAddr()) } info := mgo.DialInfo{ Addrs: []string{"localhost:40012"}, DialServer: dial, } // Use hostname here rather than IP, to make things trickier. session, err := mgo.DialWithInfo(&info) c.Assert(err, IsNil) defer session.Close() const N = 3 for i := 0; i < N; i++ { select { case <-dials: case <-time.After(5 * time.Second): c.Fatalf("expected %d dials, got %d", N, i) } } select { case <-dials: c.Fatalf("got more dials than expected") case <-time.After(100 * time.Millisecond): } } func (s *S) TestPrimaryShutdownOnAuthShard(c *C) { if *fast { c.Skip("-fast") } // Dial the shard. session, err := mgo.Dial("localhost:40203") c.Assert(err, IsNil) defer session.Close() // Login and insert something to make it more realistic. session.DB("admin").Login("root", "rapadura") coll := session.DB("mydb").C("mycoll") err = coll.Insert(bson.M{"n": 1}) c.Assert(err, IsNil) // Dial the replica set to figure the master out. rs, err := mgo.Dial("root:rapadura@localhost:40031") c.Assert(err, IsNil) defer rs.Close() // With strong consistency, this will open a socket to the master. result := &struct{ Host string }{} err = rs.Run("serverStatus", result) c.Assert(err, IsNil) // Kill the master. host := result.Host s.Stop(host) // This must fail, since the connection was broken. err = rs.Run("serverStatus", result) c.Assert(err, Equals, io.EOF) // This won't work because the master just died. err = coll.Insert(bson.M{"n": 2}) c.Assert(err, NotNil) // Refresh session and wait for re-election. session.Refresh() for i := 0; i < 60; i++ { err = coll.Insert(bson.M{"n": 3}) if err == nil { break } c.Logf("Waiting for replica set to elect a new master. Last error: %v", err) time.Sleep(500 * time.Millisecond) } c.Assert(err, IsNil) count, err := coll.Count() c.Assert(count > 1, Equals, true) } func (s *S) TestNearestSecondary(c *C) { defer mgo.HackPingDelay(300 * time.Millisecond)() rs1a := "127.0.0.1:40011" rs1b := "127.0.0.1:40012" rs1c := "127.0.0.1:40013" s.Freeze(rs1b) session, err := mgo.Dial(rs1a) c.Assert(err, IsNil) defer session.Close() // Wait for the sync up to run through the first couple of servers. for len(session.LiveServers()) != 2 { c.Log("Waiting for two servers to be alive...") time.Sleep(100 * time.Millisecond) } // Extra delay to ensure the third server gets penalized. time.Sleep(500 * time.Millisecond) // Release third server. s.Thaw(rs1b) // Wait for it to come up. for len(session.LiveServers()) != 3 { c.Log("Waiting for all servers to be alive...") time.Sleep(100 * time.Millisecond) } session.SetMode(mgo.Monotonic, true) var result struct{ Host string } // See which slave picks the line, several times to avoid chance. for i := 0; i < 10; i++ { session.Refresh() err = session.Run("serverStatus", &result) c.Assert(err, IsNil) c.Assert(hostPort(result.Host), Equals, hostPort(rs1c)) } if *fast { // Don't hold back for several seconds. return } // Now hold the other server for long enough to penalize it. s.Freeze(rs1c) time.Sleep(5 * time.Second) s.Thaw(rs1c) // Wait for the ping to be processed. time.Sleep(500 * time.Millisecond) // Repeating the test should now pick the former server consistently. for i := 0; i < 10; i++ { session.Refresh() err = session.Run("serverStatus", &result) c.Assert(err, IsNil) c.Assert(hostPort(result.Host), Equals, hostPort(rs1b)) } } func (s *S) TestNearestServer(c *C) { defer mgo.HackPingDelay(300 * time.Millisecond)() rs1a := "127.0.0.1:40011" rs1b := "127.0.0.1:40012" rs1c := "127.0.0.1:40013" session, err := mgo.Dial(rs1a) c.Assert(err, IsNil) defer session.Close() s.Freeze(rs1a) s.Freeze(rs1b) // Extra delay to ensure the first two servers get penalized. time.Sleep(500 * time.Millisecond) // Release them. s.Thaw(rs1a) s.Thaw(rs1b) // Wait for everyone to come up. for len(session.LiveServers()) != 3 { c.Log("Waiting for all servers to be alive...") time.Sleep(100 * time.Millisecond) } session.SetMode(mgo.Nearest, true) var result struct{ Host string } // See which server picks the line, several times to avoid chance. for i := 0; i < 10; i++ { session.Refresh() err = session.Run("serverStatus", &result) c.Assert(err, IsNil) c.Assert(hostPort(result.Host), Equals, hostPort(rs1c)) } if *fast { // Don't hold back for several seconds. return } // Now hold the two secondaries for long enough to penalize them. s.Freeze(rs1b) s.Freeze(rs1c) time.Sleep(5 * time.Second) s.Thaw(rs1b) s.Thaw(rs1c) // Wait for the ping to be processed. time.Sleep(500 * time.Millisecond) // Repeating the test should now pick the primary server consistently. for i := 0; i < 10; i++ { session.Refresh() err = session.Run("serverStatus", &result) c.Assert(err, IsNil) c.Assert(hostPort(result.Host), Equals, hostPort(rs1a)) } } func (s *S) TestConnectCloseConcurrency(c *C) { restore := mgo.HackPingDelay(500 * time.Millisecond) defer restore() var wg sync.WaitGroup const n = 500 wg.Add(n) for i := 0; i < n; i++ { go func() { defer wg.Done() session, err := mgo.Dial("localhost:40001") if err != nil { c.Fatal(err) } time.Sleep(1) session.Close() }() } wg.Wait() } func (s *S) TestSelectServers(c *C) { if !s.versionAtLeast(2, 2) { c.Skip("read preferences introduced in 2.2") } session, err := mgo.Dial("localhost:40011") c.Assert(err, IsNil) defer session.Close() session.SetMode(mgo.Eventual, true) var result struct{ Host string } session.Refresh() session.SelectServers(bson.D{{"rs1", "b"}}) err = session.Run("serverStatus", &result) c.Assert(err, IsNil) c.Assert(hostPort(result.Host), Equals, "40012") session.Refresh() session.SelectServers(bson.D{{"rs1", "c"}}) err = session.Run("serverStatus", &result) c.Assert(err, IsNil) c.Assert(hostPort(result.Host), Equals, "40013") } func (s *S) TestSelectServersWithMongos(c *C) { if !s.versionAtLeast(2, 2) { c.Skip("read preferences introduced in 2.2") } session, err := mgo.Dial("localhost:40021") c.Assert(err, IsNil) defer session.Close() ssresult := &struct{ Host string }{} imresult := &struct{ IsMaster bool }{} // Figure the master while still using the strong session. err = session.Run("serverStatus", ssresult) c.Assert(err, IsNil) err = session.Run("isMaster", imresult) c.Assert(err, IsNil) master := ssresult.Host c.Assert(imresult.IsMaster, Equals, true, Commentf("%s is not the master", master)) var slave1, slave2 string switch hostPort(master) { case "40021": slave1, slave2 = "b", "c" case "40022": slave1, slave2 = "a", "c" case "40023": slave1, slave2 = "a", "b" } // Collect op counters for everyone. q21a := s.countQueries(c, "localhost:40021") q22a := s.countQueries(c, "localhost:40022") q23a := s.countQueries(c, "localhost:40023") // Do a SlaveOk query through MongoS mongos, err := mgo.Dial("localhost:40202") c.Assert(err, IsNil) defer mongos.Close() mongos.SetMode(mgo.Monotonic, true) mongos.Refresh() mongos.SelectServers(bson.D{{"rs2", slave1}}) coll := mongos.DB("mydb").C("mycoll") result := &struct{}{} for i := 0; i != 5; i++ { err := coll.Find(nil).One(result) c.Assert(err, Equals, mgo.ErrNotFound) } mongos.Refresh() mongos.SelectServers(bson.D{{"rs2", slave2}}) coll = mongos.DB("mydb").C("mycoll") for i := 0; i != 7; i++ { err := coll.Find(nil).One(result) c.Assert(err, Equals, mgo.ErrNotFound) } // Collect op counters for everyone again. q21b := s.countQueries(c, "localhost:40021") q22b := s.countQueries(c, "localhost:40022") q23b := s.countQueries(c, "localhost:40023") switch hostPort(master) { case "40021": c.Check(q21b-q21a, Equals, 0) c.Check(q22b-q22a, Equals, 5) c.Check(q23b-q23a, Equals, 7) case "40022": c.Check(q21b-q21a, Equals, 5) c.Check(q22b-q22a, Equals, 0) c.Check(q23b-q23a, Equals, 7) case "40023": c.Check(q21b-q21a, Equals, 5) c.Check(q22b-q22a, Equals, 7) c.Check(q23b-q23a, Equals, 0) default: c.Fatal("Uh?") } } func (s *S) TestDoNotFallbackToMonotonic(c *C) { // There was a bug at some point that some functions were // falling back to Monotonic mode. This test ensures all listIndexes // commands go to the primary, as should happen since the session is // in Strong mode. if !s.versionAtLeast(3, 0) { c.Skip("command-counting logic depends on 3.0+") } session, err := mgo.Dial("localhost:40012") c.Assert(err, IsNil) defer session.Close() for i := 0; i < 15; i++ { q11a := s.countCommands(c, "localhost:40011", "listIndexes") q12a := s.countCommands(c, "localhost:40012", "listIndexes") q13a := s.countCommands(c, "localhost:40013", "listIndexes") _, err := session.DB("local").C("system.indexes").Indexes() c.Assert(err, IsNil) q11b := s.countCommands(c, "localhost:40011", "listIndexes") q12b := s.countCommands(c, "localhost:40012", "listIndexes") q13b := s.countCommands(c, "localhost:40013", "listIndexes") c.Assert(q11b, Equals, q11a+1) c.Assert(q12b, Equals, q12a) c.Assert(q13b, Equals, q13a) } } mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/dbtest/000077500000000000000000000000001276133741700224455ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/dbtest/dbserver.go000066400000000000000000000117131276133741700246130ustar00rootroot00000000000000package dbtest import ( "bytes" "fmt" "net" "os" "os/exec" "strconv" "time" "gopkg.in/mgo.v2" "gopkg.in/tomb.v2" ) // DBServer controls a MongoDB server process to be used within test suites. // // The test server is started when Session is called the first time and should // remain running for the duration of all tests, with the Wipe method being // called between tests (before each of them) to clear stored data. After all tests // are done, the Stop method should be called to stop the test server. // // Before the DBServer is used the SetPath method must be called to define // the location for the database files to be stored. type DBServer struct { session *mgo.Session output bytes.Buffer server *exec.Cmd dbpath string host string tomb tomb.Tomb } // SetPath defines the path to the directory where the database files will be // stored if it is started. The directory path itself is not created or removed // by the test helper. func (dbs *DBServer) SetPath(dbpath string) { dbs.dbpath = dbpath } func (dbs *DBServer) start() { if dbs.server != nil { panic("DBServer already started") } if dbs.dbpath == "" { panic("DBServer.SetPath must be called before using the server") } mgo.SetStats(true) l, err := net.Listen("tcp", "127.0.0.1:0") if err != nil { panic("unable to listen on a local address: " + err.Error()) } addr := l.Addr().(*net.TCPAddr) l.Close() dbs.host = addr.String() args := []string{ "--dbpath", dbs.dbpath, "--bind_ip", "127.0.0.1", "--port", strconv.Itoa(addr.Port), "--nssize", "1", "--noprealloc", "--smallfiles", "--nojournal", } dbs.tomb = tomb.Tomb{} dbs.server = exec.Command("mongod", args...) dbs.server.Stdout = &dbs.output dbs.server.Stderr = &dbs.output err = dbs.server.Start() if err != nil { panic(err) } dbs.tomb.Go(dbs.monitor) dbs.Wipe() } func (dbs *DBServer) monitor() error { dbs.server.Process.Wait() if dbs.tomb.Alive() { // Present some debugging information. fmt.Fprintf(os.Stderr, "---- mongod process died unexpectedly:\n") fmt.Fprintf(os.Stderr, "%s", dbs.output.Bytes()) fmt.Fprintf(os.Stderr, "---- mongod processes running right now:\n") cmd := exec.Command("/bin/sh", "-c", "ps auxw | grep mongod") cmd.Stdout = os.Stderr cmd.Stderr = os.Stderr cmd.Run() fmt.Fprintf(os.Stderr, "----------------------------------------\n") panic("mongod process died unexpectedly") } return nil } // Stop stops the test server process, if it is running. // // It's okay to call Stop multiple times. After the test server is // stopped it cannot be restarted. // // All database sessions must be closed before or while the Stop method // is running. Otherwise Stop will panic after a timeout informing that // there is a session leak. func (dbs *DBServer) Stop() { if dbs.session != nil { dbs.checkSessions() if dbs.session != nil { dbs.session.Close() dbs.session = nil } } if dbs.server != nil { dbs.tomb.Kill(nil) dbs.server.Process.Signal(os.Interrupt) select { case <-dbs.tomb.Dead(): case <-time.After(5 * time.Second): panic("timeout waiting for mongod process to die") } dbs.server = nil } } // Session returns a new session to the server. The returned session // must be closed after the test is done with it. // // The first Session obtained from a DBServer will start it. func (dbs *DBServer) Session() *mgo.Session { if dbs.server == nil { dbs.start() } if dbs.session == nil { mgo.ResetStats() var err error dbs.session, err = mgo.Dial(dbs.host + "/test") if err != nil { panic(err) } } return dbs.session.Copy() } // checkSessions ensures all mgo sessions opened were properly closed. // For slightly faster tests, it may be disabled setting the // environmnet variable CHECK_SESSIONS to 0. func (dbs *DBServer) checkSessions() { if check := os.Getenv("CHECK_SESSIONS"); check == "0" || dbs.server == nil || dbs.session == nil { return } dbs.session.Close() dbs.session = nil for i := 0; i < 100; i++ { stats := mgo.GetStats() if stats.SocketsInUse == 0 && stats.SocketsAlive == 0 { return } time.Sleep(100 * time.Millisecond) } panic("There are mgo sessions still alive.") } // Wipe drops all created databases and their data. // // The MongoDB server remains running if it was prevoiusly running, // or stopped if it was previously stopped. // // All database sessions must be closed before or while the Wipe method // is running. Otherwise Wipe will panic after a timeout informing that // there is a session leak. func (dbs *DBServer) Wipe() { if dbs.server == nil || dbs.session == nil { return } dbs.checkSessions() sessionUnset := dbs.session == nil session := dbs.Session() defer session.Close() if sessionUnset { dbs.session.Close() dbs.session = nil } names, err := session.DatabaseNames() if err != nil { panic(err) } for _, name := range names { switch name { case "admin", "local", "config": default: err = session.DB(name).DropDatabase() if err != nil { panic(err) } } } } mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/dbtest/dbserver_test.go000066400000000000000000000041541276133741700256530ustar00rootroot00000000000000package dbtest_test import ( "os" "testing" "time" . "gopkg.in/check.v1" "gopkg.in/mgo.v2" "gopkg.in/mgo.v2/dbtest" ) type M map[string]interface{} func TestAll(t *testing.T) { TestingT(t) } type S struct { oldCheckSessions string } var _ = Suite(&S{}) func (s *S) SetUpTest(c *C) { s.oldCheckSessions = os.Getenv("CHECK_SESSIONS") os.Setenv("CHECK_SESSIONS", "") } func (s *S) TearDownTest(c *C) { os.Setenv("CHECK_SESSIONS", s.oldCheckSessions) } func (s *S) TestWipeData(c *C) { var server dbtest.DBServer server.SetPath(c.MkDir()) defer server.Stop() session := server.Session() err := session.DB("mydb").C("mycoll").Insert(M{"a": 1}) session.Close() c.Assert(err, IsNil) server.Wipe() session = server.Session() names, err := session.DatabaseNames() session.Close() c.Assert(err, IsNil) for _, name := range names { if name != "local" && name != "admin" { c.Fatalf("Wipe should have removed this database: %s", name) } } } func (s *S) TestStop(c *C) { var server dbtest.DBServer server.SetPath(c.MkDir()) defer server.Stop() // Server should not be running. process := server.ProcessTest() c.Assert(process, IsNil) session := server.Session() addr := session.LiveServers()[0] session.Close() // Server should be running now. process = server.ProcessTest() p, err := os.FindProcess(process.Pid) c.Assert(err, IsNil) p.Release() server.Stop() // Server should not be running anymore. session, err = mgo.DialWithTimeout(addr, 500*time.Millisecond) if session != nil { session.Close() c.Fatalf("Stop did not stop the server") } } func (s *S) TestCheckSessions(c *C) { var server dbtest.DBServer server.SetPath(c.MkDir()) defer server.Stop() session := server.Session() defer session.Close() c.Assert(server.Wipe, PanicMatches, "There are mgo sessions still alive.") } func (s *S) TestCheckSessionsDisabled(c *C) { var server dbtest.DBServer server.SetPath(c.MkDir()) defer server.Stop() os.Setenv("CHECK_SESSIONS", "0") // Should not panic, although it looks to Wipe like this session will leak. session := server.Session() defer session.Close() server.Wipe() } mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/dbtest/export_test.go000066400000000000000000000002301276133741700253470ustar00rootroot00000000000000package dbtest import ( "os" ) func (dbs *DBServer) ProcessTest() *os.Process { if dbs.server == nil { return nil } return dbs.server.Process } mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/doc.go000066400000000000000000000023171276133741700222570ustar00rootroot00000000000000// Package mgo offers a rich MongoDB driver for Go. // // Details about the mgo project (pronounced as "mango") are found // in its web page: // // http://labix.org/mgo // // Usage of the driver revolves around the concept of sessions. To // get started, obtain a session using the Dial function: // // session, err := mgo.Dial(url) // // This will establish one or more connections with the cluster of // servers defined by the url parameter. From then on, the cluster // may be queried with multiple consistency rules (see SetMode) and // documents retrieved with statements such as: // // c := session.DB(database).C(collection) // err := c.Find(query).One(&result) // // New sessions are typically created by calling session.Copy on the // initial session obtained at dial time. These new sessions will share // the same cluster information and connection pool, and may be easily // handed into other methods and functions for organizing logic. // Every session created must have its Close method called at the end // of its life time, so its resources may be put back in the pool or // collected, depending on the case. // // For more details, see the documentation for the types and methods. // package mgo mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/export_test.go000066400000000000000000000011121276133741700240620ustar00rootroot00000000000000package mgo import ( "time" ) func HackPingDelay(newDelay time.Duration) (restore func()) { globalMutex.Lock() defer globalMutex.Unlock() oldDelay := pingDelay restore = func() { globalMutex.Lock() pingDelay = oldDelay globalMutex.Unlock() } pingDelay = newDelay return } func HackSyncSocketTimeout(newTimeout time.Duration) (restore func()) { globalMutex.Lock() defer globalMutex.Unlock() oldTimeout := syncSocketTimeout restore = func() { globalMutex.Lock() syncSocketTimeout = oldTimeout globalMutex.Unlock() } syncSocketTimeout = newTimeout return } mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/gridfs.go000066400000000000000000000513271276133741700227750ustar00rootroot00000000000000// mgo - MongoDB driver for Go // // Copyright (c) 2010-2012 - Gustavo Niemeyer // // All rights reserved. // // Redistribution and use in source and binary forms, with or without // modification, are permitted provided that the following conditions are met: // // 1. Redistributions of source code must retain the above copyright notice, this // list of conditions and the following disclaimer. // 2. Redistributions in binary form must reproduce the above copyright notice, // this list of conditions and the following disclaimer in the documentation // and/or other materials provided with the distribution. // // THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND // ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED // WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE // DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR // ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES // (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; // LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND // ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT // (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS // SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. package mgo import ( "crypto/md5" "encoding/hex" "errors" "hash" "io" "os" "sync" "time" "gopkg.in/mgo.v2/bson" ) type GridFS struct { Files *Collection Chunks *Collection } type gfsFileMode int const ( gfsClosed gfsFileMode = 0 gfsReading gfsFileMode = 1 gfsWriting gfsFileMode = 2 ) type GridFile struct { m sync.Mutex c sync.Cond gfs *GridFS mode gfsFileMode err error chunk int offset int64 wpending int wbuf []byte wsum hash.Hash rbuf []byte rcache *gfsCachedChunk doc gfsFile } type gfsFile struct { Id interface{} "_id" ChunkSize int "chunkSize" UploadDate time.Time "uploadDate" Length int64 ",minsize" MD5 string Filename string ",omitempty" ContentType string "contentType,omitempty" Metadata *bson.Raw ",omitempty" } type gfsChunk struct { Id interface{} "_id" FilesId interface{} "files_id" N int Data []byte } type gfsCachedChunk struct { wait sync.Mutex n int data []byte err error } func newGridFS(db *Database, prefix string) *GridFS { return &GridFS{db.C(prefix + ".files"), db.C(prefix + ".chunks")} } func (gfs *GridFS) newFile() *GridFile { file := &GridFile{gfs: gfs} file.c.L = &file.m //runtime.SetFinalizer(file, finalizeFile) return file } func finalizeFile(file *GridFile) { file.Close() } // Create creates a new file with the provided name in the GridFS. If the file // name already exists, a new version will be inserted with an up-to-date // uploadDate that will cause it to be atomically visible to the Open and // OpenId methods. If the file name is not important, an empty name may be // provided and the file Id used instead. // // It's important to Close files whether they are being written to // or read from, and to check the err result to ensure the operation // completed successfully. // // A simple example inserting a new file: // // func check(err error) { // if err != nil { // panic(err.String()) // } // } // file, err := db.GridFS("fs").Create("myfile.txt") // check(err) // n, err := file.Write([]byte("Hello world!")) // check(err) // err = file.Close() // check(err) // fmt.Printf("%d bytes written\n", n) // // The io.Writer interface is implemented by *GridFile and may be used to // help on the file creation. For example: // // file, err := db.GridFS("fs").Create("myfile.txt") // check(err) // messages, err := os.Open("/var/log/messages") // check(err) // defer messages.Close() // err = io.Copy(file, messages) // check(err) // err = file.Close() // check(err) // func (gfs *GridFS) Create(name string) (file *GridFile, err error) { file = gfs.newFile() file.mode = gfsWriting file.wsum = md5.New() file.doc = gfsFile{Id: bson.NewObjectId(), ChunkSize: 255 * 1024, Filename: name} return } // OpenId returns the file with the provided id, for reading. // If the file isn't found, err will be set to mgo.ErrNotFound. // // It's important to Close files whether they are being written to // or read from, and to check the err result to ensure the operation // completed successfully. // // The following example will print the first 8192 bytes from the file: // // func check(err error) { // if err != nil { // panic(err.String()) // } // } // file, err := db.GridFS("fs").OpenId(objid) // check(err) // b := make([]byte, 8192) // n, err := file.Read(b) // check(err) // fmt.Println(string(b)) // check(err) // err = file.Close() // check(err) // fmt.Printf("%d bytes read\n", n) // // The io.Reader interface is implemented by *GridFile and may be used to // deal with it. As an example, the following snippet will dump the whole // file into the standard output: // // file, err := db.GridFS("fs").OpenId(objid) // check(err) // err = io.Copy(os.Stdout, file) // check(err) // err = file.Close() // check(err) // func (gfs *GridFS) OpenId(id interface{}) (file *GridFile, err error) { var doc gfsFile err = gfs.Files.Find(bson.M{"_id": id}).One(&doc) if err != nil { return } file = gfs.newFile() file.mode = gfsReading file.doc = doc return } // Open returns the most recently uploaded file with the provided // name, for reading. If the file isn't found, err will be set // to mgo.ErrNotFound. // // It's important to Close files whether they are being written to // or read from, and to check the err result to ensure the operation // completed successfully. // // The following example will print the first 8192 bytes from the file: // // file, err := db.GridFS("fs").Open("myfile.txt") // check(err) // b := make([]byte, 8192) // n, err := file.Read(b) // check(err) // fmt.Println(string(b)) // check(err) // err = file.Close() // check(err) // fmt.Printf("%d bytes read\n", n) // // The io.Reader interface is implemented by *GridFile and may be used to // deal with it. As an example, the following snippet will dump the whole // file into the standard output: // // file, err := db.GridFS("fs").Open("myfile.txt") // check(err) // err = io.Copy(os.Stdout, file) // check(err) // err = file.Close() // check(err) // func (gfs *GridFS) Open(name string) (file *GridFile, err error) { var doc gfsFile err = gfs.Files.Find(bson.M{"filename": name}).Sort("-uploadDate").One(&doc) if err != nil { return } file = gfs.newFile() file.mode = gfsReading file.doc = doc return } // OpenNext opens the next file from iter for reading, sets *file to it, // and returns true on the success case. If no more documents are available // on iter or an error occurred, *file is set to nil and the result is false. // Errors will be available via iter.Err(). // // The iter parameter must be an iterator on the GridFS files collection. // Using the GridFS.Find method is an easy way to obtain such an iterator, // but any iterator on the collection will work. // // If the provided *file is non-nil, OpenNext will close it before attempting // to iterate to the next element. This means that in a loop one only // has to worry about closing files when breaking out of the loop early // (break, return, or panic). // // For example: // // gfs := db.GridFS("fs") // query := gfs.Find(nil).Sort("filename") // iter := query.Iter() // var f *mgo.GridFile // for gfs.OpenNext(iter, &f) { // fmt.Printf("Filename: %s\n", f.Name()) // } // if iter.Close() != nil { // panic(iter.Close()) // } // func (gfs *GridFS) OpenNext(iter *Iter, file **GridFile) bool { if *file != nil { // Ignoring the error here shouldn't be a big deal // as we're reading the file and the loop iteration // for this file is finished. _ = (*file).Close() } var doc gfsFile if !iter.Next(&doc) { *file = nil return false } f := gfs.newFile() f.mode = gfsReading f.doc = doc *file = f return true } // Find runs query on GridFS's files collection and returns // the resulting Query. // // This logic: // // gfs := db.GridFS("fs") // iter := gfs.Find(nil).Iter() // // Is equivalent to: // // files := db.C("fs" + ".files") // iter := files.Find(nil).Iter() // func (gfs *GridFS) Find(query interface{}) *Query { return gfs.Files.Find(query) } // RemoveId deletes the file with the provided id from the GridFS. func (gfs *GridFS) RemoveId(id interface{}) error { err := gfs.Files.Remove(bson.M{"_id": id}) if err != nil { return err } _, err = gfs.Chunks.RemoveAll(bson.D{{"files_id", id}}) return err } type gfsDocId struct { Id interface{} "_id" } // Remove deletes all files with the provided name from the GridFS. func (gfs *GridFS) Remove(name string) (err error) { iter := gfs.Files.Find(bson.M{"filename": name}).Select(bson.M{"_id": 1}).Iter() var doc gfsDocId for iter.Next(&doc) { if e := gfs.RemoveId(doc.Id); e != nil { err = e } } if err == nil { err = iter.Close() } return err } func (file *GridFile) assertMode(mode gfsFileMode) { switch file.mode { case mode: return case gfsWriting: panic("GridFile is open for writing") case gfsReading: panic("GridFile is open for reading") case gfsClosed: panic("GridFile is closed") default: panic("internal error: missing GridFile mode") } } // SetChunkSize sets size of saved chunks. Once the file is written to, it // will be split in blocks of that size and each block saved into an // independent chunk document. The default chunk size is 255kb. // // It is a runtime error to call this function once the file has started // being written to. func (file *GridFile) SetChunkSize(bytes int) { file.assertMode(gfsWriting) debugf("GridFile %p: setting chunk size to %d", file, bytes) file.m.Lock() file.doc.ChunkSize = bytes file.m.Unlock() } // Id returns the current file Id. func (file *GridFile) Id() interface{} { return file.doc.Id } // SetId changes the current file Id. // // It is a runtime error to call this function once the file has started // being written to, or when the file is not open for writing. func (file *GridFile) SetId(id interface{}) { file.assertMode(gfsWriting) file.m.Lock() file.doc.Id = id file.m.Unlock() } // Name returns the optional file name. An empty string will be returned // in case it is unset. func (file *GridFile) Name() string { return file.doc.Filename } // SetName changes the optional file name. An empty string may be used to // unset it. // // It is a runtime error to call this function when the file is not open // for writing. func (file *GridFile) SetName(name string) { file.assertMode(gfsWriting) file.m.Lock() file.doc.Filename = name file.m.Unlock() } // ContentType returns the optional file content type. An empty string will be // returned in case it is unset. func (file *GridFile) ContentType() string { return file.doc.ContentType } // ContentType changes the optional file content type. An empty string may be // used to unset it. // // It is a runtime error to call this function when the file is not open // for writing. func (file *GridFile) SetContentType(ctype string) { file.assertMode(gfsWriting) file.m.Lock() file.doc.ContentType = ctype file.m.Unlock() } // GetMeta unmarshals the optional "metadata" field associated with the // file into the result parameter. The meaning of keys under that field // is user-defined. For example: // // result := struct{ INode int }{} // err = file.GetMeta(&result) // if err != nil { // panic(err.String()) // } // fmt.Printf("inode: %d\n", result.INode) // func (file *GridFile) GetMeta(result interface{}) (err error) { file.m.Lock() if file.doc.Metadata != nil { err = bson.Unmarshal(file.doc.Metadata.Data, result) } file.m.Unlock() return } // SetMeta changes the optional "metadata" field associated with the // file. The meaning of keys under that field is user-defined. // For example: // // file.SetMeta(bson.M{"inode": inode}) // // It is a runtime error to call this function when the file is not open // for writing. func (file *GridFile) SetMeta(metadata interface{}) { file.assertMode(gfsWriting) data, err := bson.Marshal(metadata) file.m.Lock() if err != nil && file.err == nil { file.err = err } else { file.doc.Metadata = &bson.Raw{Data: data} } file.m.Unlock() } // Size returns the file size in bytes. func (file *GridFile) Size() (bytes int64) { file.m.Lock() bytes = file.doc.Length file.m.Unlock() return } // MD5 returns the file MD5 as a hex-encoded string. func (file *GridFile) MD5() (md5 string) { return file.doc.MD5 } // UploadDate returns the file upload time. func (file *GridFile) UploadDate() time.Time { return file.doc.UploadDate } // SetUploadDate changes the file upload time. // // It is a runtime error to call this function when the file is not open // for writing. func (file *GridFile) SetUploadDate(t time.Time) { file.assertMode(gfsWriting) file.m.Lock() file.doc.UploadDate = t file.m.Unlock() } // Close flushes any pending changes in case the file is being written // to, waits for any background operations to finish, and closes the file. // // It's important to Close files whether they are being written to // or read from, and to check the err result to ensure the operation // completed successfully. func (file *GridFile) Close() (err error) { file.m.Lock() defer file.m.Unlock() if file.mode == gfsWriting { if len(file.wbuf) > 0 && file.err == nil { file.insertChunk(file.wbuf) file.wbuf = file.wbuf[0:0] } file.completeWrite() } else if file.mode == gfsReading && file.rcache != nil { file.rcache.wait.Lock() file.rcache = nil } file.mode = gfsClosed debugf("GridFile %p: closed", file) return file.err } func (file *GridFile) completeWrite() { for file.wpending > 0 { debugf("GridFile %p: waiting for %d pending chunks to complete file write", file, file.wpending) file.c.Wait() } if file.err == nil { hexsum := hex.EncodeToString(file.wsum.Sum(nil)) if file.doc.UploadDate.IsZero() { file.doc.UploadDate = bson.Now() } file.doc.MD5 = hexsum file.err = file.gfs.Files.Insert(file.doc) } if file.err != nil { file.gfs.Chunks.RemoveAll(bson.D{{"files_id", file.doc.Id}}) } if file.err == nil { index := Index{ Key: []string{"files_id", "n"}, Unique: true, } file.err = file.gfs.Chunks.EnsureIndex(index) } } // Abort cancels an in-progress write, preventing the file from being // automically created and ensuring previously written chunks are // removed when the file is closed. // // It is a runtime error to call Abort when the file was not opened // for writing. func (file *GridFile) Abort() { if file.mode != gfsWriting { panic("file.Abort must be called on file opened for writing") } file.err = errors.New("write aborted") } // Write writes the provided data to the file and returns the // number of bytes written and an error in case something // wrong happened. // // The file will internally cache the data so that all but the last // chunk sent to the database have the size defined by SetChunkSize. // This also means that errors may be deferred until a future call // to Write or Close. // // The parameters and behavior of this function turn the file // into an io.Writer. func (file *GridFile) Write(data []byte) (n int, err error) { file.assertMode(gfsWriting) file.m.Lock() debugf("GridFile %p: writing %d bytes", file, len(data)) defer file.m.Unlock() if file.err != nil { return 0, file.err } n = len(data) file.doc.Length += int64(n) chunkSize := file.doc.ChunkSize if len(file.wbuf)+len(data) < chunkSize { file.wbuf = append(file.wbuf, data...) return } // First, flush file.wbuf complementing with data. if len(file.wbuf) > 0 { missing := chunkSize - len(file.wbuf) if missing > len(data) { missing = len(data) } file.wbuf = append(file.wbuf, data[:missing]...) data = data[missing:] file.insertChunk(file.wbuf) file.wbuf = file.wbuf[0:0] } // Then, flush all chunks from data without copying. for len(data) > chunkSize { size := chunkSize if size > len(data) { size = len(data) } file.insertChunk(data[:size]) data = data[size:] } // And append the rest for a future call. file.wbuf = append(file.wbuf, data...) return n, file.err } func (file *GridFile) insertChunk(data []byte) { n := file.chunk file.chunk++ debugf("GridFile %p: adding to checksum: %q", file, string(data)) file.wsum.Write(data) for file.doc.ChunkSize*file.wpending >= 1024*1024 { // Hold on.. we got a MB pending. file.c.Wait() if file.err != nil { return } } file.wpending++ debugf("GridFile %p: inserting chunk %d with %d bytes", file, n, len(data)) // We may not own the memory of data, so rather than // simply copying it, we'll marshal the document ahead of time. data, err := bson.Marshal(gfsChunk{bson.NewObjectId(), file.doc.Id, n, data}) if err != nil { file.err = err return } go func() { err := file.gfs.Chunks.Insert(bson.Raw{Data: data}) file.m.Lock() file.wpending-- if err != nil && file.err == nil { file.err = err } file.c.Broadcast() file.m.Unlock() }() } // Seek sets the offset for the next Read or Write on file to // offset, interpreted according to whence: 0 means relative to // the origin of the file, 1 means relative to the current offset, // and 2 means relative to the end. It returns the new offset and // an error, if any. func (file *GridFile) Seek(offset int64, whence int) (pos int64, err error) { file.m.Lock() debugf("GridFile %p: seeking for %s (whence=%d)", file, offset, whence) defer file.m.Unlock() switch whence { case os.SEEK_SET: case os.SEEK_CUR: offset += file.offset case os.SEEK_END: offset += file.doc.Length default: panic("unsupported whence value") } if offset > file.doc.Length { return file.offset, errors.New("seek past end of file") } if offset == file.doc.Length { // If we're seeking to the end of the file, // no need to read anything. This enables // a client to find the size of the file using only the // io.ReadSeeker interface with low overhead. file.offset = offset return file.offset, nil } chunk := int(offset / int64(file.doc.ChunkSize)) if chunk+1 == file.chunk && offset >= file.offset { file.rbuf = file.rbuf[int(offset-file.offset):] file.offset = offset return file.offset, nil } file.offset = offset file.chunk = chunk file.rbuf = nil file.rbuf, err = file.getChunk() if err == nil { file.rbuf = file.rbuf[int(file.offset-int64(chunk)*int64(file.doc.ChunkSize)):] } return file.offset, err } // Read reads into b the next available data from the file and // returns the number of bytes written and an error in case // something wrong happened. At the end of the file, n will // be zero and err will be set to io.EOF. // // The parameters and behavior of this function turn the file // into an io.Reader. func (file *GridFile) Read(b []byte) (n int, err error) { file.assertMode(gfsReading) file.m.Lock() debugf("GridFile %p: reading at offset %d into buffer of length %d", file, file.offset, len(b)) defer file.m.Unlock() if file.offset == file.doc.Length { return 0, io.EOF } for err == nil { i := copy(b, file.rbuf) n += i file.offset += int64(i) file.rbuf = file.rbuf[i:] if i == len(b) || file.offset == file.doc.Length { break } b = b[i:] file.rbuf, err = file.getChunk() } return n, err } func (file *GridFile) getChunk() (data []byte, err error) { cache := file.rcache file.rcache = nil if cache != nil && cache.n == file.chunk { debugf("GridFile %p: Getting chunk %d from cache", file, file.chunk) cache.wait.Lock() data, err = cache.data, cache.err } else { debugf("GridFile %p: Fetching chunk %d", file, file.chunk) var doc gfsChunk err = file.gfs.Chunks.Find(bson.D{{"files_id", file.doc.Id}, {"n", file.chunk}}).One(&doc) data = doc.Data } file.chunk++ if int64(file.chunk)*int64(file.doc.ChunkSize) < file.doc.Length { // Read the next one in background. cache = &gfsCachedChunk{n: file.chunk} cache.wait.Lock() debugf("GridFile %p: Scheduling chunk %d for background caching", file, file.chunk) // Clone the session to avoid having it closed in between. chunks := file.gfs.Chunks session := chunks.Database.Session.Clone() go func(id interface{}, n int) { defer session.Close() chunks = chunks.With(session) var doc gfsChunk cache.err = chunks.Find(bson.D{{"files_id", id}, {"n", n}}).One(&doc) cache.data = doc.Data cache.wait.Unlock() }(file.doc.Id, file.chunk) file.rcache = cache } debugf("Returning err: %#v", err) return } mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/gridfs_test.go000066400000000000000000000375431276133741700240400ustar00rootroot00000000000000// mgo - MongoDB driver for Go // // Copyright (c) 2010-2012 - Gustavo Niemeyer // // All rights reserved. // // Redistribution and use in source and binary forms, with or without // modification, are permitted provided that the following conditions are met: // // 1. Redistributions of source code must retain the above copyright notice, this // list of conditions and the following disclaimer. // 2. Redistributions in binary form must reproduce the above copyright notice, // this list of conditions and the following disclaimer in the documentation // and/or other materials provided with the distribution. // // THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND // ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED // WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE // DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR // ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES // (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; // LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND // ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT // (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS // SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. package mgo_test import ( "io" "os" "time" . "gopkg.in/check.v1" "gopkg.in/mgo.v2" "gopkg.in/mgo.v2/bson" ) func (s *S) TestGridFSCreate(c *C) { session, err := mgo.Dial("localhost:40011") c.Assert(err, IsNil) defer session.Close() db := session.DB("mydb") before := bson.Now() gfs := db.GridFS("fs") file, err := gfs.Create("") c.Assert(err, IsNil) n, err := file.Write([]byte("some data")) c.Assert(err, IsNil) c.Assert(n, Equals, 9) err = file.Close() c.Assert(err, IsNil) after := bson.Now() // Check the file information. result := M{} err = db.C("fs.files").Find(nil).One(result) c.Assert(err, IsNil) fileId, ok := result["_id"].(bson.ObjectId) c.Assert(ok, Equals, true) c.Assert(fileId.Valid(), Equals, true) result["_id"] = "" ud, ok := result["uploadDate"].(time.Time) c.Assert(ok, Equals, true) c.Assert(ud.After(before) && ud.Before(after), Equals, true) result["uploadDate"] = "" expected := M{ "_id": "", "length": 9, "chunkSize": 255 * 1024, "uploadDate": "", "md5": "1e50210a0202497fb79bc38b6ade6c34", } c.Assert(result, DeepEquals, expected) // Check the chunk. result = M{} err = db.C("fs.chunks").Find(nil).One(result) c.Assert(err, IsNil) chunkId, ok := result["_id"].(bson.ObjectId) c.Assert(ok, Equals, true) c.Assert(chunkId.Valid(), Equals, true) result["_id"] = "" expected = M{ "_id": "", "files_id": fileId, "n": 0, "data": []byte("some data"), } c.Assert(result, DeepEquals, expected) // Check that an index was created. indexes, err := db.C("fs.chunks").Indexes() c.Assert(err, IsNil) c.Assert(len(indexes), Equals, 2) c.Assert(indexes[1].Key, DeepEquals, []string{"files_id", "n"}) } func (s *S) TestGridFSFileDetails(c *C) { session, err := mgo.Dial("localhost:40011") c.Assert(err, IsNil) defer session.Close() db := session.DB("mydb") gfs := db.GridFS("fs") file, err := gfs.Create("myfile1.txt") c.Assert(err, IsNil) n, err := file.Write([]byte("some")) c.Assert(err, IsNil) c.Assert(n, Equals, 4) c.Assert(file.Size(), Equals, int64(4)) n, err = file.Write([]byte(" data")) c.Assert(err, IsNil) c.Assert(n, Equals, 5) c.Assert(file.Size(), Equals, int64(9)) id, _ := file.Id().(bson.ObjectId) c.Assert(id.Valid(), Equals, true) c.Assert(file.Name(), Equals, "myfile1.txt") c.Assert(file.ContentType(), Equals, "") var info interface{} err = file.GetMeta(&info) c.Assert(err, IsNil) c.Assert(info, IsNil) file.SetId("myid") file.SetName("myfile2.txt") file.SetContentType("text/plain") file.SetMeta(M{"any": "thing"}) c.Assert(file.Id(), Equals, "myid") c.Assert(file.Name(), Equals, "myfile2.txt") c.Assert(file.ContentType(), Equals, "text/plain") err = file.GetMeta(&info) c.Assert(err, IsNil) c.Assert(info, DeepEquals, bson.M{"any": "thing"}) err = file.Close() c.Assert(err, IsNil) c.Assert(file.MD5(), Equals, "1e50210a0202497fb79bc38b6ade6c34") ud := file.UploadDate() now := time.Now() c.Assert(ud.Before(now), Equals, true) c.Assert(ud.After(now.Add(-3*time.Second)), Equals, true) result := M{} err = db.C("fs.files").Find(nil).One(result) c.Assert(err, IsNil) result["uploadDate"] = "" expected := M{ "_id": "myid", "length": 9, "chunkSize": 255 * 1024, "uploadDate": "", "md5": "1e50210a0202497fb79bc38b6ade6c34", "filename": "myfile2.txt", "contentType": "text/plain", "metadata": M{"any": "thing"}, } c.Assert(result, DeepEquals, expected) } func (s *S) TestGridFSSetUploadDate(c *C) { session, err := mgo.Dial("localhost:40011") c.Assert(err, IsNil) defer session.Close() db := session.DB("mydb") gfs := db.GridFS("fs") file, err := gfs.Create("") c.Assert(err, IsNil) t := time.Date(2014, 1, 1, 1, 1, 1, 0, time.Local) file.SetUploadDate(t) err = file.Close() c.Assert(err, IsNil) // Check the file information. result := M{} err = db.C("fs.files").Find(nil).One(result) c.Assert(err, IsNil) ud := result["uploadDate"].(time.Time) if !ud.Equal(t) { c.Fatalf("want upload date %s, got %s", t, ud) } } func (s *S) TestGridFSCreateWithChunking(c *C) { session, err := mgo.Dial("localhost:40011") c.Assert(err, IsNil) defer session.Close() db := session.DB("mydb") gfs := db.GridFS("fs") file, err := gfs.Create("") c.Assert(err, IsNil) file.SetChunkSize(5) // Smaller than the chunk size. n, err := file.Write([]byte("abc")) c.Assert(err, IsNil) c.Assert(n, Equals, 3) // Boundary in the middle. n, err = file.Write([]byte("defg")) c.Assert(err, IsNil) c.Assert(n, Equals, 4) // Boundary at the end. n, err = file.Write([]byte("hij")) c.Assert(err, IsNil) c.Assert(n, Equals, 3) // Larger than the chunk size, with 3 chunks. n, err = file.Write([]byte("klmnopqrstuv")) c.Assert(err, IsNil) c.Assert(n, Equals, 12) err = file.Close() c.Assert(err, IsNil) // Check the file information. result := M{} err = db.C("fs.files").Find(nil).One(result) c.Assert(err, IsNil) fileId, _ := result["_id"].(bson.ObjectId) c.Assert(fileId.Valid(), Equals, true) result["_id"] = "" result["uploadDate"] = "" expected := M{ "_id": "", "length": 22, "chunkSize": 5, "uploadDate": "", "md5": "44a66044834cbe55040089cabfc102d5", } c.Assert(result, DeepEquals, expected) // Check the chunks. iter := db.C("fs.chunks").Find(nil).Sort("n").Iter() dataChunks := []string{"abcde", "fghij", "klmno", "pqrst", "uv"} for i := 0; ; i++ { result = M{} if !iter.Next(result) { if i != 5 { c.Fatalf("Expected 5 chunks, got %d", i) } break } c.Assert(iter.Close(), IsNil) result["_id"] = "" expected = M{ "_id": "", "files_id": fileId, "n": i, "data": []byte(dataChunks[i]), } c.Assert(result, DeepEquals, expected) } } func (s *S) TestGridFSAbort(c *C) { session, err := mgo.Dial("localhost:40011") c.Assert(err, IsNil) defer session.Close() db := session.DB("mydb") gfs := db.GridFS("fs") file, err := gfs.Create("") c.Assert(err, IsNil) file.SetChunkSize(5) n, err := file.Write([]byte("some data")) c.Assert(err, IsNil) c.Assert(n, Equals, 9) var count int for i := 0; i < 10; i++ { count, err = db.C("fs.chunks").Count() if count > 0 || err != nil { break } } c.Assert(err, IsNil) c.Assert(count, Equals, 1) file.Abort() err = file.Close() c.Assert(err, ErrorMatches, "write aborted") count, err = db.C("fs.chunks").Count() c.Assert(err, IsNil) c.Assert(count, Equals, 0) } func (s *S) TestGridFSCloseConflict(c *C) { session, err := mgo.Dial("localhost:40011") c.Assert(err, IsNil) defer session.Close() db := session.DB("mydb") db.C("fs.files").EnsureIndex(mgo.Index{Key: []string{"filename"}, Unique: true}) // For a closing-time conflict err = db.C("fs.files").Insert(M{"filename": "foo.txt"}) c.Assert(err, IsNil) gfs := db.GridFS("fs") file, err := gfs.Create("foo.txt") c.Assert(err, IsNil) _, err = file.Write([]byte("some data")) c.Assert(err, IsNil) err = file.Close() c.Assert(mgo.IsDup(err), Equals, true) count, err := db.C("fs.chunks").Count() c.Assert(err, IsNil) c.Assert(count, Equals, 0) } func (s *S) TestGridFSOpenNotFound(c *C) { session, err := mgo.Dial("localhost:40011") c.Assert(err, IsNil) defer session.Close() db := session.DB("mydb") gfs := db.GridFS("fs") file, err := gfs.OpenId("non-existent") c.Assert(err == mgo.ErrNotFound, Equals, true) c.Assert(file, IsNil) file, err = gfs.Open("non-existent") c.Assert(err == mgo.ErrNotFound, Equals, true) c.Assert(file, IsNil) } func (s *S) TestGridFSReadAll(c *C) { session, err := mgo.Dial("localhost:40011") c.Assert(err, IsNil) defer session.Close() db := session.DB("mydb") gfs := db.GridFS("fs") file, err := gfs.Create("") c.Assert(err, IsNil) id := file.Id() file.SetChunkSize(5) n, err := file.Write([]byte("abcdefghijklmnopqrstuv")) c.Assert(err, IsNil) c.Assert(n, Equals, 22) err = file.Close() c.Assert(err, IsNil) file, err = gfs.OpenId(id) c.Assert(err, IsNil) b := make([]byte, 30) n, err = file.Read(b) c.Assert(n, Equals, 22) c.Assert(err, IsNil) n, err = file.Read(b) c.Assert(n, Equals, 0) c.Assert(err == io.EOF, Equals, true) err = file.Close() c.Assert(err, IsNil) } func (s *S) TestGridFSReadChunking(c *C) { session, err := mgo.Dial("localhost:40011") c.Assert(err, IsNil) defer session.Close() db := session.DB("mydb") gfs := db.GridFS("fs") file, err := gfs.Create("") c.Assert(err, IsNil) id := file.Id() file.SetChunkSize(5) n, err := file.Write([]byte("abcdefghijklmnopqrstuv")) c.Assert(err, IsNil) c.Assert(n, Equals, 22) err = file.Close() c.Assert(err, IsNil) file, err = gfs.OpenId(id) c.Assert(err, IsNil) b := make([]byte, 30) // Smaller than the chunk size. n, err = file.Read(b[:3]) c.Assert(err, IsNil) c.Assert(n, Equals, 3) c.Assert(b[:3], DeepEquals, []byte("abc")) // Boundary in the middle. n, err = file.Read(b[:4]) c.Assert(err, IsNil) c.Assert(n, Equals, 4) c.Assert(b[:4], DeepEquals, []byte("defg")) // Boundary at the end. n, err = file.Read(b[:3]) c.Assert(err, IsNil) c.Assert(n, Equals, 3) c.Assert(b[:3], DeepEquals, []byte("hij")) // Larger than the chunk size, with 3 chunks. n, err = file.Read(b) c.Assert(err, IsNil) c.Assert(n, Equals, 12) c.Assert(b[:12], DeepEquals, []byte("klmnopqrstuv")) n, err = file.Read(b) c.Assert(n, Equals, 0) c.Assert(err == io.EOF, Equals, true) err = file.Close() c.Assert(err, IsNil) } func (s *S) TestGridFSOpen(c *C) { session, err := mgo.Dial("localhost:40011") c.Assert(err, IsNil) defer session.Close() db := session.DB("mydb") gfs := db.GridFS("fs") file, err := gfs.Create("myfile.txt") c.Assert(err, IsNil) file.Write([]byte{'1'}) file.Close() file, err = gfs.Create("myfile.txt") c.Assert(err, IsNil) file.Write([]byte{'2'}) file.Close() file, err = gfs.Open("myfile.txt") c.Assert(err, IsNil) defer file.Close() var b [1]byte _, err = file.Read(b[:]) c.Assert(err, IsNil) c.Assert(string(b[:]), Equals, "2") } func (s *S) TestGridFSSeek(c *C) { session, err := mgo.Dial("localhost:40011") c.Assert(err, IsNil) defer session.Close() db := session.DB("mydb") gfs := db.GridFS("fs") file, err := gfs.Create("") c.Assert(err, IsNil) id := file.Id() file.SetChunkSize(5) n, err := file.Write([]byte("abcdefghijklmnopqrstuv")) c.Assert(err, IsNil) c.Assert(n, Equals, 22) err = file.Close() c.Assert(err, IsNil) b := make([]byte, 5) file, err = gfs.OpenId(id) c.Assert(err, IsNil) o, err := file.Seek(3, os.SEEK_SET) c.Assert(err, IsNil) c.Assert(o, Equals, int64(3)) _, err = file.Read(b) c.Assert(err, IsNil) c.Assert(b, DeepEquals, []byte("defgh")) o, err = file.Seek(5, os.SEEK_CUR) c.Assert(err, IsNil) c.Assert(o, Equals, int64(13)) _, err = file.Read(b) c.Assert(err, IsNil) c.Assert(b, DeepEquals, []byte("nopqr")) o, err = file.Seek(0, os.SEEK_END) c.Assert(err, IsNil) c.Assert(o, Equals, int64(22)) n, err = file.Read(b) c.Assert(err, Equals, io.EOF) c.Assert(n, Equals, 0) o, err = file.Seek(-10, os.SEEK_END) c.Assert(err, IsNil) c.Assert(o, Equals, int64(12)) _, err = file.Read(b) c.Assert(err, IsNil) c.Assert(b, DeepEquals, []byte("mnopq")) o, err = file.Seek(8, os.SEEK_SET) c.Assert(err, IsNil) c.Assert(o, Equals, int64(8)) _, err = file.Read(b) c.Assert(err, IsNil) c.Assert(b, DeepEquals, []byte("ijklm")) // Trivial seek forward within same chunk. Already // got the data, shouldn't touch the database. sent := mgo.GetStats().SentOps o, err = file.Seek(1, os.SEEK_CUR) c.Assert(err, IsNil) c.Assert(o, Equals, int64(14)) c.Assert(mgo.GetStats().SentOps, Equals, sent) _, err = file.Read(b) c.Assert(err, IsNil) c.Assert(b, DeepEquals, []byte("opqrs")) // Try seeking past end of file. file.Seek(3, os.SEEK_SET) o, err = file.Seek(23, os.SEEK_SET) c.Assert(err, ErrorMatches, "seek past end of file") c.Assert(o, Equals, int64(3)) } func (s *S) TestGridFSRemoveId(c *C) { session, err := mgo.Dial("localhost:40011") c.Assert(err, IsNil) defer session.Close() db := session.DB("mydb") gfs := db.GridFS("fs") file, err := gfs.Create("myfile.txt") c.Assert(err, IsNil) file.Write([]byte{'1'}) file.Close() file, err = gfs.Create("myfile.txt") c.Assert(err, IsNil) file.Write([]byte{'2'}) id := file.Id() file.Close() err = gfs.RemoveId(id) c.Assert(err, IsNil) file, err = gfs.Open("myfile.txt") c.Assert(err, IsNil) defer file.Close() var b [1]byte _, err = file.Read(b[:]) c.Assert(err, IsNil) c.Assert(string(b[:]), Equals, "1") n, err := db.C("fs.chunks").Find(M{"files_id": id}).Count() c.Assert(err, IsNil) c.Assert(n, Equals, 0) } func (s *S) TestGridFSRemove(c *C) { session, err := mgo.Dial("localhost:40011") c.Assert(err, IsNil) defer session.Close() db := session.DB("mydb") gfs := db.GridFS("fs") file, err := gfs.Create("myfile.txt") c.Assert(err, IsNil) file.Write([]byte{'1'}) file.Close() file, err = gfs.Create("myfile.txt") c.Assert(err, IsNil) file.Write([]byte{'2'}) file.Close() err = gfs.Remove("myfile.txt") c.Assert(err, IsNil) _, err = gfs.Open("myfile.txt") c.Assert(err == mgo.ErrNotFound, Equals, true) n, err := db.C("fs.chunks").Find(nil).Count() c.Assert(err, IsNil) c.Assert(n, Equals, 0) } func (s *S) TestGridFSOpenNext(c *C) { session, err := mgo.Dial("localhost:40011") c.Assert(err, IsNil) defer session.Close() db := session.DB("mydb") gfs := db.GridFS("fs") file, err := gfs.Create("myfile1.txt") c.Assert(err, IsNil) file.Write([]byte{'1'}) file.Close() file, err = gfs.Create("myfile2.txt") c.Assert(err, IsNil) file.Write([]byte{'2'}) file.Close() var f *mgo.GridFile var b [1]byte iter := gfs.Find(nil).Sort("-filename").Iter() ok := gfs.OpenNext(iter, &f) c.Assert(ok, Equals, true) c.Check(f.Name(), Equals, "myfile2.txt") _, err = f.Read(b[:]) c.Assert(err, IsNil) c.Assert(string(b[:]), Equals, "2") ok = gfs.OpenNext(iter, &f) c.Assert(ok, Equals, true) c.Check(f.Name(), Equals, "myfile1.txt") _, err = f.Read(b[:]) c.Assert(err, IsNil) c.Assert(string(b[:]), Equals, "1") ok = gfs.OpenNext(iter, &f) c.Assert(ok, Equals, false) c.Assert(iter.Close(), IsNil) c.Assert(f, IsNil) // Do it again with a more restrictive query to make sure // it's actually taken into account. iter = gfs.Find(bson.M{"filename": "myfile1.txt"}).Iter() ok = gfs.OpenNext(iter, &f) c.Assert(ok, Equals, true) c.Check(f.Name(), Equals, "myfile1.txt") ok = gfs.OpenNext(iter, &f) c.Assert(ok, Equals, false) c.Assert(iter.Close(), IsNil) c.Assert(f, IsNil) } mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/000077500000000000000000000000001276133741700226235ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/certs/000077500000000000000000000000001276133741700237435ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/certs/client.crt000066400000000000000000000022141276133741700257320ustar00rootroot00000000000000-----BEGIN CERTIFICATE----- MIIDLjCCAhYCAQcwDQYJKoZIhvcNAQELBQAwXDELMAkGA1UEBhMCR08xDDAKBgNV BAgMA01HTzEMMAoGA1UEBwwDTUdPMQwwCgYDVQQKDANNR08xDzANBgNVBAsMBlNl cnZlcjESMBAGA1UEAwwJbG9jYWxob3N0MCAXDTE1MDkyOTA4NDAzMFoYDzIxMTUw OTA1MDg0MDMwWjBcMQswCQYDVQQGEwJHTzEMMAoGA1UECAwDTUdPMQwwCgYDVQQH DANNR08xDDAKBgNVBAoMA01HTzEPMA0GA1UECwwGQ2xpZW50MRIwEAYDVQQDDAls b2NhbGhvc3QwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC0UiQhmT+H 4IIqrn8SMESDzvcl3rwImwUoRIHlmXkovCIZCbvBCJ1nAu6X5zIN89EPPOjfNrgZ 616wPgVV/YEQXp+D7+jTAsE5s8JepRXFdecResmvh/+0i2DSuI4QFsuyVAPM1O0I AQ5EKgr0weZZmsX6lhPD4uYehV4DxDE0i/8aTAlDoNgRCAJrYFMharRTDdY7bQzd 7ZYab/pK/3DSmOKxl/AFJ8Enmcj9w1bsvy0fgAgoGEBnBru80PRFpFiqk72TJkXO Hx7zcYFpegtKPbAreTCModaCnjP//fskCp4XJrkfH5+01NeeX/r1OfEbjgE/wzzx l8NaWnPCmxNfAgMBAAEwDQYJKoZIhvcNAQELBQADggEBAFwYpje3dCLDOIHYjd+5 CpFOEb+bJsS4ryqm/NblTjIhCLo58hNpMsBqdJHRbHAFRCOE8fvY8yiWtdHeFZcW DgVRAXfHONLtN7faZaZQnhy/YzOhLfC/8dUMB0gQA8KXhBCPZqQmexE28AfkEO47 PwICAxIWINfjm5VnFMkA3b7bDNLHon/pev2m7HqVQ3pRUJQNK3XgFOdDgRrnuXpR OKAfHORHVGTh1gf1DVwc0oM+0gnkSiJ1VG0n5pE3zhZ24fmZxu6JQ6X515W7APQI /nKVH+f1Fo+ustyTNLt8Bwxi1XmwT7IXwnkVSE9Ff6VejppXRF01V0aaWsa3kU3r z3A= -----END CERTIFICATE----- mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/certs/client.key000066400000000000000000000032131276133741700257320ustar00rootroot00000000000000-----BEGIN RSA PRIVATE KEY----- MIIEogIBAAKCAQEAtFIkIZk/h+CCKq5/EjBEg873Jd68CJsFKESB5Zl5KLwiGQm7 wQidZwLul+cyDfPRDzzo3za4GetesD4FVf2BEF6fg+/o0wLBObPCXqUVxXXnEXrJ r4f/tItg0riOEBbLslQDzNTtCAEORCoK9MHmWZrF+pYTw+LmHoVeA8QxNIv/GkwJ Q6DYEQgCa2BTIWq0Uw3WO20M3e2WGm/6Sv9w0pjisZfwBSfBJ5nI/cNW7L8tH4AI KBhAZwa7vND0RaRYqpO9kyZFzh8e83GBaXoLSj2wK3kwjKHWgp4z//37JAqeFya5 Hx+ftNTXnl/69TnxG44BP8M88ZfDWlpzwpsTXwIDAQABAoIBADzCjOAxZkHfuZyu La0wTHXpkEfXdJ6ltagq5WY7P6MlOYwcRoK152vlhgXzZl9jL6ely4YjRwec0swq KdwezpV4fOGVPmuTuw45bx47HEnr/49ZQ4p9FgF9EYQPofbz53FQc/NaMACJcogv bn+osniw+VMFrOVNmGLiZ5p3Smk8zfXE7GRHO8CL5hpWLWO/aK236yytbfWOjM2f Pr76ICb26TPRNzYaYUEThU6DtgdLU8pLnJ6QKKaDsjn+zqQzRa+Nvc0c0K8gvWwA Afq7t0325+uMSwfpLgCOFldcaZQ5uvteJ0CAVRq1MvStnSHBmMzPlgS+NzsDm6lp QH5+rIkCgYEA5j3jrWsv7TueTNbk8Hr/Zwywc+fA2Ex0pBURBHlHyc6ahSXWSCqo DtvRGX0GDoK1lCfaIf1qb/DLlGaoHpkEeqcNhXQ+hHs+bZAxfbfBY9+ikit5ZTtl QN1tIlhaiyLDnwhkpi/hMw1tiouxJUf84Io61z0sCL4hyZSPCpjn0H0CgYEAyH6F Mwl+bCD3VDL/Dr5WSoOr2B/M3bF5SfvdStwy2IPcDJ716je1Ud/2qFCnKGgqvWhJ +HU15c7CjAWo7/pXq2/pEMD8fDKTYww4Hr4p6duEA7DpbOGkwcUX8u3eknxUWT9F jOSbTCvAxuDOC1K3AElyMxVVTNUrFFe8M84R9gsCgYBXmb6RkdG3WlKde7m5gaLB K4PLZabq5RQQBe/mmtpkfxYtiLrh1FEC7kG9h+MRDExX5V3KRugDVUOv3+shUSjy HbM4ToUm1NloyE78PTj4bfMl2CKlEJcyucy3H5S7kWuKi5/31wnA6d/+sa2huKUP Lai7kgu5+9VRJBPUfV7d5QKBgCnhk/13TDtWH5QtGu5/gBMMskbxTaA5xHZZ8H4E xXJJCRxx0Dje7jduK145itF8AQGT2W/XPC0HJciOHh4TE2EyfWMMjTF8dyFHmimB 28uIGWmT+Q7Pi9UWUMxkOAwtgIksGGE4F+CvexOQPjpLSwL6VKqrGCh2lwsm0J+Z ulLFAoGAKlC93c6XEj1A31c1+usdEhUe9BrmTqtSYLYpDNpeMLdZ3VctrAZuOQPZ 4A4gkkQkqqwZGBYYSEqwqiLU6MsBdHPPZ9u3JXLLOQuh1xGeaKylvHj7qx6iT0Xo I+FkJ6/3JeMgOina/+wlzD4oyQpqR4Mnh+TuLkDfQTgY+Lg0WPk= -----END RSA PRIVATE KEY----- mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/certs/client.pem000066400000000000000000000062651276133741700257350ustar00rootroot00000000000000To regenerate the key: openssl req -newkey rsa:2048 -new -x509 -days 36500 -nodes -out server.crt -keyout server.key cat server.key server.crt > server.pem openssl genrsa -out client.key 2048 openssl req -key client.key -new -out client.req openssl x509 -req -in client.req -CA server.crt -CAkey server.key -days 36500 -CAserial file.srl -out client.crt cat client.key client.crt > client.pem -----BEGIN RSA PRIVATE KEY----- MIIEogIBAAKCAQEAtFIkIZk/h+CCKq5/EjBEg873Jd68CJsFKESB5Zl5KLwiGQm7 wQidZwLul+cyDfPRDzzo3za4GetesD4FVf2BEF6fg+/o0wLBObPCXqUVxXXnEXrJ r4f/tItg0riOEBbLslQDzNTtCAEORCoK9MHmWZrF+pYTw+LmHoVeA8QxNIv/GkwJ Q6DYEQgCa2BTIWq0Uw3WO20M3e2WGm/6Sv9w0pjisZfwBSfBJ5nI/cNW7L8tH4AI KBhAZwa7vND0RaRYqpO9kyZFzh8e83GBaXoLSj2wK3kwjKHWgp4z//37JAqeFya5 Hx+ftNTXnl/69TnxG44BP8M88ZfDWlpzwpsTXwIDAQABAoIBADzCjOAxZkHfuZyu La0wTHXpkEfXdJ6ltagq5WY7P6MlOYwcRoK152vlhgXzZl9jL6ely4YjRwec0swq KdwezpV4fOGVPmuTuw45bx47HEnr/49ZQ4p9FgF9EYQPofbz53FQc/NaMACJcogv bn+osniw+VMFrOVNmGLiZ5p3Smk8zfXE7GRHO8CL5hpWLWO/aK236yytbfWOjM2f Pr76ICb26TPRNzYaYUEThU6DtgdLU8pLnJ6QKKaDsjn+zqQzRa+Nvc0c0K8gvWwA Afq7t0325+uMSwfpLgCOFldcaZQ5uvteJ0CAVRq1MvStnSHBmMzPlgS+NzsDm6lp QH5+rIkCgYEA5j3jrWsv7TueTNbk8Hr/Zwywc+fA2Ex0pBURBHlHyc6ahSXWSCqo DtvRGX0GDoK1lCfaIf1qb/DLlGaoHpkEeqcNhXQ+hHs+bZAxfbfBY9+ikit5ZTtl QN1tIlhaiyLDnwhkpi/hMw1tiouxJUf84Io61z0sCL4hyZSPCpjn0H0CgYEAyH6F Mwl+bCD3VDL/Dr5WSoOr2B/M3bF5SfvdStwy2IPcDJ716je1Ud/2qFCnKGgqvWhJ +HU15c7CjAWo7/pXq2/pEMD8fDKTYww4Hr4p6duEA7DpbOGkwcUX8u3eknxUWT9F jOSbTCvAxuDOC1K3AElyMxVVTNUrFFe8M84R9gsCgYBXmb6RkdG3WlKde7m5gaLB K4PLZabq5RQQBe/mmtpkfxYtiLrh1FEC7kG9h+MRDExX5V3KRugDVUOv3+shUSjy HbM4ToUm1NloyE78PTj4bfMl2CKlEJcyucy3H5S7kWuKi5/31wnA6d/+sa2huKUP Lai7kgu5+9VRJBPUfV7d5QKBgCnhk/13TDtWH5QtGu5/gBMMskbxTaA5xHZZ8H4E xXJJCRxx0Dje7jduK145itF8AQGT2W/XPC0HJciOHh4TE2EyfWMMjTF8dyFHmimB 28uIGWmT+Q7Pi9UWUMxkOAwtgIksGGE4F+CvexOQPjpLSwL6VKqrGCh2lwsm0J+Z ulLFAoGAKlC93c6XEj1A31c1+usdEhUe9BrmTqtSYLYpDNpeMLdZ3VctrAZuOQPZ 4A4gkkQkqqwZGBYYSEqwqiLU6MsBdHPPZ9u3JXLLOQuh1xGeaKylvHj7qx6iT0Xo I+FkJ6/3JeMgOina/+wlzD4oyQpqR4Mnh+TuLkDfQTgY+Lg0WPk= -----END RSA PRIVATE KEY----- -----BEGIN CERTIFICATE----- MIIDLjCCAhYCAQcwDQYJKoZIhvcNAQELBQAwXDELMAkGA1UEBhMCR08xDDAKBgNV BAgMA01HTzEMMAoGA1UEBwwDTUdPMQwwCgYDVQQKDANNR08xDzANBgNVBAsMBlNl cnZlcjESMBAGA1UEAwwJbG9jYWxob3N0MCAXDTE1MDkyOTA4NDAzMFoYDzIxMTUw OTA1MDg0MDMwWjBcMQswCQYDVQQGEwJHTzEMMAoGA1UECAwDTUdPMQwwCgYDVQQH DANNR08xDDAKBgNVBAoMA01HTzEPMA0GA1UECwwGQ2xpZW50MRIwEAYDVQQDDAls b2NhbGhvc3QwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC0UiQhmT+H 4IIqrn8SMESDzvcl3rwImwUoRIHlmXkovCIZCbvBCJ1nAu6X5zIN89EPPOjfNrgZ 616wPgVV/YEQXp+D7+jTAsE5s8JepRXFdecResmvh/+0i2DSuI4QFsuyVAPM1O0I AQ5EKgr0weZZmsX6lhPD4uYehV4DxDE0i/8aTAlDoNgRCAJrYFMharRTDdY7bQzd 7ZYab/pK/3DSmOKxl/AFJ8Enmcj9w1bsvy0fgAgoGEBnBru80PRFpFiqk72TJkXO Hx7zcYFpegtKPbAreTCModaCnjP//fskCp4XJrkfH5+01NeeX/r1OfEbjgE/wzzx l8NaWnPCmxNfAgMBAAEwDQYJKoZIhvcNAQELBQADggEBAFwYpje3dCLDOIHYjd+5 CpFOEb+bJsS4ryqm/NblTjIhCLo58hNpMsBqdJHRbHAFRCOE8fvY8yiWtdHeFZcW DgVRAXfHONLtN7faZaZQnhy/YzOhLfC/8dUMB0gQA8KXhBCPZqQmexE28AfkEO47 PwICAxIWINfjm5VnFMkA3b7bDNLHon/pev2m7HqVQ3pRUJQNK3XgFOdDgRrnuXpR OKAfHORHVGTh1gf1DVwc0oM+0gnkSiJ1VG0n5pE3zhZ24fmZxu6JQ6X515W7APQI /nKVH+f1Fo+ustyTNLt8Bwxi1XmwT7IXwnkVSE9Ff6VejppXRF01V0aaWsa3kU3r z3A= -----END CERTIFICATE----- mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/certs/client.req000066400000000000000000000017351276133741700257400ustar00rootroot00000000000000-----BEGIN CERTIFICATE REQUEST----- MIICoTCCAYkCAQAwXDELMAkGA1UEBhMCR08xDDAKBgNVBAgMA01HTzEMMAoGA1UE BwwDTUdPMQwwCgYDVQQKDANNR08xDzANBgNVBAsMBkNsaWVudDESMBAGA1UEAwwJ bG9jYWxob3N0MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtFIkIZk/ h+CCKq5/EjBEg873Jd68CJsFKESB5Zl5KLwiGQm7wQidZwLul+cyDfPRDzzo3za4 GetesD4FVf2BEF6fg+/o0wLBObPCXqUVxXXnEXrJr4f/tItg0riOEBbLslQDzNTt CAEORCoK9MHmWZrF+pYTw+LmHoVeA8QxNIv/GkwJQ6DYEQgCa2BTIWq0Uw3WO20M 3e2WGm/6Sv9w0pjisZfwBSfBJ5nI/cNW7L8tH4AIKBhAZwa7vND0RaRYqpO9kyZF zh8e83GBaXoLSj2wK3kwjKHWgp4z//37JAqeFya5Hx+ftNTXnl/69TnxG44BP8M8 8ZfDWlpzwpsTXwIDAQABoAAwDQYJKoZIhvcNAQELBQADggEBAKbOFblIscxlXalV sEGNm2oz380RN2QoLhN6nKtAiv0jWm6iKhdAhOIQIeaRPhUP3cyi8bcBvLdMeQ3d ZYIByB55/R0VSP1vs4qkXJCQegHcpMpyuIzsMV8p3Q4lxzGKyKtPA6Bb5c49p8Sk ncD+LL4ymrMEia4cBPsHL9hhFOm4gqDacbU8+ETLTpuoSvUZiw7OwngqhE2r+kMv KDweq5TOPeb+ftKzQKrrfB+XVdBoTKYw6CwARpogbc0/7mvottVcJ/0yAgC1fBbM vupkohkXwKfjxKl6nKNL3R2GkzHQOh91hglAx5zyybKQn2YMM328Vk4X6csBg+pg tb1s0MA= -----END CERTIFICATE REQUEST----- mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/certs/server.crt000066400000000000000000000024161276133741700257660ustar00rootroot00000000000000-----BEGIN CERTIFICATE----- MIIDjTCCAnWgAwIBAgIJAMW+wDfcdzC+MA0GCSqGSIb3DQEBCwUAMFwxCzAJBgNV BAYTAkdPMQwwCgYDVQQIDANNR08xDDAKBgNVBAcMA01HTzEMMAoGA1UECgwDTUdP MQ8wDQYDVQQLDAZTZXJ2ZXIxEjAQBgNVBAMMCWxvY2FsaG9zdDAgFw0xNTA5Mjkw ODM0MTBaGA8yMTE1MDkwNTA4MzQxMFowXDELMAkGA1UEBhMCR08xDDAKBgNVBAgM A01HTzEMMAoGA1UEBwwDTUdPMQwwCgYDVQQKDANNR08xDzANBgNVBAsMBlNlcnZl cjESMBAGA1UEAwwJbG9jYWxob3N0MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB CgKCAQEA/T5W1vTsAF+2gTXP1JKygjM7T/2BXHiJc6DRKVjlshTtPYuC3rpTddDm 6d86d17LWEo+T2bCT4MzZJhSGAun9peFvehdElRMr57xs7j5V1QYjwadMTBkLQuK IAg6cISN1KPUzpUTUKsWIsbx97sA0t0wiEPifROb7nfSMIVQsdz/c9LlY2UNYI+5 GiU88iDGg2wrdsa3U+l2G2KSx/9uE3c5iFki6bdequLiWmBZ6rxfoaLe4gk1INji fKssNsn2i3uJ4i4Tmr3PUc4kxx0mMKuWK3HdlQsMqtpq++HQmHSvsPrbgcjl9HyP JiHDsoJ+4O5bbtcE51oQbLh1bZAhYwIDAQABo1AwTjAdBgNVHQ4EFgQUhku/u9Kd OAc1L0OR649vCCuQT+0wHwYDVR0jBBgwFoAUhku/u9KdOAc1L0OR649vCCuQT+0w DAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAw7Bgw3hlWXWSZjLhnSOu 2mW/UJ2Sj31unHngmgtXwW/04cyzoULb+qmzPe/Z06QMgGIsku1jFBcu0JabQtUG TyalpfW77tfnvz238CYdImYwE9ZcIGuZGfhs6ySFN9XpW43B8YM7R8wTNPvOcSPw nfjqU6kueN4TTspQg9cKhDss5DcMTIdgJgLbITXhIsrCu6GlKOgtX3HrdMGpQX7s UoMXtZVG8pK32vxKWGTZ6DPqESeKjjq74NbYnB3H5U/kDU2dt7LF90C/Umdr9y+C W2OJb1WBrf6RTcbt8D6d7P9kOfLPOtyn/cbaA/pfXBMQMHqr7XNXzjnaNU+jB7hL yQ== -----END CERTIFICATE----- mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/certs/server.key000066400000000000000000000032501276133741700257630ustar00rootroot00000000000000-----BEGIN PRIVATE KEY----- MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQD9PlbW9OwAX7aB Nc/UkrKCMztP/YFceIlzoNEpWOWyFO09i4LeulN10Obp3zp3XstYSj5PZsJPgzNk mFIYC6f2l4W96F0SVEyvnvGzuPlXVBiPBp0xMGQtC4ogCDpwhI3Uo9TOlRNQqxYi xvH3uwDS3TCIQ+J9E5vud9IwhVCx3P9z0uVjZQ1gj7kaJTzyIMaDbCt2xrdT6XYb YpLH/24TdzmIWSLpt16q4uJaYFnqvF+hot7iCTUg2OJ8qyw2yfaLe4niLhOavc9R ziTHHSYwq5Yrcd2VCwyq2mr74dCYdK+w+tuByOX0fI8mIcOygn7g7ltu1wTnWhBs uHVtkCFjAgMBAAECggEASRAfRc1L+Z+jrAu2doIMdnwJdL6S//bW0UFolyFKw+I9 wC/sBg6D3c3zkS4SVDZJPKPO7mGbVg1oWnGH3eAfCYoV0ACmOY+QwGp/GXcYmRVu MHWcDIEFpelaZHt7QNM9iEfsMd3YwMFblZUIYozVZADk66uKQMPTjS2Muur7qRSi wuVfSmsVZ5afH3B1Tr96BbmPsHrXLjvNpjO44k2wrnnSPQjUL7+YiZPvtnNW8Fby yuo2uoAyjg3+68PYZftOvvNneMsv1uyGlUs6Bk+DVWaqofIztWFdFZyXbHnK2PTk eGQt5EsL+RwIck5eoqd5vSE+KyzhhydL0zcpngVQoQKBgQD/Yelvholbz5NQtSy3 ZoiW1y7hL1BKzvVNHuAMKJ5WOnj5szhjhKxt/wZ+hk0qcAmlV9WAPbf4izbEwPRC tnMBQzf1uBxqqbLL6WZ4YAyGrcX3UrT7GXsGfVT4zJjz7oYSw8aPircecw5V4exB xa4NF+ki8IycXSkHwvW2R56fRwKBgQD92xpxXtte/rUnmENbQmr0aKg7JEfMoih6 MdX+f6mfgjMmqj+L4jPTI8/ql8HEy13SQS1534aDSHO+nBqBK5aHUCRMIgSLnTP9 Xyx9Ngg03SZIkPfykqxQmnZgWkTPMhYS+K1Ao9FGVs8W5jVi7veyAdhHptAcxhP3 IuxvrxVTBQKBgQCluMPiu0snaOwP04HRAZhhSgIB3tIbuXE1OnPpb/JPwmH+p25Q Jig+uN9d+4jXoRyhTv4c2fAoOS6xPwVCxWKbzyLhMTg/fx+ncy4rryhxvRJaDDGl QEO1Ul9xlFMs9/vI8YJIY5uxBrimwpStmbn4hSukoLSeQ1X802bfglpMwQKBgD8z GTY4Y20XBIrDAaHquy32EEwJEEcF6AXj+l7N8bDgfVOW9xMgUb6zH8RL29Xeu5Do 4SWCXL66fvZpbr/R1jwB28eIgJExpgvicfUKSqi+lhVi4hfmJDg8/FOopZDf61b1 ykxZfHSCkDQnRAtJaylKBEpyYUWImtfgPfTgJfLxAoGAc8A/Tl2h/DsdTA+cA5d7 1e0l64m13ObruSWRczyru4hy8Yq6E/K2rOFw8cYCcFpy24NqNlk+2iXPLRpWm2zt 9R497zAPvhK/bfPXjvm0j/VjB44lvRTC9hby/RRMHy9UJk4o/UQaD+1IodxZovvk SruEA1+5bfBRMW0P+h7Qfe4= -----END PRIVATE KEY----- mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/certs/server.pem000066400000000000000000000056661276133741700257710ustar00rootroot00000000000000-----BEGIN PRIVATE KEY----- MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQD9PlbW9OwAX7aB Nc/UkrKCMztP/YFceIlzoNEpWOWyFO09i4LeulN10Obp3zp3XstYSj5PZsJPgzNk mFIYC6f2l4W96F0SVEyvnvGzuPlXVBiPBp0xMGQtC4ogCDpwhI3Uo9TOlRNQqxYi xvH3uwDS3TCIQ+J9E5vud9IwhVCx3P9z0uVjZQ1gj7kaJTzyIMaDbCt2xrdT6XYb YpLH/24TdzmIWSLpt16q4uJaYFnqvF+hot7iCTUg2OJ8qyw2yfaLe4niLhOavc9R ziTHHSYwq5Yrcd2VCwyq2mr74dCYdK+w+tuByOX0fI8mIcOygn7g7ltu1wTnWhBs uHVtkCFjAgMBAAECggEASRAfRc1L+Z+jrAu2doIMdnwJdL6S//bW0UFolyFKw+I9 wC/sBg6D3c3zkS4SVDZJPKPO7mGbVg1oWnGH3eAfCYoV0ACmOY+QwGp/GXcYmRVu MHWcDIEFpelaZHt7QNM9iEfsMd3YwMFblZUIYozVZADk66uKQMPTjS2Muur7qRSi wuVfSmsVZ5afH3B1Tr96BbmPsHrXLjvNpjO44k2wrnnSPQjUL7+YiZPvtnNW8Fby yuo2uoAyjg3+68PYZftOvvNneMsv1uyGlUs6Bk+DVWaqofIztWFdFZyXbHnK2PTk eGQt5EsL+RwIck5eoqd5vSE+KyzhhydL0zcpngVQoQKBgQD/Yelvholbz5NQtSy3 ZoiW1y7hL1BKzvVNHuAMKJ5WOnj5szhjhKxt/wZ+hk0qcAmlV9WAPbf4izbEwPRC tnMBQzf1uBxqqbLL6WZ4YAyGrcX3UrT7GXsGfVT4zJjz7oYSw8aPircecw5V4exB xa4NF+ki8IycXSkHwvW2R56fRwKBgQD92xpxXtte/rUnmENbQmr0aKg7JEfMoih6 MdX+f6mfgjMmqj+L4jPTI8/ql8HEy13SQS1534aDSHO+nBqBK5aHUCRMIgSLnTP9 Xyx9Ngg03SZIkPfykqxQmnZgWkTPMhYS+K1Ao9FGVs8W5jVi7veyAdhHptAcxhP3 IuxvrxVTBQKBgQCluMPiu0snaOwP04HRAZhhSgIB3tIbuXE1OnPpb/JPwmH+p25Q Jig+uN9d+4jXoRyhTv4c2fAoOS6xPwVCxWKbzyLhMTg/fx+ncy4rryhxvRJaDDGl QEO1Ul9xlFMs9/vI8YJIY5uxBrimwpStmbn4hSukoLSeQ1X802bfglpMwQKBgD8z GTY4Y20XBIrDAaHquy32EEwJEEcF6AXj+l7N8bDgfVOW9xMgUb6zH8RL29Xeu5Do 4SWCXL66fvZpbr/R1jwB28eIgJExpgvicfUKSqi+lhVi4hfmJDg8/FOopZDf61b1 ykxZfHSCkDQnRAtJaylKBEpyYUWImtfgPfTgJfLxAoGAc8A/Tl2h/DsdTA+cA5d7 1e0l64m13ObruSWRczyru4hy8Yq6E/K2rOFw8cYCcFpy24NqNlk+2iXPLRpWm2zt 9R497zAPvhK/bfPXjvm0j/VjB44lvRTC9hby/RRMHy9UJk4o/UQaD+1IodxZovvk SruEA1+5bfBRMW0P+h7Qfe4= -----END PRIVATE KEY----- -----BEGIN CERTIFICATE----- MIIDjTCCAnWgAwIBAgIJAMW+wDfcdzC+MA0GCSqGSIb3DQEBCwUAMFwxCzAJBgNV BAYTAkdPMQwwCgYDVQQIDANNR08xDDAKBgNVBAcMA01HTzEMMAoGA1UECgwDTUdP MQ8wDQYDVQQLDAZTZXJ2ZXIxEjAQBgNVBAMMCWxvY2FsaG9zdDAgFw0xNTA5Mjkw ODM0MTBaGA8yMTE1MDkwNTA4MzQxMFowXDELMAkGA1UEBhMCR08xDDAKBgNVBAgM A01HTzEMMAoGA1UEBwwDTUdPMQwwCgYDVQQKDANNR08xDzANBgNVBAsMBlNlcnZl cjESMBAGA1UEAwwJbG9jYWxob3N0MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB CgKCAQEA/T5W1vTsAF+2gTXP1JKygjM7T/2BXHiJc6DRKVjlshTtPYuC3rpTddDm 6d86d17LWEo+T2bCT4MzZJhSGAun9peFvehdElRMr57xs7j5V1QYjwadMTBkLQuK IAg6cISN1KPUzpUTUKsWIsbx97sA0t0wiEPifROb7nfSMIVQsdz/c9LlY2UNYI+5 GiU88iDGg2wrdsa3U+l2G2KSx/9uE3c5iFki6bdequLiWmBZ6rxfoaLe4gk1INji fKssNsn2i3uJ4i4Tmr3PUc4kxx0mMKuWK3HdlQsMqtpq++HQmHSvsPrbgcjl9HyP JiHDsoJ+4O5bbtcE51oQbLh1bZAhYwIDAQABo1AwTjAdBgNVHQ4EFgQUhku/u9Kd OAc1L0OR649vCCuQT+0wHwYDVR0jBBgwFoAUhku/u9KdOAc1L0OR649vCCuQT+0w DAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAw7Bgw3hlWXWSZjLhnSOu 2mW/UJ2Sj31unHngmgtXwW/04cyzoULb+qmzPe/Z06QMgGIsku1jFBcu0JabQtUG TyalpfW77tfnvz238CYdImYwE9ZcIGuZGfhs6ySFN9XpW43B8YM7R8wTNPvOcSPw nfjqU6kueN4TTspQg9cKhDss5DcMTIdgJgLbITXhIsrCu6GlKOgtX3HrdMGpQX7s UoMXtZVG8pK32vxKWGTZ6DPqESeKjjq74NbYnB3H5U/kDU2dt7LF90C/Umdr9y+C W2OJb1WBrf6RTcbt8D6d7P9kOfLPOtyn/cbaA/pfXBMQMHqr7XNXzjnaNU+jB7hL yQ== -----END CERTIFICATE----- mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/000077500000000000000000000000001276133741700242515ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/.env000066400000000000000000000022371276133741700250460ustar00rootroot00000000000000 set -e MONGOVERSION=$(mongod --version | sed -n 's/.*v\([0-9]\+\.[0-9]\+\)\..*/\1/p') MONGOMAJOR=$(echo $MONGOVERSION | sed 's/\([0-9]\+\)\..*/\1/') MONGOMINOR=$(echo $MONGOVERSION | sed 's/[0-9]\+\.\([0-9]\+\)/\1/') versionAtLeast() { TESTMAJOR="$1" TESTMINOR="$2" if [ "$MONGOMAJOR" -gt "$TESTMAJOR" ]; then return 0 fi if [ "$MONGOMAJOR" -lt "$TESTMAJOR" ]; then return 100 fi if [ "$MONGOMINOR" -ge "$TESTMINOR" ]; then return 0 fi return 100 } COMMONDOPTSNOIP=" --nohttpinterface --noprealloc --nojournal --smallfiles --nssize=1 --oplogSize=1 --dbpath ./db " COMMONDOPTS=" $COMMONDOPTSNOIP --bind_ip=127.0.0.1 " COMMONCOPTS=" $COMMONDOPTS " COMMONSOPTS=" --chunkSize 1 --bind_ip=127.0.0.1 " if versionAtLeast 3 2; then # 3.2 doesn't like --nojournal on config servers. #COMMONCOPTS="$(echo "$COMMONCOPTS" | sed '/--nojournal/d')" # Using a hacked version of MongoDB 3.2 for now. # Go back to MMAPv1 so it's not super sluggish. :-( COMMONDOPTSNOIP="--storageEngine=mmapv1 $COMMONDOPTSNOIP" COMMONDOPTS="--storageEngine=mmapv1 $COMMONDOPTS" COMMONCOPTS="--storageEngine=mmapv1 $COMMONCOPTS" fi if [ "$TRAVIS" = true ]; then set -x fi mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/cfg1/000077500000000000000000000000001276133741700250715ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/cfg1/db/000077500000000000000000000000001276133741700254565ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/cfg1/db/.empty000066400000000000000000000000001276133741700266030ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/cfg1/db/journal/000077500000000000000000000000001276133741700271305ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/cfg1/db/journal/tempLatencyTest000066400000000000000000006200001276133741700321760ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/cfg1/db/mongod.lock000077500000000000000000000000001276133741700276040ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/cfg1/log/000077500000000000000000000000001276133741700256525ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/cfg1/log/run000077500000000000000000000000401276133741700263760ustar00rootroot00000000000000#!/bin/sh exec cat - > log.txt mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/cfg1/run000077500000000000000000000001171276133741700256220ustar00rootroot00000000000000#!/bin/sh . ../.env exec mongod $COMMONCOPTS \ --port 40101 \ --configsvr mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/cfg2/000077500000000000000000000000001276133741700250725ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/cfg2/db/000077500000000000000000000000001276133741700254575ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/cfg2/db/.empty000066400000000000000000000000001276133741700266040ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/cfg2/log/000077500000000000000000000000001276133741700256535ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/cfg2/log/run000077500000000000000000000000401276133741700263770ustar00rootroot00000000000000#!/bin/sh exec cat - > log.txt mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/cfg2/run000077500000000000000000000001171276133741700256230ustar00rootroot00000000000000#!/bin/sh . ../.env exec mongod $COMMONCOPTS \ --port 40102 \ --configsvr mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/cfg3/000077500000000000000000000000001276133741700250735ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/cfg3/db/000077500000000000000000000000001276133741700254605ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/cfg3/db/.empty000066400000000000000000000000001276133741700266050ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/cfg3/log/000077500000000000000000000000001276133741700256545ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/cfg3/log/run000077500000000000000000000000401276133741700264000ustar00rootroot00000000000000#!/bin/sh exec cat - > log.txt mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/cfg3/run000077500000000000000000000001711276133741700256240ustar00rootroot00000000000000#!/bin/sh . ../.env exec mongod $COMMONCOPTS \ --port 40103 \ --configsvr \ --auth \ --keyFile=../../certs/keyfile mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/db1/000077500000000000000000000000001276133741700247175ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/db1/db/000077500000000000000000000000001276133741700253045ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/db1/db/.empty000066400000000000000000000000001276133741700264310ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/db1/log/000077500000000000000000000000001276133741700255005ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/db1/log/run000077500000000000000000000000401276133741700262240ustar00rootroot00000000000000#!/bin/sh exec cat - > log.txt mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/db1/run000077500000000000000000000003001276133741700254420ustar00rootroot00000000000000#!/bin/sh . ../.env if [ x$NOIPV6 = x1 ]; then BINDIP="127.0.0.1" else BINDIP="127.0.0.1,::1" fi exec mongod $COMMONDOPTSNOIP \ --shardsvr \ --bind_ip=$BINDIP \ --port 40001 \ --ipv6 mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/db2/000077500000000000000000000000001276133741700247205ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/db2/db/000077500000000000000000000000001276133741700253055ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/db2/db/.empty000066400000000000000000000000001276133741700264320ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/db2/log/000077500000000000000000000000001276133741700255015ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/db2/log/run000077500000000000000000000000401276133741700262250ustar00rootroot00000000000000#!/bin/sh exec cat - > log.txt mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/db2/run000077500000000000000000000001271276133741700254520ustar00rootroot00000000000000#!/bin/sh . ../.env exec mongod $COMMONDOPTS \ --shardsvr \ --port 40002 \ --auth mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/db3/000077500000000000000000000000001276133741700247215ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/db3/db/000077500000000000000000000000001276133741700253065ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/db3/db/.empty000066400000000000000000000000001276133741700264330ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/db3/log/000077500000000000000000000000001276133741700255025ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/db3/log/run000077500000000000000000000000401276133741700262260ustar00rootroot00000000000000#!/bin/sh exec cat - > log.txt mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/db3/run000077500000000000000000000002771276133741700254610ustar00rootroot00000000000000#!/bin/sh . ../.env exec mongod $COMMONDOPTS \ --shardsvr \ --port 40003 \ --auth \ --sslMode preferSSL \ --sslCAFile ../../certs/server.pem \ --sslPEMKeyFile ../../certs/server.pem mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs1a/000077500000000000000000000000001276133741700251175ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs1a/db/000077500000000000000000000000001276133741700255045ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs1a/db/.empty000066400000000000000000000000001276133741700266310ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs1a/log/000077500000000000000000000000001276133741700257005ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs1a/log/run000077500000000000000000000000401276133741700264240ustar00rootroot00000000000000#!/bin/sh exec cat - > log.txt mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs1a/run000077500000000000000000000001361276133741700256510ustar00rootroot00000000000000#!/bin/sh . ../.env exec mongod $COMMONDOPTS \ --shardsvr \ --replSet rs1 \ --port 40011 mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs1b/000077500000000000000000000000001276133741700251205ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs1b/db/000077500000000000000000000000001276133741700255055ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs1b/db/.empty000066400000000000000000000000001276133741700266320ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs1b/log/000077500000000000000000000000001276133741700257015ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs1b/log/run000077500000000000000000000000401276133741700264250ustar00rootroot00000000000000#!/bin/sh exec cat - > log.txt mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs1b/run000077500000000000000000000001361276133741700256520ustar00rootroot00000000000000#!/bin/sh . ../.env exec mongod $COMMONDOPTS \ --shardsvr \ --replSet rs1 \ --port 40012 mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs1c/000077500000000000000000000000001276133741700251215ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs1c/db/000077500000000000000000000000001276133741700255065ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs1c/db/.empty000066400000000000000000000000001276133741700266330ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs1c/log/000077500000000000000000000000001276133741700257025ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs1c/log/run000077500000000000000000000000401276133741700264260ustar00rootroot00000000000000#!/bin/sh exec cat - > log.txt mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs1c/run000077500000000000000000000001361276133741700256530ustar00rootroot00000000000000#!/bin/sh . ../.env exec mongod $COMMONDOPTS \ --shardsvr \ --replSet rs1 \ --port 40013 mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs2a/000077500000000000000000000000001276133741700251205ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs2a/db/000077500000000000000000000000001276133741700255055ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs2a/db/.empty000066400000000000000000000000001276133741700266320ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs2a/log/000077500000000000000000000000001276133741700257015ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs2a/log/run000077500000000000000000000000401276133741700264250ustar00rootroot00000000000000#!/bin/sh exec cat - > log.txt mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs2a/run000077500000000000000000000001361276133741700256520ustar00rootroot00000000000000#!/bin/sh . ../.env exec mongod $COMMONDOPTS \ --shardsvr \ --replSet rs2 \ --port 40021 mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs2b/000077500000000000000000000000001276133741700251215ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs2b/db/000077500000000000000000000000001276133741700255065ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs2b/db/.empty000066400000000000000000000000001276133741700266330ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs2b/log/000077500000000000000000000000001276133741700257025ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs2b/log/run000077500000000000000000000000401276133741700264260ustar00rootroot00000000000000#!/bin/sh exec cat - > log.txt mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs2b/run000077500000000000000000000001361276133741700256530ustar00rootroot00000000000000#!/bin/sh . ../.env exec mongod $COMMONDOPTS \ --shardsvr \ --replSet rs2 \ --port 40022 mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs2c/000077500000000000000000000000001276133741700251225ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs2c/db/000077500000000000000000000000001276133741700255075ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs2c/db/.empty000066400000000000000000000000001276133741700266340ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs2c/log/000077500000000000000000000000001276133741700257035ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs2c/log/run000077500000000000000000000000401276133741700264270ustar00rootroot00000000000000#!/bin/sh exec cat - > log.txt mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs2c/run000077500000000000000000000001361276133741700256540ustar00rootroot00000000000000#!/bin/sh . ../.env exec mongod $COMMONDOPTS \ --shardsvr \ --replSet rs2 \ --port 40023 mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs3a/000077500000000000000000000000001276133741700251215ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs3a/db/000077500000000000000000000000001276133741700255065ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs3a/db/.empty000066400000000000000000000000001276133741700266330ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs3a/log/000077500000000000000000000000001276133741700257025ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs3a/log/run000077500000000000000000000000401276133741700264260ustar00rootroot00000000000000#!/bin/sh exec cat - > log.txt mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs3a/run000077500000000000000000000001771276133741700256600ustar00rootroot00000000000000#!/bin/sh . ../.env exec mongod $COMMONDOPTS \ --shardsvr \ --replSet rs3 \ --port 40031 \ --keyFile=../../certs/keyfile mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs3b/000077500000000000000000000000001276133741700251225ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs3b/db/000077500000000000000000000000001276133741700255075ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs3b/db/.empty000066400000000000000000000000001276133741700266340ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs3b/log/000077500000000000000000000000001276133741700257035ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs3b/log/run000077500000000000000000000000401276133741700264270ustar00rootroot00000000000000#!/bin/sh exec cat - > log.txt mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs3b/run000077500000000000000000000001771276133741700256610ustar00rootroot00000000000000#!/bin/sh . ../.env exec mongod $COMMONDOPTS \ --shardsvr \ --replSet rs3 \ --port 40032 \ --keyFile=../../certs/keyfile mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs3c/000077500000000000000000000000001276133741700251235ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs3c/db/000077500000000000000000000000001276133741700255105ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs3c/db/.empty000066400000000000000000000000001276133741700266350ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs3c/log/000077500000000000000000000000001276133741700257045ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs3c/log/run000077500000000000000000000000401276133741700264300ustar00rootroot00000000000000#!/bin/sh exec cat - > log.txt mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs3c/run000077500000000000000000000001771276133741700256620ustar00rootroot00000000000000#!/bin/sh . ../.env exec mongod $COMMONDOPTS \ --shardsvr \ --replSet rs3 \ --port 40033 \ --keyFile=../../certs/keyfile mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs4a/000077500000000000000000000000001276133741700251225ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs4a/db/000077500000000000000000000000001276133741700255075ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs4a/db/.empty000066400000000000000000000000001276133741700266340ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs4a/log/000077500000000000000000000000001276133741700257035ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs4a/log/run000077500000000000000000000000401276133741700264270ustar00rootroot00000000000000#!/bin/sh exec cat - > log.txt mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/rs4a/run000077500000000000000000000001361276133741700256540ustar00rootroot00000000000000#!/bin/sh . ../.env exec mongod $COMMONDOPTS \ --shardsvr \ --replSet rs4 \ --port 40041 mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/s1/000077500000000000000000000000001276133741700245745ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/s1/log/000077500000000000000000000000001276133741700253555ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/s1/log/run000077500000000000000000000000401276133741700261010ustar00rootroot00000000000000#!/bin/sh exec cat - > log.txt mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/s1/run000077500000000000000000000001351276133741700253250ustar00rootroot00000000000000#!/bin/sh . ../.env exec mongos $COMMONSOPTS \ --port 40201 \ --configdb 127.0.0.1:40101 mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/s2/000077500000000000000000000000001276133741700245755ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/s2/log/000077500000000000000000000000001276133741700253565ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/s2/log/run000077500000000000000000000000401276133741700261020ustar00rootroot00000000000000#!/bin/sh exec cat - > log.txt mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/s2/run000077500000000000000000000001351276133741700253260ustar00rootroot00000000000000#!/bin/sh . ../.env exec mongos $COMMONSOPTS \ --port 40202 \ --configdb 127.0.0.1:40102 mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/s3/000077500000000000000000000000001276133741700245765ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/s3/log/000077500000000000000000000000001276133741700253575ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/s3/log/run000077500000000000000000000000401276133741700261030ustar00rootroot00000000000000#!/bin/sh exec cat - > log.txt mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/daemons/s3/run000077500000000000000000000001761276133741700253340ustar00rootroot00000000000000#!/bin/sh . ../.env exec mongos $COMMONSOPTS \ --port 40203 \ --configdb 127.0.0.1:40103 \ --keyFile=../../certs/keyfile mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/mongojs/000077500000000000000000000000001276133741700242775ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/mongojs/dropall.js000066400000000000000000000036561276133741700263040ustar00rootroot00000000000000 var ports = [40001, 40002, 40011, 40012, 40013, 40021, 40022, 40023, 40041, 40101, 40102, 40103, 40201, 40202, 40203] var auth = [40002, 40103, 40203, 40031] var db1 = new Mongo("localhost:40001") if (db1.getDB("admin").serverBuildInfo().OpenSSLVersion) { ports.push(40003) auth.push(40003) } for (var i in ports) { var port = ports[i] var server = "localhost:" + port var mongo = new Mongo("localhost:" + port) var admin = mongo.getDB("admin") for (var j in auth) { if (auth[j] == port) { admin.auth("root", "rapadura") admin.system.users.find().forEach(function(u) { if (u.user == "root" || u.user == "reader") { return; } if (typeof admin.dropUser == "function") { mongo.getDB(u.db).dropUser(u.user); } else { admin.removeUser(u.user); } }) break } } var result = admin.runCommand({"listDatabases": 1}) for (var j = 0; j != 100; j++) { if (typeof result.databases != "undefined" || notMaster(result)) { break } result = admin.runCommand({"listDatabases": 1}) } if (notMaster(result)) { continue } if (typeof result.databases == "undefined") { print("Could not list databases. Command result:") print(JSON.stringify(result)) quit(12) } var dbs = result.databases for (var j = 0; j != dbs.length; j++) { var db = dbs[j] switch (db.name) { case "admin": case "local": case "config": break default: mongo.getDB(db.name).dropDatabase() } } } function notMaster(result) { return typeof result.errmsg != "undefined" && (result.errmsg.indexOf("not master") >= 0 || result.errmsg.indexOf("no master found")) } // vim:ts=4:sw=4:et mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/mongojs/init.js000066400000000000000000000106711276133741700256050ustar00rootroot00000000000000//var settings = {heartbeatSleep: 0.05, heartbeatTimeout: 0.5} var settings = {}; // We know the master of the first set (pri=1), but not of the second. var rs1cfg = {_id: "rs1", members: [{_id: 1, host: "127.0.0.1:40011", priority: 1, tags: {rs1: "a"}}, {_id: 2, host: "127.0.0.1:40012", priority: 0, tags: {rs1: "b"}}, {_id: 3, host: "127.0.0.1:40013", priority: 0, tags: {rs1: "c"}}], settings: settings} var rs2cfg = {_id: "rs2", members: [{_id: 1, host: "127.0.0.1:40021", priority: 1, tags: {rs2: "a"}}, {_id: 2, host: "127.0.0.1:40022", priority: 1, tags: {rs2: "b"}}, {_id: 3, host: "127.0.0.1:40023", priority: 1, tags: {rs2: "c"}}], settings: settings} var rs3cfg = {_id: "rs3", members: [{_id: 1, host: "127.0.0.1:40031", priority: 1, tags: {rs3: "a"}}, {_id: 2, host: "127.0.0.1:40032", priority: 1, tags: {rs3: "b"}}, {_id: 3, host: "127.0.0.1:40033", priority: 1, tags: {rs3: "c"}}], settings: settings} for (var i = 0; i != 60; i++) { try { db1 = new Mongo("127.0.0.1:40001").getDB("admin") db2 = new Mongo("127.0.0.1:40002").getDB("admin") rs1a = new Mongo("127.0.0.1:40011").getDB("admin") rs2a = new Mongo("127.0.0.1:40021").getDB("admin") rs3a = new Mongo("127.0.0.1:40031").getDB("admin") break } catch(err) { print("Can't connect yet...") } sleep(1000) } function hasSSL() { return Boolean(db1.serverBuildInfo().OpenSSLVersion) } rs1a.runCommand({replSetInitiate: rs1cfg}) rs2a.runCommand({replSetInitiate: rs2cfg}) rs3a.runCommand({replSetInitiate: rs3cfg}) function configShards() { cfg1 = new Mongo("127.0.0.1:40201").getDB("admin") cfg1.runCommand({addshard: "127.0.0.1:40001"}) cfg1.runCommand({addshard: "rs1/127.0.0.1:40011"}) cfg2 = new Mongo("127.0.0.1:40202").getDB("admin") cfg2.runCommand({addshard: "rs2/127.0.0.1:40021"}) cfg3 = new Mongo("127.0.0.1:40203").getDB("admin") cfg3.runCommand({addshard: "rs3/127.0.0.1:40031"}) } function configAuth() { var addrs = ["127.0.0.1:40002", "127.0.0.1:40203", "127.0.0.1:40031"] if (hasSSL()) { addrs.push("127.0.0.1:40003") } for (var i in addrs) { print("Configuring auth for", addrs[i]) var db = new Mongo(addrs[i]).getDB("admin") var v = db.serverBuildInfo().versionArray var timedOut = false if (v < [2, 5]) { db.addUser("root", "rapadura") } else { try { db.createUser({user: "root", pwd: "rapadura", roles: ["root"]}) } catch (err) { // 3.2 consistently fails replication of creds on 40031 (config server) print("createUser command returned an error: " + err) if (String(err).indexOf("timed out") >= 0) { timedOut = true; } } } for (var i = 0; i < 60; i++) { var ok = db.auth("root", "rapadura") if (ok || !timedOut) { break } sleep(1000); } if (v >= [2, 6]) { db.createUser({user: "reader", pwd: "rapadura", roles: ["readAnyDatabase"]}) } else if (v >= [2, 4]) { db.addUser({user: "reader", pwd: "rapadura", roles: ["readAnyDatabase"]}) } else { db.addUser("reader", "rapadura", true) } } } function countHealthy(rs) { var status = rs.runCommand({replSetGetStatus: 1}) var count = 0 var primary = 0 if (typeof status.members != "undefined") { for (var i = 0; i != status.members.length; i++) { var m = status.members[i] if (m.health == 1 && (m.state == 1 || m.state == 2)) { count += 1 if (m.state == 1) { primary = 1 } } } } if (primary == 0) { count = 0 } return count } var totalRSMembers = rs1cfg.members.length + rs2cfg.members.length + rs3cfg.members.length for (var i = 0; i != 60; i++) { var count = countHealthy(rs1a) + countHealthy(rs2a) + countHealthy(rs3a) print("Replica sets have", count, "healthy nodes.") if (count == totalRSMembers) { configShards() configAuth() quit(0) } sleep(1000) } print("Replica sets didn't sync up properly.") quit(12) // vim:ts=4:sw=4:et mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/mongojs/wait.js000066400000000000000000000042441276133741700256050ustar00rootroot00000000000000// We know the master of the first set (pri=1), but not of the second. var settings = {} var rs1cfg = {_id: "rs1", members: [{_id: 1, host: "127.0.0.1:40011", priority: 1}, {_id: 2, host: "127.0.0.1:40012", priority: 0}, {_id: 3, host: "127.0.0.1:40013", priority: 0}]} var rs2cfg = {_id: "rs2", members: [{_id: 1, host: "127.0.0.1:40021", priority: 1}, {_id: 2, host: "127.0.0.1:40022", priority: 1}, {_id: 3, host: "127.0.0.1:40023", priority: 0}]} var rs3cfg = {_id: "rs3", members: [{_id: 1, host: "127.0.0.1:40031", priority: 1}, {_id: 2, host: "127.0.0.1:40032", priority: 1}, {_id: 3, host: "127.0.0.1:40033", priority: 1}], settings: settings} for (var i = 0; i != 60; i++) { try { rs1a = new Mongo("127.0.0.1:40011").getDB("admin") rs2a = new Mongo("127.0.0.1:40021").getDB("admin") rs3a = new Mongo("127.0.0.1:40031").getDB("admin") rs3a.auth("root", "rapadura") db1 = new Mongo("127.0.0.1:40001").getDB("admin") db2 = new Mongo("127.0.0.1:40002").getDB("admin") break } catch(err) { print("Can't connect yet...") } sleep(1000) } function countHealthy(rs) { var status = rs.runCommand({replSetGetStatus: 1}) var count = 0 var primary = 0 if (typeof status.members != "undefined") { for (var i = 0; i != status.members.length; i++) { var m = status.members[i] if (m.health == 1 && (m.state == 1 || m.state == 2)) { count += 1 if (m.state == 1) { primary = 1 } } } } if (primary == 0) { count = 0 } return count } var totalRSMembers = rs1cfg.members.length + rs2cfg.members.length + rs3cfg.members.length for (var i = 0; i != 90; i++) { var count = countHealthy(rs1a) + countHealthy(rs2a) + countHealthy(rs3a) print("Replica sets have", count, "healthy nodes.") if (count == totalRSMembers) { quit(0) } sleep(1000) } print("Replica sets didn't sync up properly.") quit(12) // vim:ts=4:sw=4:et mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/harness/setup.sh000077500000000000000000000044551276133741700243320ustar00rootroot00000000000000#!/bin/sh -e LINE="---------------" start() { if [ -d _harness ]; then echo "Daemon setup already in place, stop it first." exit 1 fi mkdir -p _harness cd _harness cp -a ../harness/daemons . cp -a ../harness/certs . echo keyfile > certs/keyfile chmod 600 certs/keyfile if ! mongod --help | grep -q -- --ssl; then rm -rf daemons/db3 fi COUNT=$(ls daemons | wc -l) echo "Running daemons..." svscan daemons & SVSCANPID=$! echo $SVSCANPID > svscan.pid if ! kill -0 $SVSCANPID; then echo "Cannot execute svscan." exit 1 fi echo "Starting $COUNT processes..." for i in $(seq 30); do UP=$(svstat daemons/* | grep ' up ' | grep -v ' [0-3] seconds' | wc -l) echo "$UP processes up..." if [ x$COUNT = x$UP ]; then echo "Running setup.js with mongo..." mongo --nodb ../harness/mongojs/init.js exit 0 fi sleep 1 done echo "Failed to start processes. svstat _harness/daemons/* output:" echo $LINE svstat daemons/* echo $LINE for DAEMON in daemons/*; do if $(svstat $DAEMON | grep ' up ' | grep ' [0-3] seconds' > /dev/null); then echo "Logs for _harness/$DAEMON:" echo $LINE cat $DAEMON/log/log.txt echo $LINE fi done exit 1 } stop() { if [ -d _harness ]; then cd _harness if [ -f svscan.pid ]; then kill -9 $(cat svscan.pid) 2> /dev/null || true svc -dx daemons/* daemons/*/log > /dev/null 2>&1 || true COUNT=$(ls daemons | wc -l) echo "Shutting down $COUNT processes..." while true; do DOWN=$(svstat daemons/* | grep 'supervise not running' | wc -l) echo "$DOWN processes down..." if [ x$DOWN = x$COUNT ]; then break fi sleep 1 done rm svscan.pid echo "Done." fi cd .. rm -rf _harness fi } if [ ! -f suite_test.go ]; then echo "This script must be run from within the source directory." exit 1 fi case "$1" in start) start $2 ;; stop) stop $2 ;; esac # vim:ts=4:sw=4:et mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/internal/000077500000000000000000000000001276133741700227745ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/internal/json/000077500000000000000000000000001276133741700237455ustar00rootroot00000000000000mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/internal/json/LICENSE000066400000000000000000000027071276133741700247600ustar00rootroot00000000000000Copyright (c) 2012 The Go Authors. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of Google Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/internal/json/bench_test.go000066400000000000000000000110231276133741700264070ustar00rootroot00000000000000// Copyright 2011 The Go Authors. All rights reserved. // Use of this source code is governed by a BSD-style // license that can be found in the LICENSE file. // Large data benchmark. // The JSON data is a summary of agl's changes in the // go, webkit, and chromium open source projects. // We benchmark converting between the JSON form // and in-memory data structures. package json import ( "bytes" "compress/gzip" "io/ioutil" "os" "strings" "testing" ) type codeResponse struct { Tree *codeNode `json:"tree"` Username string `json:"username"` } type codeNode struct { Name string `json:"name"` Kids []*codeNode `json:"kids"` CLWeight float64 `json:"cl_weight"` Touches int `json:"touches"` MinT int64 `json:"min_t"` MaxT int64 `json:"max_t"` MeanT int64 `json:"mean_t"` } var codeJSON []byte var codeStruct codeResponse func codeInit() { f, err := os.Open("testdata/code.json.gz") if err != nil { panic(err) } defer f.Close() gz, err := gzip.NewReader(f) if err != nil { panic(err) } data, err := ioutil.ReadAll(gz) if err != nil { panic(err) } codeJSON = data if err := Unmarshal(codeJSON, &codeStruct); err != nil { panic("unmarshal code.json: " + err.Error()) } if data, err = Marshal(&codeStruct); err != nil { panic("marshal code.json: " + err.Error()) } if !bytes.Equal(data, codeJSON) { println("different lengths", len(data), len(codeJSON)) for i := 0; i < len(data) && i < len(codeJSON); i++ { if data[i] != codeJSON[i] { println("re-marshal: changed at byte", i) println("orig: ", string(codeJSON[i-10:i+10])) println("new: ", string(data[i-10:i+10])) break } } panic("re-marshal code.json: different result") } } func BenchmarkCodeEncoder(b *testing.B) { if codeJSON == nil { b.StopTimer() codeInit() b.StartTimer() } enc := NewEncoder(ioutil.Discard) for i := 0; i < b.N; i++ { if err := enc.Encode(&codeStruct); err != nil { b.Fatal("Encode:", err) } } b.SetBytes(int64(len(codeJSON))) } func BenchmarkCodeMarshal(b *testing.B) { if codeJSON == nil { b.StopTimer() codeInit() b.StartTimer() } for i := 0; i < b.N; i++ { if _, err := Marshal(&codeStruct); err != nil { b.Fatal("Marshal:", err) } } b.SetBytes(int64(len(codeJSON))) } func BenchmarkCodeDecoder(b *testing.B) { if codeJSON == nil { b.StopTimer() codeInit() b.StartTimer() } var buf bytes.Buffer dec := NewDecoder(&buf) var r codeResponse for i := 0; i < b.N; i++ { buf.Write(codeJSON) // hide EOF buf.WriteByte('\n') buf.WriteByte('\n') buf.WriteByte('\n') if err := dec.Decode(&r); err != nil { b.Fatal("Decode:", err) } } b.SetBytes(int64(len(codeJSON))) } func BenchmarkDecoderStream(b *testing.B) { b.StopTimer() var buf bytes.Buffer dec := NewDecoder(&buf) buf.WriteString(`"` + strings.Repeat("x", 1000000) + `"` + "\n\n\n") var x interface{} if err := dec.Decode(&x); err != nil { b.Fatal("Decode:", err) } ones := strings.Repeat(" 1\n", 300000) + "\n\n\n" b.StartTimer() for i := 0; i < b.N; i++ { if i%300000 == 0 { buf.WriteString(ones) } x = nil if err := dec.Decode(&x); err != nil || x != 1.0 { b.Fatalf("Decode: %v after %d", err, i) } } } func BenchmarkCodeUnmarshal(b *testing.B) { if codeJSON == nil { b.StopTimer() codeInit() b.StartTimer() } for i := 0; i < b.N; i++ { var r codeResponse if err := Unmarshal(codeJSON, &r); err != nil { b.Fatal("Unmarshal:", err) } } b.SetBytes(int64(len(codeJSON))) } func BenchmarkCodeUnmarshalReuse(b *testing.B) { if codeJSON == nil { b.StopTimer() codeInit() b.StartTimer() } var r codeResponse for i := 0; i < b.N; i++ { if err := Unmarshal(codeJSON, &r); err != nil { b.Fatal("Unmarshal:", err) } } } func BenchmarkUnmarshalString(b *testing.B) { data := []byte(`"hello, world"`) var s string for i := 0; i < b.N; i++ { if err := Unmarshal(data, &s); err != nil { b.Fatal("Unmarshal:", err) } } } func BenchmarkUnmarshalFloat64(b *testing.B) { var f float64 data := []byte(`3.14`) for i := 0; i < b.N; i++ { if err := Unmarshal(data, &f); err != nil { b.Fatal("Unmarshal:", err) } } } func BenchmarkUnmarshalInt64(b *testing.B) { var x int64 data := []byte(`3`) for i := 0; i < b.N; i++ { if err := Unmarshal(data, &x); err != nil { b.Fatal("Unmarshal:", err) } } } func BenchmarkIssue10335(b *testing.B) { b.ReportAllocs() var s struct{} j := []byte(`{"a":{ }}`) for n := 0; n < b.N; n++ { if err := Unmarshal(j, &s); err != nil { b.Fatal(err) } } } mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/internal/json/decode.go000066400000000000000000001173001276133741700255210ustar00rootroot00000000000000// Copyright 2010 The Go Authors. All rights reserved. // Use of this source code is governed by a BSD-style // license that can be found in the LICENSE file. // Represents JSON data structure using native Go types: booleans, floats, // strings, arrays, and maps. package json import ( "bytes" "encoding" "encoding/base64" "errors" "fmt" "reflect" "runtime" "strconv" "unicode" "unicode/utf16" "unicode/utf8" ) // Unmarshal parses the JSON-encoded data and stores the result // in the value pointed to by v. // // Unmarshal uses the inverse of the encodings that // Marshal uses, allocating maps, slices, and pointers as necessary, // with the following additional rules: // // To unmarshal JSON into a pointer, Unmarshal first handles the case of // the JSON being the JSON literal null. In that case, Unmarshal sets // the pointer to nil. Otherwise, Unmarshal unmarshals the JSON into // the value pointed at by the pointer. If the pointer is nil, Unmarshal // allocates a new value for it to point to. // // To unmarshal JSON into a struct, Unmarshal matches incoming object // keys to the keys used by Marshal (either the struct field name or its tag), // preferring an exact match but also accepting a case-insensitive match. // Unmarshal will only set exported fields of the struct. // // To unmarshal JSON into an interface value, // Unmarshal stores one of these in the interface value: // // bool, for JSON booleans // float64, for JSON numbers // string, for JSON strings // []interface{}, for JSON arrays // map[string]interface{}, for JSON objects // nil for JSON null // // To unmarshal a JSON array into a slice, Unmarshal resets the slice length // to zero and then appends each element to the slice. // As a special case, to unmarshal an empty JSON array into a slice, // Unmarshal replaces the slice with a new empty slice. // // To unmarshal a JSON array into a Go array, Unmarshal decodes // JSON array elements into corresponding Go array elements. // If the Go array is smaller than the JSON array, // the additional JSON array elements are discarded. // If the JSON array is smaller than the Go array, // the additional Go array elements are set to zero values. // // To unmarshal a JSON object into a map, Unmarshal first establishes a map to // use, If the map is nil, Unmarshal allocates a new map. Otherwise Unmarshal // reuses the existing map, keeping existing entries. Unmarshal then stores key- // value pairs from the JSON object into the map. The map's key type must // either be a string or implement encoding.TextUnmarshaler. // // If a JSON value is not appropriate for a given target type, // or if a JSON number overflows the target type, Unmarshal // skips that field and completes the unmarshaling as best it can. // If no more serious errors are encountered, Unmarshal returns // an UnmarshalTypeError describing the earliest such error. // // The JSON null value unmarshals into an interface, map, pointer, or slice // by setting that Go value to nil. Because null is often used in JSON to mean // ``not present,'' unmarshaling a JSON null into any other Go type has no effect // on the value and produces no error. // // When unmarshaling quoted strings, invalid UTF-8 or // invalid UTF-16 surrogate pairs are not treated as an error. // Instead, they are replaced by the Unicode replacement // character U+FFFD. // func Unmarshal(data []byte, v interface{}) error { // Check for well-formedness. // Avoids filling out half a data structure // before discovering a JSON syntax error. var d decodeState err := checkValid(data, &d.scan) if err != nil { return err } d.init(data) return d.unmarshal(v) } // Unmarshaler is the interface implemented by types // that can unmarshal a JSON description of themselves. // The input can be assumed to be a valid encoding of // a JSON value. UnmarshalJSON must copy the JSON data // if it wishes to retain the data after returning. type Unmarshaler interface { UnmarshalJSON([]byte) error } // An UnmarshalTypeError describes a JSON value that was // not appropriate for a value of a specific Go type. type UnmarshalTypeError struct { Value string // description of JSON value - "bool", "array", "number -5" Type reflect.Type // type of Go value it could not be assigned to Offset int64 // error occurred after reading Offset bytes } func (e *UnmarshalTypeError) Error() string { return "json: cannot unmarshal " + e.Value + " into Go value of type " + e.Type.String() } // An UnmarshalFieldError describes a JSON object key that // led to an unexported (and therefore unwritable) struct field. // (No longer used; kept for compatibility.) type UnmarshalFieldError struct { Key string Type reflect.Type Field reflect.StructField } func (e *UnmarshalFieldError) Error() string { return "json: cannot unmarshal object key " + strconv.Quote(e.Key) + " into unexported field " + e.Field.Name + " of type " + e.Type.String() } // An InvalidUnmarshalError describes an invalid argument passed to Unmarshal. // (The argument to Unmarshal must be a non-nil pointer.) type InvalidUnmarshalError struct { Type reflect.Type } func (e *InvalidUnmarshalError) Error() string { if e.Type == nil { return "json: Unmarshal(nil)" } if e.Type.Kind() != reflect.Ptr { return "json: Unmarshal(non-pointer " + e.Type.String() + ")" } return "json: Unmarshal(nil " + e.Type.String() + ")" } func (d *decodeState) unmarshal(v interface{}) (err error) { defer func() { if r := recover(); r != nil { if _, ok := r.(runtime.Error); ok { panic(r) } err = r.(error) } }() rv := reflect.ValueOf(v) if rv.Kind() != reflect.Ptr || rv.IsNil() { return &InvalidUnmarshalError{reflect.TypeOf(v)} } d.scan.reset() // We decode rv not rv.Elem because the Unmarshaler interface // test must be applied at the top level of the value. d.value(rv) return d.savedError } // A Number represents a JSON number literal. type Number string // String returns the literal text of the number. func (n Number) String() string { return string(n) } // Float64 returns the number as a float64. func (n Number) Float64() (float64, error) { return strconv.ParseFloat(string(n), 64) } // Int64 returns the number as an int64. func (n Number) Int64() (int64, error) { return strconv.ParseInt(string(n), 10, 64) } // isValidNumber reports whether s is a valid JSON number literal. func isValidNumber(s string) bool { // This function implements the JSON numbers grammar. // See https://tools.ietf.org/html/rfc7159#section-6 // and http://json.org/number.gif if s == "" { return false } // Optional - if s[0] == '-' { s = s[1:] if s == "" { return false } } // Digits switch { default: return false case s[0] == '0': s = s[1:] case '1' <= s[0] && s[0] <= '9': s = s[1:] for len(s) > 0 && '0' <= s[0] && s[0] <= '9' { s = s[1:] } } // . followed by 1 or more digits. if len(s) >= 2 && s[0] == '.' && '0' <= s[1] && s[1] <= '9' { s = s[2:] for len(s) > 0 && '0' <= s[0] && s[0] <= '9' { s = s[1:] } } // e or E followed by an optional - or + and // 1 or more digits. if len(s) >= 2 && (s[0] == 'e' || s[0] == 'E') { s = s[1:] if s[0] == '+' || s[0] == '-' { s = s[1:] if s == "" { return false } } for len(s) > 0 && '0' <= s[0] && s[0] <= '9' { s = s[1:] } } // Make sure we are at the end. return s == "" } // decodeState represents the state while decoding a JSON value. type decodeState struct { data []byte off int // read offset in data scan scanner nextscan scanner // for calls to nextValue savedError error useNumber bool ext Extension } // errPhase is used for errors that should not happen unless // there is a bug in the JSON decoder or something is editing // the data slice while the decoder executes. var errPhase = errors.New("JSON decoder out of sync - data changing underfoot?") func (d *decodeState) init(data []byte) *decodeState { d.data = data d.off = 0 d.savedError = nil return d } // error aborts the decoding by panicking with err. func (d *decodeState) error(err error) { panic(err) } // saveError saves the first err it is called with, // for reporting at the end of the unmarshal. func (d *decodeState) saveError(err error) { if d.savedError == nil { d.savedError = err } } // next cuts off and returns the next full JSON value in d.data[d.off:]. // The next value is known to be an object or array, not a literal. func (d *decodeState) next() []byte { c := d.data[d.off] item, rest, err := nextValue(d.data[d.off:], &d.nextscan) if err != nil { d.error(err) } d.off = len(d.data) - len(rest) // Our scanner has seen the opening brace/bracket // and thinks we're still in the middle of the object. // invent a closing brace/bracket to get it out. if c == '{' { d.scan.step(&d.scan, '}') } else if c == '[' { d.scan.step(&d.scan, ']') } else { // Was inside a function name. Get out of it. d.scan.step(&d.scan, '(') d.scan.step(&d.scan, ')') } return item } // scanWhile processes bytes in d.data[d.off:] until it // receives a scan code not equal to op. // It updates d.off and returns the new scan code. func (d *decodeState) scanWhile(op int) int { var newOp int for { if d.off >= len(d.data) { newOp = d.scan.eof() d.off = len(d.data) + 1 // mark processed EOF with len+1 } else { c := d.data[d.off] d.off++ newOp = d.scan.step(&d.scan, c) } if newOp != op { break } } return newOp } // value decodes a JSON value from d.data[d.off:] into the value. // it updates d.off to point past the decoded value. func (d *decodeState) value(v reflect.Value) { if !v.IsValid() { _, rest, err := nextValue(d.data[d.off:], &d.nextscan) if err != nil { d.error(err) } d.off = len(d.data) - len(rest) // d.scan thinks we're still at the beginning of the item. // Feed in an empty string - the shortest, simplest value - // so that it knows we got to the end of the value. if d.scan.redo { // rewind. d.scan.redo = false d.scan.step = stateBeginValue } d.scan.step(&d.scan, '"') d.scan.step(&d.scan, '"') n := len(d.scan.parseState) if n > 0 && d.scan.parseState[n-1] == parseObjectKey { // d.scan thinks we just read an object key; finish the object d.scan.step(&d.scan, ':') d.scan.step(&d.scan, '"') d.scan.step(&d.scan, '"') d.scan.step(&d.scan, '}') } return } switch op := d.scanWhile(scanSkipSpace); op { default: d.error(errPhase) case scanBeginArray: d.array(v) case scanBeginObject: d.object(v) case scanBeginLiteral: d.literal(v) case scanBeginName: d.name(v) } } type unquotedValue struct{} // valueQuoted is like value but decodes a // quoted string literal or literal null into an interface value. // If it finds anything other than a quoted string literal or null, // valueQuoted returns unquotedValue{}. func (d *decodeState) valueQuoted() interface{} { switch op := d.scanWhile(scanSkipSpace); op { default: d.error(errPhase) case scanBeginArray: d.array(reflect.Value{}) case scanBeginObject: d.object(reflect.Value{}) case scanBeginName: switch v := d.nameInterface().(type) { case nil, string: return v } case scanBeginLiteral: switch v := d.literalInterface().(type) { case nil, string: return v } } return unquotedValue{} } // indirect walks down v allocating pointers as needed, // until it gets to a non-pointer. // if it encounters an Unmarshaler, indirect stops and returns that. // if decodingNull is true, indirect stops at the last pointer so it can be set to nil. func (d *decodeState) indirect(v reflect.Value, decodingNull bool) (Unmarshaler, encoding.TextUnmarshaler, reflect.Value) { // If v is a named type and is addressable, // start with its address, so that if the type has pointer methods, // we find them. if v.Kind() != reflect.Ptr && v.Type().Name() != "" && v.CanAddr() { v = v.Addr() } for { // Load value from interface, but only if the result will be // usefully addressable. if v.Kind() == reflect.Interface && !v.IsNil() { e := v.Elem() if e.Kind() == reflect.Ptr && !e.IsNil() && (!decodingNull || e.Elem().Kind() == reflect.Ptr) { v = e continue } } if v.Kind() != reflect.Ptr { break } if v.Elem().Kind() != reflect.Ptr && decodingNull && v.CanSet() { break } if v.IsNil() { v.Set(reflect.New(v.Type().Elem())) } if v.Type().NumMethod() > 0 { if u, ok := v.Interface().(Unmarshaler); ok { return u, nil, v } if u, ok := v.Interface().(encoding.TextUnmarshaler); ok { return nil, u, v } } v = v.Elem() } return nil, nil, v } // array consumes an array from d.data[d.off-1:], decoding into the value v. // the first byte of the array ('[') has been read already. func (d *decodeState) array(v reflect.Value) { // Check for unmarshaler. u, ut, pv := d.indirect(v, false) if u != nil { d.off-- err := u.UnmarshalJSON(d.next()) if err != nil { d.error(err) } return } if ut != nil { d.saveError(&UnmarshalTypeError{"array", v.Type(), int64(d.off)}) d.off-- d.next() return } v = pv // Check type of target. switch v.Kind() { case reflect.Interface: if v.NumMethod() == 0 { // Decoding into nil interface? Switch to non-reflect code. v.Set(reflect.ValueOf(d.arrayInterface())) return } // Otherwise it's invalid. fallthrough default: d.saveError(&UnmarshalTypeError{"array", v.Type(), int64(d.off)}) d.off-- d.next() return case reflect.Array: case reflect.Slice: break } i := 0 for { // Look ahead for ] - can only happen on first iteration. op := d.scanWhile(scanSkipSpace) if op == scanEndArray { break } // Back up so d.value can have the byte we just read. d.off-- d.scan.undo(op) // Get element of array, growing if necessary. if v.Kind() == reflect.Slice { // Grow slice if necessary if i >= v.Cap() { newcap := v.Cap() + v.Cap()/2 if newcap < 4 { newcap = 4 } newv := reflect.MakeSlice(v.Type(), v.Len(), newcap) reflect.Copy(newv, v) v.Set(newv) } if i >= v.Len() { v.SetLen(i + 1) } } if i < v.Len() { // Decode into element. d.value(v.Index(i)) } else { // Ran out of fixed array: skip. d.value(reflect.Value{}) } i++ // Next token must be , or ]. op = d.scanWhile(scanSkipSpace) if op == scanEndArray { break } if op != scanArrayValue { d.error(errPhase) } } if i < v.Len() { if v.Kind() == reflect.Array { // Array. Zero the rest. z := reflect.Zero(v.Type().Elem()) for ; i < v.Len(); i++ { v.Index(i).Set(z) } } else { v.SetLen(i) } } if i == 0 && v.Kind() == reflect.Slice { v.Set(reflect.MakeSlice(v.Type(), 0, 0)) } } var nullLiteral = []byte("null") var textUnmarshalerType = reflect.TypeOf(new(encoding.TextUnmarshaler)).Elem() // object consumes an object from d.data[d.off-1:], decoding into the value v. // the first byte ('{') of the object has been read already. func (d *decodeState) object(v reflect.Value) { // Check for unmarshaler. u, ut, pv := d.indirect(v, false) if d.storeKeyed(pv) { return } if u != nil { d.off-- err := u.UnmarshalJSON(d.next()) if err != nil { d.error(err) } return } if ut != nil { d.saveError(&UnmarshalTypeError{"object", v.Type(), int64(d.off)}) d.off-- d.next() // skip over { } in input return } v = pv // Decoding into nil interface? Switch to non-reflect code. if v.Kind() == reflect.Interface && v.NumMethod() == 0 { v.Set(reflect.ValueOf(d.objectInterface())) return } // Check type of target: // struct or // map[string]T or map[encoding.TextUnmarshaler]T switch v.Kind() { case reflect.Map: // Map key must either have string kind or be an encoding.TextUnmarshaler. t := v.Type() if t.Key().Kind() != reflect.String && !reflect.PtrTo(t.Key()).Implements(textUnmarshalerType) { d.saveError(&UnmarshalTypeError{"object", v.Type(), int64(d.off)}) d.off-- d.next() // skip over { } in input return } if v.IsNil() { v.Set(reflect.MakeMap(t)) } case reflect.Struct: default: d.saveError(&UnmarshalTypeError{"object", v.Type(), int64(d.off)}) d.off-- d.next() // skip over { } in input return } var mapElem reflect.Value empty := true for { // Read opening " of string key or closing }. op := d.scanWhile(scanSkipSpace) if op == scanEndObject { if !empty && !d.ext.trailingCommas { d.syntaxError("beginning of object key string") } break } empty = false if op == scanBeginName { if !d.ext.unquotedKeys { d.syntaxError("beginning of object key string") } } else if op != scanBeginLiteral { d.error(errPhase) } unquotedKey := op == scanBeginName // Read key. start := d.off - 1 op = d.scanWhile(scanContinue) item := d.data[start : d.off-1] var key []byte if unquotedKey { key = item // TODO Fix code below to quote item when necessary. } else { var ok bool key, ok = unquoteBytes(item) if !ok { d.error(errPhase) } } // Figure out field corresponding to key. var subv reflect.Value destring := false // whether the value is wrapped in a string to be decoded first if v.Kind() == reflect.Map { elemType := v.Type().Elem() if !mapElem.IsValid() { mapElem = reflect.New(elemType).Elem() } else { mapElem.Set(reflect.Zero(elemType)) } subv = mapElem } else { var f *field fields := cachedTypeFields(v.Type()) for i := range fields { ff := &fields[i] if bytes.Equal(ff.nameBytes, key) { f = ff break } if f == nil && ff.equalFold(ff.nameBytes, key) { f = ff } } if f != nil { subv = v destring = f.quoted for _, i := range f.index { if subv.Kind() == reflect.Ptr { if subv.IsNil() { subv.Set(reflect.New(subv.Type().Elem())) } subv = subv.Elem() } subv = subv.Field(i) } } } // Read : before value. if op == scanSkipSpace { op = d.scanWhile(scanSkipSpace) } if op != scanObjectKey { d.error(errPhase) } // Read value. if destring { switch qv := d.valueQuoted().(type) { case nil: d.literalStore(nullLiteral, subv, false) case string: d.literalStore([]byte(qv), subv, true) default: d.saveError(fmt.Errorf("json: invalid use of ,string struct tag, trying to unmarshal unquoted value into %v", subv.Type())) } } else { d.value(subv) } // Write value back to map; // if using struct, subv points into struct already. if v.Kind() == reflect.Map { kt := v.Type().Key() var kv reflect.Value switch { case kt.Kind() == reflect.String: kv = reflect.ValueOf(key).Convert(v.Type().Key()) case reflect.PtrTo(kt).Implements(textUnmarshalerType): kv = reflect.New(v.Type().Key()) d.literalStore(item, kv, true) kv = kv.Elem() default: panic("json: Unexpected key type") // should never occur } v.SetMapIndex(kv, subv) } // Next token must be , or }. op = d.scanWhile(scanSkipSpace) if op == scanEndObject { break } if op != scanObjectValue { d.error(errPhase) } } } // isNull returns whether there's a null literal at the provided offset. func (d *decodeState) isNull(off int) bool { if off+4 >= len(d.data) || d.data[off] != 'n' || d.data[off+1] != 'u' || d.data[off+2] != 'l' || d.data[off+3] != 'l' { return false } d.nextscan.reset() for i, c := range d.data[off:] { if i > 4 { return false } switch d.nextscan.step(&d.nextscan, c) { case scanContinue, scanBeginName: continue } break } return true } // name consumes a const or function from d.data[d.off-1:], decoding into the value v. // the first byte of the function name has been read already. func (d *decodeState) name(v reflect.Value) { if d.isNull(d.off-1) { d.literal(v) return } // Check for unmarshaler. u, ut, pv := d.indirect(v, false) if d.storeKeyed(pv) { return } if u != nil { d.off-- err := u.UnmarshalJSON(d.next()) if err != nil { d.error(err) } return } if ut != nil { d.saveError(&UnmarshalTypeError{"object", v.Type(), int64(d.off)}) d.off-- d.next() // skip over function in input return } v = pv // Decoding into nil interface? Switch to non-reflect code. if v.Kind() == reflect.Interface && v.NumMethod() == 0 { out := d.nameInterface() if out == nil { v.Set(reflect.Zero(v.Type())) } else { v.Set(reflect.ValueOf(out)) } return } nameStart := d.off - 1 op := d.scanWhile(scanContinue) name := d.data[nameStart : d.off-1] if op != scanParam { // Back up so the byte just read is consumed next. d.off-- d.scan.undo(op) if l, ok := d.convertLiteral(name); ok { d.storeValue(v, l) return } d.error(&SyntaxError{fmt.Sprintf("json: unknown constant %q", name), int64(d.off)}) } funcName := string(name) funcData := d.ext.funcs[funcName] if funcData.key == "" { d.error(fmt.Errorf("json: unknown function %q", funcName)) } // Check type of target: // struct or // map[string]T or map[encoding.TextUnmarshaler]T switch v.Kind() { case reflect.Map: // Map key must either have string kind or be an encoding.TextUnmarshaler. t := v.Type() if t.Key().Kind() != reflect.String && !reflect.PtrTo(t.Key()).Implements(textUnmarshalerType) { d.saveError(&UnmarshalTypeError{"object", v.Type(), int64(d.off)}) d.off-- d.next() // skip over { } in input return } if v.IsNil() { v.Set(reflect.MakeMap(t)) } case reflect.Struct: default: d.saveError(&UnmarshalTypeError{"object", v.Type(), int64(d.off)}) d.off-- d.next() // skip over { } in input return } // TODO Fix case of func field as map. //topv := v // Figure out field corresponding to function. key := []byte(funcData.key) if v.Kind() == reflect.Map { elemType := v.Type().Elem() v = reflect.New(elemType).Elem() } else { var f *field fields := cachedTypeFields(v.Type()) for i := range fields { ff := &fields[i] if bytes.Equal(ff.nameBytes, key) { f = ff break } if f == nil && ff.equalFold(ff.nameBytes, key) { f = ff } } if f != nil { for _, i := range f.index { if v.Kind() == reflect.Ptr { if v.IsNil() { v.Set(reflect.New(v.Type().Elem())) } v = v.Elem() } v = v.Field(i) } if v.Kind() == reflect.Ptr { if v.IsNil() { v.Set(reflect.New(v.Type().Elem())) } v = v.Elem() } } } // Check for unmarshaler on func field itself. u, ut, pv = d.indirect(v, false) if u != nil { d.off = nameStart err := u.UnmarshalJSON(d.next()) if err != nil { d.error(err) } return } var mapElem reflect.Value // Parse function arguments. for i := 0; ; i++ { // closing ) - can only happen on first iteration. op := d.scanWhile(scanSkipSpace) if op == scanEndParams { break } // Back up so d.value can have the byte we just read. d.off-- d.scan.undo(op) if i >= len(funcData.args) { d.error(fmt.Errorf("json: too many arguments for function %s", funcName)) } key := []byte(funcData.args[i]) // Figure out field corresponding to key. var subv reflect.Value destring := false // whether the value is wrapped in a string to be decoded first if v.Kind() == reflect.Map { elemType := v.Type().Elem() if !mapElem.IsValid() { mapElem = reflect.New(elemType).Elem() } else { mapElem.Set(reflect.Zero(elemType)) } subv = mapElem } else { var f *field fields := cachedTypeFields(v.Type()) for i := range fields { ff := &fields[i] if bytes.Equal(ff.nameBytes, key) { f = ff break } if f == nil && ff.equalFold(ff.nameBytes, key) { f = ff } } if f != nil { subv = v destring = f.quoted for _, i := range f.index { if subv.Kind() == reflect.Ptr { if subv.IsNil() { subv.Set(reflect.New(subv.Type().Elem())) } subv = subv.Elem() } subv = subv.Field(i) } } } // Read value. if destring { switch qv := d.valueQuoted().(type) { case nil: d.literalStore(nullLiteral, subv, false) case string: d.literalStore([]byte(qv), subv, true) default: d.saveError(fmt.Errorf("json: invalid use of ,string struct tag, trying to unmarshal unquoted value into %v", subv.Type())) } } else { d.value(subv) } // Write value back to map; // if using struct, subv points into struct already. if v.Kind() == reflect.Map { kt := v.Type().Key() var kv reflect.Value switch { case kt.Kind() == reflect.String: kv = reflect.ValueOf(key).Convert(v.Type().Key()) case reflect.PtrTo(kt).Implements(textUnmarshalerType): kv = reflect.New(v.Type().Key()) d.literalStore(key, kv, true) kv = kv.Elem() default: panic("json: Unexpected key type") // should never occur } v.SetMapIndex(kv, subv) } // Next token must be , or ). op = d.scanWhile(scanSkipSpace) if op == scanEndParams { break } if op != scanParam { d.error(errPhase) } } } // keyed attempts to decode an object or function using a keyed doc extension, // and returns the value and true on success, or nil and false otherwise. func (d *decodeState) keyed() (interface{}, bool) { if len(d.ext.keyed) == 0 { return nil, false } unquote := false // Look-ahead first key to check for a keyed document extension. d.nextscan.reset() var start, end int for i, c := range d.data[d.off-1:] { switch op := d.nextscan.step(&d.nextscan, c); op { case scanSkipSpace, scanContinue, scanBeginObject: continue case scanBeginLiteral, scanBeginName: unquote = op == scanBeginLiteral start = i continue } end = i break } name := d.data[d.off-1+start : d.off-1+end] var key []byte var ok bool if unquote { key, ok = unquoteBytes(name) if !ok { d.error(errPhase) } } else { funcData, ok := d.ext.funcs[string(name)] if !ok { return nil, false } key = []byte(funcData.key) } decode, ok := d.ext.keyed[string(key)] if !ok { return nil, false } d.off-- out, err := decode(d.next()) if err != nil { d.error(err) } return out, true } func (d *decodeState) storeKeyed(v reflect.Value) bool { keyed, ok := d.keyed() if !ok { return false } d.storeValue(v, keyed) return true } var ( trueBytes = []byte("true") falseBytes = []byte("false") nullBytes = []byte("null") ) func (d *decodeState) storeValue(v reflect.Value, from interface{}) { switch from { case nil: d.literalStore(nullBytes, v, false) return case true: d.literalStore(trueBytes, v, false) return case false: d.literalStore(falseBytes, v, false) return } fromv := reflect.ValueOf(from) for fromv.Kind() == reflect.Ptr && !fromv.IsNil() { fromv = fromv.Elem() } fromt := fromv.Type() for v.Kind() == reflect.Ptr && !v.IsNil() { v = v.Elem() } vt := v.Type() if fromt.AssignableTo(vt) { v.Set(fromv) } else if fromt.ConvertibleTo(vt) { v.Set(fromv.Convert(vt)) } else { d.saveError(&UnmarshalTypeError{"object", v.Type(), int64(d.off)}) } } func (d *decodeState) convertLiteral(name []byte) (interface{}, bool) { if len(name) == 0 { return nil, false } switch name[0] { case 't': if bytes.Equal(name, trueBytes) { return true, true } case 'f': if bytes.Equal(name, falseBytes) { return false, true } case 'n': if bytes.Equal(name, nullBytes) { return nil, true } } if l, ok := d.ext.consts[string(name)]; ok { return l, true } return nil, false } // literal consumes a literal from d.data[d.off-1:], decoding into the value v. // The first byte of the literal has been read already // (that's how the caller knows it's a literal). func (d *decodeState) literal(v reflect.Value) { // All bytes inside literal return scanContinue op code. start := d.off - 1 op := d.scanWhile(scanContinue) // Scan read one byte too far; back up. d.off-- d.scan.undo(op) d.literalStore(d.data[start:d.off], v, false) } // convertNumber converts the number literal s to a float64 or a Number // depending on the setting of d.useNumber. func (d *decodeState) convertNumber(s string) (interface{}, error) { if d.useNumber { return Number(s), nil } f, err := strconv.ParseFloat(s, 64) if err != nil { return nil, &UnmarshalTypeError{"number " + s, reflect.TypeOf(0.0), int64(d.off)} } return f, nil } var numberType = reflect.TypeOf(Number("")) // literalStore decodes a literal stored in item into v. // // fromQuoted indicates whether this literal came from unwrapping a // string from the ",string" struct tag option. this is used only to // produce more helpful error messages. func (d *decodeState) literalStore(item []byte, v reflect.Value, fromQuoted bool) { // Check for unmarshaler. if len(item) == 0 { //Empty string given d.saveError(fmt.Errorf("json: invalid use of ,string struct tag, trying to unmarshal %q into %v", item, v.Type())) return } wantptr := item[0] == 'n' // null u, ut, pv := d.indirect(v, wantptr) if u != nil { err := u.UnmarshalJSON(item) if err != nil { d.error(err) } return } if ut != nil { if item[0] != '"' { if fromQuoted { d.saveError(fmt.Errorf("json: invalid use of ,string struct tag, trying to unmarshal %q into %v", item, v.Type())) } else { d.saveError(&UnmarshalTypeError{"string", v.Type(), int64(d.off)}) } return } s, ok := unquoteBytes(item) if !ok { if fromQuoted { d.error(fmt.Errorf("json: invalid use of ,string struct tag, trying to unmarshal %q into %v", item, v.Type())) } else { d.error(errPhase) } } err := ut.UnmarshalText(s) if err != nil { d.error(err) } return } v = pv switch c := item[0]; c { case 'n': // null switch v.Kind() { case reflect.Interface, reflect.Ptr, reflect.Map, reflect.Slice: v.Set(reflect.Zero(v.Type())) // otherwise, ignore null for primitives/string } case 't', 'f': // true, false value := c == 't' switch v.Kind() { default: if fromQuoted { d.saveError(fmt.Errorf("json: invalid use of ,string struct tag, trying to unmarshal %q into %v", item, v.Type())) } else { d.saveError(&UnmarshalTypeError{"bool", v.Type(), int64(d.off)}) } case reflect.Bool: v.SetBool(value) case reflect.Interface: if v.NumMethod() == 0 { v.Set(reflect.ValueOf(value)) } else { d.saveError(&UnmarshalTypeError{"bool", v.Type(), int64(d.off)}) } } case '"': // string s, ok := unquoteBytes(item) if !ok { if fromQuoted { d.error(fmt.Errorf("json: invalid use of ,string struct tag, trying to unmarshal %q into %v", item, v.Type())) } else { d.error(errPhase) } } switch v.Kind() { default: d.saveError(&UnmarshalTypeError{"string", v.Type(), int64(d.off)}) case reflect.Slice: if v.Type().Elem().Kind() != reflect.Uint8 { d.saveError(&UnmarshalTypeError{"string", v.Type(), int64(d.off)}) break } b := make([]byte, base64.StdEncoding.DecodedLen(len(s))) n, err := base64.StdEncoding.Decode(b, s) if err != nil { d.saveError(err) break } v.SetBytes(b[:n]) case reflect.String: v.SetString(string(s)) case reflect.Interface: if v.NumMethod() == 0 { v.Set(reflect.ValueOf(string(s))) } else { d.saveError(&UnmarshalTypeError{"string", v.Type(), int64(d.off)}) } } default: // number if c != '-' && (c < '0' || c > '9') { if fromQuoted { d.error(fmt.Errorf("json: invalid use of ,string struct tag, trying to unmarshal %q into %v", item, v.Type())) } else { d.error(errPhase) } } s := string(item) switch v.Kind() { default: if v.Kind() == reflect.String && v.Type() == numberType { v.SetString(s) if !isValidNumber(s) { d.error(fmt.Errorf("json: invalid number literal, trying to unmarshal %q into Number", item)) } break } if fromQuoted { d.error(fmt.Errorf("json: invalid use of ,string struct tag, trying to unmarshal %q into %v", item, v.Type())) } else { d.error(&UnmarshalTypeError{"number", v.Type(), int64(d.off)}) } case reflect.Interface: n, err := d.convertNumber(s) if err != nil { d.saveError(err) break } if v.NumMethod() != 0 { d.saveError(&UnmarshalTypeError{"number", v.Type(), int64(d.off)}) break } v.Set(reflect.ValueOf(n)) case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64: n, err := strconv.ParseInt(s, 10, 64) if err != nil || v.OverflowInt(n) { d.saveError(&UnmarshalTypeError{"number " + s, v.Type(), int64(d.off)}) break } v.SetInt(n) case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Uintptr: n, err := strconv.ParseUint(s, 10, 64) if err != nil || v.OverflowUint(n) { d.saveError(&UnmarshalTypeError{"number " + s, v.Type(), int64(d.off)}) break } v.SetUint(n) case reflect.Float32, reflect.Float64: n, err := strconv.ParseFloat(s, v.Type().Bits()) if err != nil || v.OverflowFloat(n) { d.saveError(&UnmarshalTypeError{"number " + s, v.Type(), int64(d.off)}) break } v.SetFloat(n) } } } // The xxxInterface routines build up a value to be stored // in an empty interface. They are not strictly necessary, // but they avoid the weight of reflection in this common case. // valueInterface is like value but returns interface{} func (d *decodeState) valueInterface() interface{} { switch d.scanWhile(scanSkipSpace) { default: d.error(errPhase) panic("unreachable") case scanBeginArray: return d.arrayInterface() case scanBeginObject: return d.objectInterface() case scanBeginLiteral: return d.literalInterface() case scanBeginName: return d.nameInterface() } } func (d *decodeState) syntaxError(expected string) { msg := fmt.Sprintf("invalid character '%c' looking for %s", d.data[d.off-1], expected) d.error(&SyntaxError{msg, int64(d.off)}) } // arrayInterface is like array but returns []interface{}. func (d *decodeState) arrayInterface() []interface{} { var v = make([]interface{}, 0) for { // Look ahead for ] - can only happen on first iteration. op := d.scanWhile(scanSkipSpace) if op == scanEndArray { if len(v) > 0 && !d.ext.trailingCommas { d.syntaxError("beginning of value") } break } // Back up so d.value can have the byte we just read. d.off-- d.scan.undo(op) v = append(v, d.valueInterface()) // Next token must be , or ]. op = d.scanWhile(scanSkipSpace) if op == scanEndArray { break } if op != scanArrayValue { d.error(errPhase) } } return v } // objectInterface is like object but returns map[string]interface{}. func (d *decodeState) objectInterface() interface{} { v, ok := d.keyed() if ok { return v } m := make(map[string]interface{}) for { // Read opening " of string key or closing }. op := d.scanWhile(scanSkipSpace) if op == scanEndObject { if len(m) > 0 && !d.ext.trailingCommas { d.syntaxError("beginning of object key string") } break } if op == scanBeginName { if !d.ext.unquotedKeys { d.syntaxError("beginning of object key string") } } else if op != scanBeginLiteral { d.error(errPhase) } unquotedKey := op == scanBeginName // Read string key. start := d.off - 1 op = d.scanWhile(scanContinue) item := d.data[start : d.off-1] var key string if unquotedKey { key = string(item) } else { var ok bool key, ok = unquote(item) if !ok { d.error(errPhase) } } // Read : before value. if op == scanSkipSpace { op = d.scanWhile(scanSkipSpace) } if op != scanObjectKey { d.error(errPhase) } // Read value. m[key] = d.valueInterface() // Next token must be , or }. op = d.scanWhile(scanSkipSpace) if op == scanEndObject { break } if op != scanObjectValue { d.error(errPhase) } } return m } // literalInterface is like literal but returns an interface value. func (d *decodeState) literalInterface() interface{} { // All bytes inside literal return scanContinue op code. start := d.off - 1 op := d.scanWhile(scanContinue) // Scan read one byte too far; back up. d.off-- d.scan.undo(op) item := d.data[start:d.off] switch c := item[0]; c { case 'n': // null return nil case 't', 'f': // true, false return c == 't' case '"': // string s, ok := unquote(item) if !ok { d.error(errPhase) } return s default: // number if c != '-' && (c < '0' || c > '9') { d.error(errPhase) } n, err := d.convertNumber(string(item)) if err != nil { d.saveError(err) } return n } } // nameInterface is like function but returns map[string]interface{}. func (d *decodeState) nameInterface() interface{} { v, ok := d.keyed() if ok { return v } nameStart := d.off - 1 op := d.scanWhile(scanContinue) name := d.data[nameStart : d.off-1] if op != scanParam { // Back up so the byte just read is consumed next. d.off-- d.scan.undo(op) if l, ok := d.convertLiteral(name); ok { return l } d.error(&SyntaxError{fmt.Sprintf("json: unknown constant %q", name), int64(d.off)}) } funcName := string(name) funcData := d.ext.funcs[funcName] if funcData.key == "" { d.error(fmt.Errorf("json: unknown function %q", funcName)) } m := make(map[string]interface{}) for i := 0; ; i++ { // Look ahead for ) - can only happen on first iteration. op := d.scanWhile(scanSkipSpace) if op == scanEndParams { break } // Back up so d.value can have the byte we just read. d.off-- d.scan.undo(op) if i >= len(funcData.args) { d.error(fmt.Errorf("json: too many arguments for function %s", funcName)) } m[funcData.args[i]] = d.valueInterface() // Next token must be , or ). op = d.scanWhile(scanSkipSpace) if op == scanEndParams { break } if op != scanParam { d.error(errPhase) } } return map[string]interface{}{funcData.key: m} } // getu4 decodes \uXXXX from the beginning of s, returning the hex value, // or it returns -1. func getu4(s []byte) rune { if len(s) < 6 || s[0] != '\\' || s[1] != 'u' { return -1 } r, err := strconv.ParseUint(string(s[2:6]), 16, 64) if err != nil { return -1 } return rune(r) } // unquote converts a quoted JSON string literal s into an actual string t. // The rules are different than for Go, so cannot use strconv.Unquote. func unquote(s []byte) (t string, ok bool) { s, ok = unquoteBytes(s) t = string(s) return } func unquoteBytes(s []byte) (t []byte, ok bool) { if len(s) < 2 || s[0] != '"' || s[len(s)-1] != '"' { return } s = s[1 : len(s)-1] // Check for unusual characters. If there are none, // then no unquoting is needed, so return a slice of the // original bytes. r := 0 for r < len(s) { c := s[r] if c == '\\' || c == '"' || c < ' ' { break } if c < utf8.RuneSelf { r++ continue } rr, size := utf8.DecodeRune(s[r:]) if rr == utf8.RuneError && size == 1 { break } r += size } if r == len(s) { return s, true } b := make([]byte, len(s)+2*utf8.UTFMax) w := copy(b, s[0:r]) for r < len(s) { // Out of room? Can only happen if s is full of // malformed UTF-8 and we're replacing each // byte with RuneError. if w >= len(b)-2*utf8.UTFMax { nb := make([]byte, (len(b)+utf8.UTFMax)*2) copy(nb, b[0:w]) b = nb } switch c := s[r]; { case c == '\\': r++ if r >= len(s) { return } switch s[r] { default: return case '"', '\\', '/', '\'': b[w] = s[r] r++ w++ case 'b': b[w] = '\b' r++ w++ case 'f': b[w] = '\f' r++ w++ case 'n': b[w] = '\n' r++ w++ case 'r': b[w] = '\r' r++ w++ case 't': b[w] = '\t' r++ w++ case 'u': r-- rr := getu4(s[r:]) if rr < 0 { return } r += 6 if utf16.IsSurrogate(rr) { rr1 := getu4(s[r:]) if dec := utf16.DecodeRune(rr, rr1); dec != unicode.ReplacementChar { // A valid pair; consume. r += 6 w += utf8.EncodeRune(b[w:], dec) break } // Invalid surrogate; fall back to replacement rune. rr = unicode.ReplacementChar } w += utf8.EncodeRune(b[w:], rr) } // Quote, control characters are invalid. case c == '"', c < ' ': return // ASCII case c < utf8.RuneSelf: b[w] = c r++ w++ // Coerce to well-formed UTF-8. default: rr, size := utf8.DecodeRune(s[r:]) r += size w += utf8.EncodeRune(b[w:], rr) } } return b[0:w], true } mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/internal/json/decode_test.go000066400000000000000000001041041276133741700265560ustar00rootroot00000000000000// Copyright 2010 The Go Authors. All rights reserved. // Use of this source code is governed by a BSD-style // license that can be found in the LICENSE file. package json import ( "bytes" "encoding" "errors" "fmt" "image" "net" "reflect" "strings" "testing" "time" ) type T struct { X string Y int Z int `json:"-"` } type U struct { Alphabet string `json:"alpha"` } type V struct { F1 interface{} F2 int32 F3 Number } // ifaceNumAsFloat64/ifaceNumAsNumber are used to test unmarshaling with and // without UseNumber var ifaceNumAsFloat64 = map[string]interface{}{ "k1": float64(1), "k2": "s", "k3": []interface{}{float64(1), float64(2.0), float64(3e-3)}, "k4": map[string]interface{}{"kk1": "s", "kk2": float64(2)}, } var ifaceNumAsNumber = map[string]interface{}{ "k1": Number("1"), "k2": "s", "k3": []interface{}{Number("1"), Number("2.0"), Number("3e-3")}, "k4": map[string]interface{}{"kk1": "s", "kk2": Number("2")}, } type tx struct { x int } // A type that can unmarshal itself. type unmarshaler struct { T bool } func (u *unmarshaler) UnmarshalJSON(b []byte) error { *u = unmarshaler{true} // All we need to see that UnmarshalJSON is called. return nil } type ustruct struct { M unmarshaler } type unmarshalerText struct { A, B string } // needed for re-marshaling tests func (u unmarshalerText) MarshalText() ([]byte, error) { return []byte(u.A + ":" + u.B), nil } func (u *unmarshalerText) UnmarshalText(b []byte) error { pos := bytes.Index(b, []byte(":")) if pos == -1 { return errors.New("missing separator") } u.A, u.B = string(b[:pos]), string(b[pos+1:]) return nil } var _ encoding.TextUnmarshaler = (*unmarshalerText)(nil) type ustructText struct { M unmarshalerText } var ( um0, um1 unmarshaler // target2 of unmarshaling ump = &um1 umtrue = unmarshaler{true} umslice = []unmarshaler{{true}} umslicep = new([]unmarshaler) umstruct = ustruct{unmarshaler{true}} um0T, um1T unmarshalerText // target2 of unmarshaling umpType = &um1T umtrueXY = unmarshalerText{"x", "y"} umsliceXY = []unmarshalerText{{"x", "y"}} umslicepType = new([]unmarshalerText) umstructType = new(ustructText) umstructXY = ustructText{unmarshalerText{"x", "y"}} ummapType = map[unmarshalerText]bool{} ummapXY = map[unmarshalerText]bool{unmarshalerText{"x", "y"}: true} ) // Test data structures for anonymous fields. type Point struct { Z int } type Top struct { Level0 int Embed0 *Embed0a *Embed0b `json:"e,omitempty"` // treated as named Embed0c `json:"-"` // ignored Loop Embed0p // has Point with X, Y, used Embed0q // has Point with Z, used embed // contains exported field } type Embed0 struct { Level1a int // overridden by Embed0a's Level1a with json tag Level1b int // used because Embed0a's Level1b is renamed Level1c int // used because Embed0a's Level1c is ignored Level1d int // annihilated by Embed0a's Level1d Level1e int `json:"x"` // annihilated by Embed0a.Level1e } type Embed0a struct { Level1a int `json:"Level1a,omitempty"` Level1b int `json:"LEVEL1B,omitempty"` Level1c int `json:"-"` Level1d int // annihilated by Embed0's Level1d Level1f int `json:"x"` // annihilated by Embed0's Level1e } type Embed0b Embed0 type Embed0c Embed0 type Embed0p struct { image.Point } type Embed0q struct { Point } type embed struct { Q int } type Loop struct { Loop1 int `json:",omitempty"` Loop2 int `json:",omitempty"` *Loop } // From reflect test: // The X in S6 and S7 annihilate, but they also block the X in S8.S9. type S5 struct { S6 S7 S8 } type S6 struct { X int } type S7 S6 type S8 struct { S9 } type S9 struct { X int Y int } // From reflect test: // The X in S11.S6 and S12.S6 annihilate, but they also block the X in S13.S8.S9. type S10 struct { S11 S12 S13 } type S11 struct { S6 } type S12 struct { S6 } type S13 struct { S8 } type unmarshalTest struct { in string ptr interface{} out interface{} err error useNumber bool } type Ambig struct { // Given "hello", the first match should win. First int `json:"HELLO"` Second int `json:"Hello"` } type XYZ struct { X interface{} Y interface{} Z interface{} } func sliceAddr(x []int) *[]int { return &x } func mapAddr(x map[string]int) *map[string]int { return &x } var unmarshalTests = []unmarshalTest{ // basic types {in: `true`, ptr: new(bool), out: true}, {in: `1`, ptr: new(int), out: 1}, {in: `1.2`, ptr: new(float64), out: 1.2}, {in: `-5`, ptr: new(int16), out: int16(-5)}, {in: `2`, ptr: new(Number), out: Number("2"), useNumber: true}, {in: `2`, ptr: new(Number), out: Number("2")}, {in: `2`, ptr: new(interface{}), out: float64(2.0)}, {in: `2`, ptr: new(interface{}), out: Number("2"), useNumber: true}, {in: `"a\u1234"`, ptr: new(string), out: "a\u1234"}, {in: `"http:\/\/"`, ptr: new(string), out: "http://"}, {in: `"g-clef: \uD834\uDD1E"`, ptr: new(string), out: "g-clef: \U0001D11E"}, {in: `"invalid: \uD834x\uDD1E"`, ptr: new(string), out: "invalid: \uFFFDx\uFFFD"}, {in: "null", ptr: new(interface{}), out: nil}, {in: `{"X": [1,2,3], "Y": 4}`, ptr: new(T), out: T{Y: 4}, err: &UnmarshalTypeError{"array", reflect.TypeOf(""), 7}}, {in: `{"x": 1}`, ptr: new(tx), out: tx{}}, {in: `{"F1":1,"F2":2,"F3":3}`, ptr: new(V), out: V{F1: float64(1), F2: int32(2), F3: Number("3")}}, {in: `{"F1":1,"F2":2,"F3":3}`, ptr: new(V), out: V{F1: Number("1"), F2: int32(2), F3: Number("3")}, useNumber: true}, {in: `{"k1":1,"k2":"s","k3":[1,2.0,3e-3],"k4":{"kk1":"s","kk2":2}}`, ptr: new(interface{}), out: ifaceNumAsFloat64}, {in: `{"k1":1,"k2":"s","k3":[1,2.0,3e-3],"k4":{"kk1":"s","kk2":2}}`, ptr: new(interface{}), out: ifaceNumAsNumber, useNumber: true}, // raw values with whitespace {in: "\n true ", ptr: new(bool), out: true}, {in: "\t 1 ", ptr: new(int), out: 1}, {in: "\r 1.2 ", ptr: new(float64), out: 1.2}, {in: "\t -5 \n", ptr: new(int16), out: int16(-5)}, {in: "\t \"a\\u1234\" \n", ptr: new(string), out: "a\u1234"}, // Z has a "-" tag. {in: `{"Y": 1, "Z": 2}`, ptr: new(T), out: T{Y: 1}}, {in: `{"alpha": "abc", "alphabet": "xyz"}`, ptr: new(U), out: U{Alphabet: "abc"}}, {in: `{"alpha": "abc"}`, ptr: new(U), out: U{Alphabet: "abc"}}, {in: `{"alphabet": "xyz"}`, ptr: new(U), out: U{}}, // syntax errors {in: `{"X": "foo", "Y"}`, err: &SyntaxError{"invalid character '}' after object key", 17}}, {in: `[1, 2, 3+]`, err: &SyntaxError{"invalid character '+' after array element", 9}}, {in: `{"X":12x}`, err: &SyntaxError{"invalid character 'x' after object key:value pair", 8}, useNumber: true}, // raw value errors {in: "\x01 42", err: &SyntaxError{"invalid character '\\x01' looking for beginning of value", 1}}, {in: " 42 \x01", err: &SyntaxError{"invalid character '\\x01' after top-level value", 5}}, {in: "\x01 true", err: &SyntaxError{"invalid character '\\x01' looking for beginning of value", 1}}, {in: " false \x01", err: &SyntaxError{"invalid character '\\x01' after top-level value", 8}}, {in: "\x01 1.2", err: &SyntaxError{"invalid character '\\x01' looking for beginning of value", 1}}, {in: " 3.4 \x01", err: &SyntaxError{"invalid character '\\x01' after top-level value", 6}}, {in: "\x01 \"string\"", err: &SyntaxError{"invalid character '\\x01' looking for beginning of value", 1}}, {in: " \"string\" \x01", err: &SyntaxError{"invalid character '\\x01' after top-level value", 11}}, // array tests {in: `[1, 2, 3]`, ptr: new([3]int), out: [3]int{1, 2, 3}}, {in: `[1, 2, 3]`, ptr: new([1]int), out: [1]int{1}}, {in: `[1, 2, 3]`, ptr: new([5]int), out: [5]int{1, 2, 3, 0, 0}}, // empty array to interface test {in: `[]`, ptr: new([]interface{}), out: []interface{}{}}, {in: `null`, ptr: new([]interface{}), out: []interface{}(nil)}, {in: `{"T":[]}`, ptr: new(map[string]interface{}), out: map[string]interface{}{"T": []interface{}{}}}, {in: `{"T":null}`, ptr: new(map[string]interface{}), out: map[string]interface{}{"T": interface{}(nil)}}, // composite tests {in: allValueIndent, ptr: new(All), out: allValue}, {in: allValueCompact, ptr: new(All), out: allValue}, {in: allValueIndent, ptr: new(*All), out: &allValue}, {in: allValueCompact, ptr: new(*All), out: &allValue}, {in: pallValueIndent, ptr: new(All), out: pallValue}, {in: pallValueCompact, ptr: new(All), out: pallValue}, {in: pallValueIndent, ptr: new(*All), out: &pallValue}, {in: pallValueCompact, ptr: new(*All), out: &pallValue}, // unmarshal interface test {in: `{"T":false}`, ptr: &um0, out: umtrue}, // use "false" so test will fail if custom unmarshaler is not called {in: `{"T":false}`, ptr: &ump, out: &umtrue}, {in: `[{"T":false}]`, ptr: &umslice, out: umslice}, {in: `[{"T":false}]`, ptr: &umslicep, out: &umslice}, {in: `{"M":{"T":"x:y"}}`, ptr: &umstruct, out: umstruct}, // UnmarshalText interface test {in: `"x:y"`, ptr: &um0T, out: umtrueXY}, {in: `"x:y"`, ptr: &umpType, out: &umtrueXY}, {in: `["x:y"]`, ptr: &umsliceXY, out: umsliceXY}, {in: `["x:y"]`, ptr: &umslicepType, out: &umsliceXY}, {in: `{"M":"x:y"}`, ptr: umstructType, out: umstructXY}, // Map keys can be encoding.TextUnmarshalers {in: `{"x:y":true}`, ptr: &ummapType, out: ummapXY}, // If multiple values for the same key exists, only the most recent value is used. {in: `{"x:y":false,"x:y":true}`, ptr: &ummapType, out: ummapXY}, // Overwriting of data. // This is different from package xml, but it's what we've always done. // Now documented and tested. {in: `[2]`, ptr: sliceAddr([]int{1}), out: []int{2}}, {in: `{"key": 2}`, ptr: mapAddr(map[string]int{"old": 0, "key": 1}), out: map[string]int{"key": 2}}, { in: `{ "Level0": 1, "Level1b": 2, "Level1c": 3, "x": 4, "Level1a": 5, "LEVEL1B": 6, "e": { "Level1a": 8, "Level1b": 9, "Level1c": 10, "Level1d": 11, "x": 12 }, "Loop1": 13, "Loop2": 14, "X": 15, "Y": 16, "Z": 17, "Q": 18 }`, ptr: new(Top), out: Top{ Level0: 1, Embed0: Embed0{ Level1b: 2, Level1c: 3, }, Embed0a: &Embed0a{ Level1a: 5, Level1b: 6, }, Embed0b: &Embed0b{ Level1a: 8, Level1b: 9, Level1c: 10, Level1d: 11, Level1e: 12, }, Loop: Loop{ Loop1: 13, Loop2: 14, }, Embed0p: Embed0p{ Point: image.Point{X: 15, Y: 16}, }, Embed0q: Embed0q{ Point: Point{Z: 17}, }, embed: embed{ Q: 18, }, }, }, { in: `{"hello": 1}`, ptr: new(Ambig), out: Ambig{First: 1}, }, { in: `{"X": 1,"Y":2}`, ptr: new(S5), out: S5{S8: S8{S9: S9{Y: 2}}}, }, { in: `{"X": 1,"Y":2}`, ptr: new(S10), out: S10{S13: S13{S8: S8{S9: S9{Y: 2}}}}, }, // invalid UTF-8 is coerced to valid UTF-8. { in: "\"hello\xffworld\"", ptr: new(string), out: "hello\ufffdworld", }, { in: "\"hello\xc2\xc2world\"", ptr: new(string), out: "hello\ufffd\ufffdworld", }, { in: "\"hello\xc2\xffworld\"", ptr: new(string), out: "hello\ufffd\ufffdworld", }, { in: "\"hello\\ud800world\"", ptr: new(string), out: "hello\ufffdworld", }, { in: "\"hello\\ud800\\ud800world\"", ptr: new(string), out: "hello\ufffd\ufffdworld", }, { in: "\"hello\\ud800\\ud800world\"", ptr: new(string), out: "hello\ufffd\ufffdworld", }, { in: "\"hello\xed\xa0\x80\xed\xb0\x80world\"", ptr: new(string), out: "hello\ufffd\ufffd\ufffd\ufffd\ufffd\ufffdworld", }, // Used to be issue 8305, but time.Time implements encoding.TextUnmarshaler so this works now. { in: `{"2009-11-10T23:00:00Z": "hello world"}`, ptr: &map[time.Time]string{}, out: map[time.Time]string{time.Date(2009, 11, 10, 23, 0, 0, 0, time.UTC): "hello world"}, }, // issue 8305 { in: `{"2009-11-10T23:00:00Z": "hello world"}`, ptr: &map[Point]string{}, err: &UnmarshalTypeError{"object", reflect.TypeOf(map[Point]string{}), 1}, }, { in: `{"asdf": "hello world"}`, ptr: &map[unmarshaler]string{}, err: &UnmarshalTypeError{"object", reflect.TypeOf(map[unmarshaler]string{}), 1}, }, } func TestMarshal(t *testing.T) { b, err := Marshal(allValue) if err != nil { t.Fatalf("Marshal allValue: %v", err) } if string(b) != allValueCompact { t.Errorf("Marshal allValueCompact") diff(t, b, []byte(allValueCompact)) return } b, err = Marshal(pallValue) if err != nil { t.Fatalf("Marshal pallValue: %v", err) } if string(b) != pallValueCompact { t.Errorf("Marshal pallValueCompact") diff(t, b, []byte(pallValueCompact)) return } } var badUTF8 = []struct { in, out string }{ {"hello\xffworld", `"hello\ufffdworld"`}, {"", `""`}, {"\xff", `"\ufffd"`}, {"\xff\xff", `"\ufffd\ufffd"`}, {"a\xffb", `"a\ufffdb"`}, {"\xe6\x97\xa5\xe6\x9c\xac\xff\xaa\x9e", `"日本\ufffd\ufffd\ufffd"`}, } func TestMarshalBadUTF8(t *testing.T) { for _, tt := range badUTF8 { b, err := Marshal(tt.in) if string(b) != tt.out || err != nil { t.Errorf("Marshal(%q) = %#q, %v, want %#q, nil", tt.in, b, err, tt.out) } } } func TestMarshalNumberZeroVal(t *testing.T) { var n Number out, err := Marshal(n) if err != nil { t.Fatal(err) } outStr := string(out) if outStr != "0" { t.Fatalf("Invalid zero val for Number: %q", outStr) } } func TestMarshalEmbeds(t *testing.T) { top := &Top{ Level0: 1, Embed0: Embed0{ Level1b: 2, Level1c: 3, }, Embed0a: &Embed0a{ Level1a: 5, Level1b: 6, }, Embed0b: &Embed0b{ Level1a: 8, Level1b: 9, Level1c: 10, Level1d: 11, Level1e: 12, }, Loop: Loop{ Loop1: 13, Loop2: 14, }, Embed0p: Embed0p{ Point: image.Point{X: 15, Y: 16}, }, Embed0q: Embed0q{ Point: Point{Z: 17}, }, embed: embed{ Q: 18, }, } b, err := Marshal(top) if err != nil { t.Fatal(err) } want := "{\"Level0\":1,\"Level1b\":2,\"Level1c\":3,\"Level1a\":5,\"LEVEL1B\":6,\"e\":{\"Level1a\":8,\"Level1b\":9,\"Level1c\":10,\"Level1d\":11,\"x\":12},\"Loop1\":13,\"Loop2\":14,\"X\":15,\"Y\":16,\"Z\":17,\"Q\":18}" if string(b) != want { t.Errorf("Wrong marshal result.\n got: %q\nwant: %q", b, want) } } func TestUnmarshal(t *testing.T) { for i, tt := range unmarshalTests { var scan scanner in := []byte(tt.in) if err := checkValid(in, &scan); err != nil { if !reflect.DeepEqual(err, tt.err) { t.Errorf("#%d: checkValid: %#v", i, err) continue } } if tt.ptr == nil { continue } // v = new(right-type) v := reflect.New(reflect.TypeOf(tt.ptr).Elem()) dec := NewDecoder(bytes.NewReader(in)) if tt.useNumber { dec.UseNumber() } if err := dec.Decode(v.Interface()); !reflect.DeepEqual(err, tt.err) { t.Errorf("#%d: %v, want %v", i, err, tt.err) continue } else if err != nil { continue } if !reflect.DeepEqual(v.Elem().Interface(), tt.out) { t.Errorf("#%d: mismatch\nhave: %#+v\nwant: %#+v", i, v.Elem().Interface(), tt.out) data, _ := Marshal(v.Elem().Interface()) println(string(data)) data, _ = Marshal(tt.out) println(string(data)) continue } // Check round trip. if tt.err == nil { enc, err := Marshal(v.Interface()) if err != nil { t.Errorf("#%d: error re-marshaling: %v", i, err) continue } vv := reflect.New(reflect.TypeOf(tt.ptr).Elem()) dec = NewDecoder(bytes.NewReader(enc)) if tt.useNumber { dec.UseNumber() } if err := dec.Decode(vv.Interface()); err != nil { t.Errorf("#%d: error re-unmarshaling %#q: %v", i, enc, err) continue } if !reflect.DeepEqual(v.Elem().Interface(), vv.Elem().Interface()) { t.Errorf("#%d: mismatch\nhave: %#+v\nwant: %#+v", i, v.Elem().Interface(), vv.Elem().Interface()) t.Errorf(" In: %q", strings.Map(noSpace, string(in))) t.Errorf("Marshal: %q", strings.Map(noSpace, string(enc))) continue } } } } func TestUnmarshalMarshal(t *testing.T) { initBig() var v interface{} if err := Unmarshal(jsonBig, &v); err != nil { t.Fatalf("Unmarshal: %v", err) } b, err := Marshal(v) if err != nil { t.Fatalf("Marshal: %v", err) } if !bytes.Equal(jsonBig, b) { t.Errorf("Marshal jsonBig") diff(t, b, jsonBig) return } } var numberTests = []struct { in string i int64 intErr string f float64 floatErr string }{ {in: "-1.23e1", intErr: "strconv.ParseInt: parsing \"-1.23e1\": invalid syntax", f: -1.23e1}, {in: "-12", i: -12, f: -12.0}, {in: "1e1000", intErr: "strconv.ParseInt: parsing \"1e1000\": invalid syntax", floatErr: "strconv.ParseFloat: parsing \"1e1000\": value out of range"}, } // Independent of Decode, basic coverage of the accessors in Number func TestNumberAccessors(t *testing.T) { for _, tt := range numberTests { n := Number(tt.in) if s := n.String(); s != tt.in { t.Errorf("Number(%q).String() is %q", tt.in, s) } if i, err := n.Int64(); err == nil && tt.intErr == "" && i != tt.i { t.Errorf("Number(%q).Int64() is %d", tt.in, i) } else if (err == nil && tt.intErr != "") || (err != nil && err.Error() != tt.intErr) { t.Errorf("Number(%q).Int64() wanted error %q but got: %v", tt.in, tt.intErr, err) } if f, err := n.Float64(); err == nil && tt.floatErr == "" && f != tt.f { t.Errorf("Number(%q).Float64() is %g", tt.in, f) } else if (err == nil && tt.floatErr != "") || (err != nil && err.Error() != tt.floatErr) { t.Errorf("Number(%q).Float64() wanted error %q but got: %v", tt.in, tt.floatErr, err) } } } func TestLargeByteSlice(t *testing.T) { s0 := make([]byte, 2000) for i := range s0 { s0[i] = byte(i) } b, err := Marshal(s0) if err != nil { t.Fatalf("Marshal: %v", err) } var s1 []byte if err := Unmarshal(b, &s1); err != nil { t.Fatalf("Unmarshal: %v", err) } if !bytes.Equal(s0, s1) { t.Errorf("Marshal large byte slice") diff(t, s0, s1) } } type Xint struct { X int } func TestUnmarshalInterface(t *testing.T) { var xint Xint var i interface{} = &xint if err := Unmarshal([]byte(`{"X":1}`), &i); err != nil { t.Fatalf("Unmarshal: %v", err) } if xint.X != 1 { t.Fatalf("Did not write to xint") } } func TestUnmarshalPtrPtr(t *testing.T) { var xint Xint pxint := &xint if err := Unmarshal([]byte(`{"X":1}`), &pxint); err != nil { t.Fatalf("Unmarshal: %v", err) } if xint.X != 1 { t.Fatalf("Did not write to xint") } } func TestEscape(t *testing.T) { const input = `"foobar"` + " [\u2028 \u2029]" const expected = `"\"foobar\"\u003chtml\u003e [\u2028 \u2029]"` b, err := Marshal(input) if err != nil { t.Fatalf("Marshal error: %v", err) } if s := string(b); s != expected { t.Errorf("Encoding of [%s]:\n got [%s]\nwant [%s]", input, s, expected) } } // WrongString is a struct that's misusing the ,string modifier. type WrongString struct { Message string `json:"result,string"` } type wrongStringTest struct { in, err string } var wrongStringTests = []wrongStringTest{ {`{"result":"x"}`, `json: invalid use of ,string struct tag, trying to unmarshal "x" into string`}, {`{"result":"foo"}`, `json: invalid use of ,string struct tag, trying to unmarshal "foo" into string`}, {`{"result":"123"}`, `json: invalid use of ,string struct tag, trying to unmarshal "123" into string`}, {`{"result":123}`, `json: invalid use of ,string struct tag, trying to unmarshal unquoted value into string`}, } // If people misuse the ,string modifier, the error message should be // helpful, telling the user that they're doing it wrong. func TestErrorMessageFromMisusedString(t *testing.T) { for n, tt := range wrongStringTests { r := strings.NewReader(tt.in) var s WrongString err := NewDecoder(r).Decode(&s) got := fmt.Sprintf("%v", err) if got != tt.err { t.Errorf("%d. got err = %q, want %q", n, got, tt.err) } } } func noSpace(c rune) rune { if isSpace(byte(c)) { //only used for ascii return -1 } return c } type All struct { Bool bool Int int Int8 int8 Int16 int16 Int32 int32 Int64 int64 Uint uint Uint8 uint8 Uint16 uint16 Uint32 uint32 Uint64 uint64 Uintptr uintptr Float32 float32 Float64 float64 Foo string `json:"bar"` Foo2 string `json:"bar2,dummyopt"` IntStr int64 `json:",string"` PBool *bool PInt *int PInt8 *int8 PInt16 *int16 PInt32 *int32 PInt64 *int64 PUint *uint PUint8 *uint8 PUint16 *uint16 PUint32 *uint32 PUint64 *uint64 PUintptr *uintptr PFloat32 *float32 PFloat64 *float64 String string PString *string Map map[string]Small MapP map[string]*Small PMap *map[string]Small PMapP *map[string]*Small EmptyMap map[string]Small NilMap map[string]Small Slice []Small SliceP []*Small PSlice *[]Small PSliceP *[]*Small EmptySlice []Small NilSlice []Small StringSlice []string ByteSlice []byte Small Small PSmall *Small PPSmall **Small Interface interface{} PInterface *interface{} unexported int } type Small struct { Tag string } var allValue = All{ Bool: true, Int: 2, Int8: 3, Int16: 4, Int32: 5, Int64: 6, Uint: 7, Uint8: 8, Uint16: 9, Uint32: 10, Uint64: 11, Uintptr: 12, Float32: 14.1, Float64: 15.1, Foo: "foo", Foo2: "foo2", IntStr: 42, String: "16", Map: map[string]Small{ "17": {Tag: "tag17"}, "18": {Tag: "tag18"}, }, MapP: map[string]*Small{ "19": {Tag: "tag19"}, "20": nil, }, EmptyMap: map[string]Small{}, Slice: []Small{{Tag: "tag20"}, {Tag: "tag21"}}, SliceP: []*Small{{Tag: "tag22"}, nil, {Tag: "tag23"}}, EmptySlice: []Small{}, StringSlice: []string{"str24", "str25", "str26"}, ByteSlice: []byte{27, 28, 29}, Small: Small{Tag: "tag30"}, PSmall: &Small{Tag: "tag31"}, Interface: 5.2, } var pallValue = All{ PBool: &allValue.Bool, PInt: &allValue.Int, PInt8: &allValue.Int8, PInt16: &allValue.Int16, PInt32: &allValue.Int32, PInt64: &allValue.Int64, PUint: &allValue.Uint, PUint8: &allValue.Uint8, PUint16: &allValue.Uint16, PUint32: &allValue.Uint32, PUint64: &allValue.Uint64, PUintptr: &allValue.Uintptr, PFloat32: &allValue.Float32, PFloat64: &allValue.Float64, PString: &allValue.String, PMap: &allValue.Map, PMapP: &allValue.MapP, PSlice: &allValue.Slice, PSliceP: &allValue.SliceP, PPSmall: &allValue.PSmall, PInterface: &allValue.Interface, } var allValueIndent = `{ "Bool": true, "Int": 2, "Int8": 3, "Int16": 4, "Int32": 5, "Int64": 6, "Uint": 7, "Uint8": 8, "Uint16": 9, "Uint32": 10, "Uint64": 11, "Uintptr": 12, "Float32": 14.1, "Float64": 15.1, "bar": "foo", "bar2": "foo2", "IntStr": "42", "PBool": null, "PInt": null, "PInt8": null, "PInt16": null, "PInt32": null, "PInt64": null, "PUint": null, "PUint8": null, "PUint16": null, "PUint32": null, "PUint64": null, "PUintptr": null, "PFloat32": null, "PFloat64": null, "String": "16", "PString": null, "Map": { "17": { "Tag": "tag17" }, "18": { "Tag": "tag18" } }, "MapP": { "19": { "Tag": "tag19" }, "20": null }, "PMap": null, "PMapP": null, "EmptyMap": {}, "NilMap": null, "Slice": [ { "Tag": "tag20" }, { "Tag": "tag21" } ], "SliceP": [ { "Tag": "tag22" }, null, { "Tag": "tag23" } ], "PSlice": null, "PSliceP": null, "EmptySlice": [], "NilSlice": null, "StringSlice": [ "str24", "str25", "str26" ], "ByteSlice": "Gxwd", "Small": { "Tag": "tag30" }, "PSmall": { "Tag": "tag31" }, "PPSmall": null, "Interface": 5.2, "PInterface": null }` var allValueCompact = strings.Map(noSpace, allValueIndent) var pallValueIndent = `{ "Bool": false, "Int": 0, "Int8": 0, "Int16": 0, "Int32": 0, "Int64": 0, "Uint": 0, "Uint8": 0, "Uint16": 0, "Uint32": 0, "Uint64": 0, "Uintptr": 0, "Float32": 0, "Float64": 0, "bar": "", "bar2": "", "IntStr": "0", "PBool": true, "PInt": 2, "PInt8": 3, "PInt16": 4, "PInt32": 5, "PInt64": 6, "PUint": 7, "PUint8": 8, "PUint16": 9, "PUint32": 10, "PUint64": 11, "PUintptr": 12, "PFloat32": 14.1, "PFloat64": 15.1, "String": "", "PString": "16", "Map": null, "MapP": null, "PMap": { "17": { "Tag": "tag17" }, "18": { "Tag": "tag18" } }, "PMapP": { "19": { "Tag": "tag19" }, "20": null }, "EmptyMap": null, "NilMap": null, "Slice": null, "SliceP": null, "PSlice": [ { "Tag": "tag20" }, { "Tag": "tag21" } ], "PSliceP": [ { "Tag": "tag22" }, null, { "Tag": "tag23" } ], "EmptySlice": null, "NilSlice": null, "StringSlice": null, "ByteSlice": null, "Small": { "Tag": "" }, "PSmall": null, "PPSmall": { "Tag": "tag31" }, "Interface": null, "PInterface": 5.2 }` var pallValueCompact = strings.Map(noSpace, pallValueIndent) func TestRefUnmarshal(t *testing.T) { type S struct { // Ref is defined in encode_test.go. R0 Ref R1 *Ref R2 RefText R3 *RefText } want := S{ R0: 12, R1: new(Ref), R2: 13, R3: new(RefText), } *want.R1 = 12 *want.R3 = 13 var got S if err := Unmarshal([]byte(`{"R0":"ref","R1":"ref","R2":"ref","R3":"ref"}`), &got); err != nil { t.Fatalf("Unmarshal: %v", err) } if !reflect.DeepEqual(got, want) { t.Errorf("got %+v, want %+v", got, want) } } // Test that the empty string doesn't panic decoding when ,string is specified // Issue 3450 func TestEmptyString(t *testing.T) { type T2 struct { Number1 int `json:",string"` Number2 int `json:",string"` } data := `{"Number1":"1", "Number2":""}` dec := NewDecoder(strings.NewReader(data)) var t2 T2 err := dec.Decode(&t2) if err == nil { t.Fatal("Decode: did not return error") } if t2.Number1 != 1 { t.Fatal("Decode: did not set Number1") } } // Test that a null for ,string is not replaced with the previous quoted string (issue 7046). // It should also not be an error (issue 2540, issue 8587). func TestNullString(t *testing.T) { type T struct { A int `json:",string"` B int `json:",string"` C *int `json:",string"` } data := []byte(`{"A": "1", "B": null, "C": null}`) var s T s.B = 1 s.C = new(int) *s.C = 2 err := Unmarshal(data, &s) if err != nil { t.Fatalf("Unmarshal: %v", err) } if s.B != 1 || s.C != nil { t.Fatalf("after Unmarshal, s.B=%d, s.C=%p, want 1, nil", s.B, s.C) } } func intp(x int) *int { p := new(int) *p = x return p } func intpp(x *int) **int { pp := new(*int) *pp = x return pp } var interfaceSetTests = []struct { pre interface{} json string post interface{} }{ {"foo", `"bar"`, "bar"}, {"foo", `2`, 2.0}, {"foo", `true`, true}, {"foo", `null`, nil}, {nil, `null`, nil}, {new(int), `null`, nil}, {(*int)(nil), `null`, nil}, {new(*int), `null`, new(*int)}, {(**int)(nil), `null`, nil}, {intp(1), `null`, nil}, {intpp(nil), `null`, intpp(nil)}, {intpp(intp(1)), `null`, intpp(nil)}, } func TestInterfaceSet(t *testing.T) { for _, tt := range interfaceSetTests { b := struct{ X interface{} }{tt.pre} blob := `{"X":` + tt.json + `}` if err := Unmarshal([]byte(blob), &b); err != nil { t.Errorf("Unmarshal %#q: %v", blob, err) continue } if !reflect.DeepEqual(b.X, tt.post) { t.Errorf("Unmarshal %#q into %#v: X=%#v, want %#v", blob, tt.pre, b.X, tt.post) } } } // JSON null values should be ignored for primitives and string values instead of resulting in an error. // Issue 2540 func TestUnmarshalNulls(t *testing.T) { jsonData := []byte(`{ "Bool" : null, "Int" : null, "Int8" : null, "Int16" : null, "Int32" : null, "Int64" : null, "Uint" : null, "Uint8" : null, "Uint16" : null, "Uint32" : null, "Uint64" : null, "Float32" : null, "Float64" : null, "String" : null}`) nulls := All{ Bool: true, Int: 2, Int8: 3, Int16: 4, Int32: 5, Int64: 6, Uint: 7, Uint8: 8, Uint16: 9, Uint32: 10, Uint64: 11, Float32: 12.1, Float64: 13.1, String: "14"} err := Unmarshal(jsonData, &nulls) if err != nil { t.Errorf("Unmarshal of null values failed: %v", err) } if !nulls.Bool || nulls.Int != 2 || nulls.Int8 != 3 || nulls.Int16 != 4 || nulls.Int32 != 5 || nulls.Int64 != 6 || nulls.Uint != 7 || nulls.Uint8 != 8 || nulls.Uint16 != 9 || nulls.Uint32 != 10 || nulls.Uint64 != 11 || nulls.Float32 != 12.1 || nulls.Float64 != 13.1 || nulls.String != "14" { t.Errorf("Unmarshal of null values affected primitives") } } func TestStringKind(t *testing.T) { type stringKind string var m1, m2 map[stringKind]int m1 = map[stringKind]int{ "foo": 42, } data, err := Marshal(m1) if err != nil { t.Errorf("Unexpected error marshaling: %v", err) } err = Unmarshal(data, &m2) if err != nil { t.Errorf("Unexpected error unmarshaling: %v", err) } if !reflect.DeepEqual(m1, m2) { t.Error("Items should be equal after encoding and then decoding") } } // Custom types with []byte as underlying type could not be marshalled // and then unmarshalled. // Issue 8962. func TestByteKind(t *testing.T) { type byteKind []byte a := byteKind("hello") data, err := Marshal(a) if err != nil { t.Error(err) } var b byteKind err = Unmarshal(data, &b) if err != nil { t.Fatal(err) } if !reflect.DeepEqual(a, b) { t.Errorf("expected %v == %v", a, b) } } // The fix for issue 8962 introduced a regression. // Issue 12921. func TestSliceOfCustomByte(t *testing.T) { type Uint8 uint8 a := []Uint8("hello") data, err := Marshal(a) if err != nil { t.Fatal(err) } var b []Uint8 err = Unmarshal(data, &b) if err != nil { t.Fatal(err) } if !reflect.DeepEqual(a, b) { t.Fatalf("expected %v == %v", a, b) } } var decodeTypeErrorTests = []struct { dest interface{} src string }{ {new(string), `{"user": "name"}`}, // issue 4628. {new(error), `{}`}, // issue 4222 {new(error), `[]`}, {new(error), `""`}, {new(error), `123`}, {new(error), `true`}, } func TestUnmarshalTypeError(t *testing.T) { for _, item := range decodeTypeErrorTests { err := Unmarshal([]byte(item.src), item.dest) if _, ok := err.(*UnmarshalTypeError); !ok { t.Errorf("expected type error for Unmarshal(%q, type %T): got %T", item.src, item.dest, err) } } } var unmarshalSyntaxTests = []string{ "tru", "fals", "nul", "123e", `"hello`, `[1,2,3`, `{"key":1`, `{"key":1,`, } func TestUnmarshalSyntax(t *testing.T) { var x interface{} for _, src := range unmarshalSyntaxTests { err := Unmarshal([]byte(src), &x) if _, ok := err.(*SyntaxError); !ok { t.Errorf("expected syntax error for Unmarshal(%q): got %T", src, err) } } } // Test handling of unexported fields that should be ignored. // Issue 4660 type unexportedFields struct { Name string m map[string]interface{} `json:"-"` m2 map[string]interface{} `json:"abcd"` } func TestUnmarshalUnexported(t *testing.T) { input := `{"Name": "Bob", "m": {"x": 123}, "m2": {"y": 456}, "abcd": {"z": 789}}` want := &unexportedFields{Name: "Bob"} out := &unexportedFields{} err := Unmarshal([]byte(input), out) if err != nil { t.Errorf("got error %v, expected nil", err) } if !reflect.DeepEqual(out, want) { t.Errorf("got %q, want %q", out, want) } } // Time3339 is a time.Time which encodes to and from JSON // as an RFC 3339 time in UTC. type Time3339 time.Time func (t *Time3339) UnmarshalJSON(b []byte) error { if len(b) < 2 || b[0] != '"' || b[len(b)-1] != '"' { return fmt.Errorf("types: failed to unmarshal non-string value %q as an RFC 3339 time", b) } tm, err := time.Parse(time.RFC3339, string(b[1:len(b)-1])) if err != nil { return err } *t = Time3339(tm) return nil } func TestUnmarshalJSONLiteralError(t *testing.T) { var t3 Time3339 err := Unmarshal([]byte(`"0000-00-00T00:00:00Z"`), &t3) if err == nil { t.Fatalf("expected error; got time %v", time.Time(t3)) } if !strings.Contains(err.Error(), "range") { t.Errorf("got err = %v; want out of range error", err) } } // Test that extra object elements in an array do not result in a // "data changing underfoot" error. // Issue 3717 func TestSkipArrayObjects(t *testing.T) { json := `[{}]` var dest [0]interface{} err := Unmarshal([]byte(json), &dest) if err != nil { t.Errorf("got error %q, want nil", err) } } // Test semantics of pre-filled struct fields and pre-filled map fields. // Issue 4900. func TestPrefilled(t *testing.T) { ptrToMap := func(m map[string]interface{}) *map[string]interface{} { return &m } // Values here change, cannot reuse table across runs. var prefillTests = []struct { in string ptr interface{} out interface{} }{ { in: `{"X": 1, "Y": 2}`, ptr: &XYZ{X: float32(3), Y: int16(4), Z: 1.5}, out: &XYZ{X: float64(1), Y: float64(2), Z: 1.5}, }, { in: `{"X": 1, "Y": 2}`, ptr: ptrToMap(map[string]interface{}{"X": float32(3), "Y": int16(4), "Z": 1.5}), out: ptrToMap(map[string]interface{}{"X": float64(1), "Y": float64(2), "Z": 1.5}), }, } for _, tt := range prefillTests { ptrstr := fmt.Sprintf("%v", tt.ptr) err := Unmarshal([]byte(tt.in), tt.ptr) // tt.ptr edited here if err != nil { t.Errorf("Unmarshal: %v", err) } if !reflect.DeepEqual(tt.ptr, tt.out) { t.Errorf("Unmarshal(%#q, %s): have %v, want %v", tt.in, ptrstr, tt.ptr, tt.out) } } } var invalidUnmarshalTests = []struct { v interface{} want string }{ {nil, "json: Unmarshal(nil)"}, {struct{}{}, "json: Unmarshal(non-pointer struct {})"}, {(*int)(nil), "json: Unmarshal(nil *int)"}, } func TestInvalidUnmarshal(t *testing.T) { buf := []byte(`{"a":"1"}`) for _, tt := range invalidUnmarshalTests { err := Unmarshal(buf, tt.v) if err == nil { t.Errorf("Unmarshal expecting error, got nil") continue } if got := err.Error(); got != tt.want { t.Errorf("Unmarshal = %q; want %q", got, tt.want) } } } var invalidUnmarshalTextTests = []struct { v interface{} want string }{ {nil, "json: Unmarshal(nil)"}, {struct{}{}, "json: Unmarshal(non-pointer struct {})"}, {(*int)(nil), "json: Unmarshal(nil *int)"}, {new(net.IP), "json: cannot unmarshal string into Go value of type *net.IP"}, } func TestInvalidUnmarshalText(t *testing.T) { buf := []byte(`123`) for _, tt := range invalidUnmarshalTextTests { err := Unmarshal(buf, tt.v) if err == nil { t.Errorf("Unmarshal expecting error, got nil") continue } if got := err.Error(); got != tt.want { t.Errorf("Unmarshal = %q; want %q", got, tt.want) } } } // Test that string option is ignored for invalid types. // Issue 9812. func TestInvalidStringOption(t *testing.T) { num := 0 item := struct { T time.Time `json:",string"` M map[string]string `json:",string"` S []string `json:",string"` A [1]string `json:",string"` I interface{} `json:",string"` P *int `json:",string"` }{M: make(map[string]string), S: make([]string, 0), I: num, P: &num} data, err := Marshal(item) if err != nil { t.Fatalf("Marshal: %v", err) } err = Unmarshal(data, &item) if err != nil { t.Fatalf("Unmarshal: %v", err) } } mongo-tools-r3.2.11/vendor/src/gopkg.in/mgo.v2/internal/json/encode.go000066400000000000000000001010111276133741700255230ustar00rootroot00000000000000// Copyright 2010 The Go Authors. All rights reserved. // Use of this source code is governed by a BSD-style // license that can be found in the LICENSE file. // Package json implements encoding and decoding of JSON as defined in // RFC 4627. The mapping between JSON and Go values is described // in the documentation for the Marshal and Unmarshal functions. // // See "JSON and Go" for an introduction to this package: // https://golang.org/doc/articles/json_and_go.html package json import ( "bytes" "encoding" "encoding/base64" "fmt" "math" "reflect" "runtime" "sort" "strconv" "strings" "sync" "unicode" "unicode/utf8" ) // Marshal returns the JSON encoding of v. // // Marshal traverses the value v recursively. // If an encountered value implements the Marshaler interface // and is not a nil pointer, Marshal calls its MarshalJSON method // to produce JSON. If no MarshalJSON method is present but the // value implements encoding.TextMarshaler instead, Marshal calls // its MarshalText method. // The nil pointer exception is not strictly necessary // but mimics a similar, necessary exception in the behavior of // UnmarshalJSON. // // Otherwise, Marshal uses the following type-dependent default encodings: // // Boolean values encode as JSON booleans. // // Floating point, integer, and Number values encode as JSON numbers. // // String values encode as JSON strings coerced to valid UTF-8, // replacing invalid bytes with the Unicode replacement rune. // The angle brackets "<" and ">" are escaped to "\u003c" and "\u003e" // to keep some browsers from misinterpreting JSON output as HTML. // Ampersand "&" is also escaped to "\u0026" for the same reason. // This escaping can be disabled using an Encoder with DisableHTMLEscaping. // // Array and slice values encode as JSON arrays, except that // []byte encodes as a base64-encoded string, and a nil slice // encodes as the null JSON value. // // Struct values encode as JSON objects. Each exported struct field // becomes a member of the object unless // - the field's tag is "-", or // - the field is empty and its tag specifies the "omitempty" option. // The empty values are false, 0, any // nil pointer or interface value, and any array, slice, map, or string of // length zero. The object's default key string is the struct field name // but can be specified in the struct field's tag value. The "json" key in // the struct field's tag value is the key name, followed by an optional comma // and options. Examples: // // // Field is ignored by this package. // Field int `json:"-"` // // // Field appears in JSON as key "myName". // Field int `json:"myName"` // // // Field appears in JSON as key "myName" and // // the field is omitted from the object if its value is empty, // // as defined above. // Field int `json:"myName,omitempty"` // // // Field appears in JSON as key "Field" (the default), but // // the field is skipped if empty. // // Note the leading comma. // Field int `json:",omitempty"` // // The "string" option signals that a field is stored as JSON inside a // JSON-encoded string. It applies only to fields of string, floating point, // integer, or boolean types. This extra level of encoding is sometimes used // when communicating with JavaScript programs: // // Int64String int64 `json:",string"` // // The key name will be used if it's a non-empty string consisting of // only Unicode letters, digits, dollar signs, percent signs, hyphens, // underscores and slashes. // // Anonymous struct fields are usually marshaled as if their inner exported fields // were fields in the outer struct, subject to the usual Go visibility rules amended // as described in the next paragraph. // An anonymous struct field with a name given in its JSON tag is treated as // having that name, rather than being anonymous. // An anonymous struct field of interface type is treated the same as having // that type as its name, rather than being anonymous. // // The Go visibility rules for struct fields are amended for JSON when // deciding which field to marshal or unmarshal. If there are // multiple fields at the same level, and that level is the least // nested (and would therefore be the nesting level selected by the // usual Go rules), the following extra rules apply: // // 1) Of those fields, if any are JSON-tagged, only tagged fields are considered, // even if there are multiple untagged fields that would otherwise conflict. // 2) If there is exactly one field (tagged or not according to the first rule), that is selected. // 3) Otherwise there are multiple fields, and all are ignored; no error occurs. // // Handling of anonymous struct fields is new in Go 1.1. // Prior to Go 1.1, anonymous struct fields were ignored. To force ignoring of // an anonymous struct field in both current and earlier versions, give the field // a JSON tag of "-". // // Map values encode as JSON objects. The map's key type must either be a string // or implement encoding.TextMarshaler. The map keys are used as JSON object // keys, subject to the UTF-8 coercion described for string values above. // // Pointer values encode as the value pointed to. // A nil pointer encodes as the null JSON value. // // Interface values encode as the value contained in the interface. // A nil interface value encodes as the null JSON value. // // Channel, complex, and function values cannot be encoded in JSON. // Attempting to encode such a value causes Marshal to return // an UnsupportedTypeError. // // JSON cannot represent cyclic data structures and Marshal does not // handle them. Passing cyclic structures to Marshal will result in // an infinite recursion. // func Marshal(v interface{}) ([]byte, error) { e := &encodeState{} err := e.marshal(v, encOpts{escapeHTML: true}) if err != nil { return nil, err } return e.Bytes(), nil } // MarshalIndent is like Marshal but applies Indent to format the output. func MarshalIndent(v interface{}, prefix, indent string) ([]byte, error) { b, err := Marshal(v) if err != nil { return nil, err } var buf bytes.Buffer err = Indent(&buf, b, prefix, indent) if err != nil { return nil, err } return buf.Bytes(), nil } // HTMLEscape appends to dst the JSON-encoded src with <, >, &, U+2028 and U+2029 // characters inside string literals changed to \u003c, \u003e, \u0026, \u2028, \u2029 // so that the JSON will be safe to embed inside HTML